Merge pull request #1 from pluja/dev-indep

Rebase

CHANGELOG.md

@@ -9,6 +9,12 @@
 - Play tweet videos from Parasitter.
 - Create following lists.
 
+## [0.2.3] - 2020-09-04
+### Added
+- [x] Youtube: Proxy all images through Yotter.
+- [x] General: Add server config file.
+- [x] General: @Sn0wed1 added a Docker file and Docker installation instructions.
+
 ## [0.2.2] - 2020-08-27
 ### Changed
 - [x] Twitter: Scrap nitter pages instead of using RSS.

README.md

@@ -15,13 +15,16 @@ Yotter is possible thanks to several open-source projects that are listed on the
 * [Why](#why)
 * [Features](#features)
 * [Screenshots](#screenshots)
-* [Privacy and Security](#-privacy)
+* [Privacy and Security](#privacy)
-* [Self hosting](#-self-hosting)
+* [Self hosting](#self-hosting)
-    * [Install & Test](#-test)
+    * Install & Test
-    * [Hosting on a server](#-hosting-on-a-server)
+        * [Normal installation](#test)
-    * [Update](#-updating-to-newer-versions)
+        * [Docker installation](#using-docker)
-* [Powered by](#-powered-by)
+    * [Hosting on a server](#hosting-on-a-server)
-* [Donate](#-donate)
+    * [Update](#updating-to-newer-versions)
+    * [Configure server](configure-the-server)
+* [Powered by](#powered-by)
+* [Donate](#donate)
 
 ## Why
 At first I started working on this project as a solution for following Twitter accounts (a thing that can't be done with Nitter) and getting a Twitter-like feed. Weeks later the leader of Invidious, Omar Roth, announced that he was stepping away from the project. As an Invidious active user, this made me think that a new alternative was needed for the community and also an alternative with an easier language for most people (as Invidious is written in Crystal). So I started developing a 'written-in-python Invidious alternative' and it went quite well.
@@ -42,15 +45,15 @@ I hope that this project can prosperate, gain contributors, new instances and cr
 
 > And many more to come!
 
-## 🎭 Privacy
+## Privacy
-#### 🌐 Connections
+#### Connections
 Yotter cares about your privacy, and for this it will never make any connection to Twitter or Youtube on the client. Every request is proxied through the Yotter server; video streaming, photos, data gathering, scrapping, etc.
 
 The Yotter server connects to Google (Youtube) and Nitter in order to gather all the necessary data. Then it serves it (proxyed through itself) to the client. This means that as a client, you will never connect to Google - the Yotter server will do it for you. So if you want to set up a Yotter server for privacy reasons I recommend you to set it up on a remote VPS so you don't share your IP with Google or use a VPN on the server. 
 
 If you don't mind exposing your IP making requests to Google then you can set it up wherever you want. Even with this method you will **avoid all trackers, ads, heavy-loaded pages, etc**. - Even with this method, you can stay safe if you use a VPN to hide your IP.
 
-#### 🛡️ Your data
+#### Your data
 The only things the database stores are:
 * Hash of the password
 * Username
@@ -60,7 +63,7 @@ The only things the database stores are:
 
 This data will never be used for any other purpose than offering the service to the user. It's not sent anywhere, never.
 
-#### 🔐 Security
+#### Security
 Only the hash of your password is stored on the database. Also, no personal information of any kind is required nor kept, if a hacker gets access to the database the only thing they could do would be to follow/unfollow some accounts. So there's no motivation in 'hacking' Yotter.
 
 I always recommend self-hosting, as you will be the only person with access to the data.
@@ -70,9 +73,9 @@ I always recommend self-hosting, as you will be the only person with access to t
 #### Others
 If you want to use a specific Nitter instance you can replace it on the file `app/routes.py`.
 
-## 🏠 Self hosting
+## Self hosting
 
-### 🐣 Test
+### Test
 You can test this new version.
 
 ##### IMPORTANT: Connections to googlevideo will be made to stream the videos. It is recommended to use a VPS server or a VPN to preserve your privacy. This version is intended for a remote server.
@@ -131,9 +134,10 @@ A quick deployment
 6. Go to "http://localhost:5000/" and enjoy.
 
 ### 🔗 Hosting on a server:
-`SOON`
 
-### 🐓 Updating to newer versions:
+#### [VISIT THIS FILE FOR INSTRUCTIONS](https://github.com/pluja/Yotter/blob/dev-indep/SELF-HOSTING.md)
+
+### Updating to newer versions:
 **IMPORTANT: Before updating to newer versions, always export your data on `Settings>Export Data`. A major version update could have changes on the whole database and you may be forced to remove and reset the database (only when running locally)!**
 
 1. Navigate to the git repository (the one you cloned when installing).
@@ -153,7 +157,14 @@ A quick deployment
 6. Done! You are on latest version.
 > **See [CHANGELOG](CHANGELOG.md) for a list of changes.**
 
-### ⛽ Powered by:
+### Configure the server
+You will find in the root folder of the project a file named `yotter-config.json`. This is the global config file for the Yotter server.
+
+Currently available config is:
+* **nitterInstance**: Nitter instance that will be used when fetching Twitter content. Format must be `**https://**<NitterInstance.tld>**/**`
+* **maxInstanceUsers**: Max users on the instance. When set to `0` it closes registrations.
+
+## Powered by:
 * [Nitter](https://nitter.net/)
 * [youtube-dl](https://github.com/ytdl-org/youtube-dl)
 * [Flask](https://flask.palletsprojects.com/)
@@ -164,7 +175,7 @@ A quick deployment
 * [Video.js](https://videojs.com/)
 * [My fork of youtube_search](https://github.com/pluja/youtube_search-fork)
 
-### 💌 Donate
+## Donate
 This project is completely free and Open Source and will always be.
 
 Funding will be used 100% for opening and mantaining an online public instance of Yotter, this will be hosted on Netcup and will (at first) be the *VPS 500 G8*. I mention all of this in case you want to check the prices.
@@ -174,7 +185,7 @@ Funding will be used 100% for opening and mantaining an online public instance o
 #### Fiat:
 - <a href="https://liberapay.com/pluja/donate"><img alt="Donate using Liberapay" src="https://liberapay.com/assets/widgets/donate.svg"></a>
 
-## 🖼️ Screenshots
+## Screenshots
 <p align="center"> <img width="720" src="https://i.imgur.com/6AfXO57.png"> </img></p> 
 <p align="center"> <img width="720" src="https://i.imgur.com/jipjySH.png"> </img></p> 
 <p align="center"> <img width="720" src="https://i.imgur.com/JMUW6VH.png"> </img></p> 

SELF-HOSTING.md

@@ -0,0 +1,102 @@
+<h1> UNDER CONSTRUCTION </h1>
+<a href="https://github.com/pluja/Yotter/tree/master"><img alt="Installation Working" src="https://img.shields.io/badge/Working-2020.09.04-green.svg"></img></a>
+<br>
+<a href="https://github.com/pluja/Yotter/tree/master"><img alt="Tested on Ubuntu" src="https://img.shields.io/badge/Tested On-Ubuntu 20.04LTS-blue.svg"></img></a>
+
+#### Step 1: Base setup
+1. Connect to your server via SSH or direct access.
+    * (Recommended) Set up password-less login with ssh-keys.
+
+2. Install base dependencies:
+* `sudo apt-get -y update`
+
+* `sudo apt-get -y install python3 python3-venv python3-dev`
+
+* `sudo apt-get -y install mysql-server supervisor nginx git make`
+
+> When installing MySQL-server it will prompt for a root password. Set up a password of your like, this will be the MySQL databases master password and will be required later, so don't forget it!
+
+3. Clone this repository and acccess folder:
+* `git clone https://github.com/pluja/Yotter`
+
+* `cd Yotter`
+
+4. Create a Python virtual environment and populate it with dependencies:
+* `python3 -m venv venv`
+* `source venv/bin/activate`
+
+* `pip install -r requirements.txt`
+
+> You can edit the `yotter-config` file
+
+5. Install gunicorn (production web server for Python apps) and pymysql:
+`pip install gunicorn pymysql`
+
+6. Set up `.env`
+    1. (PRE) Generate a random string and copy it to clipboard:
+        `python3 -c "import uuid; print(uuid.uuid4().hex)"`
+
+    2. Create a `.env` file on the root folder of the project (`/home/ubuntu/Yotter/.env`):
+        ```
+        SECRET_KEY=<RandomStringHere!>
+        DATABASE_URL=mysql+pymysql://yotter:<db-password>@localhost:3306/yotter
+        ```
+
+#### Step 2: Setting up the MySQL Database:
+* Open the MySQL prompt line (Use the previously set MySQL root password!)
+    `mysql -u root -p`
+
+Now you should be on the MySQL prompt line (`mysql>`). So let's create the databases:
+
+> Change `<db-password>` for a password of your like. It will be the password for the dabase user `yotter`. Don't choose the same password as the root user of MySQL for security.
+
+> The password for the **yotter** user needs to match the password that you included in the `DATABASE_URL` variable in the `.env` file. If you didn't change it, you can change it now.
+
+```
+mysql> create database yotter character set utf8 collate utf8_bin;
+mysql> create user 'yotter'@'localhost' identified by '<db-password>';
+mysql> grant all privileges on yotter.* to 'yotter'@'localhost';
+mysql> flush privileges;
+mysql> quit;
+```
+
+If your set up was correct, you should now be able to run:
+
+`flask db init`
+`flask db migrate`
+
+#### Step 3: Setting up Gunicorn and Supervisor
+When you run the server with flask run, you are using a web server that comes with Flask. This server is very useful during development, but it isn't a good choice to use for a production server because it wasn't built with performance and robustness in mind. Instead of the Flask development server, for this deployment I decided to use gunicorn, which is also a pure Python web server, but unlike Flask's, it is a robust production server that is used by a lot of people, while at the same time it is very easy to use. [ref](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xvii-deployment-on-linux)
+
+* Start yotter under Gunicorn:
+
+`gunicorn -b localhost:8000 -w 4 yotter:app`
+
+The supervisor utility uses configuration files that tell it what programs to monitor and how to restart them when necessary. Configuration files must be stored in /etc/supervisor/conf.d. Here is a configuration file for Yotter, which I'm going to call yotter.conf [ref](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xvii-deployment-on-linux).
+
+* Create a yotter.conf file on `/etc/supervisor/conf.d/`:
+
+> You can run `nano /etc/supervisor/conf.d/yotter.conf` and paste the text below:
+
+> Make sure to fit any path and user to your system.
+
+```
+[program:yotter]
+command=/home/ubuntu/yotter/venv/bin/gunicorn -b localhost:8000 -w 4 yotter:app
+directory=/home/ubuntu/yotter
+user=ubuntu
+autostart=true
+autorestart=true
+stopasgroup=true
+killasgroup=true
+```
+
+After you write this configuration file, you have to reload the supervisor service for it to be imported:
+`sudo supervisorctl reload`
+
+#### Step 4: Set up Nginx
+The Yotter application server powered by gunicorn is now running privately port 8000. Now we need to expose the application to the outside world by enabling public facing web server on ports 80 and 443, the two ports too need to be opened on the firewall to handle the web traffic of the application. I want this to be a secure deployment, so I'm going to configure port 80 to forward all traffic to port 443, which is going to be encrypted. [ref](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xvii-deployment-on-linux).
+
+* `sudo rm /etc/nginx/sites-enabled/default`
+
+Create a new Nginx site, you can run `sudo nano /etc/nginx/sites-enabled/yotter`

app/routes.py

@@ -12,6 +12,7 @@ from werkzeug.urls import url_parse
 from youtube_dl import YoutubeDL
 from numerize import numerize
 from bs4 import BeautifulSoup
+from xml.dom import minidom
 from app import app, db
 from re import findall
 import random, string
@@ -22,18 +23,21 @@ import bleach
 import urllib
 import json
 import re
-
 ##########################
 #### Config variables ####
 ##########################
-NITTERINSTANCE = "https://nitter.net/" # Must be https://.../ 
+config = json.load(open('yotter-config.json'))
+##########################
+#### Config variables ####
+##########################
+NITTERINSTANCE = config['nitterInstance'] # Must be https://.../ 
 YOUTUBERSS = "https://www.youtube.com/feeds/videos.xml?channel_id="
-REGISTRATIONS = True
+REGISTRATIONS = config['registrations']
 
 ##########################
 #### Global variables ####
 ##########################
-ALLOWED_EXTENSIONS = {'json'}
+ALLOWED_EXTENSIONS = {'json', 'db'}
 
 #########################
 #### Twitter Logic ######
@@ -372,6 +376,13 @@ def login():
         return redirect(next_page)
     return render_template('login.html', title='Sign In', form=form)
 
+#Proxy images through server
+@app.route('/img/<url>', methods=['GET', 'POST'])
+@login_required
+def img(url):
+    pic = requests.get(url.replace("~", "/"))
+    return Response(pic,mimetype="image/png")
+
 @app.route('/logout')
 def logout():
     logout_user()
@@ -430,13 +441,21 @@ def importdata():
         if file.filename == '':
             flash('No selected file')
             return redirect(request.referrer)
-        if file and allowed_file(file.filename):
+        if file and allowed_file(file.filename) or 'subscription_manager' in file.filename:
-            importAccounts(file)
+            option = request.form['import_format']
+            if option == 'yotter':
+                importYotterSubscriptions(file)
+            elif option == 'newpipe':
+                importNewPipeSubscriptions(file)
+            elif option == 'youtube':
+                importYoutubeSubscriptions(file)
+            elif option == 'freetube':
+                importFreeTubeSubscriptions(file)
             return redirect(request.referrer)
 
     return redirect(request.referrer)
 
-def importAccounts(file):
+def importYotterSubscriptions(file):
     filename = secure_filename(file.filename)
     data = json.load(file)
     for acc in data['twitter']:
@@ -445,20 +464,44 @@ def importAccounts(file):
     for acc in data['youtube']:
         r = followYoutubeChannel(acc['channelId'])
 
+def importNewPipeSubscriptions(file):
+    filename = secure_filename(file.filename)
+    data = json.load(file)
+    for acc in data['subscriptions']:
+        r = followYoutubeChannel(re.search('(UC[a-zA-Z0-9_-]{22})|(?<=user\/)[a-zA-Z0-9_-]+', acc['url']).group())
+
+def importYoutubeSubscriptions(file):
+    filename = secure_filename(file.filename)
+    itemlist = minidom.parse(file).getElementsByTagName('outline')
+    for item in itemlist[1:]:
+        r = followYoutubeChannel(re.search('UC[a-zA-Z0-9_-]{22}', item.attributes['xmlUrl'].value).group())
+
+def importFreeTubeSubscriptions(file):
+    filename = secure_filename(file.filename)
+    data = re.findall('UC[a-zA-Z0-9_-]{22}', file.read().decode('utf-8'))
+    for acc in data:
+        r = followYoutubeChannel(acc)
+
 def allowed_file(filename):
     return '.' in filename and filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIONS
 
 @app.route('/register', methods=['GET', 'POST'])
 def register():
+    global REGISTRATIONS
+    count = db.session.query(User).count()
     if current_user.is_authenticated:
         return redirect(url_for('index'))
 
+    if count >= config['maxInstanceUsers']:
+        REGISTRATIONS = False
+        
     form = RegistrationForm()
     if form.validate_on_submit():
         if User.query.filter_by(username=form.username.data).first():
             flash("This username is taken! Try with another.")
             return redirect(request.referrer)
 
+
         user = User(username=form.username.data)
         user.set_password(form.password.data)
         db.session.add(user)
@@ -553,7 +596,7 @@ def getFeed(urls):
                     for post in userFeed[:-1]:
                         date_time_str = post.find('span', attrs={'class':'tweet-date'}).find('a')['title'].replace(",","")
                         time = datetime.datetime.now() - datetime.datetime.strptime(date_time_str, '%d/%m/%Y %H:%M:%S')
-                        if time.days >=8:
+                        if time.days >=7:
                             continue
 
                         if post.find('div', attrs={'class':'pinned'}):
@@ -608,8 +651,6 @@ def getPosts(account):
             for post in userFeed[:-1]:
                 date_time_str = post.find('span', attrs={'class':'tweet-date'}).find('a')['title'].replace(",","")
                 time = datetime.datetime.now() - datetime.datetime.strptime(date_time_str, '%d/%m/%Y %H:%M:%S')
-                if time.days >=8:
-                    continue
 
                 if post.find('div', attrs={'class':'pinned'}):
                     if post.find('div', attrs={'class':'pinned'}).find('span', attrs={'icon-pin'}):
@@ -660,7 +701,7 @@ def getYoutubePosts(ids):
             for vid in rssFeed.entries:
                 time = datetime.datetime.now() - datetime.datetime(*vid.published_parsed[:6])
 
-                if time.days >=8:
+                if time.days >=7:
                     continue
                 
                 video = ytPost()
@@ -671,9 +712,9 @@ def getYoutubePosts(ids):
                 video.channelUrl = vid.author_detail.href
                 video.id = vid.yt_videoid
                 video.videoTitle = vid.title
-                video.videoThumb = vid.media_thumbnail[0]['url']
+                video.videoThumb = vid.media_thumbnail[0]['url'].replace('/', '~')
                 video.views = vid.media_statistics['views']
                 video.description = vid.summary_detail.value
                 video.description = re.sub(r'^https?:\/\/.*[\r\n]*', '', video.description[0:120]+"...", flags=re.MULTILINE)
                 videos.append(video)
-    return videos
+    return videos

app/templates/_closed_registrations.html

@@ -1,6 +1,6 @@
 <div class="ui center aligned text container">
     <div class="ui row">
-        <img alt="Closed registrations image" class="ui image" src="{{ url_for('static',filename='img/closed.png') }}">
+        <img alt="Closed registrations image" class="ui centered medium image" src="{{ url_for('static',filename='img/closed.png') }}">
     </div>
 
     <div class="ui row">

app/templates/_video_item.html

@@ -1,6 +1,6 @@
 <div class="card">
     <div class="image">
-        <img alt="Thumbnail" src="{{video.videoThumb}}">
+        <img alt="Thumbnail" src="/img/{{video.videoThumb.replace('/', '~')}}">
     </div>
     <div class="content">
         <a class="video-title break-word" href="{{url_for('watch', v=video.id, _method='GET')}}">{{video.videoTitle}}</a>

app/templates/base.html

@@ -2,7 +2,7 @@
 <html lang="en">
     <head>
         <meta charset="UTF-8">
-        <meta name="viewport" content="width=device-width, initial-scale=1.0">
+        <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
         {% if title %}
         <title>{{ title }} - Parassiter</title>
         {% else %}

app/templates/channel.html

@@ -4,7 +4,7 @@
     <div class="blue ui centered card">
         <div class="content">
             <div class="center aligned author">
-                <img alt="Avatar" class="ui avatar image" src="{{channel.avatar}}">
+                <img alt="Avatar" class="ui avatar image" src="/img/{{channel.avatar.replace('/', '~')}}">
             </div>
             <div class="center aligned header"><a href="">{{channel.name}}</a></div>
             <div class="center aligned description">

app/templates/settings.html

@@ -40,8 +40,21 @@
                 <form action = "{{ url_for('importdata') }}" method = "POST" enctype = "multipart/form-data">
                     <input type = "file" name = "file"/>
                     <input type = "submit"/>
-                </form> 
+                    <br>
-                <div class="description">Import from JSON export.</div>
+                    <label class="radio-inline">
+                        <input type="radio" name="import_format" id="yotter" value="yotter" checked> Yotter
+                    </label>
+                    <label class="radio-inline">
+                        <input type="radio" name="import_format" id="newpipe" value="newpipe"> NewPipe
+                    </label>
+                    <label class="radio-inline">
+                        <input type="radio" name="import_format" id="youtube" value="youtube"> Youtube
+                    </label>
+                    <label class="radio-inline">
+                        <input type="radio" name="import_format" id="freetube" value="freetube"> FreeTube
+                    </label>
+                </form>
+                <div class="description">Import subscription data.</div>
             </div>
         </div>
 

app/templates/ytsearch.html

@@ -58,8 +58,6 @@
                             {% include '_video_item.html' %}
                         {% endfor %}
                     </div>
-                {% else %}
-                    {% include '_empty_feed.html' %}
                 {% endif %}
             </div>
 </div>

yotter-config.json

@@ -0,0 +1,4 @@
+{
+	"nitterInstance": "https://nitter.net/",
+	"maxInstanceUsers": 1
+}