diff --git a/CHANGELOG.md b/CHANGELOG.md index ec019fb..70e3730 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,12 @@ - Play tweet videos from Parasitter. - Create following lists. +## [0.2.3] - 2020-09-04 +### Added +- [x] Youtube: Proxy all images through Yotter. +- [x] General: Add server config file. +- [x] General: @Sn0wed1 added a Docker file and Docker installation instructions. + ## [0.2.2] - 2020-08-27 ### Changed - [x] Twitter: Scrap nitter pages instead of using RSS. diff --git a/README.md b/README.md index 1a6078b..68b7625 100644 --- a/README.md +++ b/README.md @@ -15,13 +15,16 @@ Yotter is possible thanks to several open-source projects that are listed on the * [Why](#why) * [Features](#features) * [Screenshots](#screenshots) -* [Privacy and Security](#-privacy) -* [Self hosting](#-self-hosting) - * [Install & Test](#-test) - * [Hosting on a server](#-hosting-on-a-server) - * [Update](#-updating-to-newer-versions) -* [Powered by](#-powered-by) -* [Donate](#-donate) +* [Privacy and Security](#privacy) +* [Self hosting](#self-hosting) + * Install & Test + * [Normal installation](#test) + * [Docker installation](#using-docker) + * [Hosting on a server](#hosting-on-a-server) + * [Update](#updating-to-newer-versions) + * [Configure server](configure-the-server) +* [Powered by](#powered-by) +* [Donate](#donate) ## Why At first I started working on this project as a solution for following Twitter accounts (a thing that can't be done with Nitter) and getting a Twitter-like feed. Weeks later the leader of Invidious, Omar Roth, announced that he was stepping away from the project. As an Invidious active user, this made me think that a new alternative was needed for the community and also an alternative with an easier language for most people (as Invidious is written in Crystal). So I started developing a 'written-in-python Invidious alternative' and it went quite well. @@ -42,15 +45,15 @@ I hope that this project can prosperate, gain contributors, new instances and cr > And many more to come! -## 🎭 Privacy -#### 🌐 Connections +## Privacy +#### Connections Yotter cares about your privacy, and for this it will never make any connection to Twitter or Youtube on the client. Every request is proxied through the Yotter server; video streaming, photos, data gathering, scrapping, etc. The Yotter server connects to Google (Youtube) and Nitter in order to gather all the necessary data. Then it serves it (proxyed through itself) to the client. This means that as a client, you will never connect to Google - the Yotter server will do it for you. So if you want to set up a Yotter server for privacy reasons I recommend you to set it up on a remote VPS so you don't share your IP with Google or use a VPN on the server. If you don't mind exposing your IP making requests to Google then you can set it up wherever you want. Even with this method you will **avoid all trackers, ads, heavy-loaded pages, etc**. - Even with this method, you can stay safe if you use a VPN to hide your IP. -#### 🛡️ Your data +#### Your data The only things the database stores are: * Hash of the password * Username @@ -60,7 +63,7 @@ The only things the database stores are: This data will never be used for any other purpose than offering the service to the user. It's not sent anywhere, never. -#### 🔐 Security +#### Security Only the hash of your password is stored on the database. Also, no personal information of any kind is required nor kept, if a hacker gets access to the database the only thing they could do would be to follow/unfollow some accounts. So there's no motivation in 'hacking' Yotter. I always recommend self-hosting, as you will be the only person with access to the data. @@ -70,9 +73,9 @@ I always recommend self-hosting, as you will be the only person with access to t #### Others If you want to use a specific Nitter instance you can replace it on the file `app/routes.py`. -## 🏠 Self hosting +## Self hosting -### 🐣 Test +### Test You can test this new version. ##### IMPORTANT: Connections to googlevideo will be made to stream the videos. It is recommended to use a VPS server or a VPN to preserve your privacy. This version is intended for a remote server. @@ -131,9 +134,10 @@ A quick deployment 6. Go to "http://localhost:5000/" and enjoy. ### 🔗 Hosting on a server: -`SOON` -### 🐓 Updating to newer versions: +#### [VISIT THIS FILE FOR INSTRUCTIONS](https://github.com/pluja/Yotter/blob/dev-indep/SELF-HOSTING.md) + +### Updating to newer versions: **IMPORTANT: Before updating to newer versions, always export your data on `Settings>Export Data`. A major version update could have changes on the whole database and you may be forced to remove and reset the database (only when running locally)!** 1. Navigate to the git repository (the one you cloned when installing). @@ -153,7 +157,14 @@ A quick deployment 6. Done! You are on latest version. > **See [CHANGELOG](CHANGELOG.md) for a list of changes.** -### ⛽ Powered by: +### Configure the server +You will find in the root folder of the project a file named `yotter-config.json`. This is the global config file for the Yotter server. + +Currently available config is: +* **nitterInstance**: Nitter instance that will be used when fetching Twitter content. Format must be `**https://****/**` +* **maxInstanceUsers**: Max users on the instance. When set to `0` it closes registrations. + +## Powered by: * [Nitter](https://nitter.net/) * [youtube-dl](https://github.com/ytdl-org/youtube-dl) * [Flask](https://flask.palletsprojects.com/) @@ -164,7 +175,7 @@ A quick deployment * [Video.js](https://videojs.com/) * [My fork of youtube_search](https://github.com/pluja/youtube_search-fork) -### 💌 Donate +## Donate This project is completely free and Open Source and will always be. Funding will be used 100% for opening and mantaining an online public instance of Yotter, this will be hosted on Netcup and will (at first) be the *VPS 500 G8*. I mention all of this in case you want to check the prices. @@ -174,7 +185,7 @@ Funding will be used 100% for opening and mantaining an online public instance o #### Fiat: - Donate using Liberapay -## 🖼️ Screenshots +## Screenshots

diff --git a/SELF-HOSTING.md b/SELF-HOSTING.md new file mode 100644 index 0000000..2a43c9f --- /dev/null +++ b/SELF-HOSTING.md @@ -0,0 +1,102 @@ +

UNDER CONSTRUCTION

+Installation Working +
+Tested on Ubuntu + +#### Step 1: Base setup +1. Connect to your server via SSH or direct access. + * (Recommended) Set up password-less login with ssh-keys. + +2. Install base dependencies: +* `sudo apt-get -y update` + +* `sudo apt-get -y install python3 python3-venv python3-dev` + +* `sudo apt-get -y install mysql-server supervisor nginx git make` + +> When installing MySQL-server it will prompt for a root password. Set up a password of your like, this will be the MySQL databases master password and will be required later, so don't forget it! + +3. Clone this repository and acccess folder: +* `git clone https://github.com/pluja/Yotter` + +* `cd Yotter` + +4. Create a Python virtual environment and populate it with dependencies: +* `python3 -m venv venv` +* `source venv/bin/activate` + +* `pip install -r requirements.txt` + +> You can edit the `yotter-config` file + +5. Install gunicorn (production web server for Python apps) and pymysql: +`pip install gunicorn pymysql` + +6. Set up `.env` + 1. (PRE) Generate a random string and copy it to clipboard: + `python3 -c "import uuid; print(uuid.uuid4().hex)"` + + 2. Create a `.env` file on the root folder of the project (`/home/ubuntu/Yotter/.env`): + ``` + SECRET_KEY= + DATABASE_URL=mysql+pymysql://yotter:@localhost:3306/yotter + ``` + +#### Step 2: Setting up the MySQL Database: +* Open the MySQL prompt line (Use the previously set MySQL root password!) + `mysql -u root -p` + +Now you should be on the MySQL prompt line (`mysql>`). So let's create the databases: + +> Change `` for a password of your like. It will be the password for the dabase user `yotter`. Don't choose the same password as the root user of MySQL for security. + +> The password for the **yotter** user needs to match the password that you included in the `DATABASE_URL` variable in the `.env` file. If you didn't change it, you can change it now. + +``` +mysql> create database yotter character set utf8 collate utf8_bin; +mysql> create user 'yotter'@'localhost' identified by ''; +mysql> grant all privileges on yotter.* to 'yotter'@'localhost'; +mysql> flush privileges; +mysql> quit; +``` + +If your set up was correct, you should now be able to run: + +`flask db init` +`flask db migrate` + +#### Step 3: Setting up Gunicorn and Supervisor +When you run the server with flask run, you are using a web server that comes with Flask. This server is very useful during development, but it isn't a good choice to use for a production server because it wasn't built with performance and robustness in mind. Instead of the Flask development server, for this deployment I decided to use gunicorn, which is also a pure Python web server, but unlike Flask's, it is a robust production server that is used by a lot of people, while at the same time it is very easy to use. [ref](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xvii-deployment-on-linux) + +* Start yotter under Gunicorn: + +`gunicorn -b localhost:8000 -w 4 yotter:app` + +The supervisor utility uses configuration files that tell it what programs to monitor and how to restart them when necessary. Configuration files must be stored in /etc/supervisor/conf.d. Here is a configuration file for Yotter, which I'm going to call yotter.conf [ref](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xvii-deployment-on-linux). + +* Create a yotter.conf file on `/etc/supervisor/conf.d/`: + +> You can run `nano /etc/supervisor/conf.d/yotter.conf` and paste the text below: + +> Make sure to fit any path and user to your system. + +``` +[program:yotter] +command=/home/ubuntu/yotter/venv/bin/gunicorn -b localhost:8000 -w 4 yotter:app +directory=/home/ubuntu/yotter +user=ubuntu +autostart=true +autorestart=true +stopasgroup=true +killasgroup=true +``` + +After you write this configuration file, you have to reload the supervisor service for it to be imported: +`sudo supervisorctl reload` + +#### Step 4: Set up Nginx +The Yotter application server powered by gunicorn is now running privately port 8000. Now we need to expose the application to the outside world by enabling public facing web server on ports 80 and 443, the two ports too need to be opened on the firewall to handle the web traffic of the application. I want this to be a secure deployment, so I'm going to configure port 80 to forward all traffic to port 443, which is going to be encrypted. [ref](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xvii-deployment-on-linux). + +* `sudo rm /etc/nginx/sites-enabled/default` + +Create a new Nginx site, you can run `sudo nano /etc/nginx/sites-enabled/yotter` diff --git a/app/routes.py b/app/routes.py index f514312..0f249ac 100644 --- a/app/routes.py +++ b/app/routes.py @@ -12,6 +12,7 @@ from werkzeug.urls import url_parse from youtube_dl import YoutubeDL from numerize import numerize from bs4 import BeautifulSoup +from xml.dom import minidom from app import app, db from re import findall import random, string @@ -22,18 +23,21 @@ import bleach import urllib import json import re - ########################## #### Config variables #### ########################## -NITTERINSTANCE = "https://nitter.net/" # Must be https://.../ +config = json.load(open('yotter-config.json')) +########################## +#### Config variables #### +########################## +NITTERINSTANCE = config['nitterInstance'] # Must be https://.../ YOUTUBERSS = "https://www.youtube.com/feeds/videos.xml?channel_id=" -REGISTRATIONS = True +REGISTRATIONS = config['registrations'] ########################## #### Global variables #### ########################## -ALLOWED_EXTENSIONS = {'json'} +ALLOWED_EXTENSIONS = {'json', 'db'} ######################### #### Twitter Logic ###### @@ -372,6 +376,13 @@ def login(): return redirect(next_page) return render_template('login.html', title='Sign In', form=form) +#Proxy images through server +@app.route('/img/', methods=['GET', 'POST']) +@login_required +def img(url): + pic = requests.get(url.replace("~", "/")) + return Response(pic,mimetype="image/png") + @app.route('/logout') def logout(): logout_user() @@ -430,13 +441,21 @@ def importdata(): if file.filename == '': flash('No selected file') return redirect(request.referrer) - if file and allowed_file(file.filename): - importAccounts(file) + if file and allowed_file(file.filename) or 'subscription_manager' in file.filename: + option = request.form['import_format'] + if option == 'yotter': + importYotterSubscriptions(file) + elif option == 'newpipe': + importNewPipeSubscriptions(file) + elif option == 'youtube': + importYoutubeSubscriptions(file) + elif option == 'freetube': + importFreeTubeSubscriptions(file) return redirect(request.referrer) return redirect(request.referrer) -def importAccounts(file): +def importYotterSubscriptions(file): filename = secure_filename(file.filename) data = json.load(file) for acc in data['twitter']: @@ -445,20 +464,44 @@ def importAccounts(file): for acc in data['youtube']: r = followYoutubeChannel(acc['channelId']) +def importNewPipeSubscriptions(file): + filename = secure_filename(file.filename) + data = json.load(file) + for acc in data['subscriptions']: + r = followYoutubeChannel(re.search('(UC[a-zA-Z0-9_-]{22})|(?<=user\/)[a-zA-Z0-9_-]+', acc['url']).group()) + +def importYoutubeSubscriptions(file): + filename = secure_filename(file.filename) + itemlist = minidom.parse(file).getElementsByTagName('outline') + for item in itemlist[1:]: + r = followYoutubeChannel(re.search('UC[a-zA-Z0-9_-]{22}', item.attributes['xmlUrl'].value).group()) + +def importFreeTubeSubscriptions(file): + filename = secure_filename(file.filename) + data = re.findall('UC[a-zA-Z0-9_-]{22}', file.read().decode('utf-8')) + for acc in data: + r = followYoutubeChannel(acc) + def allowed_file(filename): return '.' in filename and filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIONS @app.route('/register', methods=['GET', 'POST']) def register(): + global REGISTRATIONS + count = db.session.query(User).count() if current_user.is_authenticated: return redirect(url_for('index')) + if count >= config['maxInstanceUsers']: + REGISTRATIONS = False + form = RegistrationForm() if form.validate_on_submit(): if User.query.filter_by(username=form.username.data).first(): flash("This username is taken! Try with another.") return redirect(request.referrer) + user = User(username=form.username.data) user.set_password(form.password.data) db.session.add(user) @@ -553,7 +596,7 @@ def getFeed(urls): for post in userFeed[:-1]: date_time_str = post.find('span', attrs={'class':'tweet-date'}).find('a')['title'].replace(",","") time = datetime.datetime.now() - datetime.datetime.strptime(date_time_str, '%d/%m/%Y %H:%M:%S') - if time.days >=8: + if time.days >=7: continue if post.find('div', attrs={'class':'pinned'}): @@ -608,8 +651,6 @@ def getPosts(account): for post in userFeed[:-1]: date_time_str = post.find('span', attrs={'class':'tweet-date'}).find('a')['title'].replace(",","") time = datetime.datetime.now() - datetime.datetime.strptime(date_time_str, '%d/%m/%Y %H:%M:%S') - if time.days >=8: - continue if post.find('div', attrs={'class':'pinned'}): if post.find('div', attrs={'class':'pinned'}).find('span', attrs={'icon-pin'}): @@ -660,7 +701,7 @@ def getYoutubePosts(ids): for vid in rssFeed.entries: time = datetime.datetime.now() - datetime.datetime(*vid.published_parsed[:6]) - if time.days >=8: + if time.days >=7: continue video = ytPost() @@ -671,9 +712,9 @@ def getYoutubePosts(ids): video.channelUrl = vid.author_detail.href video.id = vid.yt_videoid video.videoTitle = vid.title - video.videoThumb = vid.media_thumbnail[0]['url'] + video.videoThumb = vid.media_thumbnail[0]['url'].replace('/', '~') video.views = vid.media_statistics['views'] video.description = vid.summary_detail.value video.description = re.sub(r'^https?:\/\/.*[\r\n]*', '', video.description[0:120]+"...", flags=re.MULTILINE) videos.append(video) - return videos \ No newline at end of file + return videos diff --git a/app/templates/_closed_registrations.html b/app/templates/_closed_registrations.html index b29d202..367d594 100644 --- a/app/templates/_closed_registrations.html +++ b/app/templates/_closed_registrations.html @@ -1,6 +1,6 @@
- Closed registrations image + Closed registrations image
diff --git a/app/templates/_video_item.html b/app/templates/_video_item.html index e53696b..d48ef07 100644 --- a/app/templates/_video_item.html +++ b/app/templates/_video_item.html @@ -1,6 +1,6 @@
- Thumbnail + Thumbnail
{{video.videoTitle}} diff --git a/app/templates/base.html b/app/templates/base.html index fcc675b..d8a7afd 100644 --- a/app/templates/base.html +++ b/app/templates/base.html @@ -2,7 +2,7 @@ - + {% if title %} {{ title }} - Parassiter {% else %} diff --git a/app/templates/channel.html b/app/templates/channel.html index 171880f..a4f3c5b 100644 --- a/app/templates/channel.html +++ b/app/templates/channel.html @@ -4,7 +4,7 @@
- Avatar + Avatar
{{channel.name}}
diff --git a/app/templates/settings.html b/app/templates/settings.html index 70e02b3..bac76a0 100644 --- a/app/templates/settings.html +++ b/app/templates/settings.html @@ -40,8 +40,21 @@
-
-
Import from JSON export.
+
+ + + + + +
Import subscription data.
diff --git a/app/templates/ytsearch.html b/app/templates/ytsearch.html index 8af14a0..a7f7e23 100644 --- a/app/templates/ytsearch.html +++ b/app/templates/ytsearch.html @@ -58,8 +58,6 @@ {% include '_video_item.html' %} {% endfor %}
- {% else %} - {% include '_empty_feed.html' %} {% endif %}
diff --git a/yotter-config.json b/yotter-config.json new file mode 100644 index 0000000..bf9d1d0 --- /dev/null +++ b/yotter-config.json @@ -0,0 +1,4 @@ +{ + "nitterInstance": "https://nitter.net/", + "maxInstanceUsers": 1 +} \ No newline at end of file