Commit Graph

1781 Commits

Author SHA1 Message Date
Stypox
3708ab9ed5
[YouTube] Refactor YoutubeDashManifestCreator
- Remove all of the methods used to access caches and replace them with three caches getters
- Rename caches to shorter and more meaningful names
- Remove redundant @throws tags that just say "if this method fails to do what it should do", which is obvious
2022-05-28 12:00:51 +02:00
Stypox
5c83409039
[YouTube] Rewrite manifest test and rename long methods 2022-05-28 12:00:51 +02:00
Stypox
8226fd044f
[YouTube] Suppress Sonar security warning for XEE 2022-05-28 12:00:50 +02:00
Stypox
ba68b8c014
[YouTube] Secure DashManifestCreator against XEE attack
Also remove duplicate lines from javadoc comments, otherwise checkstyle complaints.
XEE prevention is inspired from https://github.com/ChuckerTeam/chucker/pull/201/files
For an explanation of XEE see https://rules.sonarsource.com/java/RSPEC-2755, though the solution reported there causes crashes on Android
2022-05-28 12:00:50 +02:00
Stypox
159d05c91b
Remove unused DashMpdParser (but kept in git history) 2022-05-28 12:00:49 +02:00
Stypox
50272db946
Apply reviews: improve comments, remove FILE, remove Stream#equals(Stream) 2022-05-28 12:00:49 +02:00
TiA4f8R
07b045f20d
[YouTube] Support the iOS client in YoutubeDashManifestCreator and decrypt again the n parameter, if present, for all clients
This commits reverts a new behavior introduced in this branch, which only applied the decryption if needed on streams from the WEB client.
Also fix rebase issues and documentations style in YoutubeDashManifestCreator.
2022-05-28 12:00:48 +02:00
TiA4f8R
436ddde29f
Use assertThrows in YoutubeDashManifestCreatorTest 2022-05-28 12:00:48 +02:00
TiA4f8R
d64d7bbd01
Move ManifestCreatorCache tests to a separate class and remove override of equals and hashCode methods in ManifestCreatorCache
These methods don't need to be overriden, as they are not excepted to be used in collections.
Also improve the toString method of this class, which contains also now clearFactor and maximumSize attributes and for each operations.
2022-05-28 12:00:48 +02:00
TiA4f8R
2fb1a412a6
Fix Checkstyle issues, revert resolution string changes for YouTube video streams and don't return the rn parameter in DASH manifests
This parameter is still used to get the initialization sequence of OTF and POST-live streams, but is not returned anymore in the manifests.
It has been removed in order to avoid fingerprinting based on the number sent (e.g. when starting to play a stream close to the end and using 123 as the request number where it should be 1) and should be added dynamically by clients in their requests.
The relevant test has been also updated.

Checkstyle issues in YoutubeDashManifestCreator have been fixed, and the changes in the resolution string returned for video streams in YoutubeStreamExtractor have been reverted, as they create issues on NewPipe right now.
2022-05-28 12:00:47 +02:00
TiA4f8R
f61e2092a1
[YouTube] Return a copy of the hardcoded ItagItem instead of returning the reference to the hardcoded one in ItagItem.getItag
To do so, a copy constructor has been added in the class.

This fixes, for instance, an issue in NewPipe, in which the ItagItem values where not the ones corresponsing to a stream but to another, when generating DASH manifests.
2022-05-28 12:00:47 +02:00
TiA4f8R
aa4c10e751
Improve documentation and adress most of the requested changes
Also fix some issues in several places, in the code and the documentation.
2022-05-28 12:00:46 +02:00
TiA4f8R
6985167e63
Add tests for YoutubeDashManifestCreator and ManifestCreatorCache
The test added in YoutubeDashManifestCreator uses a video of the Creative Commons channel, licenced under the Creative Commons Attribution licence (reuse allowed).
Also remove public keywords of tests in UtilsTest, as suggested by SonarLint, because they are not needed with Junit 5.
2022-05-28 12:00:46 +02:00
TiA4f8R
f6ec7f9a61
Update DefaultStreamExtractorTest and SoundcloudStreamExtractorTest to support changes made in Stream classes 2022-05-28 12:00:45 +02:00
TiA4f8R
7477ed0f3d
[YouTube] Add ability to generate manifests of progressive, OTF and post live streams
A new class has been added to do so: YoutubeDashManifestCreator.
It relies on a new class: ManifestCreatorCache, to cache the content, which relies on a new pair class named Pair.
Results are cached and there is a cache per delivery type, on which cache limit, clear factor, clearing and resetting can be applied to each cache and to all caches.
Look at code changes for more details.
2022-05-28 12:00:45 +02:00
TiA4f8R
a857684442
Apply changes in YoutubeStreamExtractor
Extract post live DVR streams as post live streams instead of live streams.

A new class has been in order to improve code: ItagInfo, which stores an itag, the content (URL) extracted and if its an URL or not.
A functional interface has been added in order to abstract the stream building: StreamBuilderHelper.
Also add the cver parameter added by the desktop web client on the corresponding streams (a new method has been added in YoutubeParsingHelper to check this and another for Android streams).

Some code in these classes has been also refactored/improved/optimized.
2022-05-28 12:00:44 +02:00
TiA4f8R
4330b5f7be
Add POST_LIVE_STREAM and POST_LIVE_AUDIO_STREAM stream types
This allows the extractor to determine if a content is an ended audio or video livestream.
2022-05-28 12:00:43 +02:00
TiA4f8R
881969f1da
Apply changes in all StreamExtractors except YouTube's one and fix extraction of PeerTube audio streams as video streams
Some code in these classes has been also refactored/improved/optimized.
Also fix the extraction of PeerTube audio streams as video streams, which are now returned as audio streams.
2022-05-28 12:00:43 +02:00
TiA4f8R
d5f3637fc3
[YouTube] Return more values returned inside the ItagItems of the player response and deprecate use of public audio and video fields
These fields can be now replaced by a getter and a setter.

New fields have been added and will allow the creation of DASH manifests for OTF and ended livestreams. There are:
- contentLength;
- approxDurationMs;
- targetDurationSec;
- sampleRate;
- audioChannels.
2022-05-28 12:00:42 +02:00
TiA4f8R
7c67d46e09
Move DashMpdParser to the YouTube package and fix extraction of streams
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.
Also update creation of AudioStreams and VideoStreams objects.
2022-05-28 12:00:41 +02:00
TiA4f8R
ad993b920f
Remove fetching of the DASH manifest extracted when getting information of a content with StreamInfo
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.

Also improve code and comments of StreamInfo (especially final use where possible).
2022-05-28 12:00:41 +02:00
TiA4f8R
2f061b8dbd
Add support of other delivery methods than progressive HTTP in Stream classes
Stream constructors are now private and streams can be constructed with new Builder classes per stream class. This change has been made to prevent creating and using several constructors in stream classes.

Some default cases have been also added in these Builder classes, so not everything has to be set, depending of the service and the content.
2022-05-28 12:00:27 +02:00
TiA4f8R
c34b5e3a8b
[YouTube] Fix extraction of YouTube Music client version and API key when using YouTube Music's website in EU
Google returns now the consent page of YouTube for YouTube Music in EU, which can be also avoided by adding the ucbcb parameter to the URL with the value 1 ("?ucbcb=1").
2022-05-15 11:20:06 +02:00
litetex
2015eb374a Removed more unused methods 2022-05-09 21:05:03 +02:00
litetex
f69b0ff77b Remove unused methods 2022-05-09 20:59:25 +02:00
TiA4f8R
3c3cd78676
Remove Checkstyle suppressions file and fix Checkstyle issues introduced in 24e8399 and 8c1041d
The Checkstyle suppressions file is now replaced by // CHECKSTYLE:OFF and // CHECKSTYLE:ON comments.
2022-05-02 21:51:25 +02:00
Stypox
2e1c5c119d
Merge pull request #822 from Stypox/more-refactors
More refactors
2022-05-02 19:03:54 +02:00
Stypox
598ebb92ea
Merge pull request #839 from TeamNewPipe/bandcamp/extract-length
Bandcamp: extract stream length
2022-05-02 15:49:41 +02:00
litetex
5db4d1faf3
Merge pull request #782 from litetex/cleanup-yt-stream-extractor
Cleanup of ``YoutubeStreamExtractor`` and some related classes
2022-05-01 16:44:11 +02:00
litetex
fe30eb43a9 Cleanup `YoutubeStreamExtractor` and some related classes
* Fixed obvious sonar(lint) warnings
* Abstracted some code (get*Streams)
* Used some new lines to make code better readable
* Chopped down brace-jungle in some methods
* Use StandardCharset (Java 8 4tw)
2022-05-01 16:39:07 +02:00
Stypox
c2b5370517
Apply suggestions: improve switch and use EMPTY_STRING 2022-04-30 16:39:51 +02:00
Stypox
7c78c39230
Merge pull request #821 from litetex/cleanup-TimeAgoParser-java
Cleanup ``TimeAgoParser``
2022-04-30 16:20:09 +02:00
TiA4f8R
9f9af35adb
[YouTube] Fix regression introduced in the order of streams used when adding more parameters to InnerTube requests, using the iOS client for livestreams and more 2022-04-25 20:23:04 +02:00
Fynn Godau
c38c016de5 Bandcamp: extract stream length 2022-04-24 21:24:19 +02:00
Nickoriginal
b5f3f9eb90
Update USER_AGENT to match the latest Firefox 2022-04-24 00:27:23 +03:00
Stypox
52fa2d939a
Fix javadoc formatting error causing deployment to fail 2022-04-16 17:07:07 +02:00
Stypox
dcb7483dcf
Fix YouTube throttling decrypter function parsing 2022-04-15 13:10:19 +02:00
TiA4f8R
ef49cd0007
[YouTube] Extract subtitles for age-restricted videos
Subtitles of age-restricted videos can be extracted since the InnerTube API migration, so there is no reason to not extract them anymore.
2022-04-11 22:09:56 +02:00
TiA4f8R
b30e341559
Add link to learn more about the issue which makes YoutubeStreamExtractorRelatedMixTest.testRelatedItems test disabled 2022-04-04 20:36:59 +02:00
TiA4f8R
70812fa611
Update YouTube stream mocks and disable YoutubeStreamExtractorRelatedMixTest
Mixes seems to be not given by YouTube anymore if you use a PENDING consent cookie value.
As mocks needs to updated, the test is always failing because of this change.
2022-04-02 19:28:49 +02:00
TiA4f8R
67288a0191
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.

This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.

The outdated code for these contents has been removed.

Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.

Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 19:06:36 +02:00
TiA4f8R
11b5a222c4
Deduplicate code of getStringResultFromRegexArray methods in Utils
Also revert indentation in Utils.mixedNumberWordToLong.
2022-04-02 18:40:00 +02:00
TiA4f8R
9ca647a750
Update mocks 2022-03-27 22:10:59 +02:00
TiA4f8R
dfa4239661
Fix missing imports and Checkstyle issues 2022-03-27 22:10:57 +02:00
TiA4f8R
6d27996ac4
Improve code of getStringResultFromRegexArray methods in Utils 2022-03-27 22:10:57 +02:00
TiA4f8R
2e3da445e6
[YouTube] Add documentation about parameters added and clients versions and key
Also move the iPhone device machine id to a constant, explain how it is used and move the licence in the header of the file, and fix missing imports in YoutubeStreamExtractor (due to a rebase issue).
2022-03-27 22:10:57 +02:00
TiA4f8R
1dad3bfe8b
[YouTube] Update again hardcoded client versions and update mobile user agents
Also provide ability to get mobile user-agents used for mobile InnerTube requests and deduplicate related code.
2022-03-27 20:52:40 +02:00
TiA4f8R
3d38459cf3
[YouTube] Reduce InnerTube response sizes by adding the prettyPrint parameter with the false value
InnerTube responses return pretty printed responses, which increase responses' size for nothing.

By using the prettyPrint parameter on requests and setting its value to false, responses are not pretty printed anymore, which reduces responses size, and so data transfer and processing times.
This usage has been recently deployed by YouTube on their websites.
2022-03-27 20:52:40 +02:00
litetex
349ba8db7f
Improve tests and randomness
- Use the existing RNG inside YoutubeParsingHelper
- Deduplicated test-setup for YouTube tests
- Minor improvements
2022-03-27 20:52:38 +02:00
TiA4f8R
52376949e5
Add setSeedForVideoTests method in YoutubeStreamExtractor tests
In order to use still use mocks with the generation of random strings in player requests, we need to use YoutubeParsingHelper.setSeedForVideoTests() method in every stream test.
2022-03-27 20:51:39 +02:00
TiA4f8R
d0d91e6690
Adress requested changes 2022-03-27 20:51:39 +02:00
TiA4f8R
b6bc521f0d
[YouTube] Update client versions again 2022-03-27 20:51:38 +02:00
TiA4f8R
26f93f5bb0
[YouTube] Extract streams of livestreams from the iOS client and disabled the Android client for livestreams
The iOS client is only enabled for livestreams and the Android client is now only enabled for videos, both by default.

A way to force, or not, the fetch of both clients have been added with two new static methods in YoutubeStreamExtractor.
2022-03-27 20:51:38 +02:00
TiA4f8R
7d07924de8
[YouTube] Try to use lighter requests when extracting client version and key from YouTube and YouTube Music
This is done by fetching https://www.youtube.com/sw.js for YouTube and https://music.youtube.com/sw.js for YouTube Music.

Two new methods in Utils class have been added which allow to try to get a match of regular expressions in a string array, or a Pattern array, on a content, on a specific index or 0.
Also some code refactoring has been made in this class.
2022-03-27 20:51:38 +02:00
TiA4f8R
05b7fee23b
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.

For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.

Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).

For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.

This commit also fixes a small bug with the Android User-Agent string.

Some code improvements have been also made.
2022-03-27 20:51:38 +02:00
TiA4f8R
83f374bff1
[YouTube] Update client versions and fix a bug when using resetClientVersionAndKey method
The boolean keyAndVersionExtracted in YoutubeParsingHelper was not set to false when resetting the client version and the key, which makes the extractor uses null on the next getting of the client version or the key if the clientVersion and the key were extracted before.
Also update client versions.
2022-03-27 20:51:38 +02:00
Stypox
8c1041def6
Add @ null annotations where Android Studio suggested it
That is, basically where the overriding function was missing an annotation from the base method.

Also apply renaming of emptyDescription to EMPTY_DESCRIPTION
2022-03-26 22:07:14 +01:00
Stypox
adbbdc7a5b
[YouTube] Fix regex warning: use ' {2}' instead of ' ' 2022-03-26 22:07:14 +01:00
Stypox
24e83997b4
[Bandcamp] Add Java 8 streams 2022-03-26 22:07:12 +01:00
Stypox
349990fd48
Fix redundant escape \\ in regex in Utils 2022-03-26 22:01:30 +01:00
litetex
3bf7aa3762 Cleanup `TimeAgoParser` 2022-03-26 21:09:31 +01:00
litetex
af82edf9dc Fix checkstyle problems 2022-03-26 20:54:20 +01:00
litetex
29408f9356 `YoutubeMusicSearchExtractorTest$Suggestion` has no mock data 2022-03-26 20:52:28 +01:00
litetex
fcee247f9f Switched to mockonly tests
* Made the reason why the tests are mockonly mandatory
2022-03-26 20:52:28 +01:00
litetex
0fceb4686b Added missing mock data 2022-03-26 20:52:27 +01:00
litetex
70b20f0d6b Updated mock data after conflicts 2022-03-26 20:52:27 +01:00
litetex
6a6c9359af Fixed kurzgesagt test - They changed their description... 2022-03-26 20:52:27 +01:00
litetex
26596215fa Refactored `MediaCCCRecentListExtractorTest` 2022-03-26 20:52:26 +01:00
litetex
53962bfd7d Disabled suggestion test for now
Has to be done once YT's backed works again...
2022-03-26 20:52:26 +01:00
litetex
66dc5e8bb8 API hardening against changes 2022-03-26 20:52:26 +01:00
litetex
5d58156cde Update mock data 2022-03-26 20:52:25 +01:00
litetex
7598b40957 Workaround for incorrect duration for "YT shorts" videos in channels
As a workaround 0 is returned as duration for such videos.
See also https://github.com/TeamNewPipe/NewPipe/issues/8034
2022-03-26 20:52:24 +01:00
litetex
164e21b5af Fixed `MediaCCCRecentKiosk`
Ignore faulty data/items (with duration <= 0)
2022-03-26 20:52:23 +01:00
litetex
49dfdae5d2 Removed useless throw declaration 2022-03-26 20:51:38 +01:00
litetex
69a58fd3d1 Don't use sysout 2022-03-26 20:51:37 +01:00
litetex
639be7adda Minimized some code 2022-03-26 20:51:37 +01:00
litetex
2bd4299563 Fixed test: Peertube - Use new comments and videos
Old comments/videos got deleted
2022-03-26 20:49:39 +01:00
litetex
606a386a98 Fix tests: What's peertube has less comments now 2022-03-26 20:49:39 +01:00
litetex
4d1a1c8fb8 Better test for `MediaCCCRecentListExtractorTest`
* Use assertAll
* Show which item is affected
2022-03-26 20:49:38 +01:00
litetex
ba43dbaa28 Fix Tests: The channel lost abos 2022-03-26 20:49:38 +01:00
litetex
ff436e5740 Fixed test: YT mix playlist 2022-03-26 20:49:38 +01:00
litetex
9c07e8a664 Fix useage of wrong object 2022-03-26 20:17:50 +01:00
litetex
804e57004f Fixed new checkstyle problems from dev 2022-03-26 19:46:10 +01:00
litetex
33347ac18b Removed unused methods
``contentFilters`` and ``sortfilter`` are get inside the ``ListLinkHandler`` and not the ``ListLinkHandlerFactory``
 ``ListLinkHandlerFactory`` only passes these values through when ``fromQuery`` is called
2022-03-26 19:43:11 +01:00
litetex
ec5b54c38b Removed unused class 2022-03-26 19:43:10 +01:00
litetex
8771af7ba5 Restored original naming 2022-03-26 19:43:09 +01:00
Stypox
bdadcfa1f7 Legitimately suppress remaining checkstyle warnings 2022-03-26 19:43:08 +01:00
Stypox
740a37a2de [YouTube] Fix checkstyle issues 2022-03-26 19:42:40 +01:00
Stypox
9dc17cd1ca [Soundcloud] Fix checkstyle issues 2022-03-26 19:40:20 +01:00
Stypox
9ab32cb2e7 [Peertube] Fix checkstyle issues 2022-03-26 19:40:19 +01:00
Stypox
9f7e06c817 [MediaCCC] Fix checkstyle issues 2022-03-26 19:40:18 +01:00
Stypox
3a94839359 [Bandcamp] Fix checkstyle issues 2022-03-26 19:40:17 +01:00
Stypox
08dff33002 Use Java 8 streams in NewPipe class 2022-03-26 19:40:15 +01:00
Stypox
c2446ecff0 Use Java 8 streams and deduplicate code in MediaFormat class 2022-03-26 19:40:15 +01:00
Stypox
d79e20340c Fix checkstyle issues in root package extractor/
Note: not all issues were fixed because MediaFormat and ServiceList use a specific formatting that makes sense for them
2022-03-26 19:40:14 +01:00
Stypox
ca7c63f273 Fix remaining checkstyle issues in utils/ subpackage 2022-03-26 19:40:13 +01:00
Stypox
1d5f22e41f Fix checkstyle issues & more in JsonUtils
Also use Java 8 streams and extract duplicate code to getInstanceOf function
2022-03-26 19:40:13 +01:00
Stypox
87d2834986 Fix checkstyle issues & more in DonationLinkHelper
Also add comment about the class being unused and replace the fixLink function with Utils.stringToUrl()
2022-03-26 19:40:12 +01:00
Stypox
bd7b362040 Fix checkstyle issues & more in DashMpdParser
Also remove useless null check on ItagItem.getItag() as that function already throws an exception if there is no itag
2022-03-26 19:40:11 +01:00
Stypox
8aba2b47b0 Fix checkstyle issues in subpackages with abstract classes 2022-03-26 19:40:10 +01:00
Stypox
e4951a0623 Refactor code handling http headers in downloader.Request 2022-03-26 19:37:47 +01:00
XiangRongLin
aa6b7272a4
Merge pull request #804 from Stypox/fix-yt-music-mix
[YouTube] Fix music mixes in some countries
2022-03-20 08:35:56 +01:00
Stypox
09ddb6adbb
[YouTube] Add MockOnly to method testing mixes in related items 2022-03-19 10:54:38 +01:00
Stypox
8201b3b90e
[YouTube] Parse any playlist (including music mixes) in related items 2022-03-19 10:48:13 +01:00
Stypox
13f7900816
[YouTube] Add test for genre mix 2022-03-19 10:48:13 +01:00
Stypox
279f3a20fe
[YouTube] Fix mix tests with invalid video ids
Replaces mix tests based on a strange mix type RDQM{videoId} (only reference I could find is https://github.com/ytdl-org/youtube-dl/issues/26228) and with an invalid video id of 13 characters (the first two characters were QM, but even after removing QM there still wasn't a video available at that id).
Also updates mocks.
2022-03-19 10:48:13 +01:00
Stypox
d660c04838
[YouTube] Also test playlist type in playlist tests 2022-03-19 10:48:13 +01:00
Stypox
401082abe4
[YouTube] Extract playlist type in playlist extractor 2022-03-19 10:48:12 +01:00
Stypox
63ed06a710
[YouTube] Differentiate genre mixes from normal mixes
Note: genre mixes already worked, now they are just considered as such in various video id extraction and in related items
Note 2: now extracting a mix id from a *normal* youtube mix id will fail if the video id wouldn't be exactly 11 characters long
2022-03-19 10:46:31 +01:00
Stypox
f19660e7d9
[YouTube] Deduplicate code extracting video id from mix id 2022-03-19 10:46:30 +01:00
Stypox
8f9d5b858e
[YouTube] Remove useless comments about mixes 2022-03-19 10:44:06 +01:00
Stypox
34a4484c72
[YouTube] Add test for a video with a mix in related items 2022-03-19 10:44:06 +01:00
Stypox
50db871d89
[YouTube] Extract mixes from streams related items 2022-03-19 10:44:06 +01:00
Stypox
638da1756c
[Mix] Create MultiInfoItemsCollector
It is a collector that can handle many extractor types, to be used when a list contains items of different types (e.g. search). It was renamed from InfoItemsSearchCollector so that it can now be used not just for search but for any extractor needing it. It supports, streams, channels, playlists and *mixes*.
2022-03-19 10:44:06 +01:00
Stypox
53673d64c6
[Mix] Add type to playlists & playlist items, to distinguish mixes 2022-03-19 10:44:06 +01:00
Stypox
d8f2031619
Merge pull request #816 from Stypox/mock-only-extension
Add `@MockOnly` Junit 5 extension
2022-03-19 10:40:38 +01:00
litetex
cc2e4d7104
Merge pull request #815 from litetex/fix-soundcloud-id-once-and-for-all
Removed hardcoded soundcloud HARDCODED_CLIENT_ID
2022-03-17 13:54:08 +01:00
TiA4f8R
c7757c0994
Apply requested changes 2022-03-16 20:14:08 +01:00
TiA4f8R
35e082248e
Fix YouTube and SoundCloud playlists tests 2022-03-16 19:40:30 +01:00
TiA4f8R
8b3f90eb7e
[YouTube] Fix extraction of series playlists and don't return the view count as the stream count for learning playlists
ITEM_COUNT_UNKNOWN is returned when the JSON array which contains usally the number of videos is less than 3 items.
Also apply the same type of optimizations done in other PlaylistExtractors in YoutubePlaylistExtractor.
2022-03-16 19:18:58 +01:00
TiA4f8R
58a247907e
Apply changes in all playlist extractors except YoutubePlaylistExtractor
Also fix some issues in the extractors, remove uneeded overrides, use the Java 8 Stream API where possible and replace usages of Utils.UTF_8 with StandardCharsets.UTF_8 in these classes.
2022-03-16 19:18:57 +01:00
TiA4f8R
fc6b45ee36
Implement some methods in PlaylistExtractor
This will prevent their override in each child class where the values corresponding to the methods could not be extracted.
2022-03-16 19:18:36 +01:00
Stypox
73d1fd472f
Add MockOnly junit 5 test extension 2022-03-16 19:03:08 +01:00
Stypox
ef71a5fa0f
static final instead of final static 2022-03-16 17:24:33 +01:00
Stypox
0c37c75981
Make getDownloader static & extract getDownloaderType 2022-03-16 17:22:42 +01:00
Stypox
40aa5104b1
Merge pull request #786 from XiangRongLin/throttling_resilience
[Youtube] Make throttling decryption more resilient to api change
2022-03-16 11:03:16 +01:00
litetex
ba56be8ef1 Removed hardcoded soundcloud id
It never works (long enough) so let's simply remove it...
2022-03-15 21:19:19 +01:00
XiangRongLin
e726437da3
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
Co-authored-by: Stypox <stypox@pm.me>
2022-03-15 17:10:05 +01:00
litetex
e7aee0ca57
Merge pull request #807 from FireMasterK/no-commentsinfo-instance
Remove the need for a CommentsInfo instance in CommentsInfo.getMoreItems and fix PeertubeCommentsExtractorTest.Default test
2022-03-15 15:06:56 +01:00
FireMasterK
60cc71e944
Remove the need for a CommentsInfo instance. 2022-03-03 11:48:41 +00:00
Stypox
dd8687f9fe
[YouTube] Fix music mixes in some countries 2022-03-01 23:02:56 +01:00
FireMasterK
e6d334765d
Apply requested codestyle improvements. 2022-02-26 17:12:51 +00:00
FireMasterK
6950b362f2
Add test values for failing tests. 2022-02-24 12:55:13 +00:00
FireMasterK
5b0ec694a6
Add requested changes. 2022-02-24 12:50:51 +00:00
FireMasterK
d290d2e393
Move variable to super Extractor classes. 2022-02-24 12:50:51 +00:00
FireMasterK
ab49cb6e18
Add requested changes. 2022-02-24 12:50:51 +00:00
Kavin
4bd59f65f4
Update exception message.
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-02-24 12:50:51 +00:00
FireMasterK
6f3f608ab6
Add support for extracting channel subscriber count in StreamInfo. 2022-02-24 12:50:51 +00:00
litetex
1a67ea100a Refactored code
according to review
2022-02-21 19:56:12 +01:00
litetex
f79ce1f52a Refactored YoutubeCommentsExtractor
* Use Java Streaming API
* Use StandardCharsets
* Prevented several NPEs/ArrayIndexOutOfBound
* Reformatted some code so that it's easier readable
2022-02-21 19:56:09 +01:00
TobiGr
d337e537dc [YouTube] Add a comment clarifying why the panelIdentfier can be null 2022-02-18 20:26:19 +01:00
Tobi
5c8fbeb166
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments() (#797)
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments()

Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
2022-02-16 23:39:26 +01:00
Tobi
0d6aebce25
[SoundCloud] Update clientId (#798) 2022-02-11 17:16:32 +01:00
XiangRongLin
545522b80f [Youtube] Make throttling decryption more resilient to api change 2022-02-01 19:57:34 +01:00
TobiGr
047d75a4c9 Improve code style 2022-02-01 12:56:59 +01:00
Abdu Ameen
50a4b026f8 Adding removal of the braces 2022-01-31 23:52:31 -08:00
Abdu Ameen
a9174f6b9f Changed the regex to account for
nonword characters
2022-01-31 22:29:00 -08:00
Stypox
59e0aeba88
Merge pull request #768 from litetex/junit-5
JUnit 5
2022-01-24 16:56:16 +01:00
Mohammed Anas
65df39b3e3
[YouTube] parse timestamps with >3 digits correctly (#775)
* [YouTube] parse timestamps with >3 digits correctly

Fixes https://github.com/TeamNewPipe/NewPipe/issues/7530; check the
issue for details.

* Remove outdated comment
2022-01-12 10:45:06 +01:00
litetex
a579ef2651 Fixed Soundcloud tests 2022-01-07 14:27:24 +01:00
litetex
a32edce750 Removed deprecation because AndroidSDK/API < 19 2022-01-07 14:15:55 +01:00
litetex
602068937d
Code cleanup
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-01-06 21:04:16 +01:00
litetex
dd3f3da828 Kurzgesagt mock data is outdated 2022-01-04 17:44:12 +01:00
litetex
a8ebd2cead Update mock resources for YT Mix playlist 2022-01-04 17:39:14 +01:00
litetex
066afd8629 Restored test 2022-01-04 17:33:08 +01:00
litetex
3712a669b1 Code cleanup / improvements
* Deprecated Utils#UTF-8; see StandardCharsets
* Added more helpful methods to ``ExtractorAsserts``
* Use parameterized (cool new) tests
* Restore functionality of some tests + updated mockdata
* Other code cleanups + Sonarlint improvements
2022-01-04 17:28:31 +01:00
litetex
4291a90251 Fixed SoundCloud's search(for tests) 2021-12-27 21:42:43 +01:00
litetex
4995709871 Fixed SoundCloud's search(for tests)
Getting the initial page was not returning initial page
2021-12-27 21:18:39 +01:00
litetex
a2cbdf0991 Updated to JUnit 5 2021-12-27 21:08:08 +01:00
litetex
148ba6641d Updated mock data 2021-12-27 17:08:14 +01:00
litetex
4d0a689cb6 Disabled dislike count in tests 2021-12-27 17:04:46 +01:00
litetex
a6961c5efa Fixed NumberFormatException when processing likes
See also #7525
2021-12-27 16:54:47 +01:00
litetex
65687f3b9b Removed YT dislike count as it no longer works 2021-12-27 16:54:46 +01:00
litetex
1e2e0029fc [StreamExtractor] Deduplicated a ton of code by using default methods 2021-12-27 16:54:45 +01:00
litetex
15b98ffdb4 Remove old ways of getting YT dis/likes
* Added additional check for averageRating (in dislikes)
2021-12-27 16:54:44 +01:00
litetex
7489502cc8 Disabled `testRelatedItems` as it's unreliable 2021-12-11 15:23:11 +01:00
litetex
d9b2a2042a Updated mock data for YoutubeCommentsExtractorTest 2021-12-08 22:11:45 +01:00
litetex
a3a0c1ee85 Fixed YoutubeCommentsExtractorTest
Old video was deleted / set to private
2021-12-08 22:11:34 +01:00
litetex
5c288eb8d7 Updated mock data for YoutubeChannelExtractorTest 2021-12-08 21:51:38 +01:00
litetex
75bb83a1b8 Fixed YoutubeChannelExtractorTest
Channel was restored, picked a new one
2021-12-08 21:51:15 +01:00
litetex
f276bacce5 Fixed SoundcloudStreamExtractorTest 2021-12-08 21:41:27 +01:00
litetex
7f99c1c193 Fixed YoutubeStreamExtractorLivestreamTest 2021-12-08 21:16:39 +01:00
litetex
652fdf2c36 Updated mock-data for YoutubeStreamExtractorAgeRestrictedTest 2021-12-08 21:08:25 +01:00
litetex
3c669d3cb5 Fixed YoutubeStreamExtractorAgeRestrictedTest
Looks like a lot of stuff (description, tags, name) got changed on the existing video.
Use a better suited video.
2021-12-08 21:08:03 +01:00
litetex
f735788a39 Fixed Soundcloud hardcoded client id 2021-12-08 21:00:53 +01:00
bopol
56c8af710e Code refactoring 2021-11-21 18:16:31 +01:00
bopol
c4eca91be9 Fix YouTube likes + dislikes 2021-11-19 21:36:03 +01:00
TobiGr
5028396405 Improve names and formatting 2021-11-14 13:44:13 +01:00
opusforlife2
cb80a646d9
Update hardcoded SoundCloud client_id
Hah! I got here first, @TobiGr!
2021-11-04 15:28:26 +00:00
bopol
396aecef19 Make sure playlists aren't accepted by PeertubeStreamLinkHandlerFactory 2021-11-03 14:41:39 +01:00
bopol
38ad1eaac5 [PeerTube] Support /w/ short video links part 2
We also need to support it for comments
2021-11-03 13:54:37 +01:00
bopol
026751624f [PeerTube] Support /w/p/ short playlist links 2021-11-03 13:35:30 +01:00
bopol
e0b8e142fc [PeerTube] Support /a/ and /c/ short links 2021-11-03 13:26:27 +01:00
bopol
599a91c88c [PeerTube] Support /w/ short video links 2021-11-03 11:51:58 +01:00
mhmdanas
3e8e2a1532 Add support for y2u.be links 2021-10-22 22:48:18 +03:00
Tobi
b425394e7a
Merge pull request #731 from FireMasterK/short-description
Extract Video Short Description in YouTube.
2021-10-15 23:10:00 +02:00
FireMaskterK
62313962a0 Add default implementation for getShortDescription. 2021-10-04 01:42:29 +01:00
Kavin
d1add05bc1 Add requested changes.
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
Add final.

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-10-04 01:36:51 +01:00
TobiGr
d8ec3ce370 [SoundCloud] Update client id 2021-10-01 02:51:38 +02:00
FireMasterK
94efe86c71
Extract Video Short Description in YouTube.
In Trending, and Search results.
2021-09-22 16:26:16 +01:00
Tobi
a9d214478d
Merge pull request #703 from FireMasterK/comment-replies
Add support for extracting comment replies continuation
2021-09-14 23:58:14 +02:00
FireMasterK
6aabdc6d16
Fix for requested changes. 2021-09-12 01:15:19 +05:30
Kavin
0aad09fa22
Update JavaDocs for comment replies continuation.
Co-authored-by: Stypox <stypox@pm.me>
2021-09-12 01:05:20 +05:30
FireMasterK
feb09e9997
Extract Channel Avatar in related videos. 2021-09-04 01:09:40 +05:30
Tobi
f0aa46b008
Merge pull request #724 from FireMasterK/streams-uploader-verified
Extract uploaderVerified to StreamInfo.
2021-09-03 18:18:04 +02:00
FireMasterK
857b2f39e8
Extract uploaderVerified to StreamInfo. 2021-09-03 21:27:58 +05:30
FireMasterK
1db463b55f
Add a default test for the Uploader's Avatar. 2021-09-03 21:05:36 +05:30
Kavin
db6b3b2c29
Extract uploader's avatar in peertube.
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-09-02 16:14:45 +05:30
FireMasterK
a0c1dcc8d8
Remove throws parsing exception. 2021-09-02 16:14:45 +05:30
FireMasterK
0ba03c552c
Improve based on review/feedback. 2021-09-02 00:35:56 +05:30
FireMasterK
b9fad4fcc8
Extract Uploader's Avatar on YouTube. 2021-09-02 00:19:00 +05:30
Stypox
68f1fa994a
Merge pull request #721 from Stypox/yt-csv-import
[YouTube] Small improvements to subscription import
2021-08-31 12:02:59 +02:00
TobiGr
b644f1d67b [SoundCloud] Update client id 2021-08-31 07:42:21 +02:00
Stypox
a5e9eeb790
[YouTube] Small improvements to subscription import 2021-08-30 15:45:45 +02:00
Tobi
9570882c73
Merge pull request #712 from magicbrothers/add-eduvid-tubus
Add invidious instance EduVid Tubus
2021-08-24 16:07:07 +02:00
talanc
94a29fd63f
[YouTube] csv and zip subscription import (Google Takeout)
csv:
Improved error messages
Exits early if it hasnt found any items in the first few lines

zip:
Now checks all CSV files instead of hard-coded paths

final qualifiers for immutable locals and parameters

Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
2021-08-24 15:42:49 +02:00
Stypox
d8179dc35a
Merge pull request #713 from TiA4f8R/regexes-yt-signatureciphers
[YouTube] Consolidate the regular expressions used to find the cipher decryption function
2021-08-17 18:58:27 +02:00
XiangRongLin
61acdae4b8 Fix failing PeertubeStreamExtractorTest 2021-08-17 17:06:36 +02:00
FireMasterK
4f86b8ce43
Add verified info to ChannelInfo. (#714) 2021-08-17 13:08:49 +02:00
Stypox
52910ce970
Keep original exception when there is an error message (#710) 2021-08-13 16:13:58 +02:00
TiA4f8R
f8309d546a
[YouTube] Consolidate current regexes of the cipher decryption function
Also add more regexes
2021-08-12 18:26:43 +02:00
magicfelix
0e16091ce0
Add invidious instance EduVid Tubus 2021-08-12 10:06:41 +02:00
gardenapple
50bba32239
Fix typos in Javadoc 2021-08-11 12:01:25 +03:00
FireMasterK
71b9fd0076
Faster iframe api based player extraction. (#694)
* Faster iframe api based player extraction.

Uses the IFrame API to reduce the required download to less than 1/50 of the size.

* Remove debug code.

* Extract to two methods.

* Add tests for player URL extraction.

* Add assertThat for tests.
2021-08-07 12:35:48 +02:00
FireMasterK
ee32317315
Update supported countries list for YouTube. 2021-08-05 23:48:38 +05:30
FireMasterK
e8e9e6cd00
Fix comment reply continuations. 2021-08-04 22:14:04 +05:30
FireMasterK
666b45958e
Fix test and update mocks. 2021-08-04 21:10:02 +05:30
FireMasterK
f6d054e5da
Add test for comments. 2021-08-04 21:03:00 +05:30
FireMasterK
ed84658055
Get it working. 2021-08-04 18:35:14 +05:30
FireMasterK
ea7e6526fd
Collect replies/page to CommentsInfoItem. 2021-08-04 18:09:42 +05:30
FireMasterK
4385cc1b7c
Add comment replies. 2021-08-04 17:56:04 +05:30
Stypox
5b38b3ae97
Merge branch 'master' into v0.21.8 2021-08-03 20:38:07 +02:00
TiA4f8R
79cc5c8d12
Fix tests of YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest
Without removing RunWith and SuiteClasses annotations (and the corresponding imports) in YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest, some mocks cannot be generated, so the CI fails because of the missing mocks. Mocks of workings tests have been also updated.
2021-08-01 17:28:01 +02:00
TiA4f8R
b74a39c176
Reformat some code and don't use the clickTrackingParams in continuations of YouTube Music search results
The clickTrackingParams of YouTube Music search results are not needed to get continuations. This commit removes their use, which may improve privacy.
2021-08-01 17:25:59 +02:00
TiA4f8R
f8197da09e
Update YouTube comments and YouTube playlist continuations mocks 2021-08-01 12:39:05 +02:00
FireMasterK
f3e4c9d689
Use the youtubei API for YouTube comments
Migrate YouTube comments to the desktop version by using the `next` endpoint of the InnerTube internal API.
With the desktop version, we are able to get the exact like count of YouTube comments (by parsing the accessibility data) (the current extraction is used as a fallback). We are also now able to get if the uploader of the comment is verified or not.

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:05 +02:00
TiA4f8R
286d839a3b
Update mocks (except mocks of YoutubeCommentsExtractorTest) 2021-08-01 12:39:04 +02:00
TiA4f8R
693f654f02
Ignore a broken test of YoutubeMixPlaylistExtractorTest 2021-08-01 12:39:03 +02:00
TiA4f8R
d55109258d
Fix YoutubeStreamExtractorAgeRestrictedTest 2021-08-01 12:39:03 +02:00
FireMasterK
2eeb0a3403
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:

- the desktop API is fetched.

If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status

If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.

If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.

We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.

Other code changes:

- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:03 +02:00
TiA4f8R
7753556e66
Adress the last requested changes + update YoutubeCommentsExtractor mocks 2021-08-01 12:39:03 +02:00
TiA4f8R
2320aecb7c
Update mocks 2021-08-01 12:39:02 +02:00
TiA4f8R
629dcd63b2
Ignore some broken tests 2021-08-01 12:39:01 +02:00
TiA4f8R
3adac6a150
Workaround for rate limits: always use the Android mobile API 2021-08-01 12:39:01 +02:00
TiA4f8R
8aa60d7e8f
Update clients version 2021-08-01 12:39:01 +02:00
TiA4f8R
27bd797f1e
Update mocks 2021-08-01 12:39:01 +02:00
TiA4f8R
609919db59
Adress again reviews, fix some rebase issues 2021-08-01 12:39:00 +02:00
TiA4f8R
4299d806a2
Adress changes 2021-08-01 12:38:59 +02:00
TiA4f8R
1a6b8da438
Annotate YoutubeParsingHelper methods with Nonnull when needed 2021-08-01 12:38:59 +02:00
TiA4f8R
d8177b57f6
Loop in all formats to check if the stream has URLs protected by signatureCiphers 2021-08-01 12:38:59 +02:00
TiA4f8R
a6a2c6eb80
Revert the use of Collections.singletonList instead of Arrays.asList in addCookieHeader of YoutubeParsingHelper 2021-08-01 12:38:59 +02:00
TiA4f8R
81013e5a8e
Add a static method to reset the YouTube deobfuscation code and use it in tests
This method is needed for YouTube stream tests, because when all YouTube tests are ran, the signatureTimestamp is known (the sts string) so a different body than the body present in the mocks is send by the extractor instance.
As a result, running all YouTube stream tests with the MockDownloader (like the CI does) will fail if this method is not called before fetching the page of a test.
2021-08-01 12:38:58 +02:00
TiA4f8R
cc798523cd
Adress requested changes in other classes 2021-08-01 12:38:58 +02:00
TiA4f8R
accd5ddef3
Adress requested changes in YoutubeStreamExtractor 2021-08-01 12:38:58 +02:00
TiA4f8R
632772d17f
Adress requested changes in YoutubeParsingHelper 2021-08-01 12:38:58 +02:00
TiA4f8R
657f165771
Update client version and mocks 2021-08-01 12:38:44 +02:00
TiA4f8R
8c1c7281b0
Real fix of the fetch of the JS player at each the signatureTimestamp is required
The strings playerJsUrl, sts and playerCode are now static in order to don't fetch again the JavaScript player at each time the signatureTimestamp is needed.
2021-08-01 12:38:43 +02:00
TiA4f8R
e97a685989
Format the YoutubeMusicSearchExtractor class to be in the 100 characters per line limit 2021-08-01 12:38:43 +02:00
TiA4f8R
34a9ccb0fd
Adress requested changes 2021-08-01 12:38:42 +02:00
TiA4f8R
54d4551ca6
Adress requested changes in YoutubeParsingHelper and update mobile client version 2021-08-01 12:38:42 +02:00
TiA4f8R
a59c2a3577
Catch every exception instead of only IOException and ExtractionException and add a Javadoc
Catch every exception instead of only IOException and ExtractionException.
Add JavaDoc for fetchAndroidMobileJsonPlayer method of YoutubeStreamExtractor
2021-08-01 12:38:42 +02:00
TiA4f8R
6921e80ded
Try again to don't fetch at each time the JavaScript player 2021-08-01 12:38:42 +02:00
TiA4f8R
70927ddade
Update client version and mocks 2021-08-01 12:38:40 +02:00
TiA4f8R
318bc46a8c
Readd the deleted code of views because watching count of livestreams was broken
The number shown was the total number of views that a livestream has. In order to fix this bug, the previous code is readded.
2021-08-01 12:38:39 +02:00
TiA4f8R
947baec805
Fetch again the desktop player JSON only if the content is protected by signatureCiphers 2021-08-01 12:38:39 +02:00
TiA4f8R
7474049fd1
Update mocks 2021-08-01 12:38:39 +02:00
TiA4f8R
c32bc6e534
Try to don't fetch again the first page when requesting a channel continuation
Try to don't fetch again the first page of a YouTube channel when requesting a continuation of it by trying to store the channel name and the channel id into the next page using the ids field of the Page class.
2021-08-01 12:38:38 +02:00
TiA4f8R
b52732a1c3
Fix the fetch of the playerCode at each time getStsFromPlayerJs is called
storePlayerJs was called even if sts was not empty in the getStsFromPlayerJs method. This commit fixes it.
2021-08-01 12:38:37 +02:00
TiA4f8R
ae5abc0c5d
Fix a typo in YoutubeChannelExtractor 2021-08-01 12:38:37 +02:00
TiA4f8R
0f9e9b8b4b
Use the youtubei API for YouTube mixes + update the corresponding test + do some improvements
Use the youtubei API for YouTube mixes. The corresponding has been updated because the new API breaks the tests of YoutubeMixPlaylistExtractorTest.
Remove some deprecated code (the old search code with the pbj JSON) and do some other improvements.
2021-08-01 12:38:37 +02:00
TiA4f8R
14569c4aa9
Readd the return of itags 134 and 136 2021-08-01 12:38:37 +02:00
TiA4f8R
3017dde67e
Fix some typos 2021-08-01 12:38:36 +02:00
TiA4f8R
013b902535
Use the Android mobile API when there are OTF streams or the content is protected by signatureCiphers
Use the Android mobile API to get the itag 22 (720p with audio), removed when the content is protected by signatureCiphers.
Also use this API when they are OTF streams, to get the itag 17 and 36, low 3GPP quality streams but also the itag 139.
Update the web client version.
2021-08-01 12:38:36 +02:00
TiA4f8R
e7d589edbf
Use the youtubei API for YouTube videos + update client version
Update the hardcoded client version to 2.20210520.09.00
Use the player and next endpoints of the Innertube API for YouTube videos
2021-08-01 12:38:36 +02:00
TiA4f8R
f73c923f60
Don't use the youtubei.googleapis.com but the websites domains + update client version of the desktop internal API
Use again www.youtube.com and music.youtube.com domains instead of youtubei.googleapis.com domain because it spoofs more a web client of YouTube or YouTube Music and may reduce Google's detection of NewPipe Extractor users.
2021-08-01 12:38:34 +02:00
TiA4f8R
effcdaa4f2
Update mocks, reenable a test and fix a test
This commit updates mocks and reenables the test invalidId of the NotAvailable class of the YoutubePlaylistExtractorTest class beacuse with the youtubei/innertube API, it returns "Not found" and doesn't redirect to the YouTube homepage.
The expectedMetaInfo test of the MetaInfoTest class of the YoutubeSearchExtractorTest class was broken because YouTube removes the vaccine progress link of the WHO from the meta info so this commit removes it and the test is now passing.
2021-08-01 12:38:04 +02:00
TiA4f8R
4d682834c3
Fix localization and update client version 2021-08-01 12:38:03 +02:00
TiA4f8R
f46cfb0f26
Adress reviews and do some improvements
Adress changes requested in reviews.
Do some improvements, remove unused imports and format some code to be in the 100 characters line limit.
2021-08-01 12:38:03 +02:00
TiA4f8R
e075dd5a63
Update client version, fix some tests, update mocks and do some improvements
Add the origin and the referer headers with the https://www.youtube.com value for YouTube JSON POST requests.
Don't add the consent cookie header for the requests which use the youtubei/innertube API because it's uneeded.
Fix some tests and update YouTube mocks
2021-08-01 12:38:02 +02:00
TiA4f8R
b49ae547a3
Do some improvements to YoutubeStreamExtractor
Get the real name of the uploader (for autogenerated channels and music artist channels), like before the migration to the JSON pbj.
Do some other improvements, especially reformatting some code to be in the 100 characters line limit and use final where possible.
2021-08-01 12:38:01 +02:00
TiA4f8R
58ce9b04a1
Fix channel extraction when channel URL is youtube.com/c/username 2021-08-01 12:38:01 +02:00
TiA4f8R
991b2c7d73
Use lightweight requests when getting and checking YouTube API key and client version 2021-08-01 12:38:01 +02:00
TiA4f8R
9ab9c66ddf
Use the youtubei.googleapis.com domain for YouTube Music searches + change a check + update client version and mocks
Change the domain from music.youtube.com to youtubei.googleapis.com.
Use a lightweight request to check if the hardcoded YouTubeMusic keys are valid. Increase the length of the response to 500 because if the key is invalid, the length of the response returned is higher than 250 and the response when the key is valid is higher than 1500.
Format the YoutubeMusicSearchExtractor file.
Update YouTube web client version and mocks
2021-08-01 12:37:56 +02:00
TiA4f8R
77c031a88a
Use the youtubei API for YouTube trends 2021-08-01 12:36:28 +02:00
TiA4f8R
f461224b2b
Use the youtubei API for YouTube searches + update mocks
Add getSearchParameter, a new method in YoutubeSearchQueryHandlerFactory class which returns the params field for a search, or an empty string if there is no one.
Update mocks of YoutubeSearchExtractorTest.
2021-08-01 12:36:28 +02:00
TiA4f8R
a12c69da7d
Use the youtubei API for YouTube channels 2021-08-01 12:36:27 +02:00
TiA4f8R
5794eb2350
Use the youtubei API for YouTube playlists 2021-08-01 12:36:12 +02:00
Stypox
c97a19d719
Merge pull request #696 from XiangRongLin/decrypt_pattern
Expand regex to match n param decrypt function
2021-08-01 10:24:22 +02:00
XiangRongLin
37df225556 Remove length check from StringUtils.matchToClosingParenthesis 2021-07-31 16:05:24 +02:00
FireMasterK
f4aad8b014
Fix tests again. 2021-07-31 14:40:51 +05:30
FireMasterK
88c11db4c5
Fix isCorrectedSearch test. 2021-07-31 02:18:18 +05:30
FireMasterK
fd19c53f65
Fix search suggestions test. 2021-07-31 02:08:50 +05:30
FireMasterK
a685941bba
Fix YouTube music search. 2021-07-31 01:33:38 +05:30
XiangRongLin
852a65ff18 Add tests for StringUtils 2021-07-30 19:56:57 +02:00
XiangRongLin
48d897e6ad Add final and adjust utils class name 2021-07-29 21:33:45 +02:00
XiangRongLin
60794aea31 [YouTube] Add parenthesis matching as way to parse decrypt function
From @Stypox
2021-07-29 21:23:00 +02:00
XiangRongLin
2967d1ae6a [YouTube] Compile YoutubeThrottlingDecrypter pattern statically 2021-07-29 20:27:29 +02:00
XiangRongLin
1c78976900 [YouTube] Expand regex to match n param decrypt function
Temporary solution
2021-07-29 20:26:36 +02:00
TobiGr
b62fe7141e Fix JDoc 2 2021-07-29 12:34:04 +02:00
TobiGr
dbc4e01c17 Fix JDoc 2021-07-29 12:31:33 +02:00
TobiGr
8bfcb0ad59 [SoundCloud] Fix hardcoded client id 2021-07-28 20:33:21 +02:00
Tobi
027dc65434 pull request #683 from XiangRongLin/yt_throttling
[YouTube] Fix buffering by decoding n parameter of stream urls
2021-07-28 20:32:38 +02:00
Tobi
14c179f343
Merge pull request #692 from TeamNewPipe/fix/sc/id
[SoundCloud] Fix hardcoded client id
2021-07-28 18:28:07 +02:00
Tobi
394c02ad06
Merge pull request #683 from XiangRongLin/yt_throttling
[YouTube] Fix buffering by decoding n parameter of stream urls
2021-07-28 18:01:57 +02:00
TobiGr
785ff4aa32 [SoundCloud] Fix hardcoded client id 2021-07-28 17:02:56 +02:00
TobiGr
79f2d74b04 Fix test
The decryption code changes over time. Only check whether the n parameter was changed in the URL
2021-07-28 16:45:31 +02:00
TobiGr
d70adfdb8f Add methods for cache control to YoutubeThrottlingDecrypter. 2021-07-27 20:32:22 +02:00
TiA4f8R
d13f531b6f Use YoutubeThrottlingDecrypter also in getAudioStreams and getVideoOnlyStreams methods of YoutubeStreamExtractor
Without this commit, the n param is only decrypted for streams extracted in getVideoStreams (so only for streams in the formats object of the player response).
2021-07-27 17:28:32 +02:00
TiA4f8R
32055147e0 Do some code improvements
Use final where possible, annotate some methods and parameters as Nonnull and format new code to be in the 100 characters limit per line.
2021-07-27 17:24:27 +02:00
TobiGr
1c30a2725e Cache nParams to prevent executing the JavaScript function for the same nParam multiple times.
Closes #689
2021-07-22 19:40:12 +02:00
XiangRongLin
3a3d1d7f2b Make YoutubeJavaScriptExtractor and JavaScript methods static
Also address review and rewrite some comments
2021-07-20 20:48:11 +02:00
XiangRongLin
a683c8d278 Delete duplicated code to load youtube javascript file 2021-07-17 19:14:57 +02:00
XiangRongLin
a02ee2e952 Rewrite youtube throttling solution and add tests 2021-07-17 19:10:09 +02:00
litetex
0c12b396e5 Review changes 2021-07-17 16:41:12 +02:00
litetex
ecf4232ce3 Using `Collections.emptyList()` 2021-07-17 16:41:12 +02:00
litetex
0f9ed020a4 Frameset has to implement Serializable or NewPipe fails
java.lang.RuntimeException: Parcelable encountered IOException writing serializable object (name = org.schabi.newpipe.extractor.stream.StreamInfo)
...
Caused by: java.io.NotSerializableException: org.schabi.newpipe.extractor.stream.Frameset
...
2021-07-17 16:41:11 +02:00
litetex
17ccaf4b87 Update YoutubeStreamExtractor.java
To fix ``java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String[] java.lang.String.split(java.lang.String)' on a null object reference``
2021-07-17 16:41:10 +02:00
litetex
4e41e172ea Update Frameset.java
Reformatted code
2021-07-17 16:41:10 +02:00
litetex
09b670d745 Renamed frames to previewFrames (so it's better understandable) 2021-07-17 16:41:09 +02:00
litetex
b086655d54 Reverted useless change 2021-07-17 16:41:08 +02:00
litetex
93b0c5b15f Added seekbar data 2021-07-17 16:41:08 +02:00
XiangRongLin
6956b72af7 Fix YoutubeThrottlingDecoder.replaceNParam
Previously it replaced the parameter itself not the value of the parameter.
2021-07-17 10:05:14 +02:00
XiangRongLin
a189f685dc cache player js code 2021-07-16 19:20:40 +02:00
XiangRongLin
fcdb9bdbeb add url to sout 2021-07-15 20:05:26 +02:00
XiangRongLin
80cf8b3acd Extract separate YoutubeThrottlingDecoder 2021-07-15 20:05:25 +02:00
XiangRongLin
a86a30103f [Youtube] bare bones version to solve throttling
Done by transforming the parameter "n" from videoplayback urls
https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-865985377
2021-07-15 20:05:25 +02:00
bopol
c38a06e8dc
Merge pull request #661 from B0pol/peertube-livestreams
[PeerTube] Support livestreams
2021-07-13 15:48:13 +02:00
litetex
fdebf3c6cd Added javadoc warnings 2021-07-07 20:41:59 +02:00
litetex
6860543b07 Added more doc 2021-07-06 21:16:31 +02:00
litetex
a59fead0d7 Fixed typos 2021-07-06 21:16:30 +02:00
litetex
688a1c316b Fixed Exception when YT comments are disabled and added `commentsDisabled` field
* Fixed code: Added missing finals (according to NewPipes Checkstyle guide)
* Fixed ``findValue`` method in ``YoutubeCommentsExtractor``
2021-07-06 21:16:29 +02:00
litetex
ca33f4f60b Improved code format (final) and added `isCommentsDisabled` method 2021-07-06 21:16:28 +02:00
TobiGr
dbdc962a07 Update mocks 2021-07-03 13:30:26 +02:00
TobiGr
9ca6dc26ed Fix invalid CONSENT cookie value 2021-07-03 13:27:40 +02:00
XiangRongLin
b9549ace25 [Youtube] Fix lofi girl test and ignore mix with id test 2021-06-29 21:12:20 +02:00
XiangRongLin
5982431ffb Remove IPv4 addresses from response during mock recording 2021-06-29 21:03:33 +02:00
bopol
7d7cc087e2 [PeerTube] Support livestreams 2021-06-24 11:02:30 +02:00
bopol
3966178979
Merge pull request #665 from B0pol/agerestricted
[youtube] Fix extraction of age restricted videos
2021-06-23 21:07:14 +02:00
bopol
b475f09ba7 don't do useless concatenation 2021-06-23 20:33:32 +02:00
bopol
b96b2a6eba Update invidious instances 2021-06-23 14:39:10 +02:00
bopol
c5e8bd368d Update Mocks 2021-06-23 12:21:57 +02:00
bopol
e9a992b0a9 [youtube] Fix extraction of age restricted videos 2021-06-23 12:06:02 +02:00
bopol
65fd8740aa
Merge pull request #658 from TeamNewPipe/revert-651-fix/streamType
[YouTube] Fix getting stream type of live streams
2021-06-23 11:33:24 +02:00
bopol
361f142621 Use badge style instead of label 2021-06-23 11:25:59 +02:00
bopol
58f109ddec adress stypox review 2021-06-23 11:15:40 +02:00
TobiGr
1f9a99c931 [SoundCloud] Update HARDCODED_CLIENT_ID 2021-06-18 15:30:54 +02:00
Tobi
c998012c28
Revert "[YouTube] Fix getting stream type of live streams" 2021-06-18 09:20:32 +02:00
TobiGr
6762fe3357 [YouTube] Fix getting stream type of live streams 2021-06-10 16:51:00 +02:00
TobiGr
f7f727d19c Add constant NO_LIKE_COUNT to CommentsInfoItem 2021-06-08 10:28:44 +02:00
TobiGr
b70c0f93c7 Add streamPosition for comments
SoundCloud is the only service which supports adding comments at a specific timestamp in the stream.
2021-06-08 10:27:52 +02:00
Tobi
d4186d100b
Merge pull request #580 from TeamNewPipe/accountTerminated
Add AccountTerminatedException for terminated channels
2021-06-08 09:55:58 +02:00
TobiGr
89e0514d8b Fix Javadoc generation fix 2021-06-08 09:30:05 +02:00
TobiGr
80d3052033 Fix Javadoc generation 2021-06-08 09:26:01 +02:00
TobiGr
fa444c8298 Remove assertations of exception messages 2021-06-06 11:38:43 +02:00
Stypox
2158ca4060
Remove two unneded warning suppressions 2021-06-05 14:15:48 +02:00
Stypox
13f192704d
Add utility method getService() to Info object 2021-06-05 14:14:19 +02:00
XiangRongLin
0ad51e76fb
Merge pull request #638 from TeamNewPipe/spelling
Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl)
2021-06-04 18:23:31 +02:00
XiangRongLin
faa503c7dd Fix PeertubeAccountExtractorTest 2021-06-03 19:54:42 +02:00
TiA4f8R
ac31f3a883
Specify UTF-8 file encoding in RecordingDownloader and MockDownloader
On Windows, mocks are recorded and read with the Cp1252 encoding so it breaks the mocks on non ASCII characters for Linux OS (and so the CI).
The project is in Java 8, so we can't use FileReader(File, Charset) and FileReader(File, Charset) because these methods require Java 11. Instead of changing the Java version of the extractor, use OutputStreamWriter and FileOutputStream instead of FileWriter and InputStreamReader and FileInputStream instead of FileReader.
2021-06-03 18:38:25 +02:00
bopol
ff11c2df2a
Merge pull request #628 from litetex/fix-broken-yt-liked-comments
Fix broken yt likes in comments
2021-05-29 11:04:11 +02:00