Commit Graph

2153 Commits

Author SHA1 Message Date
Stypox
23fa31a1ec
Merge pull request #843 from TiA4f8R/fix-checkstyle-issues
Remove Checkstyle suppressions file and fix Checkstyle issues
2022-05-02 22:07:27 +02:00
TiA4f8R
3c3cd78676
Remove Checkstyle suppressions file and fix Checkstyle issues introduced in 24e8399 and 8c1041d
The Checkstyle suppressions file is now replaced by // CHECKSTYLE:OFF and // CHECKSTYLE:ON comments.
2022-05-02 21:51:25 +02:00
Stypox
2e1c5c119d
Merge pull request #822 from Stypox/more-refactors
More refactors
2022-05-02 19:03:54 +02:00
Stypox
598ebb92ea
Merge pull request #839 from TeamNewPipe/bandcamp/extract-length
Bandcamp: extract stream length
2022-05-02 15:49:41 +02:00
litetex
5db4d1faf3
Merge pull request #782 from litetex/cleanup-yt-stream-extractor
Cleanup of ``YoutubeStreamExtractor`` and some related classes
2022-05-01 16:44:11 +02:00
litetex
fe30eb43a9 Cleanup `YoutubeStreamExtractor` and some related classes
* Fixed obvious sonar(lint) warnings
* Abstracted some code (get*Streams)
* Used some new lines to make code better readable
* Chopped down brace-jungle in some methods
* Use StandardCharset (Java 8 4tw)
2022-05-01 16:39:07 +02:00
Stypox
c2b5370517
Apply suggestions: improve switch and use EMPTY_STRING 2022-04-30 16:39:51 +02:00
Stypox
7c78c39230
Merge pull request #821 from litetex/cleanup-TimeAgoParser-java
Cleanup ``TimeAgoParser``
2022-04-30 16:20:09 +02:00
Stypox
ac1c22d81c
Release NewPipe Extractor 0.22.1 2022-04-28 11:05:52 +02:00
Stypox
3f5c8962ac
Merge pull request #841 from TiA4f8R/yt-respect-order-of-clients-for-streams
[YouTube] Use correct order of clients to get better streams first
2022-04-28 10:58:49 +02:00
TiA4f8R
9f9af35adb
[YouTube] Fix regression introduced in the order of streams used when adding more parameters to InnerTube requests, using the iOS client for livestreams and more 2022-04-25 20:23:04 +02:00
Fynn Godau
c38c016de5 Bandcamp: extract stream length 2022-04-24 21:24:19 +02:00
litetex
5f8f3929bd
Merge pull request #838 from Nickoriginal/update_user_agent
Update User-Agent in NPE
2022-04-24 14:52:18 +02:00
Nickoriginal
b5f3f9eb90
Update USER_AGENT to match the latest Firefox 2022-04-24 00:27:23 +03:00
litetex
9f665db4e1
Merge pull request #828 from TeamNewPipe/dependabot/github_actions/actions/setup-java-3
Bump actions/setup-java from 2 to 3
2022-04-23 00:09:44 +02:00
Stypox
52fa2d939a
Fix javadoc formatting error causing deployment to fail 2022-04-16 17:07:07 +02:00
Stypox
95a4b05acc
Release NewPipe Extractor 0.22.0 2022-04-16 16:57:58 +02:00
Tobi
b77c72fb88
Merge pull request #832 from Stypox/hotfix-decrypter
Fix YouTube throttling decrypter function parsing
2022-04-15 13:35:10 +02:00
Stypox
dcb7483dcf
Fix YouTube throttling decrypter function parsing 2022-04-15 13:10:19 +02:00
Robin
bebdc55ad4
Merge pull request #829 from TiA4f8R/yt-subtitles-age-restricted-videos-fix
[YouTube] Extract subtitles for age-restricted videos
2022-04-14 10:31:16 +02:00
TiA4f8R
ef49cd0007
[YouTube] Extract subtitles for age-restricted videos
Subtitles of age-restricted videos can be extracted since the InnerTube API migration, so there is no reason to not extract them anymore.
2022-04-11 22:09:56 +02:00
dependabot[bot]
00b51a2bf8
Bump actions/setup-java from 2 to 3
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 2 to 3.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-11 09:16:36 +00:00
TiA4f8R
2e92d718a2
Merge pull request #780 from TiA4f8R/yt-more-params-innertube-requests
[YouTube] Add more parameters to InnerTube requests, use the iOS client for livestreams and fix extraction of embeddable age-restricted videos and contents with a warning before playback
2022-04-09 18:15:18 +02:00
TiA4f8R
b30e341559
Add link to learn more about the issue which makes YoutubeStreamExtractorRelatedMixTest.testRelatedItems test disabled 2022-04-04 20:36:59 +02:00
TiA4f8R
70812fa611
Update YouTube stream mocks and disable YoutubeStreamExtractorRelatedMixTest
Mixes seems to be not given by YouTube anymore if you use a PENDING consent cookie value.
As mocks needs to updated, the test is always failing because of this change.
2022-04-02 19:28:49 +02:00
TiA4f8R
67288a0191
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.

This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.

The outdated code for these contents has been removed.

Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.

Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 19:06:36 +02:00
TiA4f8R
11b5a222c4
Deduplicate code of getStringResultFromRegexArray methods in Utils
Also revert indentation in Utils.mixedNumberWordToLong.
2022-04-02 18:40:00 +02:00
TiA4f8R
9ca647a750
Update mocks 2022-03-27 22:10:59 +02:00
TiA4f8R
dfa4239661
Fix missing imports and Checkstyle issues 2022-03-27 22:10:57 +02:00
TiA4f8R
6d27996ac4
Improve code of getStringResultFromRegexArray methods in Utils 2022-03-27 22:10:57 +02:00
TiA4f8R
2e3da445e6
[YouTube] Add documentation about parameters added and clients versions and key
Also move the iPhone device machine id to a constant, explain how it is used and move the licence in the header of the file, and fix missing imports in YoutubeStreamExtractor (due to a rebase issue).
2022-03-27 22:10:57 +02:00
TiA4f8R
1dad3bfe8b
[YouTube] Update again hardcoded client versions and update mobile user agents
Also provide ability to get mobile user-agents used for mobile InnerTube requests and deduplicate related code.
2022-03-27 20:52:40 +02:00
TiA4f8R
3d38459cf3
[YouTube] Reduce InnerTube response sizes by adding the prettyPrint parameter with the false value
InnerTube responses return pretty printed responses, which increase responses' size for nothing.

By using the prettyPrint parameter on requests and setting its value to false, responses are not pretty printed anymore, which reduces responses size, and so data transfer and processing times.
This usage has been recently deployed by YouTube on their websites.
2022-03-27 20:52:40 +02:00
litetex
349ba8db7f
Improve tests and randomness
- Use the existing RNG inside YoutubeParsingHelper
- Deduplicated test-setup for YouTube tests
- Minor improvements
2022-03-27 20:52:38 +02:00
TiA4f8R
52376949e5
Add setSeedForVideoTests method in YoutubeStreamExtractor tests
In order to use still use mocks with the generation of random strings in player requests, we need to use YoutubeParsingHelper.setSeedForVideoTests() method in every stream test.
2022-03-27 20:51:39 +02:00
TiA4f8R
d0d91e6690
Adress requested changes 2022-03-27 20:51:39 +02:00
TiA4f8R
b6bc521f0d
[YouTube] Update client versions again 2022-03-27 20:51:38 +02:00
TiA4f8R
26f93f5bb0
[YouTube] Extract streams of livestreams from the iOS client and disabled the Android client for livestreams
The iOS client is only enabled for livestreams and the Android client is now only enabled for videos, both by default.

A way to force, or not, the fetch of both clients have been added with two new static methods in YoutubeStreamExtractor.
2022-03-27 20:51:38 +02:00
TiA4f8R
7d07924de8
[YouTube] Try to use lighter requests when extracting client version and key from YouTube and YouTube Music
This is done by fetching https://www.youtube.com/sw.js for YouTube and https://music.youtube.com/sw.js for YouTube Music.

Two new methods in Utils class have been added which allow to try to get a match of regular expressions in a string array, or a Pattern array, on a content, on a specific index or 0.
Also some code refactoring has been made in this class.
2022-03-27 20:51:38 +02:00
TiA4f8R
05b7fee23b
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.

For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.

Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).

For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.

This commit also fixes a small bug with the Android User-Agent string.

Some code improvements have been also made.
2022-03-27 20:51:38 +02:00
TiA4f8R
83f374bff1
[YouTube] Update client versions and fix a bug when using resetClientVersionAndKey method
The boolean keyAndVersionExtracted in YoutubeParsingHelper was not set to false when resetting the client version and the key, which makes the extractor uses null on the next getting of the client version or the key if the clientVersion and the key were extracted before.
Also update client versions.
2022-03-27 20:51:38 +02:00
Stypox
8c1041def6
Add @ null annotations where Android Studio suggested it
That is, basically where the overriding function was missing an annotation from the base method.

Also apply renaming of emptyDescription to EMPTY_DESCRIPTION
2022-03-26 22:07:14 +01:00
Stypox
adbbdc7a5b
[YouTube] Fix regex warning: use ' {2}' instead of ' ' 2022-03-26 22:07:14 +01:00
Stypox
24e83997b4
[Bandcamp] Add Java 8 streams 2022-03-26 22:07:12 +01:00
Stypox
349990fd48
Fix redundant escape \\ in regex in Utils 2022-03-26 22:01:30 +01:00
litetex
3bf7aa3762 Cleanup `TimeAgoParser` 2022-03-26 21:09:31 +01:00
litetex
5a18730845
Merge pull request #813 from litetex/fix-test-2022-03
Fixed tests (+ Youtube shorts in channels)
2022-03-26 21:09:10 +01:00
litetex
af82edf9dc Fix checkstyle problems 2022-03-26 20:54:20 +01:00
litetex
29408f9356 `YoutubeMusicSearchExtractorTest$Suggestion` has no mock data 2022-03-26 20:52:28 +01:00
litetex
fcee247f9f Switched to mockonly tests
* Made the reason why the tests are mockonly mandatory
2022-03-26 20:52:28 +01:00