YouTube now returns a Shorts tab for InteractiveTabbedHeader gaming channels,
which contains Shorts about the game of the topic channel but are not uploaded
on the game topic channel.
As this tab is already supported by the extractor, fetching a gaming topic
channel now returns a tab instead of none.
Channel name and channel URL of these Shorts needs to be set to null in a
separate commit, as Shorts on this tab do not have the topic channel as their
uploader.
The new action data can return multiple contact actions instead of only one,
which will be concatenated by a new line return.
This commit fixes tests of the CrisisResources test class of
YoutubeSearchExtractorTest.
YouTube disabled the effectiveness of the parameters which were used (the
player response we get redirects to another video), but new parameters which
work around Android's client integrity checks have been found.
The API keys are not used anymore by official clients in almost all cases
(still used by the Android app until it gets a configuration) for all requests
we made.
Clients and device OS versions have been bumped to their latest stable version
known.
Methods and fields related to API keys have been renamed or deleted if they're
no longer relevant.
It’s not obvious that the function will fail in some cases and throw
an `IllegalArgumentException`.
So instead of just failing if parsing fails, return an Optional that
all callers have to decide what to do (e.g. the YoutubeExtractor can
just ignore the locale in that case, like it does with most other
fields in the json if they are unexpected).
i.e. without needing to pass through the conference/channel extractor
This was needed because clients (like NewPipe) might rely on link handlers to hold as little data as possible, since they might be kept around for long or passed around in system transactions, so this commit allows obtaining a standalone link handler that does not hold a JsonObject within itself.
YouTube provides that meta info panel when users search for really sensitive content like suicide (e.g. "blue whale").
It contains:
- an encouragement as title (e.g. "We are with you")
- a phone number as action
- details about how to call the phone number (e.g. availability)
- an url pointing to the website of an association
Also add a test that just checks if a meta info is properly extracted
This test only tests that search results are returned, when no content filters
are provided and crisis resources blocking search results should be returned.
Searches with blocking crisis resources and content filters should work too, as
the bypass has been implemented for them.
As search parameters to bypass crisis resources blocking search results have
been implemented, they need to be added to search tests, in order to pass
them.
The subscriber count is now lower than the expected count as some people
unsubscribed to the Sports system channel. The expected count has been so
lowered.
YouTube doesn't return anymore a suggestion for the query "algorythm", but does
for the query "on board ing" ("on boarding"). This search query is now used and
had to be URL-encoded.
URL encoding in the complete YoutubeSearchExtractorTest test class uses now
extractor's Util class instead of Java's URLDecoder class directly.
YouTube is rolling out or A/B testing a new date format returned inside player
responses, which are precise to the second instead of the day.
This commit makes the StreamExtractor tests use these more precise dates.
This commit fixes the testRelatedItems test method by:
- accepting consent in the test class, in order to extract mixes in
recommendations;
- removing assertion of a music mix inside the recommendations, as YouTube
doesn't seem to return such mixes anymore, at least for the video used in the
test class.
Replace the video used in this test class with another one publicly available
and update the corresponding expected test values.
The test class's mocks will be updated in a different commit.
- Change CarouselHeader test channel to Sports system one, as the Coachella one
doesn't return this channel header anymore;
- Fix InteractiveTabbedHeader test by checking whether the test's channel
description is not empty instead of containing some words, as it is changing
frequently.
Video's title and tags have been changed by its uploader, so they have to be
updated.
Also make some package-private constants private, as they are not used outside
of the class, and remove unneeded test overrides.
These crisis resources are preventing search results to be returned. See
https://support.google.com/youtube/answer/10726080?hl=en for more info on them.
This commit changes search parameters to include the property allowing to show
search results.
YouTube returns sometimes videos inside channel search results. As we only want
results corresponding to the type we requested, this commits makes
YoutubeSearchExtractor ignoring non-requested search results we get, using the
extractor LinkHandler's first content filter value.
Also remove an unneeded exception throwing declaration in
YoutubeSearchExtractor.
This query parameter for which its value is set to false was not added to two
requests made in test classes of YoutubeMixPlaylistExtractorTest.
Also remove an unneeded ParsingException exception throwing declaration in a
test method.
This should make returned dates consistent between timezones and countries on
which the extractor is ran.
It was previously only set on YouTube Music search continuations.
For every InnerTube request:
- Always add a `request` object with the following properties:
- "internalExperimentFlags" set to an empty array;
- "useSsl" set to "true";
- "lockedSafetyMode" set to "false".
- Use proper TODO comment to provide a way to enable restricted mode on every
request and add it on requests on which it wasn't present.
For YouTube Music:
- Remove alt query parameter, as it is not used anymore by the website;
- Add prettyPrint query parameter with false value on YouTube Music search
continuations.
Default image qualities were removed in image URLs with the jpg extension,
causing the addition of the image suffix to full non-JPG images URLs and so to
invalid image URLs.
Only the image quality name with its leading "-" character and the "."
character after the name is now removed and replaced by a string format
replaced itself with the image quality name for each quality.
As the image suffixes do not contain the image extension, the name of image
qualities lists has been adapted with these changes and some related comments
have been also improved.
Some services may provide different image formats using the same suffix,
without we know what format the service provide. Enforcing an image extension
could so lead to provide invalid image URLs, like for SoundCloud PNG images
currently.
With this documentation change, it is now clear that users of this class decide
of whether they want to include image extensions in the suffix. The previous
behavior described in the Javadoc was not enforced.
The signature timestamp is used as a number by HTML5 clients, so it should be
used in the same way by the extractor too instead of being a string.
As the timestamp doesn't seem to exceed 5 digits, an integer is used to store
its value.
