Commit Graph

2658 Commits

Author SHA1 Message Date
AudricV
81c0d80a54
[PeerTube] Add utility methods to get avatars and banners of accounts and channels
Four new static methods have been added in PeertubeParsingHelper to do so:
- two public methods to get the corresponding image type:
  getAvatarsFromOwnerAccountOrVideoChannelObject(String, JsonObject) and
  getBannersFromAccountOrVideoChannelObject(String, JsonObject);
- two private methods as helper methods: getImagesFromAvatarsOrBanners(String,
  JsonObject, String, String) and getImagesFromAvatarOrBannerArray(String,
  JsonArray).
2023-08-12 22:56:28 +02:00
AudricV
31da5beb51
[SoundCloud] Apply changes in Extractors 2023-08-12 22:56:28 +02:00
AudricV
a3a74cd566
[SoundCloud] Apply changes in InfoItemExtractors and return track user avatars as uploader avatars in SoundcloudStreamInfoItemExtractor 2023-08-12 22:56:28 +02:00
AudricV
7f818217d2
[SoundCloud] Add utility methods to get images from track JSON objects and image URLs
These new public and static methods, added in SoundcloudParsingHelper,
getAllImagesFromArtworkOrAvatarUrl(String) and
getAllImagesFromVisualUrl(String) (which call a common private method,
getAllImagesFromImageUrlReturned(String, List<ImageSuffix>, List<Image>)),
return an unmodifiable list of JPEG images containing almost every image
resolution provided by SoundCloud except the original size and the tiny
resolution (for artworks and avatars, as the image size is 20x20 for artworks
and 18x18 for avatars, so very close to or equal to the t20x20 resolution):

- for artworks and avatars:
  - mini: 16x16;
  - t20x20: 20x20;
  - small: 32x32;
  - badge: 47x47;
  - t50x50: 50x50;
  - t60x60: 60x60;
  - t67x67: 67x67;
  - large: 100x100;
  - t120x120: 120x120;
  - t200x200: 200x200;
  - t240x240: 240x240;
  - t250x250: 250x250;
  - t300x300: 300x300;
  - t500x500: 500x500.

- for visuals/user banners:
  - t1240x260: 1240x260;
  - t2480x520: 2480x520.

Duplicated code in two methods of SoundcloudParsingHelper
(getUsersFromApi(ChannelInfoItemsCollector, String) and
getStreamsFromApi(StreamInfoItemsCollector, String, boolean)) has been merged
into one common private method, getNextPageUrlFromResponseObject(JsonObject).
2023-08-12 22:56:28 +02:00
AudricV
266cd1f76b
[YouTube] Apply changes in YoutubeMusicSearchExtractor and split its InfoItemExtractors into separate classes
Splitting YoutubeMusicSearchExtractor's InfoItemExtractors into separate
classes (YoutubeMusicSongOrVideoInfoItemExtractor,
YoutubeMusicAlbumOrPlaylistInfoItemExtractor and
YoutubeMusicArtistInfoItemExtractor) allows to simplify
YoutubeMusicSearchExtractor,improves reading and applying changes to InfoItems
(no more losing at least quarter of a line due to indentations).

These InfoItems, in which the image changes have been applied, don't extend the
YouTube ones anymore, as most methods were overridden and the few ones that are
not don't apply in YouTube Music items responses, so it was useless to extend
them.

The code of YoutubeMusicSearchExtractor have been also improved a bit.
2023-08-12 22:56:27 +02:00
AudricV
c1981ed54f
[YouTube] Apply changes in Extractors except YoutubeMusicSearchExtractor
Also improve a bit some code related to the changes.
2023-08-12 22:56:27 +02:00
AudricV
4cc99f9ce1
[YouTube] Apply changes in InfoItemExtractors except YouTube Music ones 2023-08-12 22:56:27 +02:00
AudricV
adfad086ac
[YouTube] Add utility methods to get images from InfoItems and thumbnails arrays
Unmodifiable lists of Images are returned, parsed from a given YouTube
"thumbnails" JSON array.

These methods will be used in all YouTube extractors and InfoItems, as the
structures between content types (videos, channels, playlists, ...) are common.
2023-08-12 22:56:27 +02:00
AudricV
d56b880cae
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Infos 2023-08-12 22:56:26 +02:00
AudricV
9d8098576e
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Extractors 2023-08-12 22:56:26 +02:00
AudricV
0f4a5a8184
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemsCollectors 2023-08-12 22:56:26 +02:00
AudricV
ca1d4a6fa4
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemExtractors 2023-08-12 22:56:26 +02:00
AudricV
2f3ee8a3f2
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItems 2023-08-12 22:56:25 +02:00
AudricV
78ce65769f
Add an ImageSuffix class to the extractor
The goal of this utility class is to simply store suffixes which need to be
appended to image URLs, in order to get images at the suffix resolution.

This class contains four properties: the suffix (as a string), the height,
the width (as integers) and the estimated resolution level of the image
corresponding to the one represented by the suffix.
2023-08-12 22:56:25 +02:00
AudricV
d85454186a
Add an Image class to the extractor
Objects of this serializable class contains four properties: a URL (as a
string), a width, a height (represented as integers) and an estimated
resolution level, which can be constructed from a given height.

Possible resolution levels are:
- UNKNOWN: for unknown heights or heights <= 0;
- LOW: for heights > 0 & < 175;
- MEDIUM: for heights >= 175 & < 720;
- HIGH: for heights >= 720.

Getters of these properties are available and the constructor needs these four
properties.
2023-08-12 22:56:25 +02:00
Stypox
7294675aea
Merge pull request #1093 from AudricV/yt_support-shorts-ui-playlists
[YouTube] Support Shorts UI in playlists
2023-08-12 11:11:36 +02:00
Stypox
93a90b816d
Merge pull request #1094 from AudricV/yt_support-more-channel-headers
[YouTube] Support more channel headers
2023-08-12 11:08:30 +02:00
Stypox
44b664af15
[YouTube] Simplify Optional chains in channel 2023-08-12 11:02:51 +02:00
AudricV
2f7bfd3e7f
[YouTube] Add mocks of interactiveTabbedHeaderRenderer channel header test 2023-08-08 19:12:29 +02:00
AudricV
b147904571
[YouTube] Add test for interactiveTabbedHeaderRenderer channel header
This test uses the Minecraft game topic channel.
2023-08-08 19:12:28 +02:00
AudricV
1852031a0b
[YouTube] Support pageHeaderRenderer and interactiveTabbedHeaderRenderer channel headers
The addition of this support required to turn the isCarouselHeader boolean into
an enum containing all supported channel headers named HeaderType.

Also assert that the page has been fetched where needed to avoid
NullPointerExceptions when the channel page has been not fetched and remove the
getChannelHeaderJson method in YoutubeChannelExtractor, method for which its
code has been moved to its sole usage after the new headers support changes.
2023-08-08 19:12:27 +02:00
AudricV
698c710685
Do not require knowledge of uploader in default StreamInfoItems tests
This change is required as some services can return no uploader info, such as
YouTube for playlists with a Shorts UI.
2023-08-07 19:43:15 +02:00
AudricV
8237052ef5
Fix wrong assertion in assertNotEmpty
The non-null assertion was made on the exception message instead of the string
to check, causing a NullPointerException if the string to check was null.
2023-08-07 19:43:09 +02:00
AudricV
162c261577
[YouTube] Add mocks of the playlist with Shorts UI test 2023-08-07 19:07:53 +02:00
AudricV
e2f4ee47b9
[YouTube] Add a playlist with Shorts UI test
The system Shorts videos uploads playlist of the YouTube official channel has
been chosen for this test.
2023-08-07 19:06:09 +02:00
AudricV
e6f371fb94
[YouTube] Support Shorts UI in playlists
Also remove an outdated A/B test comment.
2023-08-07 19:01:08 +02:00
Stypox
7936987955
Merge pull request #1092 from Stypox/channel-tabs-improvements
Channel tabs code improvements
2023-08-06 21:44:10 +02:00
Stypox
6d2227111f
[YouTube] Assert that videos tab is ready after channel fetching 2023-08-06 21:14:57 +02:00
Stypox
ee625c325c
Inherit from DefaultListExtractorTest in channel tab tests 2023-08-06 21:14:56 +02:00
Stypox
276c293889
Rename assertTabsContain 2023-08-06 21:14:56 +02:00
Stypox
9d3761a371
[YouTube] Directly use playlist collector in channel tabs wrapper
Note that this introduces a "Raw use of parameterized class 'InfoItemsPage'" warning, but it can be ignored since the type missing would be <InfoItem>, and StreamInfoItem extends InfoItem
2023-08-06 21:13:25 +02:00
Stypox
95a3cc0a17
Merge pull request #1083 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.0
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
2023-08-06 18:58:27 +02:00
Stypox
e34b4f1978
[YouTube] Avoid using Consumer 2023-08-06 13:02:31 +02:00
Stypox
ef67c7cd74
[YouTube] Simplify usage of channel header json 2023-08-06 13:02:31 +02:00
Stypox
a104cf3227
[YouTube] Fix docs in channel helper 2023-08-06 13:02:31 +02:00
Stypox
bb47f05f89
Merge pull request #1091 from Stypox/update-mocks
[YouTube] Update stream mocks after #1087
2023-08-06 12:58:24 +02:00
Stypox
468bcc045d
[YouTube] Update mocks after #1087 2023-08-06 12:33:04 +02:00
Stypox
35f3a4ad01
Merge pull request #1082 from AudricV/channel-tabs-and-tags-support
Add support for channel tabs and channel tags
2023-08-06 12:21:42 +02:00
AudricV
e7d64099a7
[YouTube] Update channel mocks and add channel tabs mocks 2023-08-06 12:15:06 +02:00
AudricV
684101c47d
[YouTube] Implement age-restricted channels support, link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
eaf2600ce0
[SoundCloud] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
0ee2072de5
[PeerTube] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
d3801dd0e9
[MediaCCC] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
8baec04611
[Bandcamp] Implement link handlers and channels tabs and tags changes on tests
Tests in BandcampChannelExtractorTest and BandcampChannelLinkHandlerFactoryTest
have been also fixed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
e0ba29cd19
Add utility method to assert that given channel tabs are in the ones returned by a channel extractor
Only the first content filter of the ListLinkHandler instances provided is
used when collecting all channel tabs of the ListLinkHandler list, as channel
tabs implementations only use one content filter per ListLinkHandler instance.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
18846baba7
Add tabs and tags methods in tests interfaces and annotate all methods with the Test JUnit annotation
These changes should help to detect tests as tests, when running a subset of
tests or all tests.
They should be also implemented in these interfaces' implementations (new and
existing ones).

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
ThetaDev
c70a0e3543
Add a test for textual durations parsing using TimeAgoParser's patterns 2023-08-06 12:15:06 +02:00
AudricV
7366eab156
[YouTube] Add support for channel tabs and tags and age-restricted channels
Support of tags and videos, shorts, live, playlists and channels tabs has been
added for non-age restricted channels.

Age-restricted channels are now also supported and always returned the videos,
shorts and live tabs, accessible using system playlists. These tabs are the
only ones which can be accessed using YouTube's desktop website without being
logged-in.

The videos channel tab parameter has been updated to the one used by the
desktop website and when a channel extraction is fetched, this tab is returned
in the list of tabs as a cached one in the corresponding link handler.

Visitor data support per request has been added, as a valid visitor data is
required to fetch continuations with contents on the shorts tab. It is only
used in this case to enhance privacy.

A dedicated shorts UI elements (reelItemRenderers) extractor has been added,
YoutubeReelInfoItemExtractor. These elements do not provide the exact view
count, any uploader info (name, URL, avatar, verified status) and the upload
date.

All service's LinkHandlers are now using the singleton pattern and some code
has been also improved on the files changed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:15:04 +02:00
AudricV
4586067934
Add utility method to parse textual durations using TimeAgoParser's patterns
This is required to parse duration of YouTube's reelItemRenderers, returned
only inside accessibility data.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:13:33 +02:00
AudricV
d4bfe791ee
[SoundCloud] Add tabs support for users
Support of tracks, playlists and albums has been added for users.

Also add the declaration of the UnsupportedOperationException exception to the
service's LinkHandlers.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:13:32 +02:00