1909 Commits

Author	SHA1	Message	Date
TobiGr	aff3e795f8	[PeerTube] Fix multi level comment replies	2023-05-07 22:49:14 +02:00
ThetaDev	3673d4ae01	fix: YouTube trending name extraction	2023-05-03 21:16:35 +02:00
ThetaDev	0addb98cd7	tests: update mocks	2023-05-03 21:16:35 +02:00
ThetaDev	24eba62305	fix: extract YouTube trends from new "Videos" tab	2023-05-03 21:16:23 +02:00
Kavin	a9ca5c49e4	Merge pull request #1056 from AudricV/yt-improve-search-suggestions-extraction ... [YouTube] Switch to new search suggestion domain and improve error handling	2023-05-02 20:17:48 +01:00
dependabot[bot]	108f8a7a17	Bump org.jsoup:jsoup from 1.15.4 to 1.16.1 ... Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.15.4 to 1.16.1. - [Release notes](https://github.com/jhy/jsoup/releases) - [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES) - [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.4...jsoup-1.16.1) --- updated-dependencies: - dependency-name: org.jsoup:jsoup dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2023-05-01 09:58:32 +00:00
AudricV	cf6040ddb3	Update YoutubeSuggestionExtractorTest mocks	2023-04-30 19:53:44 +02:00
AudricV	593122342f	[YouTube] Improve YoutubeSuggestionExtractorTest ... - Remove useless concatenation on the downloader path; - Remove unneeded public test modifier; - Update license header; - Specify the service class tested instead of the generic class.	2023-04-30 19:53:43 +02:00
AudricV	e923fca440	[YouTube] Switch to new search suggestion domain and improve error handling ... - Switch to the new domain used by YouTube for search suggestions, suggestqueries-clients6.youtube.com, and add the xhr query parameter with the t value, to allow getting responses without requiring trim; - Use the Java 8 Stream API to collect search suggestions and improve invalid response detection by checking whether the content type of the response returned is JSON; - Move the licence header at the top of the file.	2023-04-30 19:53:42 +02:00
AudricV	945165a3c0	[PeerTube] Don't return "No description" when there is no description for a channel or an account ... When a description is missing, no description should be returned, even the ones indicating there is no description. This behavior is represented by a null return instead. Also update PeertubeAccountExtractorTest to reflect these changes.	2023-04-30 18:41:38 +02:00
Stypox	2deb023da4	Merge pull request #1050 from Theta-Dev/fix/channel-carousel-header ... [YouTube] Add support for CarouselHeaderRenderer	2023-04-25 15:17:31 +02:00
ThetaDev	4aada7f91b	refactor: rename carousel header channel test	2023-04-21 22:48:37 +02:00
ThetaDev	47aa9fed40	fix: set musicClientVersion regex capture group	2023-04-16 19:25:05 +02:00
ThetaDev	20370395c5	fix: add support for CarouselHeaderRenderer	2023-04-16 17:40:13 +02:00
Stypox	7dba6e3891	Merge pull request #1033 from petlyh/bandcamp-paywalled-content ... [Bandcamp] Handle paywalled tracks	2023-04-12 13:04:26 +02:00
petlyh	e6aad117e7	[Bandcamp] Throw PaidContentException on paywalled albums	2023-04-03 19:27:09 +02:00
fynngodau	69705138e4	[Bandcamp] Fix extraction of related playlist items URL (#1047) ... Small change in HTML structure	2023-04-02 22:24:29 +02:00
Björn Sigurbergsson	1b6fe5edd6	[YouTube] Fix ParsingException when comments are unavailable in a video (#1040) ... Co-authored-by: bjs <bjs@elect-it.com> Co-authored-by: Audric V. <74829229+AudricV@users.noreply.github.com> Co-authored-by: Kavin <20838718+FireMasterK@users.noreply.github.com>	2023-03-30 19:58:06 +02:00
ThetaDev	8d1303e18f	Add track types to audio streams (#1041)	2023-03-28 00:02:20 +02:00
AudricV	80a6fc2c63	[PeerTube] Fix testGetCommentsFromCommentsInfo test of PeertubeCommentsExtractorTest.Default ... The tested comment has been removed, so it couldn't be found in the comments list. This comment has been replaced by a new one from the current comments of the video. Also, in the parent class PeertubeCommentsExtractorTest, final has been used as much as possible and for-each loops of lists have been replaced by their forEach method or the Stream API, in order to simplify code.	2023-03-04 16:49:10 +01:00
petlyh	5a9b6ed2e3	[Bandcamp] Support loading additional comments (#1030)	2023-03-04 14:01:06 +01:00
Stypox	6bdd698c25	Merge pull request #1026 from AudricV/audio-streams-descriptive-and-locale-properties ... Add descriptive and locale properties to audio streams	2023-03-01 11:15:46 +01:00
Stypox	19e4b216c9	Merge pull request #1032 from AudricV/yt_fix-comments-hashtags-links-extraction ... [YouTube] Fix hashtags links extraction and escape HTML links	2023-03-01 10:47:37 +01:00
Stypox	b1298490c0	Merge pull request #1029 from AudricV/yt_fix-no-views-extraction-playlist-items ... [YouTube] Fix partial non-extraction of "No views" string in stream items	2023-03-01 10:46:52 +01:00
petlyh	9dc1832733	[Bandcamp] Handle paywalled tracks	2023-02-28 17:51:30 +01:00
fynngodau	3fdb6ee476	Merge pull request #1031 from petlyh/bandcamp-fix-radio-comments ... [Bandcamp] Show comments as disabled on radio streams	2023-02-27 20:50:12 +01:00
AudricV	bd79b921e8	[YouTube] Refactor the code to get stream items' view count ... This refactoring avoids code duplication as much as possible.	2023-02-27 10:25:46 +01:00
AudricV	51f9b39953	[YouTube] Fix partial non-extraction of no views string in stream items ... As the "No views" string is returned in the case there is no view on a video, a number cannot be parsed in this case, so -1 was returned. This string is now detected in all methods to get the view count of a stream.	2023-02-27 10:18:45 +01:00
AudricV	95b3f5e391	[MediaCCC] Test audio language property extraction	2023-02-26 19:06:18 +01:00
AudricV	30a0f8c510	[MediaCCC] Extract audio language property for single language audio tracks	2023-02-26 19:06:18 +01:00
AudricV	7f0269c4c7	[YouTube] Edit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage to test audio locale property ... The Hindi audio track language presence test has been changed from audio track label to audio locale.	2023-02-26 19:06:18 +01:00
AudricV	034f82dae7	[YouTube] Test language and descriptive audio in YoutubeDashManifestCreatorsTest	2023-02-26 19:06:17 +01:00
AudricV	05e8cb39f7	[YouTube] Add language and descriptive audio properties to DASH manifests	2023-02-26 19:06:17 +01:00
AudricV	bf30d70152	[YouTube] Add descriptive audio test ... This test uses video TjxC-evzxdk. Also improve a bit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test.	2023-02-26 19:06:17 +01:00
AudricV	76b7c19c5d	[YouTube] Extract whether a track is a descriptive audio and audio locale when available ... Also use audio track setters only for audio itags.	2023-02-26 19:06:17 +01:00
AudricV	3bb5eeef30	[YouTube] Add descriptive and locale audio support in ItagItem	2023-02-26 19:06:16 +01:00
AudricV	14bf3fb05b	Add ability to know the locale of an audio stream ... Getting audio tracks locales by parsing their ID or their label, should not be done by clients, but by the extractor. This commit adds the ability to store the Locale of an AudioStream, which is used to compare similar AudioStreams (in the equalStats method).	2023-02-26 19:06:16 +01:00
AudricV	f92426560c	Add descriptive audio properties ... Also improve AudioStream's audio language documentation	2023-02-26 19:06:16 +01:00
AudricV	a63f289667	[YouTube] Update mocks of YoutubeCommentsExtractorTest.FormattingTest	2023-02-26 18:50:07 +01:00
AudricV	9483dcd9fa	[YouTube] Update mocks of YoutubeCommentsExtractorTest.RepliesTest	2023-02-26 18:43:36 +01:00
AudricV	1556adbb2d	[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links ... webCommandMetadata object is contained inside a commandMetadata one, so it is not accessible from the root of the navigationEndpoint object. The corresponding statement has been moved at the bottom of the specific endpoints parsing, as the webCommandMetadata object is present almost everywhere, otherwise URLs of some endpoints would have be changed, such as uploader URLs (from channel IDs to handles). As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were catching ParsingExceptions thrown by these methods had to be updated. URLs got in the HTML version of getTextFromObject are now escaped properly to provide valid HTML to clients. This has been also done for attribute descriptions, with the description text for this type of descriptions. As YouTube descriptions are in HTML format (except for the fallback on the JSON player response, which is plain text and only happens when there is no visual metadata or a breaking change), all URLs returned are escaped, so tests which are testing presence of URLs with escaped characters had to be updated (it was only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).	2023-02-26 18:43:36 +01:00
petlyh	f7a7a236fb	[Bandcamp] Show comments as disabled on radio streams	2023-02-23 18:42:43 +01:00
dependabot[bot]	f5599ff08d	Bump org.jsoup:jsoup from 1.15.3 to 1.15.4 ... Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.15.3 to 1.15.4. - [Release notes](https://github.com/jhy/jsoup/releases) - [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES) - [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.3...jsoup-1.15.4) --- updated-dependencies: - dependency-name: org.jsoup:jsoup dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2023-02-20 10:05:26 +00:00
TobiGr	3f7df9536e	[YouTube] Fix getting the comment text if the comment contains a hashtag	2023-01-29 20:33:51 +01:00
Stypox	999fb7f812	Merge pull request #1024 from AudricV/snd_fix-tracks-like-count ... [SoundCloud] Fix extraction of tracks like count	2023-01-29 10:52:54 +01:00
Stypox	3519d4c367	Merge pull request #1015 from AudricV/yt_fix-channel-id-rss-feeds ... [YouTube] Fix channel ID extraction of YouTube channels RSS feeds	2023-01-29 10:41:38 +01:00
Stypox	9aca710e86	Merge pull request #1013 from Stypox/fix-music-mixes ... [YouTube] Now music mixes can be treated as normal mixes	2023-01-29 09:48:51 +01:00
Stypox	76eeabac45	Merge pull request #1020 from TeamNewPipe/fix/yt-subscriber-count ... [YouTube] Fix NPE in search when getting channel items without subscriber count	2023-01-29 09:44:22 +01:00
AudricV	676622f6df	[SoundCloud] Fix expectedLikeCountAtLeast tests of SoundcloudStreamExtractorTest test classes ... As like count is now returned by the extractor, we need to assert a positive minimum like count, which is close to the actual value, in order to avoid test failures due to lower like counts than the ones excepted.	2023-01-29 01:08:02 +01:00
AudricV	2a24d407d5	[SoundCloud] Fix extraction of tracks like count ... SoundCloud is using likes_count to return the like count of a track, like it was the case before they switched to favoritings_count.	2023-01-29 01:00:49 +01:00
AudricV	ba24976e41	[YouTube] Add live URLs test and do minor improvements to YoutubeStreamLinkHandlerFactoryTest ... - Remove unused imports; - Replace wildcard imports by single class imports; - Suppress "HTTP links are not secured" warnings from IDEA IDEs; - Replace removed video jZViOEv90dI by an existing video, 9Dpqou5cI08 (the corresponding test has been of course renamed).	2023-01-28 19:36:21 +01:00
AudricV	57f850bc2d	[YouTube] Support live URLs and do minor improvements to YoutubeStreamLinkHandlerFactory ... - Move license header at the top; - Use an unmodifiable set for the subpaths instead of a modifiable list; - Add missing Nonnull and Nullable annotations; - Improve exception messages.	2023-01-28 19:36:20 +01:00
AudricV	1f4ed9dce9	[YouTube] Fix channel ID extraction of YouTube channel RSS feeds ... The yt:channelId element doesn't provide the channel ID anymore and is empty, like the id element, so we need now to extract it from the channel URL provided in two elements: author -> uri and feed -> link. Also avoid a NullPointerException in getUrl and getName methods.	2023-01-28 11:53:33 +01:00
Tobi	c589a2c1a2	Merge pull request #1014 from TeamNewPipe/fix/yt-comments ... [YouTube] Fix getting next comments pages	2023-01-27 11:14:55 +01:00
TobiGr	72573932cf	[YouTube] Fix NPE in search when getting channel items without subscriber count	2023-01-24 23:03:45 +01:00
TobiGr	f50b7275af	[YouTube] Fix getting next comments pages	2023-01-24 22:39:08 +01:00
Kunal	9bdad40b06	Removed topStandaloneBadge	2023-01-20 02:41:21 +05:30
Stypox	5945057227	[YouTube] Add music mix test	2023-01-15 23:30:30 +01:00
Stypox	7293991832	[YouTube] Now music mixes can be treated as normal mixes ... Using a playlist extractor on them would result in "Unviewable playlist" errors	2023-01-15 23:28:59 +01:00
Stypox	ff94e9f30b	Merge pull request #1009 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10.1 ... Bump gson from 2.10 to 2.10.1	2023-01-11 15:35:36 +01:00
Stypox	c1040bccac	Merge pull request #794 from FireMasterK/comments-count ... [YouTube] Add support to extract total comment count	2023-01-11 15:32:19 +01:00
dependabot[bot]	f43049985e	Bump gson from 2.10 to 2.10.1 ... Bumps [gson](https://github.com/google/gson) from 2.10 to 2.10.1. - [Release notes](https://github.com/google/gson/releases) - [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md) - [Commits](https://github.com/google/gson/compare/gson-parent-2.10...gson-parent-2.10.1) --- updated-dependencies: - dependency-name: com.google.code.gson:gson dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2023-01-09 09:05:55 +00:00
TobiGr	56aab4d971	[YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject	2023-01-05 00:28:12 +01:00
Kavin	22a47da8c7	Fix requested change and remove outdated comment.	2023-01-02 20:42:32 +00:00
Kavin	98a90fd9c8	Don't cache comments count and return early on page fetch if no token.	2023-01-02 20:40:48 +00:00
Kavin	2974dfaa48	Only store ajaxJson for initial page and eager fetch the initial continuation.	2023-01-02 20:40:48 +00:00
Kavin	64d24aa09e	Fix request changes.	2023-01-02 20:40:48 +00:00
Kavin	67ef4f4c30	Cleanup and remove optional.	2023-01-02 20:40:48 +00:00
FireMasterK	22f71b010c	Fix for requested changes.	2023-01-02 20:40:48 +00:00
FireMasterK	656b7c1cd9	Improve method documentation.	2023-01-02 20:40:48 +00:00
FireMasterK	981aee4092	Add support to extract total comment count.	2023-01-02 20:40:48 +00:00
Stypox	45636b0d00	Merge pull request #986 from Isira-Seneviratne/Static_maps ... Use immutable Map factory methods.	2023-01-02 18:11:14 +01:00
Stypox	219c5c5be5	Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java	2023-01-02 18:11:03 +01:00
Stypox	259de3cba6	Merge pull request #995 from TeamNewPipe/feat/soundcloud-playlistinfoitemextractor ... [SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor	2023-01-02 15:10:40 +01:00
Stypox	991394b53a	Merge pull request #1005 from FireMasterK/fix-escaping-xss ... Fix for potential XSS attacks and formatting issues	2023-01-02 15:06:17 +01:00
Isira Seneviratne	d8ce08d969	Use immutable Map factory methods.	2023-01-02 07:50:31 +05:30
Kavin	01acf79436	Fix for potential XSS attacks.	2022-12-31 20:05:32 +00:00
TobiGr	292e0d8ce7	[SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor	2022-12-31 18:46:39 +01:00
TobiGr	2a8729aeb2	Apply suggestions ... Co-authored-by: Stypox <stypox@pm.me>	2022-12-31 18:24:33 +01:00
TobiGr	d75a997611	[PeerTube] Support searching for channels	2022-12-31 18:24:33 +01:00
TobiGr	dea6d8ce4c	[PeerTube] Support searching for playlists	2022-12-31 18:24:33 +01:00
Stypox	95cc6aefbb	Merge pull request #994 from TeamNewPipe/fix/peertube-subtitles-exception ... [PeerTube] Report Exceptions thrown while getting a stream's subtitles	2022-12-31 15:01:39 +01:00
Stypox	7b54457789	Merge pull request #941 from TeamNewPipe/feat/peertube-comment-replies ... [PeerTube] Support comment replies	2022-12-31 14:57:51 +01:00
AudricV	f45966d449	Merge pull request #910 from Isira-Seneviratne/Locale_forLanguageTag ... Add compat Locale.forLanguageTag() implementation.	2022-12-24 23:53:30 +01:00
AudricV	d5437e0bc5	Merge pull request #863 from AudricV/add-content-type-and-content-length-headers-to-post-requests ... Add Content-Type header to all POST requests without an empty body	2022-12-16 19:32:56 +01:00
AudricV	0766b1d211	[YouTube] Improve YoutubeStreamInfoItemExtractor ... - Return duration of video premieres; - Add another non-localized method to determine whether a stream is a running livestream; - Return view count and upload date of videos in playlists; - Store isPremiere result; - Remove shorts workaround code, as it was only useful on channels and shorts have been moved into a separated channel tab; - Improve some other code.	2022-12-08 13:59:12 +01:00
Tobi	896d7e09eb	Merge pull request #978 from Theta-Dev/fix/search-channel-handles ... [YouTube] Fix search subscriber count extraction with channel handles	2022-12-05 17:52:05 +01:00
TobiGr	cd3262745d	[PeerTube] Report Exceptions thrown while getting a stream's subtitles	2022-12-03 16:11:21 +01:00
TobiGr	4e66b2287e	[PeerTube] Add support for comment replies	2022-12-01 14:05:18 +01:00
Tobi	41c8dce452	Merge pull request #992 from Isira-Seneviratne/String_isBlank ... Use String.isBlank().	2022-11-30 17:48:54 +01:00
Isira Seneviratne	2bca56f0df	Use String.isBlank().	2022-11-30 08:26:21 +05:30
Isira Seneviratne	3b80547976	Add code review suggestions.	2022-11-30 07:57:45 +05:30
ThetaDev	016623131e	docs: update comment in YoutubeChannelInfoItemExtractor	2022-11-29 19:06:03 +01:00
Kavin	2e08eaad96	Fix complication error in comment test.	2022-11-29 16:07:48 +00:00
Kavin	abf08e1496	Merge pull request #990 from FireMasterK/bold-italic-strikethrough ... [YouTube] Implement bold/italic/strike-through support	2022-11-29 15:59:38 +00:00
Kavin	57e7a6fb7c	Add mocks test.	2022-11-28 20:27:55 +00:00
Kavin	1d3d7fa5c3	Add test for formatting.	2022-11-28 20:26:37 +00:00
Kavin	52fda37915	Implement bold/italic/strike-through support.	2022-11-28 19:06:18 +00:00
Kavin	b566084cac	Use Description object for comments text.	2022-11-28 17:02:19 +00:00
Tobi	f8162b049d	Merge pull request #984 from FireMasterK/unused-dep ... Remove unused autolink dependency	2022-11-28 11:28:42 +01:00
Tobi	1da0190056	Merge pull request #980 from TeamNewPipe/fix/yt/unavailable ... [YouTube] Fix extracting the detailed error message for unavailable streams	2022-11-28 10:07:34 +01:00
Stypox	60fb30f835	Merge pull request #928 from FireMasterK/comment-urls ... Parse YouTube comments as HTML	2022-11-27 19:16:34 +01:00
Kavin	5abea22225	Fix throwing correct reason.	2022-11-26 21:09:08 +00:00
Kavin	faf28f5c11	Remove unused dependency.	2022-11-26 20:17:25 +00:00
Kavin	c043597255	Update supported countries list.	2022-11-26 19:01:33 +00:00
TobiGr	4680df0bdf	Fix throwing correct reason	2022-11-23 17:03:22 +01:00
TobiGr	9de8405c9f	[YouTube] Fix extracting the detailed error message of streams which are unavailable	2022-11-23 08:33:06 +01:00
Stypox	34d79bd267	[YouTube] Update mocks	2022-11-22 17:10:04 +01:00
AudricV	2ec296e674	Fix YoutubeSearchExtractorTest.MetaInfoTest ... Not all the "learn more" button is uppercase anymore, that's only the case for the first letter.	2022-11-22 16:34:54 +01:00
AudricV	3891542ca1	Use Downloader's postWithContentType and postWithContentTypeJson methods in services and extractors	2022-11-22 11:37:18 +01:00
AudricV	b2862f3cd1	Add postWithContentType and postWithContentTypeJson utility methods in Downloader ... Co-authored-by: Stypox <stypox@pm.me>	2022-11-22 11:37:17 +01:00
AudricV	e9a0d3bd95	[YouTube] Send Content-Type header in all POST requests ... This header was not sent partially before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet. Some code in the modified classes has been improved and / or deduplicated, and usages of the UTF_8 constant of the Utils class has been replaced by StandardCharsets.UTF_8 where possible. Note that this header has been not added in except in YoutubeDashManifestCreatorsUtils, as an empty body is sent in the POST requests made by this class.	2022-11-22 11:37:16 +01:00
AudricV	b9e463de49	[Bandcamp] Send Content-Type header in POST requests ... This header was not sent before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet. Also make use of StandardCharsets.UTF_8 when getting bytes of bodies instead of the platform default's charset, to make sure to prevent some encoding issues on some JVMs.	2022-11-22 11:35:46 +01:00
AudricV	65d6321e3d	Fix typos in Downloader.post JavaDocs ... Post methods in Downloader return the result of a POST request and not the one of a GET request.	2022-11-22 11:35:46 +01:00
ThetaDev	5daabd1793	fix: #976 search subscriber count extraction with channel handles	2022-11-22 02:17:10 +01:00
Kavin	c953e23414	Merge pull request #968 from AudricV/yt-support-no-video-info-renderers-for-streams ... [YouTube] Support lack of video info renderers for streams	2022-11-16 20:20:01 +00:00
Tobi	2211a24b69	Merge pull request #971 from lrusso96/patch-1 ... [YouTube] Improve duration parsing	2022-11-16 16:14:54 +01:00
Kavin	86f06b333a	Address review.	2022-11-14 00:05:31 +00:00
Kavin	b16e6082e1	Add test for audio stream languages.	2022-11-13 23:10:44 +00:00
Kavin	30909da1df	Fix audio track similar comparison for IDs.	2022-11-13 23:08:54 +00:00
Kavin	6d59cdbe3a	Add support for extracting audio tracks.	2022-11-13 21:39:29 +00:00
Isira Seneviratne	e4d982c7ea	Fix license.	2022-11-12 07:29:15 +05:30
Isira Seneviratne	416089146e	Fix failing tests.	2022-11-12 07:29:15 +05:30
Isira Seneviratne	ddbce3b83d	Add Utils methods for URL encoding/decoding using UTF-8.	2022-11-12 07:29:15 +05:30
Isira Seneviratne	366f5c1632	Use StandardCharsets.UTF_8.	2022-11-12 07:29:15 +05:30
dependabot[bot]	0b82fade51	Bump gson from 2.9.1 to 2.10 ... Bumps [gson](https://github.com/google/gson) from 2.9.1 to 2.10. - [Release notes](https://github.com/google/gson/releases) - [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md) - [Commits](https://github.com/google/gson/compare/gson-parent-2.9.1...gson-parent-2.10) --- updated-dependencies: - dependency-name: com.google.code.gson:gson dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2022-11-09 16:21:16 +00:00
Luigi Russo	c9635218e2	[YouTube] Improve duration parsing	2022-11-09 09:41:29 +01:00
Isira Seneviratne	316d8573fa	Use immutable sets in YoutubeParsingHelper.	2022-11-07 07:50:26 +05:30
AudricV	6a2c680d8f	[YouTube] Add mocks for YoutubeStreamExtractorDefaultTest.NoVisualMetadataVideoTest	2022-11-04 19:43:06 +01:00
AudricV	e66fed41d6	[YouTube] Add a StreamExtractor test for a video without visual metadata ... The video "Makani’s first commercial-scale energy kite" (video ID: An8vtD1FDqs), which has this behavior, is used for the new test, NoVisualMetadataVideoTest, added in YoutubeStreamExtractorDefaultTest. Tests of elements who throw an exception in this case (subscriber count, like count, uploader avatar URL) test if the ParsingException exception is thrown by YoutubeStreamExtractor.	2022-11-04 19:42:12 +01:00
AudricV	aa9a8ca23c	[YouTube] Make non-extraction of videoPrimaryInfoRenderer and/or videoSecondaryInfoRenderer not fatal ... Also de-duplicated common code related to the obtain of these video info renderers. This change allows extraction of videos without visual metadata.	2022-11-04 18:35:53 +01:00
AudricV	20cd8e8a4a	[YouTube] Update mocks of YoutubeChannelExtractorTest.Gronkh	2022-11-03 19:46:42 +01:00
AudricV	a34f060ba0	[YouTube] Use a handle for YoutubeChannelExtractorTest.Gronkh	2022-11-03 19:46:42 +01:00
AudricV	724f669ff7	[YouTube] Add tests for handles and user IDs with non ASCII characters support ... Unneeded public modifiers in test methods of YoutubeChannelLinkHandlerFactoryTest have been also removed.	2022-11-03 19:46:42 +01:00
AudricV	61ce041bda	[YouTube] Support handles and all custom channel names ... More non-channel paths have been also added to the excluded custom name paths, documentation and exception messages have been improved and fixed in some places, and the licence header of YoutubeChannelLinkHandlerFactory has been moved to its beginning and updated.	2022-11-03 19:46:42 +01:00
AudricV	ffffb04439	Merge pull request #953 from Theta-Dev/attributed-text-desc ... [YouTube] Add support for attributed text description	2022-11-03 18:34:30 +01:00
ThetaDev	592e1d6386	fix: parsing attributed description with no command runs	2022-11-03 12:10:52 +01:00
ThetaDev	099b53cc4f	[YouTube] Add parser for attributedDescription ... Also update the mock of the next InnerTube endpoint response of the YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing test class with an attributedDescription instead of a regular description	2022-11-02 23:11:33 +01:00
AudricV	e4c24d4c36	[YouTube] Regenerate supported channels mocks	2022-11-02 19:13:59 +01:00
Theta-Dev	20e4a35814	[YouTube] Support richGridRenderer on channel pages ... YouTube is deploying a new layout on their channel pages, which uses richGridRenderer JSON objects.	2022-11-02 19:01:29 +01:00
AudricV	4cae66f1f9	Merge pull request #946 from chowder/dev ... Add ability to identify short-form `StreamInfoItem`s	2022-11-01 12:19:58 +01:00
Tobi	eb40bb8458	Merge pull request #959 from FireMasterK/playlist-info-item-uploader ... Add uploaderUrl and uploaderVerified to PlaylistInfoItem.	2022-10-31 13:10:33 +01:00
chowder	b1a899fd47	Fix null pointer exception	2022-10-31 11:12:23 +00:00
Stypox	a4db106a66	Merge pull request #960 from AudricV/yt-workaround-403-errors-android-client ... [YouTube] Workaround 403 HTTP errors of ANDROID client streams	2022-10-30 21:53:26 +01:00
Kavin	b441910257	Mark uploaderUrl as nullable.	2022-10-30 13:36:04 +00:00
chowder	3fdc0e72cc	Line breaks for long docstrings	2022-10-30 13:28:39 +00:00
Kavin	6a256d0631	Add uploader url and verified to PlaylistInfoItem.	2022-10-30 13:00:19 +00:00
Tobi	430504b4b5	Merge pull request #958 from AudricV/yt-playlists-support-new-metadata-format ... [YouTube] Support new metadata format of playlists	2022-10-30 12:31:43 +01:00
Kavin	f9bd08c649	Address reviews.	2022-10-30 01:25:30 +00:00
Caleb	9282c3c13b	Fix exception message for YoutubeStreamInfoItemExtractor#isShortFormContent ... Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>	2022-10-30 01:23:15 +00:00
Caleb	c5216f7c12	Update docstring for StreamExtractor#isShortFormContent ... Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>	2022-10-30 01:23:15 +00:00
Caleb	04795fe5d2	Use Stream API for ShortFormContent#testShortFormContent ... Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>	2022-10-30 01:23:15 +00:00
chowder	975b38a0b9	Regenerate test mocks	2022-10-30 01:23:15 +00:00
chowder	4cccd33f3d	Implement isShortFormContent for StreamExtractor and StreamInfo	2022-10-30 01:23:15 +00:00
chowder	09544ceb23	Fix tests	2022-10-30 01:23:15 +00:00
chowder	644cc6cd76	Rename test function name	2022-10-30 01:23:15 +00:00
chowder	daf5674951	Add ability to identify short-form StreamInfoItems	2022-10-30 01:23:12 +00:00
Tobi	3d314169b9	Merge pull request #943 from TeamNewPipe/fix/sc/comments ... [SoundCloud] Fix getting more comments	2022-10-29 22:19:50 +02:00
AudricV	65125a3eb4	[YouTube] Fix YoutubePlaylistExtractorTest.LearningPlaylist ... The video count is now returned for this playlist, so it isn't unknown. The testStreamCount method of this test class asserts now that the stream count is greater than 40.	2022-10-29 18:12:10 +02:00
AudricV	7258a53225	[YouTube] Support new playlist layout ... This new layout doesn't provide author thumbnails and is completely different for metadata, so the code to get them has been refactored. The code of learning playlists video count check has been also removed, as it seems to be not relevant anymore (the video count seems to be returned for these playlists with both layouts). Finally, unneeded overrides of subchannel methods, which don't apply to the YouTube service, have been removed.	2022-10-29 18:12:10 +02:00
AudricV	e37e8b358e	[YouTube] Update mocks that need to be updated	2022-10-29 18:10:18 +02:00
AudricV	60e97cd274	[YouTube] Workaround getting streaming URLs returning 403 HTTP response codes ... Using the player parameters used to get stories seems to fix the issue, which affects currently only certain countries such as UK. This is a workaround and should be fixed in a better way (by changing the InnerTube additional client used for videos or finding what is now required in Android player requests).	2022-10-29 17:58:33 +02:00
AudricV	c230d84df1	Fix Checkstyle error in YoutubeCommentsInfoItemExtractor	2022-10-29 13:24:19 +02:00
xz-dev	0ffcb32d9c	[YouTube] Add comment reply count support (#936) ... Add comment reply count support for YouTube and introduce `CommentsInfoItem.UNKNOWN_REPLY_COUNT` constant Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com> Co-authored-by: Tobi <TobiGr@users.noreply.github.com>	2022-10-15 12:40:06 +02:00
Isira Seneviratne	b90a566dd8	Add backport implementation of Locale.forLanguageTag().	2022-10-12 09:21:39 +05:30
Isira Seneviratne	b232c29d22	Use Locale.forLanguageTag().	2022-10-12 09:21:38 +05:30
TobiGr	4d136599bd	[SoundCloud] Fix getting more comments	2022-10-11 15:44:54 +02:00
Tobi	a822e91909	Merge pull request #939 from TurtleArmyMc/fix/SoundcloudPlaylistExtractor_track_order ... Fix SoundcloudPlaylistExtractor: tracks are in correct order	2022-10-10 22:28:18 +02:00
TobiGr	02810a7db7	Add a comment	2022-10-10 22:22:12 +02:00
Tobi	54092fc3c7	Merge pull request #930 from Isira-Seneviratne/Avoid_Comparator_NPE ... Avoid possible NullPointerException in MediaCCCRecentKiosk.	2022-10-10 11:43:34 +02:00
TurtleArmyMc	bf70d32eb4	Fix SoundcloudPlaylistExtractor: tracks are in correct order	2022-09-30 15:26:25 -04:00
AudricV	8067c43837	[YouTube] Don't use a specific letter for the decryption function name pattern ... Use the same possible characters for variables everywhere, in order to avoid potential future throttling parameter decryption function name parsing issues related to the usage of other letter(s) than b.	2022-09-24 21:49:22 +02:00
AudricV	abcee87167	[YouTube] Fix throttling parameter decryption function regex ... - Quote the function name, as it may contain special regex symbols, such as dollar; - Support multiple lines; - Use what looks like the end of the function for the end of the regex (this part is inspired from yt-dlp throttling parameter decryption regex); - Move the throttling function body regex into a private and static constant.	2022-09-24 21:28:09 +02:00
ThetaDev	7244be7627	[YouTube] Add JavaScript lexer to parse completely throttling decryption function (#905)	2022-09-24 19:55:46 +02:00
Isira Seneviratne	0e31c86aee	Avoid possible NullPointerException in MediaCCCRecentKiosk.	2022-09-21 05:40:07 +05:30
Stypox	a99af9bb6e	Merge pull request #927 from Stypox/fix-feed-thumbnail ... [YouTube] Return mqdefault thumbnails in fast feed	2022-09-19 08:52:01 +02:00
Kavin	4e14644c41	Ensure comments are parsed with URLs and timestamps.	2022-09-17 16:03:39 +05:30
Stypox	93ef73e2f2	[YouTube] Return mqdefault thumbnails in feed ... The hqdefault thumbnails, which are used by default, have black bars at the top and at the bottom, and are thus not optimal.	2022-09-13 16:46:54 +02:00
AudricV	d8ddeb549c	[YouTube] Support new trending structure and filter recently trending section ... This new structure allow us to filter easily Trending shorts and Recently trending sections. On the previous one, this Recently trending section is now filtered, by checking whether sections have a title, which isn't the case for normal trends contrary to the other ones. This makes that the extractor returns now only the real 50 "Now" YouTube trends. Elements inside arrays are now extracted dynamically instead of only the ones of the first index, using Java 8's Stream API. The getInitialPage() method of YoutubeTrendingExtractor can now throw a ParsingException if no selected tab (corresponding to the one of the trends type extracted) has been found. Finally, the licence header has been moved to the top of the file and updated.	2022-09-11 18:52:12 +02:00
AudricV	884da40f65	Merge pull request #926 from AudricV/fix-yt-like-count-extraction ... [YouTube] Fix extraction of like count with the new data model	2022-09-11 12:21:38 +02:00
AudricV	119b9129e2	[YouTube] Fix extraction of like count with the new data model ... In the new data model currently A/B tested or deployed, the like count button data is the same, but its path has been changed. The extraction of the like count has been also improved, by using the multiple accessibility data available instead of the only one which was used before. This accessibility data has been also deprioritized, because it is language dependent when there is no view. Like count is always known, even for age-restricted videos, so returning -1 if the like count extraction fails on this type of videos is not needed and hides an extraction error: that's the reason why this return has been removed and the exception will always be thrown, even if a video is age-restricted.	2022-09-10 23:28:38 +02:00
ThetaDev	4905f74700	[YouTube] Fix throttling parameter decryption on Android ... Escape the curly brace in the regular expression used to parse the throttling parameter decryption function to allow its compatibility on Android.	2022-09-10 17:03:39 +02:00
Stypox	fa25a7280f	Downgrade Rhino since it uses a class not available on Android ... Reverts #774, see TeamNewPipe/NewPipe#8876	2022-08-25 09:49:13 +02:00
Stypox	25f0ce7305	Merge pull request #911 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.3 ... Bump jsoup from 1.15.2 to 1.15.3	2022-08-24 14:23:44 +02:00
dependabot[bot]	1d7a3a90a4	Bump jsoup from 1.15.2 to 1.15.3 ... Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.2 to 1.15.3. - [Release notes](https://github.com/jhy/jsoup/releases) - [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES) - [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.2...jsoup-1.15.3) --- updated-dependencies: - dependency-name: org.jsoup:jsoup dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2022-08-24 09:14:26 +00:00
Isira Seneviratne	41254ae12a	Remove annotation.	2022-08-24 06:59:17 +05:30
Isira Seneviratne	943b7c033b	Remove EMPTY_STRING.	2022-08-24 06:59:17 +05:30
litetex	d6577e5e0b	Merge pull request #882 from litetex/fix-all-tests ... Fix all tests	2022-08-23 16:19:27 +02:00
AudricV	03d9a4fe9d	Fix typos in YoutubeStreamExtractor.tryDecryptUrl	2022-08-21 21:17:03 +02:00
litetex	0beb55a232	Improve `YoutubeThrottlingDecrypterTest`	2022-08-21 20:17:05 +02:00
litetex	9e93d6b193	YoutubeThrottlingDecrypter: Check if returned string is a valid JavaScript function	2022-08-21 20:16:45 +02:00
litetex	ed0a07af4e	YoutubeThrottlingDecrypter: Patch regex	2022-08-21 20:15:53 +02:00
litetex	641a447d9b	Updated consent-related mock-data	2022-08-21 18:43:48 +02:00
litetex	8ff7a90f52	Improved consent cookie related constants and documentation	2022-08-21 18:41:40 +02:00
AudricV	cb64a480cd	[YouTube] Remove deprecated code in YoutubeThrottlingDecrypter ... YoutubeThrottlingDecrypter is now a non-instantiable final class and non-static attributes have been removed. The static attributes of this class have been renamed, in order to respect the naming specification used.	2022-08-16 15:01:05 +02:00
litetex	5bbea3a8f2	Fix `YoutubeMixPlaylistExtractorTest` again ... Doesn't work for kurgesagt videos anymore, used music video (copyright free) instead	2022-08-14 15:12:27 +02:00
litetex	da06166065	Updated mock data of `YoutubeSearchExtractorTest$Suggestion`	2022-08-14 15:12:26 +02:00
litetex	2e36ab1578	Fixed `YoutubeSearchExtractorTest$Suggestion`	2022-08-14 15:12:26 +02:00
litetex	2a8a623643	`YoutubePlaylistLinkHandlerFactoryTest`: Use parameterized tests	2022-08-14 14:48:31 +02:00
litetex	938e69a16a	Fixed `YoutubePlaylistLinkHandlerFactoryTest`	2022-08-14 14:48:30 +02:00