566 Commits

Author	SHA1	Message	Date
Mauricio Colli	0c27198ba1	Include errors from child collectors in the search collector	2020-03-21 15:55:29 -03:00
TobiGr	2cf8cbf0df	Merge branch 'master' into dev	2020-03-18 22:29:10 +01:00
wb9688	222d659d9e	[SoundCloud] Don't make separate request for getAudioStreams() in StreamExtractor ... Signed-off-by: Stypox <stypox@pm.me>	2020-03-18 11:30:15 +01:00
wb9688	904c9d681f	Fix YouTube videos with no views	2020-03-18 10:07:49 +01:00
Stypox	265cfb61f3	[SoundCloud] Add test for small playlist, use defaultTestGetPageInNewExtractor	2020-03-17 21:03:50 +01:00
Stypox	45bb646480	[SoundCloud] Do not overwrite nextPageUrl in PlaylistExtractor ... Consistent with YouTube and the documentation	2020-03-17 20:56:47 +01:00
Stypox	c505d4e2b7	[SoundCloud] Remove trailing , in playlist page urls	2020-03-17 20:36:59 +01:00
Stypox	ae47c9587c	[SoundCloud] Optimize imports in edited files	2020-03-17 20:32:04 +01:00
Stypox	4389fd3b7b	[SoundCloud] Migrate StreamExtractor to api-v2	2020-03-17 18:06:13 +01:00
Stypox	5e4ddb368f	[SoundCloud] Fix extractors built from next playlist pages ... They didn't have the information to calculate another next page url. So now `nextPageUrl` contains a full link with all video ids, and `getPage` takes the first part of the url (containing 15 streams) and produces another `nextPageUrl` with the remaining streams. Also add a test for this.	2020-03-17 18:04:40 +01:00
Stypox	0e1b4bbf17	[SoundCloud] Test playlists banner: it should not exist	2020-03-17 15:53:25 +01:00
Stypox	65bdb3bc9d	[SoundCloud] Unignore ignored playlist tests	2020-03-17 15:49:58 +01:00
Stypox	1558da6f6b	[SoundCloud] Fix playlist next page generation	2020-03-17 15:46:35 +01:00
Stypox	ca8bf53b61	[SoundCloud] Fix playlist test: number of streams changed	2020-03-17 15:35:33 +01:00
Stypox	d4aa4a0763	[SoundCloud] Fix typo in ChartsExtractor	2020-03-17 15:25:11 +01:00
Stypox	4b1121aac7	[SoundCloud] Add tests for api-v2 channel urls	2020-03-17 15:23:13 +01:00
Stypox	c3d811fde5	[SoundCloud] Use api-v2 in SubscriptionExtractor ... Also added	2020-03-17 15:22:25 +01:00
Stypox	d0e66cc600	[SoundCloud] Improve thumbnail url extraction in playlists ... Prevent NullPointerExceptions and remove duplicate code	2020-03-17 15:13:28 +01:00
Stypox	f3095713f9	[SoundCloud] Use api-v2 in PlaylistExtractor ... Rewrote methods to calculate next page url and to get items from it. `api-v2` is different from `api` since the initial playlist page contains (usually) the full info of the first 3 streams and only the id of the other. Then the single tracks can be requested in batch using `/tracks?ids=id1,id2,...`.	2020-03-17 15:12:13 +01:00
TobiGr	9eca7df947	Forget to fetch the page	2020-03-16 20:50:08 +01:00
TobiGr	edb57840d7	[SoundCloud] Update hard-coded client_id	2020-03-16 19:43:36 +01:00
TobiGr	efad0be99f	Test if the extractor can get AudiStreams with the hard-coded cliend_id ... SoundCloud client_id are changed on a regular basis now. We might want to remove the hardcoded id completely on a later point of time when client_id is changed to often	2020-03-16 19:42:35 +01:00
TobiGr	5f39bc4e99	[SoundCloud] Update hardcoded client id	2020-03-16 17:26:30 +01:00
Tobias Groza	65a7eda446	Merge pull request #256 from B0pol/mediaccc ... Support for non-api mediaccc links, invidious shortened links, refactor mediaccc tests	2020-03-15 17:04:09 +01:00
TobiGr	40b3bc138f	Remove hack to fix YoutubeStreamLinkHandlerFactoryTest of vnd.youtube videos	2020-03-14 21:54:54 +01:00
Tobias Groza	1c954fbded	Merge pull request #277 from XiangRongLin/vnd ... Handle youtube stream urls in "vnd.youtube:videoId" format.	2020-03-14 21:52:49 +01:00
Tobias Groza	9edd0f2675	Merge pull request #288 from Stypox/fix-unlisted-video ... [YouTube] Fix videos without next stream throwing error and add test for unlisted videos	2020-03-14 21:36:13 +01:00
Stypox	957b78ed9b	[YouTube] Add test for an unlisted video, which has no next stream	2020-03-14 14:04:01 +01:00
Stypox	325a51f73a	[YouTube] Do not crash if there is no next stream, but return null ... Like every other extractor does if there is no next stream	2020-03-14 14:02:48 +01:00
TobiGr	aea0962cfe	[SoundCloud] Update hardcoded client id	2020-03-14 12:41:44 +01:00
Mauricio Colli	b086e9db3f	[YouTube] Fix id extraction for some channels ... Some channels had no reliable way to get the redirected id in the response, so saving it for later was a valid alternative.	2020-03-14 02:39:49 -03:00
Mauricio Colli	00d1ed439b	[YouTube] Fix channel extraction when redirects are in the response ... Some redirects were embed directly into the response as instructions for the page, instead of the usual http redirects.	2020-03-14 02:39:48 -03:00
bopol	66518ec444	check wether mediaccc streams url are secure	2020-03-13 20:07:36 +01:00
bopol	0cd5e05b7b	MediaCCCLH: use substring instead of replace; improve a bit tests, return Collections.emptyList(); instead of null where it's annotated @NonNull	2020-03-13 20:00:49 +01:00
bopol	f742a6bd3e	code optimization	2020-03-13 20:00:49 +01:00
bopol	9701c7d800	invidious shortened links	2020-03-13 20:00:49 +01:00
bopol	e8e535b815	mediaccc: update linkhandlers & refactor tests	2020-03-13 20:00:49 +01:00
Mauricio Colli	70abd57852	[YouTube] Cache commonly used stream type result	2020-03-07 16:48:32 -03:00
Mauricio Colli	1ef706f567	[YouTube] Detect LIVE videos in the trending page	2020-03-07 16:48:31 -03:00
Xiang Rong Lin	cc8fb486ee	Handle youtube stream urls in "vnd.youtube:videoId" format. ... The official YouTube app accept intents in this format, which causes other developers to use it.	2020-03-02 20:40:33 +01:00
Mauricio Colli	6aa4d59b91	[PeerTube] Update video used in age limit test (the previous was 404)	2020-03-01 10:29:30 -03:00
Mauricio Colli	cfc278317d	[YouTube] Fail-fast if status exist and is anything other than "OK"	2020-03-01 10:29:29 -03:00
Mauricio Colli	90ae5fbea2	Improve kiosk tests across services	2020-03-01 12:57:18 +01:00
Mauricio Colli	e590417cc4	Test if services recognizes their own items urls	2020-03-01 12:57:18 +01:00
TobiGr	35252235b0	Go through all badges when checking if a StreamInfoItem is a live stream	2020-03-01 12:52:37 +01:00
Mauricio Colli	5686a6f562	[YouTube] Detect when a stream is deleted or doesn't exist ... Added a test case as well.	2020-03-01 12:49:13 +01:00
Mauricio Colli	e65333c3ce	[YouTube] Detect deleted/nonexistent/invalid channels and playlists ... - Added tests for these cases.	2020-03-01 12:49:13 +01:00
Mauricio Colli	98e359438a	[YouTube] Detect simple 404s in the standard fetch method	2020-03-01 12:49:13 +01:00
Mauricio Colli	408f042127	[YouTube] Fix bug when url isn't present in the browseEndpoint object	2020-03-01 12:49:13 +01:00
Mauricio Colli	342bdbb852	[YouTube] Avoid crashing by letting exceptions bubble up	2020-03-01 12:49:13 +01:00
Mauricio Colli	e9644e6216	[YouTube] Handle video premiere's date and duration	2020-03-01 12:49:13 +01:00
Mauricio Colli	2a470ac4f7	[YouTube] Handle videos with no views or with "Recommended to you" text	2020-03-01 12:49:13 +01:00
Mauricio Colli	da2c562eb0	[YouTube] Take into account videos that have their views hidden ... Also remove catch because it may hide bugs when the page changes.	2020-03-01 12:49:13 +01:00
Mauricio Colli	f66c20de54	Ignore null-keyed entries when iterating through the response headers	2020-03-01 12:49:13 +01:00
Mauricio Colli	5edd774fc4	Add latest url to the response to make detection of a redirect possible ... Will be latest one in this commit because there's need to check the history of redirects as of now.	2020-03-01 12:49:13 +01:00
Mauricio Colli	3441946bea	Make test downloader return a response instead of throwing an exception ... The test implementation was throwing an exception instead of just returning the response and letting the caller handle it.	2020-03-01 12:49:13 +01:00
wb9688	df219a9453	Don't use StandardCharsets class	2020-03-01 09:41:06 +01:00
wb9688	2807079531	Catch NumberFormatException instead of checking whether viewCount equals "Recommended for you"	2020-02-29 21:28:38 +01:00
wb9688	3e4d2fd64e	Improve testChannelUrl()	2020-02-29 18:20:35 +01:00
wb9688	bfe3eb1409	Use suggested try-if code style	2020-02-29 17:18:50 +01:00
wb9688	a6a63e9570	Let isHardcodedClientVersionValid() throw IOException	2020-02-29 17:01:36 +01:00
wb9688	45df8248bf	Use Localization in getJsonResponse()	2020-02-29 16:55:07 +01:00
wb9688	157055fbfd	Create getJsonResponse() function	2020-02-29 16:42:04 +01:00
wb9688	4f406512c2	Use YoutubeChannelLinkHandlerFactory	2020-02-29 16:26:28 +01:00
wb9688	4ff655352d	Reimplement getAgeLimit()	2020-02-28 17:14:26 +01:00
wb9688	6ba03e542b	Cache getVideoTab()	2020-02-28 17:05:31 +01:00
wb9688	8bfbae904d	Cache getVideo(Primary|Secondary)InfoRenderer()	2020-02-28 17:03:21 +01:00
wb9688	0b940f5cf4	Use "getAgeLimit() != NO_AGE_LIMIT" instead of "getAgeLimit() == 18"	2020-02-28 16:40:50 +01:00
wb9688	beb07a4852	Add test to check whether hardcoded client version is still valid	2020-02-28 16:36:14 +01:00
wb9688	b89b6b1db2	Check if hard-coded client version is valid in separate function	2020-02-28 16:35:24 +01:00
wb9688	7898ba230d	Use pbj in YoutubeStreamExtractor	2020-02-28 15:17:47 +01:00
wb9688	612ec06839	Actually fix thumbnail URLs starting with //	2020-02-28 09:36:33 +01:00
wb9688	f946310af5	Fix search queries with a promotedSparklesTextSearchRenderer returning 0 results	2020-02-28 08:57:44 +01:00
wb9688	94a2d5be46	Fix thumbnail URLs starting with //	2020-02-27 19:08:46 +01:00
wb9688	838d834ce2	Get Uploads view of Videos tab	2020-02-27 18:54:27 +01:00
wb9688	365b0329f3	Implement getTextFromObject() function	2020-02-27 17:39:23 +01:00
wb9688	0798bdd5cd	Fix getSubscriberCount() for 0 subscribers	2020-02-27 16:26:22 +01:00
wb9688	8ebd971648	Fix channel descriptions consisting of multiple parts	2020-02-27 10:06:35 +01:00
wb9688	880b951088	Fix getUploaderAvatarUrl() in YoutubeStreamExtractor	2020-02-27 09:53:59 +01:00
wb9688	8aa560bef4	Fix video titles consisting of multiple parts due to @	2020-02-27 09:42:22 +01:00
wb9688	951ed9f0d8	Don't call getUploaderInfo() in onFetchPage() for playlists ... This way YouTube Music playlists will keep working in NewPipe	2020-02-27 09:27:56 +01:00
wb9688	2399d46ab0	Fix YoutubeChannelExtractor for channels without videos	2020-02-27 09:21:25 +01:00
wb9688	cdbc751b87	Fix getUploaderUrl() in YoutubeStreamInfoItemExtractor	2020-02-27 09:05:41 +01:00
wb9688	db305408e9	Get all expandedShelfContentsRenderers in YoutubeTrendingExtractor	2020-02-26 20:37:20 +01:00
wb9688	a2d8489e16	Parse timestamps in description	2020-02-26 20:27:30 +01:00
wb9688	8d503407ef	Parse browseEndpoint and watchEndpoint in descriptions	2020-02-26 19:37:58 +01:00
wb9688	8dc39d517a	Fix tests ... Except testGetFullLinksInDescription() in DescriptionTestUnboxing, since that requires other changes first.	2020-02-26 17:31:01 +01:00
wb9688	88282761f4	Fix YoutubeStreamExtractor not being able to getAgeLimit() in onFetchPage()	2020-02-26 16:19:42 +01:00
wb9688	bb3a3d70bf	Use pbj in YoutubeTrendingExtractor	2020-02-26 16:10:24 +01:00
wb9688	1c3a4e710b	Check if there are no related streams in getAgeLimit()	2020-02-26 16:03:06 +01:00
wb9688	985c3ec877	Improve getClientVersion()	2020-02-26 15:27:50 +01:00
wb9688	15afbea3e1	Use pbj in YoutubePlaylistExtractor	2020-02-26 15:27:50 +01:00
wb9688	89a41a7f69	Use pbj in YoutubeChannelExtractor	2020-02-26 15:27:50 +01:00
wb9688	0973263aab	Use pbj in YoutubeSearchExtractor	2020-02-26 15:27:46 +01:00
TobiGr	fc465c8bf7	Fix getUploadDate() for premiered videos with a given date ... Detect publishDate and uploadDate	2020-02-25 22:54:37 +01:00
TobiGr	96285e0815	Fix getUploadDate() for premiered videos with a given date	2020-02-25 21:51:30 +01:00
wb9688	5a35300a5e	Fix parsing upload date for premiered streams	2020-02-25 21:51:30 +01:00
wb9688	729fd2eaee	Fix parsing description for navigationEndpoint without urlEndpoint	2020-02-25 21:51:30 +01:00
wb9688	26fb44595f	Fix parsing description	2020-02-25 21:51:30 +01:00
TobiGr	0fff03038e	Get description with correct links	2020-02-25 21:51:30 +01:00