kavin/NewPipeExtractor
1
0
Fork 0
mirror of https://github.com/TeamNewPipe/NewPipeExtractor.git synced 2024-12-14 14:20:33 +05:30
Code Issues Projects Releases Wiki Activity
930 Commits 10 Branches 75 Tags 199 MiB
33b1121fc7
Commit Graph

609 Commits

Author SHA1 Message Date
Stypox
ae47c9587c
[SoundCloud] Optimize imports in edited files 2020-03-17 20:32:04 +01:00
Stypox
4389fd3b7b
[SoundCloud] Migrate StreamExtractor to api-v2 2020-03-17 18:06:13 +01:00
Stypox
5e4ddb368f
[SoundCloud] Fix extractors built from next playlist pages 
They didn't have the information to calculate another next page url. So now `nextPageUrl` contains a full link with all video ids, and `getPage` takes the first part of the url (containing 15 streams) and produces another `nextPageUrl` with the remaining streams.
Also add a test for this.
 2020-03-17 18:04:40 +01:00
Stypox
0e1b4bbf17
[SoundCloud] Test playlists banner: it should not exist 2020-03-17 15:53:25 +01:00
Stypox
65bdb3bc9d
[SoundCloud] Unignore ignored playlist tests 2020-03-17 15:49:58 +01:00
Stypox
1558da6f6b
[SoundCloud] Fix playlist next page generation 2020-03-17 15:46:35 +01:00
Stypox
ca8bf53b61
[SoundCloud] Fix playlist test: number of streams changed 2020-03-17 15:35:33 +01:00
Stypox
d4aa4a0763
[SoundCloud] Fix typo in ChartsExtractor 2020-03-17 15:25:11 +01:00
Stypox
4b1121aac7
[SoundCloud] Add tests for api-v2 channel urls 2020-03-17 15:23:13 +01:00
Stypox
c3d811fde5
[SoundCloud] Use api-v2 in SubscriptionExtractor 
Also added
 2020-03-17 15:22:25 +01:00
Stypox
d0e66cc600
[SoundCloud] Improve thumbnail url extraction in playlists 
Prevent NullPointerExceptions and remove duplicate code
 2020-03-17 15:13:28 +01:00
Stypox
f3095713f9
[SoundCloud] Use api-v2 in PlaylistExtractor 
Rewrote methods to calculate next page url and to get items from it. `api-v2` is different from `api` since the initial playlist page contains (usually) the full info of the first 3 streams and only the id of the other. Then the single tracks can be requested in batch using `/tracks?ids=id1,id2,...`.
 2020-03-17 15:12:13 +01:00
TobiGr
9eca7df947 Forget to fetch the page 2020-03-16 20:50:08 +01:00
TobiGr
edb57840d7 [SoundCloud] Update hard-coded client_id 2020-03-16 19:43:36 +01:00
TobiGr
efad0be99f Test if the extractor can get AudiStreams with the hard-coded cliend_id 
SoundCloud client_id are changed on a regular basis now. We might want to remove the hardcoded id completely on a later point of time when client_id is changed to often
 2020-03-16 19:42:35 +01:00
TobiGr
5f39bc4e99 [SoundCloud] Update hardcoded client id 2020-03-16 17:26:30 +01:00
Tobias Groza
65a7eda446
Merge pull request #256 from B0pol/mediaccc 
Support for non-api mediaccc links, invidious shortened links, refactor mediaccc tests
 2020-03-15 17:04:09 +01:00
TobiGr
40b3bc138f Remove hack to fix YoutubeStreamLinkHandlerFactoryTest of vnd.youtube videos 2020-03-14 21:54:54 +01:00
Tobias Groza
1c954fbded
Merge pull request #277 from XiangRongLin/vnd 
Handle youtube stream urls in "vnd.youtube:videoId" format.
 2020-03-14 21:52:49 +01:00
Tobias Groza
9edd0f2675
Merge pull request #288 from Stypox/fix-unlisted-video 
[YouTube] Fix videos without next stream throwing error and add test for unlisted videos
 2020-03-14 21:36:13 +01:00
Stypox
957b78ed9b
[YouTube] Add test for an unlisted video, which has no next stream 2020-03-14 14:04:01 +01:00
Stypox
325a51f73a
[YouTube] Do not crash if there is no next stream, but return null 
Like every other extractor does if there is no next stream
 2020-03-14 14:02:48 +01:00
TobiGr
aea0962cfe [SoundCloud] Update hardcoded client id 2020-03-14 12:41:44 +01:00
Mauricio Colli
b086e9db3f
[YouTube] Fix id extraction for some channels 
Some channels had no reliable way to get the redirected id in the response,
so saving it for later was a valid alternative.
 2020-03-14 02:39:49 -03:00
Mauricio Colli
00d1ed439b
[YouTube] Fix channel extraction when redirects are in the response 
Some redirects were embed directly into the response as instructions
for the page, instead of the usual http redirects.
 2020-03-14 02:39:48 -03:00
bopol
66518ec444 check wether mediaccc streams url are secure 2020-03-13 20:07:36 +01:00
bopol
0cd5e05b7b MediaCCCLH: use substring instead of replace; improve a bit tests, return Collections.emptyList(); instead of null where it's annotated @NonNull 2020-03-13 20:00:49 +01:00
bopol
f742a6bd3e code optimization 2020-03-13 20:00:49 +01:00
bopol
9701c7d800 invidious shortened links 2020-03-13 20:00:49 +01:00
bopol
e8e535b815 mediaccc: update linkhandlers & refactor tests 2020-03-13 20:00:49 +01:00
Mauricio Colli
70abd57852
[YouTube] Cache commonly used stream type result 2020-03-07 16:48:32 -03:00
Mauricio Colli
1ef706f567
[YouTube] Detect LIVE videos in the trending page 2020-03-07 16:48:31 -03:00
Xiang Rong Lin
cc8fb486ee Handle youtube stream urls in "vnd.youtube:videoId" format. 
The official YouTube app accept intents in this format, which causes other developers to use it.
 2020-03-02 20:40:33 +01:00
Mauricio Colli
6aa4d59b91
[PeerTube] Update video used in age limit test (the previous was 404) 2020-03-01 10:29:30 -03:00
Mauricio Colli
cfc278317d
[YouTube] Fail-fast if status exist and is anything other than "OK" 2020-03-01 10:29:29 -03:00
Mauricio Colli
90ae5fbea2 Improve kiosk tests across services 2020-03-01 12:57:18 +01:00
Mauricio Colli
e590417cc4 Test if services recognizes their own items urls 2020-03-01 12:57:18 +01:00
TobiGr
35252235b0 Go through all badges when checking if a StreamInfoItem is a live stream 2020-03-01 12:52:37 +01:00
Mauricio Colli
5686a6f562 [YouTube] Detect when a stream is deleted or doesn't exist 
Added a test case as well.
 2020-03-01 12:49:13 +01:00
Mauricio Colli
e65333c3ce [YouTube] Detect deleted/nonexistent/invalid channels and playlists 
- Added tests for these cases.
 2020-03-01 12:49:13 +01:00
Mauricio Colli
98e359438a [YouTube] Detect simple 404s in the standard fetch method 2020-03-01 12:49:13 +01:00
Mauricio Colli
408f042127 [YouTube] Fix bug when url isn't present in the browseEndpoint object 2020-03-01 12:49:13 +01:00
Mauricio Colli
342bdbb852 [YouTube] Avoid crashing by letting exceptions bubble up 2020-03-01 12:49:13 +01:00
Mauricio Colli
e9644e6216 [YouTube] Handle video premiere's date and duration 2020-03-01 12:49:13 +01:00
Mauricio Colli
2a470ac4f7 [YouTube] Handle videos with no views or with "Recommended to you" text 2020-03-01 12:49:13 +01:00
Mauricio Colli
da2c562eb0 [YouTube] Take into account videos that have their views hidden 
Also remove catch because it may hide bugs when the page changes.
 2020-03-01 12:49:13 +01:00
Mauricio Colli
f66c20de54 Ignore null-keyed entries when iterating through the response headers 2020-03-01 12:49:13 +01:00
Mauricio Colli
5edd774fc4 Add latest url to the response to make detection of a redirect possible 
Will be latest one in this commit because there's need to check the
history of redirects as of now.
 2020-03-01 12:49:13 +01:00
Mauricio Colli
3441946bea Make test downloader return a response instead of throwing an exception 
The test implementation was throwing an exception instead of just
returning the response and letting the caller handle it.
 2020-03-01 12:49:13 +01:00
wb9688
df219a9453 Don't use StandardCharsets class 2020-03-01 09:41:06 +01:00
wb9688
2807079531 Catch NumberFormatException instead of checking whether viewCount equals "Recommended for you" 2020-02-29 21:28:38 +01:00
wb9688
3e4d2fd64e Improve testChannelUrl() 2020-02-29 18:20:35 +01:00
wb9688
bfe3eb1409 Use suggested try-if code style 2020-02-29 17:18:50 +01:00
wb9688
a6a63e9570 Let isHardcodedClientVersionValid() throw IOException 2020-02-29 17:01:36 +01:00
wb9688
45df8248bf Use Localization in getJsonResponse() 2020-02-29 16:55:07 +01:00
wb9688
157055fbfd Create getJsonResponse() function 2020-02-29 16:42:04 +01:00
wb9688
4f406512c2 Use YoutubeChannelLinkHandlerFactory 2020-02-29 16:26:28 +01:00
wb9688
4ff655352d Reimplement getAgeLimit() 2020-02-28 17:14:26 +01:00
wb9688
6ba03e542b Cache getVideoTab() 2020-02-28 17:05:31 +01:00
wb9688
8bfbae904d Cache getVideo(Primary|Secondary)InfoRenderer() 2020-02-28 17:03:21 +01:00
wb9688
0b940f5cf4 Use "getAgeLimit() != NO_AGE_LIMIT" instead of "getAgeLimit() == 18" 2020-02-28 16:40:50 +01:00
wb9688
beb07a4852 Add test to check whether hardcoded client version is still valid 2020-02-28 16:36:14 +01:00
wb9688
b89b6b1db2 Check if hard-coded client version is valid in separate function 2020-02-28 16:35:24 +01:00
wb9688
7898ba230d Use pbj in YoutubeStreamExtractor 2020-02-28 15:17:47 +01:00
wb9688
612ec06839 Actually fix thumbnail URLs starting with // 2020-02-28 09:36:33 +01:00
wb9688
f946310af5 Fix search queries with a promotedSparklesTextSearchRenderer returning 0 results 2020-02-28 08:57:44 +01:00
wb9688
94a2d5be46 Fix thumbnail URLs starting with // 2020-02-27 19:08:46 +01:00
wb9688
838d834ce2 Get Uploads view of Videos tab 2020-02-27 18:54:27 +01:00
wb9688
365b0329f3 Implement getTextFromObject() function 2020-02-27 17:39:23 +01:00
wb9688
0798bdd5cd Fix getSubscriberCount() for 0 subscribers 2020-02-27 16:26:22 +01:00
wb9688
8ebd971648 Fix channel descriptions consisting of multiple parts 2020-02-27 10:06:35 +01:00
wb9688
880b951088 Fix getUploaderAvatarUrl() in YoutubeStreamExtractor 2020-02-27 09:53:59 +01:00
wb9688
8aa560bef4 Fix video titles consisting of multiple parts due to @ 2020-02-27 09:42:22 +01:00
wb9688
951ed9f0d8 Don't call getUploaderInfo() in onFetchPage() for playlists 
This way YouTube Music playlists will keep working in NewPipe
 2020-02-27 09:27:56 +01:00
wb9688
2399d46ab0 Fix YoutubeChannelExtractor for channels without videos 2020-02-27 09:21:25 +01:00
wb9688
cdbc751b87 Fix getUploaderUrl() in YoutubeStreamInfoItemExtractor 2020-02-27 09:05:41 +01:00
wb9688
db305408e9 Get all expandedShelfContentsRenderers in YoutubeTrendingExtractor 2020-02-26 20:37:20 +01:00
wb9688
a2d8489e16 Parse timestamps in description 2020-02-26 20:27:30 +01:00
wb9688
8d503407ef Parse browseEndpoint and watchEndpoint in descriptions 2020-02-26 19:37:58 +01:00
wb9688
8dc39d517a Fix tests 
Except testGetFullLinksInDescription() in DescriptionTestUnboxing, since that requires other changes first.
 2020-02-26 17:31:01 +01:00
wb9688
88282761f4 Fix YoutubeStreamExtractor not being able to getAgeLimit() in onFetchPage() 2020-02-26 16:19:42 +01:00
wb9688
bb3a3d70bf Use pbj in YoutubeTrendingExtractor 2020-02-26 16:10:24 +01:00
wb9688
1c3a4e710b Check if there are no related streams in getAgeLimit() 2020-02-26 16:03:06 +01:00
wb9688
985c3ec877 Improve getClientVersion() 2020-02-26 15:27:50 +01:00
wb9688
15afbea3e1 Use pbj in YoutubePlaylistExtractor 2020-02-26 15:27:50 +01:00
wb9688
89a41a7f69 Use pbj in YoutubeChannelExtractor 2020-02-26 15:27:50 +01:00
wb9688
0973263aab Use pbj in YoutubeSearchExtractor 2020-02-26 15:27:46 +01:00
TobiGr
fc465c8bf7 Fix getUploadDate() for premiered videos with a given date 
Detect publishDate and uploadDate
 2020-02-25 22:54:37 +01:00
TobiGr
96285e0815 Fix getUploadDate() for premiered videos with a given date 2020-02-25 21:51:30 +01:00
wb9688
5a35300a5e Fix parsing upload date for premiered streams 2020-02-25 21:51:30 +01:00
wb9688
729fd2eaee Fix parsing description for navigationEndpoint without urlEndpoint 2020-02-25 21:51:30 +01:00
wb9688
26fb44595f Fix parsing description 2020-02-25 21:51:30 +01:00
TobiGr
0fff03038e Get description with correct links 2020-02-25 21:51:30 +01:00
wb9688
5d08c34322 Fix parsing upload date in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688
bbe1a3cd62 Use getVideo(Primary|Secondary)InfoRenderer() 2020-02-25 21:51:30 +01:00
wb9688
8e6be880d7 Simplify getId() in YoutubeChannelExtractor 2020-02-25 21:51:30 +01:00
TobiGr
ecfc27a92c Implement fallback methods for getDescription() and getTextualUploadDate() in YouTubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688
eed29ea734 Return null for related streams at age restricted videos 2020-02-25 21:51:30 +01:00
TobiGr
9efcc61ca6 Adapt some tests to match the new URLs 2020-02-25 21:51:30 +01:00
TobiGr
f39603f6ef Implement getUploadDate() in YouTubeStreamInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688
26ea3dceb6 Fix getViewCount() in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688
76d54abdbf Reimplement more methods in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688
f13c0288cc Reimplement some methods in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688
02b59903fa Remove useless code 2020-02-25 21:51:30 +01:00
TobiGr
dd4dd849dc Get client version dynamically in YouTubeSearchExtractor 2020-02-25 21:51:30 +01:00
TobiGr
583e9c1571 Fix getPage() 
Someone should check if there are unstaged changes before committing...
 2020-02-25 21:51:30 +01:00
TobiGr
5842b9ad37 Add getClientVersion() and HARDCODED_CLIENT_VERSION to YouTubeParsingHelper 
Prefer hardcoded client version above the current one when making requests to retrieve the same JSON structure for each request.
 2020-02-25 21:51:30 +01:00
wb9688
5d883d100c Implement pagination in YoutubeSearchExtractor 2020-02-25 21:51:30 +01:00
wb9688
c0a8e01889 Implement pagination in YoutubePlaylistExtractor 2020-02-25 21:51:30 +01:00
wb9688
4039409820 Fix getLikeCount() for certain streams 2020-02-25 21:51:30 +01:00
wb9688
627ab2459f Uncomment fetchPage() 2020-02-25 21:51:30 +01:00
wb9688
2dfa2187ff Implement pagination in YoutubeChannelExtractor 2020-02-25 21:51:30 +01:00
wb9688
a38ab9b791 Return null if channel has no banner 2020-02-25 21:51:30 +01:00
wb9688
1eca63089b Count private and deleted videos as ad 2020-02-25 21:51:30 +01:00
wb9688
c7360e4a46 Fix getThumbnailUrl() in YoutubeChannelInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688
51fb26625a Implement getSubscriberCount() in YoutubeChannelInfoItemExtractor 2020-02-25 21:51:30 +01:00
TobiGr
6d20b2b1d0 Fix NPE on collecting page items 2020-02-25 21:51:30 +01:00
wb9688
4462cbe3f1 Reimplement YoutubePlaylistInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688
8aea4d445b Reimplement YoutubeChannelInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688
3187116a63 Handle premium videos 2020-02-25 21:51:30 +01:00
wb9688
7442f91aa6 Don't assume which channel tab is videos 2020-02-25 21:51:30 +01:00
wb9688
316fe0109d Fix first playlist page 2020-02-25 21:51:30 +01:00
wb9688
4e57e589ce Fix first channel page 2020-02-25 21:51:30 +01:00
TobiGr
5816202cc7 Move getInitialData() method to YouTubeParsingHelper 
Rename ytInitialData to initialData
 2020-02-25 21:51:30 +01:00
wb9688
38aabc6aca Use plain YoutubeStreamInfoItemExtractor in YoutubeTrendingExtractor 2020-02-25 21:51:30 +01:00
wb9688
957db062ef Fix getting uploader for videoRenderer 2020-02-25 21:51:30 +01:00
wb9688
b88188d419 Move stuff from extractVideoPreviewInfo() into YoutubeStreamInfoItemExtractor and partially fix search 2020-02-25 21:51:30 +01:00
wb9688
af49b3c487 Fix live streams in extractVideoPreviewInfo() 2020-02-25 21:51:30 +01:00
TobiGr
f33f9466ce Fix testDescription() in ChannelExtractor 2020-02-25 21:51:30 +01:00
TobiGr
10e9c16d8c Fix LiveStream view count 2020-02-25 21:51:30 +01:00
wb9688
127c4d5893 Fix getNextStream() in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688
7bfc0e62c6 Fix getRelatedStreams() in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
TobiGr
12166ca60c [StreamExtractor] Add fallback method for uploader avatar url 2020-02-25 21:51:30 +01:00
wb9688
3ec422a542 Fix like/dislike parsing 2020-02-25 21:51:30 +01:00
TobiGr
a34e5c18e1 Trending StreamInfoItemCollecetor 2020-02-25 21:51:30 +01:00
TobiGr
b705515da0 Adapt most playlist fields 2020-02-25 21:51:30 +01:00
TobiGr
21253abd9b Use Firefox ESR User-Agent: 68 2020-02-25 21:51:30 +01:00
TobiGr
655c999795 Detect disabled subriber count correctly 
Fix parsing of kiosk name
 2020-02-25 21:51:30 +01:00
TobiGr
7dcc9f159b Stream: fix uploader avatar url 2020-02-25 21:51:30 +01:00
TobiGr
be3e20e263 Adapt most channel changes 2020-02-25 21:51:30 +01:00
TobiGr
39bf1ff1eb Add ytInitialData regex 2020-02-25 21:51:30 +01:00
Tobias Groza
a129c65c63
more improvements of comments 
Co-Authored-By: Stypox <stypox@pm.me>
 2020-02-16 23:59:31 +01:00
TobiGr
0a4115286b Apply one suggestion 2020-02-15 13:36:54 +01:00
Tobias Groza
6822574588 Fix typos 
Co-Authored-By: B0pol <bopol@e.email>
 2020-02-15 13:36:54 +01:00
TobiGr
030465b5d4 Improve code formatting and optimise imports 2020-02-15 13:36:54 +01:00
TobiGr
9112a106ea [SoundCloud] Update client_id 2020-02-08 23:09:03 +01:00
TobiGr
008b6f505c [PeerTube] Fix subscriber count in ChannelExtractorTest 2020-02-08 23:08:37 +01:00
TobiGr
0e33249c85 Fix SoundCloud description test 2020-02-08 22:20:20 +01:00
bopol
11bcc78d9c Description implements Serializable. fix NotSerializableException 2020-02-07 14:10:54 +01:00
bopol
0f8a7f9df2 fix testGetUploadDate for PeerTubeStreamExtractor 2020-02-07 13:33:50 +01:00