kavin/NewPipeExtractor
1
0
Fork 0
mirror of https://github.com/TeamNewPipe/NewPipeExtractor.git synced 2024-12-15 14:50:33 +05:30
Code Issues Projects Releases Wiki Activity
1,444 Commits 10 Branches 75 Tags 199 MiB
a7b15b51e6
Commit Graph

1035 Commits

Author SHA1 Message Date
Stypox
1d7a86e664
[Test] Add base classes for stream extractor tests 
Refactor all stream extractor tests to use new base class.

Remove check if upload date is in the past: this does not have to hold true: youtube premieres turn up in search results even though they are in the future
 2020-10-24 18:22:44 +02:00
Bartosz Rumiński
29695aed0a Small field refactor 2020-10-23 16:42:30 +02:00
Bartosz Rumiński
0e67d820bc Use static regex pattern for excluded path segments 2020-10-22 20:13:29 +02:00
Bartosz Rumiński
d3f80d1538 Exlude links which are not channels 2020-10-20 20:06:06 +02:00
Stypox
6887d59570
[YouTube] Handle urls for Shorts 2020-10-18 12:03:01 +02:00
Scratch
e945f711c1
Fix SoundCloud test artist account name (#416) 2020-10-16 23:42:46 +02:00
wb9688
be9a6f931c Fix parsing new ytInitialData 2020-10-16 20:27:40 +02:00
Tobias Groza
1a9bc8caf1
Merge pull request #414 from Scrxtchy/soundcloud-slash 
Remove tailing slash from SoundCloud URLs
 2020-10-16 09:48:44 +02:00
Scratch
6a70cb9d50 Remove tailing slash from SoundCloud URLs 
Fixes #412
 2020-10-16 17:17:40 +11:00
Tobias Groza
9a785f9a7e
Merge pull request #413 from Scrxtchy/soundcloud-test-update 
Fix SoundCloud test artist account name
 2020-10-16 08:05:44 +02:00
Scratch
538f5d3973 Fix SoundCloud test artist account name 2020-10-16 13:45:05 +11:00
Stypox
19e862657a
[YouTube] Fix some decryption exceptions by retrying 2020-10-15 20:16:44 +02:00
Bartosz Rumiński
e3f996e014 Exlude links which are not channels 2020-10-12 20:59:56 +02:00
Bartosz Rumiński
5ab1b053d2
Update youtube channel link handler tests 
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
 2020-10-12 20:11:28 +02:00
Bartosz Rumiński
7abb4b3713
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeChannelLinkHandlerFactory.java 
Fix typos

Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
 2020-10-12 19:57:45 +02:00
Bartosz Rumiński
9d63211a66
Fix typos 
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
 2020-10-12 19:56:53 +02:00
Bartosz Rumiński
8c38a5509e Prevent attribution_link urls to be accepted by channel links handler 2020-10-09 19:07:38 +02:00
Bartosz Rumiński
4e04991762 Support short custom youtube channel urls 2020-10-09 17:06:59 +02:00
Fynn Godau
81b5e7cf3d Fix extractor 2020-10-08 17:56:03 +02:00
wb9688
ebbfe7f6d4 Skip YouTube's OTF streams 2020-09-29 10:48:02 +02:00
Tobias Groza
6633f26ec5
Merge pull request #386 from mhmdanas/remove-subtitles-stream-get-url 
Remove SubtitlesStream#getURL()
 2020-08-26 21:32:36 +02:00
wb9688
7657c2ed1a Use initSafeStandardsObjects() 2020-08-15 17:08:07 +02:00
mhmdanas
1a63dcb355 Remove overriden field url 2020-08-12 22:37:36 +03:00
mhmdanas
f0f1c009b2 Remove SubtitlesStream#getURL() 2020-08-10 19:28:56 +03:00
Fynn Godau
cfe88a74c1 Throw ContentNotSupportedException when opening radio uploader channel 2020-08-09 10:07:49 +02:00
TobiGr
0fb73301e3 [YouTube] Fix crash on empty comment 
Closes #380
 2020-08-05 18:25:40 +02:00
Fynn Godau
932d094d6a Upgrade featured playlist urls to https 2020-08-02 16:55:18 +02:00
Fynn Godau
c12ef3a02d Merge TNP/dev into fynngodau/dev 2020-08-02 16:43:04 +02:00
TobiGr
5ac80624a4 Fix detection of YouTube's decryption function 
Closes TeamNewPipe/NewPipe#3951
 2020-07-28 01:01:10 +02:00
Tobias Groza
32dff1541f
Merge pull request #370 from wb9688/fix-yt-age-restricted 
Fix age restricted YouTube videos
 2020-07-26 14:17:19 +02:00
TobiGr
ec778200d9 Add comments to the code 2020-07-26 14:15:13 +02:00
wb9688
667dce034c Only use fallback methods for clientVersion when clientVersion hasn't been successfully extracted yet 2020-07-26 13:14:25 +02:00
wb9688
f345f667e2 Extract YouTube's key 2020-07-26 12:01:59 +02:00
wb9688
4c987a5302 Support YouTube's new continuations for search 2020-07-26 10:01:03 +02:00
wb9688
81459e289f Fix age restricted YouTube videos 2020-07-18 19:52:54 +02:00
wb9688
fc3a63fec5 Update Javadoc for getRelatedStreams() 2020-07-15 18:48:37 +02:00
wb9688
ff560e907a Remove getNextStream() from StreamExtractor 2020-07-15 18:48:37 +02:00
Tobias Groza
a70cb0283f
Merge pull request #314 from wb9688/remove-getnextpageurl 
Next page stuff
 2020-07-07 20:49:38 +02:00
wb9688
0a5a905bc7 Add final at more places 2020-07-07 20:45:47 +02:00
wb9688
17ba8a57fa Clean up the code 2020-07-06 20:19:31 +02:00
wb9688
9b6fe1dea6 Throw IllegalArgumentException when Page is invalid 2020-07-06 20:19:31 +02:00
wb9688
4cc312086a Introduce Page class 2020-07-06 20:19:31 +02:00
wb9688
e3bfdba135 Remove getNextPageUrl() function from ListExtractor 2020-07-06 20:11:40 +02:00
wb9688
156a26b64b Support start= timestamps for YouTube 2020-07-06 14:08:40 +02:00
bopol
261471e7f4 remove invidiou.sh from tests 
dead instance
 2020-07-02 21:33:45 +02:00
bopol
eb3901acee update invidious instances 2020-07-02 21:31:05 +02:00
Tobias Groza
c0ceb5cb27
Merge pull request #340 from wb9688/peertube-deleted-comments 
Handle isDeleted for PeerTube comments
 2020-06-28 10:21:37 +02:00
Tobias Groza
339f1d9c43
Merge pull request #349 from wb9688/okhttp 
Use OkHttp for tests like in NewPipe
 2020-06-27 20:08:02 +02:00
Tobias Groza
970bc7f69d
Merge pull request #359 from mauriciocolli/fix-client-id-bug 
[SoundCloud] Fix concurrency issue when getting the client id
 2020-06-27 17:36:24 +02:00
Robin
576754982e
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java 
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
 2020-06-27 11:35:01 +02:00
Mauricio Colli
7505e828cc
[SoundCloud] Update client id 2020-06-24 01:50:09 -03:00
Mauricio Colli
d2f1c0f40d
[SoundCloud] Fix concurrency issue when getting the client id 2020-06-24 01:50:08 -03:00
Robin
b63ae93495
Added a comment 
and fixed a "typo" in the method parseDurationString replacing Long with Integer.

These are not the droids you are looking for wb9688 😁
 2020-06-15 11:27:44 +02:00
Tobias Groza
92f6754f0f
Merge pull request #338 from wb9688/parse-duration-string-test 
Add tests for parseDurationString()
 2020-06-15 11:01:21 +02:00
Stypox
5808439c3f
[PeerTube] Change age restricted video in tests 
The old one wasn't available anymore
 2020-06-14 19:57:39 +02:00
wb9688
3239aa84f2 Handle isDeleted for PeerTube comments 2020-06-14 19:27:20 +02:00
wb9688
a65e46e4b1 Use OkHttp for tests like in NewPipe 2020-06-11 15:06:08 +02:00
Fynn Godau
96de834b67 Bandcamp: parse date from radio info item 2020-06-04 19:36:58 +02:00
Fynn Godau
39b55b5b42 Bandcamp radio stream extractor: test uploader url 2020-06-04 19:32:19 +02:00
Fynn Godau
9a555d97e5 Remove useless bandcamp tests 2020-06-04 19:31:01 +02:00
Fynn Godau
e13f341a45 Bandcamp stream extractor: test service id 2020-06-04 19:29:37 +02:00
Fynn Godau
4dd9540782 Bandcamp license switch order 2020-06-04 19:27:39 +02:00
Fynn Godau
52103ac61f Remove "useless" comments 2020-06-04 19:27:13 +02:00
Fynn Godau
8fa81537c0 Better bandcamp stream extractor description test 2020-06-04 19:17:51 +02:00
Fynn Godau
0950a95577 Fix for testing wrong object for null or empty 2020-06-04 19:16:08 +02:00
Fynn Godau
e6ecd91929 Bandcamp: Test full date 2020-06-04 19:12:10 +02:00
Fynn Godau
27e7aad159 Bandcamp: Improve code style 2020-06-04 19:04:25 +02:00
fynngodau
ea49202f64
Don't display internal license id 
Co-authored-by: bopol <bopol@e.email>
 2020-06-04 18:26:58 +02:00
bopol
063f4be766 For #342: support PeerTube embeds 2020-06-04 14:26:42 +02:00
Fynn Godau
34b6928124 Don't print stacktrace before throwing new exception 2020-06-04 14:25:38 +02:00
Fynn Godau
1be20ceeec Bandcamp: parse date 2020-06-04 14:09:21 +02:00
Fynn Godau
6822fe3dc8 Bandcamp: Get Stream upload date 2020-06-04 13:03:16 +02:00
Fynn Godau
e98bff8ae9 Improved Bandcamp radio stream extractor tests 2020-06-04 12:30:53 +02:00
Fynn Godau
8f6c00f8d8 Use default test in BandcampSearchExtractorTest 2020-06-04 12:12:20 +02:00
Fynn Godau
de776561b0 Bandcamp: fix loading uploader from streams in search 2020-06-04 12:02:18 +02:00
Fynn Godau
3940138fc5 Fix Bandcamp capitalization in one more spot 2020-06-03 22:41:27 +02:00
Fynn Godau
9fa9d920a9 Bandcamp: Implement new methods required due to interface changes 2020-06-03 21:35:11 +02:00
Fynn Godau
dd955c7f0c Merge branch 'origin/dev' into dev 2020-06-03 21:13:40 +02:00
wb9688
33b1121fc7 Add tests for parseDurationString() 2020-05-30 17:20:54 +02:00
wb9688
e39147202c Fix YouTube video duration parsing 2020-05-30 09:31:08 +02:00
wb9688
98055a3c3c Revert "Add more decryption function regexes" 
This reverts commit 384d6acd5d.
 2020-05-29 18:41:40 +02:00
Tobias Groza
bdb0f2ae6b
Merge pull request #332 from wb9688/learning-playlist 
Support YouTube's learning playlists
 2020-05-29 10:11:08 +02:00
Tobias Groza
cf18cdb2f5
Merge pull request #329 from wb9688/decryption-regexes 
Add more decryption function regexes
 2020-05-29 10:10:50 +02:00
wb9688
ab77961b0a Extract trailer thumbnail in YT learning playlists 2020-05-29 09:55:24 +02:00
wb9688
4d683e7655 Support YouTube's learning playlists 2020-05-29 09:17:03 +02:00
Tobias Groza
0b4977bb0c
Merge pull request #333 from wb9688/playerresponse-dashmanifesturl 
Extract dashManifestUrl in playerResponse
 2020-05-28 21:17:07 +02:00
wb9688
384d6acd5d Add more decryption function regexes 2020-05-28 11:40:38 +02:00
wb9688
8f71bde776 Fix bug for YT videos longer than 1000h 2020-05-28 11:34:42 +02:00
wb9688
aa277e1b6a Upgrade other dependencies 2020-05-28 11:34:42 +02:00
wb9688
036cc7e26e Upgrade jsoup 2020-05-28 11:34:41 +02:00
Fynn Godau
89b0639faa New Bandcamp tests for channel extractor 2020-05-25 20:22:23 +02:00
Fynn Godau
e08256ebc6 Remove "url cleanup" in Bandcamp link handlers 2020-05-25 19:17:27 +02:00
Fynn Godau
692b2d06f4 Bandcamp: read info item data in place and not in advance 2020-05-25 18:53:15 +02:00
Fynn Godau
fff21caa87 Fix Bandcamp no avatar test 
I own the bandcamp artist account called "NewPipeExtractorTest", so it
shouldn't break again.
 2020-05-25 18:53:10 +02:00
wb9688
7ef3e80493 Extract dashManifestUrl in playerResponse 2020-05-15 12:07:26 +02:00
wb9688
efafbbb88a
Merge pull request #331 from wb9688/playlist-subchannel 
Add sub-channel stuff to PlaylistInfo
 2020-05-12 10:49:13 +02:00
wb9688
baf5dd3e8f
Merge pull request #311 from B0pol/fixSearchSuggestion 
Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSea…
 2020-05-12 09:29:15 +02:00
wb9688
102f04e297 Add sub-channel stuff to PlaylistInfo 2020-05-12 09:14:29 +02:00
Tobias Groza
b40ccb5075
Merge pull request #307 from B0pol/playlists_peertube 
[PeerTube] playlist support
 2020-05-11 20:53:03 +02:00
bopol
adaf196c99 remove unused import 2020-05-11 12:54:33 +02:00
bopol
de26e00079 changed all == null || isEmpty() to isNullOrEmpty() 2020-05-11 11:40:24 +02:00
bopol
202a73516c refactor: add Utils.isNullOrEmpty() 2020-05-11 11:26:18 +02:00
wb9688
1de0876fb4 Fix #327 2020-05-09 20:06:50 +02:00
bopol
7cae95bf09 add tests for subchannel stuff in playlists 2020-05-09 10:15:35 +02:00
bopol
1a6e92ebf6 add getSubChannel Name, Url and AvatarUrl for playlists 2020-05-09 09:52:24 +02:00
bopol
cf99107745 address yausername change 2020-05-09 00:52:04 +02:00
bopol
5bab9d9fc0 [PeerTube] playlist support & refactoring 2020-05-09 00:52:04 +02:00
Tobias Groza
f3913e241e
Merge pull request #324 from Stypox/music-detector-url 
[YouTube] Support stream urls in `vnd.youtube://ID` form
 2020-05-08 17:43:04 +02:00
wb9688
7320108c66 Rename SubChannel in channels back to ParentChannel 2020-05-08 16:02:59 +02:00
Roy Yosef
a408661771 Fix PR review 
Rename "ParentChannel" to "SubChannel"
Add Tests for sub-channel info fetching
 2020-05-07 20:20:12 +02:00
Roy Yosef
1de1f97cf1 Add parent channel info to StreamExtractor: name, url & avatar url 2020-05-07 20:20:12 +02:00
Roy Yosef
4234740baa Add parent channel info to ChannelExtractor: name, url & avatar url 2020-05-07 20:20:12 +02:00
bopol
6cff5de400 renaming methods for consistency 2020-05-07 15:40:41 +02:00
bopol
5760366d70 use JsonObject#isEmpty instead of JsonObject#equals(JsonUtils.EMPTY_OBJECT) 2020-05-07 15:40:41 +02:00
bopol
93e868634e clarified javadoc for isCorrectedSearch and getSearchSuggestions in SearchExtractor 2020-05-07 15:40:41 +02:00
bopol
3757541a9c adapt new nanojson version 2020-05-07 15:40:41 +02:00
bopol
79483a6dc0 «Use this in SearchInfo, for consistency's sake» @Stypox order xD 2020-05-07 15:40:41 +02:00
bopol
e8d58e09c9 getSearchSuggestion now returns the corrected query if isCorrectedSearch() is true 2020-05-07 15:40:41 +02:00
bopol
5d1c3b3fa2 Use setters in SearchInfo 2020-05-07 15:39:57 +02:00
bopol
29b639b454 Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSearchSuggestion() 
isCorrectedSearch: see the javadoc
getSearchSuggestion: in YoutubeSearchExtractor, it was giving the corrected search query. It now gives the suggested query, as it should
 2020-05-07 15:39:57 +02:00
Stypox
a02bfbeae5
[YouTube] Support stream urls in vnd.youtube://ID form 2020-05-07 15:31:31 +02:00
bopol
bc13e0c616 Merge remote-tracking branch 'upstream/dev' into soundcloudComments 2020-05-05 14:49:21 +02:00
wb9688
6aae99f832 Fix NPE in defaultAlertsCheck() 2020-05-03 10:28:45 +02:00
wb9688
a1eabc7589 Return null instead of "" in getTextFromObject() 2020-05-02 08:02:25 +02:00
Fynn Godau
e7b046db9d Merge remote-tracking branch 'origin/dev' into dev 2020-04-26 23:12:43 +02:00
Fynn Godau
a3e8e1c70e Bandcamp: move own methods to helper class 2020-04-26 23:05:56 +02:00
Fynn Godau
9bc7a470ab Replace fori with foreach loop 2020-04-26 23:02:14 +02:00
Tobias Groza
49157fcf0d
Merge pull request #317 from wb9688/tnp-nanojson 
Use our fork of nanojson
 2020-04-24 21:54:35 +02:00
bopol
2564bcf399 refactor comments 
rename methods in CommentsInfoItemExtractor interface to match the other method names across NewPipeExtractor
remove getName in (Youtube|SoundCloud|Peertube)CommentsExtractor and move it up in CommentsExtractor, return "Comments" instead
 2020-04-21 20:05:54 +02:00
Fynn Godau
f71bd29d58 Bandcamp playlist extractor: add more tests 2020-04-20 23:52:12 +02:00
Fynn Godau
5009d9f53b Bandcamp tests: remove certain tests 2020-04-20 23:25:00 +02:00
Fynn Godau
b5e251c82f Bandcamp channel extractor: package-private access for getImageUrl(…) 2020-04-20 23:12:43 +02:00
Fynn Godau
8c70dab8c5 Bandcamp search query handler factory: inline variable 2020-04-20 23:11:16 +02:00
Fynn Godau
10ae3db118 Bandcamp javadoc: Replace br tags with p tags 2020-04-20 23:03:12 +02:00
Fynn Godau
9201e0d3ea Don't declare exceptions that can't actually be thrown 2020-04-20 22:46:54 +02:00
Fynn Godau
965bce00cf Bandcamp stream extractor: return NO_AGE_LIMIT 2020-04-20 22:44:45 +02:00
Fynn Godau
82099592c7 Bandcamp: Move code from SearchExtractor to InfoItemExtractors 2020-04-20 22:44:35 +02:00
Fynn Godau
00c0333059 Bandcamp: Fetch channelInfo in onFetchPage 2020-04-20 22:41:44 +02:00
Fynn Godau
67de0285e1 Bandcamp: Generate query JSON in a proper way 2020-04-20 22:03:29 +02:00
Fynn Godau
c133190c53 Bandcamp capitalization 2020-04-20 21:55:35 +02:00
Fynn Godau
b100b9873f Bandcmap: Use nanojson fork instead of org.json 2020-04-20 21:53:04 +02:00
wb9688
b51699a20e Implement @TobiGr's suggestions 2020-04-20 14:27:33 +02:00
wb9688
74f0ab1792
Merge pull request #301 from wb9688/media-ccc-non-api-urls 
Use media.ccc.de instead of api.media.ccc.de
 2020-04-19 17:06:57 +02:00
Fynn Godau
9c239371f4 Change nanojson dependency to @wb9688's fork 2020-04-19 14:29:39 +02:00
Fynn Godau
5c0a03328a Merge 'origin/dev' into dev 2020-04-19 14:26:51 +02:00
bopol
0cbbc2a1f9 remove unused import 2020-04-19 14:16:38 +02:00
wb9688
979c5a7502 Use our fork of nanojson 2020-04-17 19:36:42 +02:00
bopol
d4352f9b84 support comments for SoundCloud 2020-04-10 20:41:41 +02:00
Roy Yosef
4afe657f6f Add tests for getUrlFromId and test for video-channels in acceptUrlTest 2020-04-10 19:05:38 +03:00
Roy Yosef
2c9f1260eb Fix wb9688 review comments 
* Rename PeertubeUserExtractor to PeertubeAccountExtractor
* Add test for video-channels in PeertubeChannelLinkHandlerFactoryTest
* Compatibility support for older versions (use "accounts/" as default)
 2020-04-10 16:40:53 +03:00
bopol
636c430743 refactor: create extractors and linkHandler packages for SoundCloud, move YoutubeParsingHelper to youtube package (not linkhandler anymore) 2020-04-10 10:51:05 +02:00
Roy Yosef
b6e6f403a8 add support for PeerTube channels extraction 2020-04-09 20:37:49 +03:00
bopol
a20d53dd98 remove the mention of Opus on wb9688 advise when 
ContentNotSupportedException is thrown
 2020-04-08 15:31:11 +02:00
bopol
9a7c6b7ab0 use ContentNotSupportedException for channels without tabs 2020-04-07 14:33:43 +02:00
bopol
a22104cbda add tests for ContentNotSupportedException 2020-04-07 14:33:43 +02:00
bopol
093762e793 throw ContentNotSupportedException when content is know to be unsupported 2020-04-07 14:33:43 +02:00
Tobias Groza
094b87c537
Merge pull request #299 from B0pol/sc_countries 
add supported countries for soundcloud
 2020-04-03 19:12:33 +02:00
wb9688
bce27a0e22 Rename getValidResponseBody() to getValidJsonResponseBody() 2020-04-03 17:23:18 +02:00
bopol
1eb3deb7fd add supported countries for soundcloud 2020-04-02 11:26:19 +02:00
wb9688
8a9e137385 Extract some code to getValidResponseBody() 2020-04-01 16:01:21 +02:00
wb9688
bf24caa5f2 Use media.ccc.de instead of api.media.ccc.de 2020-04-01 13:25:10 +02:00
wb9688
2af610e3e7 Add tests for other YT Music search types 2020-04-01 10:30:51 +02:00
wb9688
c7f7bd2442 Fix error when YT Music videos has no uploader URL 2020-04-01 10:30:51 +02:00
wb9688
ac15df4548 Move YouTube Music search to its own class 2020-04-01 10:30:51 +02:00
wb9688
cf0f2aff3e Extract uploader url from certain YouTube Music videos 2020-04-01 10:30:51 +02:00
wb9688
aa8cea47f3 Refactor YouTube Music search tests 2020-04-01 10:30:51 +02:00
wb9688
dd434cca01 Fix issue when there is no didYouMeanRenderer in itemSectionRenderer 2020-04-01 10:30:51 +02:00
wb9688
5a775a4bbe Use new way of specifying stream count 2020-04-01 10:30:51 +02:00
wb9688
d58c0f230d Improve code for YouTube Music search 2020-04-01 10:30:51 +02:00
wb9688
dc29d87962 Extract YouTube search suggestions 2020-04-01 10:30:51 +02:00
wb9688
c852b13d5a Add Referer header so that it also works with HttpsUrlConnection 2020-04-01 10:30:51 +02:00
wb9688
eb48524411 Add tests for YouTube Music search 2020-04-01 10:30:51 +02:00
wb9688
2b9b2a78e8 Handle 100+ items in playlist 2020-04-01 10:30:51 +02:00
wb9688
fb9b9691b7 Improve getYoutubeMusicKeys() 2020-04-01 10:30:51 +02:00
wb9688
4ddbdf0aee Disable artist search for now 2020-04-01 10:30:51 +02:00
wb9688
1762a527c9 Add support for YouTube Music search 2020-04-01 10:30:51 +02:00
Tobias Groza
69e0624e35
Merge pull request #298 from B0pol/feed_peertube 
[peertube] add feed url in channels
 2020-03-28 12:53:47 +01:00
bopol
0ae53c1e50 [peertube] adapt test for channel feed url 2020-03-28 11:48:24 +01:00
bopol
bc43f8977d [peertube] add feed url in channels 2020-03-28 10:22:47 +01:00
TobiGr
647e7cd450 Accept YouTube Music playlists but not YouTube Mixes 2020-03-25 22:24:18 +01:00
wb9688
bcd2a1092b Don't accept YouTube Mix playlists 2020-03-25 11:07:28 +01:00
Tobias Groza
e4874d554e
Merge pull request #296 from mauriciocolli/fix-search-errors-detection 
Fix search errors detection and refactor search tests
 2020-03-24 21:53:32 +01:00
Tobias Groza
8a9ae32e6b
Merge pull request #292 from wb9688/fix-yt-no-views 
Fix YouTube videos with no views
 2020-03-24 20:27:08 +01:00
Mauricio Colli
0a20c53f1a
Use easier-to-read asserts statements in tests 2020-03-23 18:08:59 -03:00
Mauricio Colli
c921e5d6a5
Remove unnecessary collector creation function from search extractor 2020-03-23 18:08:51 -03:00
Tobias Groza
6fd9b38ad9
Merge pull request #287 from mauriciocolli/fix-channel-redirect 
[YouTube] Fix channel with redirects directly in the response
 2020-03-23 18:41:56 +01:00
Mauricio Colli
04bfa0ec6a
[PeerTube] Update video used in age limit test (was 404 not found) 2020-03-21 15:55:40 -03:00
Mauricio Colli
921bf30bb7
[YouTube] Add check for channel items with no video count in search 2020-03-21 15:55:39 -03:00
Mauricio Colli
b7f8001a49
[YouTube] Add check for channel items without description in search 2020-03-21 15:55:38 -03:00
Mauricio Colli
9b7999fe54
[YouTube] Check if channel item has subscription count in search 2020-03-21 15:55:37 -03:00
Mauricio Colli
9704fc9952
Improve search extractor tests for services 2020-03-21 15:55:36 -03:00
Mauricio Colli
d72130edae
Always return a new item collector in the search extractor 2020-03-21 15:55:34 -03:00
Mauricio Colli
0c27198ba1
Include errors from child collectors in the search collector 2020-03-21 15:55:29 -03:00
TobiGr
2cf8cbf0df Merge branch 'master' into dev 2020-03-18 22:29:10 +01:00
wb9688
222d659d9e
[SoundCloud] Don't make separate request for getAudioStreams() in StreamExtractor 
Signed-off-by: Stypox <stypox@pm.me>
 2020-03-18 11:30:15 +01:00
wb9688
904c9d681f Fix YouTube videos with no views 2020-03-18 10:07:49 +01:00
Stypox
265cfb61f3
[SoundCloud] Add test for small playlist, use defaultTestGetPageInNewExtractor 2020-03-17 21:03:50 +01:00
Stypox
45bb646480
[SoundCloud] Do not overwrite nextPageUrl in PlaylistExtractor 
Consistent with YouTube and the documentation
 2020-03-17 20:56:47 +01:00
Fynn Godau
a1523eb293 Update Bandcamp service to latest interface changes 2020-03-17 20:40:25 +01:00
Stypox
c505d4e2b7
[SoundCloud] Remove trailing , in playlist page urls 2020-03-17 20:36:59 +01:00
Stypox
ae47c9587c
[SoundCloud] Optimize imports in edited files 2020-03-17 20:32:04 +01:00
Stypox
4389fd3b7b
[SoundCloud] Migrate StreamExtractor to api-v2 2020-03-17 18:06:13 +01:00
Stypox
5e4ddb368f
[SoundCloud] Fix extractors built from next playlist pages 
They didn't have the information to calculate another next page url. So now `nextPageUrl` contains a full link with all video ids, and `getPage` takes the first part of the url (containing 15 streams) and produces another `nextPageUrl` with the remaining streams.
Also add a test for this.
 2020-03-17 18:04:40 +01:00
fynngodau
b78f788017
Merge branch 'dev' into dev 2020-03-17 18:03:14 +01:00
Stypox
0e1b4bbf17
[SoundCloud] Test playlists banner: it should not exist 2020-03-17 15:53:25 +01:00
Stypox
65bdb3bc9d
[SoundCloud] Unignore ignored playlist tests 2020-03-17 15:49:58 +01:00
Stypox
1558da6f6b
[SoundCloud] Fix playlist next page generation 2020-03-17 15:46:35 +01:00
Stypox
ca8bf53b61
[SoundCloud] Fix playlist test: number of streams changed 2020-03-17 15:35:33 +01:00
Stypox
d4aa4a0763
[SoundCloud] Fix typo in ChartsExtractor 2020-03-17 15:25:11 +01:00
Stypox
4b1121aac7
[SoundCloud] Add tests for api-v2 channel urls 2020-03-17 15:23:13 +01:00
Stypox
c3d811fde5
[SoundCloud] Use api-v2 in SubscriptionExtractor 
Also added
 2020-03-17 15:22:25 +01:00
Stypox
d0e66cc600
[SoundCloud] Improve thumbnail url extraction in playlists 
Prevent NullPointerExceptions and remove duplicate code
 2020-03-17 15:13:28 +01:00
Stypox
f3095713f9
[SoundCloud] Use api-v2 in PlaylistExtractor 
Rewrote methods to calculate next page url and to get items from it. `api-v2` is different from `api` since the initial playlist page contains (usually) the full info of the first 3 streams and only the id of the other. Then the single tracks can be requested in batch using `/tracks?ids=id1,id2,...`.
 2020-03-17 15:12:13 +01:00
TobiGr
9eca7df947 Forget to fetch the page 2020-03-16 20:50:08 +01:00
TobiGr
edb57840d7 [SoundCloud] Update hard-coded client_id 2020-03-16 19:43:36 +01:00
TobiGr
efad0be99f Test if the extractor can get AudiStreams with the hard-coded cliend_id 
SoundCloud client_id are changed on a regular basis now. We might want to remove the hardcoded id completely on a later point of time when client_id is changed to often
 2020-03-16 19:42:35 +01:00
TobiGr
5f39bc4e99 [SoundCloud] Update hardcoded client id 2020-03-16 17:26:30 +01:00
Tobias Groza
65a7eda446
Merge pull request #256 from B0pol/mediaccc 
Support for non-api mediaccc links, invidious shortened links, refactor mediaccc tests
 2020-03-15 17:04:09 +01:00
TobiGr
40b3bc138f Remove hack to fix YoutubeStreamLinkHandlerFactoryTest of vnd.youtube videos 2020-03-14 21:54:54 +01:00
Tobias Groza
1c954fbded
Merge pull request #277 from XiangRongLin/vnd 
Handle youtube stream urls in "vnd.youtube:videoId" format.
 2020-03-14 21:52:49 +01:00
Tobias Groza
9edd0f2675
Merge pull request #288 from Stypox/fix-unlisted-video 
[YouTube] Fix videos without next stream throwing error and add test for unlisted videos
 2020-03-14 21:36:13 +01:00
Stypox
957b78ed9b
[YouTube] Add test for an unlisted video, which has no next stream 2020-03-14 14:04:01 +01:00
Stypox
325a51f73a
[YouTube] Do not crash if there is no next stream, but return null 
Like every other extractor does if there is no next stream
 2020-03-14 14:02:48 +01:00
TobiGr
aea0962cfe [SoundCloud] Update hardcoded client id 2020-03-14 12:41:44 +01:00
Mauricio Colli
b086e9db3f
[YouTube] Fix id extraction for some channels 
Some channels had no reliable way to get the redirected id in the response,
so saving it for later was a valid alternative.
 2020-03-14 02:39:49 -03:00
Mauricio Colli
00d1ed439b
[YouTube] Fix channel extraction when redirects are in the response 
Some redirects were embed directly into the response as instructions
for the page, instead of the usual http redirects.
 2020-03-14 02:39:48 -03:00
bopol
66518ec444 check wether mediaccc streams url are secure 2020-03-13 20:07:36 +01:00
bopol
0cd5e05b7b MediaCCCLH: use substring instead of replace; improve a bit tests, return Collections.emptyList(); instead of null where it's annotated @NonNull 2020-03-13 20:00:49 +01:00
bopol
f742a6bd3e code optimization 2020-03-13 20:00:49 +01:00
bopol
9701c7d800 invidious shortened links 2020-03-13 20:00:49 +01:00
bopol
e8e535b815 mediaccc: update linkhandlers & refactor tests 2020-03-13 20:00:49 +01:00
Mauricio Colli
70abd57852
[YouTube] Cache commonly used stream type result 2020-03-07 16:48:32 -03:00
Mauricio Colli
1ef706f567
[YouTube] Detect LIVE videos in the trending page 2020-03-07 16:48:31 -03:00
Xiang Rong Lin
cc8fb486ee Handle youtube stream urls in "vnd.youtube:videoId" format. 
The official YouTube app accept intents in this format, which causes other developers to use it.
 2020-03-02 20:40:33 +01:00
Mauricio Colli
6aa4d59b91
[PeerTube] Update video used in age limit test (the previous was 404) 2020-03-01 10:29:30 -03:00
Mauricio Colli
cfc278317d
[YouTube] Fail-fast if status exist and is anything other than "OK" 2020-03-01 10:29:29 -03:00
Mauricio Colli
90ae5fbea2 Improve kiosk tests across services 2020-03-01 12:57:18 +01:00
Mauricio Colli
e590417cc4 Test if services recognizes their own items urls 2020-03-01 12:57:18 +01:00
TobiGr
35252235b0 Go through all badges when checking if a StreamInfoItem is a live stream 2020-03-01 12:52:37 +01:00
Mauricio Colli
5686a6f562 [YouTube] Detect when a stream is deleted or doesn't exist 
Added a test case as well.
 2020-03-01 12:49:13 +01:00
Mauricio Colli
e65333c3ce [YouTube] Detect deleted/nonexistent/invalid channels and playlists 
- Added tests for these cases.
 2020-03-01 12:49:13 +01:00
Mauricio Colli
98e359438a [YouTube] Detect simple 404s in the standard fetch method 2020-03-01 12:49:13 +01:00
Mauricio Colli
408f042127 [YouTube] Fix bug when url isn't present in the browseEndpoint object 2020-03-01 12:49:13 +01:00
Mauricio Colli
342bdbb852 [YouTube] Avoid crashing by letting exceptions bubble up 2020-03-01 12:49:13 +01:00
Mauricio Colli
e9644e6216 [YouTube] Handle video premiere's date and duration 2020-03-01 12:49:13 +01:00
Mauricio Colli
2a470ac4f7 [YouTube] Handle videos with no views or with "Recommended to you" text 2020-03-01 12:49:13 +01:00
Mauricio Colli
da2c562eb0 [YouTube] Take into account videos that have their views hidden 
Also remove catch because it may hide bugs when the page changes.
 2020-03-01 12:49:13 +01:00
Mauricio Colli
f66c20de54 Ignore null-keyed entries when iterating through the response headers 2020-03-01 12:49:13 +01:00
Mauricio Colli
5edd774fc4 Add latest url to the response to make detection of a redirect possible 
Will be latest one in this commit because there's need to check the
history of redirects as of now.
 2020-03-01 12:49:13 +01:00
Mauricio Colli
3441946bea Make test downloader return a response instead of throwing an exception 
The test implementation was throwing an exception instead of just
returning the response and letting the caller handle it.
 2020-03-01 12:49:13 +01:00
wb9688
df219a9453 Don't use StandardCharsets class 2020-03-01 09:41:06 +01:00
wb9688
2807079531 Catch NumberFormatException instead of checking whether viewCount equals "Recommended for you" 2020-02-29 21:28:38 +01:00
wb9688
3e4d2fd64e Improve testChannelUrl() 2020-02-29 18:20:35 +01:00
wb9688
bfe3eb1409 Use suggested try-if code style 2020-02-29 17:18:50 +01:00
wb9688
a6a63e9570 Let isHardcodedClientVersionValid() throw IOException 2020-02-29 17:01:36 +01:00
wb9688
45df8248bf Use Localization in getJsonResponse() 2020-02-29 16:55:07 +01:00
wb9688
157055fbfd Create getJsonResponse() function 2020-02-29 16:42:04 +01:00
wb9688
4f406512c2 Use YoutubeChannelLinkHandlerFactory 2020-02-29 16:26:28 +01:00
wb9688
4ff655352d Reimplement getAgeLimit() 2020-02-28 17:14:26 +01:00
wb9688
6ba03e542b Cache getVideoTab() 2020-02-28 17:05:31 +01:00
wb9688
8bfbae904d Cache getVideo(Primary|Secondary)InfoRenderer() 2020-02-28 17:03:21 +01:00
wb9688
0b940f5cf4 Use "getAgeLimit() != NO_AGE_LIMIT" instead of "getAgeLimit() == 18" 2020-02-28 16:40:50 +01:00
wb9688
beb07a4852 Add test to check whether hardcoded client version is still valid 2020-02-28 16:36:14 +01:00
wb9688
b89b6b1db2 Check if hard-coded client version is valid in separate function 2020-02-28 16:35:24 +01:00
wb9688
7898ba230d Use pbj in YoutubeStreamExtractor 2020-02-28 15:17:47 +01:00
wb9688
612ec06839 Actually fix thumbnail URLs starting with // 2020-02-28 09:36:33 +01:00
wb9688
f946310af5 Fix search queries with a promotedSparklesTextSearchRenderer returning 0 results 2020-02-28 08:57:44 +01:00
wb9688
94a2d5be46 Fix thumbnail URLs starting with // 2020-02-27 19:08:46 +01:00
wb9688
838d834ce2 Get Uploads view of Videos tab 2020-02-27 18:54:27 +01:00
wb9688
365b0329f3 Implement getTextFromObject() function 2020-02-27 17:39:23 +01:00
wb9688
0798bdd5cd Fix getSubscriberCount() for 0 subscribers 2020-02-27 16:26:22 +01:00
wb9688
8ebd971648 Fix channel descriptions consisting of multiple parts 2020-02-27 10:06:35 +01:00
wb9688
880b951088 Fix getUploaderAvatarUrl() in YoutubeStreamExtractor 2020-02-27 09:53:59 +01:00
wb9688
8aa560bef4 Fix video titles consisting of multiple parts due to @ 2020-02-27 09:42:22 +01:00
wb9688
951ed9f0d8 Don't call getUploaderInfo() in onFetchPage() for playlists 
This way YouTube Music playlists will keep working in NewPipe
 2020-02-27 09:27:56 +01:00
wb9688
2399d46ab0 Fix YoutubeChannelExtractor for channels without videos 2020-02-27 09:21:25 +01:00
wb9688
cdbc751b87 Fix getUploaderUrl() in YoutubeStreamInfoItemExtractor 2020-02-27 09:05:41 +01:00
wb9688
db305408e9 Get all expandedShelfContentsRenderers in YoutubeTrendingExtractor 2020-02-26 20:37:20 +01:00
wb9688
a2d8489e16 Parse timestamps in description 2020-02-26 20:27:30 +01:00
wb9688
8d503407ef Parse browseEndpoint and watchEndpoint in descriptions 2020-02-26 19:37:58 +01:00
wb9688
8dc39d517a Fix tests 
Except testGetFullLinksInDescription() in DescriptionTestUnboxing, since that requires other changes first.
 2020-02-26 17:31:01 +01:00
wb9688
88282761f4 Fix YoutubeStreamExtractor not being able to getAgeLimit() in onFetchPage() 2020-02-26 16:19:42 +01:00
wb9688
bb3a3d70bf Use pbj in YoutubeTrendingExtractor 2020-02-26 16:10:24 +01:00
wb9688
1c3a4e710b Check if there are no related streams in getAgeLimit() 2020-02-26 16:03:06 +01:00
wb9688
985c3ec877 Improve getClientVersion() 2020-02-26 15:27:50 +01:00
wb9688
15afbea3e1 Use pbj in YoutubePlaylistExtractor 2020-02-26 15:27:50 +01:00
wb9688
89a41a7f69 Use pbj in YoutubeChannelExtractor 2020-02-26 15:27:50 +01:00
wb9688
0973263aab Use pbj in YoutubeSearchExtractor 2020-02-26 15:27:46 +01:00
TobiGr
fc465c8bf7 Fix getUploadDate() for premiered videos with a given date 
Detect publishDate and uploadDate
 2020-02-25 22:54:37 +01:00
TobiGr
96285e0815 Fix getUploadDate() for premiered videos with a given date 2020-02-25 21:51:30 +01:00
wb9688
5a35300a5e Fix parsing upload date for premiered streams 2020-02-25 21:51:30 +01:00
wb9688
729fd2eaee Fix parsing description for navigationEndpoint without urlEndpoint 2020-02-25 21:51:30 +01:00
wb9688
26fb44595f Fix parsing description 2020-02-25 21:51:30 +01:00
TobiGr
0fff03038e Get description with correct links 2020-02-25 21:51:30 +01:00
wb9688
5d08c34322 Fix parsing upload date in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688
bbe1a3cd62 Use getVideo(Primary|Secondary)InfoRenderer() 2020-02-25 21:51:30 +01:00
wb9688
8e6be880d7 Simplify getId() in YoutubeChannelExtractor 2020-02-25 21:51:30 +01:00
TobiGr
ecfc27a92c Implement fallback methods for getDescription() and getTextualUploadDate() in YouTubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688
eed29ea734 Return null for related streams at age restricted videos 2020-02-25 21:51:30 +01:00
TobiGr
9efcc61ca6 Adapt some tests to match the new URLs 2020-02-25 21:51:30 +01:00
TobiGr
f39603f6ef Implement getUploadDate() in YouTubeStreamInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688
26ea3dceb6 Fix getViewCount() in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688
76d54abdbf Reimplement more methods in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688
f13c0288cc Reimplement some methods in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688
02b59903fa Remove useless code 2020-02-25 21:51:30 +01:00
TobiGr
dd4dd849dc Get client version dynamically in YouTubeSearchExtractor 2020-02-25 21:51:30 +01:00
TobiGr
583e9c1571 Fix getPage() 
Someone should check if there are unstaged changes before committing...
 2020-02-25 21:51:30 +01:00
TobiGr
5842b9ad37 Add getClientVersion() and HARDCODED_CLIENT_VERSION to YouTubeParsingHelper 
Prefer hardcoded client version above the current one when making requests to retrieve the same JSON structure for each request.
 2020-02-25 21:51:30 +01:00
wb9688
5d883d100c Implement pagination in YoutubeSearchExtractor 2020-02-25 21:51:30 +01:00
wb9688
c0a8e01889 Implement pagination in YoutubePlaylistExtractor 2020-02-25 21:51:30 +01:00
wb9688
4039409820 Fix getLikeCount() for certain streams 2020-02-25 21:51:30 +01:00
wb9688
627ab2459f Uncomment fetchPage() 2020-02-25 21:51:30 +01:00
wb9688
2dfa2187ff Implement pagination in YoutubeChannelExtractor 2020-02-25 21:51:30 +01:00
wb9688
a38ab9b791 Return null if channel has no banner 2020-02-25 21:51:30 +01:00
wb9688
1eca63089b Count private and deleted videos as ad 2020-02-25 21:51:30 +01:00
wb9688
c7360e4a46 Fix getThumbnailUrl() in YoutubeChannelInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688
51fb26625a Implement getSubscriberCount() in YoutubeChannelInfoItemExtractor 2020-02-25 21:51:30 +01:00
TobiGr
6d20b2b1d0 Fix NPE on collecting page items 2020-02-25 21:51:30 +01:00
wb9688
4462cbe3f1 Reimplement YoutubePlaylistInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688
8aea4d445b Reimplement YoutubeChannelInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688
3187116a63 Handle premium videos 2020-02-25 21:51:30 +01:00
wb9688
7442f91aa6 Don't assume which channel tab is videos 2020-02-25 21:51:30 +01:00
wb9688
316fe0109d Fix first playlist page 2020-02-25 21:51:30 +01:00
wb9688
4e57e589ce Fix first channel page 2020-02-25 21:51:30 +01:00
TobiGr
5816202cc7 Move getInitialData() method to YouTubeParsingHelper 
Rename ytInitialData to initialData
 2020-02-25 21:51:30 +01:00
wb9688
38aabc6aca Use plain YoutubeStreamInfoItemExtractor in YoutubeTrendingExtractor 2020-02-25 21:51:30 +01:00
wb9688
957db062ef Fix getting uploader for videoRenderer 2020-02-25 21:51:30 +01:00
wb9688
b88188d419 Move stuff from extractVideoPreviewInfo() into YoutubeStreamInfoItemExtractor and partially fix search 2020-02-25 21:51:30 +01:00
wb9688
af49b3c487 Fix live streams in extractVideoPreviewInfo() 2020-02-25 21:51:30 +01:00
TobiGr
f33f9466ce Fix testDescription() in ChannelExtractor 2020-02-25 21:51:30 +01:00
TobiGr
10e9c16d8c Fix LiveStream view count 2020-02-25 21:51:30 +01:00
wb9688
127c4d5893 Fix getNextStream() in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688
7bfc0e62c6 Fix getRelatedStreams() in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
TobiGr
12166ca60c [StreamExtractor] Add fallback method for uploader avatar url 2020-02-25 21:51:30 +01:00
wb9688
3ec422a542 Fix like/dislike parsing 2020-02-25 21:51:30 +01:00
TobiGr
a34e5c18e1 Trending StreamInfoItemCollecetor 2020-02-25 21:51:30 +01:00
TobiGr
b705515da0 Adapt most playlist fields 2020-02-25 21:51:30 +01:00
TobiGr
21253abd9b Use Firefox ESR User-Agent: 68 2020-02-25 21:51:30 +01:00
TobiGr
655c999795 Detect disabled subriber count correctly 
Fix parsing of kiosk name
 2020-02-25 21:51:30 +01:00
TobiGr
7dcc9f159b Stream: fix uploader avatar url 2020-02-25 21:51:30 +01:00
TobiGr
be3e20e263 Adapt most channel changes 2020-02-25 21:51:30 +01:00
TobiGr
39bf1ff1eb Add ytInitialData regex 2020-02-25 21:51:30 +01:00
Tobias Groza
a129c65c63
more improvements of comments 
Co-Authored-By: Stypox <stypox@pm.me>
 2020-02-16 23:59:31 +01:00
TobiGr
0a4115286b Apply one suggestion 2020-02-15 13:36:54 +01:00
Tobias Groza
6822574588 Fix typos 
Co-Authored-By: B0pol <bopol@e.email>
 2020-02-15 13:36:54 +01:00
TobiGr
030465b5d4 Improve code formatting and optimise imports 2020-02-15 13:36:54 +01:00
TobiGr
9112a106ea [SoundCloud] Update client_id 2020-02-08 23:09:03 +01:00
TobiGr
008b6f505c [PeerTube] Fix subscriber count in ChannelExtractorTest 2020-02-08 23:08:37 +01:00
TobiGr
0e33249c85 Fix SoundCloud description test 2020-02-08 22:20:20 +01:00
Fynn Godau
36a316eea5
Merge branch 'dev' into dev 2020-02-08 09:43:13 +01:00