XiangRongLin
fcdb9bdbeb
add url to sout
2021-07-15 20:05:26 +02:00
XiangRongLin
80cf8b3acd
Extract separate YoutubeThrottlingDecoder
2021-07-15 20:05:25 +02:00
XiangRongLin
a86a30103f
[Youtube] bare bones version to solve throttling
...
Done by transforming the parameter "n" from videoplayback urls
https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-865985377
2021-07-15 20:05:25 +02:00
bopol
c38a06e8dc
Merge pull request #661 from B0pol/peertube-livestreams
...
[PeerTube] Support livestreams
2021-07-13 15:48:13 +02:00
litetex
fdebf3c6cd
Added javadoc warnings
2021-07-07 20:41:59 +02:00
litetex
6860543b07
Added more doc
2021-07-06 21:16:31 +02:00
litetex
a59fead0d7
Fixed typos
2021-07-06 21:16:30 +02:00
litetex
688a1c316b
Fixed Exception when YT comments are disabled and added `commentsDisabled
` field
...
* Fixed code: Added missing finals (according to NewPipes Checkstyle guide)
* Fixed ``findValue`` method in ``YoutubeCommentsExtractor``
2021-07-06 21:16:29 +02:00
litetex
ca33f4f60b
Improved code format (final) and added `isCommentsDisabled
` method
2021-07-06 21:16:28 +02:00
TobiGr
dbdc962a07
Update mocks
2021-07-03 13:30:26 +02:00
TobiGr
9ca6dc26ed
Fix invalid CONSENT cookie value
2021-07-03 13:27:40 +02:00
XiangRongLin
b9549ace25
[Youtube] Fix lofi girl test and ignore mix with id test
2021-06-29 21:12:20 +02:00
XiangRongLin
5982431ffb
Remove IPv4 addresses from response during mock recording
2021-06-29 21:03:33 +02:00
bopol
7d7cc087e2
[PeerTube] Support livestreams
2021-06-24 11:02:30 +02:00
bopol
3966178979
Merge pull request #665 from B0pol/agerestricted
...
[youtube] Fix extraction of age restricted videos
2021-06-23 21:07:14 +02:00
bopol
b475f09ba7
don't do useless concatenation
2021-06-23 20:33:32 +02:00
bopol
b96b2a6eba
Update invidious instances
2021-06-23 14:39:10 +02:00
bopol
c5e8bd368d
Update Mocks
2021-06-23 12:21:57 +02:00
bopol
e9a992b0a9
[youtube] Fix extraction of age restricted videos
2021-06-23 12:06:02 +02:00
dependabot[bot]
ad403ce73e
Bump spotbugs-annotations from 4.0.2 to 4.2.3
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.0.2 to 4.2.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.0.2...4.2.3 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-06-23 09:57:42 +00:00
bopol
65fd8740aa
Merge pull request #658 from TeamNewPipe/revert-651-fix/streamType
...
[YouTube] Fix getting stream type of live streams
2021-06-23 11:33:24 +02:00
bopol
361f142621
Use badge style instead of label
2021-06-23 11:25:59 +02:00
bopol
58f109ddec
adress stypox review
2021-06-23 11:15:40 +02:00
bopol
fe29c78fe1
Merge pull request #656 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.13
...
Bump rhino from 1.7.12 to 1.7.13
2021-06-20 00:46:42 +02:00
bopol
e9412fdadc
Merge pull request #654 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.7
...
Bump gson from 2.8.6 to 2.8.7
2021-06-20 00:35:24 +02:00
TobiGr
1f9a99c931
[SoundCloud] Update HARDCODED_CLIENT_ID
2021-06-18 15:30:54 +02:00
Tobi
c998012c28
Revert "[YouTube] Fix getting stream type of live streams"
2021-06-18 09:20:32 +02:00
dependabot[bot]
38d0339753
Bump rhino from 1.7.12 to 1.7.13
...
Bumps [rhino](https://github.com/mozilla/rhino ) from 1.7.12 to 1.7.13.
- [Release notes](https://github.com/mozilla/rhino/releases )
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/mozilla/rhino/commits )
---
updated-dependencies:
- dependency-name: org.mozilla:rhino
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-06-17 21:49:07 +00:00
dependabot[bot]
5a46878b09
Bump gson from 2.8.6 to 2.8.7
...
Bumps [gson](https://github.com/google/gson ) from 2.8.6 to 2.8.7.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.6...gson-parent-2.8.7 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-06-17 21:49:01 +00:00
TobiGr
6762fe3357
[YouTube] Fix getting stream type of live streams
2021-06-10 16:51:00 +02:00
TobiGr
f7f727d19c
Add constant NO_LIKE_COUNT to CommentsInfoItem
2021-06-08 10:28:44 +02:00
TobiGr
b70c0f93c7
Add streamPosition for comments
...
SoundCloud is the only service which supports adding comments at a specific timestamp in the stream.
2021-06-08 10:27:52 +02:00
Tobi
d4186d100b
Merge pull request #580 from TeamNewPipe/accountTerminated
...
Add AccountTerminatedException for terminated channels
2021-06-08 09:55:58 +02:00
TobiGr
89e0514d8b
Fix Javadoc generation fix
2021-06-08 09:30:05 +02:00
TobiGr
80d3052033
Fix Javadoc generation
2021-06-08 09:26:01 +02:00
TobiGr
fa444c8298
Remove assertations of exception messages
2021-06-06 11:38:43 +02:00
Stypox
2158ca4060
Remove two unneded warning suppressions
2021-06-05 14:15:48 +02:00
Stypox
13f192704d
Add utility method getService() to Info object
2021-06-05 14:14:19 +02:00
XiangRongLin
0ad51e76fb
Merge pull request #638 from TeamNewPipe/spelling
...
Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl)
2021-06-04 18:23:31 +02:00
XiangRongLin
faa503c7dd
Fix PeertubeAccountExtractorTest
2021-06-03 19:54:42 +02:00
TiA4f8R
ac31f3a883
Specify UTF-8 file encoding in RecordingDownloader and MockDownloader
...
On Windows, mocks are recorded and read with the Cp1252 encoding so it breaks the mocks on non ASCII characters for Linux OS (and so the CI).
The project is in Java 8, so we can't use FileReader(File, Charset) and FileReader(File, Charset) because these methods require Java 11. Instead of changing the Java version of the extractor, use OutputStreamWriter and FileOutputStream instead of FileWriter and InputStreamReader and FileInputStream instead of FileReader.
2021-06-03 18:38:25 +02:00
bopol
ff11c2df2a
Merge pull request #628 from litetex/fix-broken-yt-liked-comments
...
Fix broken yt likes in comments
2021-05-29 11:04:11 +02:00
TobiGr
1ff56a85df
Update junit from 4.13.1 to 4.13.2
...
https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.2.md
2021-05-28 15:09:26 +02:00
TobiGr
858e007804
Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl)
2021-05-28 14:59:44 +02:00
Tobi
519bba70b3
Merge pull request #630 from XiangRongLin/fix_tests
...
Fix MetaInfoTest.expectedMetaInfo
2021-05-28 14:16:53 +02:00
litetex
289b84e68b
Removed unused code
2021-05-27 21:27:44 +02:00
litetex
e81b0e2885
Tweaked code
2021-05-27 19:48:31 +02:00
litetex
8c96545e57
Merge branch 'TeamNewPipe:dev' into fix-broken-yt-liked-comments
2021-05-24 18:06:45 +02:00
litetex
12fb18c310
Updated mock data (only the affected)
2021-05-24 18:05:26 +02:00
litetex
8e08a2aac7
Improved docs
2021-05-24 18:05:26 +02:00
litetex
020acfed71
Fixed tests and added new ones
2021-05-24 18:05:26 +02:00
litetex
b310922fc0
Better exception message
2021-05-24 18:05:25 +02:00
litetex
86bb9efb5d
Added likeCount back to tests
2021-05-24 18:05:25 +02:00
litetex
46654f5fca
Reformatted code
2021-05-24 18:05:25 +02:00
litetex
2174685c5c
Reimplemented likeCount
2021-05-24 18:05:25 +02:00
litetex
01cfb55505
Update YoutubeCommentsInfoItemExtractor.java
2021-05-24 18:05:25 +02:00
litetex
bedcd87abb
Fixed occurrences where voteCount is not set (<=0 likes)
2021-05-24 18:05:25 +02:00
litetex
545c0a6f42
Fixed comment
2021-05-24 18:05:24 +02:00
TiA4f8R
a00fdcbd3d
Format the indentation of the definition of the apiUrl string in SoundcloudParsingHelper.resolveFor method
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-05-23 20:27:45 +02:00
TiA4f8R
6b607eb38d
Update the hardcoded client_id
2021-05-23 18:08:44 +02:00
TiA4f8R
4552ea9c9f
Use the SOUNDCLOUD_API_V2_URL constant in all the SoundCloud package
2021-05-23 18:08:44 +02:00
TiA4f8R
86308d0603
Use a lightweight request to check if the client_id is valid
...
Request the api-v2 host with the client_id instead of checking if the streams of a SoundCloud track are not empty: if it is valid, the API returns 404, otherwise it should return 401.
2021-05-23 18:08:43 +02:00
TiA4f8R
c5c190500c
Format the SoundCloud package of the extractor and fix some warnings
...
Use final where possible in the package and format code to be in the 100 caracters per line limit.
Fix some warnings generated by Android Studio and do some code improvements
2021-05-23 18:08:42 +02:00
Tobi
636e27333b
Merge pull request #603 from TeamNewPipe/peertube_hls
...
Add PeerTube HLS streams
2021-05-23 11:49:40 +02:00
XiangRongLin
172394cd52
Update mocks
2021-05-22 20:33:20 +02:00
XiangRongLin
18c8cac40c
Fix MetaInfoTest.expectedMetaInfo
...
Expected meta changed
2021-05-21 21:00:08 +02:00
Hassan Igbaria
e8b5364745
Update YoutubeStreamExtractor.java
2021-05-21 16:22:24 +03:00
litetex
c3b837fe3b
Deduped code; Using default values...
2021-05-20 20:22:40 +02:00
litetex
b934c7ccbb
Removed unused import
2021-05-20 20:11:35 +02:00
litetex
10cf081145
Removed likeCount and added textualVoteCount
2021-05-20 20:10:14 +02:00
TobiGr
1acc53b8db
[PeerTube] Add support for HLS streams
2021-05-15 12:39:23 +02:00
Tobi
3a3ade20f4
Merge pull request #620 from TeamNewPipe/bandcamp-comments
...
Add Bandcamp comments (first page)
2021-05-02 13:43:53 +02:00
Fynn Godau
85a4270bd9
[Bandcamp] Fix comments test
2021-05-02 13:07:24 +02:00
TobiGr
ec50570053
Add mocks
2021-05-01 17:50:19 +02:00
TobiGr
f4404b5cc8
Add tests for more reasons
2021-05-01 17:50:19 +02:00
TobiGr
fc998589dc
Detect channels which have been terminated due to copyright infringement
2021-05-01 17:50:19 +02:00
TobiGr
bb3861ddce
Add AccountTerminatedException for better error handling of terminated channels
2021-05-01 17:50:19 +02:00
Tobi
284362fc15
Merge pull request #608 from TeamNewPipe/youtube-fix-segment-test
...
[YouTube] fix tests
2021-05-01 17:24:41 +02:00
TobiGr
9c12dc5609
[SoundCloud] Fix SoundCloud ID extraction
...
resolveIdWithEmbedPlayer() does not work anymore because the JSON data has been extracted to an API call. For this reason, replace resolveIdWithEmbedPlayer() with resolveIdWithWidgetApi)( which performs the API call.
2021-04-26 17:58:34 +02:00
Fynn Godau
9dc7a1d5ee
[Bandcamp] Add comments (first page)
2021-04-26 17:25:04 +02:00
TobiGr
8f023c1ec7
[SoundCloud] Update client id
2021-04-26 15:14:11 +02:00
Tobi
7f202db8b1
Fix Google URL detection ( #617 )
2021-04-26 11:49:47 +02:00
fynngodau
6db4bea8ca
Rework Bandcamp comments ( #613 )
2021-04-13 21:10:59 +02:00
Fynn Godau
3671876721
Fix bandcamp radio stream extractor
2021-04-13 19:30:11 +02:00
Fynn Godau
90b5c00599
Extract related items #593
2021-04-13 17:56:19 +02:00
TobiGr
1fe645704f
[YouTube] FIx testMoreRelatedItems in RandomQueryNoMorePages
2021-04-11 23:21:35 +02:00
TobiGr
9a2b814d8f
Update mocks
2021-04-11 22:59:01 +02:00
TobiGr
50005ce937
[YouTube] Replace unavailable YouTube Stream Segments test
...
The stream is unavailable in many countries.
2021-04-11 22:58:48 +02:00
TobiGr
318cec7625
[Bandcamp] Add Radio stream segments
2021-04-11 21:19:44 +02:00
TobiGr
55e0f8e725
Merge branch 'master' into dev
2021-04-10 11:33:34 +02:00
TobiGr
33173eb3e6
Make CONSENT COOKIE constants private
2021-04-09 11:59:17 +02:00
TobiGr
b94b316558
Update mocks
2021-04-09 11:59:17 +02:00
TobiGr
4b8d4a84b6
Set the NumberGenerator for th remaining tests
2021-04-09 11:59:17 +02:00
XiangRongLin
db7ad9c27d
Update Mocks
2021-04-08 16:37:29 +02:00
XiangRongLin
4833df60e5
Add method to inject Random into YoutubeParsingHelper and use in tests
2021-04-08 16:36:55 +02:00
TobiGr
682ec27737
Update mocks
2021-04-08 14:22:31 +02:00
TobiGr
080d1e318d
Code improvement
2021-04-07 14:11:47 +02:00
TobiGr
4ee270fe01
Fix LiveStream Test
2021-04-07 14:11:41 +02:00
TobiGr
883f16e0ad
[YouTube] Set CONSENT cookie
2021-04-07 14:11:37 +02:00
XiangRongLin
aee3838ed5
Ignore unreliable SoundcloudGoPlusTrack.testRelatedItems
2021-04-06 18:26:45 +02:00
XiangRongLin
7b06c696e2
Use Page.body for YoutubeChannelExtractor
2021-04-06 18:14:38 +02:00
XiangRongLin
eda1b6e199
Use Page.body for YoutubePlaylistExtractor
2021-04-06 18:14:38 +02:00
XiangRongLin
3af26a2821
Add body field to Page
2021-04-04 11:39:58 +02:00
XiangRongLin
7b36469807
Fix SoundcloudGoPlusTrack
...
https://soundcloud.com/martinsolveig/places#t=0 has related streams
2021-04-04 09:39:26 +02:00
XiangRongLin
3d70b6d46a
Fix a test and ignore rest
2021-04-03 11:03:10 +02:00
XiangRongLin
1925dcf4dc
Merge pull request #597 from fynngodau/related-items
...
Rename getRelatedStreams to getRelatedItems and change return type
2021-04-02 19:54:22 +02:00
Fynn Godau
c877712647
Add deprecated old method calls to avoid breaking API
2021-04-01 22:32:36 +02:00
Fynn Godau
14f6f1b7c3
Generify related streams calls and rename method
2021-03-31 20:24:40 +02:00
TobiGr
1498e1905e
Reduce number of type conversions in YouTubeSearchExtractor.collectStreamsFrom by introducing new variable
2021-03-31 13:30:38 +02:00
Tobi
7ea2cd73d3
Merge pull request #589 from fynngodau/bandcamp-more-recent-pages
...
Load more featured pages and fix featured cover arts not loading
2021-03-31 12:15:12 +02:00
Fynn Godau
705f6c6e33
Apply review
2021-03-31 11:02:30 +02:00
TobiGr
657b00ca11
Smaller code improvements
2021-03-30 10:10:53 +02:00
TobiGr
8fbd1a2bcf
FIx potential NullPointerException
2021-03-30 10:07:20 +02:00
TobiGr
f2e7c7ebd6
Replace Vector with ArrayList for performance improvements
...
I could not figure out, why these two methods are necessary.
2021-03-30 10:07:20 +02:00
TobiGr
48fd8ca7b3
Remove redundant type declaration
2021-03-30 10:07:20 +02:00
TobiGr
22bd16fca3
Make vars private, final and static
2021-03-30 10:07:20 +02:00
TobiGr
6a0f6e846a
Fix possible bug when language.id field is not present and this the languageCode an empty String.
2021-03-30 10:07:20 +02:00
TobiGr
b9282bbe94
Remove checks which are always true
2021-03-30 10:07:20 +02:00
TobiGr
46eab1ec17
Remove declarations of exceptions which are not thrown from method signatures
2021-03-30 10:07:20 +02:00
TobiGr
c3e23559d7
Remove unused imports
2021-03-30 10:07:20 +02:00
TobiGr
e4c40ae6d1
Use List.isEmpty()
2021-03-30 10:07:20 +02:00
Fynn Godau
d663be5a78
[Bandcamp] Fix featured albums not showing thumbnails
2021-03-28 13:12:22 +02:00
Fynn Godau
dbcf61c6f7
[Bandcamp] Load more featured pages
2021-03-28 12:53:05 +02:00
Stypox
b4dee6d08f
Merge pull request #306 from B0pol/metadata
...
Extract metadata for youtube, soundcloud & mediaccc
2021-03-27 08:45:47 +01:00
TobiGr
070a40e181
[YouTube] Mix Playlist - Fix getting name
2021-03-25 21:47:16 +01:00
TobiGr
b029779217
[media.cc.de] Disable unreliable tests for now
2021-03-25 19:44:44 +01:00
TobiGr
bbee15474d
[PeerTube] Fix playlist extraction
2021-03-25 19:19:19 +01:00
TiA4f8R
ae283314da
Add a comment explaining subscribers count calculation
2021-03-25 17:54:46 +01:00
TiA4f8R
abb790f465
Remove println in getSubscriberCount from PeertubeAccountExtractor class
2021-03-25 17:49:22 +01:00
TiA4f8R
772de53a66
Increase the number of subscribers in PeertubeAccountExtractorTest
2021-03-25 17:49:20 +01:00
TiA4f8R
efe2b964f0
Extract real subscribers of a Peertube account
...
Apply fix provided in the PR and move all accounts/ strings to a constant, ACCOUNTS.
2021-03-25 17:47:44 +01:00
Tobi
12835bfae1
Merge pull request #551 from FireMasterK/ff-privacy
...
Change UA to privacy.resistFingerprinting.
2021-03-24 17:58:34 +01:00
Tobi
cae6cea551
Merge pull request #582 from Stypox/fix-trending
...
[YouTube] Fix trending getName()
2021-03-24 10:49:46 +01:00
Stypox
ce7cbbc9a0
[YouTube] Fix trending getName()
2021-03-24 09:14:09 +01:00
XiangRongLin
9009f1e277
Merge pull request #517 from XiangRongLin/flaky_test
...
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-03-20 18:04:14 +01:00
bopol
152221c7fb
ignore a test with encoding problems
2021-03-19 10:14:43 +01:00
bopol
8806fb4e6b
address reviews
2021-03-19 10:14:43 +01:00
bopol
c47cc54908
Extract metadata for YouTube, SoundCloud & MediaCCC
2021-03-19 10:14:43 +01:00
TiA4f8R
379d7312fa
Don't use a regular expression to find the last segment URL and do code improvements
...
Apply suggestions provided in the PR and remove a redundant import.
2021-03-14 17:54:22 +01:00
TiA4f8R
0e3e420a25
Fix tests
2021-03-14 17:54:21 +01:00
TiA4f8R
d61d9d116d
Refactor getAudioStreams method of SoundcloudStreamExtractor
...
Split the method into private methods, in order to have a better reading.
2021-03-14 17:54:20 +01:00
TiA4f8R
a7b15b51e6
Change t to transcodingJsonObject in SoundcloudStreamExtractor
2021-03-14 17:54:19 +01:00
TiA4f8R
0438828e36
Add a test for the number of audioStreams in CreativeCommonsPlaysWellWithOthers test
...
It should be only two audio streams for track "Plays Well with Others, Ep 2: What Do an Army of Ants and an Online Encyclopedia Have in Common?" by Creative Commons (https://soundcloud.com/wearecc/plays-well-with-others-ep-2-what-do-an-army-of-ants-and-an-online-encyclopedia-have-in-common ):
- one which is a progressive stream, in MP3 format with a bitrate of 128 kbps
- one which is an HLS stream, in OPUS format with a bitrate of 64 kbps.
2021-03-14 17:54:18 +01:00
TiA4f8R
3bd08a2880
Adress requested changes and use final where possible in SoundcloudStreamExtractor
...
This commit moved the HLS parsing task to a separate method, did little performance improvements and used final where possible in the SoundcloudStreamExtractor file.
2021-03-14 17:54:17 +01:00
TiA4f8R
cbacd3c0a5
Add a check to don't show MP3 128kbps stream twice and catch IOException when fetching the HLS Manifest
...
If a progressive stream is present in the transcodings, it's unnecessary to show twice an MP3 128kbps stream so if this is the case, the MP3 HLS stream will be not added to the audioStreams, else it will.
This commit also catch fetching errors in HLS manifests parsing and don't add the corresponding stream if an error occurs.
2021-03-14 17:54:16 +01:00
TiA4f8R
26f1b4e7dc
Support SoundCloud HLS by using a workaround
...
This commit tries to support SoundCloud HLS streams by parsing M3U manifests, get the last segment URL (in order to get track length) and request a segment URL equals to track's duration so it's a single URL.
2021-03-14 17:54:14 +01:00
Fynn Godau
2e57a8f24f
[Bandcamp] Fix link handler acceptance behaviour
...
* Test for bandcamp footer instead of meta tag (which is not present on
all pages)
* Accept links to /music, not just /releases
* Correctly handle uppercase URLs
2021-03-14 09:51:30 +01:00
TobiGr
70d9e389b9
[Bandcamp] Add tests for external bandcamp artists
...
Necessary, because the external pages tested before were converted to bandcamp.com pages. See f9d06252f2
2021-03-13 20:51:59 +01:00
TobiGr
f9d06252f2
[Bandcamp] Fix tests
2021-03-13 17:42:39 +01:00
Tobi
7e6f464407
Merge pull request #509 from TiA4f8R/soundcloud-improvements
...
Add new exceptions to be able to display different error messages in apps
2021-03-07 13:51:15 +01:00
Tobi
ac51134aed
Merge pull request #546 from XiangRongLin/unignore2
...
Unignore more tests
2021-03-07 13:15:24 +01:00
TobiGr
22fa131922
Merge branch 'dev' into bandcamp
2021-03-05 19:45:37 +01:00
TiA4f8R
e55284bb8f
Code style improvements
...
Use final where possible in YoutubeStreamExtractor and do some other code style improvements
2021-03-05 16:38:53 +01:00
TobiGr
d0a1041afd
Prevent NullPointerException and use equalsIgnorecase
2021-03-05 16:38:52 +01:00
TiA4f8R
890cbba625
Update PaidContent exception
2021-03-05 16:38:51 +01:00
TobiGr
da3cfa967d
Handle age-restricted videos
2021-03-05 16:38:50 +01:00
TiA4f8R
448b68700c
Use final in exceptions and add AgeRestrictedException
...
AgeRestrictedException will be thrown only if the reason message equals to "Sign in to confirm your age" and if the age limit is 18.
2021-03-05 16:38:49 +01:00
TobiGr
22e6f33f0a
Ignore SoundCloudGeoRestrictedTest until HLS stream extraction is added
...
The ContentNotSupportedException is thrown because no supported audio streams where extracted. However, SoundCLoud does not check, whether there are any streams available.
This commit should be reverted in #526
2021-03-05 16:38:47 +01:00
TobiGr
21158744b1
Add expectedDescriptionIsEmpty() to DefaultStreamExtractorTest
...
Also check if related streams are empty if they are expected to be empty.
2021-03-05 16:38:46 +01:00
TiA4f8R
f15d7837a1
Fix SoundCloud tests
2021-03-05 16:38:45 +01:00
TiA4f8R
4e45aef2b3
Fix SoundCloud CreativeCommonsPlaysWellWithOthers test
2021-03-05 16:38:43 +01:00
TiA4f8R
59d6d3f04e
Fix YouTube tests
2021-03-05 16:38:42 +01:00
TiA4f8R
3925204658
Add tests and rename SoundcloudGoPlusException to SoundcloudGoPlusContentException
2021-03-05 16:38:40 +01:00
TiA4f8R
771bb1a2cb
Add three new exceptions for unavailable YouTube contents
...
These exceptions are thrown on a test with the error messages text, because YouTube returns only "UNPLAYABLE" status in most error cases.
Tests are based with English strings, so changing the lang used by
extractor will throw the generic exception (ContentNotAvailableException).
2021-03-05 16:38:39 +01:00
TiA4f8R
35325d980d
Add GeographicRestrictionException and SoundCloudGoPlusException in NewPipe Extractor to be able to display different error messages
...
This commit adds two new exceptions in NewPipe Extractor: GeographicRestrictionException and SoundCloudGoPlusException (which extend to ContentNotAvailableException). These exceptions allow showing different error messages to user when a content isn't available in his/her/its country (only used for now by SoundCloudStreamExtractor) or when the content is a SoundCloud Go+ track.
2021-03-05 16:38:37 +01:00
Tobi
bc0cda68d1
Merge pull request #570 from TiA4f8R/fix-yt-channels-playlists
...
Fix YouTube channels contination
2021-03-05 15:26:49 +01:00
Tobi
a3c6fceef5
Merge pull request #573 from B0pol/comments-performance
...
[youtube] improve comments extraction performance
2021-03-05 14:46:27 +01:00
bopol
ff5273b882
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeCommentsExtractor.java
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-03-05 14:39:01 +01:00
Tobi
ec1127dd6a
Merge pull request #571 from TeamNewPipe/soundcloud_client_id
...
[SoundCloud] Update client_id and fix extraction
2021-03-05 14:21:14 +01:00
bopol
ed850d0688
[youtube] improve comments extraction performance
...
- do not parse responseBody twice for continuation
instead try to get commentsTokenInside with the new pattern ("sectionListRenderer")
and try again with the old pattern ("commentSectionRenderer") on failure
- do not unescape responseBody multiple times
-> parse responseBody less times
2021-03-05 13:37:08 +01:00
TobiGr
440a808b8a
Update mocks for channels
2021-03-05 13:13:36 +01:00
golfinq
05213175cd
minor edit, made variable final
2021-03-05 02:49:15 -05:00
golfinq
8bceb57a6c
formatting fix
2021-03-05 02:46:06 -05:00
golfinq
ae48bdea4c
Updated to reflect variable name change
2021-03-05 02:40:36 -05:00
TiA4f8R
b27efdc9a4
Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor
...
Because it's unused with the new request.
Co-authored-by: bopol <bopol@e.email>
2021-03-04 22:50:20 +01:00
TobiGr
2b671b15ce
[YouTube] Use new continuation API for channels
...
Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>
2021-03-04 21:47:16 +01:00
TobiGr
33594d4aff
[SoundCloud] Fix client id extraction
...
SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.
2021-03-04 21:32:34 +01:00
TobiGr
fb2f37d223
[SoundCloud] Update hardcoded client id
2021-03-04 21:25:41 +01:00
XiangRongLin
506cc5fbe9
regenerate mocks
2021-03-04 19:01:58 +01:00
XiangRongLin
03b00ff1d6
Move toJsonArray and toJsonObject to JsonUtils
2021-03-04 18:58:51 +01:00
FireMasterK
668b080b3a
Use dynamic key.
2021-03-04 08:45:14 +01:00
XiangRongLin
43b46bd408
Use new youtube continuations api for playlists
...
Requires sending a POST request instead of GET.
clientName and clientVersion, which were required as headers previously now need to be part of the request payload.
continuation id also needs to be part of request body.
quick and dirty solution.
2021-03-03 19:49:26 +01:00
XiangRongLin
beb05bd05c
Replace test url, because playlist was deleted
2021-03-03 19:46:14 +01:00
XiangRongLin
e13e237392
Fix typo and reword some explanations
2021-02-26 17:48:03 +01:00
TobiGr
a1688fe953
Move BandcampExtractorHelper.getJsonData(String, String) to JsonUtils
2021-02-21 13:51:12 +01:00
TobiGr
70814dcfef
Fix Utils.nonEmptyAndNullJoin
...
When using the index here, it the index needs to be decremented once an element is removed. To cirecumvent this, the native Collections.removeIf() method is used.
2021-02-21 12:51:47 +01:00
TobiGr
b9e8ee8450
Rename BandcampExtractorHelper.smartConcatenate(String[], String) to Utils.nonEmptyAndNullJoin(String, String[])
2021-02-21 12:35:14 +01:00
TobiGr
c07db80ef0
Add BASE_URL and BASE_API_URL to BandcampExtractorHelper
2021-02-21 12:15:45 +01:00
TobiGr
adde4332d1
Code improvements
2021-02-20 23:58:54 +01:00
FireMasterK
42bdd8adc5
Update to fix failing mock tests.
2021-02-20 17:14:41 +05:30
FireMasterK
e8bccfaf5d
Change UA to privacy.resistFingerprinting.
2021-02-20 17:14:41 +05:30
bopol
0c831afa7b
use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified
2021-02-20 11:41:30 +01:00
TobiGr
ea120a4637
remove print stacktrace
...
use interface
2021-02-20 00:23:45 +01:00
XiangRongLin
adf9d7d10f
Add reason field to MockOnly
...
This enforces developers to document why a test is skipped
2021-02-19 18:54:18 +01:00
XiangRongLin
ea52030613
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-02-19 18:54:17 +01:00
TobiGr
02920fafa8
Add isUploaderVerified()
2021-02-19 15:51:23 +01:00
TobiGr
91e9309486
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-02-19 13:37:45 +01:00
TobiGr
fa61b864f2
Code improvements
2021-02-19 13:37:24 +01:00
TobiGr
98268e351c
Move radio URL check into a function
2021-02-19 13:35:44 +01:00
FireMasterK
5333d8a98b
Add more data.
2021-02-19 00:55:03 +05:30
bopol
1a322ad8ed
Add uploader verified by service extraction
2021-02-18 16:36:40 +01:00
FireMasterK
c24afa2cbb
Permalink URL.
2021-02-18 20:16:53 +05:30
FireMasterK
8d54401233
Add fps field as well.
2021-02-18 20:16:53 +05:30
FireMasterK
5b70645d85
Add data to muxed streams aswell.
2021-02-18 20:16:53 +05:30
FireMasterK
e975d33fbe
Change jdoc / comments.
2021-02-18 20:16:53 +05:30
FireMasterK
f31b2a68fd
use getters/setters.
2021-02-18 20:16:53 +05:30
FireMasterK
d4945ac55c
Constructor cleanup.
2021-02-18 20:16:52 +05:30
FireMasterK
3f6a601be8
Fix jdoc / comments.
2021-02-18 20:16:52 +05:30
FireMasterK
a7c9905183
Fix requested changes.
2021-02-18 20:16:52 +05:30
FireMasterK
0c0f2d74bc
Make fields private
...
Co-authored-by: bopol <bopol@e.email>
2021-02-18 20:16:52 +05:30
FireMasterK
d1054338d3
Fix error with livestreams.
2021-02-18 20:16:52 +05:30
FireMasterK
11eb4932f4
Add data to respective classes.
2021-02-18 20:16:52 +05:30
FireMasterK
525e345ed8
remove duplicate field.
2021-02-18 20:16:52 +05:30
FireMasterK
e844d2aed3
Add dash fields to the ItagItem.
2021-02-18 20:16:52 +05:30
bopol
64f6b0478f
Merge pull request #545 from XiangRongLin/comment_reset
...
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-18 15:14:03 +01:00
XiangRongLin
a64579da2f
Unignore some tests that work on Linux
2021-02-17 20:31:26 +01:00
XiangRongLin
b99cb8624f
Unignore PublicBroadcasterTest
...
Tests only work on linux, mocks have to be generated on linux too
2021-02-17 20:31:26 +01:00
XiangRongLin
37366c65ca
Remove encoding of '&' to '&' in test and unignore it
2021-02-17 20:31:26 +01:00
XiangRongLin
c327922cc0
Add explanations to broken tests
2021-02-17 20:31:26 +01:00
XiangRongLin
26d50aca05
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-17 19:21:39 +01:00
TobiGr
54aa5b3042
Use propper structure in KioskExtractors
...
Made BandCampRadioExtractor a Kiosk which holds StreamInfoItems and not InfoItems.
2021-02-16 21:09:42 +01:00
TobiGr
54b8e54f80
Fix potential NPE
2021-02-16 20:59:13 +01:00
TobiGr
16973126a0
Use Collections.emptyList() instead of Collections.EMPTY_LIST or new ArrayList()
2021-02-16 20:25:48 +01:00
bopol
84380e4c3a
Merge pull request #536 from B0pol/watchendpoints
...
support /watch/, /v/ and /w/ subpaths
2021-02-16 20:23:27 +01:00
bopol
41f689b099
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeStreamLinkHandlerFactory.java
...
Co-authored-by: Stypox <stypox@pm.me>
2021-02-16 19:17:10 +01:00
XiangRongLin
a9594f531f
Merge pull request #543 from XiangRongLin/unignore1
...
Unignore and fix some tests
2021-02-16 18:00:16 +01:00
XiangRongLin
a28e0226c5
Merge pull request #519 from XiangRongLin/better_logging_error_handling
...
Better logging, error handling, doc for mock tests
2021-02-16 15:06:06 +01:00
XiangRongLin
05cb22c654
Unignore Gronkh.testDescription
...
Remove special character ü from assertion
2021-02-16 08:59:35 +01:00
XiangRongLin
d9b4c2779f
Unignore YoutubeSearchQHTest.testRegularValues
2021-02-16 08:59:35 +01:00
XiangRongLin
137272b7a4
Unignore YoutubeCommentsExtractorTest.EmptyComment
...
Mock files were missing
2021-02-16 08:59:34 +01:00
TobiGr
5bf9fddba9
Code improvements
2021-02-15 22:45:17 +01:00
TobiGr
e062c8cb0d
Merge branch 'dev' into bandcamp
2021-02-15 22:09:41 +01:00
bopol
7a3d9bdb7d
Description: add .equals() and .hashCode()
2021-02-15 18:51:47 +01:00
bopol
557934cb17
use EMPTY_STRING
2021-02-15 18:51:46 +01:00
bopol
9ca52ca68a
use only one constant UTF-8
2021-02-15 18:47:54 +01:00
XiangRongLin
50e5718bd1
Merge pull request #518 from XiangRongLin/generate_mocks
...
Use and generate mocks for youtube tests
2021-02-15 06:39:41 +01:00
Tobi
d7486f251a
Merge pull request #538 from B0pol/soundcloud-georestriction
...
Soundcloud: fix charts extraction when georestricted
2021-02-14 23:52:27 +01:00
TobiGr
432c68cdec
Update okhttp from 3.12.11 to 3.12.13
...
Changelog https://square.github.io/okhttp/changelog_3x/#version-31213
2021-02-14 23:44:19 +01:00
XiangRongLin
a88b2e6af4
Undo formatting of YoutubeStreamExtractorDefaultTest
2021-02-14 14:57:23 +01:00
bopol
8d89c82caa
Soundcloud: fix charts extraction when georestricted
2021-02-13 22:28:31 +01:00
bopol
ee2012c948
rename folder to subpath
2021-02-13 22:25:46 +01:00
bopol
8db40db4de
use substring instead of split
2021-02-13 22:20:05 +01:00
bopol
81317cf6bc
youtube: support /w/ folder
2021-02-13 22:20:05 +01:00
bopol
f15c0fcfed
support /watch/ and /v/ folders
2021-02-13 22:20:05 +01:00
XiangRongLin
971128c464
Ignore failing tests
2021-02-13 20:28:03 +01:00
XiangRongLin
92dea0806a
Regenerate json files for youtube tests
2021-02-13 20:28:03 +01:00
XiangRongLin
d6b9930c18
Use DownloaderFactory in YoutubeCommentsExtractorTest.Pinned
2021-02-13 20:12:39 +01:00
XiangRongLin
b555f38fa5
Use real downloader for YoutubeSubscriptionExtractorTest
...
It doesn't make network requests anyway
2021-02-13 20:12:39 +01:00
XiangRongLin
f352b0c7af
YoutubeMusicSearchExtractorTest can't be used with mocks
...
Requires further research
2021-02-13 20:00:51 +01:00
XiangRongLin
f45b5610bf
Unignore YoutubeChannelLocalizationTest
...
With mocks time is not a concern anymore
2021-02-13 20:00:51 +01:00
XiangRongLin
19737e06ee
Use DownloaderFactory in YoutubeChannelExtractorTest
...
KurzgesagtAdditional needed to be extracted because it ran a duplicated request.
2021-02-13 20:00:51 +01:00
XiangRongLin
84eb4b30c9
Use DownloaderFactory in YouTube tests
2021-02-13 20:00:51 +01:00
XiangRongLin
b43f3474f5
Make resetClientVersionAndKey public
2021-02-13 20:00:49 +01:00
bopol
2859c7ba5c
add comment explaining why .substring(23)
2021-02-13 12:10:41 +01:00
bopol
73353996d7
fix more tests
2021-02-12 23:18:38 +01:00
bopol
a931e31239
fix tests
2021-02-12 22:36:18 +01:00
TobiGr
d116680ea3
Merge branch 'master' into dev
2021-02-12 22:26:35 +01:00
bopol
b0f356dd76
YT channels: fix continuation
2021-02-12 14:39:09 +01:00
bopol
fcdb50b825
make getInfo from Extractor public
...
In StreamInfo and CommentsInfo
2021-02-09 14:58:43 +01:00
bopol
44c54d403a
fix Invidious channels being accepted by YoutubeStreamLinkHandlerFactory ( #527 )
...
fixes https://github.com/TeamNewPipe/NewPipeExtractor/issues/524 , see it for more details
2021-02-05 18:57:58 +01:00
TiA4f8R
1414a6f178
Fix extraction of YouTube chapters ( #531 )
...
YouTube changed the name of the chapters in the JSON from engagement-panel-macro-markers to engagement-panel-macro-markers-description-chapters, so extracting chapters doesn't work.
2021-02-03 22:07:34 +01:00
Tobias Groza
bfa639950e
Merge pull request #528 from Stypox/ytmusic-0
...
[YouTube Music] Fix extracting search item view/subscriber count when = 0
2021-01-29 23:23:23 +01:00
TobiGr
d728c4fad0
Remove unused imports
...
I searched for "StandardCharsets" to prevent possible bugs with Android KitKat. The import is unused and is therefor removed.
There are other usages of the StandardCharsets class in the test implementation, but those are not relevant.
2021-01-29 23:16:40 +01:00
Stypox
a64dfd7343
[YouTube Music] Fix extracting search item view/subscriber count when = 0
2021-01-29 16:44:24 +01:00
bopol
2662ceca22
rename getHeartedByCreator and getPinned to isPinned and isHeartedByCreator
2021-01-23 13:06:07 +01:00
bopol
66e4eb2f96
extract whether the comment is pinned
2021-01-22 23:59:22 +01:00
bopol
67ddfefdc6
keep invidious instances list in the same order
...
as suggested by @TobiGr
2021-01-22 20:58:14 +01:00
bopol
53d3f7989b
update invidious instances list
2021-01-22 19:30:25 +01:00
XiangRongLin
88e4c8667b
Add comments to RecordingDownloader about usage
2021-01-17 19:53:20 +01:00
XiangRongLin
eecfe09f2c
Check for non-existent folder in MockDownloader
2021-01-17 19:52:58 +01:00
TobiGr
07570554b6
Fix build and optimize imports
2021-01-15 21:55:40 +01:00
TobiGr
78c2113094
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-01-15 21:49:58 +01:00
Stypox
b9ba95614b
Merge pull request #514 from XiangRongLin/test_jvm_property
...
Pass on gradle JVM system property to test JVM
2021-01-15 20:44:23 +01:00
XiangRongLin
a6cfe11ae9
Pass on gradle JVM system property to test JVM
2021-01-15 20:37:14 +01:00
XiangRongLin
6888e7773f
Ignore all failing tests
2021-01-15 20:11:40 +01:00
Stypox
7c774c84ce
Merge pull request #505 from TeamNewPipe/meida.ccc.de_search_fix
...
[media.ccc.de] Fix NPE in search results if they contain a future talk
2021-01-15 14:16:59 +01:00
Stypox
c2c4d97efd
Merge pull request #482 from XiangRongLin/mock_mix_pl_test
...
Mock mix pl tests
2021-01-15 12:54:48 +01:00
XiangRongLin
4f81d9d689
Regenerate mock files
2021-01-15 09:51:23 +01:00
Stypox
951159f75f
Merge pull request #504 from TeamNewPipe/media.ccc.de_recent_order
...
[media.ccc.de] Recent kiosk: order streams by upload date
2021-01-14 23:31:09 +01:00
Tobias Groza
cc51c5f6a3
Merge pull request #507 from vkay94/add-interval-to-stream-frames
...
Add durationPerFrame to Frameset
2021-01-14 20:27:40 +01:00
XiangRongLin
6ca7123d5d
Add comment for usage of YoutubeParsingHelper.resetClientVersionAndKey
2021-01-14 20:01:52 +01:00
vkay94
11dcfe638b
Add durationPerFrame to Frameset and getFrameBoundsAt method
2021-01-14 20:01:06 +01:00
Tobias Groza
c2ff6723d7
Merge pull request #511 from B0pol/errors
...
Fix PeerTube tests by changing instance
2021-01-14 19:15:22 +01:00
Tobias Groza
6773ec8699
Merge pull request #510 from B0pol/heartedbycreator
...
comments: add heartedByUploader and extract it for YouTube
2021-01-12 19:00:04 +01:00
TobiGr
0c4836c800
[media.ccc.de] Recent kiosk: Add comment clarifying "length" and "duration" fields have the same value
2021-01-12 18:48:42 +01:00
XiangRongLin
4af50c6870
Use @link for downloader
2021-01-12 09:41:57 +01:00
XiangRongLin
3455f0f23c
Add documentation to testing with mocks
2021-01-12 09:37:48 +01:00
bopol
5292cc1636
[peertube] test: remove usage of peertube.mastodon.host
...
It's down for months
2021-01-11 19:46:27 +01:00
bopol
f3c22da618
comments: add heartedByCreator and extract it for youtube
2021-01-11 18:48:08 +01:00
XiangRongLin
63c237db41
Only create directories if they do not exist
2021-01-11 07:57:11 +01:00
XiangRongLin
e82cb1efea
Remove dependency to commons-io
2021-01-10 20:50:38 +01:00
XiangRongLin
1ea6c6ce54
Prefix mock file from RecordingDownloader with "generated_mock_"
...
Only read those files in MockDownloader
2021-01-10 20:46:42 +01:00
XiangRongLin
f91916c017
Remove unnecessary file object creation
2021-01-10 20:39:22 +01:00
XiangRongLin
255c726f20
Add resource path base to DownloaderFactory
2021-01-10 20:27:34 +01:00
XiangRongLin
35e299759e
Add method to allow resetting youtube client version and key
...
This is needed so that a request is made for each test class when running multiple at once. This way RecordingDownloader records all necessary requests.
This works as long as tests are run sequentially and not in parallel.
2021-01-10 20:24:50 +01:00
XiangRongLin
f447a7a450
fix import due to downloader package move
2021-01-10 19:49:14 +01:00
XiangRongLin
f8aa989c42
Add generated json and manually copy client_version.json
2021-01-10 19:38:47 +01:00
XiangRongLin
285c26eafa
Adjust YoutubeMixPlaylistExtractorTest to use DownloaderFactory
2021-01-10 19:38:41 +01:00
XiangRongLin
e6e8e39def
Add DownloaderFactory to return a specific downloader based on 2 variables.
...
If the system property 'downloader' is set that use that specific downloader. This is used from gradle by appending `-Ddownloader=ABCD to the command.
ABCD is one of DownloaderType.
The other variable is the static property `DEFAULT_DOWNLOADER` in DownloaderFactory, which can be easily changed as needed inside the IDE according to development needs`.
Normal workflow would be to first use the recording downloader and afterwards only use mocks, if the requests are always staying the same.
2021-01-10 19:36:24 +01:00
XiangRongLin
7c40fb8bf7
Add additional downloader implementations
...
RecordingDownloader relies on the real downloader and saves the request/response pair into a json file.
MockDownloader uses json files from above and mocks responses for specific requests.
2021-01-10 19:36:24 +01:00
XiangRongLin
1bcb9c76a7
Generate equals and hashCode for Request
2021-01-10 19:36:24 +01:00
XiangRongLin
4dad3d60d6
Move DownloaderTestImpl into downloader package
2021-01-10 19:36:21 +01:00
XiangRongLin
fe3902c669
Un-comment out 360p webm
2021-01-08 19:43:03 +01:00
TobiGr
2d93b23723
[media.ccc.de] Show conference title in search results instead of slug
2021-01-08 14:15:25 +01:00
TobiGr
52f3bd15f2
[media.ccc.de] Add conference title to "recent" kiosk
...
The "conference_title" field was added upstream recently (see https://github.com/voc/voctoweb/issues/495 )
2021-01-08 12:04:39 +01:00
Tobias Groza
ab62464e86
Merge pull request #495 from B0pol/sepiasearch
...
[PeerTube] Implement sepia search
2021-01-06 11:06:36 +01:00
bopol
0bcea9c2cd
add JDoc
2021-01-01 14:37:32 +01:00
bopol
0e45c25ea9
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamInfoItemExtractor.java
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2021-01-01 14:20:47 +01:00
TobiGr
fae67fbd45
Merge remote-tracking branch 'origin/master' into dev
2021-01-01 13:06:11 +01:00
TobiGr
d63d20f943
Fix JDoc
2020-12-31 18:05:40 +01:00
TobiGr
2cbc3ccb1a
[media.ccc.de] Fix live stream test if there are no current live streams
2020-12-31 17:03:23 +01:00
TobiGr
a896ec41b9
[media.ccc.de] Use Pattern for getting the ive stream id
2020-12-31 17:03:23 +01:00
TobiGr
5dc9a76e3c
[media.ccc.de] Recent kiosk: order streams by upload date
2020-12-30 18:38:09 +01:00
TobiGr
edf8dd0e92
[media.ccc.de] Fix NPE in search results if they contain a future talk
2020-12-30 18:34:28 +01:00
Tobias Groza
3c8c8e7307
Merge pull request #497 from TiA4f8R/fix-player-js-url-redirect
...
Fix YouTube player JavaScript URL redirection
2020-12-27 18:04:40 +01:00
TiA4f8R
d30caeb091
Fix YouTube player JavaScript URL redirection
2020-12-27 15:55:03 +01:00
TobiGr
deb9af7bf5
[media.ccc.de] live kiosk - display info of the current talk
2020-12-27 15:52:09 +01:00
TobiGr
b4e1913971
[media.ccc.de] Play live streams
2020-12-27 13:25:48 +01:00
TobiGr
80f4d42226
[Draft] Add live extractor
2020-12-27 11:01:22 +01:00
TobiGr
ed9402c002
[media.ccc.de] Escape "." in id pattern
2020-12-27 11:01:10 +01:00
TobiGr
674f2227f3
make url pattern static
2020-12-27 01:27:37 +01:00
TobiGr
090dade5b3
[media.ccc.de] Add short test for recent kiosk
2020-12-27 01:03:08 +01:00
TobiGr
0cfefe222a
Revert "[media.ccc.de] Get conference name from API URL"
...
This reverts commit d1a0686d2f
.
2020-12-26 16:55:44 +01:00
TobiGr
d1a0686d2f
[media.ccc.de] Get conference name from API URL
2020-12-26 13:15:03 +01:00
TobiGr
41699798fe
[media.ccc.de] Use web URL for uploaderUrl instead of API URL in recent
2020-12-26 12:47:32 +01:00
TobiGr
b2154c6e87
[media.ccc.de] Fix conferences kiosk test
2020-12-25 23:55:45 +01:00
TobiGr
ed6ae329b3
[media.ccc.de] Add "recent" kiosk
2020-12-25 22:26:58 +01:00
Tobias Groza
b5e50cc9fb
Merge pull request #484 from TeamNewPipe/yt_,music_search
...
Fix YouTube Music search
2020-12-25 18:30:25 +01:00
TobiGr
d9e2da53c3
Replace explicit string checks whether a playlist íd is a certain YouTube Mix type with calling the dedicated methods
2020-12-25 15:00:34 +01:00
TobiGr
abee0a8df1
Fix typos in YoutubePlaylistLinkHandlerFactory
2020-12-24 10:34:21 +01:00
TobiGr
3033c0b993
[YouTube] Music Mix: Fix playlist not being accepted
...
Regression introduced by YouTube Mix support (#280 )
2020-12-23 21:08:36 +01:00
TobiGr
b62144b49d
[YouTube] Music: Fix search extraction caused by changed JSON response
2020-12-23 21:08:36 +01:00
TobiGr
c9d9bd1e24
Fix typo in ListLinkHandlerFactory#getUrl
2020-12-23 21:05:58 +01:00
bopol
3ae924a7f1
Implement sepia search
2020-12-23 15:07:55 +01:00
TobiGr
942d840624
Fix typo in AffiliateService: rename "NO_AFILIATE" to "NO_AFFILIATE"
2020-12-23 12:50:45 +01:00
TobiGr
865c42e273
Fix line break
2020-12-23 12:49:50 +01:00
Fynn Godau
c9e9953bb0
[Bandcamp] Fix channel link handler factory
2020-12-22 20:02:53 +01:00
TobiGr
41a8ed625d
Make some vars final and add annotations to methods
2020-12-22 18:19:26 +01:00
TobiGr
bc6de14952
Extract stream and search meta info for YouTube
...
Add method to extract Google webcache URLs.
2020-12-22 18:19:26 +01:00
TobiGr
50903730b1
[Bandcamp] Fix accepting HTTP URLs
2020-12-22 08:53:10 +01:00
TobiGr
838dc1ab54
Fix vulnerability in junit
2020-12-21 23:22:16 +01:00
bopol
8d3436565f
[peertube] fix related streams api url when no tags
...
fixes #489
2020-12-19 23:25:37 +01:00
Tobias Groza
22a415156f
Merge pull request #479 from vkay94/stream-segments
...
Extract stream segments for YouTube
2020-12-19 11:12:22 +01:00
TobiGr
74b46fed2d
[Bandcamp] Fix deprecation in parseDate helper function
2020-12-19 10:40:21 +01:00
TobiGr
116e921d6c
Merge remote-tracking branch 'origin/dev' into bandcamp
2020-12-19 00:26:11 +01:00
TobiGr
df16a8646f
[Bandcamp] Improve radio stream extractor test
2020-12-18 23:15:59 +01:00
XiangRongLin
cdcb66b93e
Clean up stuff from mix pl
...
Coming from unclean merge and missing hints from IDE
2020-12-16 08:35:28 +01:00
XiangRongLin
ba8782a9ed
Use lowercase string for extracting cookies
...
The map in a real response happens to ignore upper-/lowercase differences. Other maps used in unit test may not have that behaviour.
2020-12-16 08:32:04 +01:00
Stypox
85fa006214
Merge pull request #280 from XiangRongLin/mixPL
...
Extractor for youtube mix (auto-generated playlist)
2020-12-14 18:11:07 +01:00
XiangRongLin
f90f6fcf92
[YouTube] Don't escape & in getUrlFromNavigationEndpoint for playlists
2020-12-12 20:40:13 +01:00
Xiang Rong Lin
a338e4e08e
[Youtube] Apply review suggestions and avoid channel mix edge case
2020-12-12 20:32:43 +01:00
Xiang Rong Lin
22d2f7e400
[Youtube] Add cookies to youtube mix request
...
This way youtube wont return duplicates when getting more items of the mix (but youtube can also track us)
2020-12-12 20:32:40 +01:00
Stypox
421935401f
[Youtube] Add subchannel functions to mix and fix imports
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
3ff8619bcc
[Youtube] apply wb9688 suggestion (mix)
...
Channel mix adjusments and test
Don't accept youtube music mix urls as playlist
Don't override playlistData to keep getInitialPage()
Remove json constants
Indentation
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
822cf307f7
[Youtube] Add _ITEMS constants and improve code style
...
Move thumbnail id exctraction code to getThumbnailUrlFromId
Add test for "My mix" detection to service tests
Use ITEM_COUNT_UNKNOWN everywhere instead of -1 and add some tests
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
df38b1926c
[Youtube] Add tests and take thumbnail image always from first video of mix
...
Also fix getThumbnailUrl for "My Mix"
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
68a3948af6
[Youtube] Fix get banner url
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
ec6b99c082
[Youtube] Adjust mix extractor to new user agent
...
Also extract continuation now
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
0ff054acb4
[Youtube] Extract initial playlist info
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
a376792a5d
[Youtube] Handle case where url is in "youtube.com/playlist?list=listID" format.
...
This occurs when sharing a mix from the official youtube app.
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
327a5730a8
[Youtube] Add some comments to mix
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
d74265c846
[Youtube] Extract getThumbnailUrl into method and change getUploaderName
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
0efb854d27
[Youtube] Implement mix extractor for auto-generated playlists.
...
-New YoutubeMixPlaylistExtractor, that extracts from a mix (auto-generated playlist).
-The url has the format of "youtube.com/watch?v=videoID&playlistID",
where playlistID always starts with "RD" and usually followed by the videoID.
-Change YoutubePlaylistLinkHandlerFactory to create a linkhandler with the given url if it is a mix.
-Change YoutubeService to return YoutubeMixPlaylistExtractor if the url is a mix.
2020-12-12 20:30:57 +01:00
vkay94
2ba27b39af
Stream segments: Remove replaceAll from setPreviewUrl
2020-12-12 15:00:45 +01:00
vkay94
9a7a224a54
Add tests for stream segments
2020-12-12 10:24:55 +01:00
vkay94
d3eea4f4be
Add stream segments to StreamInfo
2020-12-12 10:24:29 +01:00
Isira Seneviratne
57be1f1123
Use Objects.requireNonNull().
2020-12-11 06:09:57 +05:30
Isira Seneviratne
4c19a88612
Use Objects.toString().
2020-12-11 06:09:50 +05:30
Isira Seneviratne
b8f64595a2
Use Objects' static equals() and hashCode() methods.
2020-12-11 06:09:44 +05:30
vkay94
9dbacbc618
Fix comments parsing
2020-12-09 14:57:38 +01:00
Fynn Godau
04dd3d4d32
Rework link handlers to correctly accept external websites
2020-12-05 15:08:26 +01:00
TheAssassin
ba3e2302bc
Fix name of media.ccc.de service
2020-12-01 23:54:27 +01:00
Tobias Groza
b7a995187f
Merge pull request #463 from opusforlife2/update_invidious_URLs
...
Update Invidious URL List
2020-11-28 13:30:32 +01:00
Fynn Godau
be562b8436
Change tests
2020-11-27 16:48:40 +01:00
TobiGr
c91e21b37c
[Bandcamp] Tests - Add finals and improve code formatting
2020-11-24 14:13:31 +01:00
TobiGr
99e7ef013e
[Bandcamp] Apply small changes to code formatting and style
...
Make variables final when possible
Remove unused imports
Improve code formatting
2020-11-24 14:13:31 +01:00
fynngodau
8c369b0f79
Rephrase link in javadoc
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-11-24 14:01:31 +01:00
Tobias Groza
650f0920fe
Merge pull request #465 from XiangRongLin/playlist_continuation
...
[YouTube] Fix playlist continuations extraction
2020-11-20 20:48:20 +01:00
TobiGr
334e1e9b53
Fix YouTube comments test
2020-11-20 19:53:53 +01:00
Fynn Godau
186936d041
Various changes regarding tests
2020-11-19 22:33:52 +01:00
Fynn Godau
6bc7e3420e
Merge TNP/dev into fynngodau/dev
2020-11-19 21:32:08 +01:00
XiangRongLin
5bceff0083
[YouTube] Fix extraction of next page url for the last page of playlist
2020-11-18 19:03:12 +01:00
XiangRongLin
8347e14952
[YouTube] Fix playlist continuations extraction
2020-11-17 21:04:53 +01:00
opusforlife2
2174055ef8
Update Invidious URL list in Parsing Helper
2020-11-11 15:12:31 +00:00
opusforlife2
c0afd5213a
Update Invidious URL list in Link Handler Factory
2020-11-11 14:54:16 +00:00
Stypox
89a77ae74a
[YouTube] Fix detection of ended livestreams and parse livestream upload date
2020-11-04 16:03:08 +01:00
Stypox
827f7bd137
[YouTube] Cache deobfuscation and improve requests made
...
Fix age restriction extraction
Automatically fixes more things
2020-11-04 16:02:12 +01:00
Isira Seneviratne
4fe28d7e3a
Fix YouTube parse error when only a date is present.
2020-11-04 05:49:42 +05:30
Isira Seneviratne
9cf9e7e980
Call existing constructor in DateWrapper.
2020-11-04 05:49:41 +05:30
Isira Seneviratne
fe31a90cb3
Remove DateTimeFormatter.ISO_OFFSET_DATE_TIME usage.
2020-11-04 05:49:41 +05:30
Tobias Groza
b13c7e1c1e
Merge pull request #452 from Stypox/yt-import
...
Implement YouTube subscription import from Google takeout
2020-11-03 20:32:17 +01:00
bopol
345e136f6c
create YouTubeCommentsLinkHandlerFactoryTest
...
and remove invidious test from YouTubeCommentsExtractorTest, because it was just testing if the URL is accepted, then the extractor does the same thing, we don't need to test the same thing twice
2020-11-03 19:10:10 +01:00
Stypox
501ec30152
Implement youtube subscription import from Google takeout
2020-11-02 11:04:48 +01:00
Bri@n
82746d172f
Fix typo in DonationLinkHelper and rewrote swtich statement
2020-11-01 17:34:34 -05:00
Stypox
2f02c0e6a4
Merge pull request #450 from B0pol/fileutils
...
create FileUtils
2020-11-01 18:13:57 +01:00
bopol
f69b3ef05d
create FileUtils
2020-11-01 17:15:34 +01:00
TobiGr
be9e160333
Fix build
2020-11-01 17:04:02 +01:00
Isira Seneviratne
4f04cfccca
Switch from Calendar to OffsetDateTime in DateWrapper.
2020-11-01 06:28:54 +05:30
Isira Seneviratne
ee3af63c04
Switch to ChronoUnit.
2020-11-01 06:28:53 +05:30
Stypox
6cc50b57e3
Merge pull request #410 from Scrxtchy/sets-track-title
...
Fix parsing Soundcloud tracks that contain the term 'sets'
2020-10-31 21:43:21 +01:00
Stypox
b242e1d113
Merge pull request #362 from Stypox/google-search-urls
...
Add support for Google search redirect url
2020-10-31 21:03:48 +01:00
Stypox
9e53cf0b56
Fix parameter reassignment and other style issues
...
Also remove left-behind debug statement
2020-10-31 20:58:28 +01:00
Stypox
3fe55b30ba
Add support for Google search redirect url
2020-10-31 20:58:28 +01:00
Stypox
c190a3029b
Consider protocol as base url when it is a custom one (e.g. vnd.youtube)
2020-10-31 20:58:28 +01:00
bopol
b21e59925d
[PeerTube] fix account and channel extractors
2020-10-29 19:52:29 +01:00
Stypox
30ed4f2d63
Remove any reference to decrypt and improve error message
2020-10-28 20:26:00 +01:00
TobiGr
62912ee834
Merge branch 'master' into dev
2020-10-28 14:28:06 +01:00
Scratch
947ce3ee10
Fix parsing Soundcloud tracks that contain the term 'sets'
2020-10-28 07:55:02 +11:00
bopol
6dc5ab4015
find playerUrl in another place when assetsPattern regex fails
2020-10-27 13:48:58 +01:00
bopol
db0ef83d6b
fix youtube decryption and three attemps bug
...
fixes teamnewpipe/newpipe#4572 fixes #439
2020-10-27 13:40:24 +01:00
bopol
c1e9857960
fix subscriber count when subscribe is disabled
...
fixes #305
2020-10-26 19:06:01 +01:00
bopol
a39a2cca82
fix redirect channels
2020-10-26 19:05:19 +01:00
bopol
0a12300c5e
polish tests
2020-10-26 16:57:37 +01:00
bopol
01f49e8f66
polish strings
2020-10-26 16:32:39 +01:00
Tobias Groza
14c0c37d64
Merge pull request #309 from Stypox/improve-stream-tests
...
Improve stream extractor tests & various fixes
2020-10-25 18:49:53 +01:00
Kiril Isakov
366ed71523
[YouTube] Ignore leading characters in video id
2020-10-25 14:34:20 +01:00
Stypox
8026304a0a
Merge pull request #409 from Bartoshr/bugfix/short-channel-urls
...
Support short custom YouTube channel URLs
2020-10-25 11:43:50 +01:00
Stypox
57e7994c9e
Add some missing finals, nullables and comments
2020-10-25 08:12:41 +01:00
Stypox
f11fe87688
[YouTube] Replace outdated PewDiePie video test with another one
...
The old video was made private, and this video (wedding) is probably never going to be removed.
2020-10-24 18:41:41 +02:00
Stypox
3c55ea9321
[PeerTube] Change age restricted video in tests
...
The old one wasn't available anymore
2020-10-24 18:41:41 +02:00
Stypox
880ff04a5c
[Test] Add stream dash mpd url test
2020-10-24 18:41:41 +02:00
Stypox
d0b14644bb
[YouTube/MediaCCC] Consider dates as GMT and not as local
2020-10-24 18:41:41 +02:00
Stypox
8ce711f40f
[Test] Add sub channel name, url and thumbnail tests
2020-10-24 18:41:41 +02:00
Stypox
6127826571
[Test] Add stream metadata tests
2020-10-24 18:41:41 +02:00
Stypox
a087b092b4
[Test] Improve code style and add final
2020-10-24 18:39:55 +02:00
Stypox
a4097d8d01
[MediaCCC] Return empty list of video-only streams instead of null
2020-10-24 18:39:55 +02:00
Stypox
8dc3f28618
[PeerTube] Test one channel url with api and one without
2020-10-24 18:39:55 +02:00
Stypox
68d23defba
[YouTube] Do not catch every exception on getErrorMessage
...
@B0pol suggestion
2020-10-24 18:39:55 +02:00
Stypox
3191bd6c70
[YouTube] Use final when possible
2020-10-24 18:39:55 +02:00
Stypox
55bc01d1ce
[SoundCloud] Use final when possible, ide refactorings
2020-10-24 18:39:55 +02:00
Stypox
06430c4749
[PeerTube] Use final when possible, ide refactorings
2020-10-24 18:39:06 +02:00
Stypox
fcb9b6f855
[MediaCCC] Use final when possible, ide refactorings
...
Refactorings on `throws` clause
2020-10-24 18:36:22 +02:00
Stypox
af5b8b1915
[MediaCCC] Add tests for stream and conference link handlers
2020-10-24 18:36:21 +02:00
Stypox
07a90d116a
[MediaCCC] Use regex to parse stream and conference urls
2020-10-24 18:36:21 +02:00
Stypox
0c980b2d64
[PeerTube] Improve channel and stream link handler tests
2020-10-24 18:36:21 +02:00
Stypox
d130fd79c3
[PeerTube] Prepend "accounts/" to channel id for backward compatibility
2020-10-24 18:36:21 +02:00
Stypox
492db83ccf
[MediaCCC] Return null instead of empty items collector
...
As per the documentation in the base getRelatedStreams()
2020-10-24 18:34:57 +02:00
Stypox
3b2a1829d4
[MediaCCC] Extract tags
2020-10-24 18:34:57 +02:00
Stypox
b461da792f
[MediaCCC] Fix link handler inconsistency providing API links
2020-10-24 18:34:34 +02:00
Stypox
aeeae87641
[PeerTube] Parse timestamp from url (previously unimplemented)
2020-10-24 18:33:42 +02:00
Stypox
7ae3cb6d07
[PeerTube] Fix link handler inconsistency providing API links
2020-10-24 18:33:11 +02:00
Stypox
4349be13af
[PeerTube] Return empty audio stream list instead of null
2020-10-24 18:25:56 +02:00
Stypox
3b2cfb4ca2
[SoundCloud] Return empty video stream list instead of null
...
Also replace every instance of `return new ArrayList<>();` with `return Collections.emptyList();`
2020-10-24 18:23:46 +02:00
Stypox
072bae321f
[YouTube] Fix frame extraction for livestreams
...
Use saved playerResponse instead of parsing json every time
2020-10-24 18:23:46 +02:00
Stypox
7cd410f3fc
[YouTube] Return 0 when there is no timestamp, not -2, as per javadoc
2020-10-24 18:23:45 +02:00
Stypox
7fb867c166
[YouTube] Fix error message obtaining when there is none
2020-10-24 18:22:44 +02:00
Stypox
1d7a86e664
[Test] Add base classes for stream extractor tests
...
Refactor all stream extractor tests to use new base class.
Remove check if upload date is in the past: this does not have to hold true: youtube premieres turn up in search results even though they are in the future
2020-10-24 18:22:44 +02:00
Bartosz Rumiński
29695aed0a
Small field refactor
2020-10-23 16:42:30 +02:00
Bartosz Rumiński
0e67d820bc
Use static regex pattern for excluded path segments
2020-10-22 20:13:29 +02:00
Bartosz Rumiński
d3f80d1538
Exlude links which are not channels
2020-10-20 20:06:06 +02:00
Stypox
6887d59570
[YouTube] Handle urls for Shorts
2020-10-18 12:03:01 +02:00
Scratch
e945f711c1
Fix SoundCloud test artist account name ( #416 )
2020-10-16 23:42:46 +02:00
wb9688
be9a6f931c
Fix parsing new ytInitialData
2020-10-16 20:27:40 +02:00
Tobias Groza
1a9bc8caf1
Merge pull request #414 from Scrxtchy/soundcloud-slash
...
Remove tailing slash from SoundCloud URLs
2020-10-16 09:48:44 +02:00
Scratch
6a70cb9d50
Remove tailing slash from SoundCloud URLs
...
Fixes #412
2020-10-16 17:17:40 +11:00
Tobias Groza
9a785f9a7e
Merge pull request #413 from Scrxtchy/soundcloud-test-update
...
Fix SoundCloud test artist account name
2020-10-16 08:05:44 +02:00
Scratch
538f5d3973
Fix SoundCloud test artist account name
2020-10-16 13:45:05 +11:00
Stypox
19e862657a
[YouTube] Fix some decryption exceptions by retrying
2020-10-15 20:16:44 +02:00
Bartosz Rumiński
e3f996e014
Exlude links which are not channels
2020-10-12 20:59:56 +02:00
Bartosz Rumiński
5ab1b053d2
Update youtube channel link handler tests
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 20:11:28 +02:00
Bartosz Rumiński
7abb4b3713
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeChannelLinkHandlerFactory.java
...
Fix typos
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 19:57:45 +02:00
Bartosz Rumiński
9d63211a66
Fix typos
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 19:56:53 +02:00
Bartosz Rumiński
8c38a5509e
Prevent attribution_link urls to be accepted by channel links handler
2020-10-09 19:07:38 +02:00
Bartosz Rumiński
4e04991762
Support short custom youtube channel urls
2020-10-09 17:06:59 +02:00
Fynn Godau
81b5e7cf3d
Fix extractor
2020-10-08 17:56:03 +02:00
wb9688
ebbfe7f6d4
Skip YouTube's OTF streams
2020-09-29 10:48:02 +02:00
Tobias Groza
6633f26ec5
Merge pull request #386 from mhmdanas/remove-subtitles-stream-get-url
...
Remove SubtitlesStream#getURL()
2020-08-26 21:32:36 +02:00
wb9688
7657c2ed1a
Use initSafeStandardsObjects()
2020-08-15 17:08:07 +02:00
mhmdanas
1a63dcb355
Remove overriden field url
2020-08-12 22:37:36 +03:00
mhmdanas
f0f1c009b2
Remove SubtitlesStream#getURL()
2020-08-10 19:28:56 +03:00
Fynn Godau
cfe88a74c1
Throw ContentNotSupportedException when opening radio uploader channel
2020-08-09 10:07:49 +02:00
TobiGr
0fb73301e3
[YouTube] Fix crash on empty comment
...
Closes #380
2020-08-05 18:25:40 +02:00
Fynn Godau
932d094d6a
Upgrade featured playlist urls to https
2020-08-02 16:55:18 +02:00
Fynn Godau
c12ef3a02d
Merge TNP/dev into fynngodau/dev
2020-08-02 16:43:04 +02:00
TobiGr
5ac80624a4
Fix detection of YouTube's decryption function
...
Closes TeamNewPipe/NewPipe#3951
2020-07-28 01:01:10 +02:00
Tobias Groza
32dff1541f
Merge pull request #370 from wb9688/fix-yt-age-restricted
...
Fix age restricted YouTube videos
2020-07-26 14:17:19 +02:00
TobiGr
ec778200d9
Add comments to the code
2020-07-26 14:15:13 +02:00
wb9688
667dce034c
Only use fallback methods for clientVersion when clientVersion hasn't been successfully extracted yet
2020-07-26 13:14:25 +02:00
wb9688
f345f667e2
Extract YouTube's key
2020-07-26 12:01:59 +02:00
wb9688
4c987a5302
Support YouTube's new continuations for search
2020-07-26 10:01:03 +02:00
wb9688
81459e289f
Fix age restricted YouTube videos
2020-07-18 19:52:54 +02:00
wb9688
fc3a63fec5
Update Javadoc for getRelatedStreams()
2020-07-15 18:48:37 +02:00
wb9688
ff560e907a
Remove getNextStream() from StreamExtractor
2020-07-15 18:48:37 +02:00
Tobias Groza
a70cb0283f
Merge pull request #314 from wb9688/remove-getnextpageurl
...
Next page stuff
2020-07-07 20:49:38 +02:00
wb9688
0a5a905bc7
Add final at more places
2020-07-07 20:45:47 +02:00
wb9688
17ba8a57fa
Clean up the code
2020-07-06 20:19:31 +02:00
wb9688
9b6fe1dea6
Throw IllegalArgumentException when Page is invalid
2020-07-06 20:19:31 +02:00
wb9688
4cc312086a
Introduce Page class
2020-07-06 20:19:31 +02:00
wb9688
e3bfdba135
Remove getNextPageUrl() function from ListExtractor
2020-07-06 20:11:40 +02:00
wb9688
156a26b64b
Support start= timestamps for YouTube
2020-07-06 14:08:40 +02:00
bopol
261471e7f4
remove invidiou.sh from tests
...
dead instance
2020-07-02 21:33:45 +02:00
bopol
eb3901acee
update invidious instances
2020-07-02 21:31:05 +02:00
Tobias Groza
c0ceb5cb27
Merge pull request #340 from wb9688/peertube-deleted-comments
...
Handle isDeleted for PeerTube comments
2020-06-28 10:21:37 +02:00
Tobias Groza
339f1d9c43
Merge pull request #349 from wb9688/okhttp
...
Use OkHttp for tests like in NewPipe
2020-06-27 20:08:02 +02:00
Tobias Groza
970bc7f69d
Merge pull request #359 from mauriciocolli/fix-client-id-bug
...
[SoundCloud] Fix concurrency issue when getting the client id
2020-06-27 17:36:24 +02:00
Robin
576754982e
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-06-27 11:35:01 +02:00
Mauricio Colli
7505e828cc
[SoundCloud] Update client id
2020-06-24 01:50:09 -03:00
Mauricio Colli
d2f1c0f40d
[SoundCloud] Fix concurrency issue when getting the client id
2020-06-24 01:50:08 -03:00
Robin
b63ae93495
Added a comment
...
and fixed a "typo" in the method parseDurationString replacing Long with Integer.
These are not the droids you are looking for wb9688 😁
2020-06-15 11:27:44 +02:00
Tobias Groza
92f6754f0f
Merge pull request #338 from wb9688/parse-duration-string-test
...
Add tests for parseDurationString()
2020-06-15 11:01:21 +02:00
Stypox
5808439c3f
[PeerTube] Change age restricted video in tests
...
The old one wasn't available anymore
2020-06-14 19:57:39 +02:00
wb9688
3239aa84f2
Handle isDeleted for PeerTube comments
2020-06-14 19:27:20 +02:00
wb9688
a65e46e4b1
Use OkHttp for tests like in NewPipe
2020-06-11 15:06:08 +02:00
Fynn Godau
96de834b67
Bandcamp: parse date from radio info item
2020-06-04 19:36:58 +02:00
Fynn Godau
39b55b5b42
Bandcamp radio stream extractor: test uploader url
2020-06-04 19:32:19 +02:00
Fynn Godau
9a555d97e5
Remove useless bandcamp tests
2020-06-04 19:31:01 +02:00
Fynn Godau
e13f341a45
Bandcamp stream extractor: test service id
2020-06-04 19:29:37 +02:00
Fynn Godau
4dd9540782
Bandcamp license switch order
2020-06-04 19:27:39 +02:00
Fynn Godau
52103ac61f
Remove "useless" comments
2020-06-04 19:27:13 +02:00
Fynn Godau
8fa81537c0
Better bandcamp stream extractor description test
2020-06-04 19:17:51 +02:00
Fynn Godau
0950a95577
Fix for testing wrong object for null or empty
2020-06-04 19:16:08 +02:00
Fynn Godau
e6ecd91929
Bandcamp: Test full date
2020-06-04 19:12:10 +02:00
Fynn Godau
27e7aad159
Bandcamp: Improve code style
2020-06-04 19:04:25 +02:00
fynngodau
ea49202f64
Don't display internal license id
...
Co-authored-by: bopol <bopol@e.email>
2020-06-04 18:26:58 +02:00
bopol
063f4be766
For #342 : support PeerTube embeds
2020-06-04 14:26:42 +02:00
Fynn Godau
34b6928124
Don't print stacktrace before throwing new exception
2020-06-04 14:25:38 +02:00
Fynn Godau
1be20ceeec
Bandcamp: parse date
2020-06-04 14:09:21 +02:00
Fynn Godau
6822fe3dc8
Bandcamp: Get Stream upload date
2020-06-04 13:03:16 +02:00
Fynn Godau
e98bff8ae9
Improved Bandcamp radio stream extractor tests
2020-06-04 12:30:53 +02:00
Fynn Godau
8f6c00f8d8
Use default test in BandcampSearchExtractorTest
2020-06-04 12:12:20 +02:00
Fynn Godau
de776561b0
Bandcamp: fix loading uploader from streams in search
2020-06-04 12:02:18 +02:00
Fynn Godau
3940138fc5
Fix Bandcamp capitalization in one more spot
2020-06-03 22:41:27 +02:00
Fynn Godau
9fa9d920a9
Bandcamp: Implement new methods required due to interface changes
2020-06-03 21:35:11 +02:00
Fynn Godau
dd955c7f0c
Merge branch 'origin/dev' into dev
2020-06-03 21:13:40 +02:00
wb9688
33b1121fc7
Add tests for parseDurationString()
2020-05-30 17:20:54 +02:00
wb9688
e39147202c
Fix YouTube video duration parsing
2020-05-30 09:31:08 +02:00
wb9688
98055a3c3c
Revert "Add more decryption function regexes"
...
This reverts commit 384d6acd5d
.
2020-05-29 18:41:40 +02:00
Tobias Groza
bdb0f2ae6b
Merge pull request #332 from wb9688/learning-playlist
...
Support YouTube's learning playlists
2020-05-29 10:11:08 +02:00
Tobias Groza
cf18cdb2f5
Merge pull request #329 from wb9688/decryption-regexes
...
Add more decryption function regexes
2020-05-29 10:10:50 +02:00
wb9688
ab77961b0a
Extract trailer thumbnail in YT learning playlists
2020-05-29 09:55:24 +02:00
wb9688
4d683e7655
Support YouTube's learning playlists
2020-05-29 09:17:03 +02:00
Tobias Groza
0b4977bb0c
Merge pull request #333 from wb9688/playerresponse-dashmanifesturl
...
Extract dashManifestUrl in playerResponse
2020-05-28 21:17:07 +02:00
wb9688
384d6acd5d
Add more decryption function regexes
2020-05-28 11:40:38 +02:00
wb9688
8f71bde776
Fix bug for YT videos longer than 1000h
2020-05-28 11:34:42 +02:00
wb9688
aa277e1b6a
Upgrade other dependencies
2020-05-28 11:34:42 +02:00
wb9688
036cc7e26e
Upgrade jsoup
2020-05-28 11:34:41 +02:00
Fynn Godau
89b0639faa
New Bandcamp tests for channel extractor
2020-05-25 20:22:23 +02:00
Fynn Godau
e08256ebc6
Remove "url cleanup" in Bandcamp link handlers
2020-05-25 19:17:27 +02:00
Fynn Godau
692b2d06f4
Bandcamp: read info item data in place and not in advance
2020-05-25 18:53:15 +02:00
Fynn Godau
fff21caa87
Fix Bandcamp no avatar test
...
I own the bandcamp artist account called "NewPipeExtractorTest", so it
shouldn't break again.
2020-05-25 18:53:10 +02:00
wb9688
7ef3e80493
Extract dashManifestUrl in playerResponse
2020-05-15 12:07:26 +02:00
wb9688
efafbbb88a
Merge pull request #331 from wb9688/playlist-subchannel
...
Add sub-channel stuff to PlaylistInfo
2020-05-12 10:49:13 +02:00
wb9688
baf5dd3e8f
Merge pull request #311 from B0pol/fixSearchSuggestion
...
Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSea…
2020-05-12 09:29:15 +02:00
wb9688
102f04e297
Add sub-channel stuff to PlaylistInfo
2020-05-12 09:14:29 +02:00
Tobias Groza
b40ccb5075
Merge pull request #307 from B0pol/playlists_peertube
...
[PeerTube] playlist support
2020-05-11 20:53:03 +02:00
bopol
adaf196c99
remove unused import
2020-05-11 12:54:33 +02:00
bopol
de26e00079
changed all == null || isEmpty() to isNullOrEmpty()
2020-05-11 11:40:24 +02:00
bopol
202a73516c
refactor: add Utils.isNullOrEmpty()
2020-05-11 11:26:18 +02:00
wb9688
1de0876fb4
Fix #327
2020-05-09 20:06:50 +02:00
bopol
7cae95bf09
add tests for subchannel stuff in playlists
2020-05-09 10:15:35 +02:00
bopol
1a6e92ebf6
add getSubChannel Name, Url and AvatarUrl for playlists
2020-05-09 09:52:24 +02:00
bopol
cf99107745
address yausername change
2020-05-09 00:52:04 +02:00
bopol
5bab9d9fc0
[PeerTube] playlist support & refactoring
2020-05-09 00:52:04 +02:00
Tobias Groza
f3913e241e
Merge pull request #324 from Stypox/music-detector-url
...
[YouTube] Support stream urls in `vnd.youtube://ID` form
2020-05-08 17:43:04 +02:00
wb9688
7320108c66
Rename SubChannel in channels back to ParentChannel
2020-05-08 16:02:59 +02:00
Roy Yosef
a408661771
Fix PR review
...
Rename "ParentChannel" to "SubChannel"
Add Tests for sub-channel info fetching
2020-05-07 20:20:12 +02:00
Roy Yosef
1de1f97cf1
Add parent channel info to StreamExtractor: name, url & avatar url
2020-05-07 20:20:12 +02:00
Roy Yosef
4234740baa
Add parent channel info to ChannelExtractor: name, url & avatar url
2020-05-07 20:20:12 +02:00
bopol
6cff5de400
renaming methods for consistency
2020-05-07 15:40:41 +02:00
bopol
5760366d70
use JsonObject#isEmpty instead of JsonObject#equals(JsonUtils.EMPTY_OBJECT)
2020-05-07 15:40:41 +02:00
bopol
93e868634e
clarified javadoc for isCorrectedSearch and getSearchSuggestions in SearchExtractor
2020-05-07 15:40:41 +02:00
bopol
3757541a9c
adapt new nanojson version
2020-05-07 15:40:41 +02:00
bopol
79483a6dc0
«Use this in SearchInfo, for consistency's sake» @Stypox order xD
2020-05-07 15:40:41 +02:00
bopol
e8d58e09c9
getSearchSuggestion now returns the corrected query if isCorrectedSearch() is true
2020-05-07 15:40:41 +02:00
bopol
5d1c3b3fa2
Use setters in SearchInfo
2020-05-07 15:39:57 +02:00
bopol
29b639b454
Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSearchSuggestion()
...
isCorrectedSearch: see the javadoc
getSearchSuggestion: in YoutubeSearchExtractor, it was giving the corrected search query. It now gives the suggested query, as it should
2020-05-07 15:39:57 +02:00
Stypox
a02bfbeae5
[YouTube] Support stream urls in vnd.youtube://ID
form
2020-05-07 15:31:31 +02:00
bopol
bc13e0c616
Merge remote-tracking branch 'upstream/dev' into soundcloudComments
2020-05-05 14:49:21 +02:00
wb9688
6aae99f832
Fix NPE in defaultAlertsCheck()
2020-05-03 10:28:45 +02:00
wb9688
a1eabc7589
Return null instead of "" in getTextFromObject()
2020-05-02 08:02:25 +02:00
Fynn Godau
e7b046db9d
Merge remote-tracking branch 'origin/dev' into dev
2020-04-26 23:12:43 +02:00
Fynn Godau
a3e8e1c70e
Bandcamp: move own methods to helper class
2020-04-26 23:05:56 +02:00
Fynn Godau
9bc7a470ab
Replace fori with foreach loop
2020-04-26 23:02:14 +02:00
Tobias Groza
49157fcf0d
Merge pull request #317 from wb9688/tnp-nanojson
...
Use our fork of nanojson
2020-04-24 21:54:35 +02:00
bopol
2564bcf399
refactor comments
...
rename methods in CommentsInfoItemExtractor interface to match the other method names across NewPipeExtractor
remove getName in (Youtube|SoundCloud|Peertube)CommentsExtractor and move it up in CommentsExtractor, return "Comments" instead
2020-04-21 20:05:54 +02:00
Fynn Godau
f71bd29d58
Bandcamp playlist extractor: add more tests
2020-04-20 23:52:12 +02:00
Fynn Godau
5009d9f53b
Bandcamp tests: remove certain tests
2020-04-20 23:25:00 +02:00
Fynn Godau
b5e251c82f
Bandcamp channel extractor: package-private access for getImageUrl(…)
2020-04-20 23:12:43 +02:00
Fynn Godau
8c70dab8c5
Bandcamp search query handler factory: inline variable
2020-04-20 23:11:16 +02:00
Fynn Godau
10ae3db118
Bandcamp javadoc: Replace br tags with p tags
2020-04-20 23:03:12 +02:00
Fynn Godau
9201e0d3ea
Don't declare exceptions that can't actually be thrown
2020-04-20 22:46:54 +02:00
Fynn Godau
965bce00cf
Bandcamp stream extractor: return NO_AGE_LIMIT
2020-04-20 22:44:45 +02:00
Fynn Godau
82099592c7
Bandcamp: Move code from SearchExtractor to InfoItemExtractors
2020-04-20 22:44:35 +02:00
Fynn Godau
00c0333059
Bandcamp: Fetch channelInfo in onFetchPage
2020-04-20 22:41:44 +02:00
Fynn Godau
67de0285e1
Bandcamp: Generate query JSON in a proper way
2020-04-20 22:03:29 +02:00
Fynn Godau
c133190c53
Bandcamp capitalization
2020-04-20 21:55:35 +02:00
Fynn Godau
b100b9873f
Bandcmap: Use nanojson fork instead of org.json
2020-04-20 21:53:04 +02:00
wb9688
b51699a20e
Implement @TobiGr's suggestions
2020-04-20 14:27:33 +02:00
wb9688
74f0ab1792
Merge pull request #301 from wb9688/media-ccc-non-api-urls
...
Use media.ccc.de instead of api.media.ccc.de
2020-04-19 17:06:57 +02:00
Fynn Godau
9c239371f4
Change nanojson dependency to @wb9688's fork
2020-04-19 14:29:39 +02:00
Fynn Godau
5c0a03328a
Merge 'origin/dev' into dev
2020-04-19 14:26:51 +02:00
bopol
0cbbc2a1f9
remove unused import
2020-04-19 14:16:38 +02:00
wb9688
979c5a7502
Use our fork of nanojson
2020-04-17 19:36:42 +02:00
bopol
d4352f9b84
support comments for SoundCloud
2020-04-10 20:41:41 +02:00
Roy Yosef
4afe657f6f
Add tests for getUrlFromId and test for video-channels in acceptUrlTest
2020-04-10 19:05:38 +03:00
Roy Yosef
2c9f1260eb
Fix wb9688 review comments
...
* Rename PeertubeUserExtractor to PeertubeAccountExtractor
* Add test for video-channels in PeertubeChannelLinkHandlerFactoryTest
* Compatibility support for older versions (use "accounts/" as default)
2020-04-10 16:40:53 +03:00
bopol
636c430743
refactor: create extractors and linkHandler packages for SoundCloud, move YoutubeParsingHelper to youtube package (not linkhandler anymore)
2020-04-10 10:51:05 +02:00
Roy Yosef
b6e6f403a8
add support for PeerTube channels extraction
2020-04-09 20:37:49 +03:00
bopol
a20d53dd98
remove the mention of Opus on wb9688 advise when
...
ContentNotSupportedException is thrown
2020-04-08 15:31:11 +02:00
bopol
9a7c6b7ab0
use ContentNotSupportedException for channels without tabs
2020-04-07 14:33:43 +02:00
bopol
a22104cbda
add tests for ContentNotSupportedException
2020-04-07 14:33:43 +02:00
bopol
093762e793
throw ContentNotSupportedException when content is know to be unsupported
2020-04-07 14:33:43 +02:00
Tobias Groza
094b87c537
Merge pull request #299 from B0pol/sc_countries
...
add supported countries for soundcloud
2020-04-03 19:12:33 +02:00
wb9688
bce27a0e22
Rename getValidResponseBody() to getValidJsonResponseBody()
2020-04-03 17:23:18 +02:00
bopol
1eb3deb7fd
add supported countries for soundcloud
2020-04-02 11:26:19 +02:00
wb9688
8a9e137385
Extract some code to getValidResponseBody()
2020-04-01 16:01:21 +02:00
wb9688
bf24caa5f2
Use media.ccc.de instead of api.media.ccc.de
2020-04-01 13:25:10 +02:00
wb9688
2af610e3e7
Add tests for other YT Music search types
2020-04-01 10:30:51 +02:00
wb9688
c7f7bd2442
Fix error when YT Music videos has no uploader URL
2020-04-01 10:30:51 +02:00
wb9688
ac15df4548
Move YouTube Music search to its own class
2020-04-01 10:30:51 +02:00
wb9688
cf0f2aff3e
Extract uploader url from certain YouTube Music videos
2020-04-01 10:30:51 +02:00
wb9688
aa8cea47f3
Refactor YouTube Music search tests
2020-04-01 10:30:51 +02:00
wb9688
dd434cca01
Fix issue when there is no didYouMeanRenderer in itemSectionRenderer
2020-04-01 10:30:51 +02:00
wb9688
5a775a4bbe
Use new way of specifying stream count
2020-04-01 10:30:51 +02:00
wb9688
d58c0f230d
Improve code for YouTube Music search
2020-04-01 10:30:51 +02:00
wb9688
dc29d87962
Extract YouTube search suggestions
2020-04-01 10:30:51 +02:00
wb9688
c852b13d5a
Add Referer header so that it also works with HttpsUrlConnection
2020-04-01 10:30:51 +02:00
wb9688
eb48524411
Add tests for YouTube Music search
2020-04-01 10:30:51 +02:00
wb9688
2b9b2a78e8
Handle 100+ items in playlist
2020-04-01 10:30:51 +02:00
wb9688
fb9b9691b7
Improve getYoutubeMusicKeys()
2020-04-01 10:30:51 +02:00
wb9688
4ddbdf0aee
Disable artist search for now
2020-04-01 10:30:51 +02:00
wb9688
1762a527c9
Add support for YouTube Music search
2020-04-01 10:30:51 +02:00
Tobias Groza
69e0624e35
Merge pull request #298 from B0pol/feed_peertube
...
[peertube] add feed url in channels
2020-03-28 12:53:47 +01:00
bopol
0ae53c1e50
[peertube] adapt test for channel feed url
2020-03-28 11:48:24 +01:00
bopol
bc43f8977d
[peertube] add feed url in channels
2020-03-28 10:22:47 +01:00
TobiGr
647e7cd450
Accept YouTube Music playlists but not YouTube Mixes
2020-03-25 22:24:18 +01:00
wb9688
bcd2a1092b
Don't accept YouTube Mix playlists
2020-03-25 11:07:28 +01:00
Tobias Groza
e4874d554e
Merge pull request #296 from mauriciocolli/fix-search-errors-detection
...
Fix search errors detection and refactor search tests
2020-03-24 21:53:32 +01:00
Tobias Groza
8a9ae32e6b
Merge pull request #292 from wb9688/fix-yt-no-views
...
Fix YouTube videos with no views
2020-03-24 20:27:08 +01:00
Mauricio Colli
0a20c53f1a
Use easier-to-read asserts statements in tests
2020-03-23 18:08:59 -03:00
Mauricio Colli
c921e5d6a5
Remove unnecessary collector creation function from search extractor
2020-03-23 18:08:51 -03:00
Tobias Groza
6fd9b38ad9
Merge pull request #287 from mauriciocolli/fix-channel-redirect
...
[YouTube] Fix channel with redirects directly in the response
2020-03-23 18:41:56 +01:00
Mauricio Colli
04bfa0ec6a
[PeerTube] Update video used in age limit test (was 404 not found)
2020-03-21 15:55:40 -03:00
Mauricio Colli
921bf30bb7
[YouTube] Add check for channel items with no video count in search
2020-03-21 15:55:39 -03:00
Mauricio Colli
b7f8001a49
[YouTube] Add check for channel items without description in search
2020-03-21 15:55:38 -03:00
Mauricio Colli
9b7999fe54
[YouTube] Check if channel item has subscription count in search
2020-03-21 15:55:37 -03:00
Mauricio Colli
9704fc9952
Improve search extractor tests for services
2020-03-21 15:55:36 -03:00
Mauricio Colli
d72130edae
Always return a new item collector in the search extractor
2020-03-21 15:55:34 -03:00
Mauricio Colli
0c27198ba1
Include errors from child collectors in the search collector
2020-03-21 15:55:29 -03:00
TobiGr
2cf8cbf0df
Merge branch 'master' into dev
2020-03-18 22:29:10 +01:00
wb9688
222d659d9e
[SoundCloud] Don't make separate request for getAudioStreams() in StreamExtractor
...
Signed-off-by: Stypox <stypox@pm.me>
2020-03-18 11:30:15 +01:00
wb9688
904c9d681f
Fix YouTube videos with no views
2020-03-18 10:07:49 +01:00
Stypox
265cfb61f3
[SoundCloud] Add test for small playlist, use defaultTestGetPageInNewExtractor
2020-03-17 21:03:50 +01:00
Stypox
45bb646480
[SoundCloud] Do not overwrite nextPageUrl in PlaylistExtractor
...
Consistent with YouTube and the documentation
2020-03-17 20:56:47 +01:00
Fynn Godau
a1523eb293
Update Bandcamp service to latest interface changes
2020-03-17 20:40:25 +01:00
Stypox
c505d4e2b7
[SoundCloud] Remove trailing , in playlist page urls
2020-03-17 20:36:59 +01:00
Stypox
ae47c9587c
[SoundCloud] Optimize imports in edited files
2020-03-17 20:32:04 +01:00
Stypox
4389fd3b7b
[SoundCloud] Migrate StreamExtractor to api-v2
2020-03-17 18:06:13 +01:00
Stypox
5e4ddb368f
[SoundCloud] Fix extractors built from next playlist pages
...
They didn't have the information to calculate another next page url. So now `nextPageUrl` contains a full link with all video ids, and `getPage` takes the first part of the url (containing 15 streams) and produces another `nextPageUrl` with the remaining streams.
Also add a test for this.
2020-03-17 18:04:40 +01:00
fynngodau
b78f788017
Merge branch 'dev' into dev
2020-03-17 18:03:14 +01:00
Stypox
0e1b4bbf17
[SoundCloud] Test playlists banner: it should not exist
2020-03-17 15:53:25 +01:00
Stypox
65bdb3bc9d
[SoundCloud] Unignore ignored playlist tests
2020-03-17 15:49:58 +01:00
Stypox
1558da6f6b
[SoundCloud] Fix playlist next page generation
2020-03-17 15:46:35 +01:00
Stypox
ca8bf53b61
[SoundCloud] Fix playlist test: number of streams changed
2020-03-17 15:35:33 +01:00
Stypox
d4aa4a0763
[SoundCloud] Fix typo in ChartsExtractor
2020-03-17 15:25:11 +01:00
Stypox
4b1121aac7
[SoundCloud] Add tests for api-v2 channel urls
2020-03-17 15:23:13 +01:00
Stypox
c3d811fde5
[SoundCloud] Use api-v2 in SubscriptionExtractor
...
Also added
2020-03-17 15:22:25 +01:00
Stypox
d0e66cc600
[SoundCloud] Improve thumbnail url extraction in playlists
...
Prevent NullPointerExceptions and remove duplicate code
2020-03-17 15:13:28 +01:00
Stypox
f3095713f9
[SoundCloud] Use api-v2 in PlaylistExtractor
...
Rewrote methods to calculate next page url and to get items from it. `api-v2` is different from `api` since the initial playlist page contains (usually) the full info of the first 3 streams and only the id of the other. Then the single tracks can be requested in batch using `/tracks?ids=id1,id2,...`.
2020-03-17 15:12:13 +01:00
TobiGr
9eca7df947
Forget to fetch the page
2020-03-16 20:50:08 +01:00
TobiGr
edb57840d7
[SoundCloud] Update hard-coded client_id
2020-03-16 19:43:36 +01:00
TobiGr
efad0be99f
Test if the extractor can get AudiStreams with the hard-coded cliend_id
...
SoundCloud client_id are changed on a regular basis now. We might want to remove the hardcoded id completely on a later point of time when client_id is changed to often
2020-03-16 19:42:35 +01:00
TobiGr
5f39bc4e99
[SoundCloud] Update hardcoded client id
2020-03-16 17:26:30 +01:00
Tobias Groza
65a7eda446
Merge pull request #256 from B0pol/mediaccc
...
Support for non-api mediaccc links, invidious shortened links, refactor mediaccc tests
2020-03-15 17:04:09 +01:00
TobiGr
40b3bc138f
Remove hack to fix YoutubeStreamLinkHandlerFactoryTest of vnd.youtube videos
2020-03-14 21:54:54 +01:00
Tobias Groza
1c954fbded
Merge pull request #277 from XiangRongLin/vnd
...
Handle youtube stream urls in "vnd.youtube:videoId" format.
2020-03-14 21:52:49 +01:00
Tobias Groza
9edd0f2675
Merge pull request #288 from Stypox/fix-unlisted-video
...
[YouTube] Fix videos without next stream throwing error and add test for unlisted videos
2020-03-14 21:36:13 +01:00
Stypox
957b78ed9b
[YouTube] Add test for an unlisted video, which has no next stream
2020-03-14 14:04:01 +01:00
Stypox
325a51f73a
[YouTube] Do not crash if there is no next stream, but return null
...
Like every other extractor does if there is no next stream
2020-03-14 14:02:48 +01:00
TobiGr
aea0962cfe
[SoundCloud] Update hardcoded client id
2020-03-14 12:41:44 +01:00
Mauricio Colli
b086e9db3f
[YouTube] Fix id extraction for some channels
...
Some channels had no reliable way to get the redirected id in the response,
so saving it for later was a valid alternative.
2020-03-14 02:39:49 -03:00
Mauricio Colli
00d1ed439b
[YouTube] Fix channel extraction when redirects are in the response
...
Some redirects were embed directly into the response as instructions
for the page, instead of the usual http redirects.
2020-03-14 02:39:48 -03:00
bopol
66518ec444
check wether mediaccc streams url are secure
2020-03-13 20:07:36 +01:00
bopol
0cd5e05b7b
MediaCCCLH: use substring instead of replace; improve a bit tests, return Collections.emptyList(); instead of null where it's annotated @NonNull
2020-03-13 20:00:49 +01:00
bopol
f742a6bd3e
code optimization
2020-03-13 20:00:49 +01:00
bopol
9701c7d800
invidious shortened links
2020-03-13 20:00:49 +01:00
bopol
e8e535b815
mediaccc: update linkhandlers & refactor tests
2020-03-13 20:00:49 +01:00
Mauricio Colli
70abd57852
[YouTube] Cache commonly used stream type result
2020-03-07 16:48:32 -03:00
Mauricio Colli
1ef706f567
[YouTube] Detect LIVE videos in the trending page
2020-03-07 16:48:31 -03:00
Xiang Rong Lin
cc8fb486ee
Handle youtube stream urls in "vnd.youtube:videoId" format.
...
The official YouTube app accept intents in this format, which causes other developers to use it.
2020-03-02 20:40:33 +01:00
Mauricio Colli
6aa4d59b91
[PeerTube] Update video used in age limit test (the previous was 404)
2020-03-01 10:29:30 -03:00
Mauricio Colli
cfc278317d
[YouTube] Fail-fast if status exist and is anything other than "OK"
2020-03-01 10:29:29 -03:00
Mauricio Colli
90ae5fbea2
Improve kiosk tests across services
2020-03-01 12:57:18 +01:00
Mauricio Colli
e590417cc4
Test if services recognizes their own items urls
2020-03-01 12:57:18 +01:00
TobiGr
35252235b0
Go through all badges when checking if a StreamInfoItem is a live stream
2020-03-01 12:52:37 +01:00
Mauricio Colli
5686a6f562
[YouTube] Detect when a stream is deleted or doesn't exist
...
Added a test case as well.
2020-03-01 12:49:13 +01:00
Mauricio Colli
e65333c3ce
[YouTube] Detect deleted/nonexistent/invalid channels and playlists
...
- Added tests for these cases.
2020-03-01 12:49:13 +01:00
Mauricio Colli
98e359438a
[YouTube] Detect simple 404s in the standard fetch method
2020-03-01 12:49:13 +01:00
Mauricio Colli
408f042127
[YouTube] Fix bug when url isn't present in the browseEndpoint object
2020-03-01 12:49:13 +01:00
Mauricio Colli
342bdbb852
[YouTube] Avoid crashing by letting exceptions bubble up
2020-03-01 12:49:13 +01:00
Mauricio Colli
e9644e6216
[YouTube] Handle video premiere's date and duration
2020-03-01 12:49:13 +01:00
Mauricio Colli
2a470ac4f7
[YouTube] Handle videos with no views or with "Recommended to you" text
2020-03-01 12:49:13 +01:00
Mauricio Colli
da2c562eb0
[YouTube] Take into account videos that have their views hidden
...
Also remove catch because it may hide bugs when the page changes.
2020-03-01 12:49:13 +01:00
Mauricio Colli
f66c20de54
Ignore null-keyed entries when iterating through the response headers
2020-03-01 12:49:13 +01:00
Mauricio Colli
5edd774fc4
Add latest url to the response to make detection of a redirect possible
...
Will be latest one in this commit because there's need to check the
history of redirects as of now.
2020-03-01 12:49:13 +01:00
Mauricio Colli
3441946bea
Make test downloader return a response instead of throwing an exception
...
The test implementation was throwing an exception instead of just
returning the response and letting the caller handle it.
2020-03-01 12:49:13 +01:00
wb9688
df219a9453
Don't use StandardCharsets class
2020-03-01 09:41:06 +01:00
wb9688
2807079531
Catch NumberFormatException instead of checking whether viewCount equals "Recommended for you"
2020-02-29 21:28:38 +01:00
wb9688
3e4d2fd64e
Improve testChannelUrl()
2020-02-29 18:20:35 +01:00
wb9688
bfe3eb1409
Use suggested try-if code style
2020-02-29 17:18:50 +01:00
wb9688
a6a63e9570
Let isHardcodedClientVersionValid() throw IOException
2020-02-29 17:01:36 +01:00
wb9688
45df8248bf
Use Localization in getJsonResponse()
2020-02-29 16:55:07 +01:00
wb9688
157055fbfd
Create getJsonResponse() function
2020-02-29 16:42:04 +01:00
wb9688
4f406512c2
Use YoutubeChannelLinkHandlerFactory
2020-02-29 16:26:28 +01:00
wb9688
4ff655352d
Reimplement getAgeLimit()
2020-02-28 17:14:26 +01:00
wb9688
6ba03e542b
Cache getVideoTab()
2020-02-28 17:05:31 +01:00
wb9688
8bfbae904d
Cache getVideo(Primary|Secondary)InfoRenderer()
2020-02-28 17:03:21 +01:00
wb9688
0b940f5cf4
Use "getAgeLimit() != NO_AGE_LIMIT" instead of "getAgeLimit() == 18"
2020-02-28 16:40:50 +01:00
wb9688
beb07a4852
Add test to check whether hardcoded client version is still valid
2020-02-28 16:36:14 +01:00
wb9688
b89b6b1db2
Check if hard-coded client version is valid in separate function
2020-02-28 16:35:24 +01:00
wb9688
7898ba230d
Use pbj in YoutubeStreamExtractor
2020-02-28 15:17:47 +01:00
wb9688
612ec06839
Actually fix thumbnail URLs starting with //
2020-02-28 09:36:33 +01:00
wb9688
f946310af5
Fix search queries with a promotedSparklesTextSearchRenderer returning 0 results
2020-02-28 08:57:44 +01:00
wb9688
94a2d5be46
Fix thumbnail URLs starting with //
2020-02-27 19:08:46 +01:00
wb9688
838d834ce2
Get Uploads view of Videos tab
2020-02-27 18:54:27 +01:00
wb9688
365b0329f3
Implement getTextFromObject() function
2020-02-27 17:39:23 +01:00
wb9688
0798bdd5cd
Fix getSubscriberCount() for 0 subscribers
2020-02-27 16:26:22 +01:00
wb9688
8ebd971648
Fix channel descriptions consisting of multiple parts
2020-02-27 10:06:35 +01:00
wb9688
880b951088
Fix getUploaderAvatarUrl() in YoutubeStreamExtractor
2020-02-27 09:53:59 +01:00
wb9688
8aa560bef4
Fix video titles consisting of multiple parts due to @
2020-02-27 09:42:22 +01:00
wb9688
951ed9f0d8
Don't call getUploaderInfo() in onFetchPage() for playlists
...
This way YouTube Music playlists will keep working in NewPipe
2020-02-27 09:27:56 +01:00
wb9688
2399d46ab0
Fix YoutubeChannelExtractor for channels without videos
2020-02-27 09:21:25 +01:00
wb9688
cdbc751b87
Fix getUploaderUrl() in YoutubeStreamInfoItemExtractor
2020-02-27 09:05:41 +01:00
wb9688
db305408e9
Get all expandedShelfContentsRenderers in YoutubeTrendingExtractor
2020-02-26 20:37:20 +01:00
wb9688
a2d8489e16
Parse timestamps in description
2020-02-26 20:27:30 +01:00
wb9688
8d503407ef
Parse browseEndpoint and watchEndpoint in descriptions
2020-02-26 19:37:58 +01:00
wb9688
8dc39d517a
Fix tests
...
Except testGetFullLinksInDescription() in DescriptionTestUnboxing, since that requires other changes first.
2020-02-26 17:31:01 +01:00
wb9688
88282761f4
Fix YoutubeStreamExtractor not being able to getAgeLimit() in onFetchPage()
2020-02-26 16:19:42 +01:00
wb9688
bb3a3d70bf
Use pbj in YoutubeTrendingExtractor
2020-02-26 16:10:24 +01:00
wb9688
1c3a4e710b
Check if there are no related streams in getAgeLimit()
2020-02-26 16:03:06 +01:00
wb9688
985c3ec877
Improve getClientVersion()
2020-02-26 15:27:50 +01:00
wb9688
15afbea3e1
Use pbj in YoutubePlaylistExtractor
2020-02-26 15:27:50 +01:00
wb9688
89a41a7f69
Use pbj in YoutubeChannelExtractor
2020-02-26 15:27:50 +01:00
wb9688
0973263aab
Use pbj in YoutubeSearchExtractor
2020-02-26 15:27:46 +01:00
TobiGr
fc465c8bf7
Fix getUploadDate() for premiered videos with a given date
...
Detect publishDate and uploadDate
2020-02-25 22:54:37 +01:00
TobiGr
96285e0815
Fix getUploadDate() for premiered videos with a given date
2020-02-25 21:51:30 +01:00
wb9688
5a35300a5e
Fix parsing upload date for premiered streams
2020-02-25 21:51:30 +01:00
wb9688
729fd2eaee
Fix parsing description for navigationEndpoint without urlEndpoint
2020-02-25 21:51:30 +01:00
wb9688
26fb44595f
Fix parsing description
2020-02-25 21:51:30 +01:00
TobiGr
0fff03038e
Get description with correct links
2020-02-25 21:51:30 +01:00
wb9688
5d08c34322
Fix parsing upload date in YoutubeStreamExtractor
2020-02-25 21:51:30 +01:00
wb9688
bbe1a3cd62
Use getVideo(Primary|Secondary)InfoRenderer()
2020-02-25 21:51:30 +01:00
wb9688
8e6be880d7
Simplify getId() in YoutubeChannelExtractor
2020-02-25 21:51:30 +01:00
TobiGr
ecfc27a92c
Implement fallback methods for getDescription() and getTextualUploadDate() in YouTubeStreamExtractor
2020-02-25 21:51:30 +01:00
wb9688
eed29ea734
Return null for related streams at age restricted videos
2020-02-25 21:51:30 +01:00
TobiGr
9efcc61ca6
Adapt some tests to match the new URLs
2020-02-25 21:51:30 +01:00
TobiGr
f39603f6ef
Implement getUploadDate() in YouTubeStreamInfoItemExtractor
2020-02-25 21:51:30 +01:00
wb9688
26ea3dceb6
Fix getViewCount() in YoutubeStreamExtractor
2020-02-25 21:51:30 +01:00
wb9688
76d54abdbf
Reimplement more methods in YoutubeStreamExtractor
2020-02-25 21:51:30 +01:00
wb9688
f13c0288cc
Reimplement some methods in YoutubeStreamExtractor
2020-02-25 21:51:30 +01:00
wb9688
02b59903fa
Remove useless code
2020-02-25 21:51:30 +01:00
TobiGr
dd4dd849dc
Get client version dynamically in YouTubeSearchExtractor
2020-02-25 21:51:30 +01:00
TobiGr
583e9c1571
Fix getPage()
...
Someone should check if there are unstaged changes before committing...
2020-02-25 21:51:30 +01:00
TobiGr
5842b9ad37
Add getClientVersion() and HARDCODED_CLIENT_VERSION to YouTubeParsingHelper
...
Prefer hardcoded client version above the current one when making requests to retrieve the same JSON structure for each request.
2020-02-25 21:51:30 +01:00
wb9688
5d883d100c
Implement pagination in YoutubeSearchExtractor
2020-02-25 21:51:30 +01:00
wb9688
c0a8e01889
Implement pagination in YoutubePlaylistExtractor
2020-02-25 21:51:30 +01:00
wb9688
4039409820
Fix getLikeCount() for certain streams
2020-02-25 21:51:30 +01:00
wb9688
627ab2459f
Uncomment fetchPage()
2020-02-25 21:51:30 +01:00
wb9688
2dfa2187ff
Implement pagination in YoutubeChannelExtractor
2020-02-25 21:51:30 +01:00
wb9688
a38ab9b791
Return null if channel has no banner
2020-02-25 21:51:30 +01:00
wb9688
1eca63089b
Count private and deleted videos as ad
2020-02-25 21:51:30 +01:00
wb9688
c7360e4a46
Fix getThumbnailUrl() in YoutubeChannelInfoItemExtractor
2020-02-25 21:51:30 +01:00
wb9688
51fb26625a
Implement getSubscriberCount() in YoutubeChannelInfoItemExtractor
2020-02-25 21:51:30 +01:00
TobiGr
6d20b2b1d0
Fix NPE on collecting page items
2020-02-25 21:51:30 +01:00
wb9688
4462cbe3f1
Reimplement YoutubePlaylistInfoItemExtractor
2020-02-25 21:51:30 +01:00
wb9688
8aea4d445b
Reimplement YoutubeChannelInfoItemExtractor
2020-02-25 21:51:30 +01:00
wb9688
3187116a63
Handle premium videos
2020-02-25 21:51:30 +01:00
wb9688
7442f91aa6
Don't assume which channel tab is videos
2020-02-25 21:51:30 +01:00
wb9688
316fe0109d
Fix first playlist page
2020-02-25 21:51:30 +01:00
wb9688
4e57e589ce
Fix first channel page
2020-02-25 21:51:30 +01:00
TobiGr
5816202cc7
Move getInitialData() method to YouTubeParsingHelper
...
Rename ytInitialData to initialData
2020-02-25 21:51:30 +01:00
wb9688
38aabc6aca
Use plain YoutubeStreamInfoItemExtractor in YoutubeTrendingExtractor
2020-02-25 21:51:30 +01:00
wb9688
957db062ef
Fix getting uploader for videoRenderer
2020-02-25 21:51:30 +01:00
wb9688
b88188d419
Move stuff from extractVideoPreviewInfo() into YoutubeStreamInfoItemExtractor and partially fix search
2020-02-25 21:51:30 +01:00
wb9688
af49b3c487
Fix live streams in extractVideoPreviewInfo()
2020-02-25 21:51:30 +01:00
TobiGr
f33f9466ce
Fix testDescription() in ChannelExtractor
2020-02-25 21:51:30 +01:00
TobiGr
10e9c16d8c
Fix LiveStream view count
2020-02-25 21:51:30 +01:00
wb9688
127c4d5893
Fix getNextStream() in YoutubeStreamExtractor
2020-02-25 21:51:30 +01:00
wb9688
7bfc0e62c6
Fix getRelatedStreams() in YoutubeStreamExtractor
2020-02-25 21:51:30 +01:00
TobiGr
12166ca60c
[StreamExtractor] Add fallback method for uploader avatar url
2020-02-25 21:51:30 +01:00
wb9688
3ec422a542
Fix like/dislike parsing
2020-02-25 21:51:30 +01:00
TobiGr
a34e5c18e1
Trending StreamInfoItemCollecetor
2020-02-25 21:51:30 +01:00
TobiGr
b705515da0
Adapt most playlist fields
2020-02-25 21:51:30 +01:00
TobiGr
21253abd9b
Use Firefox ESR User-Agent: 68
2020-02-25 21:51:30 +01:00
TobiGr
655c999795
Detect disabled subriber count correctly
...
Fix parsing of kiosk name
2020-02-25 21:51:30 +01:00
TobiGr
7dcc9f159b
Stream: fix uploader avatar url
2020-02-25 21:51:30 +01:00
TobiGr
be3e20e263
Adapt most channel changes
2020-02-25 21:51:30 +01:00
TobiGr
39bf1ff1eb
Add ytInitialData regex
2020-02-25 21:51:30 +01:00
Tobias Groza
a129c65c63
more improvements of comments
...
Co-Authored-By: Stypox <stypox@pm.me>
2020-02-16 23:59:31 +01:00
TobiGr
0a4115286b
Apply one suggestion
2020-02-15 13:36:54 +01:00
Tobias Groza
6822574588
Fix typos
...
Co-Authored-By: B0pol <bopol@e.email>
2020-02-15 13:36:54 +01:00
TobiGr
030465b5d4
Improve code formatting and optimise imports
2020-02-15 13:36:54 +01:00
TobiGr
9112a106ea
[SoundCloud] Update client_id
2020-02-08 23:09:03 +01:00
TobiGr
008b6f505c
[PeerTube] Fix subscriber count in ChannelExtractorTest
2020-02-08 23:08:37 +01:00
TobiGr
0e33249c85
Fix SoundCloud description test
2020-02-08 22:20:20 +01:00
Fynn Godau
36a316eea5
Merge branch 'dev' into dev
2020-02-08 09:43:13 +01:00
bopol
11bcc78d9c
Description implements Serializable. fix NotSerializableException
2020-02-07 14:10:54 +01:00
bopol
0f8a7f9df2
fix testGetUploadDate for PeerTubeStreamExtractor
2020-02-07 13:33:50 +01:00
bopol
70a40e7388
Description: rm constructor by serviceId
2020-02-07 13:28:27 +01:00
bopol
26c65b2948
Create class Description
2020-02-06 23:35:46 +01:00