bopol
ed850d0688
[youtube] improve comments extraction performance
...
- do not parse responseBody twice for continuation
instead try to get commentsTokenInside with the new pattern ("sectionListRenderer")
and try again with the old pattern ("commentSectionRenderer") on failure
- do not unescape responseBody multiple times
-> parse responseBody less times
2021-03-05 13:37:08 +01:00
TobiGr
440a808b8a
Update mocks for channels
2021-03-05 13:13:36 +01:00
Tobi
b608587e4d
Merge pull request #572 from golfinq/fixed-comments
...
Fixes missing comments under videos
2021-03-05 11:42:40 +01:00
golfinq
05213175cd
minor edit, made variable final
2021-03-05 02:49:15 -05:00
golfinq
8bceb57a6c
formatting fix
2021-03-05 02:46:06 -05:00
golfinq
ae48bdea4c
Updated to reflect variable name change
2021-03-05 02:40:36 -05:00
TiA4f8R
b27efdc9a4
Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor
...
Because it's unused with the new request.
Co-authored-by: bopol <bopol@e.email>
2021-03-04 22:50:20 +01:00
TobiGr
2b671b15ce
[YouTube] Use new continuation API for channels
...
Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>
2021-03-04 21:47:16 +01:00
TobiGr
33594d4aff
[SoundCloud] Fix client id extraction
...
SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.
2021-03-04 21:32:34 +01:00
TobiGr
fb2f37d223
[SoundCloud] Update hardcoded client id
2021-03-04 21:25:41 +01:00
XiangRongLin
9256b3b848
Merge pull request #567 from XiangRongLin/playlist_continuations
...
Playlist continuations
2021-03-04 19:05:32 +01:00
XiangRongLin
506cc5fbe9
regenerate mocks
2021-03-04 19:01:58 +01:00
XiangRongLin
03b00ff1d6
Move toJsonArray and toJsonObject to JsonUtils
2021-03-04 18:58:51 +01:00
FireMasterK
668b080b3a
Use dynamic key.
2021-03-04 08:45:14 +01:00
XiangRongLin
43b46bd408
Use new youtube continuations api for playlists
...
Requires sending a POST request instead of GET.
clientName and clientVersion, which were required as headers previously now need to be part of the request payload.
continuation id also needs to be part of request body.
quick and dirty solution.
2021-03-03 19:49:26 +01:00
XiangRongLin
beb05bd05c
Replace test url, because playlist was deleted
2021-03-03 19:46:14 +01:00
XiangRongLin
e13e237392
Fix typo and reword some explanations
2021-02-26 17:48:03 +01:00
Tobi
5d594cfded
Merge pull request #549 from FireMasterK/gradle-update
...
Update gradle wrapper.
2021-02-24 15:59:28 +01:00
Tobi
965a66bc87
Merge pull request #557 from TiA4f8R/fix-encoding-errors-windows
...
Set extractor encoding to UTF-8
2021-02-24 15:49:58 +01:00
TiA4f8R
b2cf41496d
Set extractor encoding to UTF-8
...
When compiling the extractor on Windows, you get more than 100 errors with the message error: unmappable character for encoding Cp1252, especially in timeago-parser tasks. This commit fixes it by setting the encoding for all compileJava and compileTestJava tasks to UTF-8.
2021-02-23 14:38:20 +01:00
TobiGr
a1688fe953
Move BandcampExtractorHelper.getJsonData(String, String) to JsonUtils
2021-02-21 13:51:12 +01:00
TobiGr
70814dcfef
Fix Utils.nonEmptyAndNullJoin
...
When using the index here, it the index needs to be decremented once an element is removed. To cirecumvent this, the native Collections.removeIf() method is used.
2021-02-21 12:51:47 +01:00
TobiGr
b9e8ee8450
Rename BandcampExtractorHelper.smartConcatenate(String[], String) to Utils.nonEmptyAndNullJoin(String, String[])
2021-02-21 12:35:14 +01:00
TobiGr
c07db80ef0
Add BASE_URL and BASE_API_URL to BandcampExtractorHelper
2021-02-21 12:15:45 +01:00
TobiGr
adde4332d1
Code improvements
2021-02-20 23:58:54 +01:00
FireMasterK
42bdd8adc5
Update to fix failing mock tests.
2021-02-20 17:14:41 +05:30
FireMasterK
e8bccfaf5d
Change UA to privacy.resistFingerprinting.
2021-02-20 17:14:41 +05:30
bopol
a9aa385bb3
Merge pull request #554 from B0pol/test-verified
...
use DownloaderFactory instead of DownloaderTestImpl for search Channe…
2021-02-20 12:12:43 +01:00
bopol
0c831afa7b
use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified
2021-02-20 11:41:30 +01:00
TobiGr
ea120a4637
remove print stacktrace
...
use interface
2021-02-20 00:23:45 +01:00
XiangRongLin
adf9d7d10f
Add reason field to MockOnly
...
This enforces developers to document why a test is skipped
2021-02-19 18:54:18 +01:00
XiangRongLin
ea52030613
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-02-19 18:54:17 +01:00
TobiGr
02920fafa8
Add isUploaderVerified()
2021-02-19 15:51:23 +01:00
TobiGr
91e9309486
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-02-19 13:37:45 +01:00
TobiGr
fa61b864f2
Code improvements
2021-02-19 13:37:24 +01:00
TobiGr
98268e351c
Move radio URL check into a function
2021-02-19 13:35:44 +01:00
FireMasterK
959d5d7db7
Add dependabot.
2021-02-19 11:52:56 +05:30
bopol
27a20e41cd
Merge pull request #550 from FireMasterK/add-more-data
...
Add more data.
2021-02-18 20:28:27 +01:00
FireMasterK
5333d8a98b
Add more data.
2021-02-19 00:55:03 +05:30
FireMasterK
88890d81a5
Update gradle wrapper.
2021-02-18 21:15:12 +05:30
bopol
9c1a7f7df6
Merge pull request #521 from B0pol/verified
...
Add uploader verified by service extraction
2021-02-18 16:39:53 +01:00
bopol
1a322ad8ed
Add uploader verified by service extraction
2021-02-18 16:36:40 +01:00
bopol
db253e202b
Merge pull request #535 from TeamPiped/adaptive-parsing
...
Adaptive Stream parsing
2021-02-18 15:55:27 +01:00
FireMasterK
c24afa2cbb
Permalink URL.
2021-02-18 20:16:53 +05:30
FireMasterK
8d54401233
Add fps field as well.
2021-02-18 20:16:53 +05:30
FireMasterK
4d096be14a
Revert "Update gradle wrapper."
...
This reverts commit 591856e3a11998c4b8ba87b5e361e2b9556d760f.
2021-02-18 20:16:53 +05:30
FireMasterK
5b70645d85
Add data to muxed streams aswell.
2021-02-18 20:16:53 +05:30
FireMasterK
e975d33fbe
Change jdoc / comments.
2021-02-18 20:16:53 +05:30
FireMasterK
f31b2a68fd
use getters/setters.
2021-02-18 20:16:53 +05:30
FireMasterK
d4945ac55c
Constructor cleanup.
2021-02-18 20:16:52 +05:30
