Mauricio Colli
|
00d1ed439b
|
[YouTube] Fix channel extraction when redirects are in the response
Some redirects were embed directly into the response as instructions
for the page, instead of the usual http redirects.
|
2020-03-14 02:39:48 -03:00 |
|
bopol
|
66518ec444
|
check wether mediaccc streams url are secure
|
2020-03-13 20:07:36 +01:00 |
|
bopol
|
0cd5e05b7b
|
MediaCCCLH: use substring instead of replace; improve a bit tests, return Collections.emptyList(); instead of null where it's annotated @NonNull
|
2020-03-13 20:00:49 +01:00 |
|
bopol
|
f742a6bd3e
|
code optimization
|
2020-03-13 20:00:49 +01:00 |
|
bopol
|
9701c7d800
|
invidious shortened links
|
2020-03-13 20:00:49 +01:00 |
|
bopol
|
e8e535b815
|
mediaccc: update linkhandlers & refactor tests
|
2020-03-13 20:00:49 +01:00 |
|
Mauricio Colli
|
70abd57852
|
[YouTube] Cache commonly used stream type result
|
2020-03-07 16:48:32 -03:00 |
|
Mauricio Colli
|
1ef706f567
|
[YouTube] Detect LIVE videos in the trending page
|
2020-03-07 16:48:31 -03:00 |
|
Xiang Rong Lin
|
cc8fb486ee
|
Handle youtube stream urls in "vnd.youtube:videoId" format.
The official YouTube app accept intents in this format, which causes other developers to use it.
|
2020-03-02 20:40:33 +01:00 |
|
Mauricio Colli
|
6aa4d59b91
|
[PeerTube] Update video used in age limit test (the previous was 404)
|
2020-03-01 10:29:30 -03:00 |
|
Mauricio Colli
|
cfc278317d
|
[YouTube] Fail-fast if status exist and is anything other than "OK"
|
2020-03-01 10:29:29 -03:00 |
|
Mauricio Colli
|
90ae5fbea2
|
Improve kiosk tests across services
|
2020-03-01 12:57:18 +01:00 |
|
Mauricio Colli
|
e590417cc4
|
Test if services recognizes their own items urls
|
2020-03-01 12:57:18 +01:00 |
|
TobiGr
|
35252235b0
|
Go through all badges when checking if a StreamInfoItem is a live stream
|
2020-03-01 12:52:37 +01:00 |
|
Mauricio Colli
|
5686a6f562
|
[YouTube] Detect when a stream is deleted or doesn't exist
Added a test case as well.
|
2020-03-01 12:49:13 +01:00 |
|
Mauricio Colli
|
e65333c3ce
|
[YouTube] Detect deleted/nonexistent/invalid channels and playlists
- Added tests for these cases.
|
2020-03-01 12:49:13 +01:00 |
|
Mauricio Colli
|
98e359438a
|
[YouTube] Detect simple 404s in the standard fetch method
|
2020-03-01 12:49:13 +01:00 |
|
Mauricio Colli
|
408f042127
|
[YouTube] Fix bug when url isn't present in the browseEndpoint object
|
2020-03-01 12:49:13 +01:00 |
|
Mauricio Colli
|
342bdbb852
|
[YouTube] Avoid crashing by letting exceptions bubble up
|
2020-03-01 12:49:13 +01:00 |
|
Mauricio Colli
|
e9644e6216
|
[YouTube] Handle video premiere's date and duration
|
2020-03-01 12:49:13 +01:00 |
|
Mauricio Colli
|
2a470ac4f7
|
[YouTube] Handle videos with no views or with "Recommended to you" text
|
2020-03-01 12:49:13 +01:00 |
|
Mauricio Colli
|
da2c562eb0
|
[YouTube] Take into account videos that have their views hidden
Also remove catch because it may hide bugs when the page changes.
|
2020-03-01 12:49:13 +01:00 |
|
Mauricio Colli
|
f66c20de54
|
Ignore null-keyed entries when iterating through the response headers
|
2020-03-01 12:49:13 +01:00 |
|
Mauricio Colli
|
5edd774fc4
|
Add latest url to the response to make detection of a redirect possible
Will be latest one in this commit because there's need to check the
history of redirects as of now.
|
2020-03-01 12:49:13 +01:00 |
|
Mauricio Colli
|
3441946bea
|
Make test downloader return a response instead of throwing an exception
The test implementation was throwing an exception instead of just
returning the response and letting the caller handle it.
|
2020-03-01 12:49:13 +01:00 |
|
wb9688
|
df219a9453
|
Don't use StandardCharsets class
|
2020-03-01 09:41:06 +01:00 |
|
wb9688
|
2807079531
|
Catch NumberFormatException instead of checking whether viewCount equals "Recommended for you"
|
2020-02-29 21:28:38 +01:00 |
|
wb9688
|
3e4d2fd64e
|
Improve testChannelUrl()
|
2020-02-29 18:20:35 +01:00 |
|
wb9688
|
bfe3eb1409
|
Use suggested try-if code style
|
2020-02-29 17:18:50 +01:00 |
|
wb9688
|
a6a63e9570
|
Let isHardcodedClientVersionValid() throw IOException
|
2020-02-29 17:01:36 +01:00 |
|
wb9688
|
45df8248bf
|
Use Localization in getJsonResponse()
|
2020-02-29 16:55:07 +01:00 |
|
wb9688
|
157055fbfd
|
Create getJsonResponse() function
|
2020-02-29 16:42:04 +01:00 |
|
wb9688
|
4f406512c2
|
Use YoutubeChannelLinkHandlerFactory
|
2020-02-29 16:26:28 +01:00 |
|
wb9688
|
4ff655352d
|
Reimplement getAgeLimit()
|
2020-02-28 17:14:26 +01:00 |
|
wb9688
|
6ba03e542b
|
Cache getVideoTab()
|
2020-02-28 17:05:31 +01:00 |
|
wb9688
|
8bfbae904d
|
Cache getVideo(Primary|Secondary)InfoRenderer()
|
2020-02-28 17:03:21 +01:00 |
|
wb9688
|
0b940f5cf4
|
Use "getAgeLimit() != NO_AGE_LIMIT" instead of "getAgeLimit() == 18"
|
2020-02-28 16:40:50 +01:00 |
|
wb9688
|
beb07a4852
|
Add test to check whether hardcoded client version is still valid
|
2020-02-28 16:36:14 +01:00 |
|
wb9688
|
b89b6b1db2
|
Check if hard-coded client version is valid in separate function
|
2020-02-28 16:35:24 +01:00 |
|
wb9688
|
7898ba230d
|
Use pbj in YoutubeStreamExtractor
|
2020-02-28 15:17:47 +01:00 |
|
wb9688
|
612ec06839
|
Actually fix thumbnail URLs starting with //
|
2020-02-28 09:36:33 +01:00 |
|
wb9688
|
f946310af5
|
Fix search queries with a promotedSparklesTextSearchRenderer returning 0 results
|
2020-02-28 08:57:44 +01:00 |
|
wb9688
|
94a2d5be46
|
Fix thumbnail URLs starting with //
|
2020-02-27 19:08:46 +01:00 |
|
wb9688
|
838d834ce2
|
Get Uploads view of Videos tab
|
2020-02-27 18:54:27 +01:00 |
|
wb9688
|
365b0329f3
|
Implement getTextFromObject() function
|
2020-02-27 17:39:23 +01:00 |
|
wb9688
|
0798bdd5cd
|
Fix getSubscriberCount() for 0 subscribers
|
2020-02-27 16:26:22 +01:00 |
|
wb9688
|
8ebd971648
|
Fix channel descriptions consisting of multiple parts
|
2020-02-27 10:06:35 +01:00 |
|
wb9688
|
880b951088
|
Fix getUploaderAvatarUrl() in YoutubeStreamExtractor
|
2020-02-27 09:53:59 +01:00 |
|
wb9688
|
8aa560bef4
|
Fix video titles consisting of multiple parts due to @
|
2020-02-27 09:42:22 +01:00 |
|
wb9688
|
951ed9f0d8
|
Don't call getUploaderInfo() in onFetchPage() for playlists
This way YouTube Music playlists will keep working in NewPipe
|
2020-02-27 09:27:56 +01:00 |
|
wb9688
|
2399d46ab0
|
Fix YoutubeChannelExtractor for channels without videos
|
2020-02-27 09:21:25 +01:00 |
|
wb9688
|
cdbc751b87
|
Fix getUploaderUrl() in YoutubeStreamInfoItemExtractor
|
2020-02-27 09:05:41 +01:00 |
|
wb9688
|
db305408e9
|
Get all expandedShelfContentsRenderers in YoutubeTrendingExtractor
|
2020-02-26 20:37:20 +01:00 |
|
wb9688
|
a2d8489e16
|
Parse timestamps in description
|
2020-02-26 20:27:30 +01:00 |
|
wb9688
|
8d503407ef
|
Parse browseEndpoint and watchEndpoint in descriptions
|
2020-02-26 19:37:58 +01:00 |
|
wb9688
|
8dc39d517a
|
Fix tests
Except testGetFullLinksInDescription() in DescriptionTestUnboxing, since that requires other changes first.
|
2020-02-26 17:31:01 +01:00 |
|
wb9688
|
88282761f4
|
Fix YoutubeStreamExtractor not being able to getAgeLimit() in onFetchPage()
|
2020-02-26 16:19:42 +01:00 |
|
wb9688
|
bb3a3d70bf
|
Use pbj in YoutubeTrendingExtractor
|
2020-02-26 16:10:24 +01:00 |
|
wb9688
|
1c3a4e710b
|
Check if there are no related streams in getAgeLimit()
|
2020-02-26 16:03:06 +01:00 |
|
wb9688
|
985c3ec877
|
Improve getClientVersion()
|
2020-02-26 15:27:50 +01:00 |
|
wb9688
|
15afbea3e1
|
Use pbj in YoutubePlaylistExtractor
|
2020-02-26 15:27:50 +01:00 |
|
wb9688
|
89a41a7f69
|
Use pbj in YoutubeChannelExtractor
|
2020-02-26 15:27:50 +01:00 |
|
wb9688
|
0973263aab
|
Use pbj in YoutubeSearchExtractor
|
2020-02-26 15:27:46 +01:00 |
|
TobiGr
|
fc465c8bf7
|
Fix getUploadDate() for premiered videos with a given date
Detect publishDate and uploadDate
|
2020-02-25 22:54:37 +01:00 |
|
TobiGr
|
96285e0815
|
Fix getUploadDate() for premiered videos with a given date
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
5a35300a5e
|
Fix parsing upload date for premiered streams
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
729fd2eaee
|
Fix parsing description for navigationEndpoint without urlEndpoint
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
26fb44595f
|
Fix parsing description
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
0fff03038e
|
Get description with correct links
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
5d08c34322
|
Fix parsing upload date in YoutubeStreamExtractor
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
bbe1a3cd62
|
Use getVideo(Primary|Secondary)InfoRenderer()
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
8e6be880d7
|
Simplify getId() in YoutubeChannelExtractor
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
ecfc27a92c
|
Implement fallback methods for getDescription() and getTextualUploadDate() in YouTubeStreamExtractor
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
eed29ea734
|
Return null for related streams at age restricted videos
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
9efcc61ca6
|
Adapt some tests to match the new URLs
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
f39603f6ef
|
Implement getUploadDate() in YouTubeStreamInfoItemExtractor
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
26ea3dceb6
|
Fix getViewCount() in YoutubeStreamExtractor
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
76d54abdbf
|
Reimplement more methods in YoutubeStreamExtractor
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
f13c0288cc
|
Reimplement some methods in YoutubeStreamExtractor
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
02b59903fa
|
Remove useless code
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
dd4dd849dc
|
Get client version dynamically in YouTubeSearchExtractor
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
583e9c1571
|
Fix getPage()
Someone should check if there are unstaged changes before committing...
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
5842b9ad37
|
Add getClientVersion() and HARDCODED_CLIENT_VERSION to YouTubeParsingHelper
Prefer hardcoded client version above the current one when making requests to retrieve the same JSON structure for each request.
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
5d883d100c
|
Implement pagination in YoutubeSearchExtractor
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
c0a8e01889
|
Implement pagination in YoutubePlaylistExtractor
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
4039409820
|
Fix getLikeCount() for certain streams
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
627ab2459f
|
Uncomment fetchPage()
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
2dfa2187ff
|
Implement pagination in YoutubeChannelExtractor
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
a38ab9b791
|
Return null if channel has no banner
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
1eca63089b
|
Count private and deleted videos as ad
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
c7360e4a46
|
Fix getThumbnailUrl() in YoutubeChannelInfoItemExtractor
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
51fb26625a
|
Implement getSubscriberCount() in YoutubeChannelInfoItemExtractor
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
6d20b2b1d0
|
Fix NPE on collecting page items
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
4462cbe3f1
|
Reimplement YoutubePlaylistInfoItemExtractor
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
8aea4d445b
|
Reimplement YoutubeChannelInfoItemExtractor
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
3187116a63
|
Handle premium videos
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
7442f91aa6
|
Don't assume which channel tab is videos
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
316fe0109d
|
Fix first playlist page
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
4e57e589ce
|
Fix first channel page
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
5816202cc7
|
Move getInitialData() method to YouTubeParsingHelper
Rename ytInitialData to initialData
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
38aabc6aca
|
Use plain YoutubeStreamInfoItemExtractor in YoutubeTrendingExtractor
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
957db062ef
|
Fix getting uploader for videoRenderer
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
b88188d419
|
Move stuff from extractVideoPreviewInfo() into YoutubeStreamInfoItemExtractor and partially fix search
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
af49b3c487
|
Fix live streams in extractVideoPreviewInfo()
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
f33f9466ce
|
Fix testDescription() in ChannelExtractor
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
10e9c16d8c
|
Fix LiveStream view count
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
127c4d5893
|
Fix getNextStream() in YoutubeStreamExtractor
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
7bfc0e62c6
|
Fix getRelatedStreams() in YoutubeStreamExtractor
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
12166ca60c
|
[StreamExtractor] Add fallback method for uploader avatar url
|
2020-02-25 21:51:30 +01:00 |
|
wb9688
|
3ec422a542
|
Fix like/dislike parsing
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
a34e5c18e1
|
Trending StreamInfoItemCollecetor
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
b705515da0
|
Adapt most playlist fields
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
21253abd9b
|
Use Firefox ESR User-Agent: 68
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
655c999795
|
Detect disabled subriber count correctly
Fix parsing of kiosk name
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
7dcc9f159b
|
Stream: fix uploader avatar url
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
be3e20e263
|
Adapt most channel changes
|
2020-02-25 21:51:30 +01:00 |
|
TobiGr
|
39bf1ff1eb
|
Add ytInitialData regex
|
2020-02-25 21:51:30 +01:00 |
|
Tobias Groza
|
a129c65c63
|
more improvements of comments
Co-Authored-By: Stypox <stypox@pm.me>
|
2020-02-16 23:59:31 +01:00 |
|
TobiGr
|
0a4115286b
|
Apply one suggestion
|
2020-02-15 13:36:54 +01:00 |
|
Tobias Groza
|
6822574588
|
Fix typos
Co-Authored-By: B0pol <bopol@e.email>
|
2020-02-15 13:36:54 +01:00 |
|
TobiGr
|
030465b5d4
|
Improve code formatting and optimise imports
|
2020-02-15 13:36:54 +01:00 |
|
TobiGr
|
9112a106ea
|
[SoundCloud] Update client_id
|
2020-02-08 23:09:03 +01:00 |
|
TobiGr
|
008b6f505c
|
[PeerTube] Fix subscriber count in ChannelExtractorTest
|
2020-02-08 23:08:37 +01:00 |
|
TobiGr
|
0e33249c85
|
Fix SoundCloud description test
|
2020-02-08 22:20:20 +01:00 |
|
Fynn Godau
|
36a316eea5
|
Merge branch 'dev' into dev
|
2020-02-08 09:43:13 +01:00 |
|
bopol
|
11bcc78d9c
|
Description implements Serializable. fix NotSerializableException
|
2020-02-07 14:10:54 +01:00 |
|
bopol
|
0f8a7f9df2
|
fix testGetUploadDate for PeerTubeStreamExtractor
|
2020-02-07 13:33:50 +01:00 |
|
bopol
|
70a40e7388
|
Description: rm constructor by serviceId
|
2020-02-07 13:28:27 +01:00 |
|
bopol
|
26c65b2948
|
Create class Description
|
2020-02-06 23:35:46 +01:00 |
|
bopol
|
5756df8dc7
|
Use GMT as base time (actually fix upload date)
|
2020-02-06 22:33:48 +01:00 |
|
B0pol
|
b671a4ba18
|
Merge branch 'dev' into peertube
|
2020-02-01 18:18:23 +00:00 |
|
Mauricio Colli
|
26234a1c0f
|
Introduce FeedExtractor making fetching from dedicated feeds possible
YouTube, for example, has a dedicated feed which was built to be used
like this.
|
2020-01-28 02:44:18 -03:00 |
|
Mauricio Colli
|
be81f2945c
|
[YouTube] Use correct upload date flag in StreamExtractor
|
2020-01-28 02:44:17 -03:00 |
|
Mauricio Colli
|
5c0c35064c
|
Use clean url and id when creating a channel info
|
2020-01-28 02:44:16 -03:00 |
|
TobiGr
|
033a9fb8e3
|
Revert change which I accidentally committed while fixing the decryption
function in 7b72fd2a7d
|
2020-01-25 21:08:17 +01:00 |
|
B0pol
|
e392b6c68f
|
getLanguageInfo returns Locale instead of String
so that java can automatically translate with Locale.getDisplayLanguage(), instead of always having English name of the language
|
2020-01-25 14:12:38 +01:00 |
|
B0pol
|
341372c0d0
|
reindenting (ctrl alt l) on JsonUtils and PeertubeStreamExtractor
|
2020-01-24 20:16:24 +01:00 |
|
B0pol
|
b816e48216
|
Merge branch 'dev' into peertube
|
2020-01-24 20:10:47 +01:00 |
|
TobiGr
|
f8d117f6cd
|
Merge remote-tracking branch 'origin/master' into dev
|
2020-01-24 19:22:40 +01:00 |
|
TobiGr
|
7b72fd2a7d
|
[YouTube] Add new decryption function
|
2020-01-24 02:10:56 +01:00 |
|
TobiGr
|
bce87f3d01
|
Improve getDescriptionFuncName by removing catches and adding a loop
|
2020-01-24 02:10:38 +01:00 |
|
TobiGr
|
9b45c61103
|
Typos, comments and formatting
|
2020-01-24 02:10:18 +01:00 |
|
TobiGr
|
29a4cc78bf
|
Fix typo in decryption constant
|
2020-01-24 02:10:08 +01:00 |
|
B0pol
|
812c8e0ad2
|
authorName in comments now follow PeerTube website
https://peertube.cpy.re/videos/watch/d2a5ec78-5f85-4090-8ec5-dc1102e022ea
anonyme zirbeldrüse is his displayName, what was displayed in NewPipe.
but on the website, it's shown 777@mastodon.xyz
|
2020-01-24 00:22:05 +01:00 |
|
bopol
|
c790261eed
|
update test
|
2020-01-23 19:08:41 +01:00 |
|
bopol
|
a691d6d0fc
|
fix upload date: there was a one hour delay
on peertube.co.uk: https://i.imgur.com/8Pai1rb.png
on newpipe, before this commit: https://i.imgur.com/NIRbs4Z.png
on newpipe, with this commit: https://i.ibb.co/mhKKtRD/Screenshot-20200123-185422-New-Pipe-Debug.png
|
2020-01-23 19:02:40 +01:00 |
|
bopol
|
20da4750f8
|
empty support returns "", same for empty description
|
2020-01-23 14:37:14 +01:00 |
|
bopol
|
74439f692a
|
add extraction for support info and rename getLanguageInfo function
|
2020-01-23 14:19:22 +01:00 |
|
B0pol
|
1a15c0e750
|
agelimit now returns 18 if the video is marked as nsfw, 0 otherwise
+ created getBoolean method in JsonUtils.java
|
2020-01-23 04:42:54 +01:00 |
|
B0pol
|
bcfe7be4e6
|
Merge branch 'dev' into peertube
|
2020-01-22 19:19:41 +01:00 |
|
