[YouTube] Add support for poTokens #11955
Conversation
|
Now the PR builds fine based on TeamNewPipe/NewPipeExtractor#1247, you can download the APK which uses poTokens! Let us know if you notice any issues. |
| private val TAG = PoTokenWebView::class.simpleName | ||
| private const val GOOGLE_API_KEY = "AIzaSyDyT5W0Jh49F30Pqqtyfdf7pDLFKLJoAnw" | ||
| private const val REQUEST_KEY = "O43z0dpjhgX20SCx4KAo" | ||
| private const val USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.3" |
There was a problem hiding this comment.
Could this be Firefox ESR like in DownloaderImpl?
There was a problem hiding this comment.
No, for some reason it does not work with the Firefox user agent. It would work with the curl user agent though, I don't know why...
There was a problem hiding this comment.
Perhaps checks on Firefox are stricter.
Just a deduction of what may be the cause:
Some webservers may also check the "case" of headers (User-Agent vs user-agent; this also apply to other headers) and even the order in which the headers are sent to the server may matter.
The WebView is more likely to have an implementation similar to Chrome, so it will be more likely to fail with Firefox User-Agent since the implementation differs completely.
|
Would it be possible to move the po token implementation to a library? Currently this is in NewPipe (the app repo) which makes it inaccessible by other apps which also have the need for po tokens. This will lead to a lot of duplicate code because it needs to be implement over and over again for each YT client app. Would be cool if this can be maintained in just one place (and multiple apps could benefit like it is already the case with NewPipeExtractor). |
You can recreate this PR in your own application. This simply connects to the extractor to support the Potoken stream. You will need to do this separately in your application. It should have been like this. |
my point was this would be inefficient If you want to implement this over and over again for each app - sure, go ahead. Keep in mind that this will likely not be "done" after the initial implementation. You will have to update the implementation in many places again. And again. And again... If this was implemented in just one place as a library it would be easier for more developers to share efforts. |
app/src/main/assets/po_token.html
Outdated
| // an asynchronous function runs in the background and it will eventually call | ||
| // `vmFunctionsCallback`, however we need to manually tell JavaScript to pass | ||
| // control to the things running in the background by interrupting this async | ||
| // function in any way, e.g. with a delay of 1ms. The loop is most probably not | ||
| // needed but is there just because. | ||
| for (let i = 0; i < 10000 && !this.vmFunctions.asyncSnapshotFunction; ++i) { | ||
| await new Promise(f => setTimeout(f, 1)) | ||
| } |
There was a problem hiding this comment.
I … don’t think this is how async works. The timeout is just gonna be scheduled on a new task, but the code before the loop still runs on a microtask on the previous task.
There was a problem hiding this comment.
Yes, but this.vm.a seems to start a standalone task in the background or something like that, and we need to explicitly pass control back to the event loop by pausing this async execution, for the background task to finish executing.
There was a problem hiding this comment.
The loop actually executes only once as far as I know, I still put a loop because you never know
|
Can there be an architecture overview of this somewhere? From a skim of the code I don’t get any idea of what problem this solves or how the solution is structured. |
Let me know which places are not documented enough. |
|
@Stypox I think it would be good to include this documentation into the source code somewhere, maybe in the interface module. |
|
So that people who want to understand the code later don’t have to find this PR and looks through lots of issues first |
AudricV
added
bug
AudricV
changed the title
|
As a general comment, I don’t think it’s wise to add interfaces before even having need of a second implementation, it just makes the code more indirect and harder to read than necessary. |
app/build.gradle
Outdated
| implementation 'com.github.TeamNewPipe:NewPipeExtractor:v0.24.4' | ||
| implementation 'com.github.FireMasterK:NewPipeExtractor:d2cbd09089e8af933738f98b671ad58236a79d6e' |
There was a problem hiding this comment.
Oh I missed that the NewPipeExtractor still points to an unofficial repo, that needs to be fixed before merge
Yes I totally agree with you, but when I started working on this I hoped I would be able to create two implementations |
This will be tried only once, and afterwards an error will be thrown
This comment was marked as outdated.
This comment was marked as outdated.
This prevents non-abilities to fetch BotGuard challenge and send its result with the jnn-pa.googleapis.com domain (domain block like done on Pi-hole lists or DNS servers). That's what the official website uses to send the challenge execution result, however it uses InnerTube to fetch the challenge. Embeds still use the jnn-pa.googleapis.com domain. Also rename the makeJnnPaGoogleapisRequest method appropriately.
Implements support for locally generating PoTokens using the device webview. This is a direct port of TeamNewPipe/NewPipe#11955 to native kotlin.
globalThis was introduced only on newer versions of JS
|
@spvkgn thanks for reporting, please try the latest APK, I should have fixed that. Your WebView seems to use a really old version of JavaScript that doesn't have the |
Some old Android devices have a broken WebView implementation, that can't execute the poToken code. This is now detected and the getWebClientPoToken return null instead of throwing an error in such a case, to allow the extractor to try to extract the video data even without a poToken.
|
@Stypox Thanks, I checked and it works now. |
Implements support for locally generating PoTokens using the device webview. This is a direct port of TeamNewPipe/NewPipe#11955 to native Kotlin. Closes: libre-tube#7065
app/src/main/java/org/schabi/newpipe/util/potoken/PoTokenWebView.kt
Outdated
Show resolved
Hide resolved
Implements support for locally generating PoTokens using the device webview. This is a direct port of TeamNewPipe/NewPipe#11955 to native Kotlin. Closes: libre-tube#7065
Implements support for locally generating PoTokens using the device webview. This is a direct port of TeamNewPipe/NewPipe#11955 to native Kotlin. Closes: libre-tube#7065
|
I reworked the JavaScript code to not use any new JS features to achieve compatibility with as many WebView implementations as possible (including the API 21 emulator I was testing this on, which previously did not work). To do this, I moved all conversion functions to Kotlin (since those do not need to run in JS, so it's better if they are written in Kotlin where errors are caught at compile time and/or with nice stacktraces) and used While at it, I added more ways to detect if the WebView is not able to actually run the code, by making it so that the poToken generator returns I tested on API 21 emulated, API 34 emulated, and Android 14 on-device, and it worked everywhere (with playable videos). Please test this ASAP, and report any issues. I will make a release tomorrow morning.
Thank you, solved. |
|
Everything works great!😊🤗 |
|
Tested on a Great, thanks 👍 |
Using commit 9f83b385a since JitPack is buggy...
|
works great on A15! |
|
I am merging this now since we need to do a new release. This PR probably needs some refinement, and will especially need to adapt to future extractor changes (because the current extractor API around poTokens is going to be changed), but works well enough. Let's hope all poToken implementations behave well! |
|
|
Working great on redmi note 8 pro 🙏 |
What is it?
Description of the changes in your PR
General information about poTokens and about this PR structure:
Createto obtain the VM code,GenerateITto obtain the integrity token after running the VM code. See the README here for the detailed steps.PoTokenGeneratoris the base class for all poToken generators. It has a factory method that allows asynchronously obtaining a new instance of aPoTokenGenerator, and then two methods to generate a poToken given a specific identifier, and a method to check if the integrity token has expired.PoTokenWebViewis currently the only implementation ofPoTokenGenerator, but we might want to add other implementations in the future, e.g. ones that do not rely on WebView.PoTokenProviderImplimplements the extractor interface and is supposed to take care of possibly multiplePoTokenGenerators (although right now there is only one based on WebView). It takes care of retrying in case of problems, recreates a newPoTokenGeneratorif the current one expired, and finally returns aPoTokenResult. APoTokenResultcontains two poTokens: one for the specific requested video id (used to fetch the player), and another that can be generated only once as the first thing and is specific to a visitor data (used in streaming urls).TODO:
Obtaining a poToken with something like HtmlUnitnot doable unfortunatelyyoutube.comlike the desktop website does, this avoids requirement of thejnn-pa.googleapis.comdomain - partially done, InnerTube attestation endpoint is not used yetYou can test whether the poTokens generated work also using the latest yt-dlp commit from their git repo (older commits won't work!), this way (take PLAYER_POT, STREAMING_POT and VISITOR_DATA from logcat):
Fixes the following issue(s)
Relies on the following changes
APK testing
The APK can be found by going to the "Checks" tab below the title. On the left pane, click on "CI", scroll down to "artifacts" and click "app" to download the zip file which contains the debug APK of this PR. You can find more info and a video demonstration on this wiki page.
Due diligence