Not working for some userscripts

[plateaukao]

Hi, @warren-bank

I installed userscript (https://greasyfork.org/en/scripts/457196-immersive-translate/code), and trying to make it work.
In the Chrome debug console, it will complain:

Uncaught ReferenceError: unsafeWindow is not defined
    at <anonymous>:1:16
    at <anonymous>:1:405

I did see it's declared in

Android-WebMonkey/android-studio-project/libs/webview-gm-lib/src/main/java/at/pardus/android/webview/gm/run/WebViewClientGm.java

Line 42 in 76a7901

private static final String JSUNSAFEWINDOW = "var unsafeWindow = (function() { var el = document.createElement('p'); el.setAttribute('onclick', 'return window;'); return el.onclick(); }()); window.wrappedJSObject = unsafeWindow;\n";

Due to my lack knowledge of javascript, I am not sure whether it's because some GM APIs are missing, so it can't work properly. Could you help to look at the script, and give me some hint where this might go wrong?

Thank you.

[warren-bank]

Are you using adb with a remote dubug session to view the console.log output?

The userscript that you linked to..

• doesn't once reference the variable: unsafeWindow
  • so that log message is (almost certainly) coming from somewhere else
• does a ton of communication between the top window and iframes
  • I don't know if the script assumes that it will also run in all iframes
  • if so, that's a point of failure.. since WebMonkey requires a workaround to get scripts to run in iframes
• appears to use browser detection
  • I didn't look closely enough to see what this is all about,
    but it could be another point of failure
• uses a lot of modern JS
  • so it should probably specify the minimum version of Chrome required to run its code
  • which (for WebMonkey) would translate to the minimum version of Android System WebView

In any case, I can't possibly support every random userscript found in the wild.
Though unlikely, you could ask its author to support its ability to run in WebMonkey.

[plateaukao]

@warren-bank

Yes, I checked the console log by using Chrome debugger tool on my Mac machine, while running WebMonkey in Android Studio.

Thanks for the information! I'll look further to see if I could find the root cause.

By the way, here's more information for your reference:

• The userscript I described in this issue is a great translation tool (paragraph by paragraph) and widely used; it not only comes in format of userscript, but also provides extensions to chrome, firefox, and Apple Safari.
• I happen to find that another small browser (called xBrowser) which supports GreaseMonkey interface and it's based on Android's System WebView too; and it can run the script well on the same device I verified. So, I think it's possible to support the script without modifying it. Too bad that xBrowser is not open source, I could not leverage from it.

WebView GM library is all I could find on github, and your repo is one of the few repos that tries to enhance it. I'll look further when I have time. If I found anything close to the root cause, and it's something WebMonkey could be modified, I will let you know too.

[warren-bank]

it's serendipitous that you mentioned XBrowser..
I only just heard about it a few days ago..
because it's listed on the Greasy Fork site as a supported userscript manager for Android..
and I've been meaning to take a peak at its (decompiled) code.

I'll also give this particular userscript a look..
to see which API it uses that isn't supported..
but no promises when that will be.

[plateaukao]

@warren-bank
Thanks. Any more information from you would be great!

In fact, several weeks ago, xBrowser still couldn't load the script correctly either. It had similar behavior as WebMonkey too: seeing the pink "immersive translate" button on right side of the screen; when clicking translate, all text will be attached a loading cursor, but translation result never being shown.

And... one day, the script works on xBrowser! I thought he might reference some open source too. That's why I started searching for similar Open Source solution for Android WebView, because xBrowser proved it's possible on generic WebView too (not other heavy-size chromium clones).

[warren-bank]

quick update..
I see a few different things going on..
which I will (mostly) fix very soon..

1. missing GM_xyz APIs
   • this userscript uses one that is currently not defined
     • it's only a helper function, but being undefined would throw an error
     • I'll provide an implementation.. that simply encapsulates a few standard DOM calls
   • comparing the full list of APIs provided by Tampermonkey vs. Violentmonkey
     • each has a few methods that are unique to their own implementation
     • none are particularly useful or important
     • I'll provide no-op stubs.. just so they aren't undefined
2. Content-Security-Policy
   • this is a big one
   • this is the reason that my test of the userscript on a page of Google search results didn't do anything
   • the workaround that I use is..
     • to run PCAPdroid with its PCAPdroid mitm addon
     • to configure PCAPdroid to start mitmproxy with a custom script to remove CSP response headers
     • this issue has a long discussion on the topic of using such custom scripts
     • this repo has a collection of other such custom scripts
     • huge caveat:
       • PCAPdroid-mitm doesn't currently request permission to read from or write to external storage
       • all of my testing is done on a modified APK that adds these permissions to its manifest
       • hopefully, the next version released will include these permissions
3. mitmproxy certificate authority is untrusted
   • mitmproxy generates a unique CA for each device
   • PCAPdroid-mitm provides a really nice wizard to walk a user through installing their mitmproxy CA on the device
   • Android 7.0+ throws a wrench in this..
     • CAs added by the user are untrusted by default
     • each app needs to opt-in to trusting these CAs
   • with respect to WebMonkey and your userscript..
     • it doesn't prevent browsing web pages
     • it does, however, prevent GM_xmlhttpRequest from working..
       with: java.security.cert.CertPathValidatorException: Trust anchor for certification path not found.

I can confirm that early tests prove that:

• with CSP disabled and APIs implemented
  • the userscript adds a button to the bottom left on pages of Google search results
  • clicking this button displays a floating form with some options, and a button to start translation
  • clicking the button to start translation displays many spinning icons throughout the document
  • because XHR is failing, the translation results are never received..

so.. to summarize:

• I'll add JS implementations for missing GM_xyz API methods
• I'll configure WebMonkey to trust CAs added by the user (ex: mitmproxy)
• PCAPdroid-mitm will be needed to circumvent CSP
  • this will allow the userscript to run on all pages
  • the userscript will have all of its APIs defined, and XHR to the translation services will obtain successful replies

hopefully..

• this is all that'll be needed for the userscript to be fully functional

[plateaukao]

Wow, thanks for the thorough update. It looks like a lot of effort to make it work.
About Content-Security-Policy, does it mean that xBrowser also implements something like PCAPdroid inside?

[warren-bank]

actually.. I was wrong about Google requiring CSP to be disabled for userscripts to run on their domain.

I don't actually know why it made a difference in my earlier testing..
but I can confirm now, after making a bunch of updates..
that it runs fine on pages of Google search results without using mitmproxy.

well..
by "fine", I mean that the userscript is running..
but not that the translations are successfully being added to the DOM (yet).

at the moment, I'm not sure why the translations aren't being displayed..
all of the APIs are working properly..
no errors in the browser console..
adding debug log messages show that all networking is returning 200 OK..
I'll need to keep adding some more log messages until I can pinpoint where something isn't working as it should be.

..it's actually very possible that it's simply that my WebView needs to be updated;
this script is using JS features that were added very recently.

[warren-bank]

PS.. regarding XBrowser..
I haven't looked at their app yet..
but what a lot of apps that use WebView like to do..
is to intercept all networking requests..
and use an HTTP client library to proxy the request and return the response..
which allows the app much more control over networking..
such as modifying headers in either direction.

[warren-bank]

update..
I've found 2x API calls that weren't behaving as they should..
and have subsequently been fixed:

1. GM_addElement('style', {textContent: css})
   • my implementation of this method uses setAttribute rather than element properties (which may or may not exist)
   • I needed to add a list of special attributes that can only be handled through properties (ex: textContent)
2. GM_setValue(key = 'some string', value = {some: 'object'})
   • my implementation assumed that value would always be a string
   • I needed to add JSON serialization on the set side, and JSON deserialization on the get side

now..
this userscript appear to be doing something..
but I'm not quite sure if it's working 100% properly..
I'll need to compare its behavior:

• in: desktop OS + browser (ex: Firefox or Chrome) + Tampermonkey
• vs: Android WebMonkey

[warren-bank]

one more comment..
about unusual behavior that I've observed..
pertaining to the floating pink button

• there is a block of css that styles this button and sets its position.. in addition to other things
• in my earliest tests (when this issue was opened and I commented about CSP)
  • this CSS was not being added to the DOM
  • the button was unstyled and positioned at the bottom of the page
• in my more recent tests.. about 2 days ago
  • this CSS was being added to the DOM
  • the button looked as it should..
    pink and floating at the middle of the page on the right-hand side
• in my most recent tests.. yesterday
  • this CSS was again.. not being added to the DOM
  • I don't believe that any of my changes to the API code would have any effect
  • it's just odd behavior..

update:
this behavior is now fixed..
it did trace back to a change I made to an API method's implementation..
a validation check in GM_addElement was preventing inserting new element to a shadow DOM..
it's now adjusted to allow for this usage.

[warren-bank]

observations..

• using the userscript as-is:
  • releases
  • v0.6.21
    • the most recent release at the time of this comment
  • GM_xmlhttpRequest is a bottleneck
    • I'm not sure why only some of the XHR translation requests receive a response
    • maybe it's that I'm using a very low-end device for testing..
      and there are too many threads being started concurrently
  • it works.. but not perfectly
• making a tiny edit to the userscript:
  • prepending to the userscript (immediately following the metadata block):

      try {
        GM_xmlhttpRequest = undefined;
        GM.xmlHttpRequest = undefined;
      }
      catch(e) {}

  • causes the userscript to use the browser's JS window.fetch API
  • it works perfectly

more observations:

• rather than prepending the previous bit of code..
  • if this bit of code is used instead:

      try {
        window.GM_fetch = GM_fetch;
      }
      catch(e) {}

  • it still works perfectly
    • but it's now using GM_xmlhttpRequest.. under the hood
  • the purpose for this is..
    • to allow the userscript to detect a "native" implementation of this method,
      which prevent it from injecting its own global polyfill
    • the userscript will then use my implementation, rather than its own polyfill
    • the weird thing is.. my implementation is identical to theirs
      • I liked it so much that I copied it..
        from v0.3.17, which isn't minified
      • which appears to have originally been copied from here
  • I have no idea why mine works but theirs doesn't

[warren-bank]

debugging..

• search in userscript for text: 12e4
  • it only occurs once
  • for example, in v0.6.21:

      ;let r=e.fetchPolyfill||fetch,o=12e4;

• append some debug logging:

      ;let r=e.fetchPolyfill||fetch,o=12e4;
      window.alert(`(typeof GM_xmlhttpRequest) = ${(typeof GM_xmlhttpRequest)}`);
      window.alert(`(typeof GM_fetch) = ${(typeof GM_fetch)}`);
      window.alert(`(typeof e.fetchPolyfill) = ${(typeof e.fetchPolyfill)}`);
      window.alert(`(typeof window.GM_fetch) = ${(typeof window.GM_fetch)}`);
      window.alert(`(e.fetchPolyfill === window.GM_fetch) = ${(e.fetchPolyfill === window.GM_fetch)}`);
      window.alert(`(GM_fetch === window.GM_fetch) = ${(GM_fetch === window.GM_fetch)}`);

• without prepending any changes at the start of the userscript:

    (typeof GM_xmlhttpRequest) = function
    (typeof GM_fetch) = function
    (typeof e.fetchPolyfill) = function
    (typeof window.GM_fetch) = function
    (e.fetchPolyfill === window.GM_fetch) = true
    (GM_fetch === window.GM_fetch) = false
  

• with prepending the change to undefine GM_xmlhttpRequest:

    (typeof GM_xmlhttpRequest) = undefined
    (typeof GM_fetch) = function
    (typeof e.fetchPolyfill) = undefined
    (typeof window.GM_fetch) = undefined
    (e.fetchPolyfill === window.GM_fetch) = true
    (GM_fetch === window.GM_fetch) = false
  

• with prepending the change to globally define GM_fetch:

    (typeof GM_xmlhttpRequest) = function
    (typeof GM_fetch) = function
    (typeof e.fetchPolyfill) = function
    (typeof window.GM_fetch) = function
    (e.fetchPolyfill === window.GM_fetch) = true
    (GM_fetch === window.GM_fetch) = true
  

[warren-bank]

I released a new major version.
v1.x doesn't implement any sandbox.
v2.x does..

• the implementation is unique..
  • typically userscripts run in browser extensions.. which use a sandbox environment that is provided by the browser to extensions
  • I'm using an ES6 Proxy to expose window properties.. but store new properties in a separate object (ie: its sandbox)
• it uses the same metadata cues as the other userscript engines.. to determine when to use it.. and when to turn it off
  • @unwrap: no closure and no sandbox
  • @grant none: has closure but no sandbox
  • otherwise: has closure and sandbox

long story short.. your userscript runs perfectly in v2.0.1.

I appreciate your asking me to look into this.
Although my initial reaction was that I don't support other people's scripts..
this particular userscript is exceptionally well written.. and crazy complicated..
and seemingly has full coverage of all the APIs.. using them in very advanced ways.

I never took a very close look at the API implementations provided by the upstream WebView GM library..
and out of respect to them, I didn't want to make any changes to it..
but this little exercise threw all of that out the window..
and involved making significant changes..
which, as a result, makes WebMonkey much better..
so thanks again for giving me the needed push.

[plateaukao]

Thank you for your great work and contribution to try to make the script work!!!
That sounds wonderful!
I will have a try for version v2.0.1.