All userscripts stop working in some websites like "https://www.zhihu.com"

[shunf4]

Enabling this script:

// ==UserScript==
// @name         My Script
// @version      0.1
// @match        *://*/*
// @grant        none
// ==/UserScript==

alert("Hello world!");

Navigating to https://www.zhihu.com , no alert window appears.

Nothing special found in logcat:

06-08 23:41:12.226 12563 12563 E PhoneWindow: Horizontal progress bar not located in current window decor
06-08 23:41:12.226 12563 12563 E PhoneWindow: Horizontal progress bar not located in current window decor
06-08 23:41:12.261 12563 12563 E PhoneWindow: Horizontal progress bar not located in current window decor
06-08 23:41:14.646 12563 12563 E PhoneWindow: Horizontal progress bar not located in current window decor
06-08 23:41:14.647 12563 12563 I c.a     : Running script "My Script, greasyfork.org: null" on https://www.zhihu.com/signin?next=%2F
06-08 23:41:27.150 12563 12563 I Choreographer: Skipped 60 frames!  The application may be doing too much work on its main thread.
06-08 23:41:27.186 12563 12586 I OpenGLRenderer: Davey! duration=707ms; Flags=0, IntendedVsync=13179795072300, Vsync=13180461738960, OldestInputEvent=9223372036854775807, NewestInp
utEvent=0, HandleInputStart=13180469565907, AnimationStart=13180469604136, PerformTraversalsStart=13180471340542, DrawStart=13180482733199, SyncQueued=13180484299188, SyncStart=131
80485001688, IssueDrawCommandsStart=13180485289136, SwapBuffers=13180502223615, FrameCompleted=13180503651480, DequeueBufferDuration=1234000, QueueBufferDuration=665000,
06-08 23:41:33.035 12563 12563 E PhoneWindow: Horizontal progress bar not located in current window decor
06-08 23:41:34.211 12563 12563 I Choreographer: Skipped 37 frames!  The application may be doing too much work on its main thread.
06-08 23:41:34.887 12563 12563 E PhoneWindow: Horizontal progress bar not located in current window decor
06-08 23:41:34.906 12563 12563 E PhoneWindow: Horizontal progress bar not located in current window decor
06-08 23:41:34.930 12563 12563 E PhoneWindow: Horizontal progress bar not located in current window decor

In most other websites the script works.

[warren-bank]

  curl -I "https://www.zhihu.com/"

HTTP/2 302
location: //www.zhihu.com/signin?next=%2F

• I've never seen a server send a 302 redirect to a URL that wasn't fully qualified
  • is that allowed by the HTTP spec?
• does your System WebView follow that redirect to a login page,
  or just hang on a blank page?

[warren-bank]

hmm.. you're right.. that is interesting

• I tested with this userscript,
  which is similar to yours.. shows a Toast at every page load with a greetings message
• it works on most sites, but fails on the example you gave
  • my WebView follows the redirect
  • no Toast when either:
    • following the redirect from the home page to the login page
    • loading the login page directly

sufficed to say.. the site must be doing something that prevents the browser from firing the DOM events that the WebView listens for.. and forwards to its event listeners (in Java).

I'm curious if Tampermonkey would work on this site..
update: it does

[warren-bank]

another thought..
(off the top of my head.. having forgotten how all the pieces actually fit together.. and this could be entirely irrelevent)..
is that the site's content-security-policy might be doing something that prevents the userscript from running:

content-security-policy: default-src * blob:; img-src * data: blob: resource: t.captcha.qq.com *.dun.163yun.com *.dun.163.com *.126.net *.nosdn.127.net nos.netease.com; connect-src * wss: blob: resource:; frame-src 'self' *.zhihu.com mailto: tel: weixin: *.vzuu.com mo.m.taobao.com getpocket.com note.youdao.com safari-extension://com.evernote.safari.clipper-Q79WDW8YH9 blob: mtt: zhihujs: captcha.guard.qcloud.com pos.baidu.com dup.baidustatic.com openapi.baidu.com wappass.baidu.com passport.baidu.com *.cme.qcloud.com vs-cdn.tencent-cloud.com t.captcha.qq.com *.dun.163yun.com *.dun.163.com *.126.net *.nosdn.127.net nos.netease.com; script-src 'self' blob: *.zhihu.com g.alicdn.com qzonestyle.gtimg.cn res.wx.qq.com open.mobile.qq.com 'unsafe-eval' unpkg.zhimg.com unicom.zhimg.com resource: zhihu-live.zhimg.com captcha.gtimg.com captcha.guard.qcloud.com pagead2.googlesyndication.com cpro.baidustatic.com pos.baidu.com dup.baidustatic.com i.hao61.net jsapi.qq.com 'nonce-5a10d57a-b730-4c57-b107-1032b657271b' hm.baidu.com zz.bdstatic.com b.bdstatic.com imgcache.qq.com vs-cdn.tencent-cloud.com www.mangren.com www.yunmd.net zhihu.govwza.cn p.cnwza.cn ssl.captcha.qq.com t.captcha.qq.com *.dun.163yun.com *.dun.163.com *.126.net *.nosdn.127.net nos.netease.com; style-src 'self' 'unsafe-inline' *.zhihu.com unicom.zhimg.com resource: captcha.gtimg.com www.mangren.com ssl.captcha.qq.com t.captcha.qq.com *.dun.163yun.com *.dun.163.com *.126.net *.nosdn.127.net nos.netease.com; font-src * data:; frame-ancestors *.zhihu.com
x-content-security-policy: default-src * blob:; img-src * data: blob: resource: t.captcha.qq.com *.dun.163yun.com *.dun.163.com *.126.net *.nosdn.127.net nos.netease.com; connect-src * wss: blob: resource:; frame-src 'self' *.zhihu.com mailto: tel: weixin: *.vzuu.com mo.m.taobao.com getpocket.com note.youdao.com safari-extension://com.evernote.safari.clipper-Q79WDW8YH9 blob: mtt: zhihujs: captcha.guard.qcloud.com pos.baidu.com dup.baidustatic.com openapi.baidu.com wappass.baidu.com passport.baidu.com *.cme.qcloud.com vs-cdn.tencent-cloud.com t.captcha.qq.com *.dun.163yun.com *.dun.163.com *.126.net *.nosdn.127.net nos.netease.com; script-src 'self' blob: *.zhihu.com g.alicdn.com qzonestyle.gtimg.cn res.wx.qq.com open.mobile.qq.com 'unsafe-eval' unpkg.zhimg.com unicom.zhimg.com resource: zhihu-live.zhimg.com captcha.gtimg.com captcha.guard.qcloud.com pagead2.googlesyndication.com cpro.baidustatic.com pos.baidu.com dup.baidustatic.com i.hao61.net jsapi.qq.com 'nonce-5a10d57a-b730-4c57-b107-1032b657271b' hm.baidu.com zz.bdstatic.com b.bdstatic.com imgcache.qq.com vs-cdn.tencent-cloud.com www.mangren.com www.yunmd.net zhihu.govwza.cn p.cnwza.cn ssl.captcha.qq.com t.captcha.qq.com *.dun.163yun.com *.dun.163.com *.126.net *.nosdn.127.net nos.netease.com; style-src 'self' 'unsafe-inline' *.zhihu.com unicom.zhimg.com resource: captcha.gtimg.com www.mangren.com ssl.captcha.qq.com t.captcha.qq.com *.dun.163yun.com *.dun.163.com *.126.net *.nosdn.127.net nos.netease.com; font-src * data:; frame-ancestors *.zhihu.com
x-webkit-csp: default-src * blob:; img-src * data: blob: resource: t.captcha.qq.com *.dun.163yun.com *.dun.163.com *.126.net *.nosdn.127.net nos.netease.com; connect-src * wss: blob: resource:; frame-src 'self' *.zhihu.com mailto: tel: weixin: *.vzuu.com mo.m.taobao.com getpocket.com note.youdao.com safari-extension://com.evernote.safari.clipper-Q79WDW8YH9 blob: mtt: zhihujs: captcha.guard.qcloud.com pos.baidu.com dup.baidustatic.com openapi.baidu.com wappass.baidu.com passport.baidu.com *.cme.qcloud.com vs-cdn.tencent-cloud.com t.captcha.qq.com *.dun.163yun.com *.dun.163.com *.126.net *.nosdn.127.net nos.netease.com; script-src 'self' blob: *.zhihu.com g.alicdn.com qzonestyle.gtimg.cn res.wx.qq.com open.mobile.qq.com 'unsafe-eval' unpkg.zhimg.com unicom.zhimg.com resource: zhihu-live.zhimg.com captcha.gtimg.com captcha.guard.qcloud.com pagead2.googlesyndication.com cpro.baidustatic.com pos.baidu.com dup.baidustatic.com i.hao61.net jsapi.qq.com 'nonce-5a10d57a-b730-4c57-b107-1032b657271b' hm.baidu.com zz.bdstatic.com b.bdstatic.com imgcache.qq.com vs-cdn.tencent-cloud.com www.mangren.com www.yunmd.net zhihu.govwza.cn p.cnwza.cn ssl.captcha.qq.com t.captcha.qq.com *.dun.163yun.com *.dun.163.com *.126.net *.nosdn.127.net nos.netease.com; style-src 'self' 'unsafe-inline' *.zhihu.com unicom.zhimg.com resource: captcha.gtimg.com www.mangren.com ssl.captcha.qq.com t.captcha.qq.com *.dun.163yun.com *.dun.163.com *.126.net *.nosdn.127.net nos.netease.com; font-src * data:; frame-ancestors *.zhihu.com
strict-transport-security: max-age=15552000; includeSubDomains
x-xss-protection: 1; mode=block

[warren-bank]

what's interesting is..

• your logcat (above) clearly shows the message:
  Running script "My Script, greasyfork.org: null" on https://www.zhihu.com/signin?next=%2F
• this log statement comes from here..
• which occurs right before the script is injected into the WebView.. where it should be run immediately
• which reinforces my guess that it has to do with a content security policy..
  which tells the browser that doing so is not OK
  • but I'll need to try this with a remote debugger attached to the WebView.. to see what's really going on
  • or possibly loading the page through an HTTP debugging proxy (ex: Fiddler classic)..
    and remove the CSP response headers before they reach the WebView

[warren-bank]

I've confirmed that the issue is the result of the CSP response headers.

Rather than doing the elaborate testing suggested earlier,
I tested this assertion by simply running a local webserver..
and hosting two .html files:

1. /test-01.html
   • no CSP headers
2. /test-02.html
   • same CSP headers as served by https://www.zhihu.com/

sure enough..

• the 1st test works
• the 2nd test fails

unfortunately, the WebView API is very limited..
it doesn't provide a way to intercept and conditionally remove HTTP response headers..
so this is going to take some time to consider the options..
if there even are any.

here is the server configuration for my test.
it depends on serve being installed..
either globally, or locally.. and simply added to PATH before being called.

[warren-bank]

• searching for related info
• this Chromium issue from 2016 is exactly on target
  • they say that since WebView doesn't have a JS sandbox environment the way that WebExtensions do..
    the code passed to webview.evaluateJavascript() must run in the same security environment as the page content..
    subject to the same security policy
• not good..

[warren-bank]

hello again 😃
a lot has happened since we last spoke,
here is a quick update..

as you might have noticed,
I contacted the developer of PCAPdroid..
to ask whether his app might somehow be useful to address this issue..
and OMG, you wouldn't believe how useful it is!

it seems that I caught him at the perfect time,
because he's very actively developing his PCAPdroid mitm addon at the moment..
as is very open to ideas and suggestions.

this addon will surely see major enhancements in the near future..
I'm currently testing with v0.14 of the addon..
and even as it is right now, it can solve all of our issues.

the addon bundles an instance of the mitmproxy proxy server..
and runs it in SOCKS5 mode,
through which.. PCAPdroid tunnels all of its HTTPS traffic,
which allows PCAPdroid to see the unencrypted HTTP requests and responses.

two important points:

1. mitmproxy proxy server supports scripts that define addons
2. PCAPdroid has a settings field to enter additional parameters to the mitmproxy proxy server command-line

this enables custom scripts to be run by PCAPdroid mitm by specifying them in the additional parameters.
for example:

  -s /path/to/script.py

I've been busy playing around with this feature and writing several such scripts.

Specifically, with respect to how such scripts can be used to enhance WebMonkey..

WebMonkey has two major inherent limitations:

1. CSP can block userscripts from running
2. userscripts cannot be injected into iframe windows

PCAPdroid mitm provides a workaround for both,
which can be turned on as-needed,
and doesn't require any integration with WebMonkey..
it just works.

1. the disable_csp script fixes the 1st
2. the modify_html_files script provides a solution to the 2nd

[shunf4]

Would WebViewClient#shouldInterceptRequest help?

[warren-bank]

the short answer is: no

the longer answer is..

The suggested workaround is to use the shouldInterceptRequest hook to:

• prevent the WebView from ever making any network requests
  • intercept
  • proxy
  • make any desired changes inflight
  • hand off the response

I could do that, but rather than implementing this from within the app..
I figured that a system-level debugging proxy that hooks into Android by simulating a VPN..
would be a more generic solution to a common problem..
and sure enough, it works perfectly.

One of the (many) great things about the PCAPdroid app..
is that you can choose which apps use the proxy..
and allow all other apps to circumvent it (as per usual).

[shunf4]

0. I tried using PCAPdroid. Do I have to choose "Proceed" in "On HTTPS Certificate Error" setting in order to have it work?
1. Unfortunately I prefer not using another VpnService because I already have one always switched on (I live in China!); also I prefer a less resource-consuming solution because I'm using an underpowered E-Ink device.
2. And surprisingly I found I did the exactly same shouldInterceptRequest a year ago, which I entirely forgot. I admit it's not a good workaround because
   • it doesn't handle redirects and posts
   • and sometimes necessary headers that in usual are automatically added by the browser (cookies etc.) are not added in the request when manually doing the request.
3. So maybe I'll look into hook solution (by LSPosed or LSPatch? I don't know).

[shunf4]

A Xposed hook module which adds UserScript (as well as DevTools) support for Chromium browsers (also most WebView browsers are supported), by hooking onUpdateUrl and then loading a javascript: URL: https://github.com/JingMatrix/ChromeXt

Found the module months earlier, havn't tried it yet.

[shunf4]

-- And it seems have solved the CSP problem: https://github.com/JingMatrix/ChromeXt/blob/master/app/src/main/assets/CSP.user.js

[warren-bank]

wow.. ChromeXt is very interesting.. I'll need to take a much closer look.
My main concern is security.. any app that can intercept and tinker with network traffic is crazy dangerous..
especially when granted root.. and needs to be 100% trusted.

What I like about using PCAPdroid is that it's non-root.. only runs when you turn it on.. only intercepts the whitelist of apps you choose.. and the code is 100% transparent w/ a fair bit of scrutiny by both the developer community and f-droid. But, you're right, that if I needed to run a (real) VPN most of the time.. this solution wouldn't work for me either.. well, not unless I allowed PCAPdroid to run as root.. in which case, it could run in parallel to another (real) VPN.

Regarding hacking the WebView.. it is funny that you already implemented the exact same workaround.. and forgot about it.
Does your version work on this website.. without needing ChromeXt?
Seems like it should, since it filters the CSP response header..
although there are a few other CSP-related headers that you might also want to remove.

PS: I like your version of this hack better than the answer on SO.. cleaner code! 😃