Skip to content

Commit bda85d9

Browse files
nwebsonnwebson
committed
puppeteer: new runtimeEnabled fix (addBinding) with access to the main world, add tries to acquireContextId method
1 parent 9aa7109 commit bda85d9

File tree

3 files changed

+176
-42
lines changed

3 files changed

+176
-42
lines changed

‎README.md‎

Lines changed: 19 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -46,25 +46,31 @@ However, there's a technique that detects the usage of this command, revealing t
4646
For more details on this technique, read DataDome's blog post: [How New Headless Chrome & the CDP Signal Are Impacting Bot Detection](https://datadome.co/threat-research/how-new-headless-chrome-the-cdp-signal-are-impacting-bot-detection/).
4747
In brief, it's a few lines of JavaScript on the page that are automatically called if `Runtime.Enable` was used.
4848

49-
Our fix disables the automatic `Runtime.Enable` command on every frame. Instead, we manually create contexts with unknown IDs when a frame is created. Then, when code needs to be executed, we have implemented two approaches to get the context ID. You can choose which one to use.
49+
Our fix disables the automatic `Runtime.Enable` command on every frame. Instead, we manually create contexts with unknown IDs when a frame is created. Then, when code needs to be executed, there are multiple ways to get the context ID.
5050

51-
#### 1. Create a new isolated context via `Page.createIsolatedWorld` and save its ID from the CDP response.
51+
#### 1. Create a new binding in the main world, call it and save its context ID.
52+
🟢 Pros: The ultimate approach that keeps access to the main world and works with web workers. You don't need to change any of your existing codebase.
53+
54+
🔴 Cons: None are discovered so far.
55+
56+
#### 2. Create a new isolated context via `Page.createIsolatedWorld` and save its ID.
5257
🟢 Pros: All your code will be executed in a separate isolated world, preventing page scripts from detecting your changes via MutationObserver and other techniques.
5358

5459
🔴 Cons: You won't be able to access main context variables and code. While this is necessary for some use cases, the isolated context generally works fine for most scenarios. Also, web workers don't allow creating new worlds, so you can't execute your code inside a worker. This is a niche use case but may matter in some situations. There is a workaround for this issue, please read [How to Access Main Context Objects from Isolated Context in Puppeteer & Playwright](https://rebrowser.net/blog/how-to-access-main-context-objects-from-isolated-context-in-puppeteer-and-playwright-23741).
5560

56-
#### 2. Call `Runtime.Enable` and then immediately call `Runtime.Disable`.
61+
#### 3. Call `Runtime.Enable` and then immediately call `Runtime.Disable`.
5762
This triggers `Runtime.executionContextCreated` events, allowing us to catch the proper context ID.
5863

5964
🟢 Pros: You will have full access to the main context.
6065

6166
🔴 Cons: There's a slight chance that during this short timeframe, the page will call code that leads to the leak. The risk is low, as detection code is usually called during specific actions like CAPTCHA pages or login/registration forms, typically right after the page loads. Your business logic is usually called a bit later.
6267

63-
> 🎉 Our tests show that both approaches are currently undetectable by Cloudflare or DataDome.
68+
> 🎉 Our tests show that all these approaches are currently undetectable by Cloudflare or DataDome.
6469
6570
Note: you can change settings for this patch on the fly using an environment variable. This allows you to easily switch between patched and non-patched versions based on your business logic.
6671

67-
- `REBROWSER_PATCHES_RUNTIME_FIX_MODE=alwaysIsolated` — always run all scripts in isolated context (default)
72+
- `REBROWSER_PATCHES_RUNTIME_FIX_MODE=addBinding` — addBinding technique (default)
73+
- `REBROWSER_PATCHES_RUNTIME_FIX_MODE=alwaysIsolated` — always run all scripts in isolated context
6874
- `REBROWSER_PATCHES_RUNTIME_FIX_MODE=enableDisable` — use Enable/Disable technique
6975
- `REBROWSER_PATCHES_RUNTIME_FIX_MODE=0` — completely disable this patch
7076
- `REBROWSER_PATCHES_DEBUG=1` — enable some debugging messages
@@ -113,7 +119,7 @@ This env variable cannot be changed on the fly, you have to set it before runnin
113119
*Note: it's not detectable by external website scripts, but Google might use this information in their proprietary Chrome; we never know.*
114120

115121
## Usage
116-
This package is designed to be run against an installed library. Install the Puppeteer library, then call the patcher, and it's ready to go.
122+
This package is designed to be run against an installed library. Install the library, then call the patcher, and it's ready to go.
117123

118124
In the root folder of your project, run:
119125
```
@@ -138,24 +144,17 @@ You can see all command-line options by running `npx rebrowser-patches@latest --
138144
## How to update the patches?
139145
If you already have your package patched and want to update to the latest version of rebrowser-patches, the easiest way would be to delete `node_modules/puppeteer-core`, then run `npm install` or `yarn install --check-files`, and then run `npx rebrowser-patches@latest patch`.
140146

141-
## Puppeteer support
147+
## How to patch Java/Python/.NET versions of Playwright?
148+
All these versions are just wrappers around Node.js version of Playwright. You need to find `driver` folder inside your Playwright package and run this patch with `--packagePath=$yourDriverFolder/$yourPlatform/package`.
142149

143-
| Pptr Ver | Release Date | Chrome Ver | Patch Support |
144-
|--------------------------------------|--------------|------------|---------------|
145-
| 23.3.x | 2024-09-04 | 128 | ✅ |
146-
| 23.2.x | 2024-08-29 | 128 | ✅ |
147-
| 23.1.x | 2024-08-14 | 127 | ✅ |
148-
| 23.0.x | 2024-08-07 | 127 | ✅ |
149-
| 22.15.x | 2024-07-31 | 127 | ✅ |
150-
| 22.14.x | 2024-07-25 | 127 | ✅ |
151-
| 22.13.x | 2024-07-11 | 126 | ✅ |
152-
| 22.12.x<br/><small>and below</small> | 2024-06-21 | 126 | ❌ |
150+
## Puppeteer support
151+
Latest fully tested version: 23.6.0 (2024-10-16)
152+
✅ Versions 22.13.x and above are supported.
153+
❌ Versions 22.12.x and below are not supported.
153154

154155
## Playwright support
155156
Playwright patches support `Runtime.enable` leak (only alwaysIsolated mode) and ability to change utility world name via `REBROWSER_PATCHES_UTILITY_WORLD_NAME` env variable.
156157

157-
Only JS version of Playwright is supported. Python is coming soon.
158-
159158
| Playwright Ver | Release Date | Chrome Ver | Patch Support |
160159
|-------------------------------------|--------------|------------|---------------|
161160
| 1.47.2 | 2024-09-20 | 129 | ✅ |
@@ -170,7 +169,7 @@ import puppeteer from 'puppeteer-extra'
170169
171170
// after
172171
import { addExtra } from 'puppeteer-extra'
173-
import rebrowserPuppeteer from 'rebrowser-puppeteer'
172+
import rebrowserPuppeteer from 'rebrowser-puppeteer-core'
174173
const puppeteer = addExtra(rebrowserPuppeteer)
175174
```
176175

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "rebrowser-patches",
3-
"version": "1.0.12",
3+
"version": "1.0.13",
44
"description": "Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.",
55
"keywords": [
66
"automation",

‎patches/puppeteer-core/22.13.x.patch‎

Lines changed: 156 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@
5353
clientEmitter.on('Runtime.consoleAPICalled', this.#onConsoleAPI.bind(this));
5454
clientEmitter.on(CDPSessionEvent.Disconnected, () => {
5555
this[disposeSymbol]();
56-
@@ -351,6 +358,74 @@
56+
@@ -351,6 +358,119 @@
5757
return await this.#evaluate(false, pageFunction, ...args);
5858
}
5959

@@ -65,16 +65,57 @@
6565
+ this.#puppeteerUtil = undefined
6666
+ }
6767
+ // rebrowser-patches: get context id if it's missing
68-
+ async acquireContextId() {
68+
+ async acquireContextId(tryCount = 1): Promise<any> {
6969
+ if (this.#id > 0) {
7070
+ return
7171
+ }
7272
+
73-
+ const fixMode = process.env['REBROWSER_PATCHES_RUNTIME_FIX_MODE'] || 'alwaysIsolated'
74-
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log(`[rebrowser-patches][acquireContextId] id = ${this.#id}, name = ${this.#name}, fixMode = ${fixMode}`)
73+
+ const fixMode = process.env['REBROWSER_PATCHES_RUNTIME_FIX_MODE'] || 'addBinding'
74+
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log(`[rebrowser-patches][acquireContextId] id = ${this.#id}, name = ${this.#name}, fixMode = ${fixMode}, tryCount = ${tryCount}`)
7575
+
7676
+ let contextId: any
77-
+ if (fixMode === 'alwaysIsolated') {
77+
+ if (fixMode === 'addBinding') {
78+
+ if (this.#id === -2) {
79+
+ // isolated world
80+
+ const sendRes = await this.#client
81+
+ .send('Page.createIsolatedWorld', {
82+
+ frameId: this._frameId,
83+
+ worldName: this.#name,
84+
+ grantUniveralAccess: true,
85+
+ });
86+
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log(`[rebrowser-patches][acquireContextId] Page.createIsolatedWorld result:`, sendRes);
87+
+ contextId = sendRes.executionContextId;
88+
+ } else {
89+
+ // main world
90+
+ // add the binding
91+
+ const bindingName = Math.random().toString()
92+
+ await this.#client.send('Runtime.addBinding', {
93+
+ name: bindingName,
94+
+ })
95+
+
96+
+ // listen for 'Runtime.bindingCalled' event
97+
+ const bindingCalledHandler = ({ name, executionContextId }: any) => {
98+
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log('[rebrowser-patches][bindingCalledHandler]', { name, executionContextId })
99+
+ if (contextId > 0) {
100+
+ // already acquired the id
101+
+ return;
102+
+ }
103+
+ if (name !== bindingName) {
104+
+ // ignore irrelevant bindings
105+
+ return
106+
+ }
107+
+ contextId = executionContextId
108+
+ // remove this listener
109+
+ this.#client.off('Runtime.bindingCalled', bindingCalledHandler);
110+
+ }
111+
+ this.#client.on('Runtime.bindingCalled', bindingCalledHandler);
112+
+
113+
+ // call the binding
114+
+ await this.#client.send('Runtime.evaluate', {
115+
+ expression: `self['${bindingName}']('1')`
116+
+ })
117+
+ }
118+
+ } else if (fixMode === 'alwaysIsolated') {
78119
+ if (this.#id === -3) {
79120
+ throw new Error('[rebrowser-patches] web workers are not supported in alwaysIsolated mode')
80121
+ }
@@ -119,7 +160,11 @@
119160
+ }
120161
+
121162
+ if (!contextId) {
122-
+ throw new Error('[rebrowser-patches] acquireContextId failed')
163+
+ if (tryCount >= 3) {
164+
+ throw new Error('[rebrowser-patches] acquireContextId failed')
165+
+ }
166+
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log(`[rebrowser-patches][acquireContextId] failed, try again (tryCount = ${tryCount})`)
167+
+ return this.acquireContextId(tryCount + 1)
123168
+ }
124169
+
125170
+ this.#id = contextId
@@ -128,7 +173,7 @@
128173
async #evaluate<
129174
Params extends unknown[],
130175
Func extends EvaluateFunc<Params> = EvaluateFunc<Params>,
131-
@@ -375,6 +450,13 @@
176+
@@ -375,6 +495,13 @@
132177
pageFunction: Func | string,
133178
...args: Params
134179
): Promise<HandleFor<Awaited<ReturnType<Func>>> | Awaited<ReturnType<Func>>> {
@@ -373,7 +418,7 @@
373418
*/
374419
evaluateHandle<Params extends unknown[], Func extends EvaluateFunc<Params> = EvaluateFunc<Params>>(pageFunction: Func | string, ...args: Params): Promise<HandleFor<Awaited<ReturnType<Func>>>>;
375420
+ clear(newId: any): void;
376-
+ acquireContextId(): Promise<void>;
421+
+ acquireContextId(tryCount?: number): Promise<any>;
377422
[disposeSymbol](): void;
378423
}
379424
//# sourceMappingURL=ExecutionContext.d.ts.map
@@ -419,7 +464,7 @@
419464
clientEmitter.on('Runtime.consoleAPICalled', this.#onConsoleAPI.bind(this));
420465
clientEmitter.on(CDPSession_js_1.CDPSessionEvent.Disconnected, () => {
421466
this[disposable_js_1.disposeSymbol]();
422-
@@ -324,7 +331,75 @@
467+
@@ -324,7 +331,120 @@
423468
async evaluateHandle(pageFunction, ...args) {
424469
return await this.#evaluate(false, pageFunction, ...args);
425470
}
@@ -431,14 +476,55 @@
431476
+ this.#puppeteerUtil = undefined;
432477
+ }
433478
+ // rebrowser-patches: get context id if it's missing
434-
+ async acquireContextId() {
479+
+ async acquireContextId(tryCount = 1) {
435480
+ if (this.#id > 0) {
436481
+ return;
437482
+ }
438-
+ const fixMode = process.env['REBROWSER_PATCHES_RUNTIME_FIX_MODE'] || 'alwaysIsolated';
439-
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log(`[rebrowser-patches][acquireContextId] id = ${this.#id}, name = ${this.#name}, fixMode = ${fixMode}`);
483+
+ const fixMode = process.env['REBROWSER_PATCHES_RUNTIME_FIX_MODE'] || 'addBinding';
484+
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log(`[rebrowser-patches][acquireContextId] id = ${this.#id}, name = ${this.#name}, fixMode = ${fixMode}, tryCount = ${tryCount}`);
440485
+ let contextId;
441-
+ if (fixMode === 'alwaysIsolated') {
486+
+ if (fixMode === 'addBinding') {
487+
+ if (this.#id === -2) {
488+
+ // isolated world
489+
+ const sendRes = await this.#client
490+
+ .send('Page.createIsolatedWorld', {
491+
+ frameId: this._frameId,
492+
+ worldName: this.#name,
493+
+ grantUniveralAccess: true,
494+
+ });
495+
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log(`[rebrowser-patches][acquireContextId] Page.createIsolatedWorld result:`, sendRes);
496+
+ contextId = sendRes.executionContextId;
497+
+ }
498+
+ else {
499+
+ // main world
500+
+ // add the binding
501+
+ const bindingName = Math.random().toString();
502+
+ await this.#client.send('Runtime.addBinding', {
503+
+ name: bindingName,
504+
+ });
505+
+ // listen for 'Runtime.bindingCalled' event
506+
+ const bindingCalledHandler = ({ name, executionContextId }) => {
507+
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log('[rebrowser-patches][bindingCalledHandler]', { name, executionContextId });
508+
+ if (contextId > 0) {
509+
+ // already acquired the id
510+
+ return;
511+
+ }
512+
+ if (name !== bindingName) {
513+
+ // ignore irrelevant bindings
514+
+ return;
515+
+ }
516+
+ contextId = executionContextId;
517+
+ // remove this listener
518+
+ this.#client.off('Runtime.bindingCalled', bindingCalledHandler);
519+
+ };
520+
+ this.#client.on('Runtime.bindingCalled', bindingCalledHandler);
521+
+ // call the binding
522+
+ await this.#client.send('Runtime.evaluate', {
523+
+ expression: `self['${bindingName}']('1')`
524+
+ });
525+
+ }
526+
+ }
527+
+ else if (fixMode === 'alwaysIsolated') {
442528
+ if (this.#id === -3) {
443529
+ throw new Error('[rebrowser-patches] web workers are not supported in alwaysIsolated mode');
444530
+ }
@@ -481,7 +567,11 @@
481567
+ this.#client.off('Runtime.executionContextCreated', executionContextCreatedHandler);
482568
+ }
483569
+ if (!contextId) {
484-
+ throw new Error('[rebrowser-patches] acquireContextId failed');
570+
+ if (tryCount >= 3) {
571+
+ throw new Error('[rebrowser-patches] acquireContextId failed');
572+
+ }
573+
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log(`[rebrowser-patches][acquireContextId] failed, try again (tryCount = ${tryCount})`);
574+
+ return this.acquireContextId(tryCount + 1);
485575
+ }
486576
+ this.#id = contextId;
487577
+ }
@@ -721,7 +811,7 @@
721811
*/
722812
evaluateHandle<Params extends unknown[], Func extends EvaluateFunc<Params> = EvaluateFunc<Params>>(pageFunction: Func | string, ...args: Params): Promise<HandleFor<Awaited<ReturnType<Func>>>>;
723813
+ clear(newId: any): void;
724-
+ acquireContextId(): Promise<void>;
814+
+ acquireContextId(tryCount?: number): Promise<any>;
725815
[disposeSymbol](): void;
726816
}
727817
//# sourceMappingURL=ExecutionContext.d.ts.map
@@ -767,7 +857,7 @@
767857
clientEmitter.on('Runtime.consoleAPICalled', this.#onConsoleAPI.bind(this));
768858
clientEmitter.on(CDPSessionEvent.Disconnected, () => {
769859
this[disposeSymbol]();
770-
@@ -321,7 +328,75 @@
860+
@@ -321,7 +328,120 @@
771861
async evaluateHandle(pageFunction, ...args) {
772862
return await this.#evaluate(false, pageFunction, ...args);
773863
}
@@ -779,14 +869,55 @@
779869
+ this.#puppeteerUtil = undefined;
780870
+ }
781871
+ // rebrowser-patches: get context id if it's missing
782-
+ async acquireContextId() {
872+
+ async acquireContextId(tryCount = 1) {
783873
+ if (this.#id > 0) {
784874
+ return;
785875
+ }
786-
+ const fixMode = process.env['REBROWSER_PATCHES_RUNTIME_FIX_MODE'] || 'alwaysIsolated';
787-
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log(`[rebrowser-patches][acquireContextId] id = ${this.#id}, name = ${this.#name}, fixMode = ${fixMode}`);
876+
+ const fixMode = process.env['REBROWSER_PATCHES_RUNTIME_FIX_MODE'] || 'addBinding';
877+
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log(`[rebrowser-patches][acquireContextId] id = ${this.#id}, name = ${this.#name}, fixMode = ${fixMode}, tryCount = ${tryCount}`);
788878
+ let contextId;
789-
+ if (fixMode === 'alwaysIsolated') {
879+
+ if (fixMode === 'addBinding') {
880+
+ if (this.#id === -2) {
881+
+ // isolated world
882+
+ const sendRes = await this.#client
883+
+ .send('Page.createIsolatedWorld', {
884+
+ frameId: this._frameId,
885+
+ worldName: this.#name,
886+
+ grantUniveralAccess: true,
887+
+ });
888+
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log(`[rebrowser-patches][acquireContextId] Page.createIsolatedWorld result:`, sendRes);
889+
+ contextId = sendRes.executionContextId;
890+
+ }
891+
+ else {
892+
+ // main world
893+
+ // add the binding
894+
+ const bindingName = Math.random().toString();
895+
+ await this.#client.send('Runtime.addBinding', {
896+
+ name: bindingName,
897+
+ });
898+
+ // listen for 'Runtime.bindingCalled' event
899+
+ const bindingCalledHandler = ({ name, executionContextId }) => {
900+
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log('[rebrowser-patches][bindingCalledHandler]', { name, executionContextId });
901+
+ if (contextId > 0) {
902+
+ // already acquired the id
903+
+ return;
904+
+ }
905+
+ if (name !== bindingName) {
906+
+ // ignore irrelevant bindings
907+
+ return;
908+
+ }
909+
+ contextId = executionContextId;
910+
+ // remove this listener
911+
+ this.#client.off('Runtime.bindingCalled', bindingCalledHandler);
912+
+ };
913+
+ this.#client.on('Runtime.bindingCalled', bindingCalledHandler);
914+
+ // call the binding
915+
+ await this.#client.send('Runtime.evaluate', {
916+
+ expression: `self['${bindingName}']('1')`
917+
+ });
918+
+ }
919+
+ }
920+
+ else if (fixMode === 'alwaysIsolated') {
790921
+ if (this.#id === -3) {
791922
+ throw new Error('[rebrowser-patches] web workers are not supported in alwaysIsolated mode');
792923
+ }
@@ -829,7 +960,11 @@
829960
+ this.#client.off('Runtime.executionContextCreated', executionContextCreatedHandler);
830961
+ }
831962
+ if (!contextId) {
832-
+ throw new Error('[rebrowser-patches] acquireContextId failed');
963+
+ if (tryCount >= 3) {
964+
+ throw new Error('[rebrowser-patches] acquireContextId failed');
965+
+ }
966+
+ process.env['REBROWSER_PATCHES_DEBUG'] && console.log(`[rebrowser-patches][acquireContextId] failed, try again (tryCount = ${tryCount})`);
967+
+ return this.acquireContextId(tryCount + 1);
833968
+ }
834969
+ this.#id = contextId;
835970
+ }

0 commit comments

Comments
 (0)