Nonblocking libcurl

[karlseguin]

To get this working, it should only be necessary to re-run: make install-submodule. After this zig build test/run will automatically rebuild libcurl and its dependencies.

Missing

• Supporting the headers set via Network.setExtraHTTPHeaders (cdp)
• Supporting the proxyServer parameter of the Target.createBrowserContext (cdp). This will be tricky to implement.
• Dispatching the http_request_start, http_request_fail and http_request_complete (only used by cdp)
• Content-Type request header is weird / wrong (this is partially inherited from main). I think @sjorsdonkers said he was doing stuff with headers, so will wait until that's done to fix this.

Developer Notes

1. In http/Http.zig, setting ENABLE_DEBUG to true will cause libcurl to print information to stdout, similar to how curl -v behaves.

2. In the _wait function of page.zig, you'll find:

// defer self.printWaitAnalysis();

If you uncomment this line, it'll dump information about the http client and scheduler which might help indicating why wait is waiting so long to finish. Note: I generally think there's a lot of tweaking we can do here to hit a good balance between waiting for things to finish and not blocking unnecessarily.

3. There's a pseudo_connect branch to the implementation proxy that'll act as a MITM connect proxy. It's rough around the edges. Browser should be started with:

  --http_proxy http://127.0.0.1:3000/ \
  --insecure_disable_tls_host_verification

Future

What allowed this to be completed today was to load both static and dynamic imported modules in a [mostly] blocking manner. There's still a lot of room for performance improvement here, especially for dynamically imported modules. With debug logs, you you can identify these requests via the blocking = true argument.

[sjorsdonkers]

Since a script that is not async or defered is still executed non_blocking we currently do not wait for them, as they are not added to self.asyncs
Line 224:

    self.getList(&pending_script.script).append(&pending_script.node);

    try self.client.request(.{

[sjorsdonkers]

This problem reveals itself when we try to intercept these script requests,

[karlseguin]

i don't understand. I don't think there's any difference between defer, async and normal with respect to how they're fetched. They all just directly go to client.request. It's only the order in which the fully loaded body is executed that changes.

[sjorsdonkers]

The problem is that for async script the asynct script list is not empty until the async request is completed.
As a result as long as any async script is still being processed isDone is false. However, if anon-async process has not been completed yet isDone can return true as the not async scripts are not put in that list.
This results in a race condition where the main page may check isDone, and call self.documentIsComplete() while actually a script is still in flight.
In practice the script seems to be returning in time usually, however if a non-async script is waiting for an intercept, documentIsComplete will be called and the page closed before the intercept has received a response.

Adding a basic intercept to Playwright/connect.js:

await page.route('**/*', async (route, request) => {
  console.log(`Request ROUTE: ${request.method()} ${request.url()}`);
  route.continue();
});

should reproduce it.

< {"method":"Network.requestWillBeSent","params":{"requestId":"REQ-2","frameId":"TID-1","loaderId":"LID-1","documentUrl":"http://127.0.0.1:1234/campfire-commerce/","request":{"url":"http://127.0.0.1:1234/campfire-commerce/json/product.json","urlFragment":"","method":"GET","hasPostData":false,"headers":{"User-Agent":"Lightpanda/1.0"}}},"sessionId":"SID-1"}
< {"method":"Fetch.requestPaused","params":{"requestId":"INTERCEPT-2","request":{"url":"http://127.0.0.1:1234/campfire-commerce/json/product.json","urlFragment":"","method":"GET","hasPostData":false,"headers":{"User-Agent":"Lightpanda/1.0"}},"frameId":"TID-1","resourceType":"Document","networkId":"REQ-2"},"sessionId":"SID-1"}
< {"method":"Network.requestWillBeSent","params":{"requestId":"REQ-3","frameId":"TID-1","loaderId":"LID-1","documentUrl":"http://127.0.0.1:1234/campfire-commerce/","request":{"url":"http://127.0.0.1:1234/campfire-commerce/json/reviews.json","urlFragment":"","method":"GET","hasPostData":false,"headers":{"User-Agent":"Lightpanda/1.0"}}},"sessionId":"SID-1"}
< {"method":"Fetch.requestPaused","params":{"requestId":"INTERCEPT-3","request":{"url":"http://127.0.0.1:1234/campfire-commerce/json/reviews.json","urlFragment":"","method":"GET","hasPostData":false,"headers":{"User-Agent":"Lightpanda/1.0"}},"frameId":"TID-1","resourceType":"Document","networkId":"REQ-3"},"sessionId":"SID-1"}
< {"method":"Page.frameNavigated","params":{"type":"Navigation","frame":{"id":"TID-1","loaderId":"LID-1","url":"http://127.0.0.1:1234/campfire-commerce/","domainAndRegistry":"","securityOrigin":"://","mimeType":"text/html","adFrameStatus":{"adFrameType":"none"},"secureContextType":"InsecureScheme","crossOriginIsolatedContextType":"NotIsolated","gatedAPIFeatures":[]}},"sessionId":"SID-1"}     <<-- Target will close as it is not waiting for the sync request to complete
< {"method":"DOM.documentUpdated","params":{},"sessionId":"SID-1"}
< {"method":"Page.domContentEventFired","params":{"timestamp":55348},"sessionId":"SID-1"}
< {"method":"Page.lifecycleEvent","params":{"frameId":"TID-1","loaderId":"LID-1","name":"DOMContentLoaded","timestamp":55348},"sessionId":"SID-1"}
< {"method":"Page.loadEventFired","params":{"timestamp":55348},"sessionId":"SID-1"}
< {"method":"Page.lifecycleEvent","params":{"frameId":"TID-1","loaderId":"LID-1","name":"load","timestamp":55348},"sessionId":"SID-1"}
< {"method":"Page.frameStoppedLoading","params":{"frameId":"TID-1"},"sessionId":"SID-1"}
> {"id":39,"method":"Target.closeTarget","params":{"targetId":"TID-1"}}     <<-- Target closed as it is not waiting for the sync request to complete
> {"id":40,"method":"Fetch.continueRequest","params":{"requestId":"INTERCEPT-2"},"sessionId":"SID-1"}
> {"id":41,"method":"Fetch.continueRequest","params":{"requestId":"INTERCEPT-3"},"sessionId":"SID-1"}
< {"id":39,"result":{"success":true}}
< {"method":"Inspector.detached","params":{"reason":"Render process gone."},"sessionId":"SID-1"}
< {"method":"Target.detachedFromTarget","params":{"targetId":"TID-1","sessionId":"SID-1","reason":"Render process gone."}}
< {"id":40,"code":-32001,"message":"Unknown sessionId"}
< {"id":41,"code":-32001,"message":"Unknown sessionId"}

[karlseguin]

These are XHR requests. The issue is that Page.wait is seeing that http.active == 0 and exiting. It's impossible for active == 0 while there are active or pending requests (in nonblocking_libcurl)

But with request interception, you can have active == 0, but requests held-up by CDP.

I did a commit, then reverted it, which did self.active += 1 earlier (1). This way you could return from fn request(..) for interception and page wouldn't see active == 0. However, request interception calls request again, so now active would be incremented twice for a single request.

I think we should add a field to Page which is "pending_intercept: bool. Page.waitcan now wait foractive == 0 and pending_intercept == false. CDP can set this based on whether intercept_state.waiting.count() > 0`.

(1) 19c9080

[sjorsdonkers]

Somewhat related to the isDone discussion.
I wonder whether the code would be easier to follow if we have 1 central callback that handles xhr,scripts,document with if statements instead of heaving configurable callback function pointers for each.

So instead of having a pageDoneCallback, httpDoneCallback(xhr), doneCallback(script), There would just be 1 requestDoneCallback, that handles all 3 cases.

For example now the pageDoneCallback and the script's doneCallback both separately handle the done scenario:

if (self.isDone()) {
    self.page.documentIsComplete();
}

The code is very far apart, but needs to stay aligned. So in the case where there is a single callback function the logic would be in 1 place.