Skip to content

Add support for request dataType to network data collection #1011

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 1, 2025
Merged

Add support for request dataType to network data collection #1011

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d5a153f
Add support for request dataType to network data collection
juliandescottes Sep 26, 2025
2eca438
Address review comments about types
juliandescottes Sep 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 70 additions & 16 deletions index.bs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6876,11 +6876,40 @@ To <dfn>match collector for navigable</dfn> given |collector| and |navigable|:

</div>

<div algorithm>
To <dfn export>clone network request body</dfn> given |request|:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd propose to use typed arguments for externally used hooks: #1004

Suggested change
To <dfn export>clone network request body</dfn> given |request|:
To <dfn export>clone network request body</dfn> given [=/request=] |request|:

Copy link
Contributor

@sadym-chromium sadym-chromium Sep 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, despite the comment, it is not used externally

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory it should be called from fetch, but yes at the moment it's blocked on whatwg/fetch#1540

Alternatively I can rewrite the PR to trigger this algorithm from the beforeRequestSent event handler if you prefer?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is not a blocker (I already approved the PR).

I cannot see in the fetch RP any reference to this hook. Maybe this comment meant to underline that this is a part of the WebDriver BiDi before request sent hook, which is invoked by the fetch spec in fetch RP?

Copy link
Contributor Author

@juliandescottes juliandescottes Sep 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh no you're right, I haven't updated James' PR. Not quite sure how to do that but I can give it a try.

And in theory, WebDriver BiDi before request sent is a bit too late, because other consumers might have read the stream already, so we really need to clone it as soon as possible.


Note: This hook is intended to be triggered by the fetch spec when the request body has been safely extracted.
See step 9 of https://fetch.spec.whatwg.org/#concept-fetch

1. If |request|'s [=request/body=] is null, return.

1. For each |session| in [=active BiDi sessions=]:

1. If |session|'s [=network collectors=] is not [=list/empty=]:

1. Let |collected data| be a [=network data=] with
[=network-data/bytes=] set to null,
[=network-data/cloned body=] set to [=body/clone=] of |request|'s [=request/body=],
[=network-data/collectors=] set to an empty list,
[=network-data/pending=] set to true,
[=network-data/request=] set to |request|'s [=request id=],
[=network-data/size=] set to null,
[=network-data/type=] set to "response".

1. [=list/Append=] |collected data| to [=collected network data=].

1. Return.

</div>

<div algorithm>
To <dfn export>clone network response body</dfn> given |request| and |response body|:

Note: This hook is intended to be triggered by the fetch spec when the response is set.

1. If |response body| is null, return.

1. For each |session| in [=active BiDi sessions=]:

1. If |session|'s [=network collectors=] is not [=list/empty=]:
Expand Down Expand Up @@ -6925,19 +6954,51 @@ To <dfn>maybe abort network response body collection</dfn> given |request|:

</div>

<div algorithm>
To <dfn>maybe collect network request body</dfn> given |request|:

1. Let |collected data| be [=get collected data=] with |request|'s [=request id=] and "request".

1. If |collected data| is null, return.

NOTE: This might happen if there are no collectors setup when the request is created,
and [=clone network request body=] does not clone the corresponding body.
Or if the body was null in the first place.

1. [=Maybe collect network data=] with |request|, |collected data|, null and "request".

</div>

<div algorithm>
To <dfn>maybe collect network response body</dfn> given |request| and |response|:

1. If |response|'s [=response/status=] is a [=redirect status=], return.

Note: For redirects, only the final response body is stored.
NOTE: For redirects, only the final response body is stored.

1. Let |collected data| be [=get collected data=] with |request|'s [=request id=] and "response".

1. If |collected data| is null, return.

NOTE: This might happen if there are no collectors setup when the response is created,
and [=clone network response body=] does not clone the corresponding body.
Or if the body was null in the first place.

1. Let |size| be |response|'s [=response body info=]'s [=encoded size=].

NOTE: There is a discrepancy between the fact that the bytes retrieved from the
fetch stream correspond to the decoded data, but the encoded (network) size is
used in order to calculate size limits. Implementations might decide to use a storage
model such that it uses less size than storing the decoded data, as long as the data
returned to clients in getData is identical to the decoded data. The potential
tradeoff between storage and performance is up to the implementation.

1. [=Maybe collect network data=] with |request|, |collected data|, |size| and "response".

</div>

<div algorithm>
To <dfn>maybe collect network data</dfn> given |request|, |collected data|, |size| and |data type|:

1. Set |collected data|'s <code>pending</code> to false.

Expand All @@ -6947,7 +7008,7 @@ To <dfn>maybe collect network response body</dfn> given |request| and |response|

1. [=list/Remove=] |collected data| from [=collected network data=].

1. [=Resume=] with "<code>network data collected</code>" and (|request|'s [=request id=], "response").
1. [=Resume=] with "<code>network data collected</code>" and (|request|'s [=request id=], |data type|).

1. Return.

Expand All @@ -6963,7 +7024,7 @@ To <dfn>maybe collect network response body</dfn> given |request| and |response|

1. For each |collector| in |session|'s [=network collectors=]:

1. If |collector|'s [=network-collector/data types=] [=list/contains=] "response" and if
1. If |collector|'s [=network-collector/data types=] [=list/contains=] |data type| and if
[=match collector for navigable=] with |collector| and |top-level navigable|:

1. [=list/Append=] |collector| to |collectors|.
Expand All @@ -6972,28 +7033,19 @@ To <dfn>maybe collect network response body</dfn> given |request| and |response|

1. [=list/Remove=] |collected data| from [=collected network data=].

1. [=Resume=] with "<code>network data collected</code>" and (|request|'s [=request id=], "response").
1. [=Resume=] with "<code>network data collected</code>" and (|request|'s [=request id=], |data type|).

1. Return.

1. Let |bytes| be null.

1. Let |size| be null.

1. Let |processBody| given |nullOrBytes| be this step:

1. If |nullOrBytes| is not null:

1. Set |bytes| to [=serialize protocol bytes=] with |nullOrBytes|.

1. Set |size| to |response|'s [=response body info=]'s [=encoded size=].

Note: There is a discrepancy between the fact that the bytes retrieved from the
fetch stream correspond to the decoded data, but the encoded (network) size is
used in order to calculate size limits. Implementations might decide to use a storage
model such that it uses less size than storing the decoded data, as long as the data
returned to clients in getData is identical to the decoded data. The potential
tradeoff between storage and performance is up to the implementation.
1. If |size| is null, set |size| to |bytes|' [=byte sequence/length=].

1. Let |processBodyError| be this step: Do nothing.

Expand All @@ -7016,7 +7068,7 @@ To <dfn>maybe collect network response body</dfn> given |request| and |response|

1. Otherwise, [=list/remove=] |collected data| from [=collected network data=].

1. [=Resume=] with "<code>network data collected</code>" and (|request|'s [=request id=], "response").
1. [=Resume=] with "<code>network data collected</code>" and (|request|'s [=request id=], |data type|).

</div>

Expand Down Expand Up @@ -7524,7 +7576,7 @@ To <dfn>serialize cookie header</dfn> given |protocol cookie|:
{^Remote end definition^} and {^local end definition^}

<pre class="cddl" data-cddl-module="local-cddl,remote-cddl">
network.DataType = "response"
network.DataType = "request" / "response"
</pre>

The <code><dfn>network.DataType</dfn></code> type represents the different types of network data
Expand Down Expand Up @@ -9666,6 +9718,8 @@ request sent</dfn> steps given |request|:
are assumed to be valid at this stage; any error accessing the proxy will
be reported as a network error when handling the request.

1. [=Maybe collect network request body=] with |request|.

1. If [=before request sent map=] does not contain |request|, set [=before
request sent map=][|request|] to a new set.

Expand Down