[Bug]: PATCH crawlconfig with partial config update causes data loss

[frodstee]

Browsertrix Version

v1.20.1-6f1c342

What did you expect to happen? What happened instead?

I need to mass update a behavior url on many crawl workflows, but when I try to use PATCH with the API the result is not as expected. PATCH change values that are not defined in the request payload.

Reproduction instructions

1. Create a new single page crawl workflow with a URL to crawl.
2. Save
3. Send a PATCH request to update only "customBehaviors" and "behaviors"
4. Several fields that is not included in the patch object is changed. See attached screenshot with diff.

Endpoints used:
PATCH: {{ base_url }}/api/orgs/{{org_id}}/crawlconfigs/b6781a39-3d4f-483f-b4f4-2c913e3114a0

Payload used:
{
"config":
{
"behaviors": "autoplay,autofetch,siteSpecific",
"customBehaviors": [
"https://raw.githubusercontent.com/NationalLibraryOfNorway/nettarkivet-browsertrix/refs/heads/main/amedia_personalia.js"
]
}
}

Screenshots / Video

Environment

No response

Additional details

Issue have been reproduced on app.browsertrix.com

[tw4l]

Hi @frodstee, thanks for the report and the diff. modified (last modified date for the workflow) and rev (number of crawl revisions, which we track in the database but don't expose at this point) I would expect to be updated, the other fields changing is I agree concerning. I don't believe we've seen this on our end.

What tool are you using to send the PATCH request? cURL, or something else? Is there any chance there is additional data (even nulls) being added by that tooling to the payload?

It doesn't appear the core of that PATCH method has been updated in a while, so if there is a bug it's possible it's been around for a little while. I will do some testing on our end to see if we can reproduce the issue.

[tw4l]

I can reproduce the issue, it looks like the bug is specifically when config is being updated, the entire config is rewritten to whatever is passed in with the PATCH payload rather than only updating the specified fields, and defaulting unspecified fields to their defaults as defined in the underlying database models. The Browsertrix frontend doesn't have the same issue as it passes the entire workflow and config objects into the PATCH each time, not just the fields that are being changed.

I have assigned to myself and put on the sprint starting tomorrow so that we can fix this and get the patch in an upcoming release quickly. Thanks again for the detailed issue!