list progress rewrites the stored filtering of a filtered work dir, breaking concurrent and resumed clone --filters runs

[ydixken]

We run pgcopydb 0.18 inside a Kubernetes operator and were polling pgcopydb list progress --json --dir ... via pod exec while pgcopydb clone --filters was running. At some point every worker of the running clone exited 12 with:

Catalogs at "/work/m/schema/source.db" have been setup for a different filtering than the current command, see above for details

and the log pair right above it showed:

Current filtering setup is: {"type":"SOURCE_FILTER_TYPE_EXCL","exclude-schema":["audit"]}
Catalog filtering setup is: {"type":"SOURCE_FILTER_TYPE_EXCL"}

The exclude-schema array had vanished from the stored record mid-run. After some digging: the list progress call itself rewrites it. The command takes no --filters option, so it opens the catalogs with empty filtering, and the setup check's "Case 3" path (a filterless command meeting a filtered catalog) adopts the stored filter type and then persists its own re-serialized filtering back into the catalog via catalog_update_filters (catalog.c around line 1336 in v0.18). The pattern arrays are lost in that write-back. cli_list_progress does not set skipFilterCheck, so unlike the sentinel commands it runs the full check-and-write path, and every process that opens the catalogs afterwards with the real --filters fails the comparison and exits 12.

The not ideal part is that this is not just a race against a running clone. A single bare list progress against the work dir of an already completed filtered clone poisons it for good: any later clone --filters ... --resume --not-consistent on that work dir fails the same way.

Steps to reproduce

Self-contained, rootless podman, pgcopydb 0.18-1.pgdg12+1, PostgreSQL 17 on both sides:

podman network create pgnet
podman run -d --name src --network pgnet -e POSTGRES_PASSWORD=pw postgres:17
podman run -d --name tgt --network pgnet -e POSTGRES_PASSWORD=pw postgres:17
sleep 5
podman exec src psql -U postgres -c "
  CREATE SCHEMA audit;
  CREATE TABLE audit.events (id int);
  CREATE TABLE public.t (id int);
  INSERT INTO public.t SELECT generate_series(1, 100000);"

cat > filters.ini <<'EOF'
[exclude-schema]
audit
EOF

SRC=postgres://postgres:pw@src/postgres
TGT=postgres://postgres:pw@tgt/postgres

# 1. A filtered clone completes fine.
podman run --rm --network pgnet -v ./filters.ini:/filters.ini:ro -v work:/work \
  <pgcopydb-0.18-image> pgcopydb clone --source $SRC --target $TGT \
  --dir /work/m --filters /filters.ini

# 2. One bare list progress against the same work dir. The command fails on
#    its own (exit 12, "[SQLite] no such column: bytes", that is #1036) but
#    has already rewritten the stored filtering by then.
podman run --rm --network pgnet -v work:/work \
  <pgcopydb-0.18-image> pgcopydb list progress --json --dir /work/m

# 3. Resume the same filtered clone: exit 12, "different filtering".
podman run --rm --network pgnet -v ./filters.ini:/filters.ini:ro -v work:/work \
  <pgcopydb-0.18-image> pgcopydb clone --source $SRC --target $TGT \
  --dir /work/m --filters /filters.ini --resume --not-consistent

Live variant: run step 2 in a loop while the clone from step 1 is still running. Every clone worker exits 12 with "different filtering" and the clone dies.

select filters from setup in schema/source.db (WAL included) makes the rewrite visible:

• before step 2: {"type":"SOURCE_FILTER_TYPE_EXCL","exclude-schema":["audit"]}
• after step 2: {"type":"SOURCE_FILTER_TYPE_EXCL"}

Relation to #1036

I am aware of #1036: on 0.18 list progress always fails outright with [SQLite] no such column: bytes (with and without --json). That actually makes this bug easy to miss. The command errors out, so it looks like a harmless failed read, but by then it has already rewritten the setup record. Monitoring a filtered clone with list progress on 0.18 therefore returns nothing and breaks the migration at the same time.

Expected

A listing command should not modify the catalog setup record. Workers of a running clone --filters, and later resumed runs, should keep matching the record their supervisor registered.

Suggested fix

Either set skipFilterCheck for list progress (it only reads), or keep the Case 3 adopted filtering in memory instead of writing it back to the catalog.

Happy to test a patch, and thanks for pgcopydb.