Summary
On v0.18 (SQLite CDC store), clone --follow --plugin=test_decoding can end up applying nothing while looking healthy: receive and transform work, but every transaction is skipped as "already applied". No error at default log level; the target replication origin never moves off the clone LSN; endpos drains report apply done with zero rows applied.
Environment
pgcopydb v0.18, source PG14 / target PG18 (Cloud SQL, via Auth Proxy), --plugin=test_decoding --no-owner --no-acl --slot-name pgcopydb --origin pgcopydb.
Evidence
- replay.db is fully populated:
SELECT action, count(*) FROM replay GROUP BY action → B|6401 C|6401 U|6399 D|2 I|2 K|44, correct xids/LSNs, correct parameterized SQL in stmt.
- Running
ld_store_replay_next_event()'s own SQL by hand returns thousands of eligible transactions.
- Target:
pg_replication_origin_progress('pgcopydb', true) frozen at the clone LSN; pg_stat_user_tables counters frozen while the source writes ~20 rows/s.
pgcopydb stream sentinel set endpos --current → follow.c:488 Current endpos reached: apply done — still zero rows.
Suspected mechanism (from reading v0.18 source)
ld_store_replay_next_event() returns, in LSN order, an eligible BEGIN or a KEEPALIVE at lsn > previousLSN. Keepalives ride at the stream head while the inline transform lags writing B/C rows. When replay.db is momentarily drained, the KEEPALIVE branch of stream_apply_replaydb() advances previousLSN to the keepalive LSN — past transactions transform has not written yet. When those BEGIN rows land, Guard 1 drops them silently:
/* Guard 1: already applied — should not happen given the query,
* but be safe on restart edge cases. */
if (commitLSN <= context->previousLSN)
{
log_debug("Skip already-applied txn %u commitLSN %X/%X <= previousLSN %X/%X", ...);
continue;
}log_debug — invisible. With a ~10s keepalive cadence the watermark is re-poisoned continuously, so any transform lag means the whole stream is skipped, forever.
This is the same failure family as #274 / #282, previously fixed in the file-based pipeline; the SQLite store appears to have reintroduced a keepalive-flavored variant. (v0.17 on identical infrastructure and flags applies correctly.)
Expected
Transactions present in replay.db with commitLSN > origin progress are applied; a skip of a never-applied transaction should never happen (or at least not at debug level).
We run a fleet of 11 followers and can reproduce at will — happy to test patches.
Summary
On v0.18 (SQLite CDC store),
clone --follow --plugin=test_decodingcan end up applying nothing while looking healthy: receive and transform work, but every transaction is skipped as "already applied". No error at default log level; the target replication origin never moves off the clone LSN;endposdrains reportapply donewith zero rows applied.Environment
pgcopydb v0.18, source PG14 / target PG18 (Cloud SQL, via Auth Proxy),
--plugin=test_decoding --no-owner --no-acl --slot-name pgcopydb --origin pgcopydb.Evidence
SELECT action, count(*) FROM replay GROUP BY action→B|6401 C|6401 U|6399 D|2 I|2 K|44, correct xids/LSNs, correct parameterized SQL instmt.ld_store_replay_next_event()'s own SQL by hand returns thousands of eligible transactions.pg_replication_origin_progress('pgcopydb', true)frozen at the clone LSN;pg_stat_user_tablescounters frozen while the source writes ~20 rows/s.pgcopydb stream sentinel set endpos --current→follow.c:488 Current endpos reached: apply done— still zero rows.Suspected mechanism (from reading v0.18 source)
ld_store_replay_next_event()returns, in LSN order, an eligible BEGIN or a KEEPALIVE atlsn > previousLSN. Keepalives ride at the stream head while the inline transform lags writing B/C rows. When replay.db is momentarily drained, the KEEPALIVE branch ofstream_apply_replaydb()advancespreviousLSNto the keepalive LSN — past transactions transform has not written yet. When those BEGIN rows land, Guard 1 drops them silently:log_debug— invisible. With a ~10s keepalive cadence the watermark is re-poisoned continuously, so any transform lag means the whole stream is skipped, forever.This is the same failure family as #274 / #282, previously fixed in the file-based pipeline; the SQLite store appears to have reintroduced a keepalive-flavored variant. (v0.17 on identical infrastructure and flags applies correctly.)
Expected
Transactions present in replay.db with
commitLSN > origin progressare applied; a skip of a never-applied transaction should never happen (or at least not at debug level).We run a fleet of 11 followers and can reproduce at will — happy to test patches.