fix(tracking): S3 tracking server fails on first start with 'no such table: project' (#574)

[vaquarkhan]

Root Cause
Bug 1 — no such table: project:

On a fresh container/EKS deploy with no pre-existing snapshot in S3, the tracking server crashes immediately when the indexer runs:

sqlite3.OperationalError: no such table: project
RegisterTortoise(app, config=..., add_exception_handlers=True) does not generate schemas by default (generate_schemas defaults to False). On first start there is no snapshot DB to download, so the SQLite file is created empty (no tables). When sync_index() calls backend.update() → _update_projects() → Project.all(), it hits a nonexistent table.

Bug 2 — indexer silently drops old logs:

The max_paths batch cap in _gather_paths_to_update had a broken break that only exited the inner for loop over a single S3 page, not the outer paginator loop. The paginator continued fetching additional pages, collecting far more than max_paths files. The watermark then advanced to the last file in this oversized batch, permanently skipping files that fell between position max_paths and the actual end of the batch on subsequent cycles.

Fix
Bug 1: Call Tortoise.generate_schemas(safe=True) inside the lifespan after RegisterTortoise enters. safe=True uses CREATE TABLE IF NOT EXISTS, so it:

Creates tables on first start (no snapshot)
Is a no-op when tables already exist from a downloaded snapshot
Never clobbers existing data
Bug 2: Add a cap_reached flag that breaks the outer paginator loop when the inner break fires. The batch is now truly capped at max_paths, and the watermark only advances to the last file in a correctly-sized batch.

Files Changed
backend.py
— schema generation in lifespan (Bug 1); paginator break fix (Bug 2)
test_s3_backend_bug574.py
— regression tests (moto-based)
Tests
test_generate_schemas_safe_true_creates_tables — verifies tables are created on empty DB
test_generate_schemas_safe_true_does_not_clobber_existing — verifies safe=True preserves snapshot data
test_gather_paths_respects_max_paths_cap — verifies batch cap works (requires moto)
test_watermark_advances_only_to_last_indexed_file — verifies watermark correctness (requires moto)
Bug 1 tests pass locally. Bug 2 tests require moto (marked skip if not installed; will run in CI).

Fixes #574

[Haricharanpanjwani]

Thanks for putting this together @vaquarkhan. I think the runtime fix is headed in the right direction, especially the generate_schemas(safe=True) change for the fresh-start/no-snapshot case.

I do have two concerns around test coverage:

1. The new regression file does not appear to be exercised by CI. In .github/workflows/python-package.yml, the S3 tracking job still only runs tests/tracking/test_bip0042_s3_buffering.py, so tests/tracking/test_s3_backend_bug574.py is not currently part of the tracked workflow.

2. I don’t think the bug-2 tests actually reproduce the paginator bug from Burr S3 Tracking - Deployment and Docker Image #574: test_gather_paths_respects_max_paths_cap() creates only 15 objects and does not force multi-page pagination. The original bug was that hitting max_paths only broke the inner loop, while the outer paginator loop kept fetching later pages. If everything comes back in one page, the pre-fix code would still pass this test.

   • test_watermark_advances_only_to_last_indexed_file() never exercises _gather_paths_to_update() or _scan_and_update_db_for_project(); it just computes max(...) over a handcrafted slice, so it doesn’t validate the actual watermark advancement behavior under the broken pagination path.

Would you mind tightening this by:

• wiring the new regression file into CI, and
• adding a test that forces multiple paginator pages and then verifies we stop after max_paths and persist the expected watermark?

Once that’s in place, I think this will be in much better shape.