Skip to content
This repository was archived by the owner on Feb 6, 2026. It is now read-only.

feat: add func runs and func run logs migration to sdf #8322

Merged
vbustamante merged 5 commits intomainfrom
victor/eng-3338-migrate-func-runs-and-func-run-logs-back-to-the-si-database
Jan 23, 2026
Merged

feat: add func runs and func run logs migration to sdf #8322
vbustamante merged 5 commits intomainfrom
victor/eng-3338-migrate-func-runs-and-func-run-logs-back-to-the-si-database

Conversation

@vbustamante
Copy link
Contributor

@vbustamante vbustamante commented Jan 21, 2026

How does this PR change the system?

  • Add a new database struct for func runs and func run logs that uses si-db as a backend
  • Implement a migration mode on sdf that migrates func runs and func run logs in batches

The migration is inspired by the layer-db -> S3 backfill by has some key differences:

  • It runs the batches sequentially, creating one parallel task per table
  • It runs the batches as a single INSERT query

Like it's predecessor, it checks that the entries have already been copied before inserting, and every 10 seconds it'll print out the latest ID it has inserted so we can easily pick up on failed jobs.

to run the migration mode do

buck2 run @//mode/release //bin/sdf:sdf -- -vv --migration-mode backfillFuncRuns  --backfill-key-batch-size 10

The usages have been updated, but the new client uses a fallback strategy so we can migrate with as few access errors as possible. it will right to both databases, and read on the layerdb if it fails to find data on the si-db.

How was it tested?

  • Integration tests still pass
  • Manual test: fill up the layer cache database with func runs and func run logs, then run the migration

Does it require a docs change?

  • No

@vbustamante vbustamante requested a review from jhelwig January 21, 2026 22:11
@github-actions
Copy link

github-actions bot commented Jan 21, 2026

Dependency Review

✅ No vulnerabilities or OpenSSF Scorecard issues found.

Scanned Files

None

@github-actions
Copy link

There are luminork endpoint changes on this branch that do not appear to have associated tests. Please ensure you add or update tests under bin/si-luminork-api-tests/tests/ to satisfy this warning.

@vbustamante vbustamante changed the title feat feat: add func runs and func run logs migration to sdf Jan 21, 2026
@vbustamante vbustamante force-pushed the victor/eng-3338-migrate-func-runs-and-func-run-logs-back-to-the-si-database branch 2 times, most recently from 5df62de to 302eaad Compare January 22, 2026 17:59
@vbustamante
Copy link
Contributor Author

/try

@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Okay, starting a try! I'll update this comment once it's running...
🚀 Try running here! 🚀

@vbustamante vbustamante force-pushed the victor/eng-3338-migrate-func-runs-and-func-run-logs-back-to-the-si-database branch from 302eaad to c40d4df Compare January 23, 2026 00:10
@vbustamante vbustamante force-pushed the victor/eng-3338-migrate-func-runs-and-func-run-logs-back-to-the-si-database branch from c40d4df to 4017dad Compare January 23, 2026 00:12
@vbustamante
Copy link
Contributor Author

/try

@github-actions
Copy link

github-actions bot commented Jan 23, 2026

Okay, starting a try! I'll update this comment once it's running...
🚀 Try running here! 🚀

@vbustamante vbustamante force-pushed the victor/eng-3338-migrate-func-runs-and-func-run-logs-back-to-the-si-database branch from 8f35f6e to 5890561 Compare January 23, 2026 12:21
@vbustamante
Copy link
Contributor Author

/try

@github-actions
Copy link

github-actions bot commented Jan 23, 2026

Okay, starting a try! I'll update this comment once it's running...
🚀 Try running here! 🚀

@vbustamante vbustamante force-pushed the victor/eng-3338-migrate-func-runs-and-func-run-logs-back-to-the-si-database branch from 5890561 to e91df4c Compare January 23, 2026 12:35
@vbustamante vbustamante force-pushed the victor/eng-3338-migrate-func-runs-and-func-run-logs-back-to-the-si-database branch from e91df4c to e4d015a Compare January 23, 2026 12:43
@vbustamante
Copy link
Contributor Author

/try

@github-actions
Copy link

github-actions bot commented Jan 23, 2026

Okay, starting a try! I'll update this comment once it's running...
🚀 Try running here! 🚀

@vbustamante vbustamante added this pull request to the merge queue Jan 23, 2026
Merged via the queue into main with commit 69ff778 Jan 23, 2026
29 checks passed
@vbustamante vbustamante deleted the victor/eng-3338-migrate-func-runs-and-func-run-logs-back-to-the-si-database branch January 23, 2026 15:19
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants