feat: add checkpointing into the backfill so we can resume from the l…

[knrc]

…ast completed, contiguous index

Closes #2279

Summary

This pull request adds support for checkpointing the progress of the backfill process, allowing the process to resume from the last known, contiguous index. This addresses the issue of the backfill reprocessing information from the beginning each time it runs.

Release Note

Added support for checkpointing progress during the backfill operation, rerunning the backfill will resume from the last known, contiguous index instead of starting at the beginning.

Documentation

[codecov]

Codecov Report

❌ Patch coverage is 0% with 158 lines in your changes missing coverage. Please review.
✅ Project coverage is 26.03%. Comparing base (488eb97) to head (63b3c04).
⚠️ Report is 637 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/backfill-index/main.go	0.00%	158 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2762       +/-   ##
===========================================
- Coverage   66.46%   26.03%   -40.43%     
===========================================
  Files          92      191       +99     
  Lines        9258    20236    +10978     
===========================================
- Hits         6153     5269      -884     
- Misses       2359    14138    +11779     
- Partials      746      829       +83     

Flag	Coverage Δ
e2etests	49.49% <ø> (+1.93%)	⬆️
unittests	16.63% <0.00%> (-31.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.