Ability to persist CT indexes and resume from the saved index file #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

d-Rickyy-b merged 17 commits into d-Rickyy-b:master from messede-degod:feat/ct-index-save-restore

Aug 1, 2025

Contributor

messede-degod commented Jan 14, 2025

Allows CT index to persisted.
CT indexes are dumped to disk every 30 seconds
Future sessions resume from last saved CT index.

d-Rickyy-b reviewed

View reviewed changes

internal/certificatetransparency/ct-watcher.go Outdated Show resolved Hide resolved

d-Rickyy-b reviewed

View reviewed changes

internal/certificatetransparency/ct-watcher.go Show resolved Hide resolved

d-Rickyy-b reviewed

View reviewed changes

internal/certificatetransparency/ct-watcher.go Outdated Show resolved Hide resolved

d-Rickyy-b reviewed

View reviewed changes

internal/certificatetransparency/logmetrics.go Outdated Show resolved Hide resolved

d-Rickyy-b reviewed

View reviewed changes

internal/certificatetransparency/logmetrics.go Outdated Show resolved Hide resolved

d-Rickyy-b reviewed

View reviewed changes

internal/certificatetransparency/logmetrics.go Outdated Show resolved Hide resolved

d-Rickyy-b reviewed

View reviewed changes

internal/config/config.go Outdated Show resolved Hide resolved

Owner

d-Rickyy-b commented Jan 15, 2025

Thank you for your contribution :) I appreciate it a lot. I left some comments on the code. Feel free to reply and/or fix.

Contributor Author

messede-degod commented Jan 16, 2025

I've addressed all your comments, please do take a look and let me know what you think.

messede-degod changed the title ~~Ability to persist CT indexes and resume from the save index file~~ Ability to persist CT indexes and resume from the saved index file

Owner

d-Rickyy-b commented Feb 11, 2025

Quick update: I did not forget about this PR, I just had a lot of life going on. Tonight I thought about this PR a bit, and some things need a few more thoughts.

start_at_latest_sth and resume_from_ct_index_file should be mutually exclusive, shouldn't they? So why have two variables?
There should be a final save on exit. For that we need some signal handling (Gracefully shutdown certstream-server-go #50). I'll add that in a separate commit on master. For now let's just assume that the Stop() function of the ct-watcher gets called on a graceful shutdown
The index file should be auto-created if it doesn't exist yet
If no setting is defined, the default should be to use the latest STH

I'll add these missing features asap

messede-degod and others added 14 commits

May 4, 2025 17:38


          add: ability to persist and resume from CT indexes

9c0398d


          update: disable CT index persistenc by default

fd8b87c


          update: refactor code, add start_at_latest_sth

cb210ad


          fix: race in GetAllCTIndexes()

bb52131


          update: remove unnecessary index log

eb969db


          chore: add temp files and ct_index.json to gitignore

6887cdb


          refactor: use better names for recovery behavior

ae57b02


          refactor: add proper error handling

9386c60


          refactor: rename fileName to filePath


          refactor: pass ctIndexFilePath to LoadCTIndex function

ac74794


          refactor: create ctIndexFile if not exists

37269b7


          refactor: use new config variables

ed649ea


          refactor: extract saving file into own method

44bd15c


          feat: add feature to create initial index file

4bfea45

This is useful in order to not start pulling each log after the first run at index 0 but still have newly discovered logs download from 0.

d-Rickyy-b force-pushed the feat/ct-index-save-restore branch from 9d6d11f to 4bfea45 Compare

May 5, 2025 00:38

d-Rickyy-b added 3 commits

May 5, 2025 02:40


          chore: add missing trailing newline to gitignore

ef38487


          docs: describe recovery feature in changelog

ab49770


          refactor: remove nested loop and initialize metrics with worker

d646312

Owner

d-Rickyy-b commented May 5, 2025

Sorry for the force push. Was the easiest way to get the rebase done. This PR should now be ready to be merged. Mind taking a look at the new changes and stating your opinion? Thanks a lot!

Contributor Author

messede-degod commented May 14, 2025

looks great, but i have to admit that it looks nothing like what it started off as 😄 .

d-Rickyy-b merged commit bf24abb into d-Rickyy-b:master

3 checks passed

messede-degod mentioned this pull request

Save the last processed index of the certstream log when certstream-server-go exits #46

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet