Skip to content

rework message queue event model#124

Merged
vredchenko merged 12 commits intomainfrom
rework-mq-events
Jul 14, 2025
Merged

rework message queue event model#124
vredchenko merged 12 commits intomainfrom
rework-mq-events

Conversation

@vredchenko
Copy link
Collaborator

@vredchenko vredchenko commented Jun 30, 2025

Initial MQ events need to be reworked.

  • add missing DB indexes, Add missing indexes for 4 DB tables smartem-devtools#44
  • The API now mutates DB directly, bypassing the MQ - but it still needs to alert consumers on the MQ to every DB mutating operation
  • attach decision-making logic placeholders to listen to MQ events
    • initialise prediction model weights; inputs are: prediction_model_name and grid_uuid; would be the first thing and happen once per model per grid at the start
    • generate random model predictions; inputs are: prediction_model_name, grid_uuid, level=hole|square; when gridsquares and foilholes are first registered they need initial predictions
    • random prior updates; inputs are: grid_uuid, origin=motion_correction|?; will come from whatever metric we decide to use as feedback. The origin would likely be motion correction, ctf or particle picking, but there are a few options for metrics to send back
    • register quality prediction model; inputs are: model_name, model_description; would only happen as a setup stage, probably by hand but could be automated. I think to avoid complications any existing sessions would not be touched by that
  • clean up HTTP API DB_WRITE_MODE - only direct mode remains
  • clean up mq_consumer.py
  • update docker container (build new version and publish to reflect changes in this PR)
  • test that watcher and not just parser works correctly and persists to DB: test EPU agent watcher persists data to DB (not just parser) smartem-devtools#50

@vredchenko vredchenko force-pushed the rework-mq-events branch 3 times, most recently from 37cda1b to 947ceaa Compare July 4, 2025 15:28
@d-j-hatton
Copy link
Contributor

The placeholder logic seems correct to me. The only suggestion is the one above which is something I should have done in the original. It just makes sure that the initial weights are evenly distributed against whatever models are registered and that the weights sum to 1

@vredchenko vredchenko changed the title [WIP]: rework message queue event model rework message queue event model Jul 14, 2025
@vredchenko vredchenko merged commit d11a987 into main Jul 14, 2025
11 checks passed
@vredchenko vredchenko deleted the rework-mq-events branch July 14, 2025 10:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants