Skip to content

EPU agent data to DB via HTTP API and RabbitMQ#92

Merged
vredchenko merged 15 commits intomainfrom
http-api-full-crud
May 22, 2025
Merged

EPU agent data to DB via HTTP API and RabbitMQ#92
vredchenko merged 15 commits intomainfrom
http-api-full-crud

Conversation

@vredchenko
Copy link
Collaborator

@vredchenko vredchenko commented Apr 17, 2025

  • extend HTTP API with full CRUD endpoint placeholders
  • update DB schema to conform to shape of data found in EPU Agent parsers
  • HTTP API endpoints publish updates to MQ
  • MQ consumer propagates updates to DB
    • Acquisition
    • Grid
    • GridSquare
      • fix: No handler registered for event type: grid_square.updated
    • Foilhole
    • Micrograph
  • instrument EPU Agent to send data to the API
    • generate entity IDs agent-side using uuid and modify DB model to accept these
    • and for gridsquares, foilholes and micrographs (and possibly acquisition and grid, too) - ensure there's no
      conflict between IDs we auto-generate and use as PKs in DB and IDs generated by EPU
    • Entities:
      • Acquisition
      • Grid
      • GridSquare
      • Foilhole
      • Micrograph
  • add status and health endpoints to Core HTTP API
  • clean up src/smartem_decisions/model/mq_event.py and src/smartem_decisions/workflow.py
  • add debug logging level to CLI and change info statements to debug where appropriate
  • refactor starting at RateLimitedFilesystemEventHandler.on_any_event and update to align with the new datastore
  • add verbosity param to parse CLI

@vredchenko vredchenko self-assigned this Apr 17, 2025
@vredchenko vredchenko force-pushed the http-api-full-crud branch 11 times, most recently from d969efb to 8253569 Compare April 23, 2025 14:54
@vredchenko
Copy link
Collaborator Author

vredchenko commented Apr 23, 2025

test: python -m epu_data_intake watch -vv ../smartem-decisions-test-datasets/bi37708-28-copy

@vredchenko vredchenko force-pushed the http-api-full-crud branch 5 times, most recently from c34cad4 to 9332b0a Compare April 30, 2025 15:25
@vredchenko vredchenko force-pushed the http-api-full-crud branch 3 times, most recently from fff8c21 to 9655910 Compare May 7, 2025 11:57
@vredchenko vredchenko force-pushed the http-api-full-crud branch 3 times, most recently from 55605be to 7d6810e Compare May 13, 2025 16:58
@vredchenko vredchenko force-pushed the http-api-full-crud branch 6 times, most recently from 85daa17 to 11192ab Compare May 19, 2025 17:07
@vredchenko vredchenko force-pushed the http-api-full-crud branch from 11192ab to 38207cf Compare May 21, 2025 15:33
@vredchenko vredchenko changed the title [WIP] EPU agent data to DB via HTTP API and RabbitMQ EPU agent data to DB via HTTP API and RabbitMQ May 22, 2025
@vredchenko vredchenko merged commit 82d72db into main May 22, 2025
5 checks passed
@vredchenko vredchenko deleted the http-api-full-crud branch May 22, 2025 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant