Skip to content

Complete the transition of the ledger to a database #91

@transientlunatic

Description

@transientlunatic

We should complete the database interface and make it extensible as a persistent storage back-end for the system, so that we can easily change to a different database server in the future if required.
At present there's a partial implementation, but this should be completed.
We should also create a backend for a database system with a proper transactional system to avoid threading issues.

This will likely require the final transition of remaining objects (Subjects, Analyses, etc) to be transitioned to a strict Pydantic schema, and this will need to be appropriately documented.
Prior handling is now already schematised.

Individual pipelines may need their own schema, since each pipeline might define settings unique to itself which we do not want to control for within asimov; this poses an interesting challenge: if we are to use a database with a schema how do we maintain this flexibility?

Success criteria

  • The asimov ledger is reimplemented in a database with strong protections to prevent race conditions in multi-threaded processes
  • The asimov ledger can be queried to find analyses which suit certain criteria
  • The ledger is fully documented in the sphinx documentation for the project
  • The ledger is fully unit tested
  • The interface for the ledger is generalised; we need a reference implementation (e.g. SQLite or Postgres), but this should be extensible for larger production environments
  • End-to-end tests are implemented with HTCondor and github actions which use the database back-end for the ledger
  • The ledger is fully integrated with the RESTful API

Metadata

Metadata

Labels

ledgerPersistent storage for analysis configurations

Projects

Status

Todo

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions