Skip to content

Set up sqlite database and replicate to SpacesΒ #2

@AetherUnbound

Description

@AetherUnbound

We're hoping that the Airflow instance will be ephemeral and so we need a mechanism for managing data that needs to be backed up (e.g. the metadata database).

One quick way to do this to start would be use an SQLite database rather than Postgres, then set up an hourly DAG that copies the database file to Spaces. That would provide an easy recovery mechanism for us if we needed to tear down the instance.

Note that the official Airflow docs specify to never use sqlite for production, so ideally we'll have a better setup with PG WAL replication or something down the line (CC @mepholic @mpuckett159).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    πŸ“‹ Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions