-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Description
We're hoping that the Airflow instance will be ephemeral and so we need a mechanism for managing data that needs to be backed up (e.g. the metadata database).
One quick way to do this to start would be use an SQLite database rather than Postgres, then set up an hourly DAG that copies the database file to Spaces. That would provide an easy recovery mechanism for us if we needed to tear down the instance.
Note that the official Airflow docs specify to never use sqlite for production, so ideally we'll have a better setup with PG WAL replication or something down the line (CC @mepholic @mpuckett159).
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Status
π Backlog