enable retention

Signed-off-by: Avi Deitcher <[email protected]>

Author: deitch

README.md

@@ -4,7 +4,7 @@ Back up mysql databases to... anywhere!
 
 ## Overview
 
-mysql-backup is a simple way to do MySQL database backups and restores.
+mysql-backup is a simple way to do MySQL database backups and restores, as well as manage your backups.
 
 It has the following features:
 
@@ -14,6 +14,7 @@ It has the following features:
 * connect to any container running on the same system
 * select how often to run a dump
 * select when to start the first dump, whether time of day or relative to container start time
+* prune backups older than a specific time period or quantity
 
 Please see [CONTRIBUTORS.md](./CONTRIBUTORS.md) for a list of contributors.
 

docs/configuration.md

@@ -58,7 +58,7 @@ Various sample configuration files are available in the [sample-configs](../samp
 
 ## Configuration Options
 
-The following are the environment variables, CLI flags and configuration file options for a backup or a restore.
+The following are the environment variables, CLI flags and configuration file options for: backup(B), restore (R), prune (P).
 
 | Purpose | Backup / Restore | CLI Flag | Env Var | Config Key | Default |
 | --- | --- | --- | --- | --- | --- |
@@ -70,34 +70,26 @@ The following are the environment variables, CLI flags and configuration file op
 | names of databases to exclude from the dump | B | `exclude` | `DB_NAMES_EXCLUDE` | `database.exclude` |  |
 | do not include `USE <database>;` statement in the dump | B | `no-database-name` | `NO_DATABASE_NAME` | `database.no-database-name` | `false` |
 | restore to a specific database | R | `restore --database` | `RESTORE_DATABASE` | `restore.database` |  |
-| how often to do a dump, in minutes | B | `dump --frequency` | `DB_DUMP_FREQ` | `dump.schedule.frequency` | `1440` (in minutes), i.e. once per day |
-| what time to do the first dump | B | `dump --begin` | `DB_DUMP_BEGIN` | `dump.schedule.begin` | `0`, i.e. immediately |
-| cron schedule for dumps | B | `dump --cron` | `DB_DUMP_CRON` | `dump.schedule.cron` |  |
-| run the backup a single time and exit | B | `dump --once` | `RUN_ONCE` | `dump.schedule.once` | `false` |
-| enable debug logging | BR | `debug` | `DEBUG` | `logging: debug` | `false` |
-| where to put the dump file; see [backup](./backup.md) | B | `dump --target` | `DB_DUMP_TARGET` | `dump.targets` |  |
+| how often to do a dump or prune, in minutes | BP | `dump --frequency` | `DB_DUMP_FREQ` | `dump.schedule.frequency` | `1440` (in minutes), i.e. once per day |
+| what time to do the first dump or prune | BP | `dump --begin` | `DB_DUMP_BEGIN` | `dump.schedule.begin` | `0`, i.e. immediately |
+| cron schedule for dumps or prunes | BP | `dump --cron` | `DB_DUMP_CRON` | `dump.schedule.cron` |  |
+| run the backup or prune a single time and exit | BP | `dump --once` | `RUN_ONCE` | `dump.schedule.once` | `false` |
+| enable debug logging | BRP | `debug` | `DEBUG` | `logging: debug` | `false` |
+| where to put the dump file; see [backup](./backup.md) | BP | `dump --target` | `DB_DUMP_TARGET` | `dump.targets` |  |
 | where the restore file exists; see [restore](./restore.md) | R | `restore --target` | `DB_RESTORE_TARGET` | `restore.target` |  |
-| replace any `:` in the dump filename with `-` | B | `dump --safechars` | `DB_DUMP_SAFECHARS` | `database.safechars` | `false` |
-| AWS access key ID, used only if a target does not have one | BR | `aws-access-key-id` | `AWS_ACCESS_KEY_ID` | `dump.targets[s3-target].credentials.access-key-id` |  |
-| AWS secret access key, used only if a target does not have one | BR | `aws-secret-access-key` | `AWS_SECRET_ACCESS_KEY` | `dump.targets[s3-target].credentials.secret-access-key` |  |
-| AWS default region, used only if a target does not have one | BR | `aws-region` | `AWS_REGION` | `dump.targets[s3-target].region` |  |
+| replace any `:` in the dump filename with `-` | BP | `dump --safechars` | `DB_DUMP_SAFECHARS` | `database.safechars` | `false` |
+| AWS access key ID, used only if a target does not have one | BRP | `aws-access-key-id` | `AWS_ACCESS_KEY_ID` | `dump.targets[s3-target].credentials.access-key-id` |  |
+| AWS secret access key, used only if a target does not have one | BRP | `aws-secret-access-key` | `AWS_SECRET_ACCESS_KEY` | `dump.targets[s3-target].credentials.secret-access-key` |  |
+| AWS default region, used only if a target does not have one | BRP | `aws-region` | `AWS_REGION` | `dump.targets[s3-target].region` |  |
 | alternative endpoint URL for S3-interoperable systems, used only if a target does not have one | BR | `aws-endpoint-url` | `AWS_ENDPOINT_URL` | `dump.targets[s3-target].endpoint` |  |
-| SMB username, used only if a target does not have one | BR | `smb-user` | `SMB_USER` | `dump.targets[smb-target].credentials.username` |  |
-| SMB password, used only if a target does not have one | BR | `smb-pass` | `SMB_PASS` | `dump.targets[smb-target].credentials.password` |  |
-| compression to use, one of: `bzip2`, `gzip` | B | `compression` | `COMPRESSION` | `dump.compression` | `gzip` |
+| SMB username, used only if a target does not have one | BRP | `smb-user` | `SMB_USER` | `dump.targets[smb-target].credentials.username` |  |
+| SMB password, used only if a target does not have one | BRP | `smb-pass` | `SMB_PASS` | `dump.targets[smb-target].credentials.password` |  |
+| compression to use, one of: `bzip2`, `gzip` | BP | `compression` | `COMPRESSION` | `dump.compression` | `gzip` |
 | when in container, run the dump or restore with `nice`/`ionice` | BR | `` | `NICE` | `` | `false` |
 | tmp directory to be used during backup creation and other operations | BR | `tmp` | `TMP_PATH` | `tmp` | system-defined |
 | filename to save the target backup file | B | `dump --filename-pattern` | `DB_DUMP_FILENAME_PATTERN` | `dump.filename-pattern` |  |
 | directory with scripts to execute before backup | B | `dump --pre-backup-scripts` | `DB_DUMP_PRE_BACKUP_SCRIPTS` | `dump.scripts.pre-backup` | in container, `/scripts.d/pre-backup/` |
 | directory with scripts to execute after backup | B | `dump --post-backup-scripts` | `DB_DUMP_POST_BACKUP_SCRIPTS` | `dump.scripts.post-backup` | in container, `/scripts.d/post-backup/` |
 | directory with scripts to execute before restore | R | `restore --pre-restore-scripts` | `DB_DUMP_PRE_RESTORE_SCRIPTS` | `dump.pre-restore-scripts` | in container, `/scripts.d/pre-restore/` |
 | directory with scripts to execute after restore | R | `restore --post-restore-scripts` | `DB_DUMP_POST_RESTORE_SCRIPTS` | `dump.post-restore-scripts` | in container, `/scripts.d/post-restore/` |
-
-
-## Unsupported Options
-
-Unsupported options from the old version of `mysql-backup`:
-
-* `MYSQLDUMP_OPTS`: A string of options to pass to `mysqldump`, e.g. `MYSQLDUMP_OPTS="--opt abc --param def --max_allowed_packet=123455678"` will run `mysqldump --opt abc --param def --max_allowed_packet=123455678`. These are replaced by individual options.
-* `AWS_CLI_OPTS`: Additional arguments to be passed to the `aws` part of the `aws s3 cp` command, click [here](https://docs.aws.amazon.com/cli/latest/reference/#options) for a list. These are replaced by target-specific options.
-* `AWS_CLI_S3_CP_OPTS`: Additional arguments to be passed to the `s3 cp` part of the `aws s3 cp` command, click [here](https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html#options) for a list. If you are using AWS KMS, `sse`, `sse-kms-key-id`, etc., may be of interest. These are replaced by target-specific options
+| retention policy for backups | BP | `dump --retention` | `RETENTION` | `prune.retention` | Infinite |

docs/prune.md

@@ -0,0 +1,63 @@
+# Pruning
+
+Pruning is the process of removing backups that no longer are needed.
+
+mysql-backup does **not** do this by default; it is up to you to enable this feature, if you want it.
+
+## Launching Pruning
+
+Pruning happens only in the following scenarios:
+
+* During pruning runs
+* During backup runs
+
+It does not occur during restore runs.
+
+### Pruning Runs
+
+You can start `mysql-backup` with the command `prune` to run a pruning operation. It will prune any backups that are no longer needed.
+
+Like backups, it can run once and then exit, or it can run on a schedule.
+
+It uses the same configuration options for scheduling as backups, see the [scheduling](./scheduling.md) documentation for more information,
+specifically the section about [Scheduling Options](./scheduling.md#scheduling-options).
+
+### Backup Runs
+
+When running `mysql-backup` in backup mode, it _optionally_ can also prune older backups before each backup run.
+When enabled, it will prune any backups that fit the pruning criteria.
+
+## Pruning Criteria
+
+Pruning can be on the basis of the _age_ of a specific backup, or the _number_ of backups. Both are set by the configuration setting:
+
+* Environment variable: `RETENTION=<value>`
+* CLI flag: `dump --retention=<value>` or `prune --retention=<value>`
+* Config file:
+```yaml
+prune:
+    retention: <value>
+```
+
+The value of retention always is an integer followed by a letter. The letter can one of:
+
+* `h` - hours, e.g. `2h`
+* `d` - days, e.g. `3d`
+* `w` - weeks, e.g. `4w`
+* `m` - months, e.g. `3m`
+* `y` - years, e.g. `5y`
+* `c` - count, how many backup to keep, e.g. `10c`; this could have been simply `10`, but was kept as `c` to avoid accidental confusion with the other options.
+
+Most of these are interchangeable, e.g. `3d` is the same as `72h`, and `4w` is the same as `28d` is the same as `672h`.
+
+When calculating whether or not to prune, `mysql-backup` __always__ converts the amount to hours, and then errs on the side of caution.
+For example, if provided `7d`, it will convert that to `168h`, and then prune any backups older than 168 full hours. If it is 167 hours and 59 minutes old, it
+will not be pruned.
+
+## Determining backup age
+
+Pruning depends on the name of the backup file, rather than the timestamp on the target filesystem, as the latter can be unreliable.
+This means that the filename must be of a known pattern.
+
+As of this writing, pruning only work for backup files whose filename uses the default naming scheme, as described in
+["Dump File" in backup documentation](./backup.md#dump-file). We hope to support custom filenames in the future.