Conversation
mikeshootzz
force-pushed
the
adr-backup-encryption
branch
from
b06a4e3 to
80c3680
Compare
mikeshootzz
requested review from
a team,
Kidswiss,
TheBigLee,
mdnix and
zugao
and removed request for
a team
|
|
||
| == Decision | ||
|
|
||
| We will use rclone as an S3 encryption proxy to provide encrypted backups for MariaDB Galera and PostgreSQL services. Rclone will be deployed as a separate service in each instance namespace, configured with crypt remotes to transparently encrypt all backup data before it reaches the object storage backend. |
There was a problem hiding this comment.
Have you created a little PoC to test backup and restore with CNPG or Mariadb using the rclone solution?
There was a problem hiding this comment.
I agree, a quick test if the operators are actually compatible with this would be nice.
It doesn't even have to be via AppCat. Simply test it directly with cnpg and mariadb-operator resources.
There was a problem hiding this comment.
I agree here, we need a solid PoC, without AppCat. We have to use the CRDs from the operator to see that:
- Backup is happening and we confirm (via S3cmd for instance) the data is encrypted, safe and can be decoded.
- If I point to that S3 endpoint the restore is happening without issues.
There was a problem hiding this comment.
+1 to that from my side. Also testing the throughput with various resource settings (especially cpu limits) should be done in the PoC as well
| @@ -0,0 +1,133 @@ | |||
| = ADR 42 - Backup Encryption for MariaDB Galera and PostgreSQL | |||
There was a problem hiding this comment.
| = ADR 42 - Backup Encryption for MariaDB Galera and PostgreSQL | |
| = ADR 0042 - Backup Encryption for MariaDB Galera and PostgreSQL |
| ** xref:adr/0037-mariadb-bitnami-replacement.adoc[] | ||
| ** xref:adr/0038-appcat-redis-alternative.adoc[] No newline at end of file | ||
| ** xref:adr/0038-appcat-redis-alternative.adoc[] | ||
| ** xref:adr/42-backup-encryption-for-mariadb-galera-and-postgresql.adoc[] No newline at end of file |
There was a problem hiding this comment.
You will also have to add your document to: docs/modules/ROOT/pages/adr/index.adoc
And for consistency change the filename to 0042-backup-encryption-for-mariadb-galera-and-postgresql.adoc
|
|
||
| With the decision to use rclone as an encryption proxy for MariaDB Galera and PostgreSQL backups, we need to: | ||
|
|
||
| * Implement rclone deployment manifests for each database instance namespace |
There was a problem hiding this comment.
We might want to evaluate if we could use a side care instead of a deployment or other solutions such as a controller or an operator. Having a deployment per each instance can and will increase the resource consumption
There was a problem hiding this comment.
Or as sort of a prebackup pod that only gets started when an actual backup is run (no idea how feasible this actually is though)
zugao
left a comment
zugao
left a comment
There was a problem hiding this comment.
I would further investigate if we could use a controller or even an operator that would stay in between the service operator and crossplane. It would manage perhaps via a CR the layer in between XObjectBucket and composite.
|
|
||
| With the decision to use rclone as an encryption proxy for MariaDB Galera and PostgreSQL backups, we need to: | ||
|
|
||
| * Implement rclone deployment manifests for each database instance namespace |
There was a problem hiding this comment.
Or as sort of a prebackup pod that only gets started when an actual backup is run (no idea how feasible this actually is though)
| * Implement rclone deployment manifests for each database instance namespace | ||
| * Configure CNPG Barman and MariaDB Operator to use the rclone endpoint instead of direct S3 access | ||
| * Establish secret management for rclone encryption passwords | ||
| * Monitor rclone proxy performance and availability alongside database services |
There was a problem hiding this comment.
The performance really needs to be analyzed thoroughly. We want to use the least amount of resources as possible to not have too much overhead. However, it needs to be enough, as to not stall the backup data transfer.
|
|
||
| == Decision | ||
|
|
||
| We will use rclone as an S3 encryption proxy to provide encrypted backups for MariaDB Galera and PostgreSQL services. Rclone will be deployed as a separate service in each instance namespace, configured with crypt remotes to transparently encrypt all backup data before it reaches the object storage backend. |
There was a problem hiding this comment.
+1 to that from my side. Also testing the throughput with various resource settings (especially cpu limits) should be done in the PoC as well
mikeshootzz
force-pushed
the
adr-backup-encryption
branch
from
828d60f to
18e7b24
Compare
tobru
left a comment
tobru
left a comment
There was a problem hiding this comment.
Great research and architecture document; thanks a lot for putting the effort into it!
My comments mostly relate to improve the structure of the document, so that it follows ADR 0001 more closely.
I like the idea of putting rclone in between the backup source and destination. This could - in a future iteration - allow storing backups on other targets than just object storage, depending on customers requests and needs!
|
|
||
| === Constellation s3proxy - not viable | ||
|
|
||
| The s3proxy is part of the Constellation Kubernetes engine and provides transparent client-side encryption for S3-compatible storage backends. This would enable seamless integration with PostgreSQL and Galera. |
There was a problem hiding this comment.
Please add links to the mentioned tools for proper context, e.g.
https://s3proxy.com[s3proxy^]
|
|
||
| We want to provide encrypted backups for all AppCat services. The backup solutions for MariaDB and PostgreSQL are tightly integrated with their respective operators, which prevents us from using k8up. Neither the MariaDB Operator nor the PostgreSQL Operator support encrypted backups natively, so we need an alternative solution. | ||
|
|
||
| == Evaluated Options |
There was a problem hiding this comment.
This should be part of "Context" to keep the ADR structure intact, so:
| == Evaluated Options | |
| === Evaluated Options |
And of course the subsequent headers need to be aligned
|
|
||
| == Evaluated Options | ||
|
|
||
| === Constellation s3proxy - not viable |
There was a problem hiding this comment.
Do not add any judgment in the title yet; this comes in the conclusion section. The "Context" section is neutral in that regard.
| === Constellation s3proxy - not viable | |
| === Constellation s3proxy |
|
|
||
| ==== Storing encryption passwords | ||
|
|
||
| Since we don't plan to rotate the password used for the encryption key, the password can be saved inside of a Kubernetes secret. This is also how we do it with the repository password for K8up. |
There was a problem hiding this comment.
Just to keep in mind, password rotation could be required should we somehow leak the password or be hacked.
|
|
||
| === Rclone - recommended | ||
|
|
||
| Rclone is a command-line program for managing files on cloud storage, supporting various backends including S3-compatible storage. Using `rclone serve` with crypt remotes, we can implement a transparent encryption proxy. |
There was a problem hiding this comment.
Same here as above, link to external resources where it makes sense, e.g. to the rclone website. This is very important to get the correct context when reading.
|
|
||
| This is the case for both CNPG and MariaDB. | ||
|
|
||
| == Using Rclone with CNPG and MariaDB |
There was a problem hiding this comment.
| == Using Rclone with CNPG and MariaDB | |
| == Using Rclone with CloudNativePG and MariaDB |
Abbreviations in titles are not always good.
6 participants
Summary
Documents the decision to use rclone as an S3 encryption proxy for database backups. Rclone will deploy per-instance to transparently encrypt all backup data before it reaches object storage, working with both CNPG and MariaDB Operator.