Proper way to back up Ghost when using a `distroless` image

[ktlast]

Greetings!

To be clear, this is more of a question or discussion than a direct feature request.

Is your feature request related to a problem? Please describe.
Since the image is hardened by using a distroless base image, no commands can be executed inside the Ghost pod. This behavior is expected, as it aligns with the security goals of distroless.
But I found that it can be tedious to do a periodically backup from UI, which is the only way to backup the ghost as far as I know.

Describe the solution you'd like
I don't have a good idea so far about what kind of solutions would be better. I just hope there is a better way to automate backup periodically via CronJob or something similar without compromising security.

Describe alternatives you've considered
I have tried introducing a sidecar in the deployment to do the backup job by the ghost command after altered certain permission settings. But this reintroduce security risk that the hardened image was designed to mitigate in the first place.

---

Thank you for taking the time to read this issue 😊.

[ngeorger]

Greetings!

To be clear, this is more of a question or discussion than a direct feature request.

Is your feature request related to a problem? Please describe. Since the image is hardened by using a distroless base image, no commands can be executed inside the Ghost pod. This behavior is expected, as it aligns with the security goals of distroless. But I found that it can be tedious to do a periodically backup from UI, which is the only way to backup the ghost as far as I know.

Describe the solution you'd like I don't have a good idea so far about what kind of solutions would be better. I just hope there is a better way to automate backup periodically via CronJob or something similar without compromising security.

Describe alternatives you've considered I have tried introducing a sidecar in the deployment to do the backup job by the ghost command after altered certain permission settings. But this reintroduce security risk that the hardened image was designed to mitigate in the first place.

Thank you for taking the time to read this issue 😊.

Hi!

Thanks a lot for your suggestions and enthusiasm.

The basic idea of this implementation, as you mentioned, is to reduce the "surface" of potential security risks, so any interaction via shell is discouraged, but depending on what we want to achieve -and how- we have some options.

• First of all, there is a particular reason for you to use the ghost-cli command in order to create backups?

• In my case, I create backups for the volumes (both ghost and mysql) so they're more aligned with the "kubernetes logic", but is directly aligned on your storage provisioning.

• Other option could be, as you mentioned, a separate cronjob which mounts the ghost's volume and performs the required or desired backup operations as needed.

I think your suggestion is pretty good and we should get to a "general" and simple approach for an optional backup feature which operates separately from the Ghost pod.

PD: Now we also have a -dev image for every build, which has a shell and is not intended for production, if you want to try it out.

Please, tell me what do you think, it's really appreciated.

[ktlast]

My bad, I forgot to provide the background info of my architecture.

If we are running Ghost in a single docker environment, then

create backups for the volumes (both ghost and mysql)

is totally reasonable and understandable 🫡.

Context

The reason I am a bit confused is that I just started running multiple instances of Ghost with a shared MySQL in my small k3s cluster to let me separate different blog to record different topics.

What I (would) have

• a personal blog for tech notes
• another site as a knowledge base for my side project
• maybe one more potential Ghost site to record some daily stuff
• maybe another site for my friend.

The traffic flow is somewhat like

Browser
    -> DNS 
    -> k3s node
    -> Gateway API (Gateway and HTTPRoute)
    -> services
    -> deployments

And there are 3 k3s nodes with Longhorn installed in the cluster.

Everything works fine. I am happy, and ready to migrate more posts from other blog SaaS.

Backup

Then I stuck in how to backup content properly.

According to the official document, I thought using ghost backup and then send the zip to S3 is the easiest way to go. In this way, I could even set a different backup periods and storage destinations for different instances.

After found that there is no ghost CLI in the pod and tried introducing a sidecar, I think I should look for more proper way to do this.

I am willing to wrap some shell scripts that cover the manual backup methods from the official document to backup the multiple Ghost instances without ghost CLI, but before developing anything, I think it would be better to check if there is any better design 😊.

[ngeorger]

And there are 3 k3s nodes with Longhorn installed in the cluster.

My setup is quite similar and in my case I use Longhorn features to backup the persistent volumes of mysql and each ghost instance. In my case, I use a Google Bucket as S3 https://longhorn.io/docs/1.10.1/snapshots-and-backups/backup-and-restore/set-backup-target/

I'll add some ideas ASAP in this thread, I wanted to give you a solution in the meantime 👍🏻

[ngeorger]

Context

The reason I am a bit confused is that I just started running multiple instances of Ghost with a shared MySQL in my small k3s cluster to let me separate different blog to record different topics.

• In my personal opinion, if you are running a shared mysql server, you should implement a database-level backups logic, by scripting cron based sql dumps or something more "elegant", separated from your backup logic for each Ghost instance itself, because you are now serving different "blogs" with the same instance of mysql and the ghost-cli tool could mess up a lot of thing just by having one wrong parameter (I will get to this point later).

What I (would) have

• a personal blog for tech notes
• another site as a knowledge base for my side project
• maybe one more potential Ghost site to record some daily stuff
• maybe another site for my friend.

The traffic flow is somewhat like

Browser
    -> DNS 
    -> k3s node
    -> Gateway API (Gateway and HTTPRoute)
    -> services
    -> deployments

And there are 3 k3s nodes with Longhorn installed in the cluster.

Everything works fine. I am happy, and ready to migrate more posts from other blog SaaS.

• Take a look for a clustered mysql setup 🤓 https://www.sredevops.org/en/mysql-8-2-is-what-we-were-all-waiting-for-for-kubernetes-with-transparent-read-write-splitting/

Backup

Then I stuck in how to backup content properly.

According to the official document, I thought using ghost backup and then send the zip to S3 is the easiest way to go. In this way, I could even set a different backup periods and storage destinations for different instances.

• Their docs are meant to be used with one of their supported install method and for a single instance of Ghost, mostly inside a virtual machine and/or a docker setup in a single machine.

I think the best way to backup both mysql and ghost is leveraging Longhorn for that purpose:

• Examples from my own setup (small k3s cluster):

[ktlast]

Wow I see!

Thank you for sharing detailed information. That helps a lot for clarifying a suitable way to do the backups.

What I thought was:

• Ghost would have both DB data and volume data (files and folders) at the same time.
• So it would be better backup via their CLI to make sure everything works fine easier.

But clearly, it's even better to just backup and restore everything from longhorn 😆

One more thing I want to check if this is correct:

• Backup with longhorn volume means that I would need to backup all sites at the same time, and it's not possible (or not easy enough) to backup different sites with different periods and conditions.

I am just a bit curious about this, as I can directly backup everything then somewhat like fire and forget, and it's still going to be fine.

Other questions about this repo:

1. I guess the backup mechanism is not necessary to be a feature then?
2. I can open a PR and add some Helm templates for this repo, but I am not sure if this is a good idea, as its current scope is just covered what I need (e.g., Gateway API), and it might be not helpful to others.

[ngeorger]

One more thing I want to check if this is correct:

• Backup with longhorn volume means that I would need to backup all sites at the same time, and it's not possible (or not easy enough) to backup different sites with different periods and conditions.

I am just a bit curious about this, as I can directly backup everything then somewhat like fire and forget, and it's still going to be fine.

Depends on how you configure the backups, you can create different targets for each volume and also automate those backups with a cron based logic. Please refer to Longhorn Docs:

• https://longhorn.io/docs/1.10.1/snapshots-and-backups/backup-and-restore/
• https://longhorn.io/docs/1.10.1/snapshots-and-backups/scheduling-backups-and-snapshots/#set-up-recurring-jobs

Other questions about this repo:

1. I guess the backup mechanism is not necessary to be a feature then?

I think it could be a feature, but the potential complexity for every CSI and every provider, seems to be out of the scope of this project, but I think some documentation and/or general guidelines for backing up, should be added in the near future.

2. I can open a PR and add some Helm templates for this repo, but I am not sure if this is a good idea, as its current scope is just covered what I need (e.g., Gateway API), and it might be not helpful to others.

I've tried to create a Helm Chart some time ago but I didn't have enough time to maintain it and decided to focus on a manifest logic instead. Any contributions or new ideas are highly appreciated and welcome! At least for me it would be really helpful.

[ngeorger]

This could be a good complementary reading: https://www.jannis.io/why-distributed-storage-matters-even-when-it-hurts/