k8s friendly Caddyfile consideration

[fkromer]

ATM it's not possible to mount the Caddyfile on a file basis (#364).

When deploying services with k8s it's pretty common to use ConfigMaps as files to provide configuration in a 12 factor app compatible manner.

Would it be possible to add support for file bind mounting? What is the recommended workaround when using Caddy in a k8s context ATM?

[francislavoie]

I don't use k8s so I don't understand the problem. Can you explain in simple terms?

[fkromer]

ATM I try to workaround the limitation of no being able to bind mount the Caddyfile as file in k8s.

To do so I'm

• file mounting the Caddyfile in an init container
• copy the Caddyfile into the /etc/caddy directory via k8s deployment init container command (/etc/caddy is mounted by the init container as emptyDir)
• directory mount /etc/caddy in the k8s deployment caddy container (Caddyfile is available to caddy container)

So far I'm getting

exec container process `/usr/bin/caddy`: Operation not permitted

My current assumption is that I'm not considering Caddyfile permissions properly during copying yet.

[fkromer]

Seems like the root cause for my issue is exclusively security related and could be resolved with #396 as well. (The init container is required nevertheless to being able to set user and user group of the Caddyfile prior to caddy container startup to a non-root user.)

[francislavoie]

We'll be making a rootless image variant soon #428, would that solve it for you? This does complicate binding to low ports though (80/443)

[fkromer]

Adjusting the non-root user and group in a k8s config is no big deal.

The Caddyfile ownership topic remains I guess:

Either the non-root user would need to be able to run caddy with the Caddyfile w.o. specific ownership set.

Or the ownership needs to be set to the non-root user and group beforehand. This is what works with the init container approach. Either permissions are set with the init container command or securityContext > fsGroup can be used to change the permissions of all files in a volume. Both variants are a bit inconvenient.

[francislavoie]

Yes that's the point of the rootless variant, making it so /etc/caddy in the rootless variant is owned by the user instead of root.

[fkromer]

Might relate to Declarative update of configmaps and secrets from the contents of files as well.

[francislavoie]

Alright well I don't really see anything actionable here, I don't really understand what the ask is outside of rootless (which is coming and tracked separately). So I'll close this for now. Can be reopened if you can more clearly explain.