Log rotation in multi replica environment

[pavigor]

Describe your Issue

I have NC deployed in 2 replicas, each replica has it's own ephemeral storage for logs.

`logfile` => `/tmp/nextcloud`,
'log_rotate_size' => 10485760,
'logfile_audit' => '/tmp/audit.log'

I faced the problem that CronJob can't do log rotation, because it doesn't have access to each ephemeral storage.

My plan B is to write logs from each replica into common PVC, but then I can't differentiate logs from replicas, it doesn't have field like host / hostname.

Any ideas ?

[qdii]

The Kubernetes way of doing this is to simply let each container log to stdout, then use another service to collect, store, parse and display the logs.

I personally use kubetail which creates a daemon on each node to collect the logs and a dashboard to visualize/filter/restrict source.

Other options include fluentd, elasticsearch, etc.

[pavigor]

Hi @qdii ,

stdout is not suitable for me, because apache is already write to stdout, and I don't want to combine structed NC log and unstructed apache's

[KarlJorgensen]

The canonical way to do this in kubernetes is to run different apps in different containers - this allows their outputs to be separated (remember: logging happens from each container - not from the pod).
Admittedly, that would require nextcloud to run Apache in a separate container, which the current helm chart does not :-(

[qdii]

I agree with @KarlJorgensen here. The correct way would be to run Apache in a different container: not only would it improve logging, it would also be better for many other reasons (for instance, setting resource limits/requests separately).

@pavigor a hacky way would be to have a sidecar container that continuously reads from the cronjob /tmp/audit.log, and outputs to /pvc/$(hostname)/audit.log or similar . The helm chart contains nextcloud-logger in comments that is a good starting point.