podman should log *all* user events (regardless) to enable platform observability/visibility

[larry-cable]

Feature request description

platform monitoring agents may wish to participate in the lifecycle of containers controlled by podman.

In such scenarios such an agent, may wish to monitor all podman activity on the host system, podman 'events' (or their persistent backing store(s)) seem like a suitable mechanism for this integration via the API.

however it appears that the current implementation restricts the event stream resulting to only those related to the the user identity of the API client.

while this is highly desirable nay required behavior for unprivileged clients, platform monitoring agents with privilege should have a mechanism to obtain events for all users, w/o resorting to introspecting on the underlying event "logging" mechanisms utilized by podman to persist event occurrences to the filesystem (e.g file or journal)

Suggest potential solution

since it appears as though (by observed behavior) that the current system filters events based upon the identity of the API client, it would seem appropriate to modify this behavior to permit privileged users to obtain all user events occurring on the system.

Have you considered any alternatives?

w/o this, such agents have to introspect the r/t configuration for the logging implementation and subsequently directly query that in order to obtain all pertinent event information.

this in my opinion, is not a "public" API but an implementation artifact that well designed agent implementations should not resort to in the presence of an existing API that could satisfy the requirement appropriately.

Additional context

Add any other context or screenshots about the feature request here.

[vrothberg]

Thanks for reaching out, @larry-cable.

Privileged users can query the journal (e.g., via journalctl) to see the Podman events of other users. The identifiers used in the journal are documented in the podman-events man page: https://docs.podman.io/en/latest/markdown/podman-events.1.html#journald-identifiers

Note that individual users may turn off events or change it to the file-backend.

[Luap99]

The current per user process model from podman does not really allow this. Reading from the journal is easy easy enough but that can already be done with journalctl as @vrothberg said. To actually read from the file backend we would need to read each users database and config files to get the correct path. Very non trivial code and potentially security relevant if we try to do user things as root. And then how can we even monitor all users events? Should we loop over all users on the system. Overall I would be opposed to implement something like that.

[larry-cable]

The current per user process model from podman does not really allow this. Reading from the journal is easy easy enough but that can already be done with journalctl as @vrothberg said. To actually read from the file backend we would need to read each users database and config files to get the correct path. Very non trivial code and potentially security relevant if we try to do user things as root. And then how can we even monitor all users events? Should we loop over all users on the system. Overall I would be opposed to implement something like that.

while I agree that implementing a "per user" mechanism would be onerous in the extreme and that is not what I am suggesting), I would argue that a unprivileged user should not be able to subvert platform level logging of their activity by simply overriding the event logging backend in their local configuration.

If a system administrator or a privileged subsystem wants to monitor all activity on a particular system (of a certain type) I think there is long established precedent in multiple technologies that (non-privileged) users should not be able to subvert that monitoring or the telemetry stream that enables it.

I would suggest that regardless of user config, a global/system log of all events should be maintained, or at least some
pluggable mechanism exist that would enable such to be implemented. Current behavior for per-user behaviors can be maintained allowing them to redirect their events elsewhere (in addition to their being recorded at a system level)

this is not an abstract use case, but one that I am currently attempting to implement (not just for podman but for other container engines.

[rhatdan]

Podman is just a tool that allows you to do things in the homedir, that most people think they can do as root. An smart user could simply use skopeo copy or even CURL to pull content to the host then explode the content on to disk using tar potentially. They could write their own OCI Runtime and launch the container.

If you don't trust your users then you could use the audit subsystem, but I hesitate to start hacking up Podman in a way to prevent users from doing normal user activity.

I am working on a project to stick a user into a locked down container at login time, then you could control what containers are run within that container.

[larry-cable]

Podman is just a tool that allows you to do things in the homedir, that most people think they can do as root. An smart user could simply use skopeo copy or even CURL to pull content to the host then explode the content on to disk using tar potentially. They could write their own OCI Runtime and launch the container.

If you don't trust your users then you could use the audit subsystem, but I hesitate to start hacking up Podman in a way to prevent users from doing normal user activity.

I am working on a project to stick a user into a locked down container at login time, then you could control what containers are run within that container.

again, I am not suggesting that podman be modified in any way to prevent users from performing any task(s) they can today, what I am suggesting, (in fact recommending) is that for audit (and other monitoring) purposes, a privileged user may obtain (a current record of) all container related events that occur within a podman deployment root or rootless.

this is not about trust, this is about visibility into the actions of the podman subsystem at a global level.

If a user can completely override the logging mechanism to redirect their events (away from a single global record) this is not possible.

I would at least suggest that if the events API does not support this, the underlying logging should otherwise "observability" of all podman operations is not possible.

[vrothberg]

Thanks for sharing, @larry-cable!

This is not the first time such request for Podman come up. As mentioned above, users are free to change the events backend, use a file or even disable it entirely. Hence, Podman is not capable of monitoring/auditing what's going on globally since it's easy to workaround any such attempts.

Similarly, users can easily change the registries.conf for registries or the storage.conf for the backend storage along with almost all other knobs of Podman.

That's why I think Podman is not the right layer of abstraction or place to implement monitoring or observability. I would recommend EBPF-based approaches where no users can work around.

[vrothberg]

Assuming all users just use the defaults and do not disable events or change to a file backend my previous comment on using journalctl (#18427 (comment)) will work.

[baude]

is the events_logfile_path option in containers.conf inheritable? if so, that could be set system wide in /etc/containers/containers.conf and inherited by all users. There would be nothing preventing users from overriding this setting in a ~/.config/containers/container.conf file however.

a second approach could be to use events_logfile_path and set it to something like /var/tmp/podman_log/%user%-podman.log. Ofc this assumes that podman would honor %user% which i am not sure about. this too could be overriden by the users with a local containers.conf but would achieve the desired result?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@larry-cable were you able to try any of @baude suggestions?

[Luap99]

The ask here is for some system wide auditing via podman events that users cannot turn off. I think our journal event implementation is pretty close, all it would need is some option to always force this event logging and prevent users from turning it off via config overwrites. The admin or whoever wants to audit then just needs to look in the journal via journalctl, I don't think we should change podman events output to include all users.

[lpgc]

@larry-cable were you able to try any of @baude suggestions?

I am able to use the 'journalctl' approach to successfully monitor podman at the host level, but for a cloud vendor trying to monitor all container activity 'per host' the ability of a user to override this is still an issue.

[lpgc]

The ask here is for some system wide auditing via podman events that users cannot turn off. I think our journal event implementation is pretty close, all it would need is some option to always force this event logging and prevent users from turning it off via config overwrites.. The admin or whoever wants to audit then just needs to look in the journal via journalctl, I don't think we should change podman events output to include all users.

this!!!

[lpgc]

just to illustrate the need/use case here, what we are trying to implement is the ability to monitor all container based lifecycle events (in particular we are only interested in Java in Containers) per host in a cloud environment, using:

• docker
• podman
• k8s
• ...

the journalctl approach is perfect/ideal for this, but the ability for a user to override the event log destination or even disable it totally (none) is problematic.

If it were possible for a privileged admin to 'enforce' journalctl logging "in addition" to whatever an individual user might override in their .conf this would solve the issue.

this is not a trust issue we are trying to solve; obviously users can operate containers bypassing this as is mentioned earlier, this is about observability of podman usage.

[vrothberg]

@rhatdan WDYT?

[rhatdan]

Seems reasonable to me.

[lpgc]

is there an existing precedent for configuration values that cannot be overridden in per-user .conf files?

[rhatdan]

No

[vrothberg]

I wonder how much we should generalize the effort. In some way, it sounds reasonable to enable it for an entire containers.conf but at the same time, many of these values can be overridden on the CLI. This makes it somewhat complicated to truly enforce these values: for me a strong reason to selectively only allow it.

is there an existing precedent for configuration values that cannot be overridden in per-user .conf files?

As I wrote above, it's more complex that one .conf overriding another since some of these values can be set on the CLI which will take precedence. For instance, podman --events-backend=foobar

[Luap99]

Yeah I don't think using containers.conf will cut it, that would complicated config parsing a lot. And it is not like the current way is not already complicated enough. This will be especially problematic for the use of CONTAINERS_CONF in which case we would need to still parse the system wide config.

I think the more realistic approach would be to add a new file in /etc/containers/ which would force journal event logger for all users. That said it could still be possible for users to bypass that by creating a userns and bind mounting an empty directory over it. I am not sure what the thread model here is but I am not confident that there is a way for us to make this secure, as in there is no way to bypass it. If there is a will there is a way.

[lpgc]

Yeah I don't think using containers.conf will cut it, that would complicated config parsing a lot. And it is not like the current way is not already complicated enough. This will be especially problematic for the use of CONTAINERS_CONF in which case we would need to still parse the system wide config.

I think the more realistic approach would be to add a new file in /etc/containers/ which would force journal event logger for all users. That said it could still be possible for users to bypass that by creating a userns and bind mounting an empty directory over it. I am not sure what the thread model here is but I am not confident that there is a way for us to make this secure, as in there is no way to bypass it. If there is a will there is a way.

having spent a little time looking at the design and implementation of the current event logging; it appears to me that the only way to implement this would probably be to have a parallel "system" log event sink that could not be overridden by
either per-user or command line. - and as pointed out above can still be subverted by using kernel isolation mechanisms

this seems like a lot of work for a niche (but none the less important for cloud platform providers) use case, so all in all
I think we will have to do with best efforts here and let the "maverick" users that do not want to be observable remain
incognito!

thanks for everyone's time and thoughtful responses!

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

They can also download their own source code and build their own version of Podman without these constraints. Moving this to a discussion, since I don't see Podman doing anything about this at this time.