RFE: accept input format compatible with other logging systems

[topimiettinen]

See also systemd/systemd#15921

Is your feature request related to a problem? Please describe.
Currently auditd stores separate logs in /var/log/audit. I'd like to discontinue using auditd for logging (but not for other audit control) and use only systemd-journald logs also for ausearch etc. ausearch is able to read logs also from standard input. The output format of journalctl is quite similar to format of /var/log/audit/audit.log and traditional format used by syslogd, but ausearch does not understand it.

Describe the solution you'd like
Enhance audit tools (especially ausearch) to understand some existing journalctl output format (for example syslog/short-unix or JSON). This could be useful even when not using journald. For example, /var/log/audit/* might be damaged or lost but other logs (syslog, journal etc) could contain same information.

Describe alternatives you've considered

1. Postprocess journalctl output to match audit.log (no changes required to journalctl or audit tools):

# journalctl -b _TRANSPORT=audit --output=short-unix | awk '{ print "type="$4" msg=audit("$1") " $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17 }' | ausearch -r
----
type=SERVICE_START msg=audit(1590476099.070000) pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@rfkill comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' 

2. Add new audit format for journalctl which can be directly read by audit tools, for example: journalctl -b --output=audit | ausearch.

So, this record from journalctl --output=short-unix

1590476099.070000 loora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@rfkill comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

should be printed when --output=audit is specified in the format in /var/log/audit/audit.log:

type=SERVICE_START msg=audit(1590476099.070:34): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@rfkill comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

3. Perhaps (some) search options of ausearch could be easily implemented also in journalctl. But format compatibility between journalctl and audit tools would still be needed in order to use the other audit tools like audit2allow to process SELinux AVC denials. Reimplementing audit2allow by journalctl does not seem to make sense.

As requested by the issue form, I created an account on Red Hat Bugzilla to report this but unfortunately I wasn't able to find the right product.

[kovacs-andras]

I really like this idea!
Have you seen these?
https://bugzilla.redhat.com/show_bug.cgi?id=1227379
systemd/systemd#959
You can easily get into crossfire here... and prepare for a lecturing on how not to mix the roles of a security officer with a sysadmin.
But I'm on your side!

[burnalting]

One of the features of ausearch (--checkpoint) makes use of /var/log/audit file structure to maintain state in the situation where one wants to process new (since the last ausearch invocation) auditd events.

If one switched to systemd-journald exclusively, how would this be achieved?

I am particularly interested in the use case of an environment where there may be 200k+ hosts, all sending logs to a centralised logging capability.

[topimiettinen]

Have you seen these?

No, but these want the opposite: journald would not store audit messages. I want to stop auditd from logging. I don't have any requirements for compliance with anything.

[topimiettinen]

One of the features of ausearch (--checkpoint) makes use of /var/log/audit file structure to maintain state in the situation where one wants to process new (since the last ausearch invocation) auditd events.

Would journalctl --cursor= and related switches achieve this? I haven't used them.

[burnalting]

I've not used journalctl. I suppose if you are recommending the removal of a major element of a capability, then you need to at the same time, propose a transition mechanism, as there may be a lot of log processing capability that relies on the /var/log/audit structure (logstash, splunk, etc).

Perhaps a better way forward is to offer, along with your justified better solution, a method where one can still make use of the the older mechanisms. For example, if you are proposing the removal of auditd directly writing to /var/log/audit, then offer a means of systemd-journald and all it's alternatives replicating /var/log/audit.

[topimiettinen]

To clarify, I'm not proposing of removing anything for others, but add a new optional feature for ausearch, probably only when it's reading standard input (this feature already exists) instead of directly accessing the logs. If someone needs /var/log/audit or they are not happy with features of journalctl, they are free to continue to do whatever they do now.

[stevegrubb]

Have you seen these?

No, but these want the opposite: journald would not store audit messages. I want to stop auditd from logging. I don't have any requirements for compliance with anything.

Would uninstalling audit achieve the same goal?

[topimiettinen]

Would uninstalling audit achieve the same goal?

No, because I need auditctl to set up the auditing rules and ausearch, audit2allow and perhaps others.

[stevegrubb]

I'm confused, you said you don't have any requirements for compliance or anything, but you want rules loaded? It sounds like you do have requirements of some kind.

The problem is journald uses the multicast socket which is not reliable. Journald's events are not suitable for use off machine or after accounts have been deleted. There's a lot of stuff besides audit events in journald - meaning searches will be slower. Since anyone can write to syslog, how do you ensure someone is not spoofing audit events? There are occasions where journald gets double log entries. This is likely to give any reporting tools a problem. And then there is the issue of some of the tools require DAEMON_START, DAEMON_END events to bracket sessions that are not terminated.

I'd be willing to review patches if someone submits them. But given other tasks that are a higher priority to me, I'm not likely to write the code myself.

[topimiettinen]

I'm confused, you said you don't have any requirements for compliance or anything, but you want rules loaded? It sounds like you do have requirements of some kind.

I meant that as a home user I don't need to care about (for example) PCI requirements.

The problem is ...

Thanks for the heads up. I guess these problems may be more visible with more testing. I haven't had any problems but maybe I didn't expect any.

I'll make a patch.

[burnalting]

To clarify, I'm not proposing of removing anything for others, but add a new optional feature for ausearch, probably only when it's reading standard input (this feature already exists) instead of ...

Ah. I clearly mis-read your proposal. Thanks.

[topimiettinen]

Looking at possible implementations, I noticed some issues.

• auditd associates each audit event with a serial number (event->serial). I think for journalctl this is available as _AUDIT_ID, but in general the number may not be available when reading logs elsewhere. It looks like functions like events_are_equal() will accept this but if there would be two different events with exactly same timestamp, they may be considered as one event. This may make the use case less interesting.

• There's some code duplication, for example str2event() is found in auparse/auparse.c and src/ausearch-lol.c. Would it be OK to refactor these to one file?

Regarding the input options:

• traditional syslog format (raw log files from syslog daemons or journalctl --output=short-unix) is not so well defined but it's probably most interesting as use case (audit.log files are lost but still have some syslog files). Parsing is not so big issue.
• JSON (journalctl --output=json) would be flexible and well defined. The easiest would be to use an existing JSON parsing library, which adds a new dependency to the package. If there's not a lot of interest, this seems a bit overkill.
• another option could be to use sd_journal_get_data() from libsystemd to read journal data directly. Probably fastest but it can't be used for text log files. Also this would add a packaging dependency to libsystemd.

Overall I think it would be easier to modify journalctl to optionally generate audit log format (something like output_cat()). I still think that to get maximum benefit (new ability for audit tools to read syslog files), the change should be done on auditd side.

[topimiettinen]

I implemented this tentatively for journalctl, only to find out that audit2allow actually can already parse journalctl output. This actually covers pretty much my use case since instead of using ausearch, I can use journalctl to look at raw log entries, possibly with some filtering and grepping and then use journalctl -b -u xyz.service | grep foobar | audit2allow to process the AVC denials. Another use case was to process logs from non-root users (session D-Bus or Xorg), which may contain SELinux AVC entries when they serve as object managers but this is also covered.