Skip to content

Commit 9fe3f99

Browse files
ssahanissahani
authored
Enhance README with comprehensive documentation
Expanded README.md to provide detailed overview, installation instructions, configuration options, and examples for systemd-netlogd.
1 parent 0c79bc9 commit 9fe3f99

File tree

1 file changed

+188
-141
lines changed

1 file changed

+188
-141
lines changed

‎README.md‎

Lines changed: 188 additions & 141 deletions
Original file line numberDiff line numberDiff line change
@@ -1,197 +1,244 @@
1-
systemd-netlogd
2-
===================
1+
# systemd-netlogd
32

4-
![build](https://github.com/systemd/systemd-netlogd/actions/workflows/ci.yml/badge.svg)
3+
[![Build Status](https://github.com/systemd/systemd-netlogd/actions/workflows/ci.yml/badge.svg)](https://github.com/systemd/systemd-netlogd/actions)
54

6-
Forwards messages from the journal to other hosts over the network using
7-
the Syslog Protocol (RFC 5424 and RFC 3339). It can be configured to send messages to
8-
both unicast and multicast addresses. systemd-netlogd runs with own user
9-
systemd-journal-netlog. Starts sending logs when network is up and stops
10-
sending as soon as network is down (uses sd-network). It reads from journal
11-
and forwards to network one by one. It does not use any extra disk space.
12-
systemd-netlogd supports UDP, TCP, TLS and DTLS (Datagram Transport Layer Security RFC 6012).
5+
**systemd-netlogd** is a lightweight daemon that forwards log messages from the systemd journal to remote hosts over the network using the Syslog protocol (RFC 5424 and RFC 3339). It supports both unicast and multicast destinations, ensuring reliable log aggregation in distributed environments.
136

14-
--------------------------------------------------------------------------
7+
## Overview
158

9+
### Key Features
10+
- **Network-Aware Operation**: Automatically starts forwarding logs when the network is available and pauses when it's down (integrates with `sd-network`).
11+
- **Efficient Processing**: Reads journal entries sequentially and forwards them one-by-one without buffering or using additional disk space.
12+
- **Protocol Support**: Handles UDP, TCP, TLS (for encrypted transmission), and DTLS (Datagram Transport Layer Security, RFC 6012).
13+
- **Flexible Formatting**: Supports RFC 5424 (default), RFC 5425 (with length prefix, ideal for TLS), and RFC 3339.
14+
- **Security Options**: Certificate validation for TLS/DTLS, keepalive mechanisms, and exclusion filters for sensitive logs.
15+
- **Namespace Awareness**: Can target specific journal namespaces or aggregate from multiple ones.
16+
- **Runs as Dedicated User**: Operates under the `systemd-journal-netlog` system user for isolation.
1617

17-
Installing from source
18-
----------------------
18+
systemd-netlogd is designed for minimal overhead, making it suitable for edge devices, servers, or cloud environments where centralized logging is needed without local storage impact.
1919

20-
Install build dependencies:
20+
## Installation
2121

22-
# On Debian/Ubuntu
23-
sudo apt install build-essential gperf libcap-dev libsystemd-dev pkg-config meson python3-sphinx
24-
# On CentOS/RHEL/Fedora
25-
sudo dnf group install 'Development Tools'
26-
sudo dnf install gperf libcap-devel pkg-config systemd-devel meson python3-sphinx
22+
### Prerequisites
23+
Ensure your system has the necessary build tools and dependencies. systemd-netlogd requires a recent systemd version (v255+ recommended for full feature support).
2724

28-
Build and install:
29-
30-
make
31-
sudo make install
32-
33-
Creating user:
34-
35-
``` bash
36-
sudo useradd -r -d / -s /usr/sbin/nologin -g systemd-journal systemd-journal-netlog
25+
#### On Debian/Ubuntu
26+
```bash
27+
sudo apt update
28+
sudo apt install build-essential gperf libcap-dev libsystemd-dev pkg-config meson python3-sphinx
3729
```
38-
or via sysusers
3930

40-
``` /usr/lib/sysusers.d/systemd-netlogd.conf```
31+
#### On CentOS/RHEL/Fedora
4132
```bash
42-
#Type Name ID GECOS Home directory Shell
43-
u systemd-journal-netlog -:systemd-journal - / /bin/nologin
33+
sudo dnf group install 'Development Tools'
34+
sudo dnf install gperf libcap-devel pkg-config systemd-devel meson python3-sphinx
4435
```
4536

46-
Configuration
47-
-------------
48-
49-
systemd-netlogd reads configuration files named `/etc/systemd/netlogd.conf` and `/etc/systemd/netlogd.conf.d/*.conf`.
50-
51-
**[NETWORK]** SECTION OPTIONS
52-
53-
The "[Network]" section only applies for UDP multicast address and Port:
54-
55-
Address=
56-
Controls whether log messages received by the systemd-netlogd daemon shall be forwarded to a unicast UDP address or multicast UDP network group in syslog RFC 5424 format. The the address string format is similar to socket units. See systemd.socket(1)
57-
58-
Protocol=
59-
Specifies whether to use udp, tcp, tls or dtls (Datagram Transport Layer Security) protocol. Defaults to udp.
60-
61-
LogFormat=
62-
Specifies whether to use RFC 5424, RFC 5425, or RFC 3339 format. Takes one of rfc5424, rfc5425, or rfc3339. Defaults to rfc5424. RFC 5425 is mainly useful for sending over TLS; it prepends a message length field to the RFC 5424 format.
63-
64-
Directory=
65-
Takes a directory path. Specifies whether to operate on the specified journal directory DIR instead of the default runtime and system journal paths.
66-
67-
Namespace=
68-
Takes a journal namespace identifier string as argument. If not specified the data collected by the default namespace is shown. If specified shows the log data of the specified namespace instead. If the namespace is specified as "*" data from all namespaces is shown, interleaved. If the namespace identifier is prefixed with "+" data from the specified namespace and the default namespace is shown, interleaved, but no other.
69-
70-
ConnectionRetrySec=
71-
Specifies the minimum delay before subsequent attempts to contact a Log server are made. Takes a time span value. The default unit is seconds, but other units may be specified, see systemd.time(5). Defaults to 30 seconds and must not be smaller than 1 second.
72-
73-
TLSCertificateAuthMode=
74-
Specifies whether to validate the certificate. Takes one of no, allow, deny, warn. Defaults to 'deny' which rejects certificates failed to validate.
75-
76-
TLSServerCertificate=
77-
Specify a custom certificate to validate the server against. Takes a path to a certificate file in PEM format.
78-
79-
KeepAlive=
80-
Takes a boolean argument. If true, the TCP/IP stack will send a keep alive message after 2h (depending on the configuration of /proc/sys/net/ipv4/tcp_keepalive_time) for all TCP streams accepted on this socket. This controls the SO_KEEPALIVE socket option (see socket(7) and the TCP Keepalive HOWTO for details.) Defaults to false.
81-
82-
KeepAliveTimeSec=
83-
Takes time (in seconds) as argument. The connection needs to remain idle before TCP starts sending keepalive probes. This controls the TCP_KEEPIDLE socket option (see socket(7) and the TCP Keepalive HOWTO for details.) Default value is 7200 seconds (2 hours).
84-
85-
KeepAliveIntervalSec=
86-
Takes time (in seconds) as argument between individual keepalive probes, if the socket option SO_KEEPALIVE has been set on this socket. This controls the TCP_KEEPINTVL socket option (see socket(7) and the TCP Keepalive HOWTO for details.) Default value is 75 seconds.
87-
88-
KeepAliveProbes=
89-
Takes an integer as argument. It is the number of unacknowledged probes to send before considering the connection dead and notifying the application layer. This controls the TCP_KEEPCNT socket option (see socket(7) and the TCP Keepalive HOWTO for details.) Default value is 9.
90-
91-
SendBuffer=
92-
Takes an integer argument controlling the receive or send buffer sizes of this socket, respectively. This controls the SO_SNDBUF socket options (see socket(7) for details.). The usual suffixes K, M, G are supported and are understood to the base of 1024.
93-
94-
NoDelay=
95-
Takes a boolean argument. TCP Nagle's algorithm works by combining a number of small outgoing messages, and sending them all at once. This controls the TCP_NODELAY socket option (see tcp(7)). Defaults to false.
96-
97-
Optional settings
37+
### Building from Source
38+
1. Clone the repository:
39+
```bash
40+
git clone https://github.com/systemd/systemd-netlogd.git
41+
cd systemd-netlogd
42+
```
43+
44+
2. Build and install:
45+
```bash
46+
meson setup build
47+
meson compile -C build
48+
sudo meson install -C build
49+
```
50+
51+
*Note*: If using traditional `make`, run `make` followed by `sudo make install`. Meson is preferred for modern builds.
52+
53+
3. Create the dedicated system user:
54+
- **Manual Creation**:
55+
```bash
56+
sudo useradd -r -d / -s /usr/sbin/nologin -g systemd-journal systemd-journal-netlog
57+
```
58+
- **Via Sysusers** (recommended, if supported):
59+
Include the following in `/etc/sysusers.d/systemd-netlogd.conf` or use the provided file:
60+
```
61+
#Type Name ID GECOS Home directory Shell
62+
u systemd-journal-netlog - - / /bin/nologin
63+
```
64+
Then run:
65+
```bash
66+
sudo systemd-sysusers
67+
```
68+
69+
### Package Managers
70+
- **Ubuntu**: Available in the universe repository for supported releases (e.g., Plucky Puffin, Questing Quokka, Resolute Raccoon) with version 1.4.4-1. Install with `sudo apt update && sudo apt install systemd-netlogd`.
71+
- **Fedora**: Available in COPR repositories (search for `systemd-netlogd`).
72+
- **Arch Linux**: Build from AUR (`systemd-netlogd-git`).
73+
- Check your distro's repositories for pre-built packages to simplify installation.
74+
75+
## Running the Service
76+
77+
After installation, enable and start the systemd service:
9878
99-
StructuredData=
100-
Meta information about the syslog message, which can be used for Cloud Based syslog servers, such as Loggly
101-
102-
UseSysLogStructuredData=
103-
A boolean. Specifies whether to extract SYSLOG_STRUCTURED_DATA= from journal. Defaults to false.
104-
105-
UseSysLogMsgId=
106-
A boolean. Specifies whether to extract SYSLOG_MSGID= from journal. Defaults to false.
107-
108-
ExcludeSyslogFacility=
109-
A list of strings. Specifies the syslog facilities to skip forwarding. Possible values are: "kern", "user", "mail", "daemon", "auth", "syslog", "lpr", "news", "uucp", "cron", "authpriv", "ftp", "ntp", "security", "console", "solaris-cron", "local0", "local1", "local2", "local3", "local4", "local5", "local6" and "local7".
110-
111-
ExcludeSyslogLevel=
112-
A list of strings. Specifies the syslog levels to skip forwarding. Possible values are: "emerg", "alert", "crit", "err", "warning", "notice", "info" and "debug".
113-
114-
**EXAMPLE**
115-
116-
Example 1.UDP Multicast
79+
```bash
80+
sudo systemctl daemon-reload # Reload after installation
81+
sudo systemctl enable --now systemd-netlogd.service
82+
```
11783
118-
``` toml
84+
- **Service File Location**: `/lib/systemd/system/systemd-netlogd.service` (installed during build).
85+
- **Logs**: Monitor with `journalctl -u systemd-netlogd.service`.
86+
- **Manual Start**: `sudo systemd-netlogd` (for testing; use the service in production).
87+
88+
The daemon will bind to the configured address/port and begin forwarding journal entries immediately upon network availability.
89+
90+
## Configuration
91+
92+
systemd-netlogd uses drop-in configuration files:
93+
- Main file: `/etc/systemd/netlogd.conf`
94+
- Drop-ins: `/etc/systemd/netlogd.conf.d/*.conf` (INI format, processed in lexicographical order)
95+
96+
Configurations are parsed as INI files with a `[Network]` section. Reload changes with `sudo systemctl reload systemd-netlogd.service`.
97+
98+
### [Network] Section Options
99+
100+
| Option | Description | Default | Example |
101+
|-------------------------|-------------|---------|---------|
102+
| `Address=` | Destination for forwarding (unicast IP:port or multicast group:port, e.g., `192.168.1.100:514` or `239.0.0.1:6000`). Supports socket-unit-like syntax (see `systemd.socket(5)`). | None (required) | `Address=192.168.1.100:514` |
103+
| `Protocol=` | Transport protocol: `udp` (default), `tcp`, `tls`, or `dtls`. | `udp` | `Protocol=tls` |
104+
| `LogFormat=` | Output format: `rfc5424` (default), `rfc5425` (length-prefixed for TLS), or `rfc3339`. | `rfc5424` | `LogFormat=rfc5425` |
105+
| `Directory=` | Custom journal directory path (overrides default runtime/system journals). | System default | `Directory=/var/log/journal-custom` |
106+
| `Namespace=` | Journal namespace filter: string ID, `*` (all), or `+ID` (ID + default). | Default namespace | `Namespace=*` |
107+
| `ConnectionRetrySec=` | Delay between retry attempts to the log server (time span, e.g., `1min`). Minimum 1s. | `30s` | `ConnectionRetrySec=1min` |
108+
| `TLSCertificateAuthMode=` | TLS/DTLS cert validation: `no` (skip), `allow` (accept invalid), `deny` (reject invalid), `warn` (log but accept). | `deny` | `TLSCertificateAuthMode=warn` |
109+
| `TLSServerCertificate=` | Path to PEM-formatted CA/server certificate for validation. | None | `TLSServerCertificate=/etc/ssl/ca-cert.pem` |
110+
| `KeepAlive=` | Enable TCP keepalives (boolean). | `false` | `KeepAlive=true` |
111+
| `KeepAliveTimeSec=` | Idle time before sending keepalive probes (seconds). | `7200` (2h) | `KeepAliveTimeSec=3600` |
112+
| `KeepAliveIntervalSec=` | Interval between keepalive probes (seconds). | `75` | `KeepAliveIntervalSec=60` |
113+
| `KeepAliveProbes=` | Number of unacknowledged probes before closing connection. | `9` | `KeepAliveProbes=5` |
114+
| `SendBuffer=` | Socket send buffer size (e.g., `64K`, supports K/M/G suffixes). | System default | `SendBuffer=1M` |
115+
| `NoDelay=` | Disable Nagle's algorithm for low-latency TCP (boolean). | `false` | `NoDelay=true` |
116+
| `StructuredData=` | Custom syslog structured data ID (e.g., for cloud providers like Loggly). | None | `StructuredData=[1ab456b6-90bb-6578-abcd-5b734584aaaa@41058]` |
117+
| `UseSysLogStructuredData=` | Extract and include `SYSLOG_STRUCTURED_DATA` from journal (boolean). | `false` | `UseSysLogStructuredData=yes` |
118+
| `UseSysLogMsgId=` | Extract and include `SYSLOG_MSGID` from journal (boolean). | `false` | `UseSysLogMsgId=yes` |
119+
| `ExcludeSyslogFacility=` | Comma-separated list of facilities to skip (e.g., `auth,authpriv`). See syslog facilities list below. | None | `ExcludeSyslogFacility=auth,daemon` |
120+
| `ExcludeSyslogLevel=` | Comma-separated list of levels to skip (e.g., `debug`). See syslog levels list below. | None | `ExcludeSyslogLevel=debug,info` |
121+
122+
#### Syslog Facilities
123+
Supported values: `kern`, `user`, `mail`, `daemon`, `auth`, `syslog`, `lpr`, `news`, `uucp`, `cron`, `authpriv`, `ftp`, `ntp`, `security`, `console`, `solaris-cron`, `local0`–`local7`.
124+
125+
#### Syslog Levels
126+
Supported values: `emerg`, `alert`, `crit`, `err`, `warning`, `notice`, `info`, `debug`.
127+
128+
### Configuration Examples
129+
130+
#### Example 1: UDP Multicast
131+
For broadcasting to a multicast group:
132+
```ini
119133
[Network]
120134
Address=239.0.0.1:6000
121-
#Protocol=udp
122-
#LogFormat=rfc5424
135+
# Protocol=udp (default)
136+
# LogFormat=rfc5424 (default)
123137
```
124138

125-
Example 2.UDP
126-
127-
``` toml
139+
#### Example 2: Unicast UDP with RFC 3339
140+
```ini
128141
[Network]
129142
Address=192.168.8.101:514
130-
#Protocol=udp
143+
# Protocol=udp (default)
131144
LogFormat=rfc3339
132145
```
133146

134-
Example 3. Structured data
135-
136-
``` toml
147+
#### Example 3: RFC 5424 with Custom Structured Data
148+
Useful for cloud syslog services:
149+
```ini
137150
[Network]
138151
Address=192.168.8.101:514
139-
#Protocol=udp
152+
# Protocol=udp (default)
140153
LogFormat=rfc5424
141154
StructuredData=[1ab456b6-90bb-6578-abcd-5b734584aaaa@41058]
142155
```
143156

144-
Example 4. Custom syslog structured data and message ID
145-
146-
``` toml
157+
#### Example 4: Extracting Journal Metadata
158+
Include structured data and message IDs from journal entries:
159+
```ini
147160
[Network]
148161
Address=192.168.8.101:514
149-
#Protocol=udp
162+
# Protocol=udp (default)
150163
LogFormat=rfc5424
151164
UseSysLogStructuredData=yes
152165
UseSysLogMsgId=yes
153166
```
154167

155-
Example 5. Skipping messages with facility AUTH or AUTHPRIV and messages with level DEBUG
156-
157-
``` toml
168+
#### Example 5: Filtering Sensitive Logs
169+
Skip auth-related facilities and debug levels:
170+
```ini
158171
[Network]
159172
Address=192.168.8.101:514
160-
#Protocol=udp
173+
# Protocol=udp (default)
161174
LogFormat=rfc3339
162-
ExcludeSyslogFacility=auth authpriv
175+
ExcludeSyslogFacility=auth,authpriv
163176
ExcludeSyslogLevel=debug
164177
```
165178

166-
Example 6. TLS with certificate authentocation mode
167-
168-
``` toml
179+
#### Example 6: TLS with Relaxed Certificate Validation
180+
For secure transmission with warning on invalid certs:
181+
```ini
169182
[Network]
170183
Address=192.168.8.101:4433
171184
Protocol=tls
172-
#LogFormat=rfc5424
185+
# LogFormat=rfc5424 (default)
173186
TLSCertificateAuthMode=warn
187+
TLSServerCertificate=/etc/ssl/my-ca.pem
188+
KeepAlive=true
174189
```
175190

176-
Example 7. DTLS with certificate authentocation mode
177-
178-
``` toml
191+
#### Example 7: DTLS for UDP-Like Security
192+
Datagram-based encryption:
193+
```ini
179194
[Network]
180195
Address=192.168.8.101:4433
181196
Protocol=dtls
182-
#LogFormat=rfc5424
197+
# LogFormat=rfc5424 (default)
183198
TLSCertificateAuthMode=allow
184199
```
185200

186-
Use case of ```UseSysLogStructuredData=``` and ```UseSysLogMsgId=```
187-
188-
```C
189-
sd_journal_send(
190-
"MESSAGE=%s", "Message to process",
191-
"PRIORITY=%s", "4",
192-
"SYSLOG_FACILITY=%s", "1",
193-
"SYSLOG_MSGID=%s", "1011",
194-
"SYSLOG_STRUCTURED_DATA=%s", R"([exampleSDID@32473 iut="3" eventSource="Application"])",
195-
NULL
196-
);
201+
### Using Structured Data and Message IDs
202+
To leverage `UseSysLogStructuredData` and `UseSysLogMsgId`, tag journal entries with metadata via `sd_journal_send()`:
203+
204+
```c
205+
#include <systemd/sd-journal.h>
206+
207+
int main() {
208+
sd_journal_send(
209+
"MESSAGE=%s", "Message to process",
210+
"PRIORITY=%s", "4", // warning level
211+
"SYSLOG_FACILITY=%s", "1", // user facility
212+
"SYSLOG_MSGID=%s", "1011",
213+
"SYSLOG_STRUCTURED_DATA=%s", R"([exampleSDID@32473 iut="3" eventSource="Application"])",
214+
NULL
215+
);
216+
return 0;
217+
}
197218
```
219+
220+
Compile with: `gcc example.c -lsystemd`.
221+
222+
This embeds metadata that systemd-netlogd can extract and forward in syslog headers.
223+
224+
## Security Considerations
225+
- **TLS/DTLS**: Always use certificate validation (`deny` mode) in production. Provide custom CAs via `TLSServerCertificate`.
226+
- **Firewall**: Open only necessary ports (e.g., 514/UDP for syslog, 4433/TCP for TLS).
227+
- **Exclusions**: Filter sensitive facilities (e.g., `authpriv`) to avoid leaking credentials.
228+
- **Multicast**: Limit to trusted networks to prevent unauthorized log access.
229+
- **User Isolation**: The `systemd-journal-netlog` user has minimal privileges; audit with `systemd-analyze security systemd-netlogd.service`.
230+
231+
## Troubleshooting
232+
- **No Logs Forwarded**: Check `journalctl -u systemd-netlogd` for errors. Verify network connectivity and journal permissions.
233+
- **Connection Failures**: Increase `ConnectionRetrySec` or inspect TLS certs with `openssl verify`.
234+
- **High Latency**: Enable `NoDelay=true` for TCP; monitor buffer overflows with `SendBuffer`.
235+
- **Testing**: Use `nc -u 192.168.8.101 514` to simulate a receiver and `logger -p user.info "Test message"` to generate journal entries.
236+
- **Debug Mode**: Add `StandardOutput=journal+console` to the service override for verbose output.
237+
238+
## Contributing
239+
Fork the repo, submit PRs for features/bugfixes. See `CONTRIBUTING.md` for guidelines.
240+
241+
## License
242+
LGPL-2.1-or-later (same as systemd). See `LICENSE` file.
243+
244+
For questions, open an issue on GitHub.

0 commit comments

Comments
 (0)