Skip to content

Commit 5a809f5

Browse files
author
Bob Furu
authored
Merge pull request #33959 from bgilbert/journald
2 parents 0caeb07 + 23bc3b5 commit 5a809f5

File tree

2 files changed

+76
-108
lines changed

2 files changed

+76
-108
lines changed

modules/cluster-logging-systemd-scaling.adoc

Lines changed: 47 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -17,45 +17,62 @@ and other settings.
1717

1818
.Procedure
1919

20-
. Create a `journald.conf` file with the required settings:
20+
. Create a Butane config file, `40-worker-custom-journald.bu`, that includes an `/etc/systemd/journald.conf` file with the required settings.
2121
+
22-
[source,terminal]
22+
[NOTE]
23+
====
24+
See "Creating machine configs with Butane" for information about Butane.
25+
====
26+
+
27+
[source,yaml]
2328
----
24-
Compress=yes <1>
25-
ForwardToConsole=no <2>
26-
ForwardToSyslog=no
27-
MaxRetentionSec=1month <3>
28-
RateLimitBurst=10000 <4>
29-
RateLimitIntervalSec=30s
30-
Storage=persistent <5>
31-
SyncIntervalSec=1s <6>
32-
SystemMaxUse=8g <7>
33-
SystemKeepFree=20% <8>
34-
SystemMaxFileSize=10M <9>
29+
variant: openshift
30+
version: 4.8.0
31+
metadata:
32+
name: 40-worker-custom-journald
33+
labels:
34+
machineconfiguration.openshift.io/role: worker
35+
storage:
36+
files:
37+
- path: /etc/systemd/journald.conf
38+
mode: 0644 <1>
39+
overwrite: true
40+
contents:
41+
inline: |
42+
Compress=yes <2>
43+
ForwardToConsole=no <3>
44+
ForwardToSyslog=no
45+
MaxRetentionSec=1month <4>
46+
RateLimitBurst=10000 <5>
47+
RateLimitIntervalSec=30s
48+
Storage=persistent <6>
49+
SyncIntervalSec=1s <7>
50+
SystemMaxUse=8g <8>
51+
SystemKeepFree=20% <9>
52+
SystemMaxFileSize=10M <10>
3553
----
3654
+
37-
<1> Specify whether you want logs compressed before they are written to the file system.
55+
<1> Set the permissions for the `journal.conf` file. It is recommended to set `0644` permissions.
56+
<2> Specify whether you want logs compressed before they are written to the file system.
3857
Specify `yes` to compress the message or `no` to not compress. The default is `yes`.
39-
<2> Configure whether to forward log messages. Defaults to `no` for each. Specify:
58+
<3> Configure whether to forward log messages. Defaults to `no` for each. Specify:
4059
* `ForwardToConsole` to forward logs to the system console.
4160
* `ForwardToKsmg` to forward logs to the kernel log buffer.
4261
* `ForwardToSyslog` to forward to a syslog daemon.
4362
* `ForwardToWall` to forward messages as wall messages to all logged-in users.
44-
<3> Specify the maximum time to store journal entries. Enter a number to specify seconds. Or
63+
<4> Specify the maximum time to store journal entries. Enter a number to specify seconds. Or
4564
include a unit: "year", "month", "week", "day", "h" or "m". Enter `0` to disable. The default is `1month`.
46-
<4> Configure rate limiting. If, during the time interval defined by `RateLimitIntervalSec`, more logs than specified in `RateLimitBurst`
47-
are received, all further messages within the interval are dropped until the interval is over. It is recommended to set
48-
`RateLimitIntervalSec=30s` and `RateLimitBurst=10000`, which are the defaults.
49-
<5> Specify how logs are stored. The default is `persistent`:
65+
<5> Configure rate limiting. If more logs are received than what is specified in `RateLimitBurst` during the time interval defined by `RateLimitIntervalSec`, all further messages within the interval are dropped until the interval is over. It is recommended to set `RateLimitIntervalSec=30s` and `RateLimitBurst=10000`, which are the defaults.
66+
<6> Specify how logs are stored. The default is `persistent`:
5067
* `volatile` to store logs in memory in `/var/log/journal/`.
51-
* `persistent` to store logs to disk in `/var/log/journal/`. systemd creates the directory if it does not exist.
52-
* `auto` to store logs in in `/var/log/journal/` if the directory exists. If it does not exist, systemd temporarily stores logs in `/run/systemd/journal`.
68+
* `persistent` to store logs to disk in `/var/log/journal/`. systemd creates the directory if it does not exist.
69+
* `auto` to store logs in `/var/log/journal/` if the directory exists. If it does not exist, systemd temporarily stores logs in `/run/systemd/journal`.
5370
* `none` to not store logs. systemd drops all logs.
54-
<6> Specify the timeout before synchronizing journal files to disk for *ERR*, *WARNING*, *NOTICE*, *INFO*, and *DEBUG* logs.
71+
<7> Specify the timeout before synchronizing journal files to disk for *ERR*, *WARNING*, *NOTICE*, *INFO*, and *DEBUG* logs.
5572
systemd immediately syncs after receiving a *CRIT*, *ALERT*, or *EMERG* log. The default is `1s`.
56-
<7> Specify the maximum size the journal can use. The default is `8g`.
57-
<8> Specify how much disk space systemd must leave free. The default is `20%`.
58-
<9> Specify the maximum size for individual journal files stored persistently in `/var/log/journal`. The default is `10M`.
73+
<8> Specify the maximum size the journal can use. The default is `8g`.
74+
<9> Specify how much disk space systemd must leave free. The default is `20%`.
75+
<10> Specify the maximum size for individual journal files stored persistently in `/var/log/journal`. The default is `10M`.
5976
+
6077
[NOTE]
6178
====
@@ -68,47 +85,18 @@ For more information on systemd settings, see link:https://www.freedesktop.org/s
6885
+
6986
// Defaults from https://github.com/openshift/openshift-ansible/pull/3753/files#diff-40b7a7231e77d95ca6009dc9bcc0f470R33-R34
7087

71-
. Convert the `journal.conf` file to base64:
88+
. Use Butane to generate a `MachineConfig` object file, `40-worker-custom-journald.yaml`, containing the configuration to be delivered to the nodes:
7289
+
7390
[source,terminal]
7491
----
75-
$ export jrnl_cnf=$( cat /journald.conf | base64 -w0 )
92+
$ butane 40-worker-custom-journald.bu -o 40-worker-custom-journald.yaml
7693
----
7794

78-
. Create a `MachineConfig` object that includes the `jrnl_cnf` variable, which is the encoded contents of the `journald.conf` you created in the previous step.
79-
+
80-
For example:
81-
+
82-
[source,terminal]
83-
----
84-
$ cat > /tmp/40-worker-custom-journald.yaml <<EOF
85-
apiVersion: machineconfiguration.openshift.io/v1
86-
kind: MachineConfig
87-
metadata:
88-
labels:
89-
machineconfiguration.openshift.io/role: worker
90-
name: 50-corp-journald
91-
spec:
92-
config:
93-
ignition:
94-
version: 3.1.0
95-
storage:
96-
files:
97-
- contents:
98-
source: data:text/plain;charset=utf-8;base64,${jrnl_cnf}
99-
mode: 0644 <1>
100-
overwrite: true
101-
path: /etc/systemd/journald.conf <2>
102-
EOF
103-
----
104-
<1> Set the permissions for the `journal.conf` file. It is recommended to set `0644` permissions.
105-
<2> Specify the path to the base64-encoded `journal.conf` file.
106-
107-
. Create the machine config. For example:
95+
. Apply the machine config. For example:
10896
+
10997
[source,terminal]
11098
----
111-
$ oc apply -f /tmp/40-worker-custom-journald.yaml
99+
$ oc apply -f 40-worker-custom-journald.yaml
112100
----
113101
+
114102
The controller detects the new `MachineConfig` object and generates a new `rendered-worker-<hash>` version.

modules/machineconfig-modify-journald.adoc

Lines changed: 29 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -11,73 +11,53 @@ If you need to configure settings for the `journald` service on {product-title}
1111
This procedure describes how to modify `journald` rate limiting settings in the `/etc/systemd/journald.conf` file and apply them to worker nodes. See the `journald.conf` man page for information on how to use that file.
1212

1313
.Prerequisites
14-
* Have a running {product-title} cluster (version 4.4 or later).
14+
* Have a running {product-title} cluster.
1515
* Log in to the cluster as a user with administrative privileges.
1616

1717
.Procedure
1818

19-
. Create the contents of the `/etc/systemd/journald.conf` file and encode it as base64. For example:
19+
. Create a Butane config file, `40-worker-custom-journald.bu`, that includes an `/etc/systemd/journald.conf` file with the required settings.
2020
+
21-
[source,terminal]
22-
----
23-
$ cat > /tmp/jrnl.conf <<EOF
24-
# Disable rate limiting
25-
RateLimitInterval=1s
26-
RateLimitBurst=10000
27-
Storage=volatile
28-
Compress=no
29-
MaxRetentionSec=30s
30-
EOF
31-
----
32-
33-
. Convert the temporary `journal.conf` file to base64 and save it into a variable (`jrnl_cnf`):
21+
[NOTE]
22+
====
23+
See "Creating machine configs with Butane" for information about Butane.
24+
====
3425
+
35-
[source,terminal]
26+
[source,yaml]
3627
----
37-
$ export jrnl_cnf=$( cat /tmp/jrnl.conf | base64 -w0 )
38-
$ echo $jrnl_cnf
39-
IyBEaXNhYmxlIHJhdGUgbGltaXRpbmcKUmF0ZUxpbWl0SW50ZXJ2YWw9MXMKUmF0ZUxpbWl0QnVyc3Q9MTAwMDAKU3RvcmFnZT12b2xhdGlsZQpDb21wcmVzcz1ubwpNYXhSZXRlbnRpb25TZWM9MzBzCg==
28+
variant: openshift
29+
version: 4.8.0
30+
metadata:
31+
name: 40-worker-custom-journald
32+
labels:
33+
machineconfiguration.openshift.io/role: worker
34+
storage:
35+
files:
36+
- path: /etc/systemd/journald.conf
37+
mode: 0644
38+
overwrite: true
39+
contents:
40+
inline: |
41+
# Disable rate limiting
42+
RateLimitInterval=1s
43+
RateLimitBurst=10000
44+
Storage=volatile
45+
Compress=no
46+
MaxRetentionSec=30s
4047
----
4148

42-
. Create the machine config, including the encoded contents of `journald.conf` (`jrnl_cnf` variable):
49+
. Use Butane to generate a `MachineConfig` object file, `40-worker-custom-journald.yaml`, containing the configuration to be delivered to the worker nodes:
4350
+
4451
[source,terminal]
4552
----
46-
$ cat > /tmp/40-worker-custom-journald.yaml <<EOF
47-
apiVersion: machineconfiguration.openshift.io/v1
48-
kind: MachineConfig
49-
metadata:
50-
labels:
51-
machineconfiguration.openshift.io/role: worker
52-
name: 40-worker-custom-journald
53-
spec:
54-
config:
55-
ignition:
56-
config: {}
57-
security:
58-
tls: {}
59-
timeouts: {}
60-
version: 3.1.0
61-
networkd: {}
62-
passwd: {}
63-
storage:
64-
files:
65-
- contents:
66-
source: data:text/plain;charset=utf-8;base64,${jrnl_cnf}
67-
verification: {}
68-
filesystem: root
69-
mode: 420
70-
path: /etc/systemd/journald.conf
71-
systemd: {}
72-
osImageURL: ""
73-
EOF
53+
$ butane 40-worker-custom-journald.bu -o 40-worker-custom-journald.yaml
7454
----
7555

7656
. Apply the machine config to the pool:
7757
+
7858
[source,terminal]
7959
----
80-
$ oc apply -f /tmp/40-worker-custom-journald.yaml
60+
$ oc apply -f 40-worker-custom-journald.yaml
8161
----
8262

8363
. Check that the new machine config is applied and that the nodes are not in a degraded state. It might take a few minutes. The worker pool will show the updates in progress, as each node successfully has the new machine config applied:

0 commit comments

Comments
 (0)