Enable persistent journaling

In the cloud images, /var/log/journal doesn't exist. When this
directory does not exist, journald will not write any files to
disk, and as such, logging data is lost accross reboots.

Persistent logging is useful for debugging issues that caused the nodes
to reboot (provided that the logs are flushed to disk).

Author: jovial

ansible/.gitignore

@@ -98,3 +98,5 @@ roles/*
 !roles/eessi/**
 !roles/topology/
 !roles/topology/**
+!roles/journald/
+!roles/journald/**

ansible/bootstrap.yml

@@ -28,6 +28,15 @@
           to update these variable names. ** NB: The actual secrets will not be changed.**
       when: "'secrets_openhpc_' in (hostvars[inventory_hostname] | join)"
 
+- hosts: journald
+  gather_facts: false
+  tags:
+    - logging
+    - journald
+  tasks:
+    - import_role:
+        name: journald
+
 - hosts: resolv_conf
   become: yes
   gather_facts: false

ansible/roles/journald/README.md

@@ -0,0 +1,4 @@
+# journald
+
+This role is used to configue journald. Please see the [role
+defaults](defaults/main.yml) for a full list of configuration options.

ansible/roles/journald/defaults/main.yml

@@ -0,0 +1,4 @@
+---
+# Journald storage. One of: volatile, persistent, auto, or none. Defaults to
+# `persistent`.
+journald_storage: persistent

ansible/roles/journald/tasks/main.yml

@@ -0,0 +1,32 @@
+---
+- name: Create /var/log/journal
+  become: true
+  file:
+    path: /var/log/journal
+    owner: "root"
+    group: "systemd-journal"
+    mode: 02755
+    state: directory
+  notify: Flush journal to disk
+  when: journald_storage == "persistent"
+
+- name: Ensure journald drop in directory exists
+  file:
+    path: "/etc/systemd/journald.conf.d/"
+    owner: "root"
+    group: "root"
+    mode: 0770
+    state: directory
+  become: true
+
+- name: Ensure journald.conf overrides are set
+  copy:
+    content: |
+       [Journal]
+       Storage={{ journald_storage }}
+    dest: /etc/systemd/journald.conf.d/ansible-slurm-appliance.conf
+    owner: root
+    group: root
+    mode: 0660
+  become: true
+  notify: Restart journald

docs/monitoring-and-logging.md

@@ -301,6 +301,9 @@ slurm-stats is configured `slurm-stats` role in the [slurm_openstack_tools colle
 The `slurm_stats` group controls the placement of the `slurm_stats` service.
 This should be configured to be a group with a single host. That host must be co-located on the same host as the `filebeat` service that scrapes its output.
 
+## Logging configuration
 
+### Journald
 
-
+The [journald](../ansible/roles/journald/README.md) role is used to customise
+journald configuration.

environments/common/inventory/groups

@@ -26,6 +26,10 @@ login
 openhpc
 additional
 
+[journald:childen]
+# Hosts where journald should be configured. See ansible/roles/journald/README.md.
+cluster
+
 [builder]
 # Do not add hosts here manually - used as part of Packer image build pipeline. See packer/README.md.