feat: add TLS certificate and key support for Grafana for HTTPS

Feature:
Add the ability to configure TLS/HTTPS for the Grafana graphing service
using the following new role variables:

* metrics_grafana_certificates - use the certificate role to generate certs
* metrics_grafana_cert/metrics_grafana_private_key - paths to existing certs
* metrics_grafana_cert_src/metrics_grafana_private_key_src - copy local files

Reason:
Users need the ability to secure Grafana connections with TLS/HTTPS to
protect metrics data in transit, which is a common security requirement
for production deployments.

Result:
The metrics role can now configure Grafana to use HTTPS by either:

1. Generating certificates via the certificate system role
2. Using existing certificate files on the target system
3. Copying certificate files from the control node

All tests updated to verify HTTPS functionality, and a new test added
to verify certificate role integration.

In addition - the tests were written incorrectly to run the handlers.
In order to refresh the services, the handlers must be run immediately
after running the role, so that the checks and verify tasks are testing
the actual result of running the role.  This fixes several test flakes
that are caused when the services were not refreshed when doing the
verification.

Signed-off-by: Rich Megginson <rmeggins@redhat.com>

Author: richm

README.md

@@ -185,6 +185,88 @@ enable additional metrics, export to alternate data sinks, and so on.
 metrics_optional_packages: [pcp-pmda-apache]
 ```
 
+### metrics_grafana_certificates: []
+
+For generating a new certificate for grafana it is recommended to set the
+`metrics_grafana_certificates` variable.  If you have your own certs/keys, or
+will create them from your own CA provider, see below `metrics_grafana_cert` et
+al. for information about how to pass in and/or use your own certs.
+
+The value of `metrics_grafana_certificates` is passed on to the
+`certificate_requests` variable of the `certificate` role called internally in
+the `metrics` role and it generates the private key and certificate. For the
+supported parameters of `metrics_grafana_certificates`, see the
+[`certificate_requests` role documentation section](https://github.com/linux-system-roles/certificate/#certificate_requests).
+
+When you set `metrics_grafana_certificates`, you must not set
+`metrics_grafana_private_key` and `metrics_grafana_cert` variables because they
+are ignored.
+
+This example installs grafana with an IdM-issued web server certificate assuming
+your machines are joined to a FreeIPA domain.
+
+```yaml
+    - name: Install grafana with server certificate and key
+      include_role:
+        name: linux-system-roles.metrics
+      vars:
+        metrics_graph_service: true
+        metrics_grafana_certificates:
+          - name: grafana-server
+            dns: ['localhost', 'www.example.com']
+            ca: ipa
+```
+
+NOTE: The `certificate` role, unless using IPA and joining the systems to an IPA
+domain, creates self-signed certificates, so you will need to explicitly
+configure trust, which is not currently supported by the system roles. To use
+`ca: self-sign` or `ca: local`, depending on your certmonger usage, see the
+[linux-system-roles.certificate documentation](https://github.com/linux-system-roles/certificate/#cas-and-providers) for details.
+
+NOTE: Creating a self-signed certificate is not supported on RHEL/CentOS-7.
+
+### metrics_grafana_cert: ''
+
+TLS certificate file for the grafana server.  This should be the full absolute path
+on the grafana server machine e.g. `/etc/pki/tls/certs/grafana-server.crt`.  This
+file should already exist.  If you want to copy a local file, see `metrics_grafana_cert_src`.
+
+```yaml
+metrics_grafana_cert: /etc/pki/tls/certs/grafana-server.crt
+```
+
+### metrics_grafana_cert_src: ''
+
+TLS certificate file from the local machine to copy to `metrics_grafana_cert` on
+the grafana server machine.  If `metrics_grafana_cert` is not specified, then
+`metrics_grafana_cert_src` will be copied to `/etc/pki/tls/certs/basename.crt` on
+the grafana server machine, where `basename` is the basename of `metrics_grafana_cert_src`.
+
+```yaml
+metrics_grafana_cert_src: /my/local/grafana-server.crt
+```
+
+### metrics_grafana_private_key: ''
+
+TLS private key file for the grafana server.  This should be the full absolute path
+on the grafana server machine e.g. `/etc/pki/tls/private/grafana-server.key`.  This
+file should already exist.  If you want to copy a local file, see `metrics_grafana_private_key_src`.
+
+```yaml
+metrics_grafana_private_key: /etc/pki/tls/private/grafana-server.key
+```
+
+### metrics_grafana_private_key_src: ''
+
+TLS private key file from the local machine to copy to `metrics_grafana_private_key` on
+the grafana server machine.  If `metrics_grafana_private_key` is not specified, then
+`metrics_grafana_private_key_src` will be copied to `/etc/pki/tls/private/basename.key` on
+the grafana server machine, where `basename` is the basename of `metrics_grafana_private_key_src`.
+
+```yaml
+metrics_grafana_private_key_src: /my/local/grafana-server.key
+```
+
 ## Example Playbook
 
 Basic metric recording setup for each managed host only, with one

defaults/main.yml

@@ -68,3 +68,10 @@ metrics_optional_domains: []
 # Additional metrics packages that should be installed, beyond the default set,
 # to enable additional metrics, export to alternate data sinks, and so on.
 metrics_optional_packages: []
+
+# Certificate and key for grafana server
+metrics_grafana_certificates: []
+metrics_grafana_cert: ""
+metrics_grafana_private_key: ""
+metrics_grafana_cert_src: ""
+metrics_grafana_private_key_src: ""

tasks/main.yml

@@ -1,30 +1,7 @@
 # SPDX-License-Identifier: MIT
 ---
-- name: Ensure ansible_facts used by role
-  setup:
-    gather_subset: "{{ __metrics_required_facts_subsets }}"
-  when: __metrics_required_facts |
-    difference(ansible_facts.keys() | list) | length > 0
-
-- name: Determine if system is booted with systemd
-  when: __metrics_is_booted is not defined
-  block:
-    - name: Run systemctl
-      # noqa command-instead-of-module
-      command: systemctl is-system-running
-      register: __is_system_running
-      changed_when: false
-      failed_when: false
-
-    - name: Require installed systemd
-      fail:
-        msg: "Error: This role requires systemd to be installed."
-      when: '"No such file or directory" in __is_system_running.msg | d("")'
-
-    - name: Set flag to indicate that systemd runtime operations are available
-      set_fact:
-        # see https://www.man7.org/linux/man-pages/man1/systemctl.1.html#:~:text=is-system-running%20output
-        __metrics_is_booted: "{{ __is_system_running.stdout != 'offline' }}"
+- name: Ensure facts and vars used by role are set
+  include_tasks: set_vars.yml
 
 - name: Add Elasticsearch to metrics domain list
   set_fact:
@@ -137,13 +114,71 @@
     name: "{{ role_path }}/roles/pcp"
   when: metrics_provider == 'pcp'
 
-- name: Setup metric graphing service.
+- name: Manage metrics graphing service
   vars:
-    grafana_metrics_provider: "{{ metrics_provider }}"
-  include_role:
-    # noqa role-name[path]
-    name: "{{ role_path }}/roles/grafana"
+    grafana_cert: "{{ __metrics_grafana_cert_dir + '/' + metrics_grafana_certificates.0.name + '.crt'
+      if metrics_grafana_certificates | length > 0
+      else metrics_grafana_cert if metrics_grafana_cert.startswith('/')
+      else __metrics_grafana_cert_dir + '/' + metrics_grafana_cert if metrics_grafana_cert | length > 0
+      else __metrics_grafana_cert_dir + '/' + metrics_grafana_cert_src | basename
+      if metrics_grafana_cert_src | length > 0
+      else '' }}"
+    grafana_private_key: "{{ __metrics_grafana_private_key_dir + '/' + metrics_grafana_certificates.0.name + '.key'
+      if metrics_grafana_certificates | length > 0
+      else metrics_grafana_private_key if metrics_grafana_private_key.startswith('/')
+      else __metrics_grafana_private_key_dir + '/' + metrics_grafana_private_key if metrics_grafana_private_key | length > 0
+      else __metrics_grafana_private_key_dir + '/' + metrics_grafana_private_key_src | basename
+      if metrics_grafana_private_key_src | length > 0
+      else '' }}"
   when: metrics_graph_service | bool
+  block:
+    - name: Create certificates using the certificate role
+      when:
+        - metrics_grafana_certificates | length > 0
+        - ansible_facts['os_family'] == 'RedHat'
+      block:
+        - name: Check the OS version for self-sign
+          when:
+            - (ansible_facts['distribution_version'] | int == 7 and
+              metrics_grafana_certificates.0.ca == 'self-sign')
+          fail:
+            msg: >-
+              Creating a self-signed certificate is not supported on
+              {{ ansible_facts['distribution'] }}-{{
+                ansible_facts['distribution_version'] }}
+
+        - name: Create certificates using the certificate role
+          include_role:
+            name: fedora.linux_system_roles.certificate
+          vars:
+            certificate_requests: "{{ metrics_grafana_certificates }}"
+
+    - name: Copy grafana cert
+      copy:
+        src: "{{ metrics_grafana_cert_src }}"
+        dest: "{{ grafana_cert }}"
+        mode: "0644"
+        owner: root
+        group: root
+      when: metrics_grafana_cert_src | length > 0
+
+    - name: Copy grafana private key
+      copy:
+        src: "{{ metrics_grafana_private_key_src }}"
+        dest: "{{ grafana_private_key }}"
+        mode: "0600"
+        owner: root
+        group: root
+      when: metrics_grafana_private_key_src | length > 0
+      no_log: true
+
+    - name: Setup metric graphing service.
+      vars:
+        grafana_metrics_provider: "{{ metrics_provider }}"
+      include_role:
+        # noqa role-name[path]
+        name: "{{ role_path }}/roles/grafana"
+      when: metrics_graph_service | bool
 
 - name: Configure firewall
   include_tasks: firewall.yml

tasks/set_vars.yml

@@ -0,0 +1,27 @@
+# SPDX-License-Identifier: MIT
+---
+- name: Ensure ansible_facts used by role
+  setup:
+    gather_subset: "{{ __metrics_required_facts_subsets }}"
+  when: __metrics_required_facts |
+    difference(ansible_facts.keys() | list) | length > 0
+
+- name: Determine if system is booted with systemd
+  when: __metrics_is_booted is not defined
+  block:
+    - name: Run systemctl
+      # noqa command-instead-of-module
+      command: systemctl is-system-running
+      register: __is_system_running
+      changed_when: false
+      failed_when: false
+
+    - name: Require installed systemd
+      fail:
+        msg: "Error: This role requires systemd to be installed."
+      when: '"No such file or directory" in __is_system_running.msg | d("")'
+
+    - name: Set flag to indicate that systemd runtime operations are available
+      set_fact:
+        # see https://www.man7.org/linux/man-pages/man1/systemctl.1.html#:~:text=is-system-running%20output
+        __metrics_is_booted: "{{ __is_system_running.stdout != 'offline' }}"

tests/check_grafana.yml

@@ -2,7 +2,8 @@
 ---
 - name: Check if Grafana works
   uri:
-    url: http://localhost:3000/login
+    url: "{{ __metrics_grafana_protocol | d('http') }}://localhost:3000/login"
     method: GET
     status_code: 200
+    validate_certs: false
   when: __metrics_is_booted | bool

tests/restore_services_state.yml

@@ -5,7 +5,6 @@
   service_facts:
   register: final_state
 
-# yamllint disable rule:line-length
 - name: Restore state of services
   tags: tests::cleanup
   service:
@@ -26,7 +25,6 @@
     - redis
     - valkey
     - grafana-server
-# yamllint enable rule:line-length
 
 - name: Stop firewall
   service:

tests/tests_bz1855539.yml

@@ -25,6 +25,9 @@
             name: linux-system-roles.metrics
             public: true
 
+        - name: Flush handlers
+          meta: flush_handlers
+
         - name: >-
             Check if pmie configuration file on remote host is the secondary one
           command: |-
@@ -44,9 +47,6 @@
             grep -E '^\s*\S+\s+y\s+n\s+' {{ pcp_pmie_control_path }}/local
           changed_when: false
 
-        - name: Flush handlers
-          meta: flush_handlers
-
       rescue:
         - name: Handle failure case
           include_tasks: handle_test_failure.yml

tests/tests_bz1855544.yml

@@ -26,6 +26,9 @@
             name: linux-system-roles.metrics
             public: true
 
+        - name: Flush handlers
+          meta: flush_handlers
+
         - name: Check if all default datasources are configured
           include_tasks: check_default_datasources.yml
 
@@ -38,9 +41,6 @@
           changed_when: false
           when: __metrics_is_booted | bool
 
-        - name: Flush handlers
-          meta: flush_handlers
-
       rescue:
         - name: Handle test failure
           include_tasks: handle_test_failure.yml

tests/tests_verify_auth.yml

@@ -28,21 +28,15 @@
             name: linux-system-roles.metrics
             public: true
 
-        - name: Restart PMCD
-          # noqa command-instead-of-module
-          shell: systemctl restart pmcd && sleep 5
-          changed_when: false
-          when: __metrics_is_booted | bool
+        - name: Flush handlers
+          meta: flush_handlers
 
         - name: Check if SASL works
           include_tasks: "{{ item }}"
           loop:
             - check_sasl.yml
             - check_firewall_selinux.yml
 
-        - name: Flush handlers
-          meta: flush_handlers
-
       rescue:
         - name: Handle failure case
           include_tasks: handle_test_failure.yml

tests/tests_verify_basic.yml

@@ -19,6 +19,9 @@
             name: linux-system-roles.metrics
             public: true
 
+        - name: Flush handlers
+          meta: flush_handlers
+
         - name: Check if basic metrics role setup works
           include_tasks: "{{ item }}"
           loop:
@@ -27,9 +30,6 @@
             - check_pmie.yml
             - check_firewall_selinux.yml
 
-        - name: Flush handlers
-          meta: flush_handlers
-
       rescue:
         - name: Handle failure case
           include_tasks: handle_test_failure.yml