make eeesi/squid autoconfigure for proxy

Author: sjpb

ansible/roles/eessi/README.md

@@ -8,10 +8,20 @@ None.
 
 ## Role Variables
 
-- `cvmfs_quota_limit_mb`: Optional int. Maximum size of local package cache on each node in MB.
-- `cvmfs_config_overrides`: Optional dict. Set of key-value pairs for additional CernVM-FS settings see [official docs](https://cvmfs.readthedocs.io/en/stable/cpt-configure.html) for list of options.
-  Each dict key should correspond to a valid config variable (e.g. `CVMFS_HTTP_PROXY`) and the corresponding dict value will be set as the variable value (e.g. `https://my-proxy.com`).
-  These configuration parameters will be written to the `/etc/cvmfs/default.local` config file on each host in the form `KEY=VALUE`.
+All variables relate to [CernVM-FS configuration](https://cvmfs.readthedocs.io/en/stable/cpt-configure.html).
+By default, the configuration is that [recommended by EESSI for single clients](https://www.eessi.io/docs/getting_access/native_installation/#installation-for-single-clients).
+However if `cvmfs_http_proxy` is set to a non-empty string then a configuration
+suitable for using a [squid proxy](https://www.eessi.io/docs/getting_access/native_installation/#configuring-your-client-to-use-a-squid-proxy)
+is applied instead. See [docs/production](../../../docs/eessi.md#eessi-proxy-configuration)
+for guidance on appliance configuration.
+
+- `cvmfs_quota_limit_mb`: Optional int. Maximum size of local package cache on
+  each node in MB. Default 10GB.
+- `cvmfs_http_proxy`: Optional string. Value for [CVMFS_HTTP_PROXY](https://cvmfs.readthedocs.io/en/stable/cpt-configure.html#proxy-lists). Quotes are added around the provided value. Default empty string.
+- `cvmfs_config_overrides`: Optional dict. Set of key-value pairs for additional
+  CernVM-FS settings, written to `/etc/cvmfs/default.local`. Keys are
+  [CVMFS configuration options](https://cvmfs.readthedocs.io/en/stable/cpt-configure.html)
+  (e.g. `CVMFS_TIMEOUT_DIRECT`). Default empty dict.
 
 ## Dependencies
 

ansible/roles/eessi/defaults/main.yaml

@@ -1,13 +1,22 @@
 ---
 cvmfs_release_version: "6-3"
 
-# Default to 10GB
-cvmfs_quota_limit_mb: 10000
+cvmfs_quota_limit_mb: 10000 # local cache soft quota in MB (default 10GB)
 
-cvmfs_config_default:
+# NB: The string omit removes the option. Defined here for both default configs
+# so make swapping between them work properly.
+# TODO explain the omits
+cvmfs_config_single:
   CVMFS_CLIENT_PROFILE: single
   CVMFS_QUOTA_LIMIT: "{{ cvmfs_quota_limit_mb }}"
 
+cvmfs_http_proxy: '' # as per docs, quotes are added automatically
+# See https://www.eessi.io/docs/getting_access/native_installation/#configuring-your-client-to-use-a-squid-proxy
+cvmfs_config_proxy:
+  CVMFS_QUOTA_LIMIT: "{{ cvmfs_quota_limit_mb }}"
+  CVMFS_HTTP_PROXY: "'{{ cvmfs_http_proxy }}'"
+
+cvmfs_config_default: "{{ cvmfs_config_single if cvmfs_http_proxy == '' else cvmfs_config_proxy }}"
 cvmfs_config_overrides: {}
 cvmfs_config: "{{ cvmfs_config_default | combine(cvmfs_config_overrides) }}"
 

ansible/roles/eessi/tasks/configure.yml

@@ -1,21 +1,23 @@
 ---
 
 - name: Add base CVMFS config
-  community.general.ini_file:
+  ansible.builtin.template:
     dest: /etc/cvmfs/default.local
-    section: null
-    option: "{{ item.key }}"
-    value: "{{ item.value }}"
-    no_extra_spaces: true
-    mode: "0644"
-  loop: "{{ cvmfs_config | dict2items }}"
-
+    src: cvmfs.config.j2
+    mode: u=rw,go=r
+    owner: root
+  register: cvmfs_config
 
 # NOTE: Not clear how to make this idempotent
 - name: Ensure CVMFS config is setup # noqa: no-changed-when
   ansible.builtin.command:
     cmd: "cvmfs_config setup"
 
+- name: Reload CVMFS config
+  ansible.builtin.command:
+    cmd: cvmfs_config reload
+  when: cvmfs_config.changed
+
 # configure gpus
 - name: Check for NVIDIA GPU
   ansible.builtin.stat:

ansible/roles/squid/README.md

@@ -3,13 +3,13 @@
 Deploy a caching proxy.
 
 **NB:** This role provides two default configurations, selected by setting
-`squid_conf_template`:
-- `squid.conf.j2`: This is aimed at providing a proxy for package installs etc.
+`squid_conf_mode`:
+- `default`: This is aimed at providing a proxy for package installs etc.
   for nodes which do not have direct internet connectivity. It assumes access
   to the proxy is protected by the OpenStack security groups applied to the
   cluster. The generated configuration should be reviewed if this is not case.
-- `squid-eessi.conf.j2`: This provides a proxy server for EESSI clients. It uses
-  the [recommended configuration](https://www.eessi.io/docs/tutorial/access/proxy/#configuration)
+- `eessi`: This provides a proxy server for EESSI clients. It uses the
+  [recommended configuration](https://www.eessi.io/docs/tutorial/access/proxy/#configuration)
   which assumes a server with:
       - 10Gbit link or faster to the client systems
       - a sufficiently powerful CPU
@@ -19,13 +19,15 @@ Deploy a caching proxy.
   least one for every (100-500) client nodes.
 
 ## Role Variables
+
+- `squid_conf_mode`: Optional str, `default` (the default) or `eessi`. See above.
 - `squid_conf_template`: Optional str. Path (using Ansible search paths) to
-  squid.conf template. Default is in-role `squid.conf.j2` template as above.
+  squid.conf template. Default is in-role templates. If this is overriden then
+  `squid_conf_mode` has no effect.
 
-### Role Variables for squid_conf_template=squid.conf.j2
+### Role Variables for squid_conf_mode: default
 
 Where noted these map to squid parameters of the same name without the `squid_` prefix - see [squid documentation](https://www.squid-cache.org/Doc/config) for details.
-
 - `squid_started`: Optional bool. Whether to start squid service. Default `true`.
 - `squid_enabled`: Optional bool. Whether squid service is enabled on boot. Default `true`.
 - `squid_cache_mem`: Required str. Size of memory cache, e.g "1024 KB", "12 GB" etc. See squid parameter.
@@ -52,11 +54,13 @@ Where noted these map to squid parameters of the same name without the `squid_`
 
   See squid parameter.
 
-### Role Variables for squid_conf_template=squid-eessi.conf.j2
+### Role Variables for squid_conf_mode: eessi
 
-- `squid_eessi_clients`: Optional string. CIDR specifying clients allowed to
-  access this proxy. The default is to use the CIDR of the host's default IPv4
-  interface, which should allow access from the [cluster network](../../../docs/networks.md).
-  For clusters with multiple networks this may not be appropriate.
-- `squid_eessi_stratum_1`: Optional string. Domain (squid `acl dstdomain` format)
-  of Stratum 1 replica servers. Default is the upstream EEESI Stratum 1 servers.
+- `squid_eessi_clients`: Optional str. CIDR specifying clients allowed to access
+  this proxy. Default is the CIDR for the subnet of the [access network](../../../docs/networks.md),
+  i.e. the first cluster network. For clusters with multiple networks this may
+  need overriding.
+- `squid_eessi_stratum_1`: Optional str. Domain (in squid `acl dstdomain`
+  format) of Stratum 1 replica servers. Defaults to upstream EEESI Stratum 1
+  servers.
+- `squid_cache_dir`: See definition for default mode above.

ansible/roles/squid/defaults/main.yml

@@ -1,6 +1,8 @@
 ---
-## squid dnf configuration:
-squid_conf_template: squid.conf.j2
+squid_conf_mode: default # or 'eessi'
+
+## squid_conf_mode=default:
+squid_conf_template: "squid-{{ squid_conf_mode }}.conf.j2"
 squid_started: true
 squid_enabled: true
 
@@ -25,7 +27,6 @@ squid_http_access: |
   # Finally deny all other access to this proxy
   http_access deny all
 
-## squid eeesi configuration:
-#squid_conf_template: squid-eessi.conf.j2
-squid_eessi_clients: "{{ ansible_default_ipv4.network }}/{{ ansible_default_ipv4.prefix }}"
+#squid_conf_mode=eessi:
+squid_eessi_clients: "{{ cluster_subnets[0].cidr | mandatory('squid_eessi_clients must be defined when using eeesi squid config') }}"
 squid_eessi_stratum_1: '.eessi.science'

docs/eessi.md

@@ -2,7 +2,7 @@
 
 ## How to Load EESSI
 
-The EESSI environment can be initialise by running:
+The EESSI environment can be initialised by running:
 
 ```bash
 source /cvmfs/software.eessi.io/versions/2023.06/init/bash
@@ -109,3 +109,46 @@ cmake ..
 make
 ./deviceQuery
 ```
+
+## EESSI Proxy Configuration
+
+EESSI recommend that clusters use a proxy to reduce latency for clients and
+avoid excessive load on the EESSI Stratum 1 servers. Squid can be deployed and
+configured appropriately as this proxy on separate node(s) using
+the OpenTofu variable `additional_nodegroups`, e.g.:
+
+```hcl
+additional_nodegroups = {
+    # EESSI squid proxy
+    squid = {
+      nodes = ["squid-0"]
+      flavor = squid.flavor
+    }
+  }
+```
+
+EESSI [recommend](https://www.eessi.io/docs/tutorial/access/proxy/#general-recommendations)
+that:
+> The proxy server should have a 10Gbit link to the client systems, a
+sufficiently powerful CPU, a decent amount of memory for the kernel cache (tens
+of GBs), and fast local storage (SSD or NVMe).
+>
+> As a rule of thumb, it is recommended to have (at least) one proxy server for
+every couple of hundred worker nodes (100-500).
+
+Generally, both the `squid` nodes and the `eeesi` client nodes can be
+appropriately configured simply by setting the squid mode to `eessi`:
+
+  ```yaml
+  # environments/site/inventory/group_vars/all/squid.yml:
+  squid_conf_mode: eessi
+  ```
+
+In this mode, by default:
+- `squid` is configured to allow clients from the access network's CIDR, using
+  the EESSI-recommended cache configuration
+- `eessi` is configured to use the `squid` node IPs on the access network (the
+  first network in `cluster_networks`) as proxies.
+
+If this is not suitable then override the defaults provided by `environments/common/inventory/group_vars/all/eessi.yml`
+and the `eeesi` and `squid` roles.

docs/production.md

@@ -163,33 +163,47 @@ will have been generated for you already under
 
 ## Define and deploy infrastructure
 
-Create an OpenTofu variables file to define the required infrastructure, e.g.:
-
-```text
-# environments/$ENV/tofu/terraform.tfvars
-cluster_name = "mycluster"
-cluster_networks = [
-  {
-    network = "some_network" # *
-    subnet = "some_subnet" # *
-  }
-]
-key_pair = "my_key" # *
-control_node_flavor = "some_flavor_name"
-login = {
-    # Arbitrary group name for these login nodes
-    interactive = {
-        nodes: ["login-0"]
-        flavor: "login_flavor_name" # *
+Modify the cookiecutter-templtaed OpenTofu configuration to define the required
+infrastructure, e.g.:
+
+```hcl
+# environments/$ENV/tofu/main.tf
+module "cluster" {
+  source           = "../../site/tofu/"
+  environment_root = var.environment_root
+
+  cluster_name = "mycluster"
+  cluster_networks = [
+    {
+      network = "some_network" # *
+      subnet = "some_subnet" # *
     }
-}
-cluster_image_id = "rocky_linux_9_image_uuid"
-compute = {
+  ]
+  key_pair = "my_key" # *
+  control_node_flavor = "some_flavor_name"
+  login = {
+      # Arbitrary group name for these login nodes
+      head = {
+        nodes = ["login-0"]
+        flavor = "login_flavor_name" # *
+      }
+  }
+  cluster_image_id = "rocky_linux_9_image_uuid"
+  compute = {
     # Group name used for compute node partition definition
     general = {
-        nodes: ["compute-0", "compute-1"]
-        flavor: "compute_flavor_name" # *
+      nodes = ["compute-0", "compute-1"]
+      flavor = "compute_flavor_name" # *
+    }
+  }
+  additional_nodes = {
+    # Nodes configured to provide a squid proxy for EESSI - for guidance
+    # on number and sizing see [docs/eessi.md](./eessi.md#eessi-proxy-configuration)
+    squid = {
+      nodes = ["squid-0"]
+      flavor = squid_flavor_name # *
     }
+  }
 }
 ```
 
@@ -203,7 +217,7 @@ Note that:
 - Environment-specific variables (`cluster_name`) should be hardcoded into
   the cluster module block.
 
-- Environment-independent variables (e.g. maybe `cluster_net` if the same
+- Environment-independent variables (e.g. maybe `cluster_networks` if the same
   is used for staging and production) should be set as _defaults_ in
   `environments/site/tofu/variables.tf`, and then don't need to be passed
   in to the module.
@@ -356,6 +370,16 @@ environments which should be unique, e.g. production and staging.
   not, remove `grafana_auth_anonymous` in
   `environments/$ENV/inventory/group_vars/all/grafana.yml`
 
+- Configure EESSI to be proxied via the `squid` node(s) defined in the OpenTofu
+  configuration:
+
+  ```yaml
+  # environments/site/inventory/group_vars/all/squid.yml:
+  squid_conf_mode: eessi
+  ```
+
+  See [docs/eessi](./eessi.md#eessi-proxy-configuration) for more information.
+
 - See the [hpctests docs](../ansible/roles/hpctests/README.md) for advice on
   raising `hpctests_hpl_mem_frac` during tests.
 
@@ -409,3 +433,5 @@ Once it completes you can log in to the cluster using:
 
 For further information, including additional configuration guides and
 operations instructions, see the [docs](README.md) directory.
+
+TODO: Add stuff on eessi proxy.

environments/common/inventory/group_vars/all/eessi.yml

@@ -0,0 +1,17 @@
+# Automatically configure EESSI clients to use any 'squid' node(s) with squid_conf_mode == eessi as a proxy:
+cvmfs_proxy_ip_var: ansible_host # hostvar with proxy IP - default is IP on access network
+cvmfs_proxy_ips: >- # list of IPs for squid nodes with squid_conf_mode == eessi:
+  {{
+    hostvars.values() |
+    selectattr('group_names', 'contains', 'squid') |
+    selectattr('squid_conf_mode', 'eq', 'eessi') |
+    map(attribute=cvmfs_proxy_ip_var)
+  }}
+# proxy string as per EESSI docs (but unquoted), or empty string:
+# TODO: just check final format eg "http://10.8.1.16:3128|http://10.8.1.17:3128" looks ok
+cvmfs_http_proxy: >-
+  {{
+    cvmfs_proxy_ips |
+    map('regex_replace', '^(.*)$', 'http://\1:' ~ (squid_http_port | string)) |
+    join('|')
+  }}

environments/common/inventory/group_vars/all/squid.yml

@@ -1,2 +1,3 @@
 ---
-squid_http_port: 3128 # defined here for proxy role
+squid_http_port: 3128 # defined here for proxy/eeesi roles
+squid_conf_mode: default # defined here for eeesi config