You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The full list of supported roles/groups is below, with those marked "*"
27
-
enabled by default in the common environment or "everything template":
28
-
- alertmanager *
29
-
- ansible_init *
30
-
- basic_users *
31
-
- cacerts
32
-
- chrony
33
-
- eessi *
34
-
- etc_hosts *
35
-
- filebeat *
36
-
- grafana *
37
-
- mysql *
38
-
- nfs *
39
-
- node_exporter *
40
-
- openhpc *
41
-
- opensearch *
42
-
- podman *
43
-
- prometheus *
44
-
- proxy
45
-
- rebuild
46
-
- selinux **
47
-
- slurm_exporter *
48
-
- slurm_stats *
49
-
- systemd **
50
-
- tuned
51
-
- fail2ban *
52
-
- firewalld *
53
-
- hpctests *
54
-
- openondemand *
55
-
- persist_hostkeys *
56
-
- compute_init
57
-
- nhc *
58
-
- openondemand_desktop *
59
-
60
-
Note that for this to work, all dnf repositories are disabled at the end of
61
-
image builds, so that `ansible.builtin.dnf` tasks work when running against
62
-
packages already installed in the image.
63
-
64
-
## Outbound internet via proxy not available to cluster users
65
-
If additional functionality is required it is possible configure Ansible to use
66
-
an authenticated http/https proxy (e.g. [squid](https://www.squid-cache.org/)).
67
-
The proxy credentials are not written to the cluster nodes so the proxy cannot
68
-
be used by cluster users.
69
-
70
-
To do this the proxy variables required in the remote environment must be
71
-
defined for the Ansible variable `appliances_remote_environment_vars`. Note
72
-
some default proxy variables are provided in `environments/common/inventory/group_vars/all/proxy.yml` so generally it will be sufficient set the proxy user, password and address and to add these to the remote environment:
This uses Ansible's [remote environment support](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_environment.html). Currrently this is suported for the following roles/groups:
93
-
- eessi: TODO: is this right though??
94
-
- manila
95
-
96
-
97
-
Although EESSI will install with the above configuration, as there is no
98
-
outbound internet access except for Ansible tasks, making it functional will
99
-
require [configuring a proxy for CVMFS](https://multixscale.github.io/cvmfs-tutorial-hpc-best-practices/access/proxy/#client-system-configuration).
100
-
101
-
102
-
103
-
## Deploying Squid using the appliance
104
-
If an external squid is not available, one can be deployed by the cluster on a
105
-
dual-homed host. See [docs/networks.md#proxies](../networks.md#proxies) for
106
-
guidance, but note a separate host should be used rather than a Slurm node, to
107
-
avoid users on that node getting direct access.
108
-
109
-
If the deploy host is RockyLinux, this could be used as the squid host by adding
2. Using [EESSI](https://www.eessi.io/docs/) necessarily requires outbound
22
+
network access for the CernVM File System. However this can be provided
23
+
via an authenticated proxy. While the proxy configuration on the cluster node
24
+
is readable by all users, this proxy can provide access only to EESSI's
25
+
CVMFS Stratum 1 servers.
26
+
27
+
## Support by feature for isolated networks
28
+
29
+
See above for definition of "Default" features. In the "Isolated?" column:
30
+
- "Y": Feature works without outbound internet access.
31
+
- "N": Known not to work.
32
+
- "?": Not investigated at present.
33
+
34
+
| Inventory group/role | Default? | Isolated? |
35
+
| ----------------------| -------- | --------- |
36
+
| alertmanager | Y | Y |
37
+
| ansible_init | Y | Y |
38
+
| basic_users | Y | Y |
39
+
| block_devices | Y | No (depreciated) |
40
+
| cacerts | - | Y |
41
+
| chrony | - | Y |
42
+
| compute_init | - | Y |
43
+
| cuda | - | ? |
44
+
| eessi | Y | Y - see above |
45
+
| etc_hosts | Y | Y |
46
+
| extra_packages | - | No |
47
+
| fail2ban | Y | Y |
48
+
| filebeat | Y | Y |
49
+
| firewalld | Y | Y |
50
+
| gateway | n/a | n/a - build only |
51
+
| grafana | Y | Y |
52
+
| hpctests | Y | Y - except hpl-solo, see above |
53
+
| k3s_agent | - | ? |
54
+
| k3s_server | - | ? |
55
+
| k9s | - | ? |
56
+
| lustre | - | ? |
57
+
| manila | Y | Y |
58
+
| mysql | Y | Y |
59
+
| nfs | Y | Y |
60
+
| nhc | Y | Y |
61
+
| node_exporter | Y | Y |
62
+
| openhpc | Y | Y |
63
+
| openondemand | Y | Y |
64
+
| openondemand_desktop | Y | Y |
65
+
| openondemand_jupyter | Y | Y |
66
+
| opensearch | Y | Y |
67
+
| podman | Y | Y |
68
+
| persist_hostkeys | Y | Y |
69
+
| prometheus | Y | Y |
70
+
| proxy | - | Y |
71
+
| resolv_conf | - | ? |
72
+
| slurm_exporter | Y | Y |
73
+
| slurm_stats | Y | Y |
74
+
| squid | - | ? |
75
+
| sshd | - | ? |
76
+
| sssd | - | ? |
77
+
| systemd | Y | Y |
78
+
| tuned | - | Y |
79
+
| update | - | No |
207
80
208
81
## Network considerations
209
82
210
-
Note that even when outbound internet access is not required, the following
211
-
(shown as OpenStack security groups/rules as displayed by Horizon) outbound access from nodes is still required to enable deployment
83
+
Even when outbound internet access is not required, nodes do require some outbound access, as well as connectivity inbound from the deploy host and
84
+
inbound connectivity for users. This section documents the minimal connectivity required, in the form of the minimally-permissive security group rules. Often default security groups are less restrictive than these.
212
85
213
-
Assuming nodes have a security group `isolated` applied:
86
+
Assuming nodes and the deploy host have a security group `isolated` applied then the following rules are required:
214
87
215
88
# allow outbound DNS
216
89
ALLOW IPv4 53/tcp to 0.0.0.0/0
@@ -223,19 +96,18 @@ Assuming nodes have a security group `isolated` applied:
223
96
# allow hosts to reach metadata server (e.g. for cloud-init keys):
224
97
ALLOW IPv4 80/tcp to 169.254.169.254/32
225
98
226
-
# allow hosts to reach squid proxy:
99
+
# optionally: allow hosts to reach squid proxy for EESSI:
227
100
ALLOW IPv4 3128/tcp to <squid cidr>
228
101
229
-
Note that DNS is required (and is configured by OpenStack if the subnet has
230
-
a gateway) because name resolution happens on the hosts, not on the proxy.
102
+
Note that name resolution happens on the hosts, not on the proxy, hence DNS is required for nodes even with a proxy.
231
103
232
-
For nodes running OpenOndemand, inbound ssh and https are also required:
104
+
For nodes running OpenOndemand, inbound ssh and https are also required
105
+
(e.g. in a security group called `isolated-ssh-https`):
233
106
234
107
ALLOW IPv4 443/tcp from 0.0.0.0/0
235
108
ALLOW IPv4 22/tcp from 0.0.0.0/0
236
109
237
-
Note the OpenTofu variables `login_security_groups` and
238
-
`nonlogin_security_groups` can be used to set security groups if requried:
110
+
If non-default security groups are required, then the OpenTofu variables `login_security_groups` and `nonlogin_security_groups` can be used to set these, e.g.:
0 commit comments