Skip to content

Commit 0e10965

Browse files
committed
README: Reorganise and various fixes
1 parent 2d38500 commit 0e10965

File tree

1 file changed

+107
-99
lines changed

1 file changed

+107
-99
lines changed

README.rst

Lines changed: 107 additions & 99 deletions
Original file line numberDiff line numberDiff line change
@@ -2,16 +2,43 @@
22
Terraform Kayobe Multinode
33
==========================
44

5-
This Terraform configuration deploys a requested amount of Instances on an OpenStack cloud, to be
6-
used as a Multinode Kayobe test environment.
5+
This Terraform configuration deploys a requested amount of instances on an OpenStack cloud, to be
6+
used as a Multinode Kayobe test environment. This includes:
77

8-
Usage
9-
=====
8+
* 1x Ansible control host
9+
* 1x seed host
10+
* controller hosts
11+
* compute hosts
12+
* Ceph storage hosts
13+
* Optional Wazuh manager host
14+
15+
The high-level workflow to deploy a cluster is as follows:
16+
17+
* Prerequisites
18+
* Configure Terraform and Ansible
19+
* Deploy infrastructure on OpenStack using Terraform
20+
* Configure Ansible control host using Ansible
21+
* Deploy multi-node OpenStack using Kayobe
22+
23+
This configuration is typically used with the `ci-multinode` environment in the
24+
`StackHPC Kayobe Configuration
25+
<https://stackhpc-kayobe-config.readthedocs.io/en/stackhpc-yoga/contributor/environments/ci-multinode.html>`__
26+
repository.
27+
28+
Prerequisites
29+
=============
1030

1131
These instructions show how to use this Terraform configuration manually. They
1232
assume you are running an Ubuntu host that will be used to run Terraform. The
13-
machine should have network access to the environment that will be created by this
14-
configuration.
33+
machine should have access to the API of the OpenStack cloud that will host the
34+
infrastructure, and network access to the Ansible control host once it has been
35+
deployed. This may be achieved by direct SSH access, a floating IP on the
36+
Ansible control host, or using an SSH bastion.
37+
38+
The OpenStack cloud should have sufficient capacity to deploy the
39+
infrastructure, and a suitable image registered in Glance. Ideally the image
40+
should be one of the overcloud host images defined in StackHPC Kayobe
41+
configuration and available in `Ark <https://ark.stackhpc.com>`__.
1542

1643
Install Terraform:
1744

@@ -22,21 +49,24 @@ Install Terraform:
2249
sudo apt update
2350
sudo apt install git terraform
2451
25-
Clone and initialise the Kayobe config:
52+
Clone and initialise this Terraform config repository:
2653

2754
.. code-block:: console
2855
2956
git clone https://github.com/stackhpc/terraform-kayobe-multinode
3057
cd terraform-kayobe-multinode
3158
32-
3359
Initialise Terraform:
3460

3561
.. code-block:: console
3662
3763
terraform init
3864
39-
Generate an SSH keypair:
65+
Generate an SSH keypair. The public key will be registered in OpenStack as a
66+
keypair and authorised by the instances deployed by Terraform. The private and
67+
public keys will be transferred to the Ansible control host to allow it to
68+
connect to the other hosts. Note that password-protected keys are not currently
69+
supported.
4070

4171
.. code-block:: console
4272
@@ -94,56 +124,51 @@ Or you can source the provided `init.sh` script which shall initialise terraform
94124
OpenStack Cloud Name: sms-lab
95125
Password:
96126
97-
Generate Terraform variables:
127+
You must ensure that you have `Ansible installed <https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html>`_ on your local machine.
98128

99129
.. code-block:: console
100130
101-
cat << EOF > terraform.tfvars
102-
prefix = "changeme"
131+
pip install --user ansible
103132
104-
ansible_control_vm_flavor = "general.v1.small"
105-
ansible_control_vm_name = "ansible-control"
106-
ansible_control_disk_size = 100
133+
Install the Ansible galaxy requirements.
107134

108-
seed_vm_flavor = "general.v1.small"
109-
seed_disk_size = 100
135+
.. code-block:: console
110136
111-
multinode_flavor = "general.v1.medium"
112-
multinode_image = "Rocky9-lvm"
113-
multinode_keypair = "changeme"
114-
multinode_vm_network = "stackhpc-ipv4-geneve"
115-
multinode_vm_subnet = "stackhpc-ipv4-geneve-subnet"
116-
compute_count = "2"
117-
controller_count = "3"
118-
compute_disk_size = 100
119-
controller_disk_size = 100
137+
ansible-galaxy install -r ansible/requirements.yml
120138
121-
ssh_public_key = "~/.ssh/changeme.pub"
122-
ssh_user = "cloud-user"
139+
If the deployed instances are behind an SSH bastion you must ensure that your SSH config is setup appropriately with a proxy jump.
123140

124-
storage_count = "3"
125-
storage_flavor = "general.v1.small"
126-
storage_disk_size = 100
141+
.. code-block:: console
127142
128-
deploy_wazuh = true
129-
infra_vm_flavor = "general.v1.small"
130-
infra_vm_disk_size = 100
143+
Host lab-bastion
144+
HostName BastionIPAddr
145+
User username
146+
IdentityFile ~/.ssh/key
131147
132-
add_ansible_control_fip = false
133-
ansible_control_fip_pool = ""
134-
EOF
148+
Host 10.*
149+
ProxyJump=lab-bastion
150+
ForwardAgent no
151+
IdentityFile ~/.ssh/key
152+
UserKnownHostsFile /dev/null
153+
StrictHostKeyChecking no
154+
155+
Configure Terraform variables
156+
=============================
157+
158+
Populate Terraform variables in `terraform.tfvars`. Examples are provided in
159+
files named `*.tfvars.example`.
135160

136161
You will need to set the `multinode_keypair`, `prefix`, and `ssh_public_key`.
137162
By default, Rocky Linux 9 will be used but Ubuntu Jammy is also supported by
138-
changing `multinode_image` to `Ubuntu-22.04-lvm` and `ssh_user` to `ubuntu`.
139-
Other LVM images should also work but are untested.
163+
changing `multinode_image` to `overcloud-ubuntu-jammy-<release>-<datetime>` and
164+
`ssh_user` to `ubuntu`.
140165

141166
The `multinode_flavor` will change the flavor used for controller and compute
142167
nodes. Both virtual machines and baremetal are supported, but the `*_disk_size`
143168
variables must be set to 0 when using baremetal host. This will stop a block
144169
device being allocated. When any baremetal hosts are deployed, the
145170
`multinode_vm_network` and `multinode_vm_subnet` should also be changed to
146-
`stackhpc-ipv4-vlan-v2` and `stackhpc-ipv4-vlan-subnet-v2` respectively.
171+
a VLAN network and associated subnet.
147172

148173
If `deploy_wazuh` is set to true, an infrastructure VM will be created that
149174
hosts the Wazuh manager. The Wazuh deployment playbooks will also be triggered
@@ -155,6 +180,17 @@ and attached to the Ansible control host. In that case
155180
which to allocate the floating IP, and the floating IP will be used for SSH
156181
access to the control host.
157182

183+
Configure Ansible variables
184+
===========================
185+
186+
Review the vars defined within `ansible/vars/defaults.yml`. In here you can customise the version of kayobe, kayobe-config or openstack-config.
187+
Make sure to define `ssh_key_path` to point to the location of the SSH key in use by the nodes and also `vxlan_vni` which should be unique value between 1 to 100,000.
188+
VNI should be much smaller than the officially supported limit of 16,777,215 as we encounter errors when attempting to bring interfaces up that use a high VNI.
189+
You must set `vault_password_path`; this should be set to the path to a file containing the Ansible vault password.
190+
191+
Deployment
192+
==========
193+
158194
Generate a plan:
159195

160196
.. code-block:: console
@@ -167,91 +203,63 @@ Apply the changes:
167203
168204
terraform apply -auto-approve
169205
170-
You should have requested a number of resources spawned on Openstack, and an ansible_inventory file produced as output for Kayobe.
171-
172-
Copy your generated id_rsa and id_rsa.pub to ~/.ssh/ on Ansible control host if you want Kayobe to automatically pick them up during bootstrap.
206+
You should have requested a number of resources to be spawned on Openstack.
173207

174208
Configure Ansible control host
209+
==============================
175210

176-
Using the `deploy-openstack-config.yml` playbook you can setup the Ansible control host to include the kayobe/kayobe-config repositories with `hosts` and `admin-oc-networks`.
177-
It shall also setup the kayobe virtual environment, allowing for immediate configuration and deployment of OpenStack.
178-
179-
First you must ensure that you have `Ansible installed <https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html>`_ on your local machine.
211+
Run the configure-hosts.yml playbook to configure the Ansible control host.
180212

181213
.. code-block:: console
182214
183-
pip install --user ansible
184-
185-
Secondly if the machines are behind an SSH bastion you must ensure that your ssh config is setup appropriately with a proxy jump
215+
ansible-playbook -i ansible/inventory.yml ansible/configure-hosts.yml
186216
187-
.. code-block:: console
217+
This playbook sequentially executes 2 other playbooks:
188218

189-
Host lab-bastion
190-
HostName BastionIPAddr
191-
User username
192-
IdentityFile ~/.ssh/key
219+
#. ``grow-control-host.yml`` - Applies LVM configuration to the control host to ensure it has enough space to continue with the rest of the deployment. Tag: ``lvm``
220+
#. ``deploy-openstack-config.yml`` - Prepares the Ansible control host as a Kayobe control host, cloning the Kayobe configuration and installing virtual environments. Tag: ``deploy``
193221

194-
Host 10.*
195-
ProxyJump=lab-bastion
196-
ForwardAgent no
197-
IdentityFile ~/.ssh/key
198-
UserKnownHostsFile /dev/null
199-
StrictHostKeyChecking no
222+
These playbooks are tagged so that they can be invoked or skipped using `tags` or `--skip-tags` as required.
200223

201-
Install the ansible requirements.
224+
Deploy OpenStack
225+
================
202226

203-
.. code-block:: console
227+
Once the Ansible control host has been configured with a Kayobe/OpenStack configuration you can then begin the process of deploying OpenStack.
228+
This can be achieved by either manually running the various commands to configure the hosts and deploy the services or automated by using the generated `deploy-openstack.sh` script.
229+
`deploy-openstack.sh` should be available within the home directory on your Ansible control host provided you ran `deploy-openstack-config.yml` earlier.
230+
This script will go through the process of performing the following tasks:
204231

205-
ansible-galaxy install -r ansible/requirements.yml
232+
* kayobe control host bootstrap
233+
* kayobe seed host configure
234+
* kayobe overcloud host configure
235+
* cephadm deployment
236+
* kayobe overcloud service deploy
237+
* openstack configuration
238+
* tempest testing
206239

207-
Review the vars defined within `ansible/vars/defaults.yml`. In here you can customise the version of kayobe, kayobe-config or openstack-config.
208-
However, make sure to define `ssh_key_path` to point to the location of the SSH key in use amongst the nodes and also `vxlan_vni` which should be unique value between 1 to 100,000.
209-
VNI should be much smaller than the officially supported limit of 16,777,215 as we encounter errors when attempting to bring interfaces up that use a high VNI. You must set``vault_password_path``; this should be set to the path to a file containing the Ansible vault password.
240+
Tempest test results will be written to ~/tempest-artifacts.
210241

211-
Finally, run the configure-hosts playbook.
242+
If you choose to opt for automated method you must first SSH into your Ansible control host.
212243

213244
.. code-block:: console
214245
215-
ansible-playbook -i ansible/inventory.yml ansible/configure-hosts.yml
216-
217-
This playbook sequentially executes 2 other playbooks:
218-
219-
#. ``grow-control-host.yml`` - Applies LVM configuration to the control host to ensure it has enough space to continue with the rest of the deployment. Tag: ``lvm``
220-
#. ``deploy-openstack-config.yml`` - Deploys the OpenStack configuration to the control host. Tag: ``deploy``
246+
ssh $(terraform output -raw ssh_user)@$(terraform output -raw ansible_control_access_ip_v4)
247+
~/deploy-openstack.sh
221248
222-
These playbooks are tagged so that they can be invoked or skipped as required. For example, if designate is not being deployed, some time can be saved by skipping the FQDN playbook:
249+
Start a `tmux` session to avoid halting the deployment if you are disconnected.
223250

224251
.. code-block:: console
225252
226-
ansible-playbook -i ansible/inventory.yml ansible/configure-hosts.yml --skip-tags lvm
253+
tmux
227254
228-
Deploy OpenStack
229-
----------------
230-
231-
Once the Ansible control host has been configured with a Kayobe/OpenStack configuration you can then begin the process of deploying OpenStack.
232-
This can be achieved by either manually running the various commands to configures the hosts and deploy the services or automated by using `deploy-openstack.sh`,
233-
which should be available within the homedir on your Ansible control host provided you ran `deploy-openstack-config.yml` earlier.
234-
235-
If you choose to opt for automated method you must first SSH into your Ansible control host and then run the `deploy-openstack.sh` script
255+
Run the `deploy-openstack.sh` script.
236256

237257
.. code-block:: console
238258
239-
ssh $(terraform output -raw ssh_user)@$(terraform output -raw ansible_control_access_ip_v4)
240259
~/deploy-openstack.sh
241260
242-
This script will go through the process of performing the following tasks
243-
* kayobe control host bootstrap
244-
* kayobe seed host configure
245-
* kayobe overcloud host configure
246-
* cephadm deployment
247-
* kayobe overcloud service deploy
248-
* openstack configuration
249-
* tempest testing
250-
251-
Tempest test results are written to ~/tempest-artifacts.
252-
253261
Accessing OpenStack
254-
-------------------
262+
===================
255263

256264
After a successful deployment of OpenStack you make access the OpenStack API and Horizon by proxying your connection via the seed node, as it has an interface on the public network (192.168.39.X).
257265
Using software such as sshuttle will allow for easy access.
@@ -268,15 +276,15 @@ Important to node this will proxy all DNS requests from your machine to the firs
268276
sshuttle -r $(terraform output -raw ssh_user)@$(terraform output -raw seed_access_ip_v4) 192.168.39.0/24 --dns --to-ns 192.168.39.4
269277
270278
Tear Down
271-
---------
279+
=========
272280

273281
After you are finished with the multinode environment please destroy the nodes to free up resources for others.
274282
This can acomplished by using the provided `scripts/tear-down.sh` which will destroy your controllers, compute, seed and storage nodes whilst leaving your Ansible control host and keypair intact.
275283

276284
If you would like to delete your Ansible control host then you can pass the `-a` flag however if you would also like to remove your keypair then pass `-a -k`
277285

278286
Issues & Fixes
279-
--------------
287+
==============
280288

281289
Sometimes a compute instance fails to be provisioned by Terraform or fails on boot for any reason.
282290
If this happens the solution is to mark the resource as tainted and perform terraform apply again which shall destroy and rebuild the failed instance.

0 commit comments

Comments
 (0)