From e7a581ff86f3dc5f15b09d3d2b0cc28a7505eba8 Mon Sep 17 00:00:00 2001
From: Steve Brasier <steveb@stackhpc.com>
Date: Fri, 12 Sep 2025 14:58:27 +0000
Subject: [PATCH 1/9] wip: add TF remote state docs

---
 docs/opentofu-remote-state.md | 76 +++++++++++++++++++++++++++++++++++
 docs/production.md            |  3 ++
 2 files changed, 79 insertions(+)
 create mode 100644 docs/opentofu-remote-state.md

diff --git a/docs/opentofu-remote-state.md b/docs/opentofu-remote-state.md
new file mode 100644
index 000000000..e8dc8b931
--- /dev/null
+++ b/docs/opentofu-remote-state.md
@@ -0,0 +1,76 @@
+# OpenTofu remote state
+
+Generally, using [remote state](https://opentofu.org/docs/language/state/remote/)
+is recommended.
+
+This can be configured by adding additional files into the
+`environments/{production,staging,...}/tofu` directories. Some guidance for
+different backends is given here.
+
+
+## GitLab
+
+The following creates a state per environment.
+
+1. Create a backend file and commit it:
+
+    ```terraform
+    # environments/$ENV/tofu/backend.tf:
+    terraform {
+    backend "http" {}
+    }
+    ```
+
+2. Create a personal access token with API access (note Project tokens do not
+   appear to work):
+   - In GitLab, click on your user button at top left and select 'Preferences'.
+   - Select 'Access tokens', 'Add new token'.
+   - Optionally set an expiry date, select 'API' scope and click 'Create token'.
+   - Copy the generated secret and set an environment variable in your terminal:
+   
+        ```shell
+        export TF_PASSWORD=$SECRET
+        ```
+
+3. Create this script and commit it:
+   it:
+
+    ```shell
+    # environments/$ENV/tofu/init-gitlab-backend.sh
+    PROJECT_ID="<gitlab-project-id>"
+    TF_USERNAME="<gitlab-username>"
+    TF_STATE_NAME="$(basename $APPLIANCES_ENVIRONMENT_ROOT)"
+    TF_ADDRESS="https://gitlab.com/api/v4/projects/${PROJECT_ID}/terraform/state/${TF_STATE_NAME}"
+
+    tofu init \
+    -backend-config=address=${TF_ADDRESS} \
+    -backend-config=lock_address=${TF_ADDRESS}/lock \
+    -backend-config=unlock_address=${TF_ADDRESS}/lock \
+    -backend-config=username=${TF_USERNAME} \
+    -backend-config=password=${TF_PASSWORD} \
+    -backend-config=lock_method=POST \
+    -backend-config=unlock_method=DELETE \
+    -backend-config=retry_wait_min=5
+    ```
+
+    The project id can be found by clicking the 3-dot menu at the top right of
+    the GitLab project page.
+
+4. With the environment activated, run the script:
+
+    ```shell
+    cd environments/$ENV/tofu/
+    source init-gitlab-backend.sh
+    ```
+    
+    OpenTofu is now configured to use GitLab to store state for this environment.
+
+Repeat for each environment needing remote state. Generally, `dev` environments
+will be personal so should not need this.
+
+If the project token expires, follow the above again by add an option `-reconfigure`
+to the script.
+
+## S3
+
+TODO.
diff --git a/docs/production.md b/docs/production.md
index 83587f99f..abebf4fce 100644
--- a/docs/production.md
+++ b/docs/production.md
@@ -316,6 +316,9 @@ The value chosen should be the highest value demonstrated during testing. Note
 that any time spent blocked due to this parallelism limit does not count
 against the (un-overridable) internal OpenTofu timeout of 30 minutes
 
+Consider configuring [OpenTofu remote state](./opentofu-remote-state.md) for any
+environments which should be unique, e.g. production and staging.
+
 ## Configure appliance
 
 ### Production configuration to consider

From cd38066f6f00b94abc250588f44646d624c4bce2 Mon Sep 17 00:00:00 2001
From: Steve Brasier <steveb@stackhpc.com>
Date: Fri, 12 Sep 2025 16:27:23 +0000
Subject: [PATCH 2/9] wip s3 remote state

---
 docs/opentofu-remote-state.md | 125 +++++++++++++++++++++++++++++-----
 1 file changed, 108 insertions(+), 17 deletions(-)

diff --git a/docs/opentofu-remote-state.md b/docs/opentofu-remote-state.md
index e8dc8b931..0b5b9b874 100644
--- a/docs/opentofu-remote-state.md
+++ b/docs/opentofu-remote-state.md
@@ -1,39 +1,49 @@
 # OpenTofu remote state
 
-Generally, using [remote state](https://opentofu.org/docs/language/state/remote/)
-is recommended.
+OpenTofu supports a number of [remote state backends](https://opentofu.org/docs/language/state/remote/)
+which can be used to persist state independently of where a deployment is run.
+This allows deployments to be made from anywhere that can access the state
+without corrupting or conflicting with any existing resources from previous
+deployments.
 
-This can be configured by adding additional files into the
-`environments/{production,staging,...}/tofu` directories. Some guidance for
-different backends is given here.
+Using remote state is therefore strongly recommended for  environments which
+should only be instantiated once, e.g. `production` and `staging`.
 
+This page provides some guidance for configuring remote states using
+commonly-available backends.
+
+> [!IMPORTANT]
+> In the below replace `$ENV` with the relevant environment name.
 
 ## GitLab
 
-The following creates a state per environment.
+With the environment activated:
 
 1. Create a backend file and commit it:
 
     ```terraform
     # environments/$ENV/tofu/backend.tf:
     terraform {
-    backend "http" {}
+        backend "http" {}
     }
     ```
 
 2. Create a personal access token with API access (note Project tokens do not
    appear to work):
+   
+   TODO: appears maybe the do with `project_$ID_bot` as the username
+   
    - In GitLab, click on your user button at top left and select 'Preferences'.
    - Select 'Access tokens', 'Add new token'.
    - Optionally set an expiry date, select 'API' scope and click 'Create token'.
-   - Copy the generated secret and set an environment variable in your terminal:
-   
+   - Copy the generated secret and set an environment variable in your terminal
         ```shell
         export TF_PASSWORD=$SECRET
         ```
 
+    TODO: how does this get persisted??
+
 3. Create this script and commit it:
-   it:
 
     ```shell
     # environments/$ENV/tofu/init-gitlab-backend.sh
@@ -56,21 +66,102 @@ The following creates a state per environment.
     The project id can be found by clicking the 3-dot menu at the top right of
     the GitLab project page.
 
-4. With the environment activated, run the script:
+4. Run the script:
 
     ```shell
     cd environments/$ENV/tofu/
     source init-gitlab-backend.sh
     ```
     
-    OpenTofu is now configured to use GitLab to store state for this environment.
+OpenTofu is now configured to use GitLab to store state for this environment.
 
-Repeat for each environment needing remote state. Generally, `dev` environments
-will be personal so should not need this.
+Repeat for each environment needing remote state.
 
-If the project token expires, follow the above again by add an option `-reconfigure`
-to the script.
+If the project token expires repeat the above but with the option `-reconfigure`
+added to the script.
+
+> [!CAUTION]
+> The GitLab credentials are [persisted](https://opentofu.org/docs/language/settings/backends/configuration/#credentials-and-sensitive-data)
+> into a file `environments/$ENV/tofu/.terraform/terraform.tfstate` and any
+> plan files. These should therefore not be committed.
 
 ## S3
 
-TODO.
+For clouds with S3-compatible object storage (e.g. Ceph with [radosgw](https://docs.ceph.com/en/latest/radosgw/))
+the S3 backend can be used. This approach uses a bucket per environment.
+
+With the environment activated:
+
+1. Create a bucket:
+
+    ```shell
+    export TF_STATE_NAME="$(basename $APPLIANCES_ENVIRONMENT_ROOT)"
+    openstack container create $TF_STATE_NAME
+
+2. Create credentials:
+
+    ```shell
+    openstack ec2 credentials create
+    ```
+
+    From the returned values, set:
+    
+    ```shell
+    export AWS_ACCESS_KEY_ID= # "access" value
+    export AWS_SECRET_ACCESS_KEY= # "secret" value
+    ```
+
+    Note these are available any time by running:
+
+    ```shell
+    openstack ec2 credentials list
+    ```
+
+    TODO: Think about automating these into the activate script??
+
+3. Create a backend file and commit it, for example:
+
+    ```terraform
+    # environments/$ENV/tofu/backend.tf:
+    terraform {
+        backend "s3" {
+            endpoint = "leafcloud.store"
+            bucket = "$ENV" # ** replace with environment name **
+            key    = "environment.tfstate"
+            region = "dummy"
+            
+            skip_region_validation = true
+            skip_credentials_validation = true
+            force_path_style = true
+        }
+    }
+    ```
+
+    Note that:
+    - `endpoint` is the radosgw address. If not known this can be determined by
+      creating a public bucket, and then getting the URL using
+      Project > Containers > (your public container) > Link, which provides an
+      URL of the form `https://$ENDPOINT/swift/...`.
+      `/swift`.
+    - `region` is required but not used in radosgw, hence `skip_region_validation`.
+    - `key` is an arbitrary state file name
+    - `skip_credentials_validation`: Disables STS - this may or may not be
+      required depending on the radosgw configuration.
+    - `force_path_style`: May or may not be required depending on the radosgw
+      configuration.
+      
+4. Run:
+
+    ```shell
+    tofu init
+    ```
+
+    OpenTofu is now configured to use the cloud's S3 to store state for this
+    environment.
+
+
+TODO: consider bucket versioning??
+TODO: consider whether we should use a single bucket for both stg and prd to make
+testing better??
+
+TODO: understand -reconfigure vs -migrate-state?

From 0b24cd8bf3026c2556c67964a800534d184fb318 Mon Sep 17 00:00:00 2001
From: Steve Brasier <steveb@stackhpc.com>
Date: Thu, 18 Sep 2025 11:44:04 +0000
Subject: [PATCH 3/9] improve gitlab backend configuration

---
 docs/opentofu-remote-state.md                 | 101 ++++++++++--------
 .../site/tofu/example-backends/gitlab.tf      |  41 +++++++
 2 files changed, 99 insertions(+), 43 deletions(-)
 create mode 100644 environments/site/tofu/example-backends/gitlab.tf

diff --git a/docs/opentofu-remote-state.md b/docs/opentofu-remote-state.md
index 0b5b9b874..c4f9b64d7 100644
--- a/docs/opentofu-remote-state.md
+++ b/docs/opentofu-remote-state.md
@@ -9,17 +9,35 @@ deployments.
 Using remote state is therefore strongly recommended for  environments which
 should only be instantiated once, e.g. `production` and `staging`.
 
-This page provides some guidance for configuring remote states using
-commonly-available backends.
+This page provides guidance for configuring remote states using backends
+commonly available on OpenStack deployments.
 
 > [!IMPORTANT]
 > In the below replace `$ENV` with the relevant environment name.
 
 ## GitLab
 
-With the environment activated:
+GitLab can be used with the [http backend](https://opentofu.org/docs/language/settings/backends/http/)
+to store separate states for each environment within the GitLab project.
+Access is protected by GitLab access tokens, which in the approach below are
+persisted to local files. Therefore each repository checkout will need to
+authenticate separately, using either a separate token or a shared token from
+some external secret store.
+
+The below is based on the [official docs](https://docs.gitlab.com/user/infrastructure/iac/terraform_state/)
+but includes some missing details and is modified for common appliance workflows.
+
+### Initial setup
 
-1. Create a backend file and commit it:
+1. Create the backend file:
+
+    ```shell
+    cp environments/site/tofu/example-backends/gitlab.tf environments/$ENV/tofu
+    ```
+
+2. Modify `environments/$ENV/tofu/gitlab.tf` file to set the default for the
+   project ID. This can be found by clicking the 3-dot menu at the top right of
+   the GitLab project page.
 
     ```terraform
     # environments/$ENV/tofu/backend.tf:
@@ -28,63 +46,60 @@ With the environment activated:
     }
     ```
 
-2. Create a personal access token with API access (note Project tokens do not
-   appear to work):
-   
-   TODO: appears maybe the do with `project_$ID_bot` as the username
-   
-   - In GitLab, click on your user button at top left and select 'Preferences'.
-   - Select 'Access tokens', 'Add new token'.
-   - Optionally set an expiry date, select 'API' scope and click 'Create token'.
-   - Copy the generated secret and set an environment variable in your terminal
-        ```shell
-        export TF_PASSWORD=$SECRET
-        ```
+3. Commit it.
 
-    TODO: how does this get persisted??
+4. Follow the per-checkout steps below.
 
-3. Create this script and commit it:
+### Per-checkout configuration
 
-    ```shell
-    # environments/$ENV/tofu/init-gitlab-backend.sh
-    PROJECT_ID="<gitlab-project-id>"
-    TF_USERNAME="<gitlab-username>"
-    TF_STATE_NAME="$(basename $APPLIANCES_ENVIRONMENT_ROOT)"
-    TF_ADDRESS="https://gitlab.com/api/v4/projects/${PROJECT_ID}/terraform/state/${TF_STATE_NAME}"
-
-    tofu init \
-    -backend-config=address=${TF_ADDRESS} \
-    -backend-config=lock_address=${TF_ADDRESS}/lock \
-    -backend-config=unlock_address=${TF_ADDRESS}/lock \
-    -backend-config=username=${TF_USERNAME} \
-    -backend-config=password=${TF_PASSWORD} \
-    -backend-config=lock_method=POST \
-    -backend-config=unlock_method=DELETE \
-    -backend-config=retry_wait_min=5
-    ```
+1. Create an access token in the GitLab UI, using either:
 
-    The project id can be found by clicking the 3-dot menu at the top right of
-    the GitLab project page.
+   a. If project access tokens are available, create one via
+      Project > Settings > Access tokens.
+      The token must have `Maintainer` role and `api` scope.
+
+   b. Otherwise create a personal access token via
+      User profile > Preferences > Access tokens.
+      The token must have `api` scope.
+    
+   Copy the generated secret and set an environment variable:
 
-4. Run the script:
+   ```shell
+   export TF_VAR_gitlab_access_token=$secret
+   ```
 
+2. If using a personal access token, set the GitLab username as an environment variable:
+
+   ```shell
+   export TF_VAR_gitlab_username=$your_username
+   ```
+
+4. With the environment activated, initialise OpenTofu.
+
+    If no local state exists run:
+    
     ```shell
     cd environments/$ENV/tofu/
-    source init-gitlab-backend.sh
+    tofu init
     ```
-    
+  
+    otherwise append `-migrate-state` to the `init` command to attempt to copy
+    local state to the new backend.
+
 OpenTofu is now configured to use GitLab to store state for this environment.
 
 Repeat for each environment needing remote state.
 
-If the project token expires repeat the above but with the option `-reconfigure`
-added to the script.
-
 > [!CAUTION]
 > The GitLab credentials are [persisted](https://opentofu.org/docs/language/settings/backends/configuration/#credentials-and-sensitive-data)
 > into a file `environments/$ENV/tofu/.terraform/terraform.tfstate` and any
 > plan files. These should therefore not be committed.
 
+### Token expiry
+
+If the project token expires repeat the per-checkout configuration, but using
+`opentofu init -reconfigure` instead.
+
 ## S3
 
 For clouds with S3-compatible object storage (e.g. Ceph with [radosgw](https://docs.ceph.com/en/latest/radosgw/))
diff --git a/environments/site/tofu/example-backends/gitlab.tf b/environments/site/tofu/example-backends/gitlab.tf
new file mode 100644
index 000000000..7725ab647
--- /dev/null
+++ b/environments/site/tofu/example-backends/gitlab.tf
@@ -0,0 +1,41 @@
+variable "gitlab_username" {
+    type = string
+    description = <<-EOF
+        Username of actual GitLab user, for personal access token only.
+        Default uses bot account name, for project access token.
+    EOF
+    default = null
+}
+
+variable "gitlab_access_token" {
+    type = string
+    description = <<-EOF
+        GitLab Project or Personal access token.
+        Must have Maintainer role (for Project token) and API scope
+    EOF
+}
+
+variable "gitlab_project_id" {
+    type = string
+    description = "GitLab project ID - click 3-dot menu at the top right of project page"
+    default = # Set this here
+}
+
+locals {
+    gitlab_username = coalesce(var.gitlab_username, "project_${var.gitlab_project_id}_bot")
+    gitlab_state_name = basename(var.environment_root)
+    gitlab_state_address = "https://gitlab.com/api/v4/projects/${var.gitlab_project_id}/terraform/state/${local.gitlab_state_name}"
+}
+
+terraform {
+    backend "http" {
+        address = local.gitlab_state_address
+        lock_address = "${local.gitlab_state_address}/lock"
+        unlock_address = "${local.gitlab_state_address}/lock"
+        username = local.gitlab_username
+        password = var.gitlab_access_token
+        lock_method = "POST"
+        unlock_method = "DELETE"
+        retry_wait_min = 5
+    }
+}

From 0b39ce86e5500868cf06aa65b639e260b65b2c2a Mon Sep 17 00:00:00 2001
From: Steve Brasier <steveb@stackhpc.com>
Date: Thu, 18 Sep 2025 12:27:12 +0000
Subject: [PATCH 4/9] automate s3 creds

---
 docs/opentofu-remote-state.md                 | 98 +++++++++----------
 environments/site/tofu/example-backends/s3.tf | 26 +++++
 2 files changed, 73 insertions(+), 51 deletions(-)
 create mode 100644 environments/site/tofu/example-backends/s3.tf

diff --git a/docs/opentofu-remote-state.md b/docs/opentofu-remote-state.md
index c4f9b64d7..ba948e125 100644
--- a/docs/opentofu-remote-state.md
+++ b/docs/opentofu-remote-state.md
@@ -35,7 +35,7 @@ but includes some missing details and is modified for common appliance workflows
     cp environments/site/tofu/example-backends/gitlab.tf environments/$ENV/tofu
     ```
 
-2. Modify `environments/$ENV/tofu/gitlab.tf` file to set the default for the
+2. Modify `environments/$ENV/tofu/gitlab.tf` to set the default for the
    project ID. This can be found by clicking the 3-dot menu at the top right of
    the GitLab project page.
 
@@ -103,80 +103,76 @@ If the project token expires repeat the per-checkout configuration, but using
 ## S3
 
 For clouds with S3-compatible object storage (e.g. Ceph with [radosgw](https://docs.ceph.com/en/latest/radosgw/))
-the S3 backend can be used. This approach uses a bucket per environment.
+the S3 backend can be used. This approach uses a bucket per environment and
+derives credentials from OpenStack credentials, meaning no backend-specific
+per-checkout configuration is required.
 
-With the environment activated:
+### Initial setup
 
-1. Create a bucket:
+1. Create an S3 bucket matching the current environment name:
 
     ```shell
-    export TF_STATE_NAME="$(basename $APPLIANCES_ENVIRONMENT_ROOT)"
-    openstack container create $TF_STATE_NAME
+    openstack container create $(basename $APPLIANCES_ENVIRONMENT_ROOT)
+    ```
 
-2. Create credentials:
+2. Create `ec2` credentials:
 
     ```shell
     openstack ec2 credentials create
     ```
-
-    From the returned values, set:
     
-    ```shell
-    export AWS_ACCESS_KEY_ID= # "access" value
-    export AWS_SECRET_ACCESS_KEY= # "secret" value
-    ```
+    and make a note of the `access` field returned.
 
-    Note these are available any time by running:
+3. Create the backend file:
 
     ```shell
-    openstack ec2 credentials list
+    cp environments/site/tofu/example-backends/s3.tf environments/$ENV/tofu
     ```
 
-    TODO: Think about automating these into the activate script??
+4. Modify `environments/$ENV/tofu/s3.tf` to set the default for `s3_backend_endpoint`.
+   This is the radosgw address. If not known it can be determined by creating a
+   public bucket, and then getting the URL using
+    Project > Containers > (your public bucket) > Link
+   which provides an URL of the form `https://$ENDPOINT/swift/...`.
 
-3. Create a backend file and commit it, for example:
+5. Add the following to `environments/$ENV/activate`:
 
-    ```terraform
-    # environments/$ENV/tofu/backend.tf:
-    terraform {
-        backend "s3" {
-            endpoint = "leafcloud.store"
-            bucket = "$ENV" # ** replace with environment name **
-            key    = "environment.tfstate"
-            region = "dummy"
-            
-            skip_region_validation = true
-            skip_credentials_validation = true
-            force_path_style = true
-        }
-    }
+    ```bash
+    export AWS_ACCESS_KEY_ID=$EC2_CREDENTIALS_ACCESS
+    export AWS_SECRET_ACCESS_KEY=$(openstack ec2 credentials show $AWS_ACCESS_KEY_ID -f value -c secret)
     ```
+  
+    replacing `$EC2_CREDENTIALS_ACCESS` with the `access` field of the created
+    credentials.
+
+    This avoids these credentials being persisted in local files.
+
+6. Copy the lines above into your shell to set them for your current shell.
 
-    Note that:
-    - `endpoint` is the radosgw address. If not known this can be determined by
-      creating a public bucket, and then getting the URL using
-      Project > Containers > (your public container) > Link, which provides an
-      URL of the form `https://$ENDPOINT/swift/...`.
-      `/swift`.
-    - `region` is required but not used in radosgw, hence `skip_region_validation`.
-    - `key` is an arbitrary state file name
-    - `skip_credentials_validation`: Disables STS - this may or may not be
-      required depending on the radosgw configuration.
-    - `force_path_style`: May or may not be required depending on the radosgw
-      configuration.
-      
-4. Run:
+7. With the environment activated, initialise OpenTofu.
 
+    If no local state exists run:
+    
     ```shell
+    cd environments/$ENV/tofu/
     tofu init
     ```
+  
+    otherwise append `-migrate-state` to the `init` command to attempt to copy
+    local state to the new backend.
 
-    OpenTofu is now configured to use the cloud's S3 to store state for this
-    environment.
+8. If this fails, try setting `use_path_style = true` in `environments/$ENV/tofu/s3.tf`.
 
+9. Once it works, commit `environments/$ENV/tofu/s3.tf` and `environments/$ENV/activate`.
 
-TODO: consider bucket versioning??
-TODO: consider whether we should use a single bucket for both stg and prd to make
-testing better??
+OpenTofu is now configured to use the cloud's S3-compatible storage to store
+state for this environment. 
+
+Repeat for each environment needing remote state.
+
+For more configuration options, see the OpenTofu [s3 backend docs](https://opentofu.org/docs/language/settings/backends/s3/).
+
+### Per-checkout configuration
 
-TODO: understand -reconfigure vs -migrate-state?
+The ec2 credentials will automatically be loaded when activating the environment.
+For a new checkout simply initialise OpenTofu as normal as described in step 7 above.
diff --git a/environments/site/tofu/example-backends/s3.tf b/environments/site/tofu/example-backends/s3.tf
new file mode 100644
index 000000000..ccbfcbf48
--- /dev/null
+++ b/environments/site/tofu/example-backends/s3.tf
@@ -0,0 +1,26 @@
+variable "s3_backend_endpoint" {
+    type = string
+    description = "radosgw address without protocol or path e.g. leafcloud.store"
+    default = # Set this here
+}
+
+variable "s3"
+
+terraform {
+    backend "s3" {
+        endpoint = var.s3_backend_endpoint
+        bucket = basename(var.environment_root)
+        key    = "environment.tfstate"
+        
+        # Reginon is required but not used in radosgw:
+        region = "dummy"
+        skip_region_validation = true
+
+        # Normally STS is not configured in radosgw:
+        skip_credentials_validation = true
+        
+        # Enable path-style S3 URLs (https://<HOST>/<BUCKET> instead of https://<BUCKET>.<HOST>)
+        # may or may not be required depending on radosgw configuration
+        use_path_style = true
+    }
+}

From 4fb43830cbd0ca68e3ec053fa6efd98d943261a4 Mon Sep 17 00:00:00 2001
From: Steve Brasier <steveb@stackhpc.com>
Date: Thu, 18 Sep 2025 13:34:56 +0000
Subject: [PATCH 5/9] make s3 buckets clearer

---
 docs/opentofu-remote-state.md                 | 9 +++++++--
 environments/site/tofu/example-backends/s3.tf | 4 +---
 2 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/docs/opentofu-remote-state.md b/docs/opentofu-remote-state.md
index ba948e125..ec1d82977 100644
--- a/docs/opentofu-remote-state.md
+++ b/docs/opentofu-remote-state.md
@@ -109,10 +109,15 @@ per-checkout configuration is required.
 
 ### Initial setup
 
-1. Create an S3 bucket matching the current environment name:
+1. Create an S3 bucket with a name `${cluster_name}-${environment_name}-tfstate`
+   where:
+    - `CLUSTER_NAME` is defined in `environments/$ENV/tofu/main.tf`
+    - `$ENVIRONMENT_NAME` is the name of the environment directory
+
+    e.g.
 
     ```shell
-    openstack container create $(basename $APPLIANCES_ENVIRONMENT_ROOT)
+    openstack container create research-staging-tfstate
     ```
 
 2. Create `ec2` credentials:
diff --git a/environments/site/tofu/example-backends/s3.tf b/environments/site/tofu/example-backends/s3.tf
index ccbfcbf48..5d74dbef8 100644
--- a/environments/site/tofu/example-backends/s3.tf
+++ b/environments/site/tofu/example-backends/s3.tf
@@ -4,12 +4,10 @@ variable "s3_backend_endpoint" {
     default = # Set this here
 }
 
-variable "s3"
-
 terraform {
     backend "s3" {
         endpoint = var.s3_backend_endpoint
-        bucket = basename(var.environment_root)
+        bucket = "${var.cluster_name}-${basename(var.environment_root)}-tfstate"
         key    = "environment.tfstate"
         
         # Reginon is required but not used in radosgw:

From 10876326cb4c0e9ecd6aefaae1a0fe5fa6cb8085 Mon Sep 17 00:00:00 2001
From: Steve Brasier <steveb@stackhpc.com>
Date: Thu, 18 Sep 2025 16:29:21 +0000
Subject: [PATCH 6/9] fix linting

---
 docs/opentofu-remote-state.md                     | 6 +++---
 environments/site/tofu/example-backends/gitlab.tf | 2 +-
 environments/site/tofu/example-backends/s3.tf     | 2 +-
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/docs/opentofu-remote-state.md b/docs/opentofu-remote-state.md
index ec1d82977..5bfe1cbd0 100644
--- a/docs/opentofu-remote-state.md
+++ b/docs/opentofu-remote-state.md
@@ -61,7 +61,7 @@ but includes some missing details and is modified for common appliance workflows
    b. Otherwise create a personal access token via
       User profile > Preferences > Access tokens.
       The token must have `api` scope.
-    
+
    Copy the generated secret and set an environment variable:
 
    ```shell
@@ -77,7 +77,7 @@ but includes some missing details and is modified for common appliance workflows
 4. With the environment activated, initialise OpenTofu.
 
     If no local state exists run:
-    
+
     ```shell
     cd environments/$ENV/tofu/
     tofu init
@@ -138,7 +138,7 @@ per-checkout configuration is required.
    This is the radosgw address. If not known it can be determined by creating a
    public bucket, and then getting the URL using
     Project > Containers > (your public bucket) > Link
-   which provides an URL of the form `https://$ENDPOINT/swift/...`.
+   which provides a URL of the form `https://$ENDPOINT/swift/...`.
 
 5. Add the following to `environments/$ENV/activate`:
 
diff --git a/environments/site/tofu/example-backends/gitlab.tf b/environments/site/tofu/example-backends/gitlab.tf
index 7725ab647..50e2715ea 100644
--- a/environments/site/tofu/example-backends/gitlab.tf
+++ b/environments/site/tofu/example-backends/gitlab.tf
@@ -18,7 +18,7 @@ variable "gitlab_access_token" {
 variable "gitlab_project_id" {
     type = string
     description = "GitLab project ID - click 3-dot menu at the top right of project page"
-    default = # Set this here
+    #default = # add here
 }
 
 locals {
diff --git a/environments/site/tofu/example-backends/s3.tf b/environments/site/tofu/example-backends/s3.tf
index 5d74dbef8..04c6f86b7 100644
--- a/environments/site/tofu/example-backends/s3.tf
+++ b/environments/site/tofu/example-backends/s3.tf
@@ -1,7 +1,7 @@
 variable "s3_backend_endpoint" {
     type = string
     description = "radosgw address without protocol or path e.g. leafcloud.store"
-    default = # Set this here
+    #default = # add here
 }
 
 terraform {

From ab0047be3ffee3c741262b02b662ef1b2a537248 Mon Sep 17 00:00:00 2001
From: Steve Brasier <steveb@stackhpc.com>
Date: Thu, 18 Sep 2025 16:36:27 +0000
Subject: [PATCH 7/9] try to allow same headings at different levels in
 markdown

---
 .markdownlint.json | 5 +++++
 1 file changed, 5 insertions(+)
 create mode 100644 .markdownlint.json

diff --git a/.markdownlint.json b/.markdownlint.json
new file mode 100644
index 000000000..fba9b6e23
--- /dev/null
+++ b/.markdownlint.json
@@ -0,0 +1,5 @@
+{
+  "no-duplicate-heading": {
+    "siblings_only": true
+  }
+}

From 09836068d3249d458926080d641fa3839c9f09c8 Mon Sep 17 00:00:00 2001
From: Steve Brasier <steveb@stackhpc.com>
Date: Thu, 18 Sep 2025 16:54:31 +0000
Subject: [PATCH 8/9] fix tf lint errors

---
 docs/opentofu-remote-state.md                 |  6 +--
 .../site/tofu/example-backends/gitlab.tf      | 43 ++++++++++---------
 environments/site/tofu/example-backends/s3.tf | 37 ++++++++--------
 3 files changed, 44 insertions(+), 42 deletions(-)

diff --git a/docs/opentofu-remote-state.md b/docs/opentofu-remote-state.md
index 5bfe1cbd0..8717bf1dc 100644
--- a/docs/opentofu-remote-state.md
+++ b/docs/opentofu-remote-state.md
@@ -125,7 +125,7 @@ per-checkout configuration is required.
     ```shell
     openstack ec2 credentials create
     ```
-    
+
     and make a note of the `access` field returned.
 
 3. Create the backend file:
@@ -157,7 +157,7 @@ per-checkout configuration is required.
 7. With the environment activated, initialise OpenTofu.
 
     If no local state exists run:
-    
+
     ```shell
     cd environments/$ENV/tofu/
     tofu init
@@ -171,7 +171,7 @@ per-checkout configuration is required.
 9. Once it works, commit `environments/$ENV/tofu/s3.tf` and `environments/$ENV/activate`.
 
 OpenTofu is now configured to use the cloud's S3-compatible storage to store
-state for this environment. 
+state for this environment.
 
 Repeat for each environment needing remote state.
 
diff --git a/environments/site/tofu/example-backends/gitlab.tf b/environments/site/tofu/example-backends/gitlab.tf
index 50e2715ea..722744ac4 100644
--- a/environments/site/tofu/example-backends/gitlab.tf
+++ b/environments/site/tofu/example-backends/gitlab.tf
@@ -1,41 +1,42 @@
 variable "gitlab_username" {
-    type = string
-    description = <<-EOF
+  type        = string
+  description = <<-EOF
         Username of actual GitLab user, for personal access token only.
         Default uses bot account name, for project access token.
     EOF
-    default = null
+  default     = null
 }
 
 variable "gitlab_access_token" {
-    type = string
-    description = <<-EOF
+  type        = string
+  description = <<-EOF
         GitLab Project or Personal access token.
         Must have Maintainer role (for Project token) and API scope
     EOF
 }
 
 variable "gitlab_project_id" {
-    type = string
-    description = "GitLab project ID - click 3-dot menu at the top right of project page"
-    #default = # add here
+  type        = string
+  description = "GitLab project ID - click 3-dot menu at the top right of project page"
+  #default = # add here
 }
 
 locals {
-    gitlab_username = coalesce(var.gitlab_username, "project_${var.gitlab_project_id}_bot")
-    gitlab_state_name = basename(var.environment_root)
-    gitlab_state_address = "https://gitlab.com/api/v4/projects/${var.gitlab_project_id}/terraform/state/${local.gitlab_state_name}"
+  gitlab_username      = coalesce(var.gitlab_username, "project_${var.gitlab_project_id}_bot")
+  gitlab_state_name    = basename(var.environment_root)
+  gitlab_state_address = "https://gitlab.com/api/v4/projects/${var.gitlab_project_id}/terraform/state/${local.gitlab_state_name}"
 }
 
+# tflint-ignore: terraform_required_version
 terraform {
-    backend "http" {
-        address = local.gitlab_state_address
-        lock_address = "${local.gitlab_state_address}/lock"
-        unlock_address = "${local.gitlab_state_address}/lock"
-        username = local.gitlab_username
-        password = var.gitlab_access_token
-        lock_method = "POST"
-        unlock_method = "DELETE"
-        retry_wait_min = 5
-    }
+  backend "http" {
+    address        = local.gitlab_state_address
+    lock_address   = "${local.gitlab_state_address}/lock"
+    unlock_address = "${local.gitlab_state_address}/lock"
+    username       = local.gitlab_username
+    password       = var.gitlab_access_token
+    lock_method    = "POST"
+    unlock_method  = "DELETE"
+    retry_wait_min = 5
+  }
 }
diff --git a/environments/site/tofu/example-backends/s3.tf b/environments/site/tofu/example-backends/s3.tf
index 04c6f86b7..d47113540 100644
--- a/environments/site/tofu/example-backends/s3.tf
+++ b/environments/site/tofu/example-backends/s3.tf
@@ -1,24 +1,25 @@
 variable "s3_backend_endpoint" {
-    type = string
-    description = "radosgw address without protocol or path e.g. leafcloud.store"
-    #default = # add here
+  type        = string
+  description = "radosgw address without protocol or path e.g. leafcloud.store"
+  #default = # add here
 }
 
+# tflint-ignore: terraform_required_version
 terraform {
-    backend "s3" {
-        endpoint = var.s3_backend_endpoint
-        bucket = "${var.cluster_name}-${basename(var.environment_root)}-tfstate"
-        key    = "environment.tfstate"
-        
-        # Reginon is required but not used in radosgw:
-        region = "dummy"
-        skip_region_validation = true
+  backend "s3" {
+    endpoint = var.s3_backend_endpoint
+    bucket   = "${var.cluster_name}-${basename(var.environment_root)}-tfstate"
+    key      = "environment.tfstate"
 
-        # Normally STS is not configured in radosgw:
-        skip_credentials_validation = true
-        
-        # Enable path-style S3 URLs (https://<HOST>/<BUCKET> instead of https://<BUCKET>.<HOST>)
-        # may or may not be required depending on radosgw configuration
-        use_path_style = true
-    }
+    # Reginon is required but not used in radosgw:
+    region                 = "dummy"
+    skip_region_validation = true
+
+    # Normally STS is not configured in radosgw:
+    skip_credentials_validation = true
+
+    # Enable path-style S3 URLs (https://<HOST>/<BUCKET> instead of https://<BUCKET>.<HOST>)
+    # may or may not be required depending on radosgw configuration
+    use_path_style = true
+  }
 }

From 87213153b972af50e941fae2a773fb5dbcaf2cd8 Mon Sep 17 00:00:00 2001
From: Steve Brasier <steveb@stackhpc.com>
Date: Thu, 18 Sep 2025 17:18:11 +0000
Subject: [PATCH 9/9] fix prettier errors

---
 docs/opentofu-remote-state.md | 107 +++++++++++++++++-----------------
 1 file changed, 54 insertions(+), 53 deletions(-)

diff --git a/docs/opentofu-remote-state.md b/docs/opentofu-remote-state.md
index 8717bf1dc..c70a53f29 100644
--- a/docs/opentofu-remote-state.md
+++ b/docs/opentofu-remote-state.md
@@ -6,7 +6,7 @@ This allows deployments to be made from anywhere that can access the state
 without corrupting or conflicting with any existing resources from previous
 deployments.
 
-Using remote state is therefore strongly recommended for  environments which
+Using remote state is therefore strongly recommended for environments which
 should only be instantiated once, e.g. `production` and `staging`.
 
 This page provides guidance for configuring remote states using backends
@@ -31,20 +31,20 @@ but includes some missing details and is modified for common appliance workflows
 
 1. Create the backend file:
 
-    ```shell
-    cp environments/site/tofu/example-backends/gitlab.tf environments/$ENV/tofu
-    ```
+   ```shell
+   cp environments/site/tofu/example-backends/gitlab.tf environments/$ENV/tofu
+   ```
 
 2. Modify `environments/$ENV/tofu/gitlab.tf` to set the default for the
    project ID. This can be found by clicking the 3-dot menu at the top right of
    the GitLab project page.
 
-    ```terraform
-    # environments/$ENV/tofu/backend.tf:
-    terraform {
-        backend "http" {}
-    }
-    ```
+   ```terraform
+   # environments/$ENV/tofu/backend.tf:
+   terraform {
+       backend "http" {}
+   }
+   ```
 
 3. Commit it.
 
@@ -55,12 +55,12 @@ but includes some missing details and is modified for common appliance workflows
 1. Create an access token in the GitLab UI, using either:
 
    a. If project access tokens are available, create one via
-      Project > Settings > Access tokens.
-      The token must have `Maintainer` role and `api` scope.
+   Project > Settings > Access tokens.
+   The token must have `Maintainer` role and `api` scope.
 
    b. Otherwise create a personal access token via
-      User profile > Preferences > Access tokens.
-      The token must have `api` scope.
+   User profile > Preferences > Access tokens.
+   The token must have `api` scope.
 
    Copy the generated secret and set an environment variable:
 
@@ -74,17 +74,17 @@ but includes some missing details and is modified for common appliance workflows
    export TF_VAR_gitlab_username=$your_username
    ```
 
-4. With the environment activated, initialise OpenTofu.
+3. With the environment activated, initialise OpenTofu.
 
-    If no local state exists run:
+   If no local state exists run:
 
-    ```shell
-    cd environments/$ENV/tofu/
-    tofu init
-    ```
-  
-    otherwise append `-migrate-state` to the `init` command to attempt to copy
-    local state to the new backend.
+   ```shell
+   cd environments/$ENV/tofu/
+   tofu init
+   ```
+
+   otherwise append `-migrate-state` to the `init` command to attempt to copy
+   local state to the new backend.
 
 OpenTofu is now configured to use GitLab to store state for this environment.
 
@@ -111,60 +111,61 @@ per-checkout configuration is required.
 
 1. Create an S3 bucket with a name `${cluster_name}-${environment_name}-tfstate`
    where:
-    - `CLUSTER_NAME` is defined in `environments/$ENV/tofu/main.tf`
-    - `$ENVIRONMENT_NAME` is the name of the environment directory
 
-    e.g.
+   - `CLUSTER_NAME` is defined in `environments/$ENV/tofu/main.tf`
+   - `$ENVIRONMENT_NAME` is the name of the environment directory
+
+   e.g.
 
-    ```shell
-    openstack container create research-staging-tfstate
-    ```
+   ```shell
+   openstack container create research-staging-tfstate
+   ```
 
 2. Create `ec2` credentials:
 
-    ```shell
-    openstack ec2 credentials create
-    ```
+   ```shell
+   openstack ec2 credentials create
+   ```
 
-    and make a note of the `access` field returned.
+   and make a note of the `access` field returned.
 
 3. Create the backend file:
 
-    ```shell
-    cp environments/site/tofu/example-backends/s3.tf environments/$ENV/tofu
-    ```
+   ```shell
+   cp environments/site/tofu/example-backends/s3.tf environments/$ENV/tofu
+   ```
 
 4. Modify `environments/$ENV/tofu/s3.tf` to set the default for `s3_backend_endpoint`.
    This is the radosgw address. If not known it can be determined by creating a
    public bucket, and then getting the URL using
-    Project > Containers > (your public bucket) > Link
+   Project > Containers > (your public bucket) > Link
    which provides a URL of the form `https://$ENDPOINT/swift/...`.
 
 5. Add the following to `environments/$ENV/activate`:
 
-    ```bash
-    export AWS_ACCESS_KEY_ID=$EC2_CREDENTIALS_ACCESS
-    export AWS_SECRET_ACCESS_KEY=$(openstack ec2 credentials show $AWS_ACCESS_KEY_ID -f value -c secret)
-    ```
-  
-    replacing `$EC2_CREDENTIALS_ACCESS` with the `access` field of the created
-    credentials.
+   ```bash
+   export AWS_ACCESS_KEY_ID=$EC2_CREDENTIALS_ACCESS
+   export AWS_SECRET_ACCESS_KEY=$(openstack ec2 credentials show $AWS_ACCESS_KEY_ID -f value -c secret)
+   ```
 
-    This avoids these credentials being persisted in local files.
+   replacing `$EC2_CREDENTIALS_ACCESS` with the `access` field of the created
+   credentials.
+
+   This avoids these credentials being persisted in local files.
 
 6. Copy the lines above into your shell to set them for your current shell.
 
 7. With the environment activated, initialise OpenTofu.
 
-    If no local state exists run:
+   If no local state exists run:
+
+   ```shell
+   cd environments/$ENV/tofu/
+   tofu init
+   ```
 
-    ```shell
-    cd environments/$ENV/tofu/
-    tofu init
-    ```
-  
-    otherwise append `-migrate-state` to the `init` command to attempt to copy
-    local state to the new backend.
+   otherwise append `-migrate-state` to the `init` command to attempt to copy
+   local state to the new backend.
 
 8. If this fails, try setting `use_path_style = true` in `environments/$ENV/tofu/s3.tf`.