docs: update the README.md, justfile

Author: johnalotoski

Justfile

@@ -761,7 +761,7 @@ show-nameservers:
   print $"Nameservers for domain: ($domain) \(hosted zone id: ($id)) are:"
   print ($ns | to text)
 
-# Decrypt a file to stdout
+# Decrypt a file to stdout using .sops.yaml rules
 sops-decrypt-binary FILE:
   #!/usr/bin/env bash
   set -euo pipefail
@@ -772,7 +772,7 @@ sops-decrypt-binary FILE:
   # This supports the common use case of obtaining decrypted state for cmd arg input while leaving the encrypted file intact on disk.
   sops --config "$(sops_config {{FILE}})" --input-type binary --output-type binary --decrypt {{FILE}}
 
-# Decrypt a file in place
+# Decrypt a file in place using .sops.yaml rules
 sops-decrypt-binary-in-place FILE:
   #!/usr/bin/env bash
   set -euo pipefail
@@ -781,7 +781,7 @@ sops-decrypt-binary-in-place FILE:
 
   sops --config "$(sops_config {{FILE}})" --input-type binary --output-type binary --decrypt {{FILE}} | sponge {{FILE}}
 
-# Encrypt a file in place
+# Encrypt a file in place using .sops.yaml rules
 sops-encrypt-binary FILE:
   #!/usr/bin/env bash
   set -euo pipefail
@@ -792,7 +792,7 @@ sops-encrypt-binary FILE:
   # This supports the common use case of first time encrypting plaintext state for public storage, ex: git repo commit.
   sops --config "$(sops_config {{FILE}})" --input-type binary --output-type binary --encrypt {{FILE}} | sponge {{FILE}}
 
-# Rotate sops encryption
+# Rotate sops encryption using .sops.yaml rules
 sops-rotate-binary FILE:
   #!/usr/bin/env bash
   set -euo pipefail

README.md

@@ -37,45 +37,17 @@ cardano-playground organization, then store your access key in
     aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
     aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 
-## AGE
+## AGE Admin
 
-While cluster secrets are handled using AWS KMS, per machine secrets are
-handled using sops-nix age.  For sops-nix age secrets access, place the
-SRE cluster secret in `~/.age/credentials`:
+While cluster secrets shared by all machines are generally handled using AWS
+KMS, per machine secrets are handled using sops-nix age.  However, an admin age
+key is still typically desired so that all per machine secrets can be decrypted
+by an admin or SRE.  A new age admin key can be generated with `age-keygen` and
+this should be placed in `~/.age/credentials`:
 
     # cardano-playground: sre
     AGE-SECRET-KEY-***********************************************************
 
-If needed, a new secret can be generated with `age-keygen`.
-
-## SSH
-
-If your credentials are correct, and the cluster is already provisioned with
-openTofu infrastructure, you will be able to access SSH after creating an
-`./.ssh_config` using:
-
-    just save-ssh-config
-
-With that you can then get started with:
-
-    # List machines
-    just list-machines
-
-    # Ssh to a newly provisioned machine
-    just ssh-bootstrap $MACHINE
-
-    # Deploy to a newly provisioned machine
-    just apply-bootstrap $MACHINE
-
-    # Ssh to a machine already deployed
-    just ssh $MACHINE
-
-    # Deploy to a machine already deployed
-    just apply $MACHINE
-
-    # Find many other operations recipes to use
-    just --list
-
 ## Cloudformation
 
 We bootstrap our infrastructure using AWS Cloudformation, it creates resources
@@ -116,6 +88,29 @@ Similarly, for monitoring resources:
     just tofu grafana plan
     just tofu grafana apply
 
+## SSH
+
+If your credentials are correct, and the cluster is already provisioned with
+openTofu infrastructure, you will be able to access SSH after creating an
+`./.ssh_config` and nix ip module information using:
+
+    just save-ssh-config
+    just update-ips
+
+With that you can then get started with:
+
+    # List machines
+    just list-machines
+
+    # Ssh to a newly provisioned machine
+    just ssh-bootstrap $MACHINE
+
+    # Ssh to a machine already deployed
+    just ssh $MACHINE
+
+    # Find many other operations recipes to use
+    just --list
+
 ## Colmena
 
 To deploy changes on an OS level, we use the excellent
@@ -133,16 +128,62 @@ To subsequently deploy a machine:
 
 ## Secrets
 
-Secrets are encrypted using [SOPS](https://github.com/getsops/sops) and [KMS](https://aws.amazon.com/kms/).
+Secrets are encrypted using [SOPS](https://github.com/getsops/sops) with
+[KMS](https://aws.amazon.com/kms/) and
+[AGE](https://github.com/FiloSottile/age).
 
 All secrets live in `./secrets/`
 
-You should be able to edit a KMS or sops age secret using:
+KMS encryption is generally used for secrets intended to be consumed by all
+machines as it has the benefit over age encryption of not needing re-encryption
+every time a machine in the cluster changes. To sops encrypt a secret file
+intended for all machines with KMS:
+
+    sops --encrypt \
+      --kms "$KMS" \
+      --config /dev/null \
+      --input-type binary \
+      --output-type binary \
+      $SECRET_FILE \
+    > secrets/$SECRET_FILE.enc
+
+    rm unencrypted-secret-file
+
+For per-machine secrets, age encryption is preferred, where each secret is
+typically encrypted only for the target machine and an admin such as an SRE.
+
+Age public and private keys will be automatically derived for each deployed
+machine from the machine's `/etc/ssh/ssh_host_ed25519_key` file.  Therefore, no
+manual generation of private age keys for machines is required and the public
+age key for each machine is printed during each `colmena` deployment, example:
+
+    > just apply machine
+    ...
+    machine | sops-install-secrets: Imported /etc/ssh/ssh_host_ed25519_key as age key with fingerprint $AGE_PUBLIC_KEY
+    ...
+
+These machine public age keys become the basis for access assignment of
+per-machine secrets declared in [.sops.yaml](.sops.yaml)
+
+A machine's age public key can also be generated on demand:
+
+    just ssh machine -- "'ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub'"
+
+A KMS or age sops secret file can generally be edited using:
 
     sops ./secrets/github-token.enc
 
-Or simply decrypt a KMS or sops age secret with:
+Or simply decrypt a KMS or age sops secret with:
 
     sops -d ./secrets/github-token.enc
 
-See also the `just sops-<encrypt|decrypt>-binary` and similar recipes for encrypting or decrypting age binary blobs.
+In cases where the decrypted data is in json format, sops args of `--input-type
+binary --output-type binary` may also be required to avoid decryption embedded
+in json.
+
+See also related sops encryption and decryption recipes:
+
+    just sops-decrypt-binary "$FILE"                          # Decrypt a file to stdout using .sops.yaml rules
+    just sops-decrypt-binary-in-place "$FILE"                 # Decrypt a file in place using .sops.yaml rules
+    just sops-encrypt-binary "$FILE"                          # Encrypt a file in place using .sops.yaml rules
+    just sops-rotate-binary "$FILE"                           # Rotate sops encryption using .sops.yaml rules