feat: iterate on ami, packer, ansible, ssh to investigate (#1928)

* feat: iterate on ami, packer, ansible, ssh to investigate

* chore: remove stray char that I had put in to force failure for testing

* fix: cleaning up function of running phases etc

* docs: update docs

Author: samrose

.gitignore

@@ -31,3 +31,5 @@ common-nix.vars.pkr.hcl
 # pre-commit config is managed in nix
 .pre-commit-config.yaml
 nixos.qcow2
+nix/packages/pg-ami-builder/pg-ami-builder
+nix/packages/pg-ami-builder/vendor/

docs/ami-local-development.md

@@ -0,0 +1,284 @@
+# Local AMI Development with pg-ami-builder
+
+This guide explains how to use `pg-ami-builder` for local AMI development and iteration.
+
+ Summary
+
+  | Aspect             | CI/CD Workflows                 | pg-ami-builder                          |
+  |--------------------|---------------------------------|-----------------------------------------|
+  | AMI Creation       | Packer auto-creates only        | Packer auto-creates + manual create-ami |
+  | Workflow           | Linear, automated               | Iterative, debuggable                   |
+  | State              | Stateless, ephemeral            | Stateful, persistent                    |
+  | Error Handling     | Terminate and restart           | Preserve, debug, fix, continue          |
+  | Use Case           | Production releases, CI testing | Local development, iteration            |
+  | Instance Lifecycle | Always terminated               | Preserved for debugging                 |
+
+
+## Prerequisites
+
+### Required Tools
+
+- AWS CLI v2
+- aws-vault (for credential management)
+- SSM Session Manager plugin
+- Git
+- Nix
+
+### Installing SSM Session Manager Plugin
+
+**macOS:**
+```bash
+brew install --cask session-manager-plugin
+```
+
+**Linux:**
+See [AWS documentation](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html)
+
+### AWS Permissions
+
+Your AWS user/role needs these permissions:
+- EC2: RunInstances, TerminateInstances, DescribeInstances, CreateTags
+- EC2: CreateSecurityGroup, DeleteSecurityGroup, AuthorizeSecurityGroupIngress
+- SSM: StartSession, DescribeSessions
+- EC2: CreateImage, DescribeImages (if using --create-ami)
+
+## Quick Start
+
+### Building Phase 1
+
+```bash
+# Run phase 1 build (launches instance and runs packer build)
+aws-vault exec <profile> -- nix run .#pg-ami-builder -- build phase1 --postgres-version 15
+
+# If packer build fails, instance stays alive for debugging
+# SSH to investigate
+aws-vault exec <profile> -- nix run .#pg-ami-builder -- ssh
+
+# Make local changes and re-run with file sync
+vim ansible/playbook.yml
+aws-vault exec <profile> -- nix run .#pg-ami-builder -- ansible-rerun phase1 --sync-files
+
+# Cleanup when done
+aws-vault exec <profile> -- nix run .#pg-ami-builder -- cleanup
+```
+
+### Building Phase 2
+
+```bash
+# Run phase 2 with existing stage-1 AMI
+aws-vault exec <profile> -- nix run .#pg-ami-builder -- build phase2 \
+  --source-ami ami-stage1-xyz \
+  --postgres-version 15
+```
+
+## Commands
+
+### build phase1
+
+Launch EC2 instance and run phase 1 ansible playbook.
+
+```bash
+nix run .#pg-ami-builder -- build phase1 --postgres-version 15 [flags]
+```
+
+**Flags:**
+- `--postgres-version` (required) - PostgreSQL major version (15, 16, 17)
+- `--region` - AWS region (default: us-east-1)
+- `--create-ami` - Create AMI on success (default: false)
+- `--ansible-args` - Additional ansible arguments
+- `--instance-type` - EC2 instance type (default: c6g.4xlarge)
+- `--state-file` - Custom state file path
+
+### build phase2
+
+Launch EC2 instance from stage-1 AMI and run phase 2 ansible playbook.
+
+```bash
+nix run .#pg-ami-builder -- build phase2 --source-ami ami-xyz --postgres-version 15 [flags]
+```
+
+**Flags:**
+- `--source-ami` (required) - Stage-1 AMI ID
+- `--postgres-version` (required) - PostgreSQL major version
+- `--git-sha` - Git SHA for nix packages (default: current HEAD)
+- Plus all flags from phase1
+
+### ansible-rerun
+
+Re-run ansible playbook on existing instance. Optionally sync local file changes first.
+
+```bash
+nix run .#pg-ami-builder -- ansible-rerun phase1 [flags]
+```
+
+**Flags:**
+- `--instance-id` - Target specific instance (default: from state file)
+- `--sync-files` - Sync local ansible/, scripts/, and migrations/ files before running (default: false)
+- `--ansible-args` - Additional ansible arguments
+- `--skip-tags` - Ansible tags to skip
+- `--region` - AWS region (default: us-east-1)
+
+**Examples:**
+
+```bash
+# Re-run without syncing files (use existing files on instance)
+nix run .#pg-ami-builder -- ansible-rerun phase1
+
+# Re-run with local file changes
+nix run .#pg-ami-builder -- ansible-rerun phase1 --sync-files
+
+# Re-run with skip tags
+nix run .#pg-ami-builder -- ansible-rerun phase1 --skip-tags migrations
+```
+
+### ssh
+
+Connect to instance via AWS SSM Session Manager (default) or EC2 Instance Connect.
+
+```bash
+nix run .#pg-ami-builder -- ssh [flags]
+```
+
+**Flags:**
+- `--instance-id` - Target specific instance for SSM (default: from state file)
+- `--region` - AWS region for SSM (default: us-east-1)
+- `--aws-ec2-connect-cmd` - Full AWS EC2 Instance Connect command string
+
+**Examples:**
+
+```bash
+# Connect via SSM (default)
+nix run .#pg-ami-builder -- ssh
+
+# Connect via EC2 Instance Connect
+nix run .#pg-ami-builder -- ssh \
+  --aws-ec2-connect-cmd "aws ec2-instance-connect ssh --instance-id i-024bba2db43e4b41f --region us-east-1"
+```
+
+### cleanup
+
+Terminate instance and remove associated resources.
+
+```bash
+nix run .#pg-ami-builder -- cleanup [flags]
+```
+
+**Flags:**
+- `--instance-id` - Target specific instance (default: from state file)
+- `--force` - Skip confirmation prompt
+
+## Workflows
+
+### Workflow 1: Develop and test phase 1 changes
+
+```bash
+# Run phase 1 build (launches instance and runs packer build)
+aws-vault exec <profile> -- nix run .#pg-ami-builder -- build phase1 --postgres-version 15
+
+# If packer fails, instance stays up for debugging
+# SSH to investigate
+aws-vault exec <profile> -- nix run .#pg-ami-builder -- ssh
+
+# Make local changes to ansible files
+vim ansible/playbook.yml
+
+# Re-run with your local changes
+aws-vault exec <profile> -- nix run .#pg-ami-builder -- ansible-rerun phase1 --sync-files
+
+# Repeat until working, then create AMI
+aws-vault exec <profile> -- nix run .#pg-ami-builder -- build phase1 --postgres-version 15 --create-ami
+
+# Cleanup
+aws-vault exec <profile> -- nix run .#pg-ami-builder -- cleanup
+```
+
+### Workflow 2: Parallel builds for multiple postgres versions
+
+```bash
+# Build PG 15
+aws-vault exec <profile> -- nix run .#pg-ami-builder -- build phase1 \
+  --postgres-version 15 \
+  --state-file ~/.pg-ami-build/pg15.json
+
+# Build PG 16 in parallel
+aws-vault exec <profile> -- nix run .#pg-ami-builder -- build phase1 \
+  --postgres-version 16 \
+  --state-file ~/.pg-ami-build/pg16.json
+
+# SSH into PG 15 instance
+aws-vault exec <profile> -- nix run .#pg-ami-builder -- ssh \
+  --state-file ~/.pg-ami-build/pg15.json
+```
+
+## Troubleshooting
+
+### SSM Connection Fails
+
+1. Check SSM agent status on the instance
+2. Verify instance profile has SSM permissions
+3. Ensure session-manager-plugin is installed
+
+### Ansible Fails
+
+The instance is kept running on failure. Check logs:
+
+```bash
+# SSH into instance
+nix run .#pg-ami-builder -- ssh
+
+# Check ansible logs
+sudo journalctl -u ansible-provisioner
+```
+
+### State File Issues
+
+If state file references non-existent instance:
+
+```bash
+# Override with specific instance
+nix run .#pg-ami-builder -- ssh --instance-id i-xxxxx
+
+# Or clear state and start fresh
+rm ~/.pg-ami-build/state.json
+```
+
+## Advanced Usage
+
+### Custom State Files for Parallel Builds
+
+Use `--state-file` to manage multiple builds:
+
+```bash
+nix run .#pg-ami-builder -- build phase1 \
+  --postgres-version 15 \
+  --state-file ~/.pg-ami-build/custom.json
+```
+
+### Additional Ansible Arguments
+
+Pass custom arguments to ansible:
+
+```bash
+nix run .#pg-ami-builder -- build phase1 \
+  --postgres-version 15 \
+  --ansible-args="--skip-tags=migrations"
+```
+
+## State File
+
+Location: `~/.pg-ami-build/state.json`
+
+The state file tracks the current build instance, allowing subsequent commands to operate on the same instance without specifying `--instance-id`.
+
+Example state:
+```json
+{
+  "instance_id": "i-1234567890abcdef0",
+  "phase": "phase1",
+  "execution_id": "1731672000-15",
+  "region": "us-east-1",
+  "postgres_version": "15",
+  "timestamp": "2025-11-15T10:30:00Z",
+  "git_sha": "abc123def456"
+}
+```

nix/apps.nix

@@ -24,6 +24,7 @@
         update-readme = mkApp "update-readme" "update-readme";
         show-commands = mkApp "show-commands" "show-commands";
         build-test-ami = mkApp "build-test-ami" "build-test-ami";
+        pg-ami-builder = mkApp "pg-ami-builder" "pg-ami-builder";
         run-testinfra = mkApp "run-testinfra" "run-testinfra";
         cleanup-ami = mkApp "cleanup-ami" "cleanup-ami";
         trigger-nix-build = mkApp "trigger-nix-build" "trigger-nix-build";

nix/devShells.nix

@@ -47,6 +47,16 @@
               ansible-lint
               self'.packages.packer
 
+              # Go development tools
+              go
+              gopls
+              gotools
+              go-tools
+              delve
+
+              # AWS tools
+              awscli2
+
               self'.packages.start-server
               self'.packages.start-client
               self'.packages.start-replica
@@ -55,6 +65,7 @@
               self'.packages.build-test-ami
               self'.packages.run-testinfra
               self'.packages.cleanup-ami
+              self'.packages.pg-ami-builder
               dbmate
               nushell
               pythonEnv

nix/fmt.nix

@@ -11,6 +11,10 @@
           package = pkgs.nixfmt-rfc-style;
         };
         ruff-format.enable = true;
+        gofumpt = {
+          enable = true;
+          package = pkgs.gofumpt;
+        };
       };
     };
 }

nix/packages/default.nix

@@ -28,8 +28,12 @@
     in
     {
       packages = (
-        {
+        rec {
           build-test-ami = pkgs.callPackage ./build-test-ami.nix { };
+          packer = pkgs.callPackage ./packer.nix { inherit inputs; };
+          pg-ami-builder = inputs'.nixpkgs-go124.legacyPackages.callPackage ./pg-ami-builder.nix {
+            inherit packer;
+          };
           cleanup-ami = pkgs.callPackage ./cleanup-ami.nix { };
           dbmate-tool = pkgs.callPackage ./dbmate-tool.nix { inherit (self.supabase) defaults; };
           docs = pkgs.callPackage ./docs.nix { };
@@ -39,7 +43,6 @@
           mecab-naist-jdic = pkgs.callPackage ./mecab-naist-jdic.nix { };
           migrate-tool = pkgs.callPackage ./migrate-tool.nix { psql_15 = self'.packages."psql_15/bin"; };
           overlayfs-on-package = pkgs.callPackage ./overlayfs-on-package.nix { };
-          packer = pkgs.callPackage ./packer.nix { inherit inputs; };
           pg-backrest = inputs.nixpkgs-pgbackrest.legacyPackages.${pkgs.system}.pgbackrest;
           pg-restore = pkgs.callPackage ./pg-restore.nix { psql_15 = self'.packages."psql_15/bin"; };
           pg_prove = pkgs.perlPackages.TAPParserSourceHandlerpgTAP;