feat(ppg-packer): extend AMI factory to Rocky Linux 8/9/10

- Generalize the refresh template (oracle-linux.pkr.hcl -> refresh.pkr.hcl)
  with an os dimension (oraclelinux|rocky) threaded through tags, AMI names,
  the SSH login user, and the lineage source filter
- Add a seed path (-var seed=true) sourcing the official Rocky community
  AMIs (include_deprecated covers the frozen Rocky 8) to root each lineage
  once per combo
- Make validate.sh and the smoke template distro-aware (/etc/rocky-release,
  powertools/crb vs ol*_codeready_builder, rocky vs ec2-user login)
- Rename the workflow to ppg-ami-factory.yml, add an os matrix dimension and
  a pull_request trigger for the no-AWS check job; default combos stay
  OL-only until the Rocky seeds are promoted
- Thread os through the justfile (validate/build/bake/all/latest, os-aware
  prune sweeps) and add seed-rocky + ci-seed-rocky recipes

Author: nogueiraanderson

…hub/workflows/ppg-oracle-ami-factory.yml .github/workflows/ppg-ami-factory.yml.github/workflows/ppg-oracle-ami-factory.yml renamed to .github/workflows/ppg-ami-factory.yml .github/workflows/ppg-oracle-ami-factory.yml renamed to .github/workflows/ppg-ami-factory.yml

@@ -1,22 +1,24 @@
-# Oracle Linux package-test AMI factory.
-# Bakes OL8/OL9 (and later OL10) x86_64+arm64 AMIs with Packer over AWS Session
+# EL package-test AMI factory: Oracle Linux + Rocky Linux.
+# Bakes OL and Rocky 8/9/10 x86_64+arm64 AMIs with Packer over AWS Session
 # Manager (no inbound SSH), authenticated via GitHub OIDC (no static keys), and
 # promotes each via a fresh-boot smoke test. Consumer: pg.cd molecule jobs select
 # the newest role=ppg-package-test AMI by tag (vars/moleculeEnvPPG.groovy).
+# Rocky lineage roots are seeded once per combo via dispatch (just ci-seed-rocky).
 #
-# Triggers: recipe change (paths), weekly security rebake (cron), manual.
+# Triggers: PR validate-only (pull_request -> check job, no AWS), recipe change
+# (paths), weekly security rebake (cron), manual.
 # Actions are pinned to commit SHAs; the trailing "# vX.Y.Z" records the tag.
 
-name: ppg-oracle-ami-factory
-run-name: ppg OL AMI factory (${{ inputs.env || 'prod' }}, ${{ github.event_name }})
+name: ppg-ami-factory
+run-name: ppg AMI factory (${{ inputs.env || 'prod' }}, ${{ github.event_name }})
 
 on:
   workflow_dispatch:
     inputs:
       combos:
-        description: 'JSON matrix include list, e.g. [{"os_major":"9","arch":"x86_64"}]'
+        description: 'JSON matrix include list, e.g. [{"os":"rocky","os_major":"9","arch":"x86_64","seed":true}]; os defaults to oraclelinux, seed to false'
         required: false
-        default: '[{"os_major":"9","arch":"x86_64"}]'
+        default: '[{"os":"oraclelinux","os_major":"9","arch":"x86_64"}]'
       env:
         description: 'prod (real factory tags) or test (isolated tags, never consumed by pg.cd)'
         required: false
@@ -27,9 +29,13 @@ on:
     branches: [master]
     paths:
       - 'ppg/packer/**'
-      - '.github/workflows/ppg-oracle-ami-factory.yml'
+      - '.github/workflows/ppg-ami-factory.yml'
+  pull_request:
+    paths:
+      - 'ppg/packer/**'
+      - '.github/workflows/ppg-ami-factory.yml'
   schedule:
-    - cron: '0 6 * * 1' # weekly Mon 06:00 UTC - absorb Oracle security errata
+    - cron: '0 6 * * 1' # weekly Mon 06:00 UTC - absorb OL + Rocky security errata
 
 permissions:
   contents: read # workflow-level default: read-only
@@ -67,35 +73,48 @@ jobs:
           packer fmt -check -diff .
           packer fmt -check -diff smoke
           packer init .
-          for m in 8 9 10; do
-            for a in x86_64 arm64; do
-              echo "validate OL$m $a"
-              packer validate -var "os_major=$m" -var "arch=$a" .
+          for os_name in oraclelinux rocky; do
+            for major in 8 9 10; do
+              for arch in x86_64 arm64; do
+                echo "validate $os_name $major $arch"
+                packer validate -var "os=$os_name" -var "os_major=$major" -var "arch=$arch" .
+              done
             done
           done
-          ( cd smoke && packer init . && packer validate \
+          for major in 8 9 10; do
+            for arch in x86_64 arm64; do
+              echo "validate rocky seed $major $arch"
+              packer validate -var os=rocky -var seed=true -var "os_major=$major" -var "arch=$arch" .
+            done
+          done
+          ( cd smoke && packer init . \
+            && packer validate \
+              -var candidate_ami=ami-00000000000000000 \
+              -var os=oraclelinux -var os_major=9 -var arch=x86_64 . \
+            && packer validate \
               -var candidate_ami=ami-00000000000000000 \
-              -var os_major=9 -var arch=x86_64 . )
+              -var os=rocky -var os_major=8 -var arch=x86_64 . )
           # drift guard (mirrors `just check`): justfile root_gib must equal var.volume_size,
           # else a reimaged base would be the wrong root size for the refresh to launch.
           rg=$(awk -F'"' '/^root_gib[[:space:]]*:=/{print $2; exit}' justfile)
-          vs=$(awk '/variable "volume_size"/{f=1} f&&/default/{gsub(/[^0-9]/,"");print;exit}' oracle-linux.pkr.hcl)
+          vs=$(awk '/variable "volume_size"/{f=1} f&&/default/{gsub(/[^0-9]/,"");print;exit}' refresh.pkr.hcl)
           [[ -n "$rg" && "$rg" == "$vs" ]] || { echo "DRIFT: justfile root_gib=$rg != var.volume_size=$vs"; exit 1; }
           echo "check OK"
 
   bake:
-    name: OL${{ matrix.os_major }} ${{ matrix.arch }}
+    name: ${{ matrix.os || 'oraclelinux' }} ${{ matrix.os_major }} ${{ matrix.arch }}
     needs: check # no EC2 launches until templates validate
+    if: github.event_name != 'pull_request' # PRs run the no-AWS check only
     runs-on: ubuntu-latest
     timeout-minutes: 60
     permissions:
       contents: read
       id-token: write # ONLY this job mints the OIDC token
     strategy:
       fail-fast: false
-      max-parallel: 2
+      max-parallel: 12
       matrix:
-        include: ${{ fromJSON(github.event.inputs.combos || '[{"os_major":"8","arch":"x86_64"},{"os_major":"8","arch":"arm64"},{"os_major":"9","arch":"x86_64"},{"os_major":"9","arch":"arm64"},{"os_major":"10","arch":"x86_64"},{"os_major":"10","arch":"arm64"}]') }}
+        include: ${{ fromJSON(github.event.inputs.combos || '[{"os":"oraclelinux","os_major":"8","arch":"x86_64"},{"os":"oraclelinux","os_major":"8","arch":"arm64"},{"os":"oraclelinux","os_major":"9","arch":"x86_64"},{"os":"oraclelinux","os_major":"9","arch":"arm64"},{"os":"oraclelinux","os_major":"10","arch":"x86_64"},{"os":"oraclelinux","os_major":"10","arch":"arm64"}]') }}
     steps:
       - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
 
@@ -129,18 +148,22 @@ jobs:
         working-directory: ppg/packer
         run: packer init .
 
-      - name: Build candidate AMI (OL${{ matrix.os_major }} ${{ matrix.arch }}, env=${{ env.FACTORY_ENV }})
+      - name: Build candidate AMI (${{ matrix.os || 'oraclelinux' }} ${{ matrix.os_major }} ${{ matrix.arch }}, env=${{ env.FACTORY_ENV }})
         id: build
         working-directory: ppg/packer
         # Matrix values via env, not inline ${{ }}, so they reach packer as data
         # (quoted shell vars) and cannot break out of the run shell.
         env:
+          OS: ${{ matrix.os || 'oraclelinux' }}
+          SEED: ${{ matrix.seed || false }}
           OS_MAJOR: ${{ matrix.os_major }}
           ARCH: ${{ matrix.arch }}
         run: |
           set -euo pipefail
           rm -f manifest.json
           packer build -color=false \
+            -var "os=${OS}" \
+            -var "seed=${SEED}" \
             -var "os_major=${OS_MAJOR}" \
             -var "arch=${ARCH}" \
             -var "region=${AWS_REGION}" \
@@ -149,22 +172,23 @@ jobs:
           echo "ami=$AMI" >> "$GITHUB_OUTPUT"
           echo "built candidate: $AMI"
           {
-            echo "### OL${OS_MAJOR} ${ARCH}"
+            echo "### ${OS} ${OS_MAJOR} ${ARCH}"
             echo "- built: $AMI"
           } >> "$GITHUB_STEP_SUMMARY"
 
       - name: Smoke test (boot + install)
         working-directory: ppg/packer
         env:
           AMI: ${{ steps.build.outputs.ami }}
+          OS: ${{ matrix.os || 'oraclelinux' }}
           OS_MAJOR: ${{ matrix.os_major }}
           ARCH: ${{ matrix.arch }}
         run: |
           set -euo pipefail
           echo "smoke candidate: $AMI"
           # Deregister ONLY on a real boot/install failure (never on a later promote-tag failure).
           ( cd smoke && packer init . && packer build -color=false \
-              -var "candidate_ami=$AMI" -var "os_major=${OS_MAJOR}" \
+              -var "candidate_ami=$AMI" -var "os=${OS}" -var "os_major=${OS_MAJOR}" \
               -var "arch=${ARCH}" -var "region=${AWS_REGION}" . ) \
           || { echo "smoke (boot+install) failed; deregistering $AMI + its snapshots"; aws ec2 deregister-image --delete-associated-snapshots --region "$AWS_REGION" --image-id "$AMI"; echo "- smoke: FAIL (deregistered $AMI)" >> "$GITHUB_STEP_SUMMARY"; exit 1; }
           echo "- smoke: pass" >> "$GITHUB_STEP_SUMMARY"
@@ -202,4 +226,4 @@ jobs:
           webhook: ${{ env.SLACK_WEBHOOK }}
           webhook-type: incoming-webhook
           payload: |
-            text: "PPG-OL-AMI factory FAILED - ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+            text: "PPG AMI factory FAILED - ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"

ppg/packer/README.md

@@ -1,12 +1,14 @@
-# PPG Oracle Linux AMI factory
+# PPG EL AMI factory (Oracle Linux + Rocky Linux)
 
-Builds the Oracle Linux package-test target AMIs that PG release testing runs
-against, so we stop hand-maintaining them (launch base, `dnf update`, snapshot).
+Builds the Oracle Linux and Rocky Linux package-test target AMIs that PG release
+testing runs against, so we stop hand-maintaining them (launch base,
+`dnf update`, snapshot).
 
 ## Why this exists
 
-There is no off-the-shelf AWS image that is all of: genuine Oracle Linux,
-OL8/OL9/OL10, x86_64 **and** arm64, free of software fees, and maintained.
+**Oracle Linux**: there is no off-the-shelf AWS image that is all of: genuine
+Oracle Linux, OL8/OL9/OL10, x86_64 **and** arm64, free of software fees, and
+maintained.
 
 - Oracle stopped publishing official AWS AMIs. The newest public Oracle images
   (owner `131827586825`) are OL8.9 / OL9.3 from Feb 2024 and were all deprecated
@@ -17,6 +19,16 @@ OL8/OL9/OL10, x86_64 **and** arm64, free of software fees, and maintained.
 - AlmaLinux is free + current + both-arch, but it is an EL clone, fine as
   supplemental smoke coverage, not a replacement for the Oracle Linux gate.
 
+**Rocky Linux**: official community AMIs exist (owner `792107900819`, no fees,
+no product codes), but upstream stopped publishing Rocky 8 images at
+`8.10-20240528` and AWS auto-deprecated them in 2026, so the previously
+hand-pinned Rocky 8 AMI aged two years of errata into every molecule run. The
+factory seeds each Rocky lineage once from the official AMI, then refreshes it
+weekly like OL, which keeps Rocky 8 current even though upstream no longer
+ships it. The official 10 GiB roots restore fine onto the factory's
+`var.volume_size` (EBS only refuses to shrink), so Rocky needs no
+bootstrap/re-image machinery.
+
 So we bake our own, on a schedule, in the CI build account (eu-central-1). The
 same Packer templates + scripts run whether driven locally (`justfile`) or by the
 GitHub Actions workflow. Builds connect over AWS Session Manager (no inbound SSH)
@@ -26,7 +38,8 @@ and authenticate via GitHub OIDC (no static keys).
 
 | Path | Covers | Mechanism |
 |------|--------|-----------|
-| **Refresh** (`oracle-linux.pkr.hcl`) | OL8, OL9, OL10 (x86_64 + arm64) | `amazon-ebs`: launch the latest self-owned base of the same major+arch, `dnf update`, validate (fail-closed), snapshot. Each build's output is the next build's source. |
+| **Refresh** (`refresh.pkr.hcl`) | OL + Rocky 8/9/10 (x86_64 + arm64) | `amazon-ebs`: launch the latest self-owned base of the same os+major+arch, `dnf update`, validate (fail-closed), snapshot. Each build's output is the next build's source. |
+| **Seed** (`refresh.pkr.hcl -var seed=true`) | Rocky lineage roots, one-time per combo | Same bake, but sourced from the official Rocky community AMI (`include_deprecated` covers the frozen Rocky 8). Promoted seeds become the refresh lineage. |
 | **Bootstrap** (`scripts/bootstrap-ol10.sh`) | OL10 lineage root, one-time per arch | Import Oracle's official OL10 cloud image (arm64 `*-kvm-cloud-*.qcow2`, x86_64 `*-aws-*.vmdk`) via `aws ec2 import-snapshot`, register the raw base, then `packer build bootstrap/finalize-ol10.pkr.hcl` (adds `ec2-user` + `amazon-ssm-agent` + `dnf update`). Needed because Oracle ships no OL10 AWS AMI. Promotes to a `prebase` role: Oracle's full-size root cannot launch on `var.volume_size`, so `reimage-ol10` shrinks it before it becomes the consumed base. Idempotent. |
 | **Re-image** (`scripts/reimage-ol10.sh`) | OL10 root shrink, one-time per arch | Copy the current base's root onto a fresh `var.volume_size` GiB volume on a builder, snapshot + register, then `reimage-ol10-verify` boot-tests two sizes + smoke + size-gate + promote. Needed because EBS cannot restore a volume below its source snapshot and XFS cannot shrink in place. Full internals (`dd` /boot, LVM-to-plain, the gates): [docs/reimage.md](docs/reimage.md). |
 
@@ -35,24 +48,30 @@ and authenticate via GitHub OIDC (no static keys).
 ```bash
 export AWS_PROFILE=percona-dev-admin
 just check                              # fmt-check + validate every combo (no AWS)
-just bake 9 x86_64                      # build + smoke + promote one combo
+just bake 9 x86_64                      # build + smoke + promote one OL combo
+just bake 9 x86_64 prod rocky           # same for a seeded Rocky combo
+just seed-rocky 9 x86_64 prod           # one-time Rocky lineage root (bake + smoke + promote)
 just all                                # build + promote every OL major x arch
+just all prod rocky                     # same for Rocky (after seeding)
 just bootstrap-ol10-prep                # one-time: import bucket + vmimport role + boot-test SG/key
 just bootstrap-ol10 x86_64              # one-time OL10 lineage root for an arch
 just bootstrap-ol10-verify <ami> x86_64 # boot-validate a candidate then promote it (prebase)
 just reimage-ol10 x86_64                # shrink the OL10 base root to var.volume_size (lineage shrink)
 just reimage-ol10-verify <ami> x86_64   # boot-test two sizes + smoke + size-gate + promote
 ```
 
-Each build registers `OL<major>-<arch>-<UTCstamp>` tagged `os=oraclelinux`,
+Each build registers `OL<major>-<arch>-<UTCstamp>` / `Rocky<major>-<arch>-<UTCstamp>`
+tagged `os=oraclelinux|rocky`,
 `os_major`, `arch`, `source=factory` with **`role=ppg-candidate`**, then the
 native-Packer smoke (`smoke/smoke.pkr.hcl`) fresh-boot-tests it. The smoke
 template ONLY validates; **promotion to `role=ppg-package-test` is a separate,
 retried step** in the `justfile`/workflow that runs after a passing smoke, so a
 transient tag-API failure can never deregister a smoke-passed AMI. A real
 boot/install failure deregisters the candidate + its snapshots.
 Consumers and the next build's source filter select only `role=ppg-package-test`,
-so a non-booting image is never selectable and AMI IDs are never pinned. "Latest"
+so a non-booting image is never selectable and AMI IDs are never pinned.
+Login user: `ec2-user` on OL, `rocky` on Rocky (the images keep their distro's
+cloud-init default user, matching what the molecule scenarios already use). "Latest"
 is the newest `role=ppg-package-test` AMI by `CreationDate` (no SSM parameter).
 
 ## Housekeeping (all via `just`, fail-safe)
@@ -74,7 +93,8 @@ just ci-validate         # trigger the GHA workflow matrix (env=test)
 ## Consumer lookup (replaces hardcoded IDs)
 
 `vars/moleculeEnvPPG.groovy` resolves each Oracle target at pipeline time
-(fail-closed: the job aborts if no base AMI is found):
+(fail-closed: the job aborts if no base AMI is found); Rocky targets switch to
+the same lookup with `Name=tag:os,Values=rocky` once the seeds are promoted:
 
 ```bash
 ami_ol9_x86_64=$(aws ec2 describe-images --region eu-central-1 --owners self \
@@ -89,11 +109,12 @@ ami_ol9_x86_64=$(aws ec2 describe-images --region eu-central-1 --owners self \
 `scripts/validate.sh` runs as the last provisioner; any failure aborts the build
 so a broken/mislabeled image is never registered:
 
-1. `/etc/oracle-release` major matches the target (fidelity).
+1. `/etc/oracle-release` / `/etc/rocky-release` major matches the target (fidelity).
 2. `uname -m` matches the target arch.
-3. `ol<major>_codeready_builder` repo is defined (PG CRB deps resolvable).
+3. The CRB repo is defined: `ol<major>_codeready_builder` (OL), `powertools`
+   (Rocky 8) or `crb` (Rocky 9/10). PG -devel deps resolve from it.
 4. `cloud-init` present.
-5. SELinux `enforcing` in `/etc/selinux/config` (fail-closed; stock OL ships enforcing).
+5. SELinux `enforcing` in `/etc/selinux/config` (fail-closed; stock OL and Rocky ship enforcing).
 6. `dnf makecache` succeeds (repos healthy).
 
 De-instancing is native: Packer's `ssh_clear_authorized_keys` strips the temp key,