Skip to content

Commit cfbb171

Browse files
samroseolirice
samrose
and
olirice
authored
Sam/15 16 nix package, build and test of 15.8 and 16.3 (#1266)
* feat: build and flake check of pg 16.3 with exts/wrappers * pg_partman test 15/16 compat * merge sql interface test * tests: build test and cache both versions * chore: run checks individually * pg 15 and 16 packer/ansible/ghactions (#1268) * fix: reformat ec2 cleanup commands (#1267) Co-authored-by: Sam Rose <[email protected]> * feat: build and flake check of pg 16.3 with exts/wrappers * pg_partman test 15/16 compat * merge sql interface test * tests: build test and cache both versions * chore: run checks individually * feat: realease 15 and 16 to staging * chore: update versions * chore: make yq available * chore: run yq from nix * chore: more setup for staging AMI * fix: yq usage * chore: shell vars * fix: When --init none is used, only users who can elevate to sudo privileges can run Nix * fix: no -i * fix: quote correction * fix: newline extra quotes * fix: no need for pg major version on packer * fix: postgresql_major * fix: ql * fix: no ansible args in stage to invocation * fix: unique val * fix: adjustments to build scripts * chore: env var handling * fix: bump to build * chore: set up more required vars * chore: bump var * feat: pg 16 debug symbols * feat: matrix pg versions build on testinfra * feat: matrix on Test Database * chore: running nix in the right context * feat: just use existing Dockerfile + pg version * chore: refer to var * fix: read name without including quotes * chore: try format function * fix: strip quotes from version number * chore: env var * fix: pg client * fix * fix: try to use psql from our own corresponding pkg * fix: try psql from ppa * fix: dbmate per pg version * build dbmate and then install client * fix: account for architecture * chore: limit changes detection migrations/schema.sql * missing docker compose call * ore: drop tests while investigating * test: try on pg15 only * chore: schema needs update * chore: now run on all versions in matrix * test: trying a version of schema per major pg version as there are type diffs --------- Co-authored-by: Sam Rose <[email protected]> Co-authored-by: Oliver Rice <[email protected]> * chore: rm dead code * chore: resolve versions from ansible/vars.yml instead of packer file create a matrix build for 15 16 (or how ever many versions there are) * fix: don not lose if statements used for hotfix when needed * chore: cleanup --------- Co-authored-by: Sam Rose <[email protected]> Co-authored-by: Oliver Rice <[email protected]>
1 parent 578e275 commit cfbb171

39 files changed

+2957
-148
lines changed

‎.github/workflows/ami-release-nix.yml‎

Lines changed: 47 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,31 @@ on:
88
paths:
99
- '.github/workflows/ami-release-nix.yml'
1010
- 'common-nix.vars.pkr.hcl'
11+
- 'ansible/vars.yml'
1112
workflow_dispatch:
1213

1314
jobs:
15+
prepare:
16+
runs-on: ubuntu-latest
17+
outputs:
18+
postgres_versions: ${{ steps.set-versions.outputs.postgres_versions }}
19+
steps:
20+
- name: Checkout Repo
21+
uses: actions/checkout@v3
22+
23+
- uses: DeterminateSystems/nix-installer-action@main
24+
25+
- name: Set PostgreSQL versions
26+
id: set-versions
27+
run: |
28+
VERSIONS=$(nix run nixpkgs#yq -- '.postgres_major[]' ansible/vars.yml | nix run nixpkgs#jq -- -R -s -c 'split("\n")[:-1]')
29+
echo "postgres_versions=$VERSIONS" >> $GITHUB_OUTPUT
30+
1431
build:
32+
needs: prepare
1533
strategy:
1634
matrix:
35+
postgres_version: ${{ fromJson(needs.prepare.outputs.postgres_versions) }}
1736
include:
1837
- runner: arm-runner
1938
arch: arm64
@@ -31,42 +50,55 @@ jobs:
3150
- name: Checkout Repo
3251
uses: actions/checkout@v3
3352

53+
- uses: DeterminateSystems/nix-installer-action@main
54+
3455
- name: Run checks if triggered manually
3556
if: ${{ github.event_name == 'workflow_dispatch' }}
36-
# Update `ci.yaml` too if changing constraints.
3757
run: |
38-
SUFFIX=$(sed -E 's/postgres-version = "[0-9\.]+(.*)"/\1/g' common-nix.vars.pkr.hcl)
58+
SUFFIX=$(sudo nix run nixpkgs#yq -- '.postgres_release["postgres${{ matrix.postgres_version }}"]' ansible/vars.yml | sed -E 's/[0-9\.]+(.*)$/\1/')
3959
if [[ -z $SUFFIX ]] ; then
4060
echo "Version must include non-numeric characters if built manually."
4161
exit 1
4262
fi
4363
44-
# extensions are build in nix prior to this step
45-
# so we can just use the binaries from the nix store
46-
# for postgres, extensions and wrappers
64+
- name: Set PostgreSQL version environment variable
65+
run: echo "POSTGRES_MAJOR_VERSION=${{ matrix.postgres_version }}" >> $GITHUB_ENV
66+
67+
- name: Generate common-nix.vars.pkr.hcl
68+
run: |
69+
PG_VERSION=$(sudo nix run nixpkgs#yq -- '.postgres_release["postgres'${{ matrix.postgres_version }}'"]' ansible/vars.yml)
70+
PG_VERSION=$(echo $PG_VERSION | tr -d '"') # Remove any surrounding quotes
71+
echo 'postgres-version = "'$PG_VERSION'"' > common-nix.vars.pkr.hcl
72+
# Ensure there's a newline at the end of the file
73+
echo "" >> common-nix.vars.pkr.hcl
4774
4875
- name: Build AMI stage 1
76+
env:
77+
POSTGRES_MAJOR_VERSION: ${{ env.POSTGRES_MAJOR_VERSION }}
4978
run: |
5079
packer init amazon-arm64-nix.pkr.hcl
5180
GIT_SHA=${{github.sha}}
52-
packer build -var "git-head-version=${GIT_SHA}" -var "packer-execution-id=${GITHUB_RUN_ID}" -var-file="development-arm.vars.pkr.hcl" -var-file="common-nix.vars.pkr.hcl" -var "ansible_arguments=" amazon-arm64-nix.pkr.hcl
81+
packer build -var "git-head-version=${GIT_SHA}" -var "packer-execution-id=${GITHUB_RUN_ID}" -var-file="development-arm.vars.pkr.hcl" -var-file="common-nix.vars.pkr.hcl" -var "ansible_arguments=-e postgresql_major=${POSTGRES_MAJOR_VERSION}" amazon-arm64-nix.pkr.hcl
5382
5483
- name: Build AMI stage 2
84+
env:
85+
POSTGRES_MAJOR_VERSION: ${{ env.POSTGRES_MAJOR_VERSION }}
5586
run: |
5687
packer init stage2-nix-psql.pkr.hcl
5788
GIT_SHA=${{github.sha}}
58-
packer build -var "git_sha=${GIT_SHA}" -var "git-head-version=${GIT_SHA}" -var "packer-execution-id=${GITHUB_RUN_ID}" -var-file="development-arm.vars.pkr.hcl" -var-file="common-nix.vars.pkr.hcl" stage2-nix-psql.pkr.hcl
89+
POSTGRES_MAJOR_VERSION=${{ env.POSTGRES_MAJOR_VERSION }}
90+
packer build -var "git_sha=${GIT_SHA}" -var "git-head-version=${GIT_SHA}" -var "packer-execution-id=${GITHUB_RUN_ID}" -var "postgres_major_version=${POSTGRES_MAJOR_VERSION}" -var-file="development-arm.vars.pkr.hcl" -var-file="common-nix.vars.pkr.hcl" stage2-nix-psql.pkr.hcl
5991
6092
- name: Grab release version
6193
id: process_release_version
6294
run: |
63-
VERSION=$(sed -e 's/postgres-version = "\(.*\)"/\1/g' common-nix.vars.pkr.hcl)
64-
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
95+
VERSION=$(cat common-nix.vars.pkr.hcl | sed -e 's/postgres-version = "\(.*\)"/\1/g')
96+
echo "version=$VERSION" >> $GITHUB_OUTPUT
6597
6698
- name: Create nix flake revision tarball
6799
run: |
68100
GIT_SHA=${{github.sha}}
69-
MAJOR_VERSION=$(echo "${{ steps.process_release_version.outputs.version }}" | cut -d. -f1)
101+
MAJOR_VERSION=${{ env.POSTGRES_MAJOR_VERSION }}
70102
71103
mkdir -p "/tmp/pg_upgrade_bin/${MAJOR_VERSION}"
72104
echo "$GIT_SHA" >> "/tmp/pg_upgrade_bin/${MAJOR_VERSION}/nix_flake_version"
@@ -84,17 +116,13 @@ jobs:
84116
ansible-playbook -i localhost \
85117
-e "ami_release_version=${{ steps.process_release_version.outputs.version }}" \
86118
-e "internal_artifacts_bucket=${{ secrets.ARTIFACTS_BUCKET }}" \
119+
-e "postgres_major_version=${{ env.POSTGRES_MAJOR_VERSION }}" \
87120
manifest-playbook.yml
88121
89122
- name: Upload nix flake revision to s3 staging
90123
run: |
91124
aws s3 cp /tmp/pg_binaries.tar.gz s3://${{ secrets.ARTIFACTS_BUCKET }}/upgrades/postgres/supabase-postgres-${{ steps.process_release_version.outputs.version }}/20.04.tar.gz
92125
93-
#Our self hosted github runner already has permissions to publish images
94-
#but they're limited to only that;
95-
#so if we want s3 access we'll need to config credentials with the below steps
96-
# (which overwrites existing perms) after the ami build
97-
98126
- name: configure aws credentials - prod
99127
uses: aws-actions/configure-aws-credentials@v4
100128
with:
@@ -107,6 +135,7 @@ jobs:
107135
ansible-playbook -i localhost \
108136
-e "ami_release_version=${{ steps.process_release_version.outputs.version }}" \
109137
-e "internal_artifacts_bucket=${{ secrets.PROD_ARTIFACTS_BUCKET }}" \
138+
-e "postgres_major_version=${{ env.POSTGRES_MAJOR_VERSION }}" \
110139
manifest-playbook.yml
111140
112141
- name: Upload nix flake revision to s3 prod
@@ -130,12 +159,12 @@ jobs:
130159
SLACK_MESSAGE: 'Building Postgres AMI failed'
131160
SLACK_FOOTER: ''
132161

133-
- name: Cleanup resources on build cancellation
162+
- name: Cleanup resources after build
134163
if: ${{ always() }}
135164
run: |
136-
aws ec2 describe-instances --filters "Name=tag:packerExecutionId,Values=${GITHUB_RUN_ID}" --query "Reservations[].Instances[].InstanceId" --output text | xargs -n 1 -I {} aws ec2 terminate-instances --instance-ids {}
165+
aws ec2 describe-instances --filters "Name=tag:packerExecutionId,Values=${GITHUB_RUN_ID}" --query "Reservations[].Instances[].InstanceId" --output text | xargs -r aws ec2 terminate-instances --instance-ids
137166
138167
- name: Cleanup resources on build cancellation
139168
if: ${{ cancelled() }}
140169
run: |
141-
aws ec2 describe-instances --filters "Name=tag:packerExecutionId,Values=${GITHUB_RUN_ID}" --query "Reservations[].Instances[].InstanceId" --output text | xargs -n 1 -I {} aws ec2 terminate-instances --instance-ids {}
170+
aws ec2 describe-instances --filters "Name=tag:packerExecutionId,Values=${GITHUB_RUN_ID}" --query "Reservations[].Instances[].InstanceId" --output text | xargs -r aws ec2 terminate-instances --instance-ids

‎.github/workflows/publish-nix-pgupgrade-bin-flake-version.yml‎

Lines changed: 28 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,28 @@ permissions:
1111
id-token: write
1212

1313
jobs:
14+
prepare:
15+
runs-on: ubuntu-latest
16+
outputs:
17+
postgres_versions: ${{ steps.set-versions.outputs.postgres_versions }}
18+
steps:
19+
- name: Checkout Repo
20+
uses: actions/checkout@v3
21+
22+
- uses: DeterminateSystems/nix-installer-action@main
23+
24+
- name: Set PostgreSQL versions
25+
id: set-versions
26+
run: |
27+
VERSIONS=$(nix run nixpkgs#yq -- '.postgres_major[]' ansible/vars.yml | nix run nixpkgs#jq -- -R -s -c 'split("\n")[:-1]')
28+
echo "postgres_versions=$VERSIONS" >> $GITHUB_OUTPUT
29+
1430
publish-staging:
31+
needs: prepare
1532
runs-on: ubuntu-latest
33+
strategy:
34+
matrix:
35+
postgres_version: ${{ fromJson(needs.prepare.outputs.postgres_versions) }}
1636

1737
steps:
1838
- name: Checkout Repo
@@ -21,7 +41,8 @@ jobs:
2141
- name: Grab release version
2242
id: process_release_version
2343
run: |
24-
VERSION=$(grep 'postgres-version' common-nix.vars.pkr.hcl | sed -e 's/postgres-version = "\(.*\)"/\1/g')
44+
VERSION=$(sudo nix run nixpkgs#yq -- '.postgres_release["postgres'${{ matrix.postgres_version }}'"]' ansible/vars.yml)
45+
VERSION=$(echo $PG_VERSION | tr -d '"') # Remove any surrounding quotes
2546
if [[ "${{ inputs.postgresVersion }}" != "" ]]; then
2647
VERSION=${{ inputs.postgresVersion }}
2748
fi
@@ -58,6 +79,10 @@ jobs:
5879
publish-prod:
5980
runs-on: ubuntu-latest
6081
if: github.ref_name == 'develop' || contains( github.ref, 'release' )
82+
needs: prepare
83+
strategy:
84+
matrix:
85+
postgres_version: ${{ fromJson(needs.prepare.outputs.postgres_versions) }}
6186

6287
steps:
6388
- name: Checkout Repo
@@ -66,10 +91,8 @@ jobs:
6691
- name: Grab release version
6792
id: process_release_version
6893
run: |
69-
VERSION=$(grep 'postgres-version' common-nix.vars.pkr.hcl | sed -e 's/postgres-version = "\(.*\)"/\1/g')
70-
if [[ "${{ inputs.postgresVersion }}" != "" ]]; then
71-
VERSION=${{ inputs.postgresVersion }}
72-
fi
94+
VERSION=$(sudo nix run nixpkgs#yq -- '.postgres_release["postgres'${{ matrix.postgres_version }}'"]' ansible/vars.yml)
95+
VERSION=$(echo $PG_VERSION | tr -d '"') # Remove any surrounding quotes
7396
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
7497
echo "major_version=$(echo $VERSION | cut -d'.' -f1)" >> "$GITHUB_OUTPUT"
7598

‎.github/workflows/publish-nix-pgupgrade-scripts.yml‎

Lines changed: 32 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,6 @@ on:
77
- release/*
88
paths:
99
- '.github/workflows/publish-nix-pgupgrade-scripts.yml'
10-
- 'common-nix.vars.pkr.hcl'
1110
workflow_dispatch:
1211
inputs:
1312
postgresVersion:
@@ -18,17 +17,40 @@ permissions:
1817
id-token: write
1918

2019
jobs:
20+
prepare:
21+
runs-on: ubuntu-latest
22+
outputs:
23+
postgres_versions: ${{ steps.set-versions.outputs.postgres_versions }}
24+
steps:
25+
- name: Checkout Repo
26+
uses: actions/checkout@v3
27+
28+
- uses: DeterminateSystems/nix-installer-action@main
29+
30+
- name: Set PostgreSQL versions
31+
id: set-versions
32+
run: |
33+
VERSIONS=$(nix run nixpkgs#yq -- '.postgres_major[]' ansible/vars.yml | nix run nixpkgs#jq -- -R -s -c 'split("\n")[:-1]')
34+
echo "postgres_versions=$VERSIONS" >> $GITHUB_OUTPUT
35+
2136
publish-staging:
37+
needs: prepare
2238
runs-on: ubuntu-latest
39+
strategy:
40+
matrix:
41+
postgres_version: ${{ fromJson(needs.prepare.outputs.postgres_versions) }}
2342

2443
steps:
2544
- name: Checkout Repo
2645
uses: actions/checkout@v3
2746

47+
- uses: DeterminateSystems/nix-installer-action@main
48+
2849
- name: Grab release version
2950
id: process_release_version
3051
run: |
31-
VERSION=$(grep 'postgres-version' common-nix.vars.pkr.hcl | sed -e 's/postgres-version = "\(.*\)"/\1/g')
52+
VERSION=$(sudo nix run nixpkgs#yq -- '.postgres_release["postgres'${{ matrix.postgres_version }}'"]' ansible/vars.yml)
53+
VERSION=$(echo $PG_VERSION | tr -d '"') # Remove any surrounding quotes
3254
if [[ "${{ inputs.postgresVersion }}" != "" ]]; then
3355
VERSION=${{ inputs.postgresVersion }}
3456
fi
@@ -61,20 +83,24 @@ jobs:
6183
SLACK_FOOTER: ''
6284

6385
publish-prod:
86+
needs: prepare
6487
runs-on: ubuntu-latest
6588
if: github.ref_name == 'develop' || contains( github.ref, 'release' )
6689

90+
strategy:
91+
matrix:
92+
postgres_version: ${{ fromJson(needs.prepare.outputs.postgres_versions) }}
93+
94+
6795
steps:
6896
- name: Checkout Repo
6997
uses: actions/checkout@v3
7098

7199
- name: Grab release version
72100
id: process_release_version
73101
run: |
74-
VERSION=$(grep 'postgres-version' common-nix.vars.pkr.hcl | sed -e 's/postgres-version = "\(.*\)"/\1/g')
75-
if [[ "${{ inputs.postgresVersion }}" != "" ]]; then
76-
VERSION=${{ inputs.postgresVersion }}
77-
fi
102+
VERSION=$(sudo nix run nixpkgs#yq -- '.postgres_release["postgres'${{ matrix.postgres_version }}'"]' ansible/vars.yml)
103+
VERSION=$(echo $PG_VERSION | tr -d '"') # Remove any surrounding quotes
78104
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
79105
80106
- name: Create a tarball containing pg_upgrade scripts

‎.github/workflows/test.yml‎

Lines changed: 56 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,27 @@ on:
88
workflow_dispatch:
99

1010
jobs:
11+
prepare:
12+
runs-on: ubuntu-latest
13+
outputs:
14+
postgres_versions: ${{ steps.set-versions.outputs.postgres_versions }}
15+
steps:
16+
- name: Checkout Repo
17+
uses: actions/checkout@v4
18+
19+
- uses: DeterminateSystems/nix-installer-action@main
20+
21+
- name: Set PostgreSQL versions
22+
id: set-versions
23+
run: |
24+
VERSIONS=$(nix run nixpkgs#yq -- '.postgres_major[]' ansible/vars.yml | nix run nixpkgs#jq -- -R -s -c 'split("\n")[:-1]')
25+
echo "postgres_versions=$VERSIONS" >> $GITHUB_OUTPUT
26+
1127
build:
28+
needs: prepare
1229
strategy:
1330
matrix:
31+
postgres_version: ${{ fromJson(needs.prepare.outputs.postgres_versions) }}
1432
include:
1533
- runner: [self-hosted, X64]
1634
arch: amd64
@@ -23,14 +41,36 @@ jobs:
2341
POSTGRES_PASSWORD: password
2442
steps:
2543
- uses: actions/checkout@v3
44+
45+
- uses: DeterminateSystems/nix-installer-action@main
46+
47+
- name: Set PostgreSQL version environment variable
48+
run: echo "POSTGRES_MAJOR_VERSION=${{ matrix.postgres_version }}" >> $GITHUB_ENV
49+
50+
- name: Strip quotes from pg major and set env var
51+
run: |
52+
stripped_version=$(echo ${{ matrix.postgres_version }} | sed 's/^"\(.*\)"$/\1/')
53+
echo "PGMAJOR=$stripped_version" >> $GITHUB_ENV
54+
55+
- name: Generate common-nix.vars.pkr.hcl
56+
run: |
57+
PG_VERSION=$(sudo nix run nixpkgs#yq -- '.postgres_release["postgres'${{ matrix.postgres_version }}'"]' ansible/vars.yml)
58+
PG_VERSION=$(echo $PG_VERSION | tr -d '"') # Remove any surrounding quotes
59+
echo 'postgres-version = "'$PG_VERSION'"' > common-nix.vars.pkr.hcl
60+
# Ensure there's a newline at the end of the file
61+
echo "" >> common-nix.vars.pkr.hcl
62+
2663
- id: settings
2764
# Remove spaces and quotes to get the raw version string
2865
run: sed -r 's/(\s|\")+//g' common-nix.vars.pkr.hcl >> $GITHUB_OUTPUT
2966

30-
- id: args
31-
uses: mikefarah/yq@master
32-
with:
33-
cmd: yq 'to_entries | map(select(.value|type == "!!str")) | map(.key + "=" + .value) | join("\n")' 'ansible/vars.yml'
67+
- name: Generate args
68+
id: args
69+
run: |
70+
ARGS=$(sudo nix run nixpkgs#yq -- 'to_entries | map(select(.value|type == "!!str")) | map(.key + "=" + .value) | join("\n")' ansible/vars.yml)
71+
echo "result<<EOF" >> $GITHUB_OUTPUT
72+
echo "$ARGS" >> $GITHUB_OUTPUT
73+
echo "EOF" >> $GITHUB_OUTPUT
3474
3575
- run: docker context create builders
3676
- uses: docker/setup-buildx-action@v3
@@ -40,7 +80,7 @@ jobs:
4080
with:
4181
load: true
4282
context: .
43-
file: "Dockerfile-156"
83+
file: Dockerfile-${{ env.PGMAJOR }}
4484
target: production
4585
build-args: |
4686
${{ steps.args.outputs.result }}
@@ -57,10 +97,13 @@ jobs:
5797
-p ${{ env.POSTGRES_PORT }}:5432 \
5898
--name supabase_postgres \
5999
-d supabase/postgres:${{ steps.settings.outputs.postgres-version }}
100+
60101
- name: Install psql
61102
run: |
103+
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
104+
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
62105
sudo apt update
63-
sudo apt install -y --no-install-recommends postgresql-client
106+
sudo apt install -y --no-install-recommends postgresql-client-${{ env.PGMAJOR }}
64107
65108
- name: Install pg_prove
66109
run: sudo cpan -T TAP::Parser::SourceHandler::pgTAP
@@ -107,11 +150,15 @@ jobs:
107150
PGUSER: supabase_admin
108151
PGPASSWORD: ${{ env.POSTGRES_PASSWORD }}
109152

153+
- name: Update Dockerfile.dbmate version
154+
run: |
155+
sed -i 's/%VERSION%/${{ env.PGMAJOR }}/g' migrations/Dockerfile.dbmate
156+
110157
- name: verify schema.sql is committed
111158
run: |
112159
docker compose -f migrations/docker-compose.yaml up db dbmate --abort-on-container-exit
113-
if ! git diff --ignore-space-at-eol --exit-code --quiet migrations/schema.sql; then
114-
echo "Detected uncommitted changes after build. See status below:"
115-
git diff
160+
if ! git diff --exit-code --quiet migrations/schema-${{ env.PGMAJOR }}.sql; then
161+
echo "Detected changes in schema.sql:"
162+
git diff migrations/schema-${{ env.PGMAJOR }}.sql
116163
exit 1
117164
fi

0 commit comments

Comments
 (0)