Skip to content

Commit bb3a307

Browse files
andhreljaKernandhreljaKern
authored
Postgres DB backup (#23)
1 parent 99e2f40 commit bb3a307

File tree

7 files changed

+373
-39
lines changed

7 files changed

+373
-39
lines changed

‎.github/workflows/pg_dump.yml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
name: 'Postgres: Dump Database'
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
pg_dump_name:
7+
description: 'Dump Name'
8+
required: true
9+
type: string
10+
database_name:
11+
description: 'Database Name'
12+
required: true
13+
default: refinery
14+
type: string
15+
16+
jobs:
17+
pg-dump:
18+
name: 'Postgres: Dump ${{ inputs.database_name }}'
19+
runs-on: [self-hosted, "${{ github.ref_name }}"]
20+
environment: ${{ github.ref_name }}
21+
env:
22+
ENVIRONMENT_NAME: ${{ github.ref_name }}
23+
FILE_SHARE_RESOURCE_GROUP: "${{ vars.FILE_SHARE_RESOURCE_GROUP }}"
24+
STORAGE_ACCOUNT_NAME: "${{ vars.STORAGE_ACCOUNT_NAME }}"
25+
STORAGE_ACCOUNT_KEY: "${{ secrets.STORAGE_ACCOUNT_KEY }}"
26+
FILE_SHARE_NAME: "${{ vars.FILE_SHARE_NAME }}"
27+
PG_HOST: "${{ secrets.PG_HOST }}"
28+
PG_USER: "${{ secrets.PG_USER }}"
29+
PG_PASSWORD: "${{ secrets.PG_PASSWORD }}"
30+
steps:
31+
- name: Checkout Repository
32+
uses: actions/checkout@v2
33+
with:
34+
repository: '${{ github.repository_owner }}/cicd-deployment-scripts'
35+
ref: 'pg-dump-restore'
36+
37+
- name: Generate Dump
38+
id: generate-dump
39+
shell: bash
40+
run: |
41+
bash ./pg/dump.sh \
42+
-h "${{ env.PG_HOST }}" \
43+
-u "${{ env.PG_USER }}" \
44+
-d "${{ inputs.database_name }}" \
45+
-n "${{ inputs.pg_dump_name }}" \
46+
-p "${{ env.PG_PASSWORD }}"
47+
48+
- name: Upload Dump
49+
shell: bash
50+
run: |
51+
az storage file upload-batch \
52+
--destination ${{ env.FILE_SHARE_NAME }} \
53+
--destination-path pg_dump/${{ inputs.database_name }}/${{ inputs.pg_dump_name}} \
54+
--source ${{ steps.generate-dump.outputs.PG_DUMP_PATH }} \
55+
--account-name ${{ env.STORAGE_ACCOUNT_NAME }} \
56+
--account-key ${{ env.STORAGE_ACCOUNT_KEY }}

‎.github/workflows/pg_restore.yml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
name: 'Postgres: Restore Database'
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
pg_dump_name:
7+
description: 'Dump Name'
8+
required: true
9+
type: string
10+
database_name:
11+
description: 'Database Name'
12+
required: true
13+
default: refinery
14+
type: string
15+
16+
jobs:
17+
pg-restore:
18+
name: 'Postgres: Restore ${{ inputs.database_name }}'
19+
runs-on: [self-hosted, "${{ github.ref_name }}"]
20+
environment: ${{ github.ref_name }}
21+
env:
22+
ENVIRONMENT_NAME: ${{ github.ref_name }}
23+
FILE_SHARE_RESOURCE_GROUP: "${{ vars.FILE_SHARE_RESOURCE_GROUP }}"
24+
STORAGE_ACCOUNT_NAME: "${{ vars.STORAGE_ACCOUNT_NAME }}"
25+
STORAGE_ACCOUNT_KEY: "${{ secrets.STORAGE_ACCOUNT_KEY }}"
26+
FILE_SHARE_NAME: "${{ vars.FILE_SHARE_NAME }}"
27+
PG_HOST: "${{ secrets.PG_HOST }}"
28+
PG_USER: "${{ secrets.PG_USER }}"
29+
PG_PASSWORD: "${{ secrets.PG_PASSWORD }}"
30+
steps:
31+
- name: Checkout Repository
32+
uses: actions/checkout@v2
33+
with:
34+
repository: '${{ github.repository_owner }}/cicd-deployment-scripts'
35+
ref: 'pg-dump-restore'
36+
37+
- name: Download Dump
38+
id: download-dump
39+
shell: bash
40+
run: |
41+
az storage file download-batch \
42+
--destination . \
43+
--source ${{ env.FILE_SHARE_NAME }} \
44+
--pattern "pg_dump/${{ inputs.database_name }}/${{ inputs.pg_dump_name}}/*" \
45+
--account-name ${{ env.STORAGE_ACCOUNT_NAME }} \
46+
--account-key ${{ env.STORAGE_ACCOUNT_KEY }} \
47+
--no-progress
48+
49+
echo "PG_DUMP_PATH=$(pwd)/pg_dump/${{ inputs.database_name }}/${{ inputs.pg_dump_name}}" >> $GITHUB_OUTPUT
50+
51+
- name: Restore Dump
52+
shell: bash
53+
run: |
54+
bash ./pg/restore.sh \
55+
-h "${{ env.PG_HOST }}" \
56+
-u "${{ env.PG_USER }}" \
57+
-d "${{ inputs.database_name }}" \
58+
-n "${{ steps.download-dump.outputs.PG_DUMP_PATH }}" \
59+
-p "${{ env.PG_PASSWORD }}"

‎README.md‎

Lines changed: 85 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ Scripts used for Kern AI CI/CD efforts.
2323
- [K8: Release](#k8-release)
2424
- [K8: Restart](#k8-restart)
2525
- [K8: Test](#k8-test)
26+
- [Postgres: Dump Database](#postgres-dump-database)
27+
- [Postgres: Restore Database](#postgres-restore-database)
2628
- [Parent Images: Build](#parent-images-build)
2729
- [Parent Images: Matrix](#parent-images-matrix)
2830
- [Parent Images: Submodule Merge](#parent-images-submodule-merge)
@@ -190,7 +192,7 @@ Outputs:
190192

191193
### Azure: Function App Deployment
192194

193-
Workflow file: `az_fnapp_deploy.yml`
195+
Workflow file: `az_fa_deploy.yml`
194196

195197
Triggers:
196198
- workflow_dispatch
@@ -590,6 +592,66 @@ Inputs:
590592

591593

592594

595+
### Postgres: Dump Database
596+
597+
Workflow file: `pg_dump.yml`
598+
599+
Triggers:
600+
- workflow_dispatch
601+
602+
Inputs:
603+
- pg_dump_name
604+
- database_name
605+
606+
607+
608+
**Description:**
609+
610+
- generates a PostgreSQL dump of the database specified by the workflow input
611+
- the dump is stored in the Azure File Storage configured by GitHub Actions Environment Variables
612+
613+
614+
615+
**Jobs:**
616+
617+
- Postgres: Dump ${{ inputs.database_name }}
618+
- `Generate Dump`
619+
- `Upload Dump`
620+
621+
622+
623+
624+
625+
### Postgres: Restore Database
626+
627+
Workflow file: `pg_restore.yml`
628+
629+
Triggers:
630+
- workflow_dispatch
631+
632+
Inputs:
633+
- pg_dump_name
634+
- database_name
635+
636+
637+
638+
**Description:**
639+
640+
- restores a PostgreSQL dump of the database specified by the workflow input
641+
- the dump is downloaded from the Azure File Storage configured by GitHub Actions Environment Variables
642+
643+
644+
645+
**Jobs:**
646+
647+
- Postgres: Restore ${{ inputs.database_name }}
648+
- `Download Dump`
649+
- `Restore Dump`
650+
651+
652+
653+
654+
593655
### Parent Images: Build
594656

595657
Workflow file: `pi_build.yml`
@@ -618,8 +680,18 @@ Triggers:
618680
- `Set up Python`
619681
- `Install Dependencies`
620682
- `Compile Requirements`
621-
- `Build & Push refinery-parent-images:${{ needs.configure-branch-name.outputs.gh_head_ref }}-${{ matrix.parent_image_type }}`
622-
- `Build & Push refinery-parent-images:${{ needs.configure-branch-name.outputs.gh_head_ref }}-${{ matrix.parent_image_type }}-arm64`
683+
- `Build & Push ${{ env.PARENT_IMAGE_NAME }}:${{ env.HEAD_REF }}-${{ matrix.parent_image_type }}`
684+
- `Build & Push ${{ env.PARENT_IMAGE_NAME }}:${{ env.HEAD_REF }}-${{ matrix.parent_image_type }}-arm64`
685+
686+
- Parent Images: App
687+
- `Set up Python`
688+
- `Install Dependencies`
689+
- `Compile Requirements`
690+
- `Clone ${{ matrix.app }}`
691+
- `Compile Requirements (Python)`
692+
- `Compile Requirements (Next)`
693+
- `Perform Edit/Git Operations (Python)`
694+
- `Perform Edit/Git Operations (Next)`
623695

624696

625697

@@ -636,6 +708,7 @@ Inputs:
636708
- repository
637709
- checkout_ref
638710
- parent_image_type
711+
- edit_dockerfile
639712

640713
Outputs:
641714
- parent_image_type
@@ -713,7 +786,10 @@ Triggers:
713786
- `Install Dependencies`
714787
- `Perform Edit/Git Operations`
715788

716-
- GitHub: Delete Branch
789+
- GitHub: Delete Submodule Branch
790+
- `Delete Branch`
791+
792+
- GitHub: Delete App Branch
717793
- `Delete Branch`
718794

719795

@@ -743,19 +819,14 @@ Triggers:
743819

744820
**Jobs:**
745821

746-
- Configure Head Branch Name
747-
- `Configure branch name`
748-
749822
- pi-matrix
750823

751824
- Parent Images: Docker Build
752825
- `Set up Python`
753826
- `Install Dependencies`
754827
- `Compile Requirements`
755-
- `Build & Push refinery-parent-images:${{ github.event.pull_request.base.ref }}-${{ env.PARENT_IMAGE_TYPE }}`
756-
- `Build & Push refinery-parent-images:${{ github.event.pull_request.base.ref }}-${{ env.PARENT_IMAGE_TYPE }}-arm64`
757-
- `Build & Push refinery-parent-images:sha-${{ env.PARENT_IMAGE_TYPE }}`
758-
- `Build & Push refinery-parent-images:sha-${{ env.PARENT_IMAGE_TYPE }}-arm64`
828+
- `Build & Push ${{ env.PARENT_IMAGE_NAME }}:${{ github.event.pull_request.base.ref }}-${{ env.PARENT_IMAGE_TYPE }}`
829+
- `Build & Push ${{ env.PARENT_IMAGE_NAME }}:${{ github.sha }}-${{ env.PARENT_IMAGE_TYPE }}`
759830

760831
- Parent Images: App
761832
- `Set up Python`
@@ -769,9 +840,6 @@ Triggers:
769840
- GitHub: Delete Branch
770841
- `Delete Branch`
771842

772-
- GitHub: Delete Branch
773-
- `Delete Branch`
774-
775843

776844

777845

@@ -798,9 +866,11 @@ Triggers:
798866

799867
- pi-matrix
800868

801-
- Parent Images: Dockerfile
869+
- Parent Images: Dockerfile
802870
- `Perform Edit/Git Operations`
803871

872+
- call-gh-release
873+
804874

805875

806876

0 commit comments

Comments
 (0)