Skip to content

Commit 56f0a18

Browse files
authored
Migrate DB to VM (bcgov#2423)
Add psql container for testing
1 parent eb6c16c commit 56f0a18

File tree

11 files changed

+320
-4
lines changed

11 files changed

+320
-4
lines changed

api/net/Areas/Editor/Controllers/ContentController.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
using TNO.Core.Extensions;
1717
using TNO.Core.Storage;
1818
using TNO.DAL.Config;
19+
using TNO.DAL.Helpers;
1920
using TNO.DAL.Services;
2021
using TNO.Elastic;
2122
using TNO.Entities;
@@ -26,7 +27,6 @@
2627
using TNO.Keycloak;
2728
using TNO.Models.Extensions;
2829
using TNO.Models.Filters;
29-
using TNO.DAL.Helpers;
3030
namespace TNO.API.Areas.Editor.Controllers;
3131

3232
/// <summary>

db/postgres/README.md

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,3 +70,121 @@ Update the root configuration `.env` file and point it to the Redhat files.
7070
DB_CONTEXT=db/postgres/rhel8
7171
DB_VOLUME=/var/lib/pgsql/data
7272
```
73+
74+
## Backup Commands
75+
76+
```bash
77+
# Backup and zip
78+
pg_dump -h postgres -U admin -C -Fc -v -d tno | gzip > /mnt/data/dev.tar.gz
79+
80+
# Unzip
81+
gzip -dk dev.tar.gz
82+
83+
# Copy file to local
84+
oc -n 9b301c-dev rsync psql-4-zdbv6:/mnt/data/dev.tar.gz /D/db
85+
86+
# Copy file to database server
87+
scp -v /D/db/dev.tar.gz jerfos_a@142.34.249.231:/u02/data/postgres
88+
```
89+
90+
## Backup Database and Restore to Remote Server
91+
92+
Here are the steps to backup a full database and migrate to another location.
93+
94+
First create run a container with the same Postgres version as the one you want to backup.
95+
96+
```bash
97+
# Create a volume for the database backup
98+
docker volume create postgres-backup
99+
100+
# Start the container
101+
docker run \
102+
--name postgres \
103+
-l 15.10 \
104+
-p 5432:5432 \
105+
-e POSTGRES_USER=admin \
106+
-e POSTGRES_PASSWORD=password \
107+
-e POSTGRES_DB=mmi \
108+
-v postgres-backup:/var/lib/postgresql/data \
109+
-d --rm \
110+
postgres:15.10
111+
112+
# Create variable for environment
113+
podenv="dev"
114+
115+
# Map a port to the remote database in Openshift
116+
oc port-forward postgres-0 22222:5432 -n 9b301c-${podenv}
117+
118+
# SSH into the container
119+
docker exec -it postgres bash
120+
121+
# Move to shared volume
122+
cd /var/lib/postgresql/data
123+
124+
# Within the postgres container connect to the remote database
125+
psql -U admin -h host.docker.internal -p 22222 -d tno
126+
\q
127+
128+
# Start backup. This will take 10-30 minutes.
129+
pg_dump -h host.docker.internal -p 22222 -U admin -C -Fc -v -d tno > backup.sql
130+
131+
# Connect to the government VPN
132+
# Connect to the destination database.
133+
psql -U mmiadmin -h 142.34.249.231 -d mmi
134+
135+
# Clear out the database if one exists.
136+
\c mmi
137+
\dt
138+
drop schema public cascade;
139+
create schema public;
140+
grant all on schema public to postgres;
141+
grant all on schema public to mmiadmin;
142+
\q
143+
144+
# Restore the database to the new remote database. This will take 10-30 minutes.
145+
pg_restore -U mmiadmin -h 142.34.249.231 -d mmi -v -Fc backup.sql
146+
147+
# Exit the local container
148+
exit
149+
150+
# Stop the local postgres container
151+
docker stop postgres
152+
153+
# Remove the volume when done to recover space
154+
docker volume rm postgres-backup -f
155+
```
156+
157+
Configure Openshift environment to use remote database
158+
159+
Create a `.env` file that will contain your database secrets.
160+
This is to ensure it does not get added to source code.
161+
162+
```bash
163+
164+
# Encode username and password
165+
echo "username" | base64
166+
echo "password" | base64
167+
168+
# Place encoded values into yaml and create secret in openshift
169+
oc create -f db-secret.yaml.env -n 9b301c-${podenv}
170+
```
171+
172+
Update the API environment variables to use the new secret.
173+
174+
```bash
175+
# Update the API ConfigMap connection string
176+
# Old value = Host=postgres:5432;Database=tno;Include Error Detail=true;Log Parameters=true;
177+
oc patch -n 9b301c-${podenv} configmap api --type='merge' -p '{ "data": { "CONNECTION_STRING": "Host=142.34.249.231:5432;Database=mmi;Include Error Detail=true;Log Parameters=true;" }}'
178+
179+
# Update the statefulset
180+
oc patch -n 9b301c-${podenv} sts/api -p '{ "spec": { "template": { "spec": { "containers": [{ "name": "api", "env": [{ "name": "DB_POSTGRES_USERNAME", "valueFrom": { "secretKeyRef": { "name": "montford", "key": "USERNAME" }}}]}]}}}}'
181+
oc patch -n 9b301c-${podenv} sts/api -p '{ "spec": { "template": { "spec": { "containers": [{ "name": "api", "env": [{ "name": "DB_POSTGRES_PASSWORD", "valueFrom": { "secretKeyRef": { "name": "montford", "key": "PASSWORD" }}}]}]}}}}'
182+
183+
# Rollout change to statefulset
184+
oc rollout restart sts/api -n 9b301c-${podenv}
185+
oc rollout latest dc/api-services -n 9b301c-${podenv}
186+
187+
# Update the deployment config
188+
oc patch -n 9b301c-${podenv} dc/api-services -p '{ "spec": { "template": { "spec": { "containers": [{ "name": "api-services", "env": [{ "name": "DB_POSTGRES_USERNAME", "valueFrom": { "secretKeyRef": { "name": "montford", "key": "USERNAME" }}}]}]}}}}'
189+
oc patch -n 9b301c-${podenv} dc/api-services -p '{ "spec": { "template": { "spec": { "containers": [{ "name": "api-services", "env": [{ "name": "DB_POSTGRES_PASSWORD", "valueFrom": { "secretKeyRef": { "name": "montford", "key": "PASSWORD" }}}]}]}}}}'
190+
```

db/tools/Dockerfile

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
FROM alpine/psql:latest
2+
3+
RUN apk update && apk add -y --no-cache iputils-ping bind-tools wget
4+
5+
ENTRYPOINT ["tail", "-f", "/dev/null"]

openshift/README.md

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -206,11 +206,11 @@ oc run some-pod --overrides='{"spec": {"containers": [{"command": ["/bin/bash",
206206

207207
## Helpful Information on Docker Permissions
208208

209-
(Documentation)[https://developers.redhat.com/blog/2020/10/26/adapting-docker-and-kubernetes-containers-to-run-on-red-hat-openshift-container-platform#executable_permissions]
209+
[Documentation](https://developers.redhat.com/blog/2020/10/26/adapting-docker-and-kubernetes-containers-to-run-on-red-hat-openshift-container-platform#executable_permissions)
210210

211211
## Open a remote shell to containers
212212

213-
(Documentat)[https://docs.openshift.com/container-platform/3.11/dev_guide/ssh_environment.html]
213+
[Documentation](https://docs.openshift.com/container-platform/3.11/dev_guide/ssh_environment.html)
214214

215215
```bash
216216
oc rsh <pod>
@@ -227,3 +227,13 @@ for pod in $(oc get pods | grep Error | awk '{print $1}'); do oc delete pod --gr
227227
```bash
228228
oc get pods api-0 -o jsonpath="{..imageID}"
229229
```
230+
231+
## Sysdig
232+
233+
<https://app.sysdigcloud.com/#/login>
234+
235+
Login with OpenID.
236+
237+
Enter `BCDevOps` for the authentication.
238+
239+
Login with IDIR.
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# How the app will be deployed to the pod.
2+
kind: DeploymentConfig
3+
apiVersion: apps.openshift.io/v1
4+
metadata:
5+
name: psql
6+
namespace: default
7+
annotations:
8+
description: Defines how to deploy psql
9+
labels:
10+
name: psql
11+
part-of: tno
12+
version: 1.0.0
13+
component: database
14+
managed-by: kustomize
15+
created-by: jeremy.foster
16+
spec:
17+
replicas: 1
18+
selector:
19+
name: psql
20+
part-of: tno
21+
component: database
22+
strategy:
23+
rollingParams:
24+
intervalSeconds: 1
25+
maxSurge: 25%
26+
maxUnavailable: 25%
27+
timeoutSeconds: 600
28+
updatePeriodSeconds: 1
29+
type: Rolling
30+
template:
31+
metadata:
32+
name: psql
33+
labels:
34+
name: psql
35+
part-of: tno
36+
component: database
37+
spec:
38+
volumes:
39+
- name: backup-verification
40+
persistentVolumeClaim:
41+
claimName: backup-verification
42+
containers:
43+
- name: psql
44+
image: ""
45+
imagePullPolicy: Always
46+
ports:
47+
- containerPort: 8080
48+
protocol: TCP
49+
volumeMounts:
50+
- name: backup-verification
51+
mountPath: /mnt/data
52+
resources:
53+
requests:
54+
cpu: 20m
55+
memory: 50Mi
56+
limits:
57+
cpu: 50m
58+
memory: 100Mi
59+
dnsPolicy: ClusterFirst
60+
restartPolicy: Always
61+
securityContext: {}
62+
terminationGracePeriodSeconds: 30
63+
test: false
64+
triggers:
65+
- type: ConfigChange
66+
- type: ImageChange
67+
imageChangeParams:
68+
automatic: true
69+
containerNames:
70+
- psql
71+
from:
72+
kind: ImageStreamTag
73+
namespace: 9b301c-tools
74+
name: psql:dev
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1beta1
3+
kind: Kustomization
4+
5+
resources:
6+
- deploy.yaml
7+
8+
generatorOptions:
9+
disableNameSuffixHash: true
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
---
2+
# The final build image.
3+
kind: ImageStream
4+
apiVersion: image.openshift.io/v1
5+
metadata:
6+
name: psql
7+
annotations:
8+
description: Destination for built images.
9+
labels:
10+
name: psql
11+
part-of: tno
12+
version: 1.0.0
13+
component: database
14+
managed-by: kustomize
15+
created-by: jeremy.foster
16+
17+
---
18+
# The build config that will be created will be named for the branch you created it for.
19+
kind: BuildConfig
20+
apiVersion: build.openshift.io/v1
21+
metadata:
22+
name: psql.dev
23+
annotations:
24+
description: Build image from Dockerfile in git repo.
25+
labels:
26+
name: psql
27+
part-of: tno
28+
version: 1.0.0
29+
component: database
30+
managed-by: kustomize
31+
created-by: jeremy.foster
32+
branch: dev
33+
spec:
34+
completionDeadlineSeconds: 1800
35+
triggers:
36+
- type: ImageChange
37+
- type: ConfigChange
38+
runPolicy: Serial
39+
source:
40+
git:
41+
uri: https://github.com/bcgov/tno.git
42+
ref: dev
43+
contextDir: db/tools
44+
strategy:
45+
type: Docker
46+
dockerStrategy:
47+
imageOptimizationPolicy: SkipLayers
48+
dockerfilePath: Dockerfile
49+
output:
50+
to:
51+
kind: ImageStreamTag
52+
name: psql:latest
53+
resources:
54+
requests:
55+
cpu: 20m
56+
memory: 100Mi
57+
limits:
58+
cpu: 100m
59+
memory: 1Gi
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1beta1
3+
kind: Kustomization
4+
5+
resources:
6+
- build.yaml
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1beta1
3+
kind: Kustomization
4+
namespace: 9b301c-tools
5+
6+
resources:
7+
- ../../base
8+
9+
patches:
10+
- target:
11+
kind: BuildConfig
12+
name: psql.dev
13+
patch: |-
14+
- op: replace
15+
path: /spec/source/git/uri
16+
value: https://github.com/bcgov/tno.git
17+
- op: replace
18+
path: /spec/source/git/ref
19+
value: dev
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1beta1
3+
kind: Kustomization
4+
namespace: 9b301c-dev
5+
6+
resources:
7+
- ../../base
8+
9+
patches:
10+
- target:
11+
kind: DeploymentConfig
12+
name: nginx
13+
patch: |-
14+
- op: replace
15+
path: /spec/replicas
16+
value: 1

0 commit comments

Comments
 (0)