Can't increase PVC size for postgresql

[h00jraq]

Official Helm Chart version

1.9.0 (latest released)

Apache Airflow version

2.5.0

Kubernetes Version

1.26

Helm Chart configuration

Here is the part of postgresql in my values.yaml file:

airflow:
....
postgresql:
  primary:
    persistence:
      enabled: true
      size: 50Gi
      existingClaim: airflow-postgresql-db
  resources:
    limits:
      cpu: 1000m
      memory: 2048Mi
    requests:
      cpu: 200m
      memory: 256Mi
....

I've also tried with below but it has not changed anything.

airflow:
....
postgresql:
  persistence:
    enabled: true
    size: 50Gi
    existingClaim: airflow-postgresql-db
  resources:
    limits:
      cpu: 1000m
      memory: 2048Mi
    requests:
      cpu: 200m
      memory: 256Mi
....

tried also with different indentation:

airflow:
....
  postgresql:
    persistence:
      enabled: true
      size: 50Gi
      existingClaim: airflow-postgresql-db
    resources:
      limits:
        cpu: 1000m
        memory: 2048Mi
      requests:
        cpu: 200m
        memory: 256Mi
....

PVC and PV are created and in bound state:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: disk.csi.azure.com
    volume.kubernetes.io/storage-provisioner: disk.csi.azure.com
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/component: primary
    app.kubernetes.io/instance: airflow
    app.kubernetes.io/name: postgresql
  name: airflow-postgresql-db
  namespace: airflow-sandbox
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: default
  volumeMode: Filesystem
  volumeName: postgres-db-pv

PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: disk.csi.azure.com
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/disk-csi-azure-com
  name: postgres-db-pv
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: airflow-postgresql-db
    namespace: airflow-sandbox
  csi:
    driver: disk.csi.azure.com
    volumeAttributes:
      csi.storage.k8s.io/pvc/namespace: airflow-sandbox
      requestedsizegib: "50"
      skuname: StandardSSD_LRS
    volumeHandle: /mysub/myrg/providers/Microsoft.Compute/disks/postgres-db-pv
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.disk.csi.azure.com/zone
          operator: In
          values:
          - uksouth-3
  persistentVolumeReclaimPolicy: Delete
  storageClassName: default
  volumeMode: Filesystem

Docker Image customizations

I use standard docker image from Airflow

What happened

For some reason, postgres is always creating pvc named data-airflow-postgresql-0. Is it possible that this comes from volumeClaimTemplates or something? This is what I see when running kubectl get statefulsets.apps -n airflow-sandbox airflow-postgresql -o yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    [meta.helm.sh/release-name](http://meta.helm.sh/release-name): airflow
    [meta.helm.sh/release-namespace](http://meta.helm.sh/release-namespace): airflow-sandbox
  creationTimestamp: "2023-05-22T07:08:01Z"
  generation: 1
  labels:
    [app.kubernetes.io/component](http://app.kubernetes.io/component): primary
    [app.kubernetes.io/instance](http://app.kubernetes.io/instance): airflow
    [app.kubernetes.io/managed-by](http://app.kubernetes.io/managed-by): Helm
    [app.kubernetes.io/name](http://app.kubernetes.io/name): postgresql
    [helm.sh/chart](http://helm.sh/chart): postgresql-12.1.9
  name: airflow-postgresql
  namespace: airflow-sandbox
  resourceVersion: "88975048"
  uid: 58c0d7d5-5dd3-428a-bba0-fd3a300b4a6f
spec:
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      [app.kubernetes.io/component](http://app.kubernetes.io/component): primary
      [app.kubernetes.io/instance](http://app.kubernetes.io/instance): airflow
      [app.kubernetes.io/name](http://app.kubernetes.io/name): postgresql
  serviceName: airflow-postgresql-hl
  template:
    metadata:
      creationTimestamp: null
      labels:
        [app.kubernetes.io/component](http://app.kubernetes.io/component): primary
        [app.kubernetes.io/instance](http://app.kubernetes.io/instance): airflow
        [app.kubernetes.io/managed-by](http://app.kubernetes.io/managed-by): Helm
        [app.kubernetes.io/name](http://app.kubernetes.io/name): postgresql
        [helm.sh/chart](http://helm.sh/chart): postgresql-12.1.9
      name: airflow-postgresql
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  [app.kubernetes.io/component](http://app.kubernetes.io/component): primary
                  [app.kubernetes.io/instance](http://app.kubernetes.io/instance): airflow
                  [app.kubernetes.io/name](http://app.kubernetes.io/name): postgresql
              topologyKey: [kubernetes.io/hostname](http://kubernetes.io/hostname)
            weight: 1
      containers:
      - env:
        - name: BITNAMI_DEBUG
          value: "false"
        - name: POSTGRESQL_PORT_NUMBER
          value: "5432"
        - name: POSTGRESQL_VOLUME_DIR
          value: /bitnami/postgresql
        - name: PGDATA
          value: /bitnami/postgresql/data
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              key: postgres-password
              name: airflow-postgresql
        - name: POSTGRESQL_ENABLE_LDAP
          value: "no"
        - name: POSTGRESQL_ENABLE_TLS
          value: "no"
        - name: POSTGRESQL_LOG_HOSTNAME
          value: "false"
        - name: POSTGRESQL_LOG_CONNECTIONS
          value: "false"
        - name: POSTGRESQL_LOG_DISCONNECTIONS
          value: "false"
        - name: POSTGRESQL_PGAUDIT_LOG_CATALOG
          value: "off"
        - name: POSTGRESQL_CLIENT_MIN_MESSAGES
          value: error
        - name: POSTGRESQL_SHARED_PRELOAD_LIBRARIES
          value: pgaudit
        image: [docker.io/bitnami/postgresql:11](http://docker.io/bitnami/postgresql:11)
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - exec pg_isready -U "postgres" -h 127.0.0.1 -p 5432
          failureThreshold: 6
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: postgresql
        ports:
        - containerPort: 5432
          name: tcp-postgresql
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - -e
            - |
              exec pg_isready -U "postgres" -h 127.0.0.1 -p 5432
              [ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]
          failureThreshold: 6
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          requests:
            cpu: 250m
            memory: 256Mi
        securityContext:
          runAsUser: 1001
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
        - mountPath: /bitnami/postgresql
          name: data
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir:
          medium: Memory
        name: dshm
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 8Gi
      volumeMode: Filesystem
    status:
      phase: Pending
status:
  availableReplicas: 1
  collisionCount: 0
  currentReplicas: 1
  currentRevision: airflow-postgresql-78588bb7f9
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updateRevision: airflow-postgresql-78588bb7f9
  updatedReplicas: 1

What you think should happen instead

I would like postgresql to use the PVC which I have declared in existingClaim in my values.yaml

How to reproduce

1. Create PVC based on the one I have provided
2. Add your PVC name to existingClaim: in postgresql part of values.yaml
3. Uninstall your Airflow release and delete pvc previously used by postgresql pod (you will loose your data!!)
4. Install new Airflow release - Airflow will ignore the PVC which you had used in existingClaim: and will create and use data-airflow-postgresql-0

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

Postgres in the Helm Chart is specificaly called out as development only for quick start. It's clearly written in the chart docs. Making statefull database works as K8S entity is a difficult task and we are not even pretending what we provide is ready for that. It is not. there are plenty of other things missing there that shoudl be there if you want to make a "production ready" database. We are not going to add it.

Do not use it for production. It's not customizable, don't try to fine tune it. This is a feature, not a bug that it is not customizable.

If you want to make your installation works for anything else than quick start and playing with the chart, you need to have Postgres instance outside of the chart. There are plenty of options for managing your database somewhere else. This what you need.