initdb: error: directory "/var/snap/charmed-postgresql/common/var/lib/postgresql" exists but is not empty #1336
Description
Steps to reproduce
- Deploy a Charmed PostgreSQL 16 cluster that uses a second disk as the data storage. A sample bundle has been provided below, please customize this as per your environment configuration:
default-base: [email protected]/stable
applications:
postgresql:
charm: postgresql
channel: 16/stable
revision: 952
num_units: 3
to:
- "0"
- "1"
- "2"
options:
profile: production
storage:
data: nvme,1,150G
machines:
"0":
constraints: arch=amd64 instance-type=c1.large zones=az1 root-disk-source=volume root-disk=50000
"1":
constraints: arch=amd64 instance-type=c1.large zones=az1 root-disk-source=volume root-disk=50000
"2":
constraints: arch=amd64 instance-type=c1.large zones=az1 root-disk-source=volume root-disk=50000
Expected behavior
A three node cluster is expected to form and come online.
Actual behavior
Cluster never properly forms because nodes fail to initialize PostgreSQL due to a lost+found directory being present in the data storage disk (default ext4 behavior):
initdb: error: directory "/var/snap/charmed-postgresql/common/var/lib/postgresql" exists but is not empty
initdb: detail: It contains a lost+found directory, perhaps due to it being a mount point.
initdb: hint: Using a mount point directly as the data directory is not recommended.
Create a subdirectory under the mount point.
The files belonging to this database system will be owned by user "_daemon_".
This user must also own the server process.
The database cluster will be initialized with locale "C.UTF-8".
The default text search configuration will be set to "english".
Data page checksums are enabled.
initdb: error: directory "/var/snap/charmed-postgresql/common/var/lib/postgresql" exists but is not empty
initdb: detail: It contains a lost+found directory, perhaps due to it being a mount point.
initdb: hint: Using a mount point directly as the data directory is not recommended.
Create a subdirectory under the mount point.
Versions
Operating system: Ubuntu 24.04 LTS (Noble)
Juju CLI: 3.6.12
Juju agent: 3.6.12
Charm revision: 952 on 16/stable channel
LXD: Not Applicable
Workaround
Use a bash script tied to a systemd timer to monitor the data disk and delete lost+found to allow the cluster to form properly:
# Script to monitor the mountpoint
#!/usr/bin/env bash
set -euo pipefail
TARGET="/var/snap/charmed-postgresql/common/var/lib/postgresql"
SERVICE_NAME="pg-lostfound-watcher.service"
TIMER_NAME="pg-lostfound-watcher.timer"
echo "[pg-lostfound-watcher] Started, watching for mount: $TARGET"
while true; do
if [ -d "$TARGET" ]; then
if awk -v p="$TARGET" '$2 == p {found=1} END {exit !found}' /proc/mounts; then
echo "[pg-lostfound-watcher] Detected mount at $TARGET"
break
fi
fi
sleep 5
done
LF="$TARGET/lost+found"
if [ -d "$LF" ]; then
echo "[pg-lostfound-watcher] Removing $LF"
rm -rf -- "$LF"
else
echo "[pg-lostfound-watcher] No lost+found found at $LF (nothing to do)"
fi
echo "[pg-lostfound-watcher] Disabling $SERVICE_NAME and $TIMER_NAME"
systemctl disable --now "$TIMER_NAME" "$SERVICE_NAME" || true
echo "[pg-lostfound-watcher] Done, exiting."
exit 0
# Systemd Service
[Unit]
Description=Watch for charmed-postgresql data mount and remove lost+found once
After=local-fs.target
Wants=local-fs.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/pg-lostfound-watcher.sh
User=root
Group=root
# Systemd Timer
[Unit]
Description=Start pg-lostfound-watcher at boot
[Timer]
OnBootSec=10sec
Persistent=true
[Install]
WantedBy=timers.target
Incorporate the above elements in the juju model cloud init userdata:
cloudinit-userdata: |
write_files:
- path: /usr/local/sbin/pg-lostfound-watcher.sh
permissions: '0755'
encoding: b64
content: |
IyEvdXNyL2Jpbi9lbnYgYmFzaApzZXQgLWV1byBwaXBlZmFpbAoKVEFSR0VUPSIvdmFyL3NuYXAvY2hhcm1lZC1wb3N0Z3Jlc3FsL2NvbW1vbi92YXIvbGliL3Bvc3RncmVzcWwiClNFUlZJQ0VfTkFNRT0icGctbG9zdGZvdW5kLXdhdGNoZXIuc2VydmljZSIKVElNRVJfTkFNRT0icGctbG9zdGZvdW5kLXdhdGNoZXIudGltZXIiCgplY2hvICJbcGctbG9zdGZvdW5kLXdhdGNoZXJdIFN0YXJ0ZWQsIHdhdGNoaW5nIGZvciBtb3VudDogJFRBUkdFVCIKCndoaWxlIHRydWU7IGRvCiAgICBpZiBbIC1kICIkVEFSR0VUIiBdOyB0aGVuCiAgICAgICAgaWYgYXdrIC12IHA9IiRUQVJHRVQiICckMiA9PSBwIHtmb3VuZD0xfSBFTkQge2V4aXQgIWZvdW5kfScgL3Byb2MvbW91bnRzOyB0aGVuCiAgICAgICAgICAgIGVjaG8gIltwZy1sb3N0Zm91bmQtd2F0Y2hlcl0gRGV0ZWN0ZWQgbW91bnQgYXQgJFRBUkdFVCIKICAgICAgICAgICAgYnJlYWsKICAgICAgICBmaQogICAgZmkKICAgIHNsZWVwIDUKZG9uZQoKTEY9IiRUQVJHRVQvbG9zdCtmb3VuZCIKaWYgWyAtZCAiJExGIiBdOyB0aGVuCiAgICBlY2hvICJbcGctbG9zdGZvdW5kLXdhdGNoZXJdIFJlbW92aW5nICRMRiIKICAgIHJtIC1yZiAtLSAiJExGIgplbHNlCiAgICBlY2hvICJbcGctbG9zdGZvdW5kLXdhdGNoZXJdIE5vIGxvc3QrZm91bmQgZm91bmQgYXQgJExGIChub3RoaW5nIHRvIGRvKSIKZmkKCmVjaG8gIltwZy1sb3N0Zm91bmQtd2F0Y2hlcl0gRGlzYWJsaW5nICRTRVJWSUNFX05BTUUgYW5kICRUSU1FUl9OQU1FIgpzeXN0ZW1jdGwgZGlzYWJsZSAtLW5vdyAiJFRJTUVSX05BTUUiICIkU0VSVklDRV9OQU1FIiB8fCB0cnVlCgplY2hvICJbcGctbG9zdGZvdW5kLXdhdGNoZXJdIERvbmUsIGV4aXRpbmcuIgpleGl0IDAK
- path: /etc/systemd/system/pg-lostfound-watcher.service
permissions: '0644'
encoding: b64
content: |
W1VuaXRdCkRlc2NyaXB0aW9uPVdhdGNoIGZvciBjaGFybWVkLXBvc3RncmVzcWwgZGF0YSBtb3VudCBhbmQgcmVtb3ZlIGxvc3QrZm91bmQgb25jZQpBZnRlcj1sb2NhbC1mcy50YXJnZXQKV2FudHM9bG9jYWwtZnMudGFyZ2V0CgpbU2VydmljZV0KVHlwZT1vbmVzaG90CkV4ZWNTdGFydD0vdXNyL2xvY2FsL3NiaW4vcGctbG9zdGZvdW5kLXdhdGNoZXIuc2gKVXNlcj1yb290Ckdyb3VwPXJvb3QK
- path: /etc/systemd/system/pg-lostfound-watcher.timer
permissions: '0644'
encoding: b64
content: |
W1VuaXRdCkRlc2NyaXB0aW9uPVN0YXJ0IHBnLWxvc3Rmb3VuZC13YXRjaGVyIGF0IGJvb3QKCltUaW1lcl0KT25Cb290U2VjPTEwc2VjClBlcnNpc3RlbnQ9dHJ1ZQoKW0luc3RhbGxdCldhbnRlZEJ5PXRpbWVycy50YXJnZXQK
postruncmd:
- systemctl daemon-reload
- systemctl enable --now pg-lostfound-watcher.timer
juju model-config --file model-userdata.yaml
Please let me know if need anything else.