fix: restore defaults to /var/lib/postgresql

[samrose]

Security Analysis of /var/lib/postgresql/ Permissions

What was there BEFORE commit 388c2da (Oct 22, 2025)?

The directory had NO explicit mode set, meaning it got default permissions from the system when created:

• When created by ansible.builtin.file without a mode, directories typically get 0755 (drwxr-xr-x)
• When created as a user's home directory, it also typically gets 0755

So yes, it was effectively 0755 before the refactor.

What's in /var/lib/postgresql/?

Based on the codebase:

1. .nix-profile/ - Symlinks to Nix-managed PostgreSQL binaries and libraries (read-only, no sensitive data)
2. .bashrc - Shell configuration for postgres user (no sensitive data)
3. data/ - SYMLINK to /data/pgdata (the actual data directory, which has 0750)
4. extension/ - PostgreSQL extensions (read-only, no sensitive data)

Security Implications

Setting /var/lib/postgresql/ to 0755 is safe because:

1. The actual database data is NOT in /var/lib/postgresql/
   - Real data is in /data/pgdata (mode 0750, secure)
   - /var/lib/postgresql/data is just a symlink to /data/pgdata
   - Traversing through 0755 directory to reach a symlink doesn't bypass the target's permissions
2. What's exposed with 0755:
   - Nix profile binaries (already public in /usr/bin/)
   - .bashrc (just environment variables)
   - Extension files (no credentials)
3. What's NOT exposed:
   - Database files (in /data/pgdata with 0750)
   - Connection passwords (in /etc/postgresql/)
   - SSL keys (in /etc/ssl/private/ with 0750)

The October refactor (388c2da) introduced a regression

The refactor changed the behavior from:

• Before: /var/lib/postgresql had no explicit mode → system default (0755)
• After: /var/lib/postgresql/data created with recurse: true and mode 0750 → parent directory also got 0750

This broke symlinks without improving security.

Best Practice Recommendation

Recommended solution: Set /var/lib/postgresql/ to 0755

• This is the standard PostgreSQL practice
• Most distributions ship with this (check Debian, Ubuntu, RHEL PostgreSQL packages)
• Actual data security comes from /data/pgdata being 0750
• Allows necessary symlink traversal

Security boundaries are correct:

/var/lib/postgresql/           (0755 - traversable, contains no secrets)
├── .nix-profile/             (0755 - public binaries)
├── .bashrc                   (0644 - environment vars)
└── data -> /data/pgdata      (symlink, target is secure)

/data/pgdata/                 (0750 - SECURE, actual database)
├── base/                     (database files)
├── pg_wal/                   (WAL files)
└── ...

/etc/postgresql/              (0775 - configs, accessible)
/etc/ssl/private/             (0750 - SSL keys, SECURE)

Analysis of documented practices

PostgreSQL Directory Permissions - Security Documentation

Executive Summary

This document provides evidence that setting /var/lib/postgresql/ to 0755 permissions is the standard practice for PostgreSQL installations across major Linux distributions, and explains why this is secure.

Problem Statement

After commit 388c2da4a (October 22, 2025), which refactored ansible tasks to meet modern ansible-lint standards, the /var/lib/postgresql/ directory was inadvertently set to 0750 permissions. This broke symlink traversal for PostgreSQL binaries, preventing users from executing commands like psql through symlinks in /usr/bin/.

Security Analysis

Critical Distinction: Parent Directory vs Data Directory

PostgreSQL's security model distinguishes between:

1. Parent/Home Directory (/var/lib/postgresql/)

   • Contains non-sensitive files: binaries, scripts, shell configuration
   • Can safely be 0755 (world-readable/executable)
   • Does NOT contain database data

2. Data Directory (/var/lib/postgresql/data → /data/pgdata)

   • Contains actual database files, WAL logs, configurations with sensitive data
   • MUST be 0700 or 0750 (PostgreSQL enforces this, will refuse to start otherwise)
   • Contains all sensitive information

Official PostgreSQL Requirements

From PostgreSQL official documentation (v11+):

The data directory permissions must be u=rwx (0700) or u=rwx,g=rx (0750). The PostgreSQL server will refuse to start if the data directory has any other permissions.

Important: This requirement applies to the PGDATA directory (/var/lib/postgresql/data), not the parent directory.

Distribution Survey Results

We tested actual PostgreSQL installations across major Linux distributions to document standard practices:

Distribution	Parent Directory	Permissions	User:Group
Debian Bookworm	/var/lib/postgresql	0755	postgres:postgres
Ubuntu 24.04	/var/lib/postgresql	0755	postgres:postgres
Alpine Linux	/var/lib/postgresql	0750	postgres:postgres
Rocky Linux 9	/var/lib/pgsql	0700	postgres:postgres

Test Commands Used

# Debian
docker run --rm debian:bookworm bash -c \
  "apt-get update -qq && apt-get install -y -qq postgresql && \
   stat -c '%a %U:%G %n' /var/lib/postgresql"
# Output: 755 postgres:postgres /var/lib/postgresql

# Ubuntu 24.04
docker run --rm ubuntu:24.04 bash -c \
  "apt-get update -qq && apt-get install -y -qq postgresql && \
   stat -c '%a %U:%G %n' /var/lib/postgresql"
# Output: 755 postgres:postgres /var/lib/postgresql

# Alpine
docker run --rm alpine:latest sh -c \
  "apk add -q postgresql && \
   stat -c '%a %U:%G %n' /var/lib/postgresql"
# Output: 750 postgres:postgres /var/lib/postgresql

# Rocky Linux
docker run --rm rockylinux:9 bash -c \
  "dnf install -y -q postgresql-server && \
   stat -c '%a %U:%G %n' /var/lib/pgsql"
# Output: 700 postgres:postgres /var/lib/pgsql

Analysis of Distribution Choices

• Debian/Ubuntu (our platform): Uses 0755 - most permissive, allows tab completion and directory listing
• Alpine: Uses 0750 - group-readable
• RHEL/Rocky: Uses 0700 - most restrictive

Conclusion: Debian-based distributions (which we use) standardize on 0755 for the parent directory.

Our Directory Structure

/var/lib/postgresql/           (0755 - SAFE TO BE WORLD-READABLE)
├── .nix-profile/              (symlinks to binaries)
│   ├── bin/
│   │   ├── psql               (executable)
│   │   ├── pg_dump            (executable)
│   │   └── ...
│   ├── lib/                   (libraries)
│   └── share/                 (documentation, extensions)
├── .bashrc                    (shell config - no secrets)
└── data -> /data/pgdata       (SYMLINK - target has strict permissions)

/data/pgdata/                  (0750 - SECURE, ACTUAL DATA)
├── base/                      (database files - SENSITIVE)
├── global/                    (cluster-wide data - SENSITIVE)
├── pg_wal/                    (write-ahead logs - SENSITIVE)
├── pg_xact/                   (transaction data - SENSITIVE)
└── postgresql.auto.conf       (runtime config - SENSITIVE)

/etc/postgresql/               (0775 - configs, adminapi writable)
├── postgresql.conf            (main config)
├── pg_hba.conf               (authentication config)
└── pg_ident.conf             (ident map)

/etc/ssl/private/              (0750 - SSL keys, SECURE)
└── server.key                (SSL private key - VERY SENSITIVE)

What's Exposed with 0755 on /var/lib/postgresql/?

Safe to be World-Readable:

1. Nix profile binaries - Already publicly accessible via /usr/bin/ symlinks
2. Shell configuration (.bashrc) - Just environment variables (LANG, LOCALE_ARCHIVE)
3. Extension metadata - Public information about installed extensions
4. Directory structure - Allows tab completion and ls commands

NOT Exposed (Still Protected):

1. Database files - In /data/pgdata with strict 0750 permissions
2. Connection passwords - In /etc/postgresql/ and encrypted in database
3. SSL private keys - In /etc/ssl/private/ with 0750 permissions
4. User data - All in the secure data directory

Security Boundaries

PostgreSQL's security model relies on multiple layers:

Layer 1: Parent Directory   (/var/lib/postgresql - 0755 - Public traversal allowed)
Layer 2: Data Directory     (/data/pgdata - 0750 - Strict access control)
Layer 3: File Permissions   (Database files - 0600/0640 - Postgres user only)
Layer 4: PostgreSQL ACLs    (Row-level security, role permissions)

Opening Layer 1 does NOT compromise Layers 2-4.

Recommended Solution

Set /var/lib/postgresql/ to 0755

This is implemented in ansible/tasks/setup-postgres.yml:

- name: Set /var/lib/postgresql to 0755 for nix-profile symlink traversal
  ansible.builtin.file:
    group: 'postgres'
    mode: '0755'
    owner: 'postgres'
    path: '/var/lib/postgresql'
    state: 'directory'

Why This is Safe

1. Follows Debian/Ubuntu standard practice - Our production platform
2. Maintains strict data directory permissions - /data/pgdata remains 0750
3. Enables necessary functionality - Symlinks work for all users
4. Defense in depth - Multiple security layers remain intact
5. No sensitive data exposure - Parent directory contains only public binaries/configs

Testing and Validation

We added comprehensive permission checks in ansible/files/permission_check.py:

expected_directory_permissions = {
    "/var/lib/postgresql": ("0755", "postgres", "postgres",
        "PostgreSQL home - must be traversable for nix-profile symlinks"),
    "/var/lib/postgresql/data": ("0750", "postgres", "postgres",
        "PostgreSQL data directory symlink - secure, postgres only"),
    "/data/pgdata": ("0750", "postgres", "postgres",
        "Actual PostgreSQL data directory - secure, postgres only"),
    "/etc/ssl/private": ("0750", "root", "ssl-cert",
        "SSL private keys directory - secure, ssl-cert group only"),
    # ... additional checks
}

This test runs during AMI builds and will fail if any directory has incorrect permissions.

Historical Context

Before October 22, 2025 (Commit 388c2da)

- name: Create relevant directories
  file:
    path: '{{ item }}'
    recurse: yes
    state: directory
    owner: postgres
    group: postgres
    # NO MODE SPECIFIED - defaults to 0755
  with_items:
    - '/var/lib/postgresql'

Result: /var/lib/postgresql had 0755 (system default)

After October 22, 2025 (Commit 388c2da)

- name: Create relevant directories
  ansible.builtin.file:
    mode: '0750'  # Applied to ALL directories
    path: "{{ pg_dir_item }}"
    recurse: true  # Also sets parent directory to 0750!
  loop:
    - '/var/lib/postgresql/data'

Result: Both /var/lib/postgresql and /var/lib/postgresql/data got 0750

Regression: Broke symlink traversal without improving security

References

Official Documentation

• PostgreSQL: Creating a Database Cluster
  • Documents 0700/0750 requirement for PGDATA directory
• PostgreSQL Mailing List: /var/lib/pgsql 0755
  • Community discussion on parent directory permissions

Distribution Packages

• Debian PostgreSQL Package: postgresql-common
• Ubuntu PostgreSQL Package: Default in postgresql
• RHEL/Rocky PostgreSQL: postgresql-server

Related Commits

• 388c2da4a - "refactor(ansible): bring our ansible up to modern ansible-lint standards" (Oct 22, 2025)
  • Introduced the regression
• Current fix - Explicitly sets /var/lib/postgresql to 0755

Conclusion

Setting /var/lib/postgresql/ to 0755 is:

1. Standard practice - Matches Debian/Ubuntu defaults
2. Secure - Data directory remains protected at 0750
3. Functional - Enables symlink traversal for all users
4. Tested - Automated permission checks prevent regressions
5. Documented - Clear security boundaries and rationale

fix to be submitted

diff --git a/ansible/files/permission_check.py b/ansible/files/permission_check.py
index 72a1a2fe..afb3bba5 100644
--- a/ansible/files/permission_check.py
+++ b/ansible/files/permission_check.py
@@ -2,6 +2,8 @@ import subprocess
 import json
 import sys
 import argparse
+import os
+import stat
 
 
 # Expected groups for each user
@@ -103,6 +105,59 @@ expected_results = {
 # postgresql.service is expected to mount /etc as read-only
 expected_mount = "/etc ro"
 
+# Expected directory permissions for security-critical paths
+# Format: path -> (expected_mode, expected_owner, expected_group, description)
+expected_directory_permissions = {
+    "/var/lib/postgresql": (
+        "0755",
+        "postgres",
+        "postgres",
+        "PostgreSQL home - must be traversable for nix-profile symlinks",
+    ),
+    "/var/lib/postgresql/data": (
+        "0750",
+        "postgres",
+        "postgres",
+        "PostgreSQL data directory symlink - secure, postgres only",
+    ),
+    "/data/pgdata": (
+        "0750",
+        "postgres",
+        "postgres",
+        "Actual PostgreSQL data directory - secure, postgres only",
+    ),
+    "/etc/postgresql": (
+        "0775",
+        "postgres",
+        "postgres",
+        "PostgreSQL configuration directory - adminapi writable",
+    ),
+    "/etc/postgresql-custom": (
+        "0775",
+        "postgres",
+        "postgres",
+        "PostgreSQL custom configuration - adminapi writable",
+    ),
+    "/etc/ssl/private": (
+        "0750",
+        "root",
+        "ssl-cert",
+        "SSL private keys directory - secure, ssl-cert group only",
+    ),
+    "/home/postgres": (
+        "0750",
+        "postgres",
+        "postgres",
+        "postgres user home directory - secure, postgres only",
+    ),
+    "/var/log/postgresql": (
+        "0750",
+        "postgres",
+        "postgres",
+        "PostgreSQL logs directory - secure, postgres only",
+    ),
+}
+
 
 # This program depends on osquery being installed on the system
 # Function to run osquery
@@ -189,6 +244,69 @@ def check_postgresql_mount():
     print("postgresql.service mounts /etc as read-only.")
 
 
+def check_directory_permissions():
+    """Check that security-critical directories have the correct permissions."""
+    errors = []
+
+    for path, (expected_mode, expected_owner, expected_group, description) in expected_directory_permissions.items():
+        # Skip if path doesn't exist (might be a symlink or not created yet)
+        if not os.path.exists(path):
+            print(f"Warning: {path} does not exist, skipping permission check")
+            continue
+
+        # Get actual permissions
+        try:
+            stat_info = os.stat(path)
+            actual_mode = oct(stat.S_IMODE(stat_info.st_mode))[2:]  # Remove '0o' prefix
+
+            # Get owner and group names
+            import pwd
+            import grp
+            actual_owner = pwd.getpwuid(stat_info.st_uid).pw_name
+            actual_group = grp.getgrgid(stat_info.st_gid).gr_name
+
+            # Check permissions
+            if actual_mode != expected_mode:
+                errors.append(
+                    f"ERROR: {path} has mode {actual_mode}, expected {expected_mode}\n"
+                    f"  Description: {description}\n"
+                    f"  Fix: sudo chmod {expected_mode} {path}"
+                )
+
+            # Check ownership
+            if actual_owner != expected_owner:
+                errors.append(
+                    f"ERROR: {path} has owner {actual_owner}, expected {expected_owner}\n"
+                    f"  Description: {description}\n"
+                    f"  Fix: sudo chown {expected_owner}:{actual_group} {path}"
+                )
+
+            # Check group
+            if actual_group != expected_group:
+                errors.append(
+                    f"ERROR: {path} has group {actual_group}, expected {expected_group}\n"
+                    f"  Description: {description}\n"
+                    f"  Fix: sudo chown {actual_owner}:{expected_group} {path}"
+                )
+
+            if not errors or not any(path in err for err in errors):
+                print(f"✓ {path}: {actual_mode} {actual_owner}:{actual_group} - OK")
+
+        except Exception as e:
+            errors.append(f"ERROR: Failed to check {path}: {str(e)}")
+
+    if errors:
+        print("\n" + "="*80)
+        print("DIRECTORY PERMISSION ERRORS DETECTED:")
+        print("="*80)
+        for error in errors:
+            print(error)
+        print("="*80)
+        sys.exit(1)
+
+    print("\nAll directory permissions are correct.")
+
+
 def main():
     parser = argparse.ArgumentParser(
         prog="Supabase Postgres Artifact Permissions Checker",
@@ -258,6 +376,9 @@ def main():
     # Check if postgresql.service is using a read-only mount for /etc
     check_postgresql_mount()
 
+    # Check directory permissions for security-critical paths
+    check_directory_permissions()
+
 
 if __name__ == "__main__":
     main()
diff --git a/ansible/tasks/setup-postgres.yml b/ansible/tasks/setup-postgres.yml
index 916116c7..d97f351c 100644
--- a/ansible/tasks/setup-postgres.yml
+++ b/ansible/tasks/setup-postgres.yml
@@ -118,6 +118,14 @@
       loop_control:
         loop_var: 'pg_dir_item'
 
+    - name: Set /var/lib/postgresql to 0755 for nix-profile symlink traversal
+      ansible.builtin.file:
+        group: 'postgres'
+        mode: '0755'
+        owner: 'postgres'
+        path: '/var/lib/postgresql'
+        state: 'directory'
+
     - name: Allow adminapi to write custom config
       ansible.builtin.file:
         group: 'postgres'

root ownership analysis

Evidence from Distribution Survey:

Distribution	Binary/Symlink	Ownership	Type
Debian Bookworm	/usr/bin/psql	root:root	symlink
Ubuntu 24.04	/usr/bin/psql	root:root	symlink
Alpine Linux	/usr/bin/psql	root:root	symlink
Rocky Linux 9	/usr/bin/psql	root:root	binary
Ubuntu 24.04	/usr/bin/ls	root:root	-
Ubuntu 24.04	/usr/bin/bash	root:root	-

FHS (Filesystem Hierarchy Standard) Requirements:

1. /usr/bin must be shareable, read-only data
2. /usr should be mountable as read-only (can't do this if user-owned files exist)
3. System binaries are managed by root, not application users

Why postgres:postgres Was Wrong:

1. Violates FHS - User-owned files in system directories
2. Breaks read-only mounts - Can't mount /usr as read-only with user-owned files
3. Security issue - If postgres user is compromised, attacker could modify system-wide binaries
4. Non-standard - No major distribution does this

Why root:root Is Correct:

1. Standard practice - ALL distributions use root:root for /usr/bin contents
2. FHS compliant - Follows Linux filesystem standards
3. Security - System binaries can't be modified by application users
4. Read-only mountable - Allows proper /usr mount options

This Commit Was Correct:

     - name: Create symlinks for Nix files into /usr/bin
       ansible.builtin.file:
-        group: 'postgres'
-        owner: 'postgres'
+        group: 'root'
+        owner: 'root'

Verdict: Keep this change. It fixes a security and compliance issue.

The symlink ownership doesn't affect functionality (symlinks are always traversable regardless of ownership), but having user-owned files in /usr/bin is:

• Against FHS standards
• A security risk
• Prevents read-only /usr mounts
• Non-standard across all distributions

[hunleyd]

1. my bad. sorry @samrose
2. i don't think recurse: true # Also sets parent directory to 0750! is true. however, the file module acts like mkdir -p and creates any missing path from / to the path specified, so if the parent dir isn't there, it'll get created with the stated perms.

[samrose]

@hunleyd no worries. Wanted to dig into rationale behind it, since many times some of these things hadn't been modified for a long time. I am going to try and do more analysis of system permissions like this soon, and document it somewhere + incorporate into the tests we created on this PR