2025-10-09 postgres - master branch - PR 1 of 2

Updated service definition to explicitly pull Postgres v18 and remap
the persistent store to be v18-friendly.

My assumptions are:

1. The adjusted service definition is appropriate for **new** users
   installing the Postgres container for the first time.

2. Existing users will be running v17 (or earlier) and will not bang
   into the problem reported in #808 until they do a `pull` from
   DockerHub. There is no easy way to avoid the migration problem.
   Users will just have to read the Wiki.

Wiki updated to explain how to restore database access by reverting to
v17, then performing a controlled migration to v18.

Signed-off-by: Phill Kelley <[email protected]>

Author: Paraphraser

.templates/postgres/service.yml

@@ -1,6 +1,7 @@
 postgres:
   container_name: postgres
-  image: postgres
+  image: postgres:18
+  x-comment: Please READ the Postgres document in the IOTstack Wiki BEFORE changing the "18"
   restart: unless-stopped
   environment:
     - TZ=${TZ:-Etc/UTC}
@@ -10,6 +11,5 @@ postgres:
   ports:
     - "5432:5432"
   volumes:
-    - ./volumes/postgres/data:/var/lib/postgresql/data
+    - ./volumes/postgres/data:/var/lib/postgresql
     - ./volumes/postgres/db_backup:/backup
-

docs/Containers/PostgreSQL.md

@@ -79,6 +79,223 @@ Notes:
 	$ docker-compose up -d postgresql
 	```
 
+## Postgres v17 to v18 migration  { #v1718migration }
+
+Prior to Postgres v17, the external path to the persistent store holding the database file structures was at:
+
+```
+~/IOTstack/volumes/postgres/data
+```
+
+Inside the container, that mapped to the following path, which is also where your database files were located:
+
+```
+/var/lib/postgresql/data
+```
+
+Prior to October 2025, the service definition for Postgres provided with IOTstack contained:
+
+``` yaml
+    image: postgres
+```
+
+That is a synonym for:
+
+``` yaml
+    image: postgres:latest
+```
+
+Postgres v18 was released on September 26, 2025. If you are running Postgres v17 and "pull" the latest image from DockerHub after that date, you will get v18. When you try to instantiate the v18 image, you will get this error:
+
+```
+Error response from daemon: failed to create task for container: failed
+to create shim task: OCI runtime create failed: runc create failed:
+unable to start container process: error during container init: error
+mounting "/home/pi/IOTstack/volumes/postgres/data" to rootfs at
+"/var/lib/postgresql/data": change mount propagation through procfd:
+open o_path procfd: open
+/var/lib/docker/overlay2/«hex string»/merged/var/lib/postgresql/data:
+no such file or directory: unknown
+```
+
+### Fixing the immediate problem
+
+To resolve that error and restore access to your database, you need to revert to Postgres v17. Proceed like this:
+
+1. Move into the correct directory:
+
+	``` console
+	$ cd ~/IOTstack
+	```
+
+2. Stop the (broken) container:
+
+	``` console
+	$ docker compose down postgresql
+	```
+
+3. Use your favourite text editor to open `docker-compose.yml`.
+
+	Find the `image` clause and change it to be:
+
+	``` yaml
+	image: postgres:17
+	```
+
+	Save your work.
+
+4. Start the container again:
+
+	``` console
+	$ docker compose up -d postgresql
+	```
+
+	The container should start normally and you regain access to your databases.
+
+### Anonymous volume mounts
+
+One side-effect of upgrading a v17 directory structure to v18 is that Docker creates an anonymous volume mount.
+
+Anonymous volume mounts are the result of:
+
+1. A `VOLUME` declaration in the Dockerfile used to create the image; and
+2. Launching the container *without* providing a corresponding volume or bind mount mapping.
+
+As such, and depending on the age of your system and your propensity to experiment with Docker containers, you may find that you have multiple anonymous volume mounts of which you were previously unaware.
+
+Anonymous volume mounts persist on your system until you remove them. They are of use to the containers that created them while those containers are running but they are not re-attached when the container is re-created. They simply occupy disk space, indefinately, for no benefit.
+
+Removing an anonymous volume mount is a two-step process:
+
+1. List the anonymous volume mounts:
+
+	``` console
+	$ docker volume ls --filter "dangling=true" 
+	```
+
+	You can expect to see something like this:
+
+	```
+	DRIVER    VOLUME NAME
+	local     6c2862fdbc36187c0e858be7d8ebb81c653b796f2158f450136a00e7c8eca62d
+	```
+
+	A 64-character hexadecimal string in the `VOLUME NAME` column is the signature of an anonymous volume mount.
+
+	Note:
+
+	* The list may also contain rows where the `VOLUME NAME` is not a 64-character hexadecimal string. Those are *named* volume mounts. By convention, IOTstack does not use named volume mounts so those *may* be the results of previous experiments with Docker containers. If you do not recognise a named volume mount then you can consider removing it along with any anonymous volume mounts.
+
+2. For each anonymous volume mount, remove it by passing the 64-character hexadecimal string. For example:
+
+		``` console
+		$ docker volume rm 6c2862fdbc36187c0e858be7d8ebb81c653b796f2158f450136a00e7c8eca62d
+		```
+
+### Migrating to v18
+
+When you are ready to migrate to Postgres v18, proceed like this:
+
+1. Move into the correct directory:
+
+	``` console
+	$ cd ~/IOTstack
+	```
+
+2. Take a backup of your databases:
+
+	``` console
+	$ docker exec postgres bash -c 'pg_dumpall -U $POSTGRES_USER | gzip >/backup/postgres_backup.sql.gz'
+	```
+
+3. Stop the v17 Postgres container:
+
+	```
+	$ docker compose down postgres
+	```
+
+4. Use your favourite text editor to open `docker-compose.yml` and make the following changes:
+
+	1. Find the `image` clause and change it to be:
+
+		``` yaml
+		image: postgres:18
+		```
+
+	2. The first two lines of the `volumes:` clause look like this:
+
+		``` yaml
+		volumes:
+		  - ./volumes/postgres/data:/var/lib/postgresql/data
+		```
+
+		Change the second line to look like this:
+
+		``` yaml
+		  - ./volumes/postgres/data:/var/lib/postgresql
+		```
+
+	3. Save your work.
+
+4. Carefully (repeat **carefully**) remove the `data` portion of the Postgres container's persistent store:
+
+	``` console
+	$ sudo rm -rf ./volumes/postgres/data
+	```
+
+	Key point:
+
+	* Do **not** remove `./volumes/postgres` because that will also destroy your backup.
+
+5. Start the container:
+
+	``` console
+	$ docker compose up -d postgres
+	```
+
+	This will instantiate v18 and create a new, empty database structure.
+
+6. Restore your data:
+
+	```
+	$ docker exec postgres bash -c 'gunzip -c /backup/postgres_backup.sql.gz | psql -U $POSTGRES_USER postgres'
+	```
+
+After the migration, the external path to the persistent store remains unchanged at:
+
+```
+~/IOTstack/volumes/postgres/data
+```
+
+Within the container, that path maps to:
+
+```
+/var/lib/postgresql
+```
+
+However, that path is not where your database files are located. Instead, the files are at the internal path:
+
+```
+/var/lib/postgresql/18/docker
+```
+
+which maps to the external path:
+
+```
+~/IOTstack/volumes/postgres/data/18/docker
+```
+
+### Postgres v19 and beyond
+
+I recommend leaving `image: postgres:18` in place. That way, you are unlikely to be surprised when Postgres is upgraded to v19. It *may* be that v19 will silently handle the v18-to-v19 upgrade. On the other hand, you *may* need to adopt the approach shown above:
+
+* Take a backup of your data.
+* Down the container.
+* Hand-edit your compose file to `image: postgres:19`
+* Remove the `data` portion of the persistent store.
+* Up the container.
+* Restore your data.
+
 ## Getting a clean slate { #cleanSlate }
 
 If you need to start over, proceed like this: