Processing stalled on flatnode install of nominatim-docker

[ebergerson]

I am trying to set up a local-only docker based nominatim server for Europe. I am running docker on a 64Gb mac running Tahoe 26.3. I am using an external hard drive for the PBF, PostgreSql, and flatnodes. I am installing and running nominatim into docker using this command:

docker run -it -e PBF_PATH=/data/europe.pbf \
-e NOMINATIM_FLATNODE_FILE=/flatnodes/flatnodes.bin\
-v /Volumes/Work01/Gis/pbfs/europe.pbf:/data/europe.pbf\
-v /Volumes/Work01/Gis/nominatim-data:/var/lib/postgresql/16/main\
-v /Volumes/Work01/Gis/nominatim-flatnodes:/flatnodes\
-p 8080:8080\
--name nominatim  mediagis/nominatim:5.2

Everything ran well up to a point.

I was closely monitoring memory on my local SSD and on the external hard drive. The flatnodes.bin file grew to be 108.8 GB, the size of my local SSD did not change, and the nominatim-data folder looked like a normal database install. This assured me that the configuration for using the external HD was working and I believed that the flatnodes option was working as opposed to putting all of the data into the database.

The processing ran for about two hours and has been stopped for over a day on this:

$ docker run -it -e PBF_PATH=/data/europe.pbf -e NOMINATIM_FLATNODE_FILE=/flatnodes/flatnodes.bin -v /Volumes/Work01/Gis/pbfs/europe.pbf:/data/europe.pbf -v /Volumes/Work01/Gis/nominatim-data:/var/lib/postgresql/16/main -v /Volumes/Work01/Gis/nominatim-flatnodes:/flatnodes -p 8080:8080 --name nominatim  mediagis/nominatim:5.2
+ tailpid=0
+ replicationpid=0
+ GUNICORN_PID_FILE=/tmp/gunicorn.pid
+ export PYTHONUNBUFFERED=1
+ PYTHONUNBUFFERED=1
+ trap stopServices SIGTERM TERM INT
+ /app/config.sh
+ id nominatim
+ useradd -m -p qaIACxO6wMR3 nominatim
+ IMPORT_FINISHED=/var/lib/postgresql/16/main/import-finished
+ '[' '!' -f /var/lib/postgresql/16/main/import-finished ']'
+ /app/init.sh
+ OSMFILE=/nominatim/data.osm.pbf
+ CURL=("curl" "-L" "-A" "${USER_AGENT}" "--fail-with-body")
+ SCP='sshpass -p DMg5bmLPY7npHL2Q scp -o StrictHostKeyChecking=no u355874-sub1@u355874-sub1.your-storagebox.de'
+ '[' -z '' ']'
++ nproc
+ THREADS=10
+ '[' '' = true ']'
+ '[' -f '' ']'
+ echo 'Skipping optional Wikipedia importance import'
Skipping optional Wikipedia importance import
+ '[' '' = true ']'
+ '[' -f '' ']'
+ echo 'Skipping optional Wikipedia secondary importance import'
Skipping optional Wikipedia secondary importance import
+ '[' '' = true ']'
+ '[' -f '' ']'
+ echo 'Skipping optional GB postcode import'
Skipping optional GB postcode import
+ '[' '' = true ']'
+ '[' -f '' ']'
+ echo 'Skipping optional US postcode import'
Skipping optional US postcode import
+ '[' '' = true ']'
+ '[' -f '' ']'
+ echo 'Skipping optional Tiger addresses import'
Skipping optional Tiger addresses import
+ '[' '' '!=' '' ']'
+ '[' /data/europe.pbf '!=' '' ']'
+ echo Reading OSM extract from /data/europe.pbf
Reading OSM extract from /data/europe.pbf
+ OSMFILE=/data/europe.pbf
+ '[' '!' -f /var/lib/postgresql/16/main/PG_VERSION ']'
+ chown postgres:postgres /var/lib/postgresql/16/main
+ sudo -u postgres /usr/lib/postgresql/16/bin/initdb -D /var/lib/postgresql/16/main
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/16/main ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/lib/postgresql/16/bin/pg_ctl -D /var/lib/postgresql/16/main -l logfile start

+ cp /etc/postgresql/16/main/conf.d/postgres-import.conf.disabled /etc/postgresql/16/main/conf.d/postgres-import.conf
+ sudo service postgresql start
 * Starting PostgreSQL 16 database server                                                                                                                                          [ OK ] 
+ sudo -E -u postgres psql postgres -tAc 'SELECT 1 FROM pg_roles WHERE rolname='\''nominatim'\'''
+ grep -q 1
+ sudo -E -u postgres createuser -s nominatim
+ sudo -E -u postgres psql postgres -tAc 'SELECT 1 FROM pg_roles WHERE rolname='\''www-data'\'''
+ grep -q 1
+ sudo -E -u postgres createuser -SDR www-data
+ sudo -E -u postgres psql postgres -tAc 'ALTER USER nominatim WITH ENCRYPTED PASSWORD '\''<redacted>'\'''
ALTER ROLE
+ sudo -E -u postgres psql postgres -tAc 'ALTER USER "www-data" WITH ENCRYPTED PASSWORD '\''<redacted>'\'''
ALTER ROLE
+ sudo -E -u postgres psql postgres -c 'DROP DATABASE IF EXISTS nominatim'
NOTICE:  database "nominatim" does not exist, skipping
DROP DATABASE
+ chown -R nominatim:nominatim /nominatim
+ cd /nominatim
+ '[' '' = true ']'
+ sudo -E -u nominatim nominatim import --osm-file /data/europe.pbf --threads 10
2026-02-26 21:19:58: Using project directory: /nominatim
2026-02-26 21:19:58: Creating database
2026-02-26 21:19:59: Setting up country tables
2026-02-26 21:20:01: Importing OSM data file
2026-02-26 21:20:01  osm2pgsql version 1.11.0
2026-02-26 21:20:01  Database version: 16.11 (Ubuntu 16.11-0ubuntu0.24.04.1)
2026-02-26 21:20:01  PostGIS version: 3.4
2026-02-26 21:20:01  Storing properties to table '"public"."osm2pgsql_properties"'.
Processing: Node(3712889k 706.3k/s) Way(502k 0.04k/s) Relation(0 0.0/s)

When I look at the Activity Monitor on the mac, there is very little pressure on the machine. Only two cores are busy, and they are not that busy and there is almost no pressure on the memory.

When I look at Docker, the container CPU usage is very low (<1% / 1000% - 10 CPUs available), but the Container memory usage is red and close to what appears the max (7.03GB / 7.47GB).

Am I right in concluding that the process is actually still working but at a crawl because the processing is probably all tied up in swapping inside the container?

Is the solution to increase the memory available to the container?

If so, what should I make it and how would I do that in the arguments to the docker command?

If not, does anyone know what is going on?

Thank you.

[mtmail]

The last line indicates osm2psql is running. It's single threaded and only uses one CPU core. The "0.04k/s" is very slow indeed. OSM ways are properties (key-value pairs) plus a list of nodes. osm2pgsql looks up the position (coordinates) of each node in the flatnode file. And other processing of the actual properties of course but that's similar to nodes. I suspect the external harddrive is too slow. It needs to be able to handle a lot of random reads. Is that a SSD/NvME drive? Is the cable fast (USB protocol version, in gigabytes/second). A tool like iotop might shine some light on what the throughput is. MacOS's Activity Monitor has a 'Disk' tab.