Skip to content

Commit 18e51d2

Browse files
enkoclaude
andcommitted
fix(database): Use RAM node cache instead of flat-nodes for OSM import
flat-nodes failed with "No space left on device": its file is sized by the global max OSM node ID (~12.5B x 8 bytes ~= 96GB), not by region size, so even a single-country extract needs ~96GB free — more than the import volume has. Switch to an in-memory node cache (--cache, default 8000MB, overridable via OSM_CACHE_MB). The host has ample RAM, so this is simpler and faster than flat-nodes and still keeps way resolution off the DB middle tables. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 70336e7 commit 18e51d2

1 file changed

Lines changed: 11 additions & 9 deletions

File tree

scripts/osm-import/import-region.sh

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -180,23 +180,25 @@ run_osm2pgsql() {
180180
psql "$DATABASE_URL" -q -c "DROP TABLE IF EXISTS public.osm_addresses_staging CASCADE;"
181181

182182
# Run osm2pgsql with flex output.
183-
# --flat-nodes stores node coords in a single file instead of DB middle
184-
# tables, turning per-way random DB lookups into fast sequential file reads.
185-
# Essential for full-country imports (100x faster way processing).
186-
# --cache 0 because flat-nodes replaces the in-RAM node cache.
183+
# --cache keeps node coords in RAM so way geometries resolve without random
184+
# DB lookups into the middle tables (the cause of ~0.7k/s way processing).
185+
# Sized to hold a full country's nodes (~7GB for Germany's ~430M nodes);
186+
# override via OSM_CACHE_MB for larger regions or tighter hosts.
187+
#
188+
# NB: --flat-nodes is the usual full-planet alternative, but its file is
189+
# sized by the *global* max node ID (~96GB), not by region size, so it
190+
# needs far more disk than a single-country extract suggests. With ample
191+
# RAM the in-memory cache is simpler and faster.
192+
local cache_mb="${OSM_CACHE_MB:-8000}"
187193
osm2pgsql \
188194
-d "$DATABASE_URL" \
189195
-O flex \
190196
-S "$SCRIPT_DIR/address-import.lua" \
191197
--slim \
192198
--drop \
193-
--cache 0 \
194-
--flat-nodes "$DATA_DIR/flat-nodes.bin" \
199+
--cache "$cache_mb" \
195200
"$pbf_file"
196201

197-
# --drop wipes middle tables but leaves the flat-nodes file behind
198-
rm -f "$DATA_DIR/flat-nodes.bin"
199-
200202
log_success "osm2pgsql import completed"
201203
}
202204

0 commit comments

Comments
 (0)