Skip to content

Commit 96adfb1

Browse files
rustprooflabsrustprooflabs
authored
Merge pull request #44 from rustprooflabs/docker
Migrate Docker components from legacy PgOSM project
2 parents ad94193 + dfe428b commit 96adfb1

File tree

4 files changed

+222
-0
lines changed

4 files changed

+222
-0
lines changed

‎Dockerfile‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
FROM postgis/postgis:13-3.1
2+
3+
LABEL maintainer="PgOSM-Flex - https://github.com/rustprooflabs/pgosm-flex"
4+
5+
RUN apt-get update \
6+
&& apt-get install -y --no-install-recommends \
7+
sqitch wget ca-certificates \
8+
git make cmake g++ \
9+
libboost-dev libboost-system-dev \
10+
libboost-filesystem-dev libexpat1-dev zlib1g-dev \
11+
libbz2-dev libpq-dev libproj-dev lua5.2 liblua5.2-dev \
12+
lua-dkjson \
13+
&& rm -rf /var/lib/apt/lists/*
14+
15+
16+
WORKDIR /tmp
17+
RUN git clone git://github.com/openstreetmap/osm2pgsql.git \
18+
&& mkdir osm2pgsql/build \
19+
&& cd osm2pgsql/build \
20+
&& cmake .. \
21+
&& make \
22+
&& make install \
23+
&& apt remove -y \
24+
git make cmake g++ \
25+
libboost-dev libboost-system-dev \
26+
libboost-filesystem-dev libexpat1-dev zlib1g-dev \
27+
libbz2-dev libpq-dev libproj-dev \
28+
&& apt autoremove -y \
29+
&& cd /tmp && rm -r /tmp/osm2pgsql
30+
31+
32+
WORKDIR /app
33+
COPY . ./

‎README-DOCKER.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# PgOSM Docker
2+
3+
Notes covering Docker and generating image for Docker Hub.
4+
5+
Uses [main Postgres image](https://hub.docker.com/_/postgres/) via the [main PostGIS image](https://hub.docker.com/r/postgis/postgis) as starting point, see that
6+
repo for full instructions on using the core Postgres functionality.
7+
8+
Build latest.
9+
10+
```
11+
docker build -t rustprooflabs/pgosm-flex .
12+
```
13+
14+
15+
Tag with Pg version.
16+
17+
```
18+
docker build -t rustprooflabs/pgosm-flex:pg13 .
19+
```
20+
21+
> Note: Update the Dockerfile to build with non-default Postgres/PostGIS version.
22+
23+
Push to Dockerhub
24+
25+
```
26+
docker push rustprooflabs/pgosm-flex:pg12
27+
docker push rustprooflabs/pgosm-flex:pg13
28+
docker push rustprooflabs/pgosm-flex:latest
29+
```

‎README.md‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -286,3 +286,58 @@ Currently only U.S. region drafted, more regions with local `maxspeed` are welco
286286
See [the README documenting using the QGIS styles](https://github.com/rustprooflabs/pgosm-flex/blob/main/db/qgis-style/README.md).
287287

288288

289+
290+
## PgOSM via Docker
291+
292+
293+
PgOSM-Flex can be deployed using the Docker image from [Docker Hub](https://hub.docker.com/r/rustprooflabs/pgosm-flex).
294+
295+
296+
Create folder for the output (``~/pgosm-data``),
297+
this stores the generated SQL file used to perform the PgOSM transformations and the
298+
output file from ``pg_dump`` containing the ``osm`` schema to load into a production database.
299+
The ``.osm.pbf`` file and associated ``md5``are saved here. Custom templates, and custom OSM file inputs can be stored here.
300+
301+
302+
```
303+
mkdir ~/pgosm-data
304+
```
305+
306+
Start the `pgosm` container to make PostgreSQL/PostGIS available. This command exposes Postgres inside Docker on port 5433 and establishes links to local directories.
307+
308+
```
309+
docker run --name pgosm -d \
310+
-v ~/pgosm-data:/app/output \
311+
-e POSTGRES_PASSWORD=mysecretpassword \
312+
-p 5433:5432 -d rustprooflabs/pgosm-flex
313+
```
314+
315+
316+
Run the PgOSM-flex processing. Using the Washington D.C. sub-region is great
317+
for testing, it runs fast even on the smallest hardware.
318+
319+
```
320+
docker exec -it \
321+
-e POSTGRES_PASSWORD=mysecretpassword -e POSTGRES_USER=postgres \
322+
pgosm bash docker/run_pgosm_flex.sh \
323+
north-america/us \
324+
district-of-columbia \
325+
400 \
326+
run-all
327+
```
328+
329+
330+
## Always download
331+
332+
To force the processing to remove existing files and re-download the latest PBF and MD5 files from Geofabrik, set the `PGOSM_ALWAYS_DOWNLOAD` env var when running the Docker
333+
container.
334+
335+
```
336+
docker run --name pgosm -d \
337+
-v ~/pgosm-data:/app/output \
338+
-e POSTGRES_PASSWORD=mysecretpassword \
339+
-e PGOSM_ALWAYS_DOWNLOAD=1 \
340+
-p 5433:5432 -d rustprooflabs/pgosm
341+
```
342+
343+
----

‎docker/run_pgosm_flex.sh‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
#!/bin/bash
2+
# Run with:
3+
# ./run_pgosm_flex.sh \
4+
# north-america/us \
5+
# district-of-columbia \
6+
# 4000 \
7+
# run-all
8+
#
9+
# $1 - Region - e.g. north-america/us
10+
# $2 - Subregion - e.g. district-of-columbia
11+
# $3 - Cache (mb) - e.g. 4000
12+
# $4 - Layers to load - must match flex-config/$4.lua and flex-config/$4.sql
13+
14+
BASE_PATH=/app/
15+
16+
OUT_PATH=/app/output/
17+
FLEX_PATH=/app/flex-config
18+
19+
# Naming scheme must match Geofabrik's for MD5 sums to validatate
20+
PBF_FILE=$OUT_PATH$2-latest.osm.pbf
21+
MD5_FILE=$OUT_PATH$2-latest.osm.pbf.md5
22+
23+
LOG_FILE=$OUT_PATH$2.log
24+
25+
echo "Monitor $LOG_FILE for progress..."
26+
echo "If paths setup as outlined in README.md, use:"
27+
echo " tail -f ~/pgosm-data/$2.log"
28+
29+
ALWAYS_DOWNLOAD=${PGOSM_ALWAYS_DOWNLOAD:-0}
30+
31+
32+
echo "" >> $LOG_FILE
33+
echo "---------------------------------" >> $LOG_FILE
34+
echo "Start PgOSM-Flex processing" >> $LOG_FILE
35+
echo "Region: $1" >> $LOG_FILE
36+
echo "Sub-Region: $2" >> $LOG_FILE
37+
echo "Cache: $3" >> $LOG_FILE
38+
echo "PgOSM Flex Style: $4" >> $LOG_FILE
39+
40+
41+
if [ $ALWAYS_DOWNLOAD == "1" ]; then
42+
echo 'Removing PBF and md5 files if exists...' >> $LOG_FILE
43+
BE_NICE = 'NOTE: Be nice to Geofabrik''s download server!'
44+
echo "$BE_NICE" >> $LOG_FILE
45+
echo "$BE_NICE"
46+
rm $PBF_FILE
47+
rm $MD5_FILE
48+
fi
49+
50+
if [ -f $PBF_FILE ]; then
51+
echo "$PBF_FILE exists. Not downloading." >> $LOG_FILE
52+
else
53+
echo "$PBF_FILE does not exist. Downloading..." >> $LOG_FILE
54+
wget https://download.geofabrik.de/$1/$2-latest.osm.pbf -O $PBF_FILE --quiet &>> $LOG_FILE
55+
fi
56+
57+
if [ -f $MD5_FILE ]; then
58+
echo "$MD5_FILE exists. Not downloading." >> $LOG_FILE
59+
else
60+
echo "$MD5_FILE does not exist. Downloading..." >> $LOG_FILE
61+
wget https://download.geofabrik.de/$1/$2-latest.osm.pbf.md5 -O $MD5_FILE --quiet &>> $LOG_FILE
62+
fi
63+
64+
65+
if cd $OUT_PATH && md5sum -c $MD5_FILE; then
66+
echo 'MD5 checksum validated' >> $LOG_FILE
67+
else
68+
ERR_MSG = 'ERROR - MD5 sum did not match. Try re-running with PGOSM_ALWAYS_DOWNLOAD=1'
69+
echo "$ERR_MSG" >> $LOG_FILE
70+
echo "$ERR_MSG"
71+
exit 1
72+
fi
73+
74+
cd $BASE_PATH
75+
76+
echo "Create empty pgosm database with extensions..." >> $LOG_FILE
77+
psql -U postgres -c "DROP DATABASE IF EXISTS pgosm;" >> $LOG_FILE
78+
psql -U postgres -c "CREATE DATABASE pgosm;" >> $LOG_FILE
79+
psql -U postgres -d pgosm -c "CREATE EXTENSION postgis;" >> $LOG_FILE
80+
psql -U postgres -d pgosm -c "CREATE SCHEMA osm;" >> $LOG_FILE
81+
82+
osm2pgsql --version >> $LOG_FILE
83+
84+
echo "Running osm2pgsql..." >> $LOG_FILE
85+
cd $FLEX_PATH
86+
osm2pgsql -U postgres --create --slim --drop \
87+
--cache $3 \
88+
--output=flex --style=./$4.lua \
89+
-d pgosm $PBF_FILE &>> $LOG_FILE
90+
91+
echo "Running post-processing SQL script..." >> $LOG_FILE
92+
psql -U postgres -d pgosm -f $4.sql >> $LOG_FILE
93+
94+
cd $BASE_PATH
95+
96+
echo "Running pg_dump..." >> $LOG_FILE
97+
pg_dump -U postgres -d pgosm \
98+
--schema=osm > /app/output/pgosm-flex-$2-$4.sql
99+
100+
101+
echo "PgOSM processing complete. Final output file: pgosm-$2-$4.sql" >> $LOG_FILE
102+
echo "PgOSM processing complete. Final output file: pgosm-$2-$4.sql"
103+
echo "If you followed the README.md it is at: ~/pgosm-data/pgosm-flex-$2-$4.sql"
104+
105+
exit 0

0 commit comments

Comments
 (0)