You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<p>Since version 0.6.98, the database can be cloned from an exisiting instance rather than created from scratch. This only takes 4 to 8 hours in comparison to 24 to 48 hours for an update from the planet file. Note that this feature is still rather experimental - please report any problems by eMail to me (roland.olbricht at gmx.de). If you don't want the entire planet or prefer a manually planet import for some other reason, use the <ahref="#startup">manual import</a> instead.</p>
52
52
53
53
<p>Download a clone of the database at <em>overpass-api.de</em> with the command:</p>
<p>If you want meta data, use <em>--meta=yes</em> instead of <em>--meta=no</em>. This downloads about 15 GB (25 GB with meta data) in several compressed files and uncompresses them to a ready-to-use database.</p>
58
58
59
59
<p>Now you can proceed with <ahref="#minute_updates">minute updates</a>.</p>
<p>The standard use case is to set up the database with the whole planet data and including meta data. If you haven't downloaded an <ahref="http://wiki.openstreetmap.org/wiki/Planet.osm">OSM XML planet file</a> yet, you can fetch one for example with:</p>
<p>This file has a size of more than 20 GB. Thus, depending on your internet connection, it may take between 4 hours (fastest possible) and 22 hours (with 2 MBit) to download the file. If you are not working on your local machine, you may want the download to continue even if you logout. Use <em>nohup</em> for this:</p>
<p>(without <em>--meta</em> if you have not processed meta data)</p>
87
87
<p>The dispatcher has been successfully started if you find a line "<em>Dispatcher just started.</em>" with correct date (in UTC) in the file <em>transactions.log</em> in the database directory.</p>
<p>This should start to fill the directory <em>"../minute-diffs/"</em> with subdirectories which have three digits as name and finally contain files ending in <em>osc.gz</em> and <em>state.txt</em>.</p>
<p>(with <em>--meta=no</em> instead if you have not processed meta data)</p>
94
-
<p>These commands don't make sense without <em>nohup</em>, because the programs become daemons and never terminate. Once again, you need to replace parameters: you always need to replace <em>id</em> by the replicate id to start from. If you have obtained your database by cloning, you find the replicate id in the file <em>replicate_id</em> in the database directory. If you have imported the database from an OSM file, search on <ahref="http://planet.openstreetmap.org/minute-replicate/">http://planet.openstreetmap.org/minute-replicate/</a> with your browser for the last replication diff that has been created before the planet creation date.</p>
94
+
<p>These commands don't make sense without <em>nohup</em>, because the programs become daemons and never terminate. Once again, you need to replace parameters: you always need to replace <em>id</em> by the replicate id to start from. If you have obtained your database by cloning, you find the replicate id in the file <em>replicate_id</em> in the database directory. If you have imported the database from an OSM file, search on <ahref="http://planet.openstreetmap.org/replication/minute/">http://planet.openstreetmap.org/replication/minute/</a> with your browser for the last replication diff that has been created before the planet creation date.</p>
95
95
96
-
<p>The other parameters need only to be adapted if you have chosen a different directory in a previous step: <em>"../db/"</em> is the directory of the database, <em>"http://planet.openstreetmap.org/minute-replicate/"</em> is the replicate diff's remote source, and <em>"../minute-diffs/"</em> is the directory where the minute diffs are stored until they have been applied.</p>
96
+
<p>The other parameters need only to be adapted if you have chosen a different directory in a previous step: <em>"../db/"</em> is the directory of the database, <em>"http://planet.openstreetmap.org/replication/minute/"</em> is the replicate diff's remote source, and <em>"../minute-diffs/"</em> is the directory where the minute diffs are stored until they have been applied.</p>
97
97
98
98
<p>Congratulations! Now you have a database mirror that can serve the entire world and is always only a few minutes behind the OSM main database. We can now startup the additional modules:</p>
99
99
@@ -109,26 +109,26 @@ <h2><a name="apache">Attach to Apache</a></h2>
<p>The essential part is to replace all occurences of the path <em>/path/to/osm-3s_v0.7.3/</em> to the real pathes. This configuration file tells Apache to serve the html files from the directory <em>/path/to/osm-3s_v0.7.3/html/</em> and to call programs in <em>/path/to/osm-3s_v0.7.3/cgi-bin/</em> by CGI. The <em>ScriptAlias</em> makes them visible externally as <em>/api/</em> instead of <em>/cgi-bin/</em>. For the remaining options, please look into the <ahref="http://httpd.apache.org/">Apache documentation</a>.</p>
122
+
<p>The essential part is to replace all occurences of the path <em>/path/to/osm-3s_v0.7.52/</em> to the real pathes. This configuration file tells Apache to serve the html files from the directory <em>/path/to/osm-3s_v0.7.52/html/</em> and to call programs in <em>/path/to/osm-3s_v0.7.52/cgi-bin/</em> by CGI. The <em>ScriptAlias</em> makes them visible externally as <em>/api/</em> instead of <em>/cgi-bin/</em>. For the remaining options, please look into the <ahref="http://httpd.apache.org/">Apache documentation</a>.</p>
123
123
124
124
<p>You need to check whether the involved directories and their parent directories have sufficient permissions for all users, because otherwise Apache (with its proxy user <em>www-data</em>) cannot access them:</p>
125
125
<pre>chmod 755 /path
126
126
chmod 755 /path/to
127
-
chmod 755 /path/to/osm-3s_v0.7.3
128
-
chmod 755 /path/to/osm-3s_v0.7.3/html
129
-
chmod 755 /path/to/osm-3s_v0.7.3/bin
130
-
chmod 755 /path/to/osm-3s_v0.7.3/cgi-bin
131
-
chmod 755 /path/to/osm-3s_v0.7.3/db</pre>
127
+
chmod 755 /path/to/osm-3s_v0.7.52
128
+
chmod 755 /path/to/osm-3s_v0.7.52/html
129
+
chmod 755 /path/to/osm-3s_v0.7.52/bin
130
+
chmod 755 /path/to/osm-3s_v0.7.52/cgi-bin
131
+
chmod 755 /path/to/osm-3s_v0.7.52/db</pre>
132
132
<p>Some directories are added later for some of the optional modules.</p>
133
133
134
134
<p>You can now (re-)start Apache to let the updated configuration come into effect:</p>
<p>The dispatcher has been successfully started if you find a line "<em>Dispatcher just started.</em>" with correct date (in UTC) in the file <em>transactions.log</em> in the database directory.</p>
156
156
157
157
<p>The third step then is to start the rule batch processor as a daemon:</p>
<p>You need to choose a directory where you put the executable files. You can later move them to a different directory. But the default choice of the installation program <em>automake</em>, /usr/bin, requires root permissions, although no root permissions are really necessary to run the program. I suggest the parent directory of the source code directory: this can be reached with "`pwd`/../". To configure this output directory and detect necessary adaptions of your system, run in the <em>build</em> subdirectory:</p>
<p>Since version 0.6.98, the database can be cloned from an exisiting instance rather than created from scratch. This only takes 4 to 8 hours in comparison to 24 to 48 hours for an update from the planet file. Note that this feature is still rather experimental - please report any problems by eMail to me (roland.olbricht at gmx.de). If you don't want the entire planet or prefer a manually planet import for some other reason, use the <ahref="#startup">manual import</a> instead.</p>
84
84
85
85
<p>Download a clone of the database at <em>overpass-api.de</em> with the command:</p>
<p>If you want meta data, use <em>--meta=yes</em> instead of <em>--meta=no</em>. This downloads about 15 GB (25 GB with meta data) in several compressed files and uncompresses them to a ready-to-use database.</p>
90
90
91
91
<p>Now you can proceed with <ahref="#minute_updates">minute updates</a>.</p>
<p>The standard use case is to set up the database with the whole planet data and including meta data. If you haven't downloaded an <ahref="http://wiki.openstreetmap.org/wiki/Planet.osm">OSM XML planet file</a> yet, you can fetch one for example with:</p>
<p>This file has a size of more than 20 GB. Thus, depending on your internet connection, it may take between 4 hours (fastest possible) and 22 hours (with 2 MBit) to download the file. If you are not working on your local machine, you may want the download to continue even if you logout. Use <em>nohup</em> for this:</p>
<p>These commands don't make sense without <em>nohup</em>, because the programs become daemons and never terminate. Once again, you need to replace parameters: you always need to replace <em>id</em> by the replicate id to start from. Search on <ahref="http://planet.openstreetmap.org/replication/minute/">http://planet.openstreetmap.org/replication/minute/</a> with your browser for the last replication diff that has been created before the planet creation date.</p>
<p>The dispatcher coordinates backend and frontend. Thus, there is no data associated with the dispatcher, but several overhead files, the process <em>dispatcher</em>, and the logfile <em>transactions.log</em>.</p>
48
48
49
-
<p>Dispatcher organizes its communication with the backend and frontend by three different means: Files ending in <em>.lock</em> are managed by the dispatcher and contain a pid. Having these as physical files ensures that after a crash the data suffices to recover correctly. The second way of communication is shared memory, in particular the shared memory file <em>/osm3s_v0.7.0_osm_base</em> (In Linux, you can access this file via <em>/dev/shm/osm3s_v0.7.0_osm_base</em>). Dispatcher announces in this directory the database location, and the frontend and backend can report their status here and request locks. The third way of communication is the Unix domain socket <em>osm3s_v0.7.0_osm_base</em> in the database directory. Each reading or writing process holds during its whole lifetime a permanent connection as a client to the dispatcher acting as server. Thus, if a process is killed, the dispatcher can notice this because the connection is suddenly lost.</p>
49
+
<p>Dispatcher organizes its communication with the backend and frontend by three different means: Files ending in <em>.lock</em> are managed by the dispatcher and contain a pid. Having these as physical files ensures that after a crash the data suffices to recover correctly. The second way of communication is shared memory, in particular the shared memory file <em>/osm3s_v0.7.52_osm_base</em> (In Linux, you can access this file via <em>/dev/shm/osm3s_v0.7.52_osm_base</em>). Dispatcher announces in this directory the database location, and the frontend and backend can report their status here and request locks. The third way of communication is the Unix domain socket <em>osm3s_v0.7.52_osm_base</em> in the database directory. Each reading or writing process holds during its whole lifetime a permanent connection as a client to the dispatcher acting as server. Thus, if a process is killed, the dispatcher can notice this because the connection is suddenly lost.</p>
50
50
51
51
<p>The other overhead files are a file ending in <em>.shadow</em> for <ahref="#database_content">each file in the database</a> ending in <em>.bin</em>, <em>.map</em>, or <em>.idx</em>. These files enable transactionality: all index information for the writing process is kept in the various <em>.shadow</em> files, and the plain <em>.idx</em> files are at the same time valid and used for all processes that request a read lock during the write operation. If the system crashes, one could simply discard the <em>.shadow</em> files, and the remaining database is still consistent with the original <em>.idx</em> files.</p>
0 commit comments