render_list multi threading is not working, it seems

[pkthogit]

Hi Team,

Its rather asking for help, presumably not a bug (not sure).

I have followed this page to configure postgres with OSM planet data and configured mod_tile and renderd.

I have a cloud VM with 350GB RAM and 46 threads to use for this exercise. Hence, I have updated the postgres configure parameters as mentioned below to take most out of available resources.

shared_buffers = 200GB
work_mem = 10GB
maintenance_work_mem = 40GB
effective_cache_size = 40GB
max_connections = 100

I was able to render the tiles successfully, however, my goal is to pre-generate all zoom level tiles and hence executed below command

render_list -m default -a -z 0 -Z 19 --num-threads=40

Till zoom level 13, it was too fast but later started getting slow, I know the number of tiles to be generated gets exponential as zoom level goes up.

So tried with just zoom level only 15 with setup, surprisingly, no major consumption of RAM and the process is too slow even with 40 threads.

render_list -m default -a -z 15 -Z 15 --num-threads=40

Just wondering if this threads usage is working fine or am I missing anything to make most out of available resources to generate tiles quickly? Any suggestions would be greatly appreciated.

-Prem

[hummeltech]

You'll probably want to use the --max-load option as well, something like --max-load 40 might help. Also, can you provide the contents of your renderd.conf file?

[pkthogit]

Thanks for the response @hummeltech . Here is the /etc/apache2/conf-available/renderd.conf file content for your reference. Please suggest.

<Directory /var/cache/renderd/tiles>

    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Require all granted

    #ModTileTileDir /var/cache/renderd/tiles
    ModTileTileDir /data/tile_cache/osm
    # You can manually configure each tile set with AddTileConfig or AddTileMimeConfig.
    # The first argument is the URL path relative to this virtual host
    # under which a tile set is served. The second argument specifies the
    # name of the tile set. This is used in the communication with renderd
    # and is the directory under which (meta)tiles are stored on disk.
    #
    # By default (AddTileConfig) mod_tile assumes you are serving png files, however,
    # mod_tile can also serve arbitrary other tile types such as javascript vector tiles,
    # assuming the backend render daemon can handle the file type.
    # To this purpose AddTileMimeConfig takes a 3rd agument, the file extension and it
    # will guess the correct mimetype from it. If the mime type is not set correctly automatically,
    # you need to use the configuration file route, where you can specify the mimetype and file extension
    # independently.
    #
    #AddTileConfig /folder/ TileSetName
    #AddTileMimeConfig /folder2/ TileSetName2 js

    # Alternatively (or in addition) you can load all the tile sets defined in the configuration file into this virtual host
    LoadTileConfigFile /etc/renderd.conf

    # Specify if mod_tile should keep tile delivery stats, which can be accessed from the URL /mod_tile
    # The default is On. As keeping stats needs to take a lock, this might have some performance impact,
    # but for nearly all intents and purposes this should be negligable and so it is safe to keep this turned on.
    ModTileEnableStats On

    # Turns on bulk mode. In bulk mode, mod_tile does not request any dirty tiles to be rerendered. Missing tiles
    # are always requested in the lowest priority. The default is Off.
    ModTileBulkMode Off

    # Timeout before giving up for a tile to be rendered
    ModTileRequestTimeout 3

    # Timeout before giving up for a tile to be rendered that is otherwise missing
    ModTileMissingRequestTimeout 10

    # If tile is out of date, don't re-render it if past this load threshold (users gets old tile)
    ModTileMaxLoadOld 2

    # If tile is missing, don't render it if past this load threshold (user gets 404 error)
    ModTileMaxLoadMissing 5

    # Socket where we connect to the rendering daemon
    ModTileRenderdSocketName /run/renderd/renderd.sock

    # Options controlling the cache proxy expiry headers. All values are in seconds.
    #
    # Caching is both important to reduce the load and bandwidth of the server, as
    # well as reduce the load time for the user. The site loads fastest if tiles can be
    # taken from the users browser cache and no round trip through the internet is needed.
    # With minutely or hourly updates, however there is a trade-off between cacheability
    # and freshness. As one can't predict the future, these are only heuristics, that
    # need tuning.
    # If there is a known update schedule such as only using weekly planet dumps to update the db,
    # this can also be taken into account through the constant PLANET_INTERVAL in render_config.h
    # but requires a recompile of mod_tile

    # The values in this sample configuration are not the same as the defaults
    # that apply if the config settings are left out. The defaults are more conservative
    # and disable most of the heuristics.

    # Caching is always a trade-off between being up to date and reducing server load or
    # client side latency and bandwidth requirements. Under some conditions, like poor
    # network conditions it might be more important to have good caching rather than the latest tiles.
    # Therefor the following config options allow to set a special hostheader for which the caching
    # behaviour is different to the normal heuristics
    #
    # The CacheExtended parameters overwrite all other caching parameters (including CacheDurationMax)
    # for tiles being requested via the hostname CacheExtendedHostname
    #
    #ModTileCacheExtendedHostname cache.tile.openstreetmap.org
    #ModTileCacheExtendedDuration 2592000

    # Upper bound on the length a tile will be set cacheable, which takes
    # precedence over other settings of cacheing
    ModTileCacheDurationMax 604800

    # Sets the time tiles can be cached for that are known to by outdated and have been
    # sent to renderd to be rerendered. This should be set to a value corresponding
    # roughly to how long it will take renderd to get through its queue. There is an additional
    # fuzz factor on top of this to not have all tiles expire at the same time
    ModTileCacheDurationDirty 900

    # Specify the minimum time mod_tile will set the cache expiry to for fresh tiles. There
    # is an additional fuzz factor of between 0 and 3 hours on top of this.
    ModTileCacheDurationMinimum 10800

    # Lower zoom levels are less likely to change noticeable, so these could be cached for longer
    # without users noticing much.
    # The heuristic offers three levels of zoom, Low, Medium and High, for which different minimum
    # cacheing times can be specified.

    #Specify the zoom level below  which Medium starts and the time in seconds for which they can be cached
    ModTileCacheDurationMediumZoom 13 86400

    #Specify the zoom level below which Low starts and the time in seconds for which they can be cached
    ModTileCacheDurationLowZoom 9 518400

    # A further heuristic to determine cacheing times is when was the last time a tile has changed.
    # If it hasn't changed for a while, it is less likely to change in the immediate future, so the
    # tiles can be cached for longer.
    # For example, if the factor is 0.20 and the tile hasn't changed in the last 5 days, it can be cached
    # for up to one day without having to re-validate.
    ModTileCacheLastModifiedFactor 0.20

    # Tile Throttling
    # Tile scrapers can often download large numbers of tiles and overly strain tileserver resources
    # mod_tile therefore offers the ability to automatically throttle requests from ip addresses that have
    # requested a lot of tiles.
    # The mechanism uses a token bucket approach to shape traffic. I.e. there is an initial pool of n tiles
    # per ip that can be requested arbitrarily fast. After that this pool gets filled up at a constant rate
    # The algorithm has two metrics. One based on overall tiles served to an ip address and a second one based on
    # the number of requests to renderd / tirex to render a new tile.

    # Overall enable or disable tile throttling
    ModTileEnableTileThrottling Off

    # Specify if you want to use the connecting IP for throtteling, or use the X-Forwarded-For header to determin the
    # 1 - use the client IP address, i.e. the first entry in the X-Forwarded-For list. This works through a cascade of proxies.
    #     However, as the X-Forwarded-For is written by the client this is open to manipulation and can be used to circumvent the throttling
    # 2 - use the last specified IP in the X-Forwarded-For list. If you know all requests come through a reverse proxy
    #     that adds an X-Forwarded-For header, you can trust this IP to be the IP the reverse proxy saw for the request
    ModTileEnableTileThrottlingXForward 0

    # Parameters (poolsize in tiles and topup rate in tiles per second) for throttling tile serving.
    ModTileThrottlingTiles 10000 1
    # Parameters (poolsize in tiles and topup rate in tiles per second) for throttling render requests.
    ModTileThrottlingRenders 128 0.2

</Directory>

[hummeltech]

Thanks @pkthogit, I meant your /etc/renderd.conf file, not your Apache configuration.

Also, please try using the --max-load option with render_list, otherwise it will sleep when it reaches a load average of the default value of 16 (your system should handle 40 without having to wait.)

[pkthogit]

Ooops, its my bad. Here is the config content. @hummeltech .

Just additional info, I have 10TB of SSD on mounted drive, presuming that I/O operation on mounted drive doesn't have any performance impact.

; BASIC AND SIMPLE CONFIGURATION:

[renderd]
stats_file=/run/renderd/renderd.stats
socketname=/run/renderd/renderd.sock
num_threads=10
tile_dir=/data/tile_cache/osm

[mapnik]
plugins_dir=/usr/lib/mapnik/3.1/input
font_dir=/usr/share/fonts/truetype
font_dir_recurse=true

; ADD YOUR LAYERS:
[default]
URI=/osm/
XML=/home/osm/openstreetmap-carto/style.xml
HOST=tile.maps.mydomain.com

Also while generating style.xml file, I found bellow warnings, just could be point of suspects.

carto project_osm.mml > style.xml

Warning: style/admin.mss:108:6 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:103:6 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:99:6 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:94:6 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:90:6 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:82:6 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:76:6 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:72:6 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:68:6 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:64:6 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:59:6 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:55:6 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:51:6 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:27:19 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:26:19 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:25:19 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:24:19 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:23:19 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:22:18 Styles do not match layer selector #admin-low-zoom.
Warning: style/admin.mss:18:6 Styles do not match layer selector #admin-low-zoom.
Warning: style/golf.mss:61:4 Styles do not match layer selector #text-point.
Warning: style/golf.mss:53:4 Styles do not match layer selector #amenity-points.

-Prem

[hummeltech]

@pkthogit , those warning messages are generally safe to ignore.

For your renderd.conf, I recommend you increase num_threads, I.E.:

[renderd]
stats_file=/run/renderd/renderd.stats
socketname=/run/renderd/renderd.sock
num_threads=40
tile_dir=/data/tile_cache/osm

Then, restart renderd, wait for it to start up and run render_list with the max-load option:

render_list --all --map default --min-zoom 0 --max-zoom 19 --num-threads 40 --max-load 40

It is going to take a very long time, but it will hopefully be able to fully utilize your system resources.

[pkthogit]

Thanks for the reply @hummeltech

Although I had made al suggested changes and kicked off the command for zooms 15 to 19. However, the process could not complete for 15th zoom level even after 2days of continuous process consuming most of the resources.

Postgres Database configuration:
shared_buffers = 200GB
work_mem = 10GB
maintenance_work_mem = 40GB
effective_cache_size = 40GB
max_connections = 100

Ubuntu machine with 350GB RAM, 46 threads and TBs of SSD storage

For your renderd.conf

[renderd]
stats_file=/run/renderd/renderd.stats
socketname=/run/renderd/renderd.sock
num_threads=40
tile_dir=/data/tile_cache/osm

rendered_list command being executed is:
render_list --all --map default --min-zoom 15 --max-zoom 19 --num-threads 40 --max-load 40

Crossed 2days of processing time, still 15th zoom level in-progress.

Any rough estimates of how much time ideally render_list command takes to process all the zoom levels from 0-19 with the above configuration?? Is it expected to see this behaviour of unending process or am I missing something?

-Prem

[SomeoneElseOSM]

On the original question - do you really want to generate tiles up to z19 for the entire planet? Do you have the disk space for that?

Before getting too bogged down with render_list, are you sure that your database has all the indices that your style needs (at some point in the past https://www.linuxbabe.com/ubuntu/openstreetmap-tile-server-ubuntu-20-04-osm was missing some, but I haven't checked recently).

[pkthogit]

Hi @SomeoneElseOSM , currently I have 10TB of SSD storage and I need to self host entire planet tiles on CDN (I know its too heavy process but trying to build the pipeline).

When it comes to Postgres database, I have built the spatial indices by using the below lines to make sure the queries are faster.

cd ~/src/openstreetmap-carto/
psql -d gis -f indexes.sql

Essentially, I wanted to replicate the tile service what https://tile.openstreetmap.org/{zoom}/{x}/{y}.png does without usage limit for my application.

Please correct me, if I have missed anything else. thank you.

-Prem

[hummeltech]

@pkthogit, If you are just wanting to replicate openstreetmap.org's tile services, you might want to take a look at their configuration to get an idea of how they do it:
https://github.com/openstreetmap/chef/tree/master/cookbooks/tile

I would recommend only pre-rendering the lower zoom levels and then just letting mod_tile render the higher levels on demand. You will usually not notice much of a delay.

[pnorman]

Zoom 19 tiles for the world are not practical to pre-render.

[pkthogit]

Thank you @hummeltech @SomeoneElseOSM and @pnorman for the advise.

Before we close this thread, one last question. As suggested, I could generate tiles till 15th zoom level and for the zoom 16-19, I thought of giving it a try generating tiles only for land area (taken countries data as basis) ignoring ocean and antarctica which comes 50% of overall planet area.

I could generate 2 degree by 2 degree bboxes (EPSG: 4326) and planned to loop through each bbox and run the render_list command. However, I bumped into strange issue as mentioned below.

render_list -a -m default -z 16 -Z 16 -x -180.01 -y -18.51 -X -177.99 -Y -16.49 -f --num-threads 40 --max-load 40 -f
Invalid range, x and y values must be >= 0

render_list -a -m default -z 16 -Z 16 -x -180.01,-18.51,-177.99,-16.49 -f --num-threads 40 --max-load 40
Invalid range, x and y values must be >= 0

As I understand, EPSG:4326 coordinates spans across Extent: -180,-90 : 180,90 However, tool complain the values being negative numbers.

Not sure, if I am missing anything to get the tool executed for the extended that I had computed. Any suggestions would be greatly appreciated.

-Prem

[hummeltech]

@pkthogit, sure, that will probably work. The issue you are running into there is that those values are tile coordinates, which are always positive integers. They are the same as the z/x/y values you see when requesting a tile (https://tile.openstreetmap.org/{zoom}/{x}/{y}.png)

There was recently a note added to the man page (as well as the --help output,) but it has not yet been released and/or packaged:

mod_tile/docs/man/render_list.1

Line 76 in c3d9bd2

(please note that tile coordinates must be positive integers and are not latitude and longitude values)

[ldeffenb]

@pkthogit I did a similar optimization on higher zoom level tiles by looking at the next lower zoom level tile and not rendering any of the 4 higher zoom tiles if the lower zoom tile was a single color, be it water, desert, or ice. But then, my application also runs a leaflet extension that automatically fetches and stretches a lower-zoom level tile piece if the current zoom is not found. It does this recursively, so if a single zoom 5 tile is all one color, every zoom higher than that actually uses bits of that single tile.