Multi-Core and Cluster computing with ppgdal2tiles

The standard version of GDAL2Tiles is using only one processing core at a time.

In case you have a computer with more processors (or one of the Mutli-Core processors from Intel or AMD) then several processes can be started to use the full computing power of your computer to finish the tile rendering as fast as possible.

You can also use different computers in your computer network to generate tiles concurrently on more machines and therefore dramatically decrease the time necessary to process huge datasets (such as map composed from several map sheets).

All those capabilities are available in the MapTiler Pro (former Parallel GDAL2Tiles).

How to use the Parallel GDAL2Tiles on the computers with Multi-Core processor or SMP architecture: 

The usage of the parallel GDAL2Tiles is the same like the standard GDAL2Tiles. Both utilities are using the same parameters.
You just have to call ./ppgdal2tiles.py command instead of gdal2tiles.py. It is that easy.

How to setup a computing node in a cluster:

- Install python, gdal 1.6+, ppgdal2tiles.py and pp module (parallelpython).
- Upload your source data (all nodes should have access to the same data) or mount the data directory over network. For uploading the same files to more nodes you can use udpcast utility.
- Set up an shared network storage with write privilege for storing all the generated tiles (access by NFS, Samba, ...).
- Start "ppserver.py -a" in a directory where you would like to save/read data. All local path to files given to maptiler will be relative to the directory where you started ppserver process - for every node. Check that the user under you run the ppserver have access to write the produced tiles to the shared directory.
- Computing nodes are discovered automatically on the local network. You an also specify their IP addresses explicitly. 
Requests to the port 60000 on nodes should not be blocked by the firewall.

Computing nodes can be added dynamically, even after the processing starts, and can be also turned down whenever it is necessary without loosing any data.

The software can also run a custom Amazon Machine Image (AMI) for Amazon EC2 cluster (for those who would like to rent the cluster for the geodata tile rendering).

For details related to the MapTiler Pro, for help with geodata conversion, for consultancy and commercial support please have a look at http://www.maptiler.com/ and feel free to contact us at info@klokantech.com.

Petr Přidal (author of GDAL2Tiles and MapTiler)
Klokan Technologies GmbH

Comments