How to make your own movie with OSM evolution – part I

Because of the interest on my post with the movies I have created to show OSM evolution in some locations in Romania, I have decided to write a tutorial with all the needed steps to create something similar. The tutorial will be splitted in two parts:
1. Creating a country extract from older planet files
2. Using mapnik and some simple scripts to create images for a specific area, and then compile those images in a MPEG movie.

Just to make things clear, I used a Ubuntu 9.04 to get everything done. I am not sure how easy will be to have something similar on Windows, but it will be almost identical for any Linux distro.

For the first part I would like to thank Frederik and Grant who pointed me to the right scripts, and also mentioned the differences between older OSM API’s and XML formats. The key to create a history of extracts for a country is to get older planet files and apply the correct scripts to them. For the planet files you may chose between the official provider http://planet.openstreetmap.org/ and the mirror hosted by Heanet http://ftp.heanet.ie/mirrors/openstreetmap.org/.

There are mutiple actions to be taken in order to get the extract, and these actions depends on the date of the file (which actually gives the version of the API). There is one common prerequisite, the country polygon. If you don’t have one already, you can probably get it from the Cloudmade’s download section (Romania’s polygon). Once you have the polygon file you can start to download the planet files starting with the oldest one (for Romania I started with 20061205, the country was almost a blank canvas at that time).

For planet files between 20061205 (and maybe older, but I haven’t tried it) and 20071003 the API version is 0.3 or 0.4 (the XML format is identical). So you will need a little bit of workaround to convert these to 0.5 that can be processed with current tools. Here are the steps in order to get the extract for 20061205:

$ sudo apt-get install subversion # needed if you didn't \
use SVN before on that machine
$ mkdir ~/svn
$ cd ~/svn
$ svn co -r 4779 http://svn.openstreetmap.org/\
applications/utils/osm-extract

The extract script will need the Math:Polygon Perl module, which yo can install using CPAN (I am not an expert, I followed some ideas found on the web and it worked. If you have problems I am not the guy to ask about Perl 🙂 ). You should also install some prerequisites:

$ sudo apt-get install lynx
$ sudo apt-get install ncftp
$ cpan

At the cpan prompt (that you will get after answering some questions) you should type:

cpan[1]> install Math::Polygon

When the installation is finished you should be ready to start extracting your country.

$ mkdir ~/planet
$ cd ~/planet
$ wget http://ftp.heanet.ie/mirrors/openstreetmap.org/\
planet-061205.osm.bz2

At this time you are ready to create the extract, but first get the polygon file and copy it to the new ~/planet directory. After that you can start the extract:

$ ~/svn/osm-extract/polygons/extract-polygon.pl \
-i planet-061205.osm.bz2 -o MY_COUNTRY-061205.osm \
-p MY_POLYGON_FILE

Of course you should replace MY_COUNTRY and MY_POLYGON_FILE your the correct names to have everything up  and running. I would use Romania in the examples to keep things simple, but you should use your country name. This should take a while depending on the country and of course the power of the server you use. In the end you should get a new .osm file for your country. Note 1: that this file is in 0.4 (or 0.3) XML format so we need to convert it to 0.5. For this we will need to get another script from SVN:

$ cd ~/svn
$ svn co http://svn.openstreetmap.org/applications/\
utils/conv05
$ cd ~/planet
$ mkdir 0.5
$ ~/svn/conv05/04to05.pl <romania-061205.osm \
>0.5/romania-061205.osm

After this step you should have a 0.5 compatible OSM extract of your country in the 0.5 directory. Note 2: If you get an error about a missing Perl package when starting a Perl script you should use CPAN to install it.

The next step is not mentioned anywhere from what I know but I have discovered that after the conversion you will get some ways that reference nodes without an ID. This can be fix easily using sed:

$ cd ~/planet/0.5
$ sed 's|<nd ref=""/>||g' romania-061205.osm \
>romania-061205.osm.temp
$ cp romania-061205.osm.temp romania-061205.osm

This is the final step of creating the extract. I usually validate such an extract by trying to open it in JOSM. If you get any errors while trying this leave a comment here or better post on the Talk list.

You should repeat (probably you will create a small script) the above steps in order to get the extracts until 20071003 (inclusive). For example to get the 20061213 extract  the steps will be:

$ cd ~/planet
$ wget http://ftp.heanet.ie/mirrors/openstreetmap.org/\
planet-061213.osm.bz2
$ ~/svn/osm-extract/polygons/extract-polygon.pl \
-i planet-061205.osm.bz2 -o romania-061205.osm -p romania.poly
$ ~/svn/conv05/04to05.pl <romania-061213.osm \
>0.5/romania-061213.osm
$ cd 0.5
$ sed 's|<nd ref=""/>||g' romania-061213.osm \
>romania-061213.osm.temp
$ cp romania-061213.osm.temp romania-061213.osm

This concludes the part where we describe how to extract a pre-0.5 country file. For planet files starting with 20071010 the things are a little bit simples, because we don’t have to convert it. For 0.5 version files we can use Osmosis to create the extract, but first we need to donwload the latest version:

$ cd ~
$ wget http://dev.openstreetmap.org/~bretth/osmosis-build/\
osmosis-latest-bin.tar.gz
$ tar zxf osmosis-latest-bin.tar.gz

Note 3: Osmosis needs Sun Java JRE 1.6, you should make sure you have it installed. For planet files from 20071010 up to 20090418 (inclusive) you should specify to Osmosis to use 0.5 tasks:

$ cd planet
$ wget http://ftp.heanet.ie/mirrors/openstreetmap.org/\
planet-071010.osm.bz2
$ bzcat planet-071010.osm.bz2 | ~/osmosis-0.31/bin/osmosis \
--read-xml-0.5 file=/dev/stdin --bounding-polygon-0.5 \
file=romania.poly  --write-xml-0.5 file=romania-071010.osm

Starting with 20090421 you can remove 0.5 suffix to the tasks:

$ cd planet
$ wget http://ftp.heanet.ie/mirrors/openstreetmap.org/\
planet-090429.osm.bz2
$ bzcat planet-090429.osm.bz2 | ~/osmosis-0.31/bin/osmosis \
--read-xml file=/dev/stdin --bounding-polygon file=romania.poly \
--write-xml file=romania-090429.osm

Using the above steps you should be able to get a history of .osm files for your country. We (OSM Romania) are using a Git repository to store the planet extracts but this is not necessary. I will try to create some simple scripts that will generate images with date caption that should make things simpler for those who don’t want to use Git.

To prepare for the next part of the tutorial I propose you to read the Mapnik install wiki page http://wiki.openstreetmap.org/wiki/Mapnik. We will use the generate_image.py script so you should make sure that you can generate images successfully using that script. You will probably find the PostGIS wiki page very useful as well http://wiki.openstreetmap.org/wiki/Mapnik/PostGIS. Also I recommend to compile osm2pgsql from source as described here: http://wiki.openstreetmap.org/wiki/Osm2pgsql.

I hope to have the second part ready until the end of the week, but I think in the meantime you have more than enough to play with. Questions are welcome in the comments.

UPDATE:

The readers have requested more information about using the planet weekly diff files to save bandwidth, so here it is: I used two different versions for the planetdiff tool, depending on the version of the API. For planet files/diffs before 20071010 (not included) I used the revision 4778 from SVN:

$ cd ~/svn
$ svn co -r 4778 http://svn.openstreetmap.org/\
applications/utils/planetdiff

You will need to compile the source for planetdiff, if you see any errors during compile time use the readme.txt file to find out what you need.

$ cd planetdiff; make

At this point you will have two binaries, planetdiff and planetpatch. You will use planetpatch for the following steps:

$ ./planetpatch ~/planet-070718.osm.bz2 \
~/planet-070718-070725.diff.xml.bz2 | bzip2 \
-6 > planet-070725.osm.bz2

The above is just an example on how to generate planet file from 070725 using the old planet file from 070718 and the diff file. You should make sure that the paths to the above files are correct, I had everything in the home directory.

For newer planet files (starting with 071010) you can use the latest version of the planetdiff utility (which might also work for older planets, but I didn’t want to experiment). The process is exactly the same, but you shouldn’t provide the revision when you checkout the source code and also provide another directory as location:

$ cd ~/svn
$ svn co http://svn.openstreetmap.org/applications/\
utils/planetdiff pd
$ cd pd; make
$ ./planetpatch ~/planet-080423.osm.bz2 \
~/planet-080423-080430.diff.xml.bz2 | bzip2 \
-6 > planet-080430.osm.bz2

As I mentioned in the comments applying the weekly diff file is taking a long (almost a day on my machine, not very powerful indeed), so you will have to be patient.

http://downloads.cloudmade.com/europe/romania/romania.poly