How to make your own movie with OSM evolution – part I

Because of the interest on my post with the movies I have created to show OSM evolution in some locations in Romania, I have decided to write a tutorial with all the needed steps to create something similar. The tutorial will be splitted in two parts:
1. Creating a country extract from older planet files
2. Using mapnik and some simple scripts to create images for a specific area, and then compile those images in a MPEG movie.

Just to make things clear, I used a Ubuntu 9.04 to get everything done. I am not sure how easy will be to have something similar on Windows, but it will be almost identical for any Linux distro.

For the first part I would like to thank Frederik and Grant who pointed me to the right scripts, and also mentioned the differences between older OSM API’s and XML formats. The key to create a history of extracts for a country is to get older planet files and apply the correct scripts to them. For the planet files you may chose between the official provider http://planet.openstreetmap.org/ and the mirror hosted by Heanet http://ftp.heanet.ie/mirrors/openstreetmap.org/.

There are mutiple actions to be taken in order to get the extract, and these actions depends on the date of the file (which actually gives the version of the API). There is one common prerequisite, the country polygon. If you don’t have one already, you can probably get it from the Cloudmade’s download section (Romania’s polygon). Once you have the polygon file you can start to download the planet files starting with the oldest one (for Romania I started with 20061205, the country was almost a blank canvas at that time).

For planet files between 20061205 (and maybe older, but I haven’t tried it) and 20071003 the API version is 0.3 or 0.4 (the XML format is identical). So you will need a little bit of workaround to convert these to 0.5 that can be processed with current tools. Here are the steps in order to get the extract for 20061205:

$ sudo apt-get install subversion # needed if you didn't \
use SVN before on that machine
$ mkdir ~/svn
$ cd ~/svn
$ svn co -r 4779 http://svn.openstreetmap.org/\
applications/utils/osm-extract

The extract script will need the Math:Polygon Perl module, which yo can install using CPAN (I am not an expert, I followed some ideas found on the web and it worked. If you have problems I am not the guy to ask about Perl :-) ). You should also install some prerequisites:

$ sudo apt-get install lynx
$ sudo apt-get install ncftp
$ cpan

At the cpan prompt (that you will get after answering some questions) you should type:

cpan[1]> install Math::Polygon

When the installation is finished you should be ready to start extracting your country.

$ mkdir ~/planet
$ cd ~/planet
$ wget http://ftp.heanet.ie/mirrors/openstreetmap.org/\
planet-061205.osm.bz2

At this time you are ready to create the extract, but first get the polygon file and copy it to the new ~/planet directory. After that you can start the extract:

$ ~/svn/osm-extract/polygons/extract-polygon.pl \
-i planet-061205.osm.bz2 -o MY_COUNTRY-061205.osm \
-p MY_POLYGON_FILE

Of course you should replace MY_COUNTRY and MY_POLYGON_FILE your the correct names to have everything up  and running. I would use Romania in the examples to keep things simple, but you should use your country name. This should take a while depending on the country and of course the power of the server you use. In the end you should get a new .osm file for your country. Note 1: that this file is in 0.4 (or 0.3) XML format so we need to convert it to 0.5. For this we will need to get another script from SVN:

$ cd ~/svn
$ svn co http://svn.openstreetmap.org/applications/\
utils/conv05
$ cd ~/planet
$ mkdir 0.5
$ ~/svn/conv05/04to05.pl <romania-061205.osm \
>0.5/romania-061205.osm

After this step you should have a 0.5 compatible OSM extract of your country in the 0.5 directory. Note 2: If you get an error about a missing Perl package when starting a Perl script you should use CPAN to install it.

The next step is not mentioned anywhere from what I know but I have discovered that after the conversion you will get some ways that reference nodes without an ID. This can be fix easily using sed:

$ cd ~/planet/0.5
$ sed 's|<nd ref=""/>||g' romania-061205.osm \
>romania-061205.osm.temp
$ cp romania-061205.osm.temp romania-061205.osm

This is the final step of creating the extract. I usually validate such an extract by trying to open it in JOSM. If you get any errors while trying this leave a comment here or better post on the Talk list.

You should repeat (probably you will create a small script) the above steps in order to get the extracts until 20071003 (inclusive). For example to get the 20061213 extract  the steps will be:

$ cd ~/planet
$ wget http://ftp.heanet.ie/mirrors/openstreetmap.org/\
planet-061213.osm.bz2
$ ~/svn/osm-extract/polygons/extract-polygon.pl \
-i planet-061205.osm.bz2 -o romania-061205.osm -p romania.poly
$ ~/svn/conv05/04to05.pl <romania-061213.osm \
>0.5/romania-061213.osm
$ cd 0.5
$ sed 's|<nd ref=""/>||g' romania-061213.osm \
>romania-061213.osm.temp
$ cp romania-061213.osm.temp romania-061213.osm

This concludes the part where we describe how to extract a pre-0.5 country file. For planet files starting with 20071010 the things are a little bit simples, because we don’t have to convert it. For 0.5 version files we can use Osmosis to create the extract, but first we need to donwload the latest version:

$ cd ~
$ wget http://dev.openstreetmap.org/~bretth/osmosis-build/\
osmosis-latest-bin.tar.gz
$ tar zxf osmosis-latest-bin.tar.gz

Note 3: Osmosis needs Sun Java JRE 1.6, you should make sure you have it installed. For planet files from 20071010 up to 20090418 (inclusive) you should specify to Osmosis to use 0.5 tasks:

$ cd planet
$ wget http://ftp.heanet.ie/mirrors/openstreetmap.org/\
planet-071010.osm.bz2
$ bzcat planet-071010.osm.bz2 | ~/osmosis-0.31/bin/osmosis \
--read-xml-0.5 file=/dev/stdin --bounding-polygon-0.5 \
file=romania.poly  --write-xml-0.5 file=romania-071010.osm

Starting with 20090421 you can remove 0.5 suffix to the tasks:

$ cd planet
$ wget http://ftp.heanet.ie/mirrors/openstreetmap.org/\
planet-090429.osm.bz2
$ bzcat planet-090429.osm.bz2 | ~/osmosis-0.31/bin/osmosis \
--read-xml file=/dev/stdin --bounding-polygon file=romania.poly \
--write-xml file=romania-090429.osm

Using the above steps you should be able to get a history of .osm files for your country. We (OSM Romania) are using a Git repository to store the planet extracts but this is not necessary. I will try to create some simple scripts that will generate images with date caption that should make things simpler for those who don’t want to use Git.

To prepare for the next part of the tutorial I propose you to read the Mapnik install wiki page http://wiki.openstreetmap.org/wiki/Mapnik. We will use the generate_image.py script so you should make sure that you can generate images successfully using that script. You will probably find the PostGIS wiki page very useful as well http://wiki.openstreetmap.org/wiki/Mapnik/PostGIS. Also I recommend to compile osm2pgsql from source as described here: http://wiki.openstreetmap.org/wiki/Osm2pgsql.

I hope to have the second part ready until the end of the week, but I think in the meantime you have more than enough to play with. Questions are welcome in the comments.

UPDATE:

The readers have requested more information about using the planet weekly diff files to save bandwidth, so here it is: I used two different versions for the planetdiff tool, depending on the version of the API. For planet files/diffs before 20071010 (not included) I used the revision 4778 from SVN:

$ cd ~/svn
$ svn co -r 4778 http://svn.openstreetmap.org/\
applications/utils/planetdiff

You will need to compile the source for planetdiff, if you see any errors during compile time use the readme.txt file to find out what you need.

$ cd planetdiff; make

At this point you will have two binaries, planetdiff and planetpatch. You will use planetpatch for the following steps:

$ ./planetpatch ~/planet-070718.osm.bz2 \
~/planet-070718-070725.diff.xml.bz2 | bzip2 \
-6 > planet-070725.osm.bz2

The above is just an example on how to generate planet file from 070725 using the old planet file from 070718 and the diff file. You should make sure that the paths to the above files are correct, I had everything in the home directory.

For newer planet files (starting with 071010) you can use the latest version of the planetdiff utility (which might also work for older planets, but I didn’t want to experiment). The process is exactly the same, but you shouldn’t provide the revision when you checkout the source code and also provide another directory as location:

$ cd ~/svn
$ svn co http://svn.openstreetmap.org/applications/\
utils/planetdiff pd
$ cd pd; make
$ ./planetpatch ~/planet-080423.osm.bz2 \
~/planet-080423-080430.diff.xml.bz2 | bzip2 \
-6 > planet-080430.osm.bz2

As I mentioned in the comments applying the weekly diff file is taking a long (almost a day on my machine, not very powerful indeed), so you will have to be patient.

About these ads

15 Răspunsuri

  1. This is a great tutorial!!!
    There’s a couple of things I’d like to know:
    How many of the planet files did you download.
    Could you give us an indication on the amount of file size needed for those planet files / extracts.
    How long did it take start to finish, what takes the longest amount of time?

    Cheers! :)

    Tim

  2. Fortunately we had started to create our own extract of Romania on November 2007 so what I needed to do is download planet files before that, which also have the advantage of being smaller.

    About the space needed, if you have some issues with that you can always process one planet file at a time. This means that you will need 10GB for the latest planet + space for all the extracts. In terms of space occupied by an extract I can offer some info for Romania: for all the extracts from 2006 until now, in gzip format, you need another 10GB.

    About time needed to create an extract:
    – download time of the planet file depends on your bandwidth.
    – getting the current extract takes 2.5 hours on a P4 HT 3.0GHz, 1GB RAM. It should take much less on older planet files (it was half an hour on November 2007).
    – converting from 0.4 to 0.5 API is the fastest, just a few minutes.

    If you have a slow Internet connection you might want to consider downloading just the weekly planet diffs, and apply them using planetdiff/planetpatch tool. I can offer more information about this, if it is neeeded. From my experience with patching the planet this is the most time consuming operation (took almost a day for a late 2007 planet on a Dell D630, Core Duo 7300 2.2 GHz, 2 GB RAM)

    I know that this part (getting the extracts) will be time consuming but you have to see also the advantages: once done you can create any kind of animation for your country :).

  3. Hi Ciprian,
    I have downloaded weekly diff files and don’t know how to use them to make extracts using them. Any help on using planetdiff/planetpatch would be a great!

    Cheers.

    • Sorry for the late response, I was very busy in the last days. I will prepare a response for tomorrow, and also start to work on the second part of the tutorial.

    • Valent, I have just updated the tutorial with information about installing and using the planet diff utility. Hope it helps!

  4. [...] has clearly demonstrated. UPDATE: Ciprian aka Mapguy has made tutorial explaining how to make OSM progress animation.   So in order to make this work they made git repository for planet file and that is how it [...]

  5. I hate that Tutorial… too much linux console. Who on Hell can i create custom .poly file?

  6. I’m having issues with planetdiff script. First I used old version of planetdiff with new planet files, I didn’t pay attention :(

    But now I use latest planetdiff and planet files from 2008 and I can’t make new planet file from old planet+diff :(

    I downloaded planet-080507.osm.bz2 (4GB) file and planet-080507-080514.diff.xml.bz2.

    and used this command to create new planet file:
    ~/svn/planetdiff/planetpatch planet-080507.osm.bz2 planet-080507-080514.diff.xml.bz2 | bzip2 -6 > planet-080514.osm.bz2

    The problem is that the resulting planet-080514.osm.bz2 file is 3.8GB! Smaller than original planet.

    Can you please try to reproduce this same planet+diff and see if you get the same result.

    And the file is broken because when I try to use it and apply the next diff I get an error after 580 minutes:

    $ time ~/svn/planetdiff/planetpatch planet-080514.osm.bz2 planet-080514-080521.diff.xml.bz2 | bzip2 -6 > planet-080521.osm.bz2
    File reader received error -1
    Entity: line 505772137: parser error : AttValue: ” or ‘ expected
    <node id=
    ^
    Entity: line 505772137: parser error : attributes construct error
    <node id=
    ^
    Entity: line 505772137: parser error : Couldn't find end of Start Tag node
    <node id=
    ^
    Error parsing file planet-080514.osm.bz2

    real 584m18.945s
    user 553m9.306s
    sys 3m10.832s

    Here is the list of all files I have used and their sizes:

    $ ls -lh
    total 8.7G
    88M 2009-12-09 19:40 planet-061205.osm.bz2
    90M 2009-12-09 19:40 planet-061213.osm.bz2
    665M 2009-12-09 19:41 planet-071010.osm.bz2
    45M 2009-12-09 19:41 planet-080507-080514.diff.xml.bz2
    4.0G 2009-12-09 19:48 planet-080507.osm.bz2
    42M 2009-12-09 19:48 planet-080514-080521.diff.xml.bz2
    3.8G 2009-12-11 01:10 planet-080514.osm.bz2
    33M 2009-12-09 19:48 planet-080521-080528.diff.xml.bz2
    4.3M 2009-12-09 19:48 planet-080521.osm.bz2
    34M 2009-12-09 19:48 planet-080528-080604.diff.xml.bz2
    37M 2009-12-09 19:48 planet-080604-080611.diff.xml.bz2

    Do you have any suggestion?

    • I tried to reproduce the problem, and it looks like I get the same results. The difference in file size can be produced by the compression used (-6 parameter to bzip). I have generated planet files from 080528 to 080702 and the file size is between 3.8 and 3.9 GB.

      ctalaba@ctalaba-laptop:~/svn/pd$ ls -l
      45M 2008-05-15 09:08 planet-080507-080514.diff.xml.bz2
      4.0G 2008-05-07 21:13 planet-080507.osm.bz2
      42M 2008-05-22 13:06 planet-080514-080521.diff.xml.bz2
      3.8G 2009-12-23 03:59 planet-080514.osm.bz2
      156M 2009-12-23 10:46 planet-080521.osm.bz2
      3.8G 2009-10-22 15:26 planet-080528.osm.bz2
      3.8G 2009-10-23 23:35 planet-080604.osm.bz2
      3.9G 2009-10-25 01:33 planet-080611.osm.bz2
      3.9G 2009-10-26 01:07 planet-080618.osm.bz2
      3.9G 2009-10-30 07:34 planet-080625.osm.bz2
      3.9G 2009-10-31 00:03 planet-080702.osm.bz2

      Right now I am trying to obtain planet from 080521 to see if I get the same error. If this is the case you can always download it and use it to get the next planet files (which worked for me).

  7. Thank you for your reply.

    I’m now trying this tutorial on Ubuntu, I was doing it previously on Fedora.

    I found one cpan module you need to add:

    cpan[1]> install Bit::Vector

    Hope this helps others.

    • Valent,

      I had no problem applying the diff file from 080514 to 080521, so I don’t know what might be the problem on your side (I am using Ubuntu, but I would expect this to work on Fedora without a problem).

  8. It seams that I have some hardware HDD issues. But I figured out that it is faster to download whole planet files than to make diff files. It takes me cca 3-6h to download planet file but it takes me 2 days to make a diff file.

    So I’ll download and extract files.

    I already have some 10 extract of Croatia from 2007 and 2008…

    Is there some OSM server that I can upload extracts of Croatia so that others also can use them?

    How can I make .poly files for some areas and towns?

  9. Hi,
    I think we can proceed now to part 2? ;)

  10. Code a all in One GUI… shit Linux Command Prompt

Lasă un răspuns

Completeaza detaliile de mai jos sau apasa click pe una din imagini pentru a te loga:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Schimbă )

Twitter picture

You are commenting using your Twitter account. Log Out / Schimbă )

Facebook photo

You are commenting using your Facebook account. Log Out / Schimbă )

Google+ photo

You are commenting using your Google+ account. Log Out / Schimbă )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: