OpenStreetMap: A Year of Edits

In the past weeks I've been working on some visualisations related to additions and changes being made to OpenStreetMap. To start of the new year, I hereby present: OpenStreetMap: A Year of Edits (2011):

You can see the video in HD on Vimeo. Happy New Year!

How Did He Do That?

This animation is all made with Open Source software. I haven't had the time to clean up the code to make it releasable yet, but I am intending to do that in the near feature and add it to my OpenStreetMap tools git repository on github.

The first thing I did, was download all the hourly replication files from http://planet.openstreetmap.org/hour-replicate for the year.

With a script, I scanned over all those XML files and for every three hours I created an image showing the edits of those last three hours (in white), as well as all the other edits in the background (in purple). With 4 frames per day, this gives me 1460 frames.

Each of the frames is an equirectangular projection which I then map onto a sphere with a faded background of "earthmap10k" of http://planetpixelemporium.com/earth.html (I downloaded it years ago when it was still free to download). For the 3D rendering of the Earth with the edits layed on top of it I used POV-Ray.

I generated two different loops so that I can make two full rotations at different angles. This gives me in total 2882 frames which is about two minutes. The POV-Ray generated images I post-processed by overlaying the progress bar in the lower left and added the fading and merging effects. Then I used mencoder to create a MPEG video out of it. Finally I added the sound track again by using mencoder and uploaded it to Vimeo.

Twig extension

A while ago, Fabien asked me to have a look at porting one of Twig's slowest methods, TwigTemplate::getAttribute(), into a PHP extension. It is a complex method that does a lot of different checks and look-ups. Fabien's benchmarks showed that this method was responsible for quite a large amount of time. On top of that, it didn't seem that it could be optimised any further as PHP code itself. Seeing that it was very likely that this method would be a lot faster when written in C, SensioLabs decided to sponser me for the development of a port of this method into a PHP extension.

Over the next few months I've worked on re-implementing just this method as a C-function in twig-ext: twig_template_get_attributes(). From initial benchmarks it looks like this function as implemented in the extension gives about a 15% speed increase while rendering templates. Twig's compiled templates directly contain a call to this function (as opposed to marshal it through the original TwigTemplate::getAttribute() method) for additional performance. This probably means that you'll have to remove your compiled templates cache while upgrading.

The extension at http://github.com/derickr/twig-ext has now been merged, via http://github.com/derickr/Twig, into http://github.com/fabpot/Twig. The plan is that this new improvement is going to be part of Twig 1.4.

If you are also interested in having some of your PHP code ported into a PHP extension in C, please feel free to contact me.

git clone https://github.com/fabpot/Twig.git
cd ext/twig
phpize && ./configure && make install

extension=twig.so

pecl install twig

XFCE and moving windows around with the keyboard

During a recent apt-get update && apt-get dist-upgrade I suddenly found myself with the horrors that are Gnome 3 and the Gnome shell, and of course, at the least opportune moment when I was actually planning on heading into an office (more about that later). I've always been a big Sawfish fan, and was annoyed when Gnome replaced that with Metacity, but I always managed to get Sawfish back up again. No more luck now, and without a Debian package maintainer I decided to try out XFCE.

I found XFCE to be as easy to use with the keyboard as Sawfish but with one big missing issue: I couldn't change the size and position of windows at the keyboard anymore. I hardly use a mouse as it hurts too much, and always relied on those keyboard shortcuts to position my windows so I was quite a bit at a loss. After some searching I ran into a post on nabble that contained a script that almost did what I want. It uses a bunch of standard X tools (xprop and xwininfo) to find some information about the active window and its properties. Then depending on some parameters it would use wmctrl to move and/or resize the window. For example: ./bin/window-geometry-control.sh -m left would move the window 1 pixel to the left.

That's all good, but way too slow. I basically just want to bump my windows to the top, right, bottom and left sides of my screen. So besides the simple move and resize, I added what Sawfish originally also had: move left/right/top and bottom; and well as resize-to left/right/top and bottom. Now with the command ./bin/window-geometry-control -b left I can move the active window all the way to the left, and with ./bin/window-geometry-control -s bottom I can resize my window from its current position all the way to the bottom of the screen. To each of the four directions and two methods I assigned keyboard shortcuts in xfce4-settings-manager, Keyboard, Application Shortcuts.

The screenshot above shows those assigned keyboard shortcuts. I've put the modified script on github. As you can see I hardcoded the width and height of my own screen in it. Feel free to make this dynamic and send a pull request.

Multiple PHP versions set-up

For many of my projects (both hobby and commercial) I need to support many different PHP configurations. Not only just different PHP versions, but also debug builds, ZTS builds and 32-bit builds. In order to be able to test and build extensions against all those different PHP configurations I have adopted a simple method that I'm sharing with you here.

The major part of it is that I use a different installation prefix for every configuration of PHP that I make. Right now, ls /usr/local/php shows:

5.1dev  5.3.2  5.3.8dev  5.3dev-32bit    5.3dev-zts  5.4dev      trunk
5.2dev  5.3.7  5.3dev    5.3dev-nodebug  5.4.0beta1  5.4dev-zts

There are two types of PHP installs that I make: "dev" versions from SVN branches, and "release" versions build from SVN tags. From the list above you see I have the SVN versions of 5.1, 5.2, 5.3.8, 5.3 (in various forms) and 5.4 (both normal, and ZTS).

I have a script to compile PHP which whatever combination I want: version, debug (default) or non-debug, normal (default) or ZTS; and 64-bit (default) or 32-bit. The configure options are nicely hardcoded :-)

The scripts accept arguments in a specific order:

version ["debug"|"nodebug" [, "no-zts"|"zts" [, ""|"32bit" ] ] ]

For a simple 5.3dev build I run:

./build 5.3dev

This compiles PHP 5.3 from SVN, in debug mode, without ZTS and for the 64-bit architecture. Something more exotic variants could be:

./build 5.3.8 debug zts
./build 5.4dev nodebug no-zts 32bit

Each invocation of the script will configure and build PHP, and then install into /usr/local/php/{{variant}}.

The script also has a hard coded location where the PHP sources are. In my case, that's a sparse checkout into /home/derick/dev/php/php-src.

With the help of a small .bashrc snippet:

function pe () {
        version=$1
        shift

        if [ "$#" == "0" ]; then
                export PATH=/usr/local/php/${version}/bin:/usr/local/bin:/usr/bin:/bin
        else
                PATH=/usr/local/php/${version}/bin:$PATH $@
        fi
}

I can now easily switch between PHP versions by typing on the shell:

pe 5.3dev

or:

pe 5.4dev-zts

And each version will have a totally separated environment for me to install PEAR packages and PECL extensions in, and do my own extension development. Of course, each separated environment also comes with its own php.ini file (in /usr/local/php/{{variant}}/lib/php.ini).

This set-up makes my life a whole lot easier, and I hope it is useful for you as well. Thanks for listening!

/usr/local/stow/php5.3
/usr/local/stow/php5.4-dev

/usr/local/bin/php -> /usr/local/stow/php5.4-dev/bin/php

Xdebug's Code Coverage speedup

Besides a debugging and development aid, Xdebug also implements the back-end code to provide code coverage information for use with PHPUnit. Code coverage tells you how much of your code base is actually being tested by your unit tests. It's a very useful feature, but sadly, it slows down PHP's execution quite a lot. One part of this slowdown is the overhead to record the information internally, but another part is because I have to overload lots of opcodes. (Opcodes are PHP's internal execution units, similar to assembler instructions) They are always overloaded even if code coverage is not used, because it's only safe to overload them for the whole request.

The upcoming Xdebug 2.2 has a few changes to improve code coverage performance. First of all, the speed with which information about code coverage is recorded has been improved by contributions by Taavi Burns. And secondly, Xdebug 2.2 features a new setting, xdebug.coverage_enable with which it is possible to disable the overloading of code coverage specific opcodes. By default that setting will still be on.

Just to show how those improvements have effect on the speed, I've run a benchmark (running Apache Zeta Components Graph's tests) to compare Xdebug 2.1.2 against Xdebug 2.2-dev.

The results are as follows:

Another benchmark, run by Ross McFarlane gives: @derickr Based on 3 runs each, execution of 254 tests with 1070 assertions has dropped from 19s to 12s. Nice work!.

In Xdebug 2.2 I would also like to introduce modes. Each mode will set a default configuration for Xdebug's setting to be the most optimal for that specific tasks. Examples of tasks could be: "profiling", "debugging" or "tracing". Let me know (in the comments) whether you think that's a good idea.