Derick Rethans - tag: extensions

Twig extension

Mon, 21 Nov 2011 09:20:00 +0000

Twig extension

Monday, November 21st 2011, 09:20 GMT

A while ago, Fabien asked me to have a look at porting one of Twig's slowest methods, TwigTemplate::getAttribute(), into a PHP extension. It is a complex method that does a lot of different checks and look-ups. Fabien's benchmarks showed that this method was responsible for quite a large amount of time. On top of that, it didn't seem that it could be optimised any further as PHP code itself. Seeing that it was very likely that this method would be a lot faster when written in C, SensioLabs decided to sponser me for the development of a port of this method into a PHP extension.

Over the next few months I've worked on re-implementing just this method as a C-function in twig-ext: twig_template_get_attributes(). From initial benchmarks it looks like this function as implemented in the extension gives about a 15% speed increase while rendering templates. Twig's compiled templates directly contain a call to this function (as opposed to marshal it through the original TwigTemplate::getAttribute() method) for additional performance. This probably means that you'll have to remove your compiled templates cache while upgrading.

The extension at http://github.com/derickr/twig-ext has now been merged, via http://github.com/derickr/Twig, into http://github.com/fabpot/Twig. The plan is that this new improvement is going to be part of Twig 1.4.

If you are also interested in having some of your PHP code ported into a PHP extension in C, please feel free to contact me.

Multiple PHP versions set-up

London, UK

Monday, November 7th 2011, 09:11 GMT

For many of my projects (both hobby and commercial) I need to support many different PHP configurations. Not only just different PHP versions, but also debug builds, ZTS builds and 32-bit builds. In order to be able to test and build extensions against all those different PHP configurations I have adopted a simple method that I'm sharing with you here.

The major part of it is that I use a different installation prefix for every configuration of PHP that I make. Right now, ls /usr/local/php shows:

5.1dev  5.3.2  5.3.8dev  5.3dev-32bit    5.3dev-zts  5.4dev      trunk
5.2dev  5.3.7  5.3dev    5.3dev-nodebug  5.4.0beta1  5.4dev-zts

There are two types of PHP installs that I make: "dev" versions from SVN branches, and "release" versions build from SVN tags. From the list above you see I have the SVN versions of 5.1, 5.2, 5.3.8, 5.3 (in various forms) and 5.4 (both normal, and ZTS).

I have a script to compile PHP which whatever combination I want: version, debug (default) or non-debug, normal (default) or ZTS; and 64-bit (default) or 32-bit. The configure options are nicely hardcoded :-)

The scripts accept arguments in a specific order:

version ["debug"|"nodebug" [, "no-zts"|"zts" [, ""|"32bit" ] ] ]

For a simple 5.3dev build I run:

./build 5.3dev

This compiles PHP 5.3 from SVN, in debug mode, without ZTS and for the 64-bit architecture. Something more exotic variants could be:

./build 5.3.8 debug zts
./build 5.4dev nodebug no-zts 32bit

Each invocation of the script will configure and build PHP, and then install into /usr/local/php/{{variant}}.

The script also has a hard coded location where the PHP sources are. In my case, that's a sparse checkout into /home/derick/dev/php/php-src.

With the help of a small .bashrc snippet:

function pe () {
        version=$1
        shift

        if [ "$#" == "0" ]; then
                export PATH=/usr/local/php/${version}/bin:/usr/local/bin:/usr/bin:/bin
        else
                PATH=/usr/local/php/${version}/bin:$PATH $@
        fi
}

I can now easily switch between PHP versions by typing on the shell:

pe 5.3dev

or:

pe 5.4dev-zts

And each version will have a totally separated environment for me to install PEAR packages and PECL extensions in, and do my own extension development. Of course, each separated environment also comes with its own php.ini file (in /usr/local/php/{{variant}}/lib/php.ini).

This set-up makes my life a whole lot easier, and I hope it is useful for you as well. Thanks for listening!

Valgrinding shared modules

London, UK

Monday, August 8th 2011, 09:56 BST

Over the past year I've been writing various commercial (more about that later) and open source PHP extensions (such as QuickHash, more about that later too). Most of the time they are shared extensions that are not part of PHP. While testing whether I correctly free all memory with Valgrind, I ran into the issue where I couldn't see the stack frames of where the memory leaks occurred in the extensions, and once I even ran into a Valgrind bug. The reason why Valgrind could not show the function names belonging to the stack frames is because PHP had already unloaded the shared extensions from memory.

I often found myself compiling the extensions into PHP statically so that there was nothing to unload for PHP, but that was becoming annoying. So instead I added a patch to PHP that prevents the shutdown sequence from actually unloading the modules. You can trigger this behaviour by setting the ZEND_DONT_UNLOAD_MODULES environment variable before running your script:

# export ZEND_DONT_UNLOAD_MODULES=1
# valgrind --leak-check=full --show-reachable=yes php -r 'echo "Hello World\n";'

Without ZEND_DONT_UNLOAD_MODULES exported, my Valgrind output shows a block like:

# unset ZEND_DONT_UNLOAD_MODULES
# valgrind --leak-check=full --show-reachable=yes php -r 'echo "Hello World\n";'
...
==25829== 8 bytes in 1 blocks are indirectly lost in loss record 2 of 21
==25829==    at 0x4C25E84: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==25829==    by 0xCE440DC: ???
==25829==    by 0xCE44316: ???
==25829==    by 0xCE44368: ???
==25829==    by 0xCBEE55F: ???
==25829==    by 0xCBD3F87: ???
==25829==    by 0x949A85: zend_activate_modules (zend_API.c:2285)
==25829==    by 0x8B5EBC: php_request_startup (main.c:1491)
==25829==    by 0xA83D60: do_cli (php_cli.c:954)
==25829==    by 0xA84F7B: main (php_cli.c:1356)
...

And with ZEND_DONT_UNLOAD_MODULES exported, it shows instead:

# export ZEND_DONT_UNLOAD_MODULES=1
# valgrind --leak-check=full --show-reachable=yes php -r 'echo "Hello World\n";'
...
==25824== 8 bytes in 1 blocks are still reachable in loss record 2 of 30
==25824==    at 0x4C25E84: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==25824==    by 0xCE440DC: event_base_priority_init (in /usr/lib/libevent-1.4.so.2.1.3)
==25824==    by 0xCE44316: event_base_new (in /usr/lib/libevent-1.4.so.2.1.3)
==25824==    by 0xCE44368: event_init (in /usr/lib/libevent-1.4.so.2.1.3)
==25824==    by 0xCBEE55F: zm_activate_http_request_pool (http_request_pool_api.c:58)
==25824==    by 0xCBD3F87: zm_activate_http (http.c:373)
==25824==    by 0x949A85: zend_activate_modules (zend_API.c:2285)
==25824==    by 0x8B5EBC: php_request_startup (main.c:1491)
==25824==    by 0xA83D60: do_cli (php_cli.c:954)
==25824==    by 0xA84F7B: main (php_cli.c:1356)
...

As you can see all the symbols are now nicely resolved. This patch is part of the upcoming PHP 5.4, but applies to PHP 5.2 and PHP 5.3 as well.

Autumn Conferences

Wed, 22 Sep 2010 20:14:00 +0000

Autumn Conferences

London, UK

Wednesday, September 22nd 2010, 21:14 BST

Autumn started yesterday, and that also marks the start of the 2010 autumn conference schedule. In the next two and a half months I'll be attending and speaking at several conferences. Besides some of the older topics (PHP's date/time extension, and Xdebug) I will also give a few presentations on "Geolocation and Maps with PHP".

The new presentation covers the use of geolocation information with PHP. I will start with a theoretical background of all the services, mapping systems etc involved. After that, I will discuss on how to use geo-services from PHP. From obtaining of geo-location information through databases and services, mapping services to visualize geo-aware data to geo-aware data storage, manipulation and querying.

The talks are:

Debugging with Xdebug (WebExpo, Prague, September 24th)
Geolocation and Maps with PHP (PHP NorthWest, Manchester, October 9th)
PHP Inside (IPC, Mainz, October 12th)
Geolocation and Maps with PHP (IPC, Mainz, October 13th)
Advanced Date/Time handling with PHP (ZendCon, San Jose, November 2nd)
Haystacks and Needles (Forum PHP, Paris, November 10th)
Geolocation and Maps with PHP (Forum PHP, Paris, November 10th)

I am looking forwards to meeting all my old friends, and hoping to meet some new ones too. Come and say hello!

64-bit integers in MongoDB

London, UK

Monday, August 9th 2010, 14:23 BST

The current project that I'm working on relies heavily on MongoDB, a bridge between key-value stores and traditional RDBMS systems. Users in this project are identified by their Facebook UserID, which is a "64-bit int datatype". Unfortunately, the MongoDB PHP Driver only had support for 32-bit integers causing problems for newer users of Facebook. For those users, their nice long UserID was truncated to only 32 bits which didn't quite make the application work.

MongoDB stores documents internally in something called BSON (Binary JSON). BSON has two integer types, a 32-bit signed integer type called INT and a 64-bit signed integer type called LONG. The documentation of the MongoDB PHP driver on types says (or used to say, depending on when you're reading this) that only the 32-bit signed integer type is supported because "PHP does not support 8 byte integers". That's not quite true. PHP's integer type supports 64-bit on the platforms where the C-data type long is 64 bits. That is generally on every 64-bit platform (where PHP is compiled for 64 bits); except on Windows, where the C-data type long is always only 32 bits.

Whenever a PHP integer is sent to MongoDB, the driver would use the 32 least significant bits to store the number as part of the document. The example here shows what happens (on a 64-bit platform):

selectCollection('test', 'inttest');
$c->remove(array());

$c->insert(array('number' => 1234567890123456));

$r = $c->findOne();
echo $r['number'], "\n";
?>

shows:

int(1015724736)

In binary:

1234567890123456 = 100011000101101010100111100100010101011101011000000
      1015724736 =                      111100100010101011101011000000

Truncating data is obviously not a very good idea. In order to address this issue we could just allow for the native PHP integer type to be used when storing data from PHP into MongoDB. But instead of changing how the MongoDB driver works by default I've added the new setting mongo.native_long — simply because otherwise we might be breaking applications. With the mongo.native_long setting enabled, we see the following result instead of the outcome of the script above:

insert(array('number' => 1234567890123456));

$r = $c->findOne();
var_dump($r['number']);
?>

This script shows:

int(1234567890123456)

On 64-bit platforms, the mongo.native_long setting allows for 64-bit integers to be stored in MongoDB. The MongoDB data type that is used in this case is the BSON LONG, instead of the BSON INT that is used if this setting is turned off. The setting also changes the way how BSON LONGs behave when they are read back from MongoDB. Without mongo.native_long enabled, the driver would convert every BSON LONG to a PHP double which results in the loss of precision. You can see that in the following example:

insert(array('number' => 12345678901234567));

ini_set('mongo.native_long', 0);
$r = $c->findOne();
var_dump($r['number']);
?>

This script shows:

float(1.2345678901235E+16)

On 32-bit platforms, the mongo.native_log setting changes nothing for storing integers in MongoDB: the integer is stored as a BSON INT as before. However, when the setting is enabled and a BSON LONG is read from MongoDB a MongoCursorException is thrown alerting you that the data could not be read back without losing precision:

MongoCursorException: Can not natively represent the long 1234567890123456 on this platform

If the setting is not enabled, a BSON LONG is converted to a PHP float in order to avoid breaking backwards compatibility with the current behaviour.

Although the mongo.native_long settings allows for 64-bit support on 64-bit platforms, it doesn't provide much for 32-bit platforms except preventing loss of precision while reading BSON LONGs—and then by just throwing an exception.

As part of making 64-bit integers work reliably with MongoDB, I've also added two new classes: MongoInt32 and MongoInt64. These two classes are simple wrappers around a string representation of a number. They are instantiated like this:

You can use those objects in normal insert and update queries just like normal numbers:

selectCollection('test', 'inttest');
$c->remove(array());

$c->insert(array(
        'int32' => new MongoInt32("1234567890"),
        'int64' => new MongoInt64("12345678901234567"),
));

$r = $c->findOne();
var_dump($r['int32']);
var_dump($r['int64']);
?>

Which shows:

int(1234567890)
float(1.2345678901235E+16)

As you can see, nothing is changed with how values are returned. A BSON INT is still returned as an integer, and a BSON LONG as a double. If we turn on the mongo.native_long setting then the BSON LONG that was stored through the MongoInt64 class is returned as a PHP integer on 64-bit platforms, and a MongoCursorException is thrown on 32-bit platforms.

In order to received 64-bit integers back from MongoDB on 32-bit platforms, I've added another setting: mongo.long_as_object. This will (on any platform) return a BSON LONG as stored in MongoDB as a MongoInt64 object. The following script demonstrates that:

selectCollection('test', 'inttest');
$c->remove(array());

$c->insert(array(
        'int64' => new MongoInt64("12345678901234567"),
));

ini_set('mongo.long_as_object', 1);
$r = $c->findOne();
var_dump($r['int64']);
echo $r['int64'], "\n";
echo $r['int64']->value, "\n";
?>

This script outputs:

object(MongoInt64)#7 (1) {
  ["value"]=>
  string(17) "12345678901234567"
}
12345678901234567
12345678901234567

The MongoInt32 and MongoInt64 classes implement __toString() so that their values can be echoed. You can only get their values out as strings. Please be aware that MongoDB is type-sensitive, and will not treat a number contained in a string the same way as a number that's just a number. This script shows this (on a 64-bit platform):

selectCollection('test', 'inttest');
$c->remove(array());

$nr = "12345678901234567";
$c->insert(array('int64' => new MongoInt64($nr)));

$r = $c->findOne(array('int64' => $nr)); // $nr is a string here
var_dump($r['int64']);
$r = $c->findOne(array('int64' => (int) $nr));
var_dump($r['int64']);
?>

Which shows:

NULL
int(12345678901234567)

The following tables summarises all the different conversions regarding integers, depending on which settings are enabled:

PHP to MongoDB on 32-bit systems:

From PHP	Stored in Mongo
From PHP	native_long=0	native_long=1
1234567	INT(1234567)	INT(1234567)
123456789012	FLOAT(123456789012)	FLOAT(123456789012)
MongoInt32("1234567")	INT(1234567)	INT(1234567)
MongoInt64("123456789012")	LONG(123456789012)	LONG(123456789012)

PHP to MongoDB on 64-bit systems:

From PHP	Stored in Mongo
From PHP	native_long=0	native_long=1
1234567	INT(1234567)	LONG(1234567)
123456789012	garbage	LONG(123456789012)
MongoInt32("1234567")	INT(1234567)	INT(1234567)
MongoInt64("123456789012")	LONG(123456789012)	LONG(123456789012)

Mongo to PHP on 32-bit systems:

Stored in Mongo	Returned to PHP as
	long_as_object=0		long_as_object=1
	native_long=0	native_long=1	long_as_object=1
INT(1234567)	int(1234567)	int(1234567)	int(1234567)
LONG(123456789012)	float(123456789012)	MongoCursorException	MongoInt64("123456789012")

Mongo to PHP on 64-bit systems:

Stored in Mongo	Returned to PHP as
	long_as_object=0		long_as_object=1
	native_long=0	native_long=1	long_as_object=1
INT(1234567)	int(1234567)	int(1234567)	int(1234567)
LONG(123456789012)	float(123456789012)	int(123456789012)	MongoInt64("123456789012")

Conclusion

Getting 64-bit support right with MongoDB can be tricky as we've seen. My recommendations would be to use just mongo.native_long=1 if you only deal with 64-bit platforms for your code. In this case, every integer number that you put into MongoDB will also come out as an integer number; with 64-bit integers supported.

If you however have to deal with 32-bit platforms (remember, that includes a 64-bit Windows build of PHP) then you can not reliably use just PHP's integer types and you have to use the MongoInt64 class. This comes with the restriction that you have to deal with numbers in strings for initialisation. You also need to be aware that the MongoDB shell regards all numbers as floating point numbers, and that it can not represent 64-bit integer values. Instead they will show up as floating point numbers. Do not attempt to modify those numbers on the shell, as that could change the type.

So with the script:

selectCollection('test', 'inttest');
$c->remove(array());

$c->insert(array('int64' => new MongoInt64("123456789012345678")));

The MongoDB shell mongo behaves like:

$ mongo
MongoDB shell version: 1.4.4
url: test
connecting to: test
type "help" for help
> use test
switched to db test
> db.inttest.find()
{ "_id" : ObjectId("4c5ea6d59a14ce1319000000"), "int64" : { "floatApprox" : 123456789012345680, "top" : 28744523, "bottom" : 2788225870 } }

Of course, when fetching data through a driver that supports 64-bit integers you get the proper result:

ini_set('mongo.long_as_object', 1);
$r = $c->findOne();
var_dump($r['int64']);
?>

Which shows:

object(MongoInt64)#7 (1) {
  ["value"]=>
  string(18) "123456789012345678"
}

The new functionality as outlined in this article is part of the 1.0.9 mongo release that's available through PECL with pecl install mongo. Good luck with your 64-bit integers!

Xdebug 2.1

Tue, 29 Jun 2010 14:17:00 +0000

Xdebug 2.1

London, UK

Tuesday, June 29th 2010, 15:17 BST

I've just released Xdebug 2.1. This new release contains all the bug fixes and new features that have been developed over the past three years. It provides full PHP 5.3 support, but it no longer supports PHP versions below 5.1.

A description of all the new features follows:

Error Collection

I've added xdebug_start_error_collection(), xdebug_stop_error_collection() and xdebug_get_collected_errors(), which allow you to collect all notices, warnings and error messages that Xdebug generates from PHP's error_reporting functionality so that you can output them at a later point in your script by hand.

Header Setting Interception

All functions that set HTTP headers such as with header() and setcookie() are now intercepted by Xdebug. The intercepted headers are stored internally in an array that can be retrieved by calling the xdebug_get_headers() function. This is very useful in cases where you need to test certain functionality that sets headers somewhere deep in code. This function is also used in eZ Components' test suite to test whether the correct HTTP headers are set in the MvcTools component.

Variable Assignment Tracing

The new setting xdebug.collect_assignments allows you to record changes to variables in scripts to trace files. I've already written more about it in Variable tracing with Xdebug.

"Scream" Support

The scream PECL extension disables the @ (shut-up) operator to actually see all notices, warnings and errors that PHP generates. The scream extension's functionality have been duplicated as Xdebug's xdebug.scream php.ini setting. Why disabling the @-operator is a good thing, I've already outlined in Five reasons why the shut-op operator (@) should be avoided.

Additions for Stack Traces

All HTML containers generated by Xdebug (error messages, xdebug_var_dump() output) now contain a CSS class for easier styling.

The new setting xdebug.file_link_format allows you to turn any file/line link in Xdebug's output to be decorated to a link with a specific format. For example you can set it to txmt://open/?file://%f&line=%l for opening files directly in Textmate, or gvim://%f@%l with some shell script for gvim and Firefox.

Remote Debugging

The new setting xdebug.remote_connect_back, contributed by Lucas Nealan and Brian Shire, allows Xdebug to try to make a debugging connection to the IP address from which the browser request came from. This setting is an additional solution for debugging with multiple users.

The old gdb and php3 remote debugging engines have been removed.

Overloaded var_dump

The newly introduced setting xdebug.overload_var_dump can be used to turn off Xdebug's default behavior of overriding PHP's var_dump() function with the xdebug_var_dump() function that uses pretty HTML for formatting a variable structure. It does not stop xdebug_var_dump() from working however.

Donations

You can probably imagine that writing and support Xdebug is a time-consuming effort. It's a tool that saves a lot of work during developement of applications, as well while debugging. If you think Xdebug is valuable for you, perhaps you would like to consider a donation. All donations are greatly appreciated and support future development and availablity of Xdebug.

If you want to sponsor a (new) feature in Xdebug, feel free to contact me as well.

Download and Support

The new version of Xdebug can be downloaded through the download page. A list with all changes can be found on the updates page. Support information can be found on the support page, and issues and feature requests can be filed in the bug tracker. Before asking for support, please have a look at the FAQ first.

Find my Xdebug download wizard

Mon, 03 May 2010 09:00:00 +0000

Find my Xdebug download wizard

London, UK

Monday, May 3rd 2010, 10:00 BST

Installing Xdebug seems to be a problem for some users. The main trouble is for Windows users that don't know which file to download. Over the past few days I've worked on a wizard that analyses phpinfo()'s output and for Windows users suggests:

which file to download
where to put the file
which php.ini file to modify, and how

For Unix users, it explains:

which source tarball to download
how to compile
which php.ini file to modify, and how

I hope this makes it a bit easier to get going with Xdebug. Next up (hopefully this week) is release candidate 2 for Xdebug 2.1.0.

PHP and Ordnance Survey Mapping

Tue, 27 Apr 2010 13:27:00 +0000

PHP and Ordnance Survey Mapping

London, UK

Tuesday, April 27th 2010, 14:27 BST

About a month ago, Ordnance Survey opened up some of their data for public consumption under a brand called OpenData. The data is licenced under a "Creative Commons Attribute"-like license. One of the data sets they provide is Code-Point Open and provides a dataset "that contains postcode units, each of which have a precise geographical location". I've always been a bit of a map-geek, and have always geotagged my pictures and some of my tweets. So I was wondering what cool thing I could do with this newly released data. I decided to map all the postcodes onto the UK map where more postcodes for a specific place would create a "lighter" colour. Each postcode has on average about 15 addresses, so in more densely populated areas you have more "postcodes-per-area". Doing this wasn't very difficult and it resulted in the following map:

You can very clearly see the more densely populated areas such as London.

This generated postcode-density map I wanted to overlay on a real map, such as OpenStreetMap provides. When I tried to align the image that I generated with the OSM map, I ended up with this:

Obviously, the maps don't align. In order to fix this, I ended up learning a lot more about map projections.

The Code-Point Open data uses the UK's National Grid to store location data in. The National Grid consists of 100*100 km squares that are then further subdivided into smaller squares creating grid references such as TQ3012780512, or the numerical version 530128 180512. TQ translates into the "hundred thousands" of the Eastings and Northings according to the grid that you can see here. In this case, it specifies a point 530 128 meters East and 180 512 meters North of the origin. (If you work it out, you'll end up in London).

OpenStreetMap uses a Mercator projection to visualize maps. The Mercator projection is a cylindrical map projection, and it distorts the size and shape of large objects, as the scale increases from the Equator to the poles. Therefore it only works from about 85°N to 85°S (why it is 85° only be came clear after doing all the maths for it). Google Maps uses the same projection.

There were a few challenges plotting the Code-Point Open data on an OpenStreetMap map:

The National Grid coordinates need to be converted to Latitude/Longitude pairs
Latitude/Longitude pairs need to be mapped to pixels to align with the Mercator projection of OpenStreetMap maps.

For converting National Grid references to Latitude/Longitude pairs I found some JavaScript code. I converted this code to PHP and to verify whether my code was working properly, I used this site.

Converting Latitude/Longitude pairs to the pixel coordinates of OSM required a bit of maths. OSM provides maps in different zoom levels, from 2 to 17. Each zoom level having twice the amount of pixels horizontally and vertically. Zoom level 2 has 4 times 4 tiles, where each tile is 256*256 pixels. At zoom level two, the whole world fits into 1024*1024 pixels. The number of pixels for each axis for a specific zoom-level can be calculated by:

pow(2^zoom) * 256

For a zoom level of 7 that makes 16384 pixels in each direction. To convert a longitude (in the range -180° to 180°) we can simply apply:

x = ((lon + 180) / 360) * (pow(2^zoom) * 256)

For a Longitude of 0.003117° E at zoom-level 13 that turns out to be pixel 1048594.

Longitude conversions are more difficult due to the Mercator projection itself. To convert we apply:

y = ((atanh(sin(deg2rad(-lat))) / π) + 1) * (pow(2^(zoom-1)))

For a Latitude of 51.502817° N at zoom-level 13 that turns out to be pixel 697399.

Creating an image for the whole world at zoom level 13 is impractical as it is 2097152*2097152 pixels and downloading all the 67 million tiles for this zoom level is probably not liked by the OSM people either. So instead we take a cut out for a specific area only. For the UK (61.37°N -9.49° W, 49.76°N +3.63°E) at zoom level 7 we end up with a 1536x2048 map with the North-Western pixel being 15360,9216. On this map, we can draw the latitudes and longitudes as well as the National Grid lines (full image is on flickr):

The only thing left to do now, is to map the postcode density information to the map. I picked zoom level 6 for this, and the result is (after cropping it to 640 pixels width):

As you can see, this is perfect fit to the outlines of the country. But unfortunately, when we look very closely at the plotted map data, for example for the NW10 3 postcodes, we notice that the mapping is slightly off. The blue dots are what we plotted, and the red dots are what the locations should have been:

The reason for this is that when we converted the National Grid locations to Latitude/Longitude pairs to plot on the OSM maps, I forgot to take into account the different Datums that are used in the projections. The Earth is not a perfect sphere, and an approximation of the ellipsoid of the whole Earth is not necessarily the best fitting for a specific area such as the UK. Therefore, the National Grid uses the OSGB36 Datum which fits more closely to the UK, where as OpenStreetMap uses the WGS84 Datum that is also used by GPS. The Ordnance Survey Ireland has a more thorough explanation on their site. As you can see above, using the wrong Datum can mean locations can be off. In our example about 100 meters. Converting between different Datums is possible, albeit processor intensive.

After I figured out all the maths for this, the only problem that remains that implementing those algorithms in PHP is show—calculating all the positions from the 1.6 million postcode locations takes up to 10 minutes. This is why I am not presenting any code yet. I am planning to implement all the necessary calculations in a PHP extension to speed up the calculations

VLD released

Mon, 12 Apr 2010 20:57:00 +0000

VLD released

London, UK

Monday, April 12th 2010, 21:57 BST

I just released a new version of VLD through PECL. This release features the path analytics code that I've outlined in an earlier article. There are also a few fixes for PHP's current trunk, as well as the possibility to format output slightly like a CSV file. That feature was used to make this documentation of opcodes.

The path analysis algorithm that VLD now has, can possibly also be used to get path/branch coverage working within Xdebug's code-coverage framework. This is however not a minor task.

Xdebug 2.1.0 RC1 released

Tue, 06 Apr 2010 13:41:00 +0000

Xdebug 2.1.0 RC1 released

Schiphol, Netherlands

Tuesday, April 6th 2010, 15:41 CEST

As earlier announced in my April Fools' announcement and on Xdebug's twitter feed, I'm now releasing Xdebug 2.1.0RC1. This is the first Release Candidate and it addresses a number of issues found in the latest beta. As I usually do in release announcements, I will also be introducing a new feature in Xdebug 2.1 in this one.

In an article that I wrote a few months ago, I illustrated a few different reasons on why the shut-op operator (@) should be avoided. Although there are a few cases where it simple can not be avoided if you want to write notice-free code, the main reason why this operator should be avoided, is because it makes debugging quite a bit more difficult if errors are hidden.

Xdebug is a debugging aid to make debugging easier. So from version 2.1.0 there is a new configuration setting — xdebug.scream — that simply disables the effects of the @ operator. That means that all errors will be shown, of course, still listening to the error_reporting and display_errors settings. This functionality is not a novel thing, and the idea was lifted from the PECL scream package. However, I think this functionality fits in very well with all the other things that Xdebug has to offer.

Just to mention it once more, I have no intention of making Xdebug closed source. I spend a considerable amount of time in adding new features and addressing issues, so I would appreciate it that if consider a donation in case Xdebug has saved you time and effort in debugging your application. If you have comments or suggestions to improve Xdebug, please write to the mailinglist.

First release of the D-Bus extension

Tue, 06 Apr 2010 09:33:00 +0000

First release of the D-Bus extension

London, UK

Tuesday, April 6th 2010, 10:33 BST

A few days ago I made the first beta release of the D-Bus extension that I have been working on for a while. D-Bus is a message bus system, a simple way for applications to talk to one another. I started working on this because my cellphone, an OpenMoko FreeRunner. This cellphone (or rather, a mobile Linux computing device) implements freesmartphone.org APIs to talk to all the hardware that is available on the device. This includes GSM and SIM card interfaces, but also GPS, Audio and PIM services.

However, many other applications on the Linux desktop speak D-Bus. This includes system services such as the notification daemon, the screen saver and hardware plug-in detection as well as desktop applications such as Pidgin and Empathy.

I've given a presentation on the extension at the PHP London conference earlier this year, and will also be speaking on this subject as part of the "PHP Inside" and "PHP on the D-Bus" talks at Tek-X (Chicago, May 18-21), International PHP Conference Spring (Berlin, May 30-June 2) and the Dutch PHP Conference (June 10-12).

Xdebug no longer Open Source

Thu, 01 Apr 2010 09:19:00 +0000

Xdebug no longer Open Source

London, UK

Thursday, April 1st 2010, 10:19 BST

This was obviously an April Fools' joke, Xdebug will continue to stay Open Source and free of charge.

Later today I will be releasing the first Release Candidate of Xdebug 2.1. With regret I have to announce that from this release, I will no longer provide Xdebug as an open source debugger extension. I've noticed that there are other projects making money of Xdebug's success while I only receive donations from a group of awesome supporters.

Xdebug will continue to be available for benefactors on a limited number of platforms that I have access to (Linux 32/64bit and Windows). Please refer to the FAQ for further information.

Storing Date/Times in Databases

Mon, 29 Mar 2010 15:47:00 +0000

Storing Date/Times in Databases

London, UK

Monday, March 29th 2010, 16:47 BST

After my talk during ConFoo on Advanced Date/Time Handling I received a question about whether the UTC-offset, together with the date/time in years, months, days, hours, minutes and seconds, was enough for storing a date/time in a database and still being able to do calculations with this. The answer to this question was no, but it lead to an even more interesting discussion about what would be enough to store an accurate date/time in a database.

Firstly let me explain why storing a UTC offset is not adequate at all for doing any sort of calculations. Right now (March 25th, 2010), Montreal is on the same time as Santiago (in Chile) with both a UTC offset of -4 hours. This means that for the same date/time (2010-03-25 19:03) the same timestamp is generated (1269558180). However if we want to add eight months (2010-11-25 19:03) to the current time for each of those locations then the timestamps should be different — 1290729780 for Montreal and 1290722580 for Santiago. It turns out that neither of them is exactly a full 24 hour difference from 2010-03-25 19:03 as is shown the following script:

format( 'U' );
$santiagoNow = date_create( "{$s} America/Santiago" )->format( 'U' );
$s = '2010-11-25 19:03';
$montrealThen = date_create( "{$s} America/Montreal" )->format( 'U' );
$santiagoThen = date_create( "{$s} America/Santiago" )->format( 'U' );

$hour = 60 * 60;
$day = 24 * $hour;

echo "Montreal:\n",
    $montrealNow, " - ",
    $montrealThen, "\n",
    ( ( $montrealThen - $montrealNow ) % $day ) / $hour, "\n\n";

echo "Santiago:\n",
    $santiagoNow, " - ",
    $santiagoThen, "\n",
    ( ( $santiagoThen - $santiagoNow ) % $day ) / $hour, "\n\n";

?>

which gives the output:

Montreal:
1269558180 - 1290729780
1

Santiago:
1269558180 - 1290722580
23

From this we see that Montreal has an extra hour and Santiago has an hour less. This is because Montreal changed from Daylight Savings Time to normal time and Santiago, being on the southern hemisphere, moved from normal time to Daylight Savings time.

Now if we had only the timestamp (1269558180) and the UTC-offset, the only thing we could do would be to advance a specific number of days. In this case, there are 245 days, which makes (245 * 24 * 60 * 60 = ) 21168000 seconds. If we add this to the original timestamp, we end up at 1290726180 which corresponds to 2010-11-25 18:03 Montreal time, or 2010-11-25 20:03 Santiago time. Neither of them being the correct 2010-11-25 19:03. Because from the UTC offset of 4 hours we don't know whether we're in Montreal or Santiago, we can conclude that with just this UTC offset and the original timestamp we can't calculate the timestamp of something that's 8 months in the future. Storing the UTC-offset can not change this fact either.

In order to to proper calculations, you need to keep information about the timezone itself. In PHP timezones are identified with identifiers such as America/Montreal. If we store those alongside the timestamps, we can do the proper calculations. The following example demonstrates that:

setTimeZone( new DateTimeZone( $tzid ) );
$d->modify( '+8 months' );

echo $d->format( 'Y-m-d H:i' ), "\n";
?>

which gives the output: 2010-11-25 19:03.

This seems to work, but unfortunately, even the approach of storing the timestamp and timezone identifier is not going to work correctly under certain circumstances. In order to find out why, we have to go back to January this year.

Imagine that in January this year—January 15th to be precise—we run the following script to determine the timestamp of a date/time two months in the future:

format( 'U' );

$d = new DateTime( "@$ts" );
$d->setTimeZone( new DateTimeZone( $tz ) );
$d->modify( '+2 months' );

echo $d->format( 'U Y-m-d H:i' ), "\n";
?>

This returns the timestamp 1268664060 for the date/time 2010-03-15 10:41:00.

Now skip forwards to the current date and time. If we used this calculated timestamp and the timezone identifier America/Santiago today to generate a date/time string, we would however get a different output. The following example shows this:

setTimeZone( new DateTimeZone( $tz ) );

echo $d->format( 'Y-m-d H:i:s' ), "\n";
?>

which gives the output: 2010-03-15 11:41:00 (and not 2010-03-15 10:41:00).

The difference in output is not a bug, but is caused because some countries change Daylight Savings Time rules quite frequently. In this case Chile decided (on March 4th) that instead of going forwards to DST at midnight March 14th, they will delay that to April 4th. When we ran the code on January 15th, the rules still thought that March 15th would already be on normal time again, outside of DST. But running the code now, with the updated rule set, we find that DST is still in effect until April 4th. The following example shows the transitions from/to DST for Santiago in 2010:

getTransitions(
    strtotime( '2010-01-01 00:00 UTC' ),
    strtotime( '2010-12-31 00:00 UTC' )
);

foreach ( $transitions as $t )
{
    echo $t['time'], ' ', $t['offset'] / 3600, ' ', $t['abbr'], "\n";
}
?>

If we run this script with PHP 5.3.1 or 5.3.2—which still have the old incorrect rule set—the output is:

2010-01-01T00:00:00+0000 -3 CLST
2010-03-14T03:00:00+0000 -4 CLT
2010-10-10T04:00:00+0000 -3 CLST

In the SVN repository, the rule set has been updated, so snapshots of PHP 5.3-dev have the correct rules and the script will show:

2010-01-01T00:00:00+0000 -3 CLST
2010-04-04T03:00:00+0000 -4 CLT
2010-10-10T04:00:00+0000 -3 CLST

This leads to the conclusion that storing timestamps and timezone identifiers is not good enough either, unless you want an exact point in time, as opposed to the more expected date/time in a location. So how should you store the latter then? Basically, in the same way that DateTime objects are serialized in PHP 5.3. Let us imagine again, that the following code is run on January 15th again:

modify( '+2 months' );
$s = $d->format( 'Y-m-d H:i:s' );
$ts = $d->format( 'U' );
echo "$s (ts=$ts)\n";
?>

which gives the output 2010-03-15 10:41:00 (ts=1268664060). The next example is run with a recent rule set (such as in PHP SVN) today:

format( 'Y-m-d H:i:s' );
$ts = $d->format( 'U' );
echo "$s (ts=$ts)\n";
?>

which gives the output 2010-03-15 10:41:00 (ts=1268660460). As you can see, now the date/time itself is correct, although there is a different timestamp. PHP's DateTime serialisation does something similar:

setTimeZone( new DateTimeZone( 'America/Santiago' ) );
var_dump( $d );
?>

which gives the output:

object(DateTime)#1 (3) {
  ["date"]=>
  string(19) "2010-03-29 11:32:08"
  ["timezone_type"]=>
  int(3)
  ["timezone"]=>
  string(16) "America/Santiago"
}

For things to work correctly, you need to have an up-to-date rule set. PHP versions are not released as often as the timezonedb (sometimes more than 20 times a year) and to address this issue you can install the pecl extension timezonedb with pecl install timezonedb.

To store information in a database, I would use a char column-type to store the whole America/Santiago timezone identifier and another char column-type to store the date/time (in the yyyy-mm-dd hh:ii:ss format). Alternatively, you can pick a 'datetime' column-type, as long as that type ignores timezones altogether. For MySQL that is the DATETIME column-type, and for PostgreSQL the TIMESTAMP or TIMESTAMP WITHOUT TIME ZONE column-types.

Databases rarely handle timezones, daylight savings time and rule changes correctly, so avoid the database specific functionality all together. Using either a char or a timezone-less 'datetime' column-type would still allow you to sort and by using a 'datetime' column-type you can even do calculations.

Happy summer time!

Available for PHP Extension Writing

Mon, 15 Mar 2010 12:28:00 +0000

Available for PHP Extension Writing

London, UK

Monday, March 15th 2010, 12:28 GMT

Slightly more than a month ago I left my job as team-lead of eZ Components at eZ Systems behind, to focus on something new. During the past month I've been contemplating about what to do next and realized that I do not want to take on a new full-time position right away. Instead I will be available to work on (custom) PHP extensions and internals related issues. Extensions are a great way around PHP's limitations and performance issues.

As first project I am working on a "QuickHash" extension for StumbleUpon. This extension circumvents PHP's hefty memory (and performance) overhead by providing more specific data structures. The extension currently implements integer sets and integer to integer hashes. I am now adding integer to string hashes and string to integer hashes. The QuickHash extension will be released under the PHP Licence and I will dedicate another post to it later.

If, like StumbleUpon, you are also interested in having work done on PHP or a specific extensions feel free to contact me. I'd be happy to discuss things with you.

Xdebug 2.1.0beta3 released

Sat, 27 Feb 2010 23:57:00 +0000

Xdebug 2.1.0beta3 released

London, UK

Saturday, February 27th 2010, 23:57 GMT

I've just released Xdebug 2.1.0beta3 which includes a few crash bugs as well as the issue that headers sent from PHP scripts are not actually set.

You can find the full changelog here and get the latest version from the download page.