Derick Rethans - tag: opensource

Valgrinding shared modules

Mon, 08 Aug 2011 08:56:00 +0000

Valgrinding shared modules

Monday, August 8th 2011, 09:56 BST

Over the past year I've been writing various commercial (more about that later) and open source PHP extensions (such as QuickHash, more about that later too). Most of the time they are shared extensions that are not part of PHP. While testing whether I correctly free all memory with Valgrind, I ran into the issue where I couldn't see the stack frames of where the memory leaks occurred in the extensions, and once I even ran into a Valgrind bug. The reason why Valgrind could not show the function names belonging to the stack frames is because PHP had already unloaded the shared extensions from memory.

I often found myself compiling the extensions into PHP statically so that there was nothing to unload for PHP, but that was becoming annoying. So instead I added a patch to PHP that prevents the shutdown sequence from actually unloading the modules. You can trigger this behaviour by setting the ZEND_DONT_UNLOAD_MODULES environment variable before running your script:

# export ZEND_DONT_UNLOAD_MODULES=1
# valgrind --leak-check=full --show-reachable=yes php -r 'echo "Hello World\n";'

Without ZEND_DONT_UNLOAD_MODULES exported, my Valgrind output shows a block like:

# unset ZEND_DONT_UNLOAD_MODULES
# valgrind --leak-check=full --show-reachable=yes php -r 'echo "Hello World\n";'
...
==25829== 8 bytes in 1 blocks are indirectly lost in loss record 2 of 21
==25829==    at 0x4C25E84: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==25829==    by 0xCE440DC: ???
==25829==    by 0xCE44316: ???
==25829==    by 0xCE44368: ???
==25829==    by 0xCBEE55F: ???
==25829==    by 0xCBD3F87: ???
==25829==    by 0x949A85: zend_activate_modules (zend_API.c:2285)
==25829==    by 0x8B5EBC: php_request_startup (main.c:1491)
==25829==    by 0xA83D60: do_cli (php_cli.c:954)
==25829==    by 0xA84F7B: main (php_cli.c:1356)
...

And with ZEND_DONT_UNLOAD_MODULES exported, it shows instead:

# export ZEND_DONT_UNLOAD_MODULES=1
# valgrind --leak-check=full --show-reachable=yes php -r 'echo "Hello World\n";'
...
==25824== 8 bytes in 1 blocks are still reachable in loss record 2 of 30
==25824==    at 0x4C25E84: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==25824==    by 0xCE440DC: event_base_priority_init (in /usr/lib/libevent-1.4.so.2.1.3)
==25824==    by 0xCE44316: event_base_new (in /usr/lib/libevent-1.4.so.2.1.3)
==25824==    by 0xCE44368: event_init (in /usr/lib/libevent-1.4.so.2.1.3)
==25824==    by 0xCBEE55F: zm_activate_http_request_pool (http_request_pool_api.c:58)
==25824==    by 0xCBD3F87: zm_activate_http (http.c:373)
==25824==    by 0x949A85: zend_activate_modules (zend_API.c:2285)
==25824==    by 0x8B5EBC: php_request_startup (main.c:1491)
==25824==    by 0xA83D60: do_cli (php_cli.c:954)
==25824==    by 0xA84F7B: main (php_cli.c:1356)
...

As you can see all the symbols are now nicely resolved. This patch is part of the upcoming PHP 5.4, but applies to PHP 5.2 and PHP 5.3 as well.

Translating Twitter, part 2

Tue, 31 May 2011 08:04:00 +0000

Translating Twitter, part 2

London, UK

Tuesday, May 31st 2011, 09:04 BST

A while ago I wrote in an article about translating tweets in my client Haunt. For the translating itself I was using the Google Translate API, which has sadly be deprecated. Evil after all I suppose.

I've now rewritten my translation code to use the Bing Translation APIs instead. You need to register an API key (see http://www.bing.com/developers/appids.aspx) to be able to use the APIs. The APIs that I am using are fairly simple though.

For a simple translation, requesting http://api.microsofttranslator.com/V2/Http.svc/Translate?appId=[yourappid]&to=en&text=[yourtext] is all you need to do. It will auto-detect the language for you as well.

However, it does not return the detected language, so I had to resort to using two requests in order to reimplement the same functionality that I had before with the Google APIs. I also found that it was easier to use the Http and not the Ajax variant of the API. It requires using SimpleXML to get to the data, but at least you do not have to fight with the BOM (Byte-order mark) and quoting.

The full code looks like:

with as output:

string(2) "nl"
string(34) "A high tree catches a lot of wind."

Short URLs

Tue, 22 Feb 2011 09:43:00 +0000

Short URLs

London, UK

Tuesday, February 22nd 2011, 09:43 GMT

When using twitter, or other mobile devices where space is a premium, it is quite common to use shortened URLs. Many people will not be keen on typing http://www.openstreetmap.org/?lat=51.5073&lon=-0.1278&zoom=14&layers=M, and will prefer something like http://osm.org/go/euu4gY@w- instead 1.

The URLs for my articles are automatically generated from the title. There is a little function that uses translit's transliterate() function to ASCII-ify the titles. It transforms for example "pfcongrez, P2P Конференция, php|tek, PHP Vikinger, DPC and eZ Conference and Awards" to "pfcongrez-p2p-konferenciya-phptek-php-vikinger-dpc-and-ez-conference-and-awards" with code similar to:

$titleShort = transliterate( $fullTitle,
    array(
        'cyrillic_transliterate', 'lowercase_latin',
        'normalize_ligature', 'diacritical_remove',
        'normalize_punctuation', 'remove_punctuation',
        'spaces_to_underscore', 'compact_underscores'
    ),
    'utf8', 'us-ascii'
);

However, http://derickrethans.nl/pfcongrez-p2p-konferenciya-phptek-php-vikinger-dpc-and-ez-conference-and-awards.html is hardly a short URL. What we want is something short and descriptive. First of all, I thought it would work to generate short URLs automatically, following the following algorithm:

Strip out all chars that are not A-Z, a-z, 0-9, ., space and -.
Split the title up in words, splitting on space.
If the sanitized URL is shorter than 14 chars: join the words back together with a "-", otherwise:
Loop over each word, replacing each word with the first letter of the word, unless it is one of the important words (such as "php" or "xdebug").
Add a marker for the date, to ensure that short URLs are unique. The date marker is generated like:
1. Format the date as two digit year, ISO week and ISO weekday
2. Convert the created number to base 36 with:
```
php -r "echo date_create( '20090410' )->format( 'yWN' );"
09155

php -r 'echo base_convert("09155", 10, 36), "\n";'
72b
```

Our long title from above, then turns into pp-p-php-vdaecaa-72b. Although relatively short, it is still not a descriptive URL. To be honest, it's a bit difficult to come up with a short, descriptive URL for this title, so let us examine another one, of a more recent post.

The title "64-bit integers in MongoDB" is converted into "6iim-7yp", which is again short, but not very informative; and "Xdebug 2.1 Released" is converted into "xdebug-2.1-7x2", which does make sense.

Because I did not get very good results with automatically converting titles into sort URLs, I decided that for all new articles I will just define my own.

Now, once we have the short URLs, they need to be used. Instead of my long main domain http://derickrethans.nl, I registered a new one just for short URLs: http://drck.me to reduce the URL length even more. For the current article, the length of the URL shrinks from 39 to 26 characters. The short URL is also embedded in the HTML source with a rev=canonical tag and a rel=shortlink tag, such as:

This tag is meant to provide alternative URLs for the current one. For more information on rev=canonical see http://revcanonical.appspot.com/ or Chris Shiflett's entry, and for rel=shortlink see the microformats site. The shortlink specification also recommends the use of the HTTP Link: header:

Link: ; rel=shortlink

The two tags and the HTTP header make it possible for services that prefer an as short as possible URL to automatically discover a shorter version. Sadly, it doesn't see that the twitter home page does this. I hope they will add support for that in the future. From my side, I am going to implement this in Haunt, my twitter client build in PHP-GTK.

1
This actually uses a very clever form of shortening geo-locations: http://wiki.openstreetmap.org/wiki/Shortlink

Debugging Variables

Thu, 10 Feb 2011 09:12:00 +0000

Debugging Variables

London, UK

Thursday, February 10th 2011, 09:12 GMT

Sometimes you want to inspect the contents of variables more closely, by looking at its internal properties. For this, PHP has the debug_zval_dump function.

The internal representation of a PHP variable container (called zval), contains the type and value of a variable, but also whether it is a reference and what its refcount is. Due to PHP's copy-on-write policy, one specific zval container can be used by multiple variables at the same time as we will see in a bit.

The refcount field contains how many symbols (variable names) point to a zval, regardless of whether a reference set is used or not. For example, both the following examples will have as end-result: two symbols and one zval:

In order for PHP to know the difference, there is an extra flag on the zval: is_ref, which is set if you assign a variable by reference with =&. It is important to know which of the situations actually occurred because PHP needs to be able to do the correct thing when a third symbol comes into play. In the first situation, a third symbol can be associated with the same zval only if it is assigned by assignment. If the third symbol is assigned by reference with something like $c =& $b; then PHP needs to duplicate (split) the zval into two zvals because only symbol b and c are linked, whereas a is distinct. Without the separation PHP will not be able to know which of the two symbols are referenced (b and c), and which one is not part of the reference set (a).

The opposite happens in the second situation. If a third symbol is assigned by reference with something like $c =& $b; then PHP can just update the refcount on the zval. If the third symbol is assigned by value with $c = $b then PHP needs to split the zval so that a and b still point to the original zval, and c to the newly created one without having links to either a or b.

This brings us to the very simple rule for when PHP needs to split upon assignment:

If the type of assigning (by-value with = or by-ref with =&) matches the value of is_ref: just update refcount by adding 1.
If the type of assigning does not match the value of is_ref: split the zval.

Now let us get back to the original issue: the inspection of the internal structure of variable contents with debug_zval_dump. When a variable is passed to a function as an argument, an assignment to the function's local scope is made; just like an assignment in a function. Which means that the same splitting rules apply. The following are (mostly) equivalent regarding on what happens to the is_ref and refcount values:

and:

PHP's debug_zval_dump takes a variable as argument for analysis and is thus also bound to the same splitting rules as outlined earlier. This means that's not really well suited for dumping a zval's internal structure as it would always modify the zval. Besides adding 1 to the refcount field, it could also force a split resulting in unexpected output:

shows:

long(42) refcount(2)

Xdebug has a similar function to display a zval's internal data: xdebug_debug_zval. This function does requires not a variable to be passed as its argument, but instead requires a variable name to be passed in. With this, the manipulation of the zval is avoided and the proper values are shown:

which shows:

a: (refcount=1, is_ref=0)=42

This is what is expected, and as bonus, it also shows the is_ref flag. PHP's bug #53895 mentions a few issues with the debug_zval_dump function. Gustavo is quite right to say that passing the variable name by-ref is not going to work, and any form of this function where the variable is passed in is not going to work. Johannes however is also right that passing just the variable name as a string might not be good enough; as you can't directly dump the zval information for for example array keys. There is an Xdebug feature request for this functionality to be implemented however. If implemented, you should be able to debug the foo array key with something like:

 3.1415);
xdebug_debug_zval('a["foo"]');
?>

Now if I only had some spare time...

Translating Twitter

Wed, 05 Jan 2011 09:04:00 +0000

Translating Twitter

London, UK

Wednesday, January 5th 2011, 09:04 GMT

Note: Google deprecated the API used in this article. See the new article that uses Bing Translate instead.

Note: I've made an update on January 6th to this article after a comment by a reader.

As the author of Xdebug I am interested in finding out what people think of it, and whether they have problems or compliments. I've set-up a twitter account for Xdebug, @xdebug, and my twitter client Haunt also shows me all tweets with the search term xdebug.

However, sometimes I get tweets in a language I can't read; for example Brazilian Portuguese:

Debugando aplicações PHP com Xdebug e Eclipse PDT: http://bit.ly/ffJC4G

junichi_y

or Japanese:

@pomu0325 ありがとうございます！このXdebugの書き方と各場所を調べてたんですよ！こんなふうに書くんですね。

Ken

Once in a while, I would send these tweets through Google's language tools but then my friend Elizabeth tweeted:

Hey Lazyweb, is there a twitter client that lets me filter tweets by language?

Elizabeth Naramore

Instead of a manual copy and paste in into the language tools, I thought it'd be nice to embed it directly into the client when it is requested.

Sadly, tweets don't have a language associated with them, so the first step is to actually find out which language a tweet is in. Google provides a web service called "Language Detect". To use this service, you only have to query a specific URL containing the text you want to guess the language off. and parse the returned JSON structure. The Google website has an example which basically boils down to requesting the following URL: https://ajax.googleapis.com/ajax/services/language/detect?v=1.0&q=Hola,+mi+amigo

It returns the following JSON struct:

{
        "responseData":
        {
                "language":"es",
                "isReliable":false,
                "confidence":0.08829542
        },
        "responseDetails": null,
        "responseStatus": 200
}

If the responseStatus is 200, then it worked. responseData->language contains the found language, and responseData->isReliable/responseData/confidence describe how sure Google is that the language found is actually correct. The larger the text, the easier it is to find out of course. In this case, although the confidence is low, the language is guessed correctly: es, for Spanish.

Now we have the language, we can use another web service from Google to translate the text from the guessed language to our target language which in my case is English. This Translate service wants the text and a language pair for translations. Google suggests you add a key, and an userip, but this is not strictly necessary. The language pair has the format source-language-code|destination-language-code; which is in our case es|en. The service is again very simple to use as you can see in this example. It boils again down to requesting an URL, such as: https://ajax.googleapis.com/ajax/services/language/translate?v=1.0&q=Hola, mi amigo!&langpair=es|en

It returns the following JSON struct:

{
        "responseData":
        {
                "translatedText":"Hello, my friend!"
        },
        "responseDetails": null,
        "responseStatus": 200
}

If the responseStatus is 200, then it worked and responseData->translatedText contains the translated text.

A screenshot shows the translation feature of Haunt in action:

You can find the implementation here. Haunt also has a project page.

Analysing Colours in an Image

Thu, 25 Nov 2010 18:24:00 +0000

Analysing Colours in an Image

London, UK

Thursday, November 25th 2010, 18:24 GMT

For a project that I am working on, I had to figure out which colours are primarily used in an image so that it can be indexed for sorting. Obviously, there is only a small finite set of colours that we are interested in. This article shows on how I went about analysing an image to see which of those colours were primarily used in the image.

First of all, I had to define a palette that I wanted to index on. Then I needed to check the image to see which colours existed, and map those to my finite set of colours. Then we could count them, and select the ones that were used most.

Picking the palette of colours wasn't very simple especially because pink and purple are difficult to differentiate, as well as orange and brown. It required a little bit of effort, but we settled on the following colours:

Some of the colour names have multiple colour values attached to them; this was so that we can tune the analysing algorithm a little bit to provide better results.

In order to match the colours from the image to the palette, I needed to create a mapping algorithm. It's difficult to do this with RGB values because they don't have the natural properties that we humans use for colours. However, the HSV model works a lot better. A HSV colour 'wheel' looks like:

On the X-axis we have the hue — a value that describes which colour it is on a scale from 0 to 360 (°). On The Y-axis we have both the saturation — a value that describes the intensity of a colour and value — the value describing the lightness of a colour. The respective six blocks on the y-axis correspond with saturation values of 0.0, 0.2, 0.4, 0.6, 0.8 and 1.0, and each of the ten bands of each block with the value values of 0.0, 0.1, 0.2 ... 0.9 and 1.0.

Then I cheated and used The Gimp to change this image of the colour wheel to an indexed image, using the palette from above. The option for that is: Image -> Mode -> Indexed -> "Use Custom Palette" and then pick your palette. Don't forget to change it back to RGB before you save it though. The resulting image then looks like:

The script to draw this palette is:

Each square's coordinates represent a mapping between (rounded) H, S and V values. With a little script, we can generate a mapping table:

> 16) & 0xFF;
    $g = ($color >> 8) & 0xFF;
    $b = $color & 0xFF;
}

$img = imagecreatefrompng( '/tmp/hsv-normal-palette.png' );

$hFactor = 2;
$vFactor = 12;

echo "<" . "?php\nclass colorMap {\n\tpublic \$map;\n\n";
echo "\tfunction __construct() {\n\t\t\$this->map = array(\n";

for ( $h = 0; $h < 360; $h += 15 )
{
    for ( $s = 0; $s < 5; $s += 1 )
    {
        for ( $v = 0; $v < 10; $v += 1 )
        {
            $c = imagecolorat( $img, $h * $hFactor, $s * 120 + $v * $vFactor );
            colorToRgb($c, $r, $g, $b);

            printf( "\t\t\t'x%03x%01x%01x' => '#%02x%02x%02x',\n",
                $h, $s, $v, $r, $g, $b
            );
        }
    }
}

echo "\t\t);\t}\n}\n";
?>

It would be better if we had the algorithm to convert from a random HSV value to a HSV value matching the closest colour in the palette. That's something for a next stage though.

The mapping table is used to find out which colours of our palette are actually part of the image. To sample all pixels in an image would take too much time, so we restrict ourselves to 200x200 samples, making 40000 pixels.

For each sampled pixel we:

find the RGB values
convert the RGB values to HSV (unless it's transparent, then we skip it)
normalize the HSV values by rounding the values
map the rounded value with our mapping key to produce the RGB hex value from our palette
count how much of each of the RGB hex values we've gotten
create an array where the number of pixels normalised to each RBG hex value is larger than 5%

The HSV-to-normalized-key algorithm looks like:

$key = sprintf('x%03x%01x%01x',
    max(0, min(345, floor( $h / 15 ) * 15)),
    min(4, floor($s * 5)),
    min(9, floor($v * 10))
);

To visualise this process, I've created a before and after image:

Sadly, as you can see, our typical red phone box isn't quite detected as being red, but rather black and grey. For some reason the conversion from an arbitrary HSV value to a palette colour with The Gimp isn't quite as good as we've hoped. So instead of drawing a correct HSV colour wheel, I decided to boost the colours a little bit away from grey by modifying the S and V values. This caused the following change in the palette-drawing script:

$sR = 1 - ( (1-$s) * (1-$s) );
$vR = 1 - ( (1-$v) * (1-$v) * (1-$v) * (1-$v) );
colorComponent::hsvToRgb( $h, $sR, $vR, $r, $g, $b );

In a diagram, this looks like:

We use this new algorithm, to redraw a new HSV wheel:

And with The Gimp create a new palette:

This new palette we then scan to create the color map out off. With this new color map we re-analyse the image to count our primary image colours. The result is then:

Which tells us that red was the most prevalent colour with over 50% of the sampled pixels (13916 + 7822 out of 40000), grey being second with 35% and black with 19%. Brown, orange and green also make up more than 5% of the total amount of sampled pixels.