Derick Rethans - tag: openstreetmap

OpenStreetMap: A Year of Edits

Sun, 01 Jan 2012 00:00:00 +0000

OpenStreetMap: A Year of Edits

Sunday, January 1st 2012, 00:00 GMT

In the past weeks I've been working on some visualisations related to additions and changes being made to OpenStreetMap. To start of the new year, I hereby present: OpenStreetMap: A Year of Edits (2011):

OpenStreetMap: A Year of Edits (2011)

You can see the video in HD on Vimeo. Happy New Year!

How Did He Do That?

This animation is all made with Open Source software. I haven't had the time to clean up the code to make it releasable yet, but I am intending to do that in the near feature and add it to my OpenStreetMap tools git repository on github.

The first thing I did, was download all the hourly replication files from http://planet.openstreetmap.org/hour-replicate for the year.

With a script, I scanned over all those XML files and for every three hours I created an image showing the edits of those last three hours (in white), as well as all the other edits in the background (in purple). With 4 frames per day, this gives me 1460 frames.

Each of the frames is an equirectangular projection which I then map onto a sphere with a faded background of "earthmap10k" of http://planetpixelemporium.com/earth.html (I downloaded it years ago when it was still free to download). For the 3D rendering of the Earth with the edits layed on top of it I used POV-Ray.

I generated two different loops so that I can make two full rotations at different angles. This gives me in total 2882 frames which is about two minutes. The POV-Ray generated images I post-processed by overlaying the progress bar in the lower left and added the fading and merging effects. Then I used mencoder to create a MPEG video out of it. Finally I added the sound track again by using mencoder and uploaded it to Vimeo.

South Kensington Mapping Party Animation

London, UK

Tuesday, July 19th 2011, 09:17 BST

Every two weeks or-so, the London OpenStreetMap group comes together for a mapping party. The main idea is to go to an area of London to improve the map there. Sometimes that's adding streets, but as most of those are now completed, time is spend on buildings, address information and points-of-interest.

As an experiment I recorded a snapshot of the map for the area (http://mapcraft.nanodesu.ru/pie/73) every hour to follow the progress of people adding new and improved data to the map. I then rendered an image of each map snapshot and put those together to create a video:

South Kensington Mapping Party animation — Map data © OpenStreetMap contributors. CC-BY-SA

You can see the video in HD on Vimeo.

How Did He Do That?

Well, first of all, I already have a script to keep a local version of the London XML map data (as london.osm) up to date. Each hour, this script:

Read the old london.osm file and merge in changes through the OpenStreetMap replication service and write it as london-new.osm.
Compare the old file and the new file and write out a changes file to changes/diff-.osc. This is not needed for this to work, but I am interested in this information as well.
Move the new file to the old file.
Import it into my local PostGreSQL/PostGIS instance.

In script form, that looks like:

#!/bin/bash

NOW=`date +%Y%m%d-%H%M`

export JAVACMD_OPTIONS="$JAVACMD_OPTIONS -Djava.net.preferIPv4Stack=true"
../osmosis-0.39/bin/osmosis --rri --simc --rx london.osm --ac --bb left=-0.563 right=0.28 top=51.68 bottom=51.26 --wx london-new.osm
../osmosis-0.39/bin/osmosis --rx london-new.osm --rx london.osm --derive-change --write-xml-change file="changes/diff-$NOW.osc"
mv london-new.osm london.osm
../try4/bin/osm2pgsql/osm2pgsql -S ../try4/bin/osm2pgsql/default.style --slim -d gis -C 2400 london.osm #-e 15 -o expiry.log

And from that I then can render maps of London quite easily:

cd /backup/osm/try4/bin/mapnik
./generate_tiles_london.py | grep -v " exists"

In order to make a snapshot of South Kensington, or at least, the area that we were going to map in, I added another task. I placed it just after the moving of the old file to the new file. This task cuts out the South Kensington area and puts it in a file south-ken/south-kensington-.osm:

../osmosis-0.39/bin/osmosis --rx london.osm --bb left=-0.18256 right=-0.16853 top=51.49426 bottom=51.48733 --wx south-ken/south-kensington-$NOW.osm

This generated about 100 files in the south-ken directory, where each XML file contained a snapshot of the map on each specific hour.

I then ran the following script to create a image of each of those map files:

#!/bin/bash

export MAPNIK_MAP_FILE=/backup/osm/try4/bin/mapnik/osm.xml
mkdir -p images
LAT1=-0.18256
LON1=51.48833
LAT2=-0.16853
LON2=51.49326

for i in south*osm; do
  /backup/osm/try4/bin/osm2pgsql/osm2pgsql -S /backup/osm/try4/bin/osm2pgsql/default.style --slim -d gis -C 2400 $i
  /backup/osm/try4/bin/mapnik/generate_image.py ${LAT1} ${LON1} ${LAT2} ${LON2}

  mv image.png images/${i}-0.png
  LAT1=`echo "${LAT1} + 0.000005" | bc -q`
  LON1=`echo "${LON1} + 0.00001" | bc -q`
  LAT2=`echo "${LAT2} - 0.00001" | bc -q`
  LON2=`echo "${LON2} - 0" | bc -q`

  /backup/osm/try4/bin/mapnik/generate_image.py ${LAT1} ${LON1} ${LAT2} ${LON2}
  mv image.png images/${i}-1.png
  LAT1=`echo "${LAT1} + 0.000005" | bc -q`
  LON1=`echo "${LON1} + 0.00001" | bc -q`
  LAT2=`echo "${LAT2} - 0.00001" | bc -q`
  LON2=`echo "${LON2} - 0" | bc -q`
done

I had to modify the generate_image.py script from the Mapnik distribution slightly to allow the bounding box to be passed in through the command line. I also set the image width and height to 1280 by 720:

...
ll = [];
for arg in sys.argv:
    ll.append(arg)
ll.pop(0);
for index, arg in enumerate(ll):
    ll[index] = float(arg)

z = 15
imgx = 1280
imgy = 720

m = mapnik.Map(imgx,imgy)
...

Once all the PNG files were created, I converted them to the JPEG format and ran mencoder to generate the AVI film:

for i in *png; do convert -quality 100 $i $i.jpg; done
mencoder "mf://*.jpg" -mf fps=10 -o test.avi -ovc lavc -lavcopts vcodec=msmpeg4v2:vbitrate=7200

And this video I then uploaded to Vimeo. Voilà! One OpenStreetMap mapping party progress animation!

OpenStreetMap Quality Assurance with a Garmin GPS

London, UK

Tuesday, July 5th 2011, 10:43 BST

I've recently bought myself a new Garmin eTrex Vista HCx GPS to replace my older Garmin eTrex Legend unit. Because I don't want to shell out money for outdated maps, I have spend some time building my own map images for the Garmin with mkgmap and OpenStreetMap data. I will write more on that later.

Because most of the data in OpenStreetMap has been surveyed by volunteers, it is always possible that things are missing or done incorrectly. There are quite a few different tools on-line for quality assurance of the OpenStreetMap data, but I will be looking at two of them only.

OSM Analysis

First of all there is ITO World's OSM Analysis. This tool compares the OpenStreetMap data in the UK to the Ordnance Survey's Locator data. The latter is supposed to be the authoritative source of road names in the UK. For each of the 417 areas in the UK, the tool produces two lists; for example the one for Brent is here: http://www.itoworld.com/product/data/osm_analysis/area?name=Brent.

The first column lists all the corrections that Ordnance Survey ought to make in an updated version because one of OpenStreetMap's surveyors found it incorrect. The second column lists all the streets that are missing from OpenStreetMap.

Because it would be really useful to have this sort of data available on a hand-held GPS to be able to notice any of the highlighted missing roads on the go, I set out to convert this data to a map file suitable for the Garmin devices. I've published the scripts on github so that you can try it for yourself, but I've also put a ready made osmanal.img file on-line for download at http://derick.dev.openstreetmap.org/. In the near future I will use cron to run this conversion about once a week automatically.

Musical Chairs

Another tool that compares OpenStreetMap data with Ordnance Survey's Locator data is OS Locator Musical Chairs. Documentation on this project is available on the OpenStreetMap wiki.

The author of the Musical Chairs tool has provided a data dump highlighting all the disagreements it finds (including missing roads) while comparing OpenStreetMap data against Ordnance Survey's Locator data. I've written a tool that converts this data to another Garmin map image files with the scripts that I have published at github. With those scripts you can make your own image files, but like with the OSM Analysis I have made a ready-made file muschair.img available at http://derick.dev.openstreetmap.org/.

Using the map image files

Before you can use the two Garmin map image files, you either need to convert it to a stand-alone gmapsupp.img file by running:

java -jar /backup/osm/try4/bin/mkgmap-r1946/mkgmap.jar --gmapsupp osmanal.img

or:

java -jar /backup/osm/try4/bin/mkgmap-r1946/mkgmap.jar --gmapsupp muschairs.img

These commands produce a gmapsupp.img file that you can upload to your Garmin device. If you do this, you will not have any road data.

You can also merge this file with all your other maps to keep the road data, by running something like:

java -Xmx2048M -jar /backup/osm/try4/bin/mkgmap-r1946/mkgmap.jar \
    --gmapsupp /tmp/music.img /tmp/osmanal.img \
    UK/gmapsupp.img \
    contours/gmapsupp.img

Which merges the Musical Chairs image file (music.img), the OSM Analysis image file (osmanal.img), the UK maps (UK/gmapsupp.img) as well as my UK contours line file (contours/gmapsupp.img). This again produces a gmapsupp.img file that you can upload to your Garmin GPS.

For further mkgmap usage details I would like you to point to it's page on the OpenStreetMap wiki: http://wiki.openstreetmap.org/wiki/Mkgmap or more generally to http://wiki.openstreetmap.org/wiki/OSM_Map_On_Garmin. I am also intending to publish my UK and contour map creation scripts after I've streamlined this process.

And now the maps have been on my GPS for a few days, I've identified and fixed a "disagreement" from Musical Chairs and a few others from the ITO World OSM analysis tool.

Spatial Indexes: Solr

London, UK

Tuesday, June 14th 2011, 09:04 BST

In two previous articles I introduced the spherical Earth model, using SQLite as a geographical data storage and using MySQL as a geographical data storage. In this article we're going to have a look at importing the data into something else than a relational database: the search platform Solr. (Yes, I know I've skipped PostgreSQL, but I'll come back to that).

Solr

Solr is "the popular, blazing fast open source enterprise search platform from the Apache Lucene project." Since version 3.1, Solr has support for spatial search, including geospatial search. It can store coordinates in two different field types: solr.PointType for n-dimensional points, and solr.LatLonType for a two-dimensional point for geospatial search. The main difference is that with solr.PointType, calculations are done according to the flat Earth model, and solr.LatLonType does calculations for a spherical Earth model—which is just what we need.

In this example we will use the default Solr configuration, unless noted otherwise. First we download Solr 3.1 and then untar it with tar -xvzf apache-solr-3.1.0.tgz. Then we edit example/solr/conf/solrconfig.xml to comment out the section on Query Elevation Component so that it looks like:

See here for the reason.

Setting Up Solr

Before we can start using Solr, we need to define a schema. This schema is quite analogous to a database schema, except that Solr only has one table. By default Solr comes with a schema defining lots of fields in example/solr/conf/schema.xml. I've changed the whole section to look like:

I've defined specific fields for the type of source (1=node, 2=way/area), the name of the amenity (name of the pub, hospital, etc), what sort of amenity it is (cafe, bank, etc), the address and postcode, the phone number, the type of cuisine as well as the location. For the location I've also defined two subtypes: location_0_coordinate and location_1_coordinate. Solr requires this for multi-dimensional types such as solr.LatLonType.

The types for each field are also defined in schema.xml. text is for a string that is analysed and tokenized (broken down in smaller parts) so that you can search in only parts of the text, string is for a string of characters that is not analysed or tokenized, and location contains our latitude/longitude point. Each of those types are associated with a specific Java class with an entry such as:

The subFieldSuffix in this line configures the suffix in the fields location_0_coordinate and location_1_coordinate.

Now we have changed the default schema, we can start Solr with: cd example && java -jar start.jar.

On the PHP side, we need to do a bit of set-up as well. I've been using the Solr PECL extension to talk to Solr which can be installed with pecl install solr. Please do not forget to add the extension to php.ini with extension=solr.so. Check whether it is installed correctly with: php --ri solr.

Importing

For our import, we need to change the script we used previously quite substantially. Instead of just using London, I've also downloaded the whole of the UK from Geofabrik and extracted all amenities from the dataset with Osmosis:

wget http://download.geofabrik.de/osm/europe/great_britain.osm.bz2

./osmosis-0.39/bin/osmosis -v 5 \
--read-xml file=great_britain.osm.bz2 \
--tf reject-relations \
--tf accept-nodes amenity=* \
--tf reject-ways \
outPipe.0=POI \
\
--read-xml file=great_britain.osm.bz2 \
--tf reject-relations \
--tf accept-ways amenity=* \
--used-node outPipe.0=area \
\
--merge inPipe.0=POI inPipe.1=area \
--write-xml file=great_britain_amenity.osm

This is going to take quite some time, and will result in a 500MB XML file.

The main logic of the importing script remains the same, but instead of adding each item to a database, we now build a Solr import document and add it to the search index. I've also split the address and postcode into two separate fields, and added the amenity field to store all the different amenities. For now, I'm filtering out post boxes, parking areas and grave yards. The script, parsepoi-solr.php is available here.

After downloading, and renaming the downloaded file to parsepoi-solr.php we can run the script with:

php -dmemory_limit=1G parsepoi-solr.php great_britain_amenity.osm

When done, this should have imported more than 140.000 items into Solr. You can verify this, by going to http://localhost:8983/solr/select/?q=*:*&rows=0

Querying

As you can see, you can query Solr through its HTTP interface quite easily. In fact, all queries to Solr are done over HTTP. The Solr extension however abstracts this away from you for at least the import. I couldn't find any functionality in the extension to do spatial queries, so we'll do it manually.

If we look at the URL above, we can divide it into different parts:

http://localhost:8983/solr/: The base URL for this Solr instance.
select/?: The action for running search queries.
q=*:*: q is the search query parameter, and its value, *:* means all fields (the first *) and all possible values (the second *). In this case that means return everything.
rows=0: Do not return any rows, so that we just get a count.

We get back the result as an XML file. If we want JSON instead, we can simply append &wt=json. In case you want a CSV file, append &wt=csv.

In a first example, all we want to do is to return all cafes within 100 meter. We construct the search query as follows:

http://localhost:8983/solr/select/?: The base URL with search action.
q=amenity:cafe: We search in the field amenity and look for the value cafe.
fq={!geofilt}: We set a query filter to geofilt_. This filter restricts the result set according to the location (pt) and the maximum distance from this location (d) in km.
sfield=location: The field to use for location information.
pt=51.5375,-0.1934: The point that we center our search around.
d=0.1: The maximum distance in kilometers.
wt=csv: The format to return, in our case, CSV.

Together this makes the full GET request: http://localhost:8983/solr/select/?q=amenity:cafe&fq={!geofilt}&sfield=location&pt=51.5375,-0.1934&d=0.1&wt=csv

And the result is:

id,phone,cuisine,location,address,location_1_coordinate,name,amenity,type,location_0_coordinate,postcode
w62088838,,,51.53750834,-0.19329616,75 Kilburn High Road,-0.19329616,Costa,cafe,2,51.53750834,
w78337118,,,51.53828298,-0.19410346,101 Kilburn High Road,-0.19410346,Caffè Nero,cafe,2,51.53828298,NW6 6JE
w105467205,,,51.537555925,-0.192445525,274 Belsize Road,-0.192445525,Belsize Cafe,cafe,2,51.537555925,NW6 4BT
w105467209,+44 207 3724002,,51.537624071429,-0.19228221428571,270 Belsize Road,-0.19228221428571,Lord Jim,cafe,2,51.537624071429,NW6 4BT
w107710475,+44 20 76245736,,51.53695276,-0.19263662,2 Kilburn Bridge,-0.19263662,Famished Cafe,cafe,2,51.53695276,NW6 6HT
w107710482,,,51.53717868,-0.19288912,Kilburn Bridge,-0.19288912,Mike's,cafe,2,51.53717868,NW6 6HT
w107710490,+44 20 76246942,,51.53732946,-0.1930577,12 Kilburn Bridge,-0.1930577,La Dolce Vita,cafe,2,51.53732946,NW6 6HT

Which fields are returned can be configured. In this case, we are only interested in the location and name of each amenity, so we restrict the number of returned fields with: fl=id,location,name. The result now becomes:

id,location,name
w62088838,51.53750834,-0.19329616,Costa
w78337118,51.53828298,-0.19410346,Caffè Nero
w105467205,51.537555925,-0.192445525,Belsize Cafe
w105467209,51.537624071429,-0.19228221428571,Lord Jim
w107710475,51.53695276,-0.19263662,Famished Cafe
w107710482,51.53717868,-0.19288912,Mike's
w107710490,51.53732946,-0.1930577,La Dolce Vita

Sadly, the results do not come back ordered by distance from our starting point. In order to do that, we need to add one more query parameter: sort=geodist() asc. The full query is now: http://localhost:8983/solr/select/?q=amenity:cafe&fq={!geofilt}&sfield=location&pt=51.5375,-0.1934&d=0.10&wt=csv&fl=id,location,name&sort=geodist()+asc

And the result:

id,location,name
w62088838,51.53750834,-0.19329616,Costa
w107710490,51.53732946,-0.1930577,La Dolce Vita
w107710482,51.53717868,-0.19288912,Mike's
w105467205,51.537555925,-0.192445525,Belsize Cafe
w105467209,51.537624071429,-0.19228221428571,Lord Jim
w107710475,51.53695276,-0.19263662,Famished Cafe
w78337118,51.53828298,-0.19410346,Caffè Nero

It is however not possible to return the distance from our starting point directly with Solr. With the exception being that if you use only the geodist() function as query argument. In this case, make sure to include the special field score in the fl= argument as well. An URL showing this is: http://localhost:8983/solr/select/?q={!func}geodist()&fq={!geofilt}&sfield=location&pt=51.5375,-0.1934&d=0.10&wt=csv&fl=id,location,name,score&sort=geodist()+asc Of course, in this case you can not filter for specific amenities.

Conclusion

In currently released versions of Solr (3.1 at the time of writing), it is not possible to return the distance to the starting point for the search as part of the result set. The Solr developers have already implemented the capabilities to add the results of function queries to result documents for version 4.0. You can verify this by downloading a nightly build and running the query: http://localhost:8983/solr/select/?q=amenity:cafe&fq={!geofilt}&sfield=location&pt=51.5375,-0.1934&d=0.10&fl=id,location,name,score,geodist()&sort=geodist()+asc In this query I've added geodist() to the fl= parameter. The result now includes an extra field called geodist():



  
  
        
          w62088838
          Costa
          51.53750834,-0.19329616
          4.352648
          0.0072415726934058865
        
        
          w107710490
          La Dolce Vita
          51.53732946,-0.1930577
          4.352648
          0.030333097323281075

I could not manage to alias it to a different field, or get it to work with the CSV format (wt=csv).

If the return format is xml (the default) or json, then the result includes, besides the number of found items, the total search time for this query. Look for this information in the QTime field. In all of the examples here, the QTime has been less than 5, meaning 5 milliseconds. Solr is extremely fast, even with huge amounts of data. I will try to import the amenities of the whole "planet" at some point, and report back with some benchmarking information.

In the next installment of this series on storing geospatial data, I will be looking at using MongoDB as data store for geographical information.

What is OpenStreetMap?

Thu, 12 May 2011 08:04:00 +0000

What is OpenStreetMap?

London, UK

Thursday, May 12th 2011, 09:04 BST

With people more and more complaining that Google Maps gets it wrong, I often reply that the complainers have a look at OpenStreetMap instead. But there are a few misunderstandings on what OpenStreetMap actually is.

OpenStreetMap has its main site at http://openstreetmap.org. This shows a rendering of the map that we like to call a "slippy map". This is an online map, that can interactively be zoomed and paned. The slippy map on the site is nothing more than a display of map tiles, static images rendered from OpenStreetMap data as an example of what you can do with it. The real power of OpenStreetMap is not this default rendering, but the possibility to actually access the data behind this map rendering. And this is where it differs from Google maps, Bing Maps and Yahoo! Maps and various others. OpenStreetMap is the only mapping service that allows you to do something more than just look at pretty map tiles. OpenStreetMap is a database project, with as its main purpose to have an exhaustive database of every street, city, road, building etc on the planet, and not a map display project.

Now what does that actually mean? For one thing, it means that the sample map tiles (that are displayed in the slippy map) are not meant to be used heavily from applications. It's all right to show a map tile of your neighbourhood on your web site, but it's not okay to write a mobile application that allows you to scrape map tiles of large areas. Have a look at the general tile usage policy for some background. The only reason why there is a sample rendering is to aid mappers with improving the data that is the core of OpenStreetMap.

Obviously, the OpenStreetMap project wants its data to be used. But if you're not allowed to use the map tiles en-masse, then how can you use the data?

First of all, you can query the OpenStreetMap database in various ways. In most cases, you're going to get an XML file with descriptions of nodes (points of interest, facilities such as toilets, benches, addresses), ways (roads, water ways, transport routes) and areas (buildings, lakes). There is some more info on the (very simple schema) here. This data you can parse and import (as I've shown in a previous article) and do all kinds of cool tricks with, such as finding out the closest pub. This functionality is not available on the main OpenStreetMap site, because "OpenStreetMap does not aim to create and host every webservice possible, but to provide the data so that others can" make awesome map-related applications.

Secondly there is a growing number of web sites that uses the OpenStreetMap data to render specialized maps. There is OpenCycleMap that renders information related to using your bicycle and Öpvnkarte for public transport. There is even a tool that uses the postcode data in OpenStreetMap. It's quite possible to render your own maps as well, but that I will get back to in a later article.

MapQuest is a good example of a company that uses OpenStreetMap data. They have made their own rendering with its own styles for a visually different map. Their site implements searching for pubs/bars and restaurants for example and MapQuest are also happy with you using their map tiles for your applications. Recently, they have also announced a simple API to generate a static map based on OpenStreetMap with their rendering style.

The map above is generated by just requesting http://open.mapquestapi.com/staticmap/v3/getmap?size=320,260&zoom=12¢er=51.51,-0.15

Another major difference with established mapping providers is the ability to edit the map yourself. You can add your own business, houses, addresses, postcodes, your favourite restaurants with either on-line or off-line editors. My list of recent edits is here.

There is also a whole plethora of special renderings of the map data to assist with debugging issues with the map. There is the "no name" rendering which shows roads without name, ITO World's osm analysis that shows deviations of road names from another source in the UK, keepright and OSM Inspector that have some automated checks and a tool to overlay postcodes for checking.

Obviously, there are a few things missing that would make OpenStreetMap an even better resource for mapping related applications and web-applications. Feel free to suggest things that you find missing or lacking in the comments of this article. I'd be happy to hear why you would think that OpenStreetMap is not a good fit for your applications and usage.

Spatial Indexes: MySQL

Tue, 12 Apr 2011 08:04:00 +0000

Spatial Indexes: MySQL

London, UK

Tuesday, April 12th 2011, 09:04 BST

In two previous articles I introduced The spherical Earth model and importing data into SQLite for querying geographical data. In this article we're going to have a look at importing the data into MySQL and finding out how to best store and query spatial data in the databases.

MySQL

MySQL has some support for Spatial Extensions, but it's not particularly useful. For example, there is no way to query anything within the radius around a specific point. Their community pages list a method of implementing it, but it only calculates for a flat Earth model.

Instead, we'll have to implement our own algorithms again. But first of all, let us import the data into MySQL. First we create the MySQL database:

derick@whisky:~$ mysqladmin -u root -p create poi
mysql> CREATE TABLE poi(id int, type int, lat float, lon float, name char(255), address char(255), cuisine char(64), phone char(18));

And then we take the script from the previous article on importing data, and change the third line from:

$d = ezcDbFactory::create( 'sqlite://' . dirname( __FILE__ ) . '/pois.sqlite' );

to:

$d = ezcDbFactory::create( 'mysql://root:root@localhost/poi' );

Of course, substitute the username and password (root, root) and the database name (poi) to one that suits yourself. We then run again:

php parseoi.php.txt great_britain_pubs.osm

And check that our import is complete:

mysql> SELECT count(*) from poi\G
*************************** 1. row ***************************
count(*): 28147

Now that we have all the POIs imported, we can query the database. Because MySQL doesn't have working spatial extensions, nor the register-function capabilities from SQLite such as we saw in the previous article, we have to come up with a new solution. The most obvious one is writing a stored procedure for the task, another (non-explored option) would be to write a user defined function. Writing the stored procedure is simple enough; we just have to convert the distance() function to SQL. On the MySQL command line you can enter:

delimiter //

CREATE FUNCTION distance (latA double, lonA double, latB double, LonB double)
    RETURNS double DETERMINISTIC
BEGIN
    SET @RlatA = radians(latA);
    SET @RlonA = radians(lonA);
    SET @RlatB = radians(latB);
    SET @RlonB = radians(LonB);
    SET @deltaLat = @RlatA - @RlatB;
    SET @deltaLon = @RlonA - @RlonB;
    SET @d = SIN(@deltaLat/2) * SIN(@deltaLat/2) +
        COS(@RlatA) * COS(@RlatB) * SIN(@deltaLon/2)*SIN(@deltaLon/2);
    RETURN 2 * ASIN(SQRT(@d)) * 6371.01;
END//

Fetching all the pubs within a 250 meter radius around 51.5375°N, 0.1933°W is than as easy as running:

mysql> SELECT name, address, phone
       FROM poi
       WHERE DISTANCE(lat, lon, 51.5375, -0.1933) < 0.25;

With the result being:

| name            | address                        | phone           |
+-----------------+--------------------------------+-----------------+
| Mrs Betsy Smith | Kilburn High Road 77 , NW6 6HY | +44 20 76245793 |
| The Cock Tavern | Kilburn High Road 125          | NULL            |
| The Old Bell    | NULL                           | NULL            |
| The Westbury    | Kilburn High Road 34 , NW6 5UA | +44 20 76257500 |
+-----------------+--------------------------------+-----------------+
4 rows in set (0.72 sec)

If we want to also include the distance from the centre point in the result, we need to modify the query to:

mysql> SELECT name, DISTANCE(lat, lon, 51.5375, -0.1933) AS dist
       FROM poi
       HAVING dist < 0.25
       ORDER BY dist;

with as result:

| name            | dist                |
+-----------------+---------------------+
| Mrs Betsy Smith | 0.00825473345748987 |
| The Cock Tavern |     0.2420193460511 |
| The Old Bell    |   0.103123484090313 |
| The Westbury    |   0.150294300836645 |
+-----------------+---------------------+
4 rows in set (0.72 sec)

In both cases, the query takes 0.72 seconds. This is not overly fast, and the main reason for this is that the distance() function has to be called for every row in the table. An index can not be created on this either. However, what we can do is to filter out the results roughly first, by calculating a bounding box of latitude/longitude pairs around the centre point. Calculating the latitude boundaries can be done by:

d = (distInKm / 6371.01 * 2π) * 360
lat1 = centreLat - d
lat2 = centreLat + d

In our example that becomes:

d = (0.250 / 6371.01 * 2π) * 360 = 0.0022483
lat1 = 51.5375 - 0.0022483 = 51.5352.. -> 51.5351
lat2 = 51.5375 + 0.0022483 = 51.5397.. -> 51.5398

Or in PHP:

We round slightly up and down to combat inaccuracies in the calculations—it is a rough estimate after all. After adding the index on the lat column, and the index on the lon column, we reissue the query:

mysql> CREATE index poi_lat ON poi(lat);
mysql> CREATE index poi_lon ON poi(lon);

mysql> SELECT name, DISTANCE(lat, lon, 51.5375, -0.1933) AS dist
       FROM poi
       WHERE lat BETWEEN 51.5351 AND 51.5398
       HAVING dist < 0.25;

Which returns the same result as before, but faster:

| name            | dist                |
+-----------------+---------------------+
| The Westbury    |   0.150294300836645 |
| The Old Bell    |   0.103123484090313 |
| Mrs Betsy Smith | 0.00825473345748987 |
| The Cock Tavern |     0.2420193460511 |
+-----------------+---------------------+
4 rows in set (0.01 sec)

We can pre-filter the result set even more, by also limiting on the longitude boundary. This involves a few more calculations than for the latitude boundaries. The distance in degrees longitude that belongs to a distance in km depends on the latitude of the location. So first we need to calculate the latitude boundaries as we have done above, and with that information calculate the longitude boundaries.

In the first step (red), we calculate the northern and southern boundaries of the circle. We then calculate the western and eastern boundaries for the northern boundary (green), and southern boundary (blue).

In PHP this algorithm becomes:

If we use both functions we calculate as boundaries:

Which returns the query to execute:

SELECT name, DISTANCE(lat, lon, 51.5375, -0.1933) AS dist
FROM poi
WHERE lat BETWEEN 51.535251699514 AND 51.539748300486
  AND lon BETWEEN -0.19691479627963 AND -0.18968520372037
  HAVING dist < 0.25
  ORDER BY dist;

If we run this query, we get as result:

| name            | dist                |
+-----------------+---------------------+
| Mrs Betsy Smith | 0.00825473345748987 |
| The Old Bell    |   0.103123484090313 |
| The Westbury    |   0.150294300836645 |
| The Cock Tavern |     0.2420193460511 |
+-----------------+---------------------+
4 rows in set (0.00 sec)

As you can see, we are getting the result a lot faster now—0.00 sec vs 0.72 sec. Please do note, that if we would just have used the boundaries without using the HAVING dist < 0.25 clause, we would have gotten one extra result that is slightly too far away (0.281 km):

| name            | dist                |
+-----------------+---------------------+
| Mrs Betsy Smith | 0.00825473345748987 |
| The Old Bell    |   0.103123484090313 |
| The Westbury    |   0.150294300836645 |
| The Cock Tavern |     0.2420193460511 |
| Bar Hemia       |   0.281138534935208 |
+-----------------+---------------------+

Conclusion

In this article we saw how we can use a MySQL stored procedure to find our pubs and bars within a certain distance from a central location. In the next part, we will be looking on how to solve the same problem with PostgreSQL.

Spatial Indexes: Fetching Data/SQLite

Thu, 31 Mar 2011 08:17:00 +0000

Spatial Indexes: Fetching Data/SQLite

London, UK

Thursday, March 31st 2011, 09:17 BST

In a previous article I introduced 'The Flat Earth Model' and the 'The spherical Earth model'. In this article we're going to have a look at fetching a data set and importing them into a SQLite database to query from PHP. What better data set is there to import than all of the UK's pubs?

Getting the Data

In order to get a suitable data set we are going to use the data from the OpenStreetMap project. This project is mainly concerned with making an open map of the entire globe, but it also contains a vast database of points-of-interest (POI). OpenStreetMap contributes seems to be a big fan of pubs, and hence they are mapped really well. POIs can either be stored as a node (a single point with a geographical location) such as The Long Acre or as a closed way (an ordered collection of nodes where the first and last node are the same), such as Brondes Age.

I will demonstrate two methods to fetch the pubs in an XML format containing nodes and ways. The first method is with XAPI. XAPI is an interface to the OpenStreetMap database to allow users to query and filter items. In order to fetch data through it, you specify a bounding box and a predicate. A bounding box specifies the most Eastern, Southern, Western and Northern coordinates. The UK has roughly as bounding box 9.05°W, 48.77°N, 2.19°E, 58.88°N, or short: -9.05, 48.77, 2.19, 58.88. Fetching the data can simply be done by querying the server with wget:

wget -O pubs.xml 'http://xapi.openstreetmap.org/api/0.6/*[amenity=pub][bbox=-9.05,48.77,2.19,58.88]'

This is going to take a long time; and will most likely just not work or time-out. The current XAPI server, written in an obscure programming language called MUMPS, is extremely unreliable and is really slow. A new version of XAPI build in Java is on the way, but right now it's limited to 10 square degrees.

Luckily, there is an alternative in the form of parsing (an extract of) the planet file. The planet file is an enormous database dump of OpenStreetMap's data. The people at Geofabrik have extracts for specific parts of the world at http://download.geofabrik.de/osm/. From them I downloaded the europe/great_britain.osm.pbf file (274 MB) for my example. With a tool called Osmosis we can then filter out all pubs into a similar formatted XML file. I am only giving the command to do our task at hand, but a detailed usage guide for Osmosis is available too. The command runs in about four minutes and looks like:

./osmosis-0.38/bin/osmosis -v 5 \
\
--read-pbf file=great_britain.osm.pbf \
--tf reject-relations \
--tf accept-nodes amenity=pub,bar \
--tf reject-ways \
outPipe.0=POI \
\
--read-pbf file=great_britain.osm.pbf \
--tf reject-relations \
--tf accept-ways amenity=pub,bar \
--used-node \
outPipe.0=area \
\
--merge inPipe.0=POI inPipe.1=area \
--write-xml file=great_britain_pubs.osm

Importing the Data

The resulting XML file has two important elements: nodes and ways. The XML for The Long Acre looks like:

Important here are the latitude (51.51°N) and longitude (0.12°W) and of course, the name of the pub in the tag element.

And the XML for Brondes Age (a way) is a bit more complex, and looks like:

There is no latitude and longitude, but instead there are references to nodes (the nd elements). There is also a large collection of descriptive tags, such as addr:housenumber and contact:phone. In order to calculate the latitude and longitude of Brondes Age we can either take the correct approach by calculating the centroid or we can simply take the average latitude and longitude of all the nodes. To make things simple, I will take the simple approach.

In order to find the latitude and longitude of all the nodes, we simply scan through the file again and find all the nodes that correspond to the ref attribute of each nd element. We should disregard the last one of each way though, as it is the same as the first one.

SQLite

To start, we will use a very simple RDBMS: SQLite. We will still have to define a database schema. We can simply do that with:

derick@whisky:~$ sqlite pois.sqlite
sqlite> CREATE TABLE poi(id int, type int, lat float, lon float, name char, address char, cuisine char, phone char);

The importing of data then can be done by this "simple" script (after adjusting the path to eZ Components/ Zeta Components) with:

php parsepoi.php.txt great_britain_pubs.osm

After running the import script, there should be about 28000 POIs in the database.

Querying the Data

Once we have imported the POIs into our SQLite database, we are ready to query them. SQLite does not have a very extensive set of functions so we can not do the calculation in the query directly. Just to iterate from the previous article in this series, the formula for calculating the distance in a spherical Earth model is:

One solution would be to query the database, and then use the distance() function to filter out unwanted elements, like:

createSelectQuery();
$q->select( '*' )->from( 'poi' );
$s = $q->prepare();
$s->execute();

foreach ( $s as $res )
{
  $e = distance( $lat, $lon, $res['lat'], $res['lon'] );
  if ( $e < $wantedD )
  {
    echo sprintf( '%.3f,%.3f %-40s %.2f km away',
      $res['lat'], $res['lon'], $res['name'],
      $e ), "\n";
  }
}
?>

This will show all pubs in a 250 meter radius around 51.53°N, 0.19°W:

derick@whisky:/home/httpd/html/test/maps$ php fetch-sqlite-simple.php
51.538,-0.193 Mrs Betsy Smith                          0.01 km away
51.539,-0.195 The Cock Tavern                          0.24 km away
51.537,-0.192 The Old Bell                             0.10 km away
51.537,-0.192 The Westbury                             0.15 km away

Of course, this is not very efficient as all items are selected from the database, and then filtered out depending on their calculated distance. Luckily, SQLite supports user defined functions written in PHP. This would mean that we can increase performance a bit by letting PHP's internals call the distance function for every row:

sqliteCreateFunction( 'dist', 'distance' );

// Centre point
$lat = 51.5375;
$lon = -0.1933;

// Distance (in km)
$wantedD = 0.25;

$q = $d->createSelectQuery();

// Use the user defined dist() function as additional column
$q->select( "*, dist($lat, $lon, lat, lon) as e" )
  ->from( 'poi' )
  ->where( "e < $wantedD" );

$s = $q->prepare();
$s->execute();

foreach ( $s as $res )
{
  echo sprintf( '%.3f,%.3f %-40s %.2f km away',
    $res['lat'], $res['lon'], $res['name'],
    $res['e'] ), "\n";
}
?>

The result is naturally the same as before.

Conclusion

In this installment we have seen how to retrieve information from OpenStreetMap's database and import them into SQLite for querying. In the next installment we will have a look at how to import and query with MySQL and PostgreSQL.

Spatial Indexes: Calculating Distance

Wed, 09 Mar 2011 12:17:00 +0000

Spatial Indexes: Calculating Distance

Montréal, Canada

Wednesday, March 9th 2011, 07:17 EST

During my "Geolocation and Mapping with PHP" talk that I've given a few times I briefly touch on the subject of indexes on data-sets of spatial data. This isn't as simple as just solving Pythagoras theorem and this article is meant to clarify this.

The flat Earth model

Pythagoras theorem can be used to calculate the distance between two points quite easily; you take the square root of the square of the absolute vertical distance plus the square of the absolute horizontal distance; in short:

d = √(|x1 - x2|² + |y1 - y2|²)

or in PHP:

$d = sqrt(pow(abs($x1 - $x2), 2) + pow(abs($y1 - $y2), 2));

If you take for example London's coordinates (51.50°N, 0.13°W) and Amsterdam's coordinates (52.37°N, 4.90°E) we can calculate the distance with:

d = √(|-0.13 - 4.90|² + |51.50 - 52.37|²)
d = √(5.03² + 0.87²)
d = √(26.0578)
d ≅ 5.10

And on a map:

But what does a difference of "5.10°" actually mean? How far is this in useful units, such as meters?

If we show the whole map of which the above is an extract, we come to:

The distance around the equator, and through the poles is roughly the same, 40.000km (please be aware that the blue line only shows half of it, the other half is going through the anti-meridian at 180°W/E). 5.03° in East-West difference is then about 40 000 ✕ (5.03/360) = 559 km and the North-South difference about 20 000 ✕ (0.87/180) = 97 km. Using those numbers within the Pythagoras theorem we end up with a distance of √(559² + 97²) = 567 km. Although the calculation is correct, the answer is still wrong. The real distance is closer to 360 km.

The spherical Earth model

If we look at the Earth in its original (mostly) spherical 1 shape, then it's clear that 10° longitude (East/West) at 60°N is going to be less of a distance than 10°E/W at the equator. It's actually fairly easy to calculate how much 1° longitude is at 60°N by using cos(60) * 1/360 * 6371 * 2π 2. Or in PHP:

This returns 55.597 km per ° for 60°N. The same distance in degrees on the Equator gives 6371 * 2 * M_PI / 360 = 111.195.

The following diagram shows ones more that latitudinal degrees always correspond with the same distance in kilometer, whereas longitudinal degrees differ.

In the diagram the line A is a line from 0°N, 90°W to 10°N, 90°W. It has the same length as line B, from 30°N, 90°W to 40°N, 90°W: a 36th of the circumference of the Earth through the poles. Line C, from 0°N, 30°W to 0°N, 20°W has the same length. Line D however, from 50°N, 30°W to 50°N, 20°W is shorter by a factor of cos(50°) ≅ 0.64.

If we look again at the distance between London and Amsterdam, ignore the differences in latitude and instead pick the average, we see:

Which is a bit shorter than the expected 360km, but that's because we conveniently forgot about the difference in latitude.

Sadly, we can't use Pythagoras's theorem to calculate the real distance with the latitude difference taken account as well. This is because the theorem is meant for Euclidean geometry, and a sphere does not follow the rules of this geometry. Instead we need to use a formula that is called the great-circle distance formula. The main concept behind it is that a circle is drawn across the whole sphere that connects both the start (point P) as well as the end (point V). Then with that circle the distance can be calculated. The following diagram shows this:

I will spare you how the function is derived, but the distance calculation ends up as being:

If we punch in our original numbers form London (51.50°N, 0.13°W) and Amsterdam (52.37°N, 4.90°E), we calculate the following:

Which gives us the expected result of 358.07 km.

Conclusion

I hope that the above clarified the difference between 2D spatial indexing with the flat Earth model and spatial indexing of geo-located data (the spherical Earth model). In future articles I will go into specific implementations of spatial indexing by traditional databases such as SQLite, MySQL and PostGreSQL; NoSQL databases such as MongoDB and CouchDB; and Solr.

1
The Earth is not really a sphere, but an approximation of it. However, doing the same calculations for an ellipsoid can (as far as I know) only be done by approximation. The difference would hardly matter for finding the closest pub.
2
In this article, I've used an average radius of the Earth of 6371 km.

Using OpenStreetMap tiles with Flickr

Tue, 01 Mar 2011 09:20:00 +0000

Using OpenStreetMap tiles with Flickr

London, UK

Tuesday, March 1st 2011, 09:20 GMT

Update August 4th, 2011: I've changed the script to use any of the three OpenStreetMap tile servers, instead of hard coding it just to one.

I like taking pictures, and I usually take a GPS so that I can place them on a map on my Flickr page. On my last excursion however, the battery of my GPS had died, so I did not have location information available to store in my pictures' EXIF headers. Flickr can use the EXIF headers to then show the images on the map.

Because I did not have the location information to automatically place my pictures on the map, I wanted to do that by hand. Flickr, being owned by Yahoo!, uses Yahoo! Maps. Yahoo! Maps however, is not terribly great in the country side. Actually, it is mostly empty, especially compared to OpenStreetMap's version. This made placing photos onto the map by hand quite impossible. So I set out to have only OpenStreetMap tiles as back ground in Flickr like they already do for Bejing and some other places.

Inspired by the Upside-Down-Ternet I installed Squid, and set the url_rewrite_program to a PHP script (with execute bit on):

url_rewrite_program /home/derick/bin/redirect.php

Squid will fire up a few of those scripts (depending on the url_rewrite_children setting) and then supplies them with URLs to rewrite. Each line of input is an URL to rewrite, and the script should echo a rewritten URL.

The script itself is rather simple. It just needs to account for two different formats because the Yahoo!Maps URLs are different from the Flickr ones (as indicated by the r argument in the original URL). I'm including the script here (and as download):

#!/usr/local/php/5.3dev/bin/php
 $newUrl\n" );
        }
        else
        {
            // This is the format that Yahoo!Maps uses

            // Do the math to calculate the OSM tile
            // coordinates from the Yahoo!Maps one
            $z = $queryParts['z'] - 1;
            $x = $queryParts['x'];
            $y = pow( 2, $z - 1 ) - $queryParts['y'] - 1;

            // Assemble new URL and write log line
            $range = range('a', 'c');
            $server = $range[rand(0, sizeof($range)-1)];
            $newUrl = "http://{$server}.tile.openstreetmap.org/$z/$x/$y.png";
            fwrite( $a, "REDIR: $parts[0] => $newUrl\n" );
        }
    } else {
        $newUrl = $parts[0];
        fwrite($a, "NORMAL: $newUrl\n");
    }

    // Output the rewritten (or original) URL
    echo $newUrl, "\n";
} while ( true );

After I configured my browser to use the Squid proxy running on localhost, Flickr is now shown with OpenStreetMap tiles as background:

And with the OpenStreetMap tiles in the background, I could place my photos on the correct location on the map.

PHP and Ordnance Survey Mapping

Tue, 27 Apr 2010 13:27:00 +0000

PHP and Ordnance Survey Mapping

London, UK

Tuesday, April 27th 2010, 14:27 BST

About a month ago, Ordnance Survey opened up some of their data for public consumption under a brand called OpenData. The data is licenced under a "Creative Commons Attribute"-like license. One of the data sets they provide is Code-Point Open and provides a dataset "that contains postcode units, each of which have a precise geographical location". I've always been a bit of a map-geek, and have always geotagged my pictures and some of my tweets. So I was wondering what cool thing I could do with this newly released data. I decided to map all the postcodes onto the UK map where more postcodes for a specific place would create a "lighter" colour. Each postcode has on average about 15 addresses, so in more densely populated areas you have more "postcodes-per-area". Doing this wasn't very difficult and it resulted in the following map:

You can very clearly see the more densely populated areas such as London.

This generated postcode-density map I wanted to overlay on a real map, such as OpenStreetMap provides. When I tried to align the image that I generated with the OSM map, I ended up with this:

Obviously, the maps don't align. In order to fix this, I ended up learning a lot more about map projections.

The Code-Point Open data uses the UK's National Grid to store location data in. The National Grid consists of 100*100 km squares that are then further subdivided into smaller squares creating grid references such as TQ3012780512, or the numerical version 530128 180512. TQ translates into the "hundred thousands" of the Eastings and Northings according to the grid that you can see here. In this case, it specifies a point 530 128 meters East and 180 512 meters North of the origin. (If you work it out, you'll end up in London).

OpenStreetMap uses a Mercator projection to visualize maps. The Mercator projection is a cylindrical map projection, and it distorts the size and shape of large objects, as the scale increases from the Equator to the poles. Therefore it only works from about 85°N to 85°S (why it is 85° only be came clear after doing all the maths for it). Google Maps uses the same projection.

There were a few challenges plotting the Code-Point Open data on an OpenStreetMap map:

The National Grid coordinates need to be converted to Latitude/Longitude pairs
Latitude/Longitude pairs need to be mapped to pixels to align with the Mercator projection of OpenStreetMap maps.

For converting National Grid references to Latitude/Longitude pairs I found some JavaScript code. I converted this code to PHP and to verify whether my code was working properly, I used this site.

Converting Latitude/Longitude pairs to the pixel coordinates of OSM required a bit of maths. OSM provides maps in different zoom levels, from 2 to 17. Each zoom level having twice the amount of pixels horizontally and vertically. Zoom level 2 has 4 times 4 tiles, where each tile is 256*256 pixels. At zoom level two, the whole world fits into 1024*1024 pixels. The number of pixels for each axis for a specific zoom-level can be calculated by:

pow(2^zoom) * 256

For a zoom level of 7 that makes 16384 pixels in each direction. To convert a longitude (in the range -180° to 180°) we can simply apply:

x = ((lon + 180) / 360) * (pow(2^zoom) * 256)

For a Longitude of 0.003117° E at zoom-level 13 that turns out to be pixel 1048594.

Longitude conversions are more difficult due to the Mercator projection itself. To convert we apply:

y = ((atanh(sin(deg2rad(-lat))) / π) + 1) * (pow(2^(zoom-1)))

For a Latitude of 51.502817° N at zoom-level 13 that turns out to be pixel 697399.

Creating an image for the whole world at zoom level 13 is impractical as it is 2097152*2097152 pixels and downloading all the 67 million tiles for this zoom level is probably not liked by the OSM people either. So instead we take a cut out for a specific area only. For the UK (61.37°N -9.49° W, 49.76°N +3.63°E) at zoom level 7 we end up with a 1536x2048 map with the North-Western pixel being 15360,9216. On this map, we can draw the latitudes and longitudes as well as the National Grid lines (full image is on flickr):

The only thing left to do now, is to map the postcode density information to the map. I picked zoom level 6 for this, and the result is (after cropping it to 640 pixels width):

As you can see, this is perfect fit to the outlines of the country. But unfortunately, when we look very closely at the plotted map data, for example for the NW10 3 postcodes, we notice that the mapping is slightly off. The blue dots are what we plotted, and the red dots are what the locations should have been:

The reason for this is that when we converted the National Grid locations to Latitude/Longitude pairs to plot on the OSM maps, I forgot to take into account the different Datums that are used in the projections. The Earth is not a perfect sphere, and an approximation of the ellipsoid of the whole Earth is not necessarily the best fitting for a specific area such as the UK. Therefore, the National Grid uses the OSGB36 Datum which fits more closely to the UK, where as OpenStreetMap uses the WGS84 Datum that is also used by GPS. The Ordnance Survey Ireland has a more thorough explanation on their site. As you can see above, using the wrong Datum can mean locations can be off. In our example about 100 meters. Converting between different Datums is possible, albeit processor intensive.

After I figured out all the maths for this, the only problem that remains that implementing those algorithms in PHP is show—calculating all the positions from the 1.6 million postcode locations takes up to 10 minutes. This is why I am not presenting any code yet. I am planning to implement all the necessary calculations in a PHP extension to speed up the calculations