Derick Rethans - tag: datetime

Storing Date/Times in Databases

Mon, 29 Mar 2010 15:47:00 +0000

Storing Date/Times in Databases

Monday, March 29th 2010, 16:47 BST

After my talk during ConFoo on Advanced Date/Time Handling I received a question about whether the UTC-offset, together with the date/time in years, months, days, hours, minutes and seconds, was enough for storing a date/time in a database and still being able to do calculations with this. The answer to this question was no, but it lead to an even more interesting discussion about what would be enough to store an accurate date/time in a database.

Firstly let me explain why storing a UTC offset is not adequate at all for doing any sort of calculations. Right now (March 25th, 2010), Montreal is on the same time as Santiago (in Chile) with both a UTC offset of -4 hours. This means that for the same date/time (2010-03-25 19:03) the same timestamp is generated (1269558180). However if we want to add eight months (2010-11-25 19:03) to the current time for each of those locations then the timestamps should be different — 1290729780 for Montreal and 1290722580 for Santiago. It turns out that neither of them is exactly a full 24 hour difference from 2010-03-25 19:03 as is shown the following script:

format( 'U' );
$santiagoNow = date_create( "{$s} America/Santiago" )->format( 'U' );
$s = '2010-11-25 19:03';
$montrealThen = date_create( "{$s} America/Montreal" )->format( 'U' );
$santiagoThen = date_create( "{$s} America/Santiago" )->format( 'U' );

$hour = 60 * 60;
$day = 24 * $hour;

echo "Montreal:\n",
    $montrealNow, " - ",
    $montrealThen, "\n",
    ( ( $montrealThen - $montrealNow ) % $day ) / $hour, "\n\n";

echo "Santiago:\n",
    $santiagoNow, " - ",
    $santiagoThen, "\n",
    ( ( $santiagoThen - $santiagoNow ) % $day ) / $hour, "\n\n";

?>

which gives the output:

Montreal:
1269558180 - 1290729780
1

Santiago:
1269558180 - 1290722580
23

From this we see that Montreal has an extra hour and Santiago has an hour less. This is because Montreal changed from Daylight Savings Time to normal time and Santiago, being on the southern hemisphere, moved from normal time to Daylight Savings time.

Now if we had only the timestamp (1269558180) and the UTC-offset, the only thing we could do would be to advance a specific number of days. In this case, there are 245 days, which makes (245 * 24 * 60 * 60 = ) 21168000 seconds. If we add this to the original timestamp, we end up at 1290726180 which corresponds to 2010-11-25 18:03 Montreal time, or 2010-11-25 20:03 Santiago time. Neither of them being the correct 2010-11-25 19:03. Because from the UTC offset of 4 hours we don't know whether we're in Montreal or Santiago, we can conclude that with just this UTC offset and the original timestamp we can't calculate the timestamp of something that's 8 months in the future. Storing the UTC-offset can not change this fact either.

In order to to proper calculations, you need to keep information about the timezone itself. In PHP timezones are identified with identifiers such as America/Montreal. If we store those alongside the timestamps, we can do the proper calculations. The following example demonstrates that:

setTimeZone( new DateTimeZone( $tzid ) );
$d->modify( '+8 months' );

echo $d->format( 'Y-m-d H:i' ), "\n";
?>

which gives the output: 2010-11-25 19:03.

This seems to work, but unfortunately, even the approach of storing the timestamp and timezone identifier is not going to work correctly under certain circumstances. In order to find out why, we have to go back to January this year.

Imagine that in January this year—January 15th to be precise—we run the following script to determine the timestamp of a date/time two months in the future:

format( 'U' );

$d = new DateTime( "@$ts" );
$d->setTimeZone( new DateTimeZone( $tz ) );
$d->modify( '+2 months' );

echo $d->format( 'U Y-m-d H:i' ), "\n";
?>

This returns the timestamp 1268664060 for the date/time 2010-03-15 10:41:00.

Now skip forwards to the current date and time. If we used this calculated timestamp and the timezone identifier America/Santiago today to generate a date/time string, we would however get a different output. The following example shows this:

setTimeZone( new DateTimeZone( $tz ) );

echo $d->format( 'Y-m-d H:i:s' ), "\n";
?>

which gives the output: 2010-03-15 11:41:00 (and not 2010-03-15 10:41:00).

The difference in output is not a bug, but is caused because some countries change Daylight Savings Time rules quite frequently. In this case Chile decided (on March 4th) that instead of going forwards to DST at midnight March 14th, they will delay that to April 4th. When we ran the code on January 15th, the rules still thought that March 15th would already be on normal time again, outside of DST. But running the code now, with the updated rule set, we find that DST is still in effect until April 4th. The following example shows the transitions from/to DST for Santiago in 2010:

getTransitions(
    strtotime( '2010-01-01 00:00 UTC' ),
    strtotime( '2010-12-31 00:00 UTC' )
);

foreach ( $transitions as $t )
{
    echo $t['time'], ' ', $t['offset'] / 3600, ' ', $t['abbr'], "\n";
}
?>

If we run this script with PHP 5.3.1 or 5.3.2—which still have the old incorrect rule set—the output is:

2010-01-01T00:00:00+0000 -3 CLST
2010-03-14T03:00:00+0000 -4 CLT
2010-10-10T04:00:00+0000 -3 CLST

In the SVN repository, the rule set has been updated, so snapshots of PHP 5.3-dev have the correct rules and the script will show:

2010-01-01T00:00:00+0000 -3 CLST
2010-04-04T03:00:00+0000 -4 CLT
2010-10-10T04:00:00+0000 -3 CLST

This leads to the conclusion that storing timestamps and timezone identifiers is not good enough either, unless you want an exact point in time, as opposed to the more expected date/time in a location. So how should you store the latter then? Basically, in the same way that DateTime objects are serialized in PHP 5.3. Let us imagine again, that the following code is run on January 15th again:

modify( '+2 months' );
$s = $d->format( 'Y-m-d H:i:s' );
$ts = $d->format( 'U' );
echo "$s (ts=$ts)\n";
?>

which gives the output 2010-03-15 10:41:00 (ts=1268664060). The next example is run with a recent rule set (such as in PHP SVN) today:

format( 'Y-m-d H:i:s' );
$ts = $d->format( 'U' );
echo "$s (ts=$ts)\n";
?>

which gives the output 2010-03-15 10:41:00 (ts=1268660460). As you can see, now the date/time itself is correct, although there is a different timestamp. PHP's DateTime serialisation does something similar:

setTimeZone( new DateTimeZone( 'America/Santiago' ) );
var_dump( $d );
?>

which gives the output:

object(DateTime)#1 (3) {
  ["date"]=>
  string(19) "2010-03-29 11:32:08"
  ["timezone_type"]=>
  int(3)
  ["timezone"]=>
  string(16) "America/Santiago"
}

For things to work correctly, you need to have an up-to-date rule set. PHP versions are not released as often as the timezonedb (sometimes more than 20 times a year) and to address this issue you can install the pecl extension timezonedb with pecl install timezonedb.

To store information in a database, I would use a char column-type to store the whole America/Santiago timezone identifier and another char column-type to store the date/time (in the yyyy-mm-dd hh:ii:ss format). Alternatively, you can pick a 'datetime' column-type, as long as that type ignores timezones altogether. For MySQL that is the DATETIME column-type, and for PostgreSQL the TIMESTAMP or TIMESTAMP WITHOUT TIME ZONE column-types.

Databases rarely handle timezones, daylight savings time and rule changes correctly, so avoid the database specific functionality all together. Using either a char or a timezone-less 'datetime' column-type would still allow you to sort and by using a 'datetime' column-type you can even do calculations.

Happy summer time!

Obtaining the next month in PHP

Fri, 08 Jan 2010 09:13:00 +0000

Obtaining the next month in PHP

London, UK

Friday, January 8th 2010, 09:13 GMT

Over and over again PHP users complain that next month in PHP's date-string parser doesn't go to the next month, but instead skips to the one after next month; like in the following example:

modify( 'next month' );
echo $d->format( 'F' ), "\n";
?>

The output of the little script will be March. March obviously doesn't follow January as February is in between. However, the current behavior is correct. The following happens internally:

next month increases the month number (originally 1) by one. This makes the date 2010-02-31.
The second month (February) only has 28 days in 2010, so PHP auto-corrects this by just continuing to count days from February 1st. You then end up at March 3rd.
The formatting strips off the year and day, resulting in the output March.

This can easily be seen when echoing the date with a full date format, which will output March 3rd, 2010:

format( 'F jS, Y' ), "\n";
?>

To obtain the correct behavior, you can use some of PHP 5.3's new functionality that introduces the relative time stanza first day of. This stanza can be used in combination with next month, fifth month or +8 months to go to the first day of the specified month. Instead of next month from the previous example, we use first day of next month here:

modify( 'first day of next month' );
echo $d->format( 'F' ), "\n";
?>

This script will correctly output February. The following things happen when PHP processes this first day of next month stanza:

next month increases the month number (originally 1) by one. This makes the date 2010-02-31.
first day of sets the day number to 1, resulting in the date 2010-02-01.
The formatting strips off the year and day, resulting in the output February.

Besides first day of, there is an equivalent last day of to go to the last day of a month. The following example demonstrates this:

modify( 'last day of next month' );
echo $d->format( 'F jS, Y' ), "\n";
?>

This outputs February 28th, 2010. Internally the following happens:

next month increases the month number (originally 1) by one. This makes the date 2010-02-08.
last day of increases the month number by one, and sets the day number to 0, resulting in the date 2010-03-00.
PHP then auto-corrects the invalid day number 0 by removing one from the month and skipping to the last day of that month, resulting in 2010-02-28.

I hope this clears up some of the behaviour of PHP's Date/Time handling. For more information on Date/Time Programming with PHP, please refer to my book "php|architect's Guide to Date and Time Programming" that is available through Amazon.

Unix Epoch and PHP's calendar system

Thu, 12 Nov 2009 12:32:00 +0000

Unix Epoch and PHP's calendar system

London, UK

Thursday, November 12th 2009, 12:32 GMT

I just saw a commit flying past as a response to PHP bug #50155. While right now it is proper to define the Unix Epoch at "1970-01-01 00:00:00 UTC", UTC wasn't actually defined until 1972. So it would be more correct to define the Unix Epoch as "the number of seconds elapsed since midnight proleptic Coordinated Universal Time (UTC) of January 1, 1970, not counting leap seconds." (from Wikipedia). What the bit "not counting leap seconds" means, I've already explained before in Leap Seconds and What To Do With Them.

Similarly, PHP's internal calendar is the ISO 8601 calendar. This is a modification of the proleptic Gregorian calendar. The Gregorian calendar implements the current set of leap years every 4 years, but not every 100 years, but again every 400 years (to get to an average year length of 365.2425 days). Obviously this calendar is only in use since 1582 (some countries adopted it as late as the 1900s), so using days like 1066-10-14 in the Gregorian calendar makes little sense because that calendar didn't exist back then. Now, PHP's ISO 8601-based calendar even modifies the Gregorian calendar by including the year 0. The Gregorian calendar goes straight from -1 to 1 which is a pain to do proper date calculations with. Therefore the ISO 8601 calendar uses Astronomical year numbering.

php|architect's Guide to Date and Time Programming

Wed, 03 Jun 2009 19:06:00 +0000

php|architect's Guide to Date and Time Programming

Skien, Norway

Wednesday, June 3rd 2009, 21:06 CEST

In the past year I've been working on a book, describing PHP's Date and Time functionality in great detail. It's now finally out under the name " php|architect's Guide to Date and Time Programming ". Look at the pretty cover! It's actually a picture I took some months ago from the church in Skien , where I am still living (more about that in a later post).

The book is devoted on helping you understanding dates and times, as well as how PHP deals with it. It explains how PHP parsers and reads dates and times in strings including possible pitfalls; timezones, their history, the annoyances and how to deal with them properly; formatting and displaying dates and times; and last but not least, how to handle intervals and iterating over time. There is more information on the website where you can also buy this awesome book. And in case you're wondering "what? dates and times are really simple!", then this book is definitely for you.

Rumour is that all attendees of the Dutch PHP Conference will receive a copy with their tickets, if you're asking nicely I might even sign your copy! I'll be there to give a talk on "PHP on the D-BUS" and will show you how to talk to skype, GSM chips and other desktoppy things!

Leap Seconds and What To Do With Them

Thu, 01 Jan 2009 22:03:00 +0000

Leap Seconds and What To Do With Them

Skien, Norway

Thursday, January 1st 2009, 23:03 CET

The start of this new year started with some buzz about a leap second being introduced between Dec 31st 2008, 23:59:59 and Jan 1st 2009, 00:00:00. I've had people ask where this leap second actually comes from, and whether you need to worry about it in your applications. To understand leap seconds means, unfortunately, understanding how time is actually kept.

There are many different time keeping scales. They are either defined as an arbitrary value, or obtained from astronomical observations. First of all, there are the variants of Universal Time. UT1 is the principal form of Universal Time. UT1 is the same all over the world, and is related to the Earths rotation. The rotation speed of the Earth is not precise, and is slowed down by the tidal friction of the Moon among things. Obviously, this is a bad base to build a stable and precise timekeeping scale on.

An (SI) second is nowadays defined as "the duration of 9 192 631 770 periods of the radiation corresponding to the transition between the two hyperfine levels of the ground state of the caesium 133 atom". This can be measured very accurately by atomic clocks and is the base for International Atomic Time (TAI, from Temps Atomique International). This scale however, does not take into account the gradual but increasing slowing down of the Earth's rotation and will therefore not reflect the approximation to "mean solar time" over longer periods of time. This makes it useless as a civil timekeeping scale.

For the latter purpose, Coordinated Universal Time (UTC, from Universal Time, Coordinated) has been established. UTC uses SI seconds but is adjusted for the slowing down of Earths rotation with leap seconds. Because it's not possible to predict the rotation of the Earth upfront, leap seconds are only added when the difference between UTC and UT1 approaches 0.6 SI seconds in order to keep UTC and UT1 not differ from each other for more than 0.9 seconds. UTC was defined (in the latest adjustment of its definition) as being 10 seconds different from TAI making 1972-01-01T00:00:00 UTC and 1972-01-01T00:00:10 TAI the same instant. Since 1972, 24 leap seconds have been introduced, making TAI differ from UTC with 34 seconds currently. As the Earth's rotation is slowing down at an ever increasing rate, the use of leap seconds will get more and more annoying. After the 25th century, it is likely that more than 4 leap seconds would be required every year. This is of course highly unpractical and a solution needs to be found for this.

There are two other timescales based on TAI. Terrestial Time (TT) as the modern astronomical standard for the passage of time. TT is defined as TAI + 32.184 seconds. GPS Time, as used by GPS satellites and receivers, was set to match UTC in 1980, but now deviates with 15 seconds due to the addition of leap seconds in UTC. GPS Time is defined as TAI - 19 seconds.

Greenwich Mean Time (GMT) is often used as synonym for UTC, although that is strictly not correct. GMT is an astronomical concept and therefore linked to UT1. Greenwich Mean Time is also used as timezone name for a timezone with zero seconds offset from UTC. It is for example used in winter in the UK and Ireland. During the summer, when the UK and Ireland switch to daylight savings time, GMT is not used at all. British Summer Time (BST) and Ireland Summer Time (IST) are then used instead.

A thing to realize is that leap seconds are added only at the end of June 30th and December 31st. The seconds are added at 23:59:59 UTC . This means that there is no need to add an extra second while counting down to the new year unless you're in a timezone that is UTC +0 seconds (such as GMT, or WET). Of course, in other locations the leap second is inserted as well at the same time. In Norway, the leap second was added at Jan 1st, 2009, at 00:59:59 for example.

Unix time (also called POSIX time) is defined as the number of seconds since Jan 1st 1970, 00:00 UTC, but without leap seconds. Although it is called Unix time , this timekeeping standard is used widely in many other operating systems and computing devices. Unix time simply counts the number of seconds since the epoch. A date/time expressed in those seconds is often called a Unix time stamp. Not supporting leap seconds means that Unix time does not have any way to represent the leap second in the form of 23:59:60. Instead it just uses the same second twice. In practical terms, that means that Unix time progressed like this during the year transition: 1230767999, 1230768000, 1230768000 and 1230768001. The first 1230768000 represents 2008-12-31 23:59:60 UTC, and the second 1230768000 represents 2009-01-01 00:00:00 UTC. For applications there is no way to differentiate between the two distinct times, although many applications will accept both as input, like this PHP example shows:

In order to synchronize the system time the Network Time Protocol (NTP) is widely used. NTP can use different sources of time information in an hierarchical fashion. The protocol knows how to deal with leap seconds to provide the correct Unix time stamp.

To recap the above information: UT1 is based on the actual rotational speed of the Earth and is therefore never accurate. TAI is a time scale based on the actual count of SI seconds. UTC also uses SI seconds, but requires a leap second in order not to deviate from the unstable UT1 which is in sync with the average time of when the Sun is due South (mean solar time). Unix time counts SI seconds, and resembles UTC except that instead of leap seconds, the Unix time stamp is simply used twice. GMT is both a different name for UT1 and a name for a timezone with an UTC offset of 0 seconds. PHP uses Unix time, and does not care about leap seconds.

Happy New Year!

Detecting Timezone By IP

Tue, 06 May 2008 20:09:00 +0000

Detecting Timezone By IP

Skien, Norway

Tuesday, May 6th 2008, 22:09 CEST

Through Planet PHP I found an article on Pre-populating forms with the timezone . I'd normally add a comment instead, but the comment would almost be larger then the original post, so I am instead writing up an entry myself. The post describes several ways to obtain the user's timezone and use that to pre-fill a form. None of them are working properly though. I'll try to explain for each of them why not, but first of all it is important to know what a timezone actually is.

A timezone is a set of rules that determines the UTC offset for different times around the year for a specific location. Because of daylight savings time, a specific location can have two (or more) different UTC offsets depending on what point in time you look at it. Some areas might have had different rules in the past, even in the same country. An example here is China, where currently there is only one timezone, but previously there were multiple. Thus there are different timezones for the different locations in China, even though the current UTC offset is the same for all of them. Timezones are identified by Country/City or Country/Subcountry/City.

The offering by MaxMind allows you to link a country/region combination to Timezone identifier. For the US it subdivides this per state even. However, it misses the different timezones for Russia, Australia and even Indiana, USA.

IP2Location only provides a single UTC offset per IP range, totally ignoring daylight savings time.

The Hostip.info solution I can't access because phpclasses requires some stupid registration.

As for the author's own solution, using JavaScript, is flawed at least partly as well. In his example JavaScript only returns a UTC offset. Luckily it is possible to detect the correct timezone quite a bit better by using some of PHP's functionality. The following bit of code uses both the UTC offset and the timezone abbreviation to find out the user's timezone:

This is still not perfect of course, because it would for example give the same result for most of Europe (Europe/Berlin).

Figuring out the user's timezone can be done by using a high-accuracy database of IPs to lat/longitudes such as hostip.info and MaxMind offer, as well as a mapping from location to timezone. The latter however, is not available as far as I know. If however a data file that has proper definitions of the different timezone boundaries exist (mostly, on a per-province level would be enough), then such a tool can be easily build. I saw that OpenStreetMap has such a map, but I can't really find the raw data for that unfortunately. It would however, be awesome to have such a data file.

British date format parsing

Sun, 02 Mar 2008 20:25:00 +0000

British date format parsing

London, UK

Sunday, March 2nd 2008, 20:25 GMT

During the PHP London conference (slides are here as usual), a number of British people were "complaining" that PHP's strtotime() and date_create() /new DateTime() functions do not understand the British date format "dd/mm/yyyy". Historically, the "English" text to date parsers in PHP only accept the illogical American format of "mm/dd/yyyy". Having seen this problem before, I recently added a new function/method to PHP's Date/Time support to address exactly this issue. From PHP 5.3 the new date_create_from_format() function and the DateTime::createFromFormat() factory method are available. As first argument they accept the expected format, and as second argument the string to parse. To parse a British date string, you would for example use:

format( 'd/m/Y' ), "\n";
?>

In case the passed string does not match the format, the function will return false. With the also new function date_get_last_errors() method you then can request more information about which part of the string could not be passed. The following example illustrates that:

The $errors contains a list of errors, indexed by the position in the string where the error was found. The above script would produce:

array(3) {
  [2]=>
  string(22) "Unexpected data found."
  [5]=>
  string(22) "Unexpected data found."
  [8]=>
  string(13) "Trailing data"
}

The format specifiers are mostly similar to the ones used for the date() function. Documentation on the new functions should make it into the documentation soon.

The Y2K7 problem

Tue, 06 Mar 2007 22:04:00 +0000

The Y2K7 problem

Skien, Norway

Tuesday, March 6th 2007, 23:04 CET

Right, that "Y2K7" is not a typo. It "refers to the Year 2007 problem, caused by a US-mandated change to Daylight Saving Time" ( wikipedia ). From this year the US (and Canada) changed the dates on when daylight saving time is active. DST now starts three weeks earlier, and ends one week later. This causes quite a lot of problems with software that hard coded those rules. On Linux (and other Unix-like operating systems) this can be remedied by updating the zoneinfo information - check your Operating System's documentation on how to do that.

PHP before PHP 5.1.0 relies on the Operating System to provide time zone information, while PHP versions 5.1.0 and higher come with a bundled time zone database. The database that ships with PHP 5.1.0 already covers the changes in the US and Canada, however, the US and Canada are not the only places changing rules. Many places on earth change those rules all the time which means PHP's database needs to be updated as well.

Every PHP version comes with the latest release of the time zone database at the moment PHP is released. However, it is also possible to just update this data by installing the PECL extension timezonedb . This extension is released whenever a new version of the Olson database becomes available. The current version is 2007c (2007.3).

Calculating start and end dates of a week.

Wed, 06 Sep 2006 07:25:00 +0000

Calculating start and end dates of a week.

Skien, Norway

Wednesday, September 6th 2006, 09:25 CEST

A friend asked "How do I calculate start (monday) and end (sunday) dates from a given week number for a specified year?" Instead of having to come up with your own algorithm you can simply do the following in PHP 5.1 and higher:

The format is " year 'W' weeknr ( daynr )?" where the default daynr is 1 being the Monday of that week. The daynr can be in the range 0 to 7. The weeknr is the ISO week number . Please note that the " year '-W' weeknr ( '-' daynr )?" format is only supported in PHP 5.2 and higher.

PHP lags 23 seconds

Tue, 10 Jan 2006 19:35:00 +0000

PHP lags 23 seconds

Skien, Norway

Tuesday, January 10th 2006, 20:35 CET

Bug report # 35958 must have the most obscure one ever:

"strftime usually returns a string from the number of seconds since 1 jan 1970. Now, it lags and returns a string representing 23 seconds too late."

If you know what's going on though, it isn't really that weird. Every once in a while the IERS announces a new leap second to "keep the broadcast standards for time of day close to mean solar time ". At the moment the difference is 23 seconds which is reflected in comparing the leap second adjusted time zone information file with the non-adjusted one. You can see the leap second with this little shell script:

#!/bin/bash
export TZ=Europe/Amsterdam
date --date "@1136073621"
date --date "@1136073622"
date --date "@1136073623"
export TZ=right/Europe/Amsterdam
date --date "@1136073621"
date --date "@1136073622"
date --date "@1136073623"

The output is:

Sun Jan  1 01:00:21 CET 2006
Sun Jan  1 01:00:22 CET 2006
Sun Jan  1 01:00:23 CET 2006
Sun Jan  1 00:59:59 CET 2006
Sun Jan  1 00:59:60 CET 2006
Sun Jan  1 01:00:00 CET 2006

The output with the "right/Europe/Amsterdam" timezone is actually the correct time, but this will obviously confused too many computer programs. Most often you will not see this one in use.

The new date/time code in PHP 5.1 does not support the leap second either. I was playing with it while developing, but thought it to be to confusing. Seems I was right there :)