Tony Finch

iCalendar is wrong

2009-12-09T23:20:42Z

(This article is a much-expanded version of a comment I wrote months ago on mathew's blog.)

There's a programmers' rule of thumb that timestamps should always be stored in a form that's unambiguously inter-convertible with UTC, or some reasonable approximation such as POSIX time_t. In particular, you should never store local time without also storing its timezone, and you should represent timezones as UTC offsets instead of using a familiar but ambiguous abbreviation. For textual representations the right answer is usually ISO 8601 / RFC 3339.

This rule of thumb is good if you are storing the times of events that happened in the past, such as in logs or in message headers. However it isn't good for events that happen in the future, when those events have any bearing on time as used by people in the outside world. The reason for this is the instability of time zones.

Bad solutions to timezone problems

The problem is particularly clear for repeating events. If you specify an event's time of day using a fixed offset from UTC then it will be an hour wrong for half of the year when because the time zone offset is different in winter and summer time. This is why Unix's cron scheduler works in local time.

The solution chosen for iCalendar is to store the complete timezone data (summer and winter offsets and the changeover schedule) alongside the event. It has a number of problems. Firstly bloat, though iCalendar reduces that by allowing multiple time stamps to refer to the same timezone data. Secondly, it isn't robust against changes to a timezone's DST schedule.

An outstanding example of failure caused by storing timestamps in the wrong format was provided by the US DST schedule change in 2007. People running Microsoft Exchange had to run a special tool that scanned the entire database to find timestamps that needed adjusting. If the data model had been designed properly this would not have been necessary.

The underlying error is to do the timezone data lookup too early. As we learned from David Wheeler, the fix is to add a layer of indirection so that the lookup can be delayed. Instead of storing the numerical offset, store a reference to the timezone, e.g. its name from the Olson tz database.

iCalendar TZID values are typically Olson timezone names, or something very similar, but this is not required by the specification. There is still no interoperable standard for timezone names, so iCalendar objects have to include the complete VTIMEZONE data, not just the name. There are plans to fix this, but it's unclear if the standard timezone name registry will be based on the Unicode Common Locale Data Repository, or perhaps the Olson tz database (depending on how its management changes around ADO's retirement) or something else.

Unfortunately timezone names are still not a complete solution. As well as DST schedule changes, there are often timezone boundary changes. If an event is to happen in a place that is affected by a boundary change, and its time is recorded with respect to the place's old timezone, then this time will be wrong after the change. Indiana has provided many instances of this problem, since it straddles a timezone boundary, each county in the state chooses its timezone independently, and every so often some of them will change their mind about whether they want to follow Central Time or Eastern Time or even both (depending on the time of year). The solution to unpredictable timezone boundary changes is, of course, another layer of indirection.

My solution

The time of an event in the future should be recorded in local time coupled with the event's location. The location is used to look up the timezone, and the timezone data determines the UTC offset. (I should probably clarify that Olson tz names are not locations even though they are derived from locations. It's nonsense to say that the Edinburgh Tattoo will occur in Europe/London.)

Recording the location of an event instead of its timezone makes all sorts of problems simpler, not just problems resulting from timezone mutations. A lot of the benefit comes from just making the data aware of locations and the effect they have on scheduling. Also, perhaps unexpectedly, it allows extremely simple platforms that are unaware of timezones!

Often all that is required of a PDA calendar is to keep a single person's appointments, and the times only need to be meaningful wherever that person is going to be when the event occurs. In this simple case, if all the times are stored in local time at the appointment's location, the PDA does not need to do any timezone translation in order to display them in a useful way: the stored time is good enough. In this scenario, the only timezone manipulation that occurs is the user manually resetting the PDA's clock when a timezone offset change happens (because of travel or because of DST).

It's more usual to want to share calendar events, in which case you soon encounter situations where it's useful to know when events in other timezones will occur according to your own local time. If the software knows your current location, it's a straightforward matter to translate times from place to place. This should not be done significantly earlier than when displaying the time. For example, in a calendaring app based on early binding of events to timezones, the programmer might be tempted to translate an event's time to the user's local timezone when importing the event. This optimization is clearly bogus in a location-based app, because it amounts to moving the event to a location where it is not occurring!

One case where it seems not to make sense to fix an event in a location is when it occurs in more than one place: telephone calls or (worse) conference calls. The thing to do in this situation is to decide on a primary location, such as the location of the organizer, and list the other locations as supplementary. This allows the software to display all the relevant times, so it's immediately apparent what the timing is for each participant and if it happens to be inconvenient for any of them. If politicians happen to muck around with any of the timezones the organizer is naturally responsible for any adjustments that may be necessary, so it makes sense to keep their view of the event as straight-forward as possible.

An interesting case is travel between timezones. It's usual for flight bookings to give departure and arrival times in the local time of the origin and destination locations, which I always find confusing. However if a computer has this information, it can easily display both times in both timezones and work out the total travel time. It would be even nicer if your PDA could use this information to automatically update its idea of your location, and therefore its idea of local time. If it can use this method to work out where you will be in the future it could also display future events with all three relevant times: their native time, the time according to your current timezone, and according to the timezone of your location when the event occurs.

iCalendar has the concept of a "floating" timestamp, which represents the time in whatever is your current timezone. Floating timestamps cannot be communicated reliably to another person, because the time they represent will be interpreted according to the recipient's location, not yours. One way to make them reliable would be to add another layer of indirection: attach an event to a person and provide a way of looking up the person's location. This is absurdly complicated and an invasion of privacy, and I think it shows that the concept of floating events (occurring wherever you are at the time) is unwise. They do make sense for purely personal events, such as wake-up alarms or medication reminders - you don't want your PDA to tell you to wake up in Cambridge as usual when you are currently in New York. But if an event involves more than one person and its location is in doubt, it's better to give it a provisional location so that changes have to be communicated explicitly.

With a local time plus location model, if a timezone does change, the only events that are affected by the change from the point of view of the software are also affected from the point of view of the human world. For instance, a conference call that spans multiple timezones may need to be rescheduled because its local time may change in some of the participants' locations, and this may lead to scheduling clashes that were not there when the call was originally organized. Events at a single location that occur near the old and new clock changes may need to be rescheduled to cope with inserted or omitted hours - but it's rare to schedule events for the small hours of Sunday morning. The majority of events that fall between the old clock change and the new clock change are not affected: no special bulk data fix-up tools are required.

Complications

The local time plus location model is not quite sufficient as I have described it so far. If an event is scheduled near the time the clocks go back, the local time by itself is not enough to tell if it occurs in the hour before or the hour after the change. The way to fix this is to add a disambiguation flag. However, once again the usual way this is done is wrong. POSIX struct tm, for example, has a tm_isdst flag, which states whether the broken-down time is expressed in summer time or not. The problem here is that this flag can disagree with the timezone data: it's nonsense for the flag to be zero for a time in the middle of summer. It also means correct timestamps get turned into nonsense when politicians mess around with timezones.

The correct solution is for the flag to apply only when the the time is ambiguous. At other times the flag must be ignored and should be omitted when generating timestamps. In effect the semantics of the flag are "prefer the earlier/later time if there's more than one". When phrased like this, the flag also works in weird cases. William Willett's original proposal was to phase DST in and out by skipping or repeating 20 minutes on four successive Sundays in April and September. My silly "sunrise time" idea involves changing the clock by a minute or so most nights. The isdst flag doesn't have enough bits to identify which version of local time a timestamp belongs to when there are more than two, but the disambiguation flag never needs to distinguish between more than two.

The second complication is those odd locations that do not have a single agreed idea of local time. Decades ago in the USA, arguments over DST sometimes meant that different parts of government (federal/state/local) would have different ideas of local time; see David Prerau's book "Saving the Daylight" for examples. At present the most well-known instance of this problem is Xinjiang, the Uighur Autonomous Region of China. Officially, the whole of China is on Beijing time, UTC+8. This is a bit uncomfortable in Xinjiang in the far west of the country, so the independent-minded Uighurs use their own time, UTC+6, even though their Han neighbours use the national time. (See the LA Times for a report on this subject.)

I think the way to accommodate places like Xinjiang is to treat locations as a geo-political concept rather than a purely geographical one. So you might have "Xinjiang (Han)" and "Xinjiang (Uighur)" in your location database. Xinjiang also breaks the Olson/Eggert tz naming scheme, so I think there's unlikely to be any particularly elegant way to handle it.

The third complication is how to specify locations. A significant problem for many calendaring applications is that we lack a database of which locations are in which timezone. This is usually viewed as a user-friendliness problem, but for my proposal it is more fundamental. Furthermore there's an incompatibility of scale between the kind of location that makes sense for a timezone database (e.g. centred around large cities) and the kind of location that makes sense for a meeting (e.g. room C304). I think it's reasonable to make people enter enough detail about locations to fill the gap between the room-level resolution and the city-level resolution. Only one person should ever have to enter the details of a particular location into a system (or set of connected systems) after which everyone else can re-use the data, so the burden should be small.

This leads to another problem with iCalendar: its idea of a location is both too weak and too complicated. You can specify latitude and longitude, which isn't very practical for software that lacks a built-in map, nor can it be translated into a timezone in Xinjiang. You can also (as well as or instead) specify a human-friendly location as a free text string with an optional URL pointing to a more computer-friendly representation. This latter can be anything, though you hope it is something sensible like a vCard containing a postal address. A vCard address has a fixed format which I suppose can be stretched a bit to cover meeting rooms and other ad-hoc locations in such a way that they can be tied to a timezone, but it isn't designed for the purpose.

Conclusion

Sadly it seems that the world is stuck with iCalendar, and when timezone-related problems occur calendar programmers blame politics or DST, rather than their inadequate data model. What is worse is that it appears to be very unlikely that a properly designed calendar program could interoperate with iCalendar data without loads of ad-hockery and lossage because of the mis-match between the data models.

How annoying.

FreeBSD unifdef(1) and factor(6)

2009-11-26T01:04:14Z

Earlier this week I got a bug report for unifdef from Jonathan Nieder, who now maintains the Debian package. This prompted me to do some work on it, which it sorely needed - I hadn't touched it since March last year, and I had a bunch of unincorporated patches from Anders Kaseor, and I haven't backported the fix for an embarrassing bug from FreeBSD-8 to FreeBSD-7.

I've dealt with all but the last of these, and the latest version unifdef-1.188 is available from http://dotat.at/prog/unifdef/. I have also committed it to FreeBSD-9-CURRENT. When the code freeze for FreeBSD-8-STABLE is lifted (FreeBSD-8.0 should be released RSN!) I'll backport unifdef to the 8-STABLE and 7-STABLE branches.

This evening I also did a bit of work on FreeBSD's version of factor. I noticed a while back that it has a performance bug: it's very slow to factorize some numbers, for example my phone number. I got its bignum factoring code from NetBSD back in 2002, so I went back there to see if they knew anything about this bug. It turns out that jsm28 fixed the bug in 2004. Five and a half years later I've finally committed the fix to FreeBSD...

More spam bot signatures

2009-11-07T18:29:03Z

There's another spam bot heuristic which is the exact complement to the one described in my prevous post. The idea is to keep rack of how many different HELO domains an SMTP client uses.

  defer
    message   = Probable spam bot HELO varies between $sender_rate domains
    # whitelist checks go here
    ratelimit = 2.2 / 1w / per_conn / strict \
      / unique=$sender_helo_name / $sender_host_address

This is mostly the same as the code in my previous post, but the lookup key is the client's IP address, and we only increase the measured rate if the client uses a different unique HELO domain.

This check also works very well at detecting spam bots. When testing it I noticed that one particular bot likes to use a HELO domain consisting entirely of random upper-case letters, and it only talks to one of our servers with the lowest IP address. So I added a specific (cheaper) check to deal with it. This causes the bot to go away, and legitimate senders will retry with a different server.

  defer
    message   = Probable spam bot HELO - please try another server
    condition = ${if and{{ eq{$primary_hostname}{ppsw-0.csi.cam.ac.uk} } \
                              { match{$sender_helo_name}{^[A-Z]+\$} }} }

There is a small risk of false positives with the variable HELO domain test. Some outgoing mail server clusters are behind a NAT, so we see HELO domains from multiple servers coming from the same IP address. I also found a number of false positives for the popular HELO domain check (most prominently rediffmail.com and easyjet.com). The way I'm dealing with them (which I hope will work in the long term) is as follows.

For the popular HELO domain check described in my previous post, I maintain a lookup table of legitimate HELO domains that trigger the check. The following replaces the hard-coded check for localhost.localdomain that appears in my previous entry.

    condition = ${if !match_domain{$sender_helo_name}{cdb;DB/helo_ok.cdb} }

As well as allowing through clients whose HELO domain can be verified, I now also check dnswl.org for well-known legitimate senders, and I maintain a table of sending hosts that slip through the other checks.

  ! verify   = helo
  ! hosts    = +helo_ok
  ! dnslists = list.dnswl.org

To avoid problems with false positives, my anti-spam-bot checks return a temporary error code ("defer" in Exim-speak instead of "deny"). I then have a nightly audit script which looks for hosts that appear to be repeatedly retrying a messages and which should be added to my whitelist tables. It might be possible to automate this table maintenance, again using the ratelimit feature, but I expect that will require a bit more experience to determine the right thresholds.

Spam bot signatures

2009-11-05T15:57:53Z

Recently I have been investigating spam bot signatures, specifically the characteristic domain names they choose to put in their SMTP HELO commands. A lot of spam bots use the same HELO domains from lots of different compromised PCs, which makes them quite easy to spot and block without any risk of blocking legitimate email. This kind of block can take care of about 15%-20% of spam without relying on 3rd party services like DNSBLs. Of course, this is one of the techniques used to populate the Spamhaus XBL so the only advantage of doing it yourself is if you want to spot spam bots that have not yet been spotted by the Spamhaus guys.

Steve Champeon's Enemieslist service is the highly developed commercial implementation of this idea. His patterns are much more comprehensive, covering spam bot signatures, domestic IP connectivity (like the Spamhaus PBL), and spam-infested netblocks.

Yesterday evening I was thinking about how to automatically identify spam bot signatures, when I realised that I had already written the code to do the job! I wanted to count how many different IP addresses were using the same HELO domain, and block connections that used excessively popular domains. All I needed was a few lines of Exim configuration:

  deny
    message   = Probable spam bot HELO seen from $sender_rate networks
    condition = ${if !eqi{localhost.localdomain}{$sender_helo_name} }
  ! verify    = helo
    ratelimit = 4 / 1w / per_conn / strict \
      / unique=${mask:$sender_host_address/24} / ${lc:$sender_helo_name}

Let's unpack this in reverse order.

We're measuring the rate of use of HELO domains, so the ratelimit key is ${lc:$sender_helo_name}. It's forced to lower case so that SERVER and server are treated as the same thing.

But we don't care about the total usage rate, only the rate of uses from different unique IP addresses. The unique= option invokes the Bloom filter code to avoid counting each spam bot more than once. In fact we only count different unique /24 network blocks, in order to avoid false positives from mail clusters in which all servers use the same name. For example, Facebook's MTAs all say HELO mx-out.facebook.com though they are spread across about 100 IP addresses on a couple of /24 networks.

The strict option means keep counting even when the measured rate has passed the limit. The per_conn option means only count once for each connection (which mainly helps with efficiency).

The smoothing period is set to one week, which should mean that Exim doesn't easily forget which HELO domains have been abused.

I've currently got the limit set to 4 blocks of /24. It might even be reasonable to reduce this to 3. I'll need to run it a bit longer to see if any more odd false positives sneak out of the woodwork.

We do not apply this check if the DNS agrees with the HELO domain. There are some legitimate host names which are being heavily abused by spam bots, such as mail.aol.com and mx54.mail.com, so we want to block them if the connection comes from anywhere other than the host itself. (Sadly Facebook's MTAs are misconfigured so they don't pass this check.)

The only exception to this (so far) is localhost.localdomain which is the result of a popular misconfiguration (or lack of configuration) on legitimate Unix MTAs. If I find any other false positives they'll get checked in a similar way.

ETA rediffmail.com also needs whitelisting - it's the mail service of rediff.com which is a portal for Indian expats. Also easyjet.com.

This heuristic seems to catch about three different kinds of spam bot behaviour.

The HELO domain is the same as the domain in the MAIL FROM address. Spammers like to forge email "from" surprisingly few popular sites.
The HELO domain is one of a few bare hostnames, such as pc or computer - I guess they use the name of the compromised host.
The HELO domain is a parent domain of an ISP's edge network, e.g. telesp.net.br.

I'm really pleased by how easy and effective this has turned out to be. The only annoyance is that it took me 20 months to realise that my Bloom filter ratelimit code could do this! Also I hope there aren't too many lurking gotchas that I haven't spotted yet.

This check really shows up a long-standing weakness in Exim's hints database implementation. It just uses a local DBM file to store ratelimit data, so each individual server in my SMTP cluster has to accumulate data on spam bot HELO domains without being able to benefit from the experience of the rest of the cluster. I suppose I should spend some quality time with Tokyo Tyrant...

bogon spam

2009-10-30T15:59:36Z

Someone from Craigslist posted to NANOG saying they were suffering from a lot of spam from unallocated IP address space. So I wrote a program which downloads the RIR allocation statistics tables, compiles a list of allocated /24 blocks, and scans Exim's logs for connections from unallocated space. It turns out we are not suffering from a lot of spam from unallocated IP address space - we get on average less than one bogon connection each day.

ENHANCEDSTATUSCODES FAIL

2009-10-30T13:13:21Z

I sort of expect an SMTP server that claims to support enhanced status codes to, y'know, put enhanced status codes in its responses...

220 bill.internal.[redacted] ESMTP Symantec Mail Security
EHLO ppsw-0.csi.cam.ac.uk
250-bill.internal.[redacted] says EHLO to 10.0.64.17:58126
250-ENHANCEDSTATUSCODES
250-PIPELINING
250 8BITMIME
MAIL FROM:<dot@dotat.at>
250 MAIL FROM accepted
RCPT TO:<asldkja@sjkfsajhg>
554 Recipient address rejected: User unknown
RSET
250 RSET OK
QUIT
221 bill.internal.[redacted] closing connection

Penalising senders who use invalid recipient addresses

2009-10-28T00:21:34Z

Here's an idea for penalising senders who use out of date mailing lists: temporarily reject recipients depending on the relative proportion of valid and invalid recipients. The more invalid addresses they try to send mail to, the slower they'll be able to deliver mail.

This isn't suitable for use in all situations, because there are badly-run lists that nonetheless carry legitimate and desirable traffic. There are also well-run lists that will be unreasonably penalised by my site's annual bulk cancellations of departing users. But it might be useful in borderline situations, e.g. when the sender has passed the Spamhaus check but failed a DNSWL check.

The way to implement this is using my rate measurement equation to keep track of each sender's valid and invalid recipients, two rates per sender. When a sender addresses an invalid recipient, update their invalid recipient rate and also calculate their decayed valid recipient rate (which is just a * r_old). When a sender addresses a valid recipient, do the complementary calculations. The proportion of valid recipients is r_ok / (r_bad + r_ok). Pick a random number between 0 and 1 and if it's greater than that proportion, return a 450 (temporary rejection) otherwise return 250 (ok) or 550 (bad).

If you are feeling mean you can crank up the deferral rate faster than the invalid recipient rate. The rationale for this is that a list with a 50% validity rate is shockingly bad and probably deserves more like a 90% deferral rate (say). So calculate (r_ok / (r_bad + r_ok))ⁿ for some n that increases with your meanness, and use the result for comparing with the random number.

If you are feeling kind you can pick your random numbers between 0 and slightly less than 1, so senders don't have to be whiter than white to avoid penalties. This tweak might make the technique less troublesome.

(I initially thought about deferring just valid recipients, but that has the effect of cleaning the invalid recipients out of the sender's queue, so when they retry the deferred valid recipients, the proportion of validity we measure for them will go up. Deferring both valid and invalid recipients solves this problem and makes the implementation simpler and more pleasingly symmetrical.)

This is actually fairly similar to some of the logic in SAUCE. It keeps a single annoyance number per sender which gets increased when they do something bad and decreased when they do something right, and which decays exponentially so past behaviour fades from memory. SAUCE's main penalty is teergrubing, though it will defer incoming mail if provoked enough, or if it is greylisting the sender. I don't think there's much point in teergrubing except to trigger bugs in spam bots (hence sendmail has a simple greet_pause feature, but no general teergrubing). However there might be some benefit from more intelligent dynamic greylisting.

DNSSEC

2009-09-16T13:54:49Z

Delegation, delegation! Delegation,
that's what you need.
If you want your names to nest,
and you don't want to host the rest,
ooh-ooh, delegation's what you need!

So last night I laughed out loud at a DNSSEC RFC. I blame Douglas Adams.

Near the start of Life, the Universe, and Everything, Arthur Dent and Ford Prefect escape from prehistoric Earth on Eddy's time-travelling sofa, and find themselves at Lord's cricket ground. There, shortly before all hell breaks loose, they spot Slartibartfast's spaceship hidden with a SEP. An S.E.P. is a kind of cloaking device which works at a psychological level. It prevents you from seeing something, or rather it forces you to disregard it because it is Somebody Else's Problem and nothing you need to worry about.

In DNSSEC there are (by convention) two kinds of key. Key Signing Keys are as secure as possible (the keys are big and their private parts can be kept offline) so that they can have a long lifetime. Zone Signing Keys are smaller to reduce packet sizes and CPU usage, and their private parts need to be kept online to support dynamic DNS; to compensate for their relative lack of security they have a shorter lifetime. The KSK's public part is used as a "trust anchor" that you give to other people so they can authenticate the contents of your zone. You want it to have a long lifetime so you don't have to keep bothering them with updates. Although the KSK/ZSK arrangement is just a convention, it turns out to be useful to have some explicit signalling in the protocol to distinguish them, so they added a flag bit to DNSKEYs for this purpose. However they chose a different name for the bit to emphasize that you can manage your keys in unconventional ways, so it is called the "secure entry point" bit. This indicates the key is intended to be used as a trust anchor (e.g. in RFC 5011 rollover).

To me the SEP bit is the "somebody else's problem" bit. But it has exactly the opposite effect of Douglas Adams's SEP, since it's the bit you set to make other people pay extra attention. I thought this was funny, but then I had been drinking Ardbeg.

Firewall fallout / Exchange oddity

2009-09-02T16:10:46Z

This morning my colleagues turned off their Cisco PIX SMTP fuxup mode and most of the failing email finally made it through. But not all. So I broke out tcpdump again.

BTW, a handy way to split a tcpdump output file called dump into separate TCP connections is something like the following. This assumes that one of the endpoints of each connection is always the same port on the same server IP address.

    tcpdump -vvv -r dump |
    sed '/.* clientname[.]\([0-9]*\) .*/!d;s//\1/' |
    sort -u |
    while read p; do tcpdump -r dump -w dump.$p port $p; done

I looked at the connections which ended with a timeout during the data transmission phase. It turned out that just two messages were still having problems. And what strange beasts they were. The following is a paraphrase of the key features of the messages.

MAIL FROM:<mailserver@dept.cam.ac.uk> SIZE=12345
250 OK
RCPT TO:<exchange@dept.cam.ac.uk>
250 Accepted
DATA
354 Enter message, ending with "." on a line by itself
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: application/ms-tnef;
        name="winmail.dat"
Content-Transfer-Encoding: binary
Subject: Service Unavailable
Date: Wed, 2 Sep 2009 12:34:56 +0100
Message-ID: <i6wnzxxs3ctmf9qb@mailserver.dept.cam.ac.uk>
X-MS-Has-Attach:
X-MS-TNEF-Correlator: <i6wnzxxs3ctmf9qb@mailserver.dept.cam.ac.uk>
Thread-Topic: Service Unavailable
Thread-Index: GV7mWrV6mdBQLdgGFAfkSCsA
From: "/O=IT Support/OU=Department/cn=Configuration"
      "/cn=Servers/cn=MAILSERVER/cn=Microsoft Public MDB"
      <mailserver@dept.cam.ac.uk>

... loads of binary TNEF guff with lots of UTF-16 ...

The X.400 address is funny, but what was actually causing the problem was the attempt to send a binary message. Our servers don't support the BINARYMIME extension, so this is verboten. The reason it caused a timeout is that (being binary rather than lines of text) the message didn't end with a CRLF newline sequence, so the DATA dot-CRLF terminator did not appear at the start of a line, so our server thought it was part of the data and continued waiting for more.

Bizarre. Why is the message being sent to my servers when it should have been delivered internally? (Its recipient is a departmental address.) Why isn't Exchange downgrading the binary content as required by the specification? Alternatively, why is it using DATA to send a binary message when you can only do that using the BDAT command?

Very strange.

More firewall hate

2009-09-01T18:28:03Z

I've spent most of this afternoon staring at tcpdump traces trying to work out what is causing connections from an Exchange server to my Exim servers to be lost.

The original symptom was that our sending rate measurements for the server were going crazy - increasing massively seemingly without a corresponding increase in traffic. This happens because (in our configuration) if a connection is going OK (valid sender and recipient addresses, etc.) Exim doesn't log anything until a message is accepted. Our rate measurement calculations are performed when processing RCPT commands, before the connections were lost, and also before they got logged. (This is, obviously, less than ideal.) So the Exchange server's sending rate seemed to go up for no reason.

I temporarily increased our logging which verified that this was the correct explanation. But why were the connections being lost? I ran tcpdump for a bit and looked at the contents of the connections from the problem host. This showed that a couple of messages with lots of recipients were being repeatedly retried, only to fail because the connection was lost mid-transaction. Most messages were getting through OK.

I suspected a broken firewall, so I sent a message to the relevant staff giving them some details of the problem and asking if they have a firewall that might be causing it. I turned off tcpdump to stop it filling the disk :-) and kept half an eye on the logs. Soon a new and useful bit of information turned up: (I've edited this log extract a bit.)

2009-09-01 18:00:20 +0100
    SMTP protocol synchronization error
    (next input sent too soon: pipelining was advertised):
    rejected "XXXX TO:<abc123@cam.ac.uk>"
    H=mailserver.dept.cam.ac.uk [131.111.999.999]:50555 I=[131.111.8.131]:25
    next input="RCPT TO:<abc123@cam.ac.uk>\r\nRCPT TO:<abc123@cam.ac.uk>\r\n"

Exim is complaining about the XXXX command because only certain specific commands may be pipelined and XXXX isn't one of them. So why is the Exchange server sending gibberish XXXX (sic!) instead of RCPT? In fact it isn't. I took another tcpdump and found that the problem command fell across a packet boundary: one packet ended with @cam.ac.uk>\r\nXX and the next one started with XX TO:<.

This is an absolutely classic signature of the utterly braindamaged Cisco PIX firewall's SMTP fuxup mode. Oh it aggravates me so much. It replaces SMTP commands that it doesn't know with XXXXes, and it's too stupid to handle packet boundaries correctly.

I sent another message to the department's IT staff telling them to fix their firewall ASAP. I shall probably send similar instructions to another department with a misconfigured PIX, since they brushed off my previous request to fix it. Both departments are also having problems with email from us to them, about which our servers can log more details, so I hope I have enough evidence to persuade them to heed my advice.

(a previous story of packet boundary screwups)

Duplex printers

2009-08-11T21:05:41Z

I'm annoyed by how slow our office printer is at printing on both sides of the paper. It occurs to me that it ought to be possible to make it much faster (a similar speed to single-sided printing) without too much extra complexity.

At the moment it handles one sheet at a time, in the sequence feed / print / reverse / print / eject. The feed and eject phases can be overlapped but there's otherwise little opportunity for pipelining.

If the duplex mechanism includes a paper path that goes from the print mechanism's exit to its entrance while turning over the sheet, then the path through the printer can be one-way (except inside the duplexer) and the duplexer can operate at the same time as the printer. The whole thing can then handle two sheets at once. The sequence goes like this:

feed sheet 1
print page 2 on sheet 1
sheet 1 into duplexer / feed sheet 2
sheet 1 through duplexer / print page 4 on sheet 2
sheet 1 into printer / sheet 2 into duplexer
print page 1 on sheet 1 / sheet 2 through duplexer
sheet 1 exits / sheet 2 into printer
print page 3 on sheet 2
sheet 2 exits / feed sheet 3
...

I have assumed a C-shaped or S-shaped paper path, where the printer prints on the top of the sheet which turns over before dropping into the hopper, so that you end up with the stack of paper face down in the correct order. Our printer's duplex mode presumably prints pages in even/odd order (i.e. page 2 then page 1 on sheet 1, page 4 then page 3 on sheet 2, etc.) so the output stack still ends up in the right order. My duplexer's 2/4/1/3 page ordering puts greater demands on the RIPper's memory but that shouldn't be a big deal these days. (On the other hand, why does our printer appear to wait while RIPping? Shouldn't that be instant nowadays?)

So I wonder if anyone makes a printer with a duplexer that works this way. It should be fairly obvious from the duplex / duplex / exit / exit rhythm.

Job Ad - VOIP sysadmin

2009-08-11T14:57:14Z

The University of Cambridge Computing Service is currently looking for a second VOIP sysadmin. We've just finished replacing our old analogue phone switch with a Cisco-based setup serving over 16,000 lines. There are more details (including salary level and waffle) in the job ad.

Sainsbury's self-checkout fail

2009-08-10T13:41:59Z

I hate plastic carrier bags, so I take a canvas bag when I go shopping (unless I forget). The city-centre Sainsbury's in Cambridge recently replaced their express tills with self-checkout tills, which use weighing scales under the bags to check that scanned things have been bagged properly.

If you put an unexpected object in the bagging area it flashes up an annoying message. This screen has a button on it saying "Using your own bag?" which I am pretty sure used to cancel the message and allow you to scan your shopping. It is now non-functional: if you try pressing it a member of staff has to come over and explain the correct procedure. If you want to use your own bag, before you put it on the bagging area you have to go into the "select an item" menu (for bananas and fresh bread etc.) and choose the "bag re-use" icon. Obviously.

More on O(log(log(n))) searching

2009-07-30T18:33:15Z

Thanks to everyone for the informative comments on my previous post.

I found a bug in my simulation code which meant that many of the tests were performed on arrays that were half-full of zeroes (oops). This distorted the stats a little, and more for larger N because I tested fewer different arrays for large N. This bug caused the increase in s.d. for large N: the corrected code has the same s.d. for all N. Another effect was to increase the maximum number of iterations required to find an entry. After the fix the maximum number of iterations goes down to log(log(N))+12 for the pure secant method, and (log(N)+14)/2 for Junio's mixed secant/bisection method. Altogether much more well-behaved.

Searching a sorted array faster than O(log(N))

2009-07-28T19:46:54Z

The usual way to find an element in a sorted array is using a binary search, which takes log(n) time, where logs are understood to be in base 2 and N is the size of the array.

Linus Torvalds made the clever observation that you can do better than log(N) if you know that the contents of the array are uniformly distributed. For instance, git's pack files store multiple objects identified by the SHA-1 hashes of their contents. Each pack file has an index containing a list of the pack's SHA-1 IDs and their objects' locations in the pack. The index is sorted by SHA-1 ID. Since SHA-1 is a cryptographic hash, we can assume its output is uniformly distributed.

Linus described his technique as "Newton-Raphson" which is a bit of a misnomer, since N-R works on smooth differentiable curves whereas what we have is a straight line with some stochastic variations. What we're actually doing is an iterated linear interpolation. If the SHA-1 IDs were perfectly evenly distributed then a single linear interpolation would land us right on the target item, but the random variation means we will be off by some amount, so we need to continue searching.

How far off will we be? It turns out (based on Monte Carlo simulation) that the expected error is about 0.31 * sqrt(N) with a standard deviation of about 0.26 * sqrt(N). This is a really promising result since it implies that each iteration reduces the search space to N^1/2 whereas an iteration of binary search reduces it to N/2. So we should expect a complete search to take O(log(log(N))) iterations.

I wrote a simulation to try this out, and it matches this prediction: in fact the number of iterations was about 1 + log(log(N)). However what is the variation around this expected result? In my tests it turned out that the maximum number of probes was log(N) though for small N it bottomed out at about 16. When testing lots of different randomly filled arrays, the standard deviation was about 1.2 for all values of N, but when I tested fewer arrays this number ramped up.

Junio Hamano's implementation of Linus's idea is included in git but disabled by default. He added a tweak that biases the linear interpolation towards the centre of the search range, so it's kind of a balance between binary search and linear interpolation search. In my simulator this tweaked version required (log(N)+3)/2 iterations on average with a standard deviation of 0.8. The maximum number of iterations was again log(N) but it bottomed out at about 12. Overall it's a bit slower but better behaved.

In git, where a large repository might contain two million objects, and where pack index lookups are not particularly performance-critical, this improved lookup code doesn't provide a noticeable advantage. Still, I think it's interesting and the idea might be useful in other situations. Note that unlike a binary search, which can just use comparisons returning greater / equal / less, the linear interpolation search needs to know the absolute values of the elements. Git's code actually uses a lexicographic variant that ignores any common prefix shared by the elements in the search range, and uses only the next two bytes for the interpolation.

To finish, here's a bit of code. In this example, 0.0 <= array[k] < 1.0, and I use k for keys and v for values of array elements. We are searching for vtarg.

	/* all bounds are exclusive */
	double vlo = -DBL_MIN, vhi = +1.0;
	int klo = -1, khi = N;
	while(klo - khi > 1) {
		int kmid = klo + (khi-klo) * (vtarg-vlo) / (vhi-vlo);
		/* ensure rounding does not put us out of bounds */
		if(guess_k <= min_k) guess_k = min_k + 1;
		if(guess_k >= max_k) guess_k = max_k - 1;
		double vmid = array[kmid];
		if(vmid == vtarg) return(kmid);
		if(vmid < vtarg) klo = kmid, vlo = vmid;
		if(vmid > vtarg) khi = kmid, vhi = vmid;
	}
	return(-1);

Addendum: there are a few corrections in a follow-up post.

Tempting Fate, and getting her unwanted attention

2009-06-09T22:19:04Z

Yesterday I dealt with a fairly routine question from a computer officer about sending email from a web server. It's always nice when someone asks for advice, even when they expect the answer won't contain any surprises, just in case it does. I gave our usual advice about the safest ways to set up web forms that send email. The bit that tempted fate was saying that web forms have caused spam problems for us "in the past" - and I had been reflecting recently that we haven't had a big web spam problem for quite a long time.

Our recent problems have all been account compromises of one kind or another. One that will be familiar to other postmasters is caused by users sending their login credentials in reply to a phishing message. We seem to have a relatively clued-up userbase, based on tales from other universities; very few reply to most phishing spams and most of those replies do not contain passwords - they are more or less skeptical and sometimes taunt the spammer. However there's still the occasional idiot, and even a particularly special one who had to be re-educated twice! If any passwords do escape and accounts get used to spam, our strict rate limits for remote email users shut the unwanted flows down pretty quick.

More recently we have had problems with compromised accounts on email systems other than our own. These seem to have been due to infected PCs on networks with a Microsoft Windows domain and Exchange server - I don't think remote access (e.g. via Outlook Web Access) was to blame in these cases. We have been reasonably successful applying rate limits to mail servers that relay outgoing mail through us. We set the limit for each server to be about 10% above their normal peak usage, so it should not trigger unless something suspicious is happening, and we have a monitoring script to warn us if they stray into (or past!) the 10% headroom. The first time this mechanism caught a spam run it stopped an 80,000 message flood about 2% of the way through.

So Fate observed my complacency about web form compromises and handed us a big plate full of crap today. Shortly before I got into work, one of our departments started spewing email and fairly swiftly whacked into their rate limits. I was foolishly trusting of their competence and aware of their occasional need to send large mailshots, so I tried to let the mail through - but it rapidly became clear that tens of thousands of messages was way beyond normal and their arrival rate of hundreds per second was going to cause us problems even if it was legit. Silly me for not looking at a sample message. I got in touch with the department's techies and it rapidly became apparent there was a compromise, and we went into lock-down and clean-up mode. Between us we had to delete about 100,000 queued messages, with the added joy that we had to avoid deleting the couple of dozen legitimate messages that had the same sender address as the spam. Sadly my error meant that 20,000 messages escaped (plus 6000 with invalid recipient addresses) whereas it should have been only two or three thousand :-(

As I said yesterday, there are two ways to ensure that a web form sends email safely. The first is to fix the recipient address, for example in "send feedback to the webmaster" forms, or mail to registered users of your site. The second is to fix the contents of the message, for example in signup or purchase confirmation messages. Either of these is enough to make the form useless to spammers.

Today's excitement was caused by a "mail a copy of this story" form, which should have been implemented safely in the second style. However the form allowed remote users to add their own comments above the story, and gave them plenty of space to do so - enough for a typically lengthy 419 scam.

It's fair to say this has been a learning experience, more painful for some than for others :-) I should remember that technical cockups are one thing, but even if you are technically competent political requirements can still screw the pooch.

CRSIDs and email addresses

2009-06-02T13:11:56Z

I've recently written a three page memo about the advantages of our username scheme compared to "friendly name" email addresses in large domains. I have put a copy of the PDF on the web which you can read if you like.

Use your bonce

2009-05-15T08:55:18Z

Vesta includes a purely functional programming language for specifying build rules. It has an interesting execution model which avoids unnecessary rebuilds. Unlike make, it automatically works out dependencies in a way that is independent of your programming language or tools - no manually maintained dependencies or parsing source for #include etc. Also unlike make, it doesn't use timestamps to decide if dependencies are still valid, but instead uses a hash of their contents; it can do this efficiently because of its underlying version control repository. Vesta assumes that build tools are essentially purely functional, i.e. that their output files depend only on their input files, and that any differences (e.g. embedded timestamps) don't affect the functioning of the output.

I've been wondering if Vesta's various parts can be unpicked. It occurred to me this morning that its build-once functionality could make a quite nice stand-alone tool. So here's an outline of a program called bonce that I don't have time to write.

bonce is an adverbial command, i.e. you use it like bonce gcc -c foo.c. It checks if the command has already been run, and if so it gets the results from its build results cache. It uses Vesta's dependency cache logic to decide if a command has been run. In the terminology of the paper, the primary key for the cache is a hash of the command line, and the secondary keys are all the command's dependencies as recorded in the cache. If there is a cache miss, the command is run in dependency-recording mode. (Vesta does this using its magic NFS server, which is the main interface to its repository.) This can be done using an LD_PRELOAD hack that intercepts system calls, e.g. open(O_RDONLY) is a dependency and open(O_WRONLY) is probably an output file, and exec() is modified to invoke bonce recursively. When the command completes, its dependencies and outputs are recorded in the cache.

bonce is likely to need some heuristic cleverness. For example, Vesta has some logic that simplifies the dependencies of higher-level build functions so that the dependency checking work for a top-level build invocation scales less than linearly with the size of the project. It could also be useful to look into git repositories to get SHA-1 hashes and avoid computing them.

It should then be reasonable to write very naive build scripts or makefiles, with simplified over-broad dependencies that would normally cause excessive rebuilds - e.g. every object file in a module depends on every source file - which bonce can reduce to the exact dependencies and thereby eliminate redundant work. No need for a special build language and no need to rewrite build scripts.

Define SCM

2009-05-14T11:32:33Z

Here's an abbreviation to avoid, because its various meanings overlap so heavily.

Source Code Manager: synonym for version control system, e.g. as used by the Linux crowd in the early days of git.
Software Configuration Manager: usually incorporates version control, build system, archival of build products, and CASE methodology baggage, e.g. ClearCASE and Vesta.
System Configuration Manager: for system administrators rather than developers, e.g. cfengine and Puppet.

Never delete anything

2009-05-13T22:13:25Z

How long will it be before it becomes normal to archive everything? It's already normal in some situations, and I think that's increasing. It's been the norm in software development for a long time. There's an increase in append-mostly storage systems (i.e. append-only with garbage collection) which become never-delete systems if you replace the GC with an archiver. Maybe the last hold-outs for proper deletion will be high data volume servers...

Anyway, I feel like listing some interesting append-only and append-mostly systems. A tangent that I'm not going to follow is the rise of functional programming and immutability outside the field of storage. Many of these systems rely on cryptographic hashes to identify stuff they have already stored and avoid storing it again, making append-only much more practical.

All version control systems, and software configuration management systems even more so. The former archive source code whereas the latter archive build tools and build products as well. DEC's Vesta SCM is particularly interesting, being based on a purely functional build language designed to maximize memoization - i.e. minimize unnecessary rebuilds. It's sort of ccache on steroids since it caches the results of entire module builds, not just individual source file compiles.
Nix is a purely functional package manager. Unlike most packaging systems like dpkg or rpm, Nix packages do not conflict with each other: you upgrade by installing new packages alongside your existing ones, then you stop running the old ones and start running the new ones.
Archival / backup systems, like Venti which is Plan 9's append-only filesystem. Apple's Time Machine isn't nearly as clever.
Most filesystems don't use hash-based uniquification. Append-mostly filesystems often provide cool undelete features like snapshots, e.g. NetApp's WAFL or Sun's ZFS. Early filesystems of this kind, e.g. BSD LFS tried to avoid wasting space, so didn't make old data available as snapshots, and sacrificed performance to eager garbage collection. More recently, DragonFly BSD's Hammer filesystem doesn't even have an in-kernel garbage collector, and running it is entirely optional.
Email archives: gmail's ever-increasing quotas, cyrus delayed expunge.

Some thoughts about git

2009-04-23T20:01:02Z

I was originally planning to witter about distributed version control vs. centralized version control, especially the oft-neglected problem of breaking up a large cvs / svn / p4 repository. This was partly triggered by Linus's talk about git at Google in which he didn't really address a couple of questions about how to migrate a corporate source repository to distributed version control. But in the end I don't think I have any point other than the fairly well-known one that distributed version control systems work best when your systems are split into reasonably modestly-sized and self-contained modules, one per repository. Most systems are modular, even if all the modules are in one huge central repository, but the build and system integration parts can often get tightly coupled to the repository layout making it much harder to decentralize.

Instead I'm going to wave my hands a bit about the ways in which git has unusual approaches to distributed version control, and how bzr in particular seems to take diametrically opposing attitudes. I'm not saying one is objectively better than the other, because most of these issues are fairly philosophical and for practical purposes they are dominated by things like quality of implementation and documentation and support.

Bottom-up

Git's design is very bottom-up. Linus started by designing a repository structure that he thought would support his goals of performance, semantics, and features, and worked upwards from there. The upper levels, especially the user interface, were thought to be of secondary importance and something that could be worked on and improved further down the line. As a result it has a reputation for being very unfriendly to use, but that problem is pretty much gone now.

Other VCSs take a similar approach, for example hg is based on its revlog data structure, and darcs has its patch algebra. However bzr seems to be designed from the top down, starting with a user interface and a set of supported workflows, and viewing its repository format and performance characteristics as of secondary importance and something that can be improved further down the line. As a result it has a reputation for being very slow.

Amortization

Most VCSs have a fairly intricate repository format, and every operation that writes to the repository eagerly keeps it in the canonical efficient form. Git is unusual because its write operations add data to the repository in an unpacked form which makes writing cheaper but makes reading from the repository gradually less and less efficient - until you repack the repo in a separate heavy-weight operation to make reads faster again. (Git will do this automatically for you every so often.) The advantage of this is that the packed repository format isn't constrained by any need for incremental updates, so it can optimise for read performance at the expense of greater pack write complexity because this won't slow down common write operations. Bzr being the opposite of git seems to do a lot more up-front work when writing to its repository than other VCSs, e.g. to make annotation faster.

Thus git has two parallel repository formats, loose and packed. Other VCSs may have multiple repository formats, but only one at a time, and new formats are introduced to satisfy feature or performance requirements. Repository format changes are a pain and happily git's stabilized very early on - unlike bzr's.

Laziness

As well as being slack about how it writes to its repository, git is also slack about what it writes. There has been an inclination in recent VCSs towards richer kinds of changeset, with support for file copies and renames or even things like token renames in darcs. The bzr developers think this is vital. Git, on the other hand, doesn't bother storing that kind of information at all, and instead lazily calculates it when necessary. There are some good reasons for this, in particular that developers will often not bother to be explicit about rich change information, or the information might be lost when transmitting a patch, or the change might have come from a different VCS that doesn't encode the information. This implies that even VCSs that can represent renames still need to be able to infer them in some situations.

Git's data structure helps to make this efficient: it identifies files and directories by a hash of their contents, so if the hash is the same it doesn't need to look any closer to find differences because there aren't any - and this implies a copy or rename. This means that you should not rename or copy a file and modify it in the same commit, because that makes git's rename inference harder. Similarly if you rename a directory, don't modify any of its contents (including renames and permissions changes) in the same commit.

Mercurial also uses hashes to identify things, but they aren't pure content hashes: they include historical information, so they can't be used to identify files with the same contents but different histories. Thus efficiency forces hg to represent copies explicitly.

Any more?

I should say that I know very little about bzr, and nothing about tla, mtn, or bk, so if any of the above is off the mark or over-states git's weirdness, then please correct me in a comment!

LISTSERV crapness

2009-04-08T13:18:01Z

LISTSERV uses a null return path (RFC821 MAIL FROM:<>) on its administrative mail, and some mail hosts reject this. I discard message with a null return path that do not match a few simple heuristics, so I lose things like subscription confirmations from services like JISCfail. This makes me cross.

L-Soft claim that LISTSERV is following the specifications, and they cite a couple of paragraphs from RFC 821 (published in 1982) and RFC 1123 (1989). However they fail to cite text from RFC 2821 (2001) which explicitly forbids what they are doing.

There are several types of notification messages which are required by existing and proposed standards to be sent with a null reverse path, namely non-delivery notifications, other kinds of Delivery Status Notifications (DSNs), and also Message Disposition Notifications (MDNs). [...]

All other types of messages (i.e., any message which is not required by a standards-track RFC to have a null reverse-path) SHOULD be sent with with a valid, non-null reverse-path.

The only other permitted use of null return paths that I know of is vacation notifications, described in RFC 3834 (published in 2004).

L-Soft needs to get a grip, read some RFCS, and fix their software.

Configuration management

2009-04-02T16:15:40Z

We've been discussing configuration management in the office this week. None of us are happy with the way we're doing things...

On Hermes, we do most configuration management using rdist. On our management server we have a number of file trees containing files that differ from the default - a smattering of stuff in /etc, the contents of /home and /opt, and a few other bits scattered around. These trees are cloned and hacked for each flavour of machine (ppswitch, cyrus, lists, webmail, etc.) and each version of the underlying OS. These trees are mostly kept under revision control.

This setup has the advantage of simplicity, and it's easy to push out small changes. One key feature is rdist's testing mode, which makes it easy for us to ensure that the servers are in sync with the configuration tree without changing anything. We often run a test across a cluster of 10 or 16 machines in parallel. It's easy to selectively push a change to an idle machine for testing before rolling it out to the rest of the cluster. For more tricky changes I do a phased rollout so I can check for unwanted changes in behaviour without breaking the entire service at once.

Of course it has some serious disadvantages. We have to be root on the management host to be able to push changes out to the servers. We can't easily keep file ownership and permissions under revision control. This mechanism misses significant parts of the configuration, such as installed base OS packages and which rc scripts are enabled. There's also a lot of scope for improving our initial OS install scripts to reduce the amount that they get out of sync with the rdist configuration.

So we'd like something better. Our colleagues have different system management setups with different problems, and they are also looking for something better.

A couple of my colleagues have looked at Puppet but weren't happy with it. I dislike its basic design. Managed servers pull their configurations from the master, which means you must never screw up a change on the master, and it's harder to test changes - you have to explicitly set up test server profiles. Yes, you can run Puppet in push mode but it's often a bad idea to work against a program's basic architecture. Puppet also has its own security mechanisms, whereas I'd prefer to avoid multiplying channels of trust. Finally, I really hate writing configuration files for programs that write configuration files. It's a waste of brain cells to understand this superfluous abstraction layer: I just want to write the underlying configuration file directly.

Aside: this is why I really really hate autoyast, which uses an XML configuration file to control a program that writes configuration files that control rc scripts that manipulate the underlying configuration files for the programs you actually care about. It takes hours to work out how to make it produce the correct results.

I spent some time working on a program called pstx which was to be a suped-up replacement for rdist. It was going to be an sftp client (so it would require nothing special on the target servers) that had a simple configuration file to specify which directory trees to copy where, including ownerships and permissions (bringing them under revision control and removing the need to be root on the master), and possibly with added bells and whistles like remote diff, and a reverse (pull) mode. I also intended to make it easer to combine collections of files and thereby share common files. Sadly it's only about a third written and likely won't get much further.

One of the conversations this week was about how to reduce the chance of mistakes when rolling out changes, in which we discussed the use of revision control to help with testing and rollback. At the moment Hermes mostly uses CVS and other unixy bits of the CS share a Subversion repository, so the conversation very much assumed a central repository model - which fits in well with a master configuration server. Recently I have also been investigating git more seriously, so I thought it might be part of a solution.

The key things that git provides include a flexible and efficient network protocol for moving files around (flexible in that it can use ssh and http as well as git's native protocol, and efficient in that it's incremental and compressed), it can tell us what differs between a directory tree and the state of the repository, changes can be pushed and pulled, etc. The distributed version control functionality is way better at all this than pstx was ever going to be.

The big missing part is the ability to track ownerships and permissions: git only supports normal development checkouts which should be done using the developer's uid and umask. There are also some low-level problems with the way git performs checkouts: it removes the destination file before writing the updated version in its place. I would like a special deployment command which checks all the changed files out under temporary names, fixes their ownerships and permissions, then renames them into place. The first stage almost exists in the form of git-checkout-index --temp d, though it does weird things with symlinks. It doesn't look like much work to add the other stages.

This could provide some really nice workflows. A configuration change is created on a small topic branch, then pushed to the configuration repositories on all the machines - which changes none of the live configuration. You can then switch a test machine to the new branch with a single git deploy branch, or roll back by re-deploying the master branch. You can find out the current configuration of a machine in summary using git status or git diff to make sure the summary isn't a lie.

One interesting possibility might be to tame the clone-and-hack problem by using a shared repository for all the different classes of machines. This should make it easier to see the differences between each kind of machine and spot spurious ones.

One thing that is very manual in our current setup is hupping daemons to make them load an updated configuration. It might be feasible to use a post-deploy hook to do this, provided the hook script is given enough information that it can avoid unnecessary restarts.

Now I just need to find some time to turn the idea into code...

Ada Lovelace Day

2009-03-25T12:26:31Z

This is a bit late, but I've been inspired to post after reading about the women other people admire. Here are a few of mine.

Margo Seltzer: developer of the BSD log-structured filesystem and Berkeley DB. An open source entrepreneur - founder and CTO of Sleepycat Software - who also has a successful academic career.
Dina Katabi invented XCP, the eXplicit Congestion control Protocol, which is a brilliant way for routers to make each individual flow adjust to the level of congestion without the need for any per-flow state. I think it was one of the first papers I read about advanced transport protocol research, and I have continued to follow the subject since then.
Lisa Dusseault is a WebDAV expert and IETFer, currently one of the Applications Area Directors and therefore a member of the IESG. She was one of the funner people I met at the Paris IETF meeting a few years ago. I like the new ideas she's brought to the IESG, such as monthly updates on her work as Apps AD.
And of course rmc28, who got the University Card working properly and now slaves away on the student information system.

John Taylor talks about the Corpus clock

2009-03-15T19:57:40Z

I posted a couple of articles (one, two) about the Corpus clock soon after it was unveiled, and I optained a copy of the Chronophage book soon after it became available. The book is a bit lacking in technical detail, so I was pleased to find out that John Taylor would be talking about it for the Cambridge Science Festival and I attended his talk yesterday. The lecture theatre was packed.

The talk was in three parts: a bit of autobiography, then a quick overview of the clock and its development, then questions. The talk revealed some interesting facts which do not appear in the book.

Taylor's story of how he came to make a fortune in kettle thermostats is relevant to the clock not just because that's how he made the money: his horological hero, John Harrison, invented the grasshopper escapement that inspired the Chronophage, and also invented the bimetal on which Taylor's thermostats rely. He explained his snap-action bimetallic disc, which has a C cut out to form a tongue in the middle; this tongue moves further than the centre of a disc without the cut-out does, which loosens the tolerances required to manufacture a thermostat. The calligraphic flourish engraved below his name on the clock's pendulum bob forms the shape of his bimetallic disc. (Sorry, I can't find a picture on line though it's visible in the book.)

ETA: I have found a large picture of the pendulum bob in which you can see the shape of Taylor's bimetallic disc under the date. It's overdrawn by some of the flourish so it's a bit obscure unless you know what you are looking for.

There were a couple of hints at the clock's workings. It is usual for clocks with chiming mechanisms to have a second set of drive springs or weights for the chimes. The Corpus clock has only one spring, so its hourly death-rattle is driven in an unusual way. This is why the clock ticks back and forth while rattling the chain in the coffin. There are two main mechanisms inside the Chronophage itself. It contains a mechanical pseudo-random mechanism to control the blinking and biting. The sting, however, is regular: it slowly rises then jabs down on each quarter hour.

He also mentioned that the clock "has fun" four times a year: on 24th March, which is the date of Harrison's birth and also his death; on 25th November, Taylor's birthday; on new year's eve and new year's day; and on Corpus Christi day, the Church's festival that occurs on the Thurdsday after Trinity Sunday, the 8th Sunday after Easter. The clock is away for maintenance this week, so I hope it'll be back in time to muck around on Tuesday week. The fun seems to involve even more erratic ticking than usual, with more colourful lights.

The clock's spring has to be particularly strong in order to overcome the momentum of the large escape wheel that goes around the outside of the clock. When prototyping the clock this caused a serious problem: the force of the spring is transmitted through the escapement to the pendulum, making it swing higher and higher and eventually breaking the clock. Taylor overcame this problem by adding a regulator, which also serves two other functions: it produces the clock's erratic behaviour that plays tricks with observers, and it listens to the MSF time signal to synchronize the clock every five minutes. (Taylor confirmed to me after the talk that these are all functions of the same mechanism. He also said that the hollow pendulum bob is not in fact "massive and weighty" as the book says, which answered my question about conservation of momentum.) I'm not sure how this is consistent with his assertions that the clock is purely mechanical - would it work if the computer regulator broke? News reports about the clock's teething problems suggest that it's pretty vital. I wonder if the maintenance periods this week and back in January are for software patches rather than mechanical adjustment?

I asked Taylor if he would publish more technical details about the clock, but he thinks that the mystery makes it fun. I disagree :-/