Log in

No account? Create an account


Mock the stupid

« previous entry | next entry »
29th Jun 2007 | 13:33

This is one of the worst examples of willful incompetence I have seen recently, so I thought it deserved a bit of point and laugh.

In the last couple of weeks we received two complaints that email from ABN AMRO (a large Dutch bank) to Cambridge was unreliable. According to the bounce messages that were forwarded to us, their Postfix outbound servers were complaining "conversation with mx.cam.ac.uk timed out while sending MAIL FROM". Strange: we don't do anything particularly time consuming at that point in the SMTP conversation. Perhaps it's an MTU problem?

I try turning on some extra logging, and Exim says "SMTP connection from triton10.abnamro.nl lost while reading message data (header)". This is inconsistent with an MTU problem, since the envelope commands (MAIL FROM, RCPT TO, DATA) are larger than the replies, and Exim has received the commands and sent the replies OK. It's also inconsistent with Postfix's error message, since Postfix obviously sent the MAIL FROM without a problem. It turns out there's a minor bug in Postfix that causes it to use an incorrect error message when there's a timeout waiting for a reply from the server.

OK, so ABN AMRO's Postfix is timing out while waiting for our envelope replies, but our replies are sent reasonably promptly. I resort to running a very selective tcpdump on mx.cam.ac.uk to see if that provides a clue. There is indeed no sign of an MTU problem: what is actually happening is their end is closing the connection only 15 seconds after it has sent the envelope commands. Exim doesn't check for a closed connection until it wants to read more data, which explains its error message.

So it looks like their end has an absurdly small 15 second timeout, which triggers if we take too long to emit the envelope replies - which can happen if recipient address verification takes a while. The standard requires at least a five minute timeout, and we're careful to stay within that limit. They are just asking for trouble if they reduce their timeout to such a short period, and they would have to deliberately break Postfix which ships with correct defaults.

So I tried getting in touch with their postmasters. I first tried postmaster@nl.abnamro.com since one of the problem reports came from an @nl.abnamro.com address. I received two bounces from their Lotus Notes system. I then tried postmaster@abnamro.com, and after a day without a reply I tried postmaster@abnamro.nl. I still haven't received a reply.

I also asked the people who reported the problem to chase it up with their IT staff. Eventually I got the reply that they are aware of the problem but there is no "business justification" to fix their broken systems. I bet ABN AMRO's management would do something if their post room was chucking letters in the bin and ignoring support queries, so why do they tolerate such crapness for email?

| Leave a comment |

Comments {2}


from: bslsimes
date: 29th Jun 2007 13:25 (UTC)

Having worked there, I can say that this is entirely consistent with my observations at the time. i.e. they're useless.

Reply | Thread


from: hairyears
date: 29th Jun 2007 17:19 (UTC)

There are reasons why ABN AMRO have fallen behind their competitors and become a takeover target.

I bet ABN AMRO's management would do something if their post room was chucking letters in the bin

Personally, I don't think that they would: senior managers would have no contact with the consequences, and no incentive or interest in finding out about the problem.

Is that a little harsh?

Go into the men's washroom on the postroom-and-services floor in any AMRO building and walk headfirst into the oldest management consultancy cliché of all: the hand-dryer's broken, and nobody can remember a time when it ever worked. Nobody who sees and uses it can get it fixed, and nobody with the authority to make it happen will ever go there or would have the slightest interest in sorting out the problem. Yes, it is harsh, but you can go back in a week, a month, or a year and it'll still be out-of-order. At which point, you are forced to conclude that the problem is the hand-dryer's owners and not the dryer.

Mention this to a senior manager and they'll be genuinely baffled that you think such tiny things are relevant - lightbulbs blow all the time, do you expect me to know the location of each and every one? - as if you raised the matter out of some peculiar obsession with trivia, and have become fixated with the one and only busted hand-dryer, blown bulb or misconfigured mail server that had ever happened in ABN Amro.

The truth, of course, is that it's not the only one: the entire organisation is a litany of serially-repeated errors and missed business opportunities, failed customer-service and complaints narratives, and defective transactions that never get sorted out. People who talk 'hand dryers' and screwdrivers and server config files never talk to people who talk 'mission' and 'market share', and neither group sees the other as remotely relevant to whatever they do for the bank: a fundamental gulf in communications exists. But if nobody in contact with the issues at the sharp end can get things moving, then people will stop doing business with you. And if there are senior managers with the awareness and the ability to turn it all around, and make it all work, they don't seem to be working at ABN Amro; I suspect that they all realised, pretty soon after joining the company, that there are more rewarding careers elsewhere.

Managers at Barclays, and at the rival bidding consortium let by RBOS, know exactly what they are getting into; and however hostile the general public may be to 'casino capitalism' and the all-too-frequent failures of mergers and acquisitions, everyone who deals with ABN Amro on a daily basis will welcome the obliteration of a dismal and dysfunctional organisational culture.

Reply | Thread