?

Log in

No account? Create an account

fanf

REST FAIL

« previous entry | next entry »
5th Nov 2008 | 19:07

This afternoon my colleague Andy gave a talk about Microsoft Exchange 2007, which I attended since I need to know what the competition has to offer and I have to provide support for its SMTP interfaces. One thing he mentioned which perked up my interest was "autodiscovery" for Outlook 2007. The unnecessary difficulty of configuring email software has irritated me for a long time, so after the talk I immediately went to find out more. It turns out to be a mixture of good stuff and utter FAIL.

The documentation describes three ways that Outlook 2007 can configure user accounts automatically - server names, security requirements, etc. If you are logged into a Windows Domain then it first tries querying the Active Directory. If that succeeds then it can find everything out by itself. Nice. If not, it falls back to an open protocol, which any standards-based mail server can implement, that configures the server settings automatically given an email address. More about this below.

If server-supported autodiscovery doesn't work, Outlook tries to guess the settings by attempting various combinations of host name, port number, TLS or not, POP or IMAP, etc. and stopping when it finds something that works. I think this is a great idea, so it's a damn shame that it prefers to use the lame POP not the studly IMAP if both are available. By itself that is enough to make it worth our while to implement the autodiscover protocol; but because our primary mail domain is cam.ac.uk but our service name is hermes.cam.ac.uk, Outlook's guesses will fail. This has the advantage of preventing users walking blindly into bad configurations, but the disadvantage that they're less likely to be able to configure it at all.

And so to the autodiscovery protocol itself. How can something so simple be fucked up in so many ways? It looks to me like it was bodged together by a clueless intern over two or three summer breaks, each time getting worse as the requirements evolved. The result is a case study in what RESTful protocol design is not.

The essence is simple: the client sends a request containing the user's email address, and the server sends back an XML document containing the configuration settings. The request is sent using HTTP-over-SSL to a URL derived from the domain part of the user's email address. (In fact it tries https://domain/autodiscover/autodiscover.xml first, then tries https://autodiscover.domain/autodiscover/autodiscover.xml.) So far, so good.

The first mistake is that the request is a POST with the email address contained in another XML document in the request body. It would have been more RESTful to use a GET with the email address encoded in the URL. This mistake turns into FAIL when you realise that most email services have the same settings for all users, in which case autodiscover.xml can be a static file, but many web servers do not allow POST requests to static files. If they had done it right, using a GET request with the email address in the query string would have Just Worked for both CGI scripts and static files.

It gets worse when they bodge around this mistake. A POST to a file commonly results in a "405 Method Not Allowed" error. Microsoft specify that if this is the case for your web server, you should configure it to send the autodiscover.xml file in the error reply, as if the request was successful. A foul perversion of the protocol.

It gets worse when they add support for virtual domains. The requirement seems to be that the service provider wants to host the autodiscover.xml script centrally, and doesn't want to fork out for an SSL certificate for every virtual domain. So if both of the https URLs fail, Outlook tries a GET request to the autodiscover.domain URL, which can redirect to the central autodiscover.xml. However that's not secure enough so there must also be an _autodiscover._tcp.domain SRV record in the DNS pointing to the same central web server. However that's still not secure enough so Outlook also admonishes the user to click OK in a popup dialogue box.

It gets worse when you look at the autodiscover.xml "schema". It isn't a schema, it's a commented skeleton fill-in-the-blanks autodiscover.xml file. That's just everyday lameness; the real FAIL is to be found in the various elements described therein. The redirect I mentioned above isn't an HTTP redirect, it's an autodiscover.xml document containing various redirection settings. There are also settings for controlling the expiry time of the document, because of course configuring these in a .htaccess file is too hard.

Ugh. I think that just about exhausts the rantage this stuff has triggered. I am most of the way to implementing it; I just need to get the necessary SSL certificates. It turns out to be easier than I expected because newish Apache does not get upset by POST requests aimed at static files: it just throws away the request body and returns the file with a "200 OK". No need to fart around with ErrorDocument 405 autodiscover.xml. With any luck it'll make Outlook trivial to get to work with Hermes.

| Leave a comment | Share

Comments {11}

(Deleted comment)

Ben Charlton

from: benc
date: 5th Nov 2008 21:51 (UTC)

It's an issue with 2007 parsing entire messages: http://forums.microsoft.com/TechNet/ShowPost.aspx?PostID=3942132&SiteID=17

Reply | Parent | Thread

(Deleted comment)

heliumbreath

from: heliumbreath
date: 6th Nov 2008 00:14 (UTC)

Early in your posting (before the XML mess) I was going to suggest creating a DNS entry for a server name that it'll guess and pointing that at a server interface address that supports IMAP but not POP, so that it'd stumble into the right settings.

Reply | Thread

Tony Finch

from: fanf
date: 6th Nov 2008 09:03 (UTC)

That might work, depending on the exact order in which it tries the different possibilities, except that in our setup Outlook can't guess service names like imap.hermes.cam.ac.uk from an email address like fanf2@cam.ac.uk. This decoupled naming is because cam.ac.uk is a virtual mail domain that backs on to multiple servers, though the vast majority of users are on Hermes. Configuration guessing should work if users type in their non-virtual @hermes address instead, but that is generally not preferred. Server support for autodiscovery for both @cam and @hermes is no harder than support for just one, so there's no need to try subverting the guessing algorithm into doing the right thing.

Reply | Parent | Thread

heliumbreath

from: heliumbreath
date: 6th Nov 2008 13:51 (UTC)

Evidently it's going to guess imap.cam.ac.uk for your address, so that would have been the DNS entry needed to catch it that way. If they need to go to a particular backend server rather than the pool in general, though, then that's not a solution that'd work at your site short of having it point at a front end like perdition.

Reply | Parent | Thread

Tony Finch

from: fanf
date: 6th Nov 2008 13:57 (UTC)

Yes, we are using a frontend like perdition, but only for Hermes. We won't create an imap.cam.ac.uk DNS entry for policy/architectural/aesthetic reasons. For example, the other services behind @cam are way too diverse for integration using perdition.

Reply | Parent | Thread

Results?

from: anonymous
date: 31st Dec 2008 06:19 (UTC)

I'm trying to setup autodiscovery to work with our systems here, mainly for iPhone/Treo/any ActiveSync system integration. It would be great to get our XP systems with Outlook installed (and using Samba as a domain controller) to autoconfigure.

My understanding was that Outlook / ActiveSync clients used the DNS SRV entry first, before trying the http/s services. You wouldn't happen to have gotten it to work and have a working autodiscover.xml file? I can't really find any thing that seem to work.

Reply | Thread

Tony Finch

Re: Results?

from: fanf
date: 1st Jan 2009 14:53 (UTC)

I'm afraid I don't know anything about ActiveSync, and I'm not using SRV records in our autodiscover setup because they provide no benefits. It would be cool if, as you suggest, ActiveSync allows other devices to benefit from autodiscover too.

Reply | Parent | Thread

Ian Eiloart

More breakage:

from: IanEiloart
date: 3rd Dec 2010 11:27 (UTC)

I did manage to create a PHP script to return an autodiscover.xml file for Outlook, such that it can autodiscover secure settings for our IMAP server. It improves performance, in comparison with GuessSmart, and gives more secure results, too.

POST details can be extracted like this:

if (isset($_POST)){
$body = @file_get_contents('php://input');

$xmlDoc = simplexml_load_string($body);
$email = $xmlDoc->Request[0]->EMailAddress[0];
//echo($email);
}

When testing other clients to see whether they'd work, I discovered that the address used for the lookup seems to vary, at least with respect to case:

Outlook 2011 for Mac, and 2007 for Windows both look up:
https://domain/autodiscover/autodiscover.xml

There's a test service at https://www.testexchangeconnectivity.com/ Don't use it with a real account. It looks for https://domain/AutoDiscover/AutoDiscover.xml

And the iPhone, and iPad with iOS4.2.1 look up https://domain/Autodiscover/Autodiscover.xml

So, that's all very nice, then!

Reply | Thread

Ian Eiloart

One more thing

from: IanEiloart
date: 3rd Dec 2010 11:54 (UTC)

Oh, and another thing. Outlook 2007 for Windows understands a reply that says to use our IMAP server. It'll be useful when we've deployed Exchange, but have a mix of Cyrus IMAP users and Exchange users. An LDAP lookup can tell me which server to point the requestor at.

Unfortunately, this won't work on Outlook for the Mac, which doesn't seem to understand the response. In fact, you have to tell Outlook for the Mac whether you're setting up an Exchange account or not. If you say YES, then the IMAP settings returned look like an error. If you say NO, then the lookup isn't used. Neither are the GuessSmart settings that Outlook for Windows users. Instead, the file '/Applications/Microsoft Office 2011/Microsoft Outlook.app/Contents/Resources/autoConfigDB.xml' is consulted. It's fairly easy to get this right, but again you have to explicitly tell it that SSL is required! Unfortunately the path is in the application, so you can't configure it per-user.

Apple used to do something similar with a MailAccounts.plist file, but appear to have only me.com entries nowadays. However, you can still roll your own MailAccounts.plist file, which Mail will consult if you put it in the right place.

Reply | Thread