Log in

No account? Create an account


Tempting Fate, and getting her unwanted attention

« previous entry | next entry »
9th Jun 2009 | 23:16

Yesterday I dealt with a fairly routine question from a computer officer about sending email from a web server. It's always nice when someone asks for advice, even when they expect the answer won't contain any surprises, just in case it does. I gave our usual advice about the safest ways to set up web forms that send email. The bit that tempted fate was saying that web forms have caused spam problems for us "in the past" - and I had been reflecting recently that we haven't had a big web spam problem for quite a long time.

Our recent problems have all been account compromises of one kind or another. One that will be familiar to other postmasters is caused by users sending their login credentials in reply to a phishing message. We seem to have a relatively clued-up userbase, based on tales from other universities; very few reply to most phishing spams and most of those replies do not contain passwords - they are more or less skeptical and sometimes taunt the spammer. However there's still the occasional idiot, and even a particularly special one who had to be re-educated twice! If any passwords do escape and accounts get used to spam, our strict rate limits for remote email users shut the unwanted flows down pretty quick.

More recently we have had problems with compromised accounts on email systems other than our own. These seem to have been due to infected PCs on networks with a Microsoft Windows domain and Exchange server - I don't think remote access (e.g. via Outlook Web Access) was to blame in these cases. We have been reasonably successful applying rate limits to mail servers that relay outgoing mail through us. We set the limit for each server to be about 10% above their normal peak usage, so it should not trigger unless something suspicious is happening, and we have a monitoring script to warn us if they stray into (or past!) the 10% headroom. The first time this mechanism caught a spam run it stopped an 80,000 message flood about 2% of the way through.

So Fate observed my complacency about web form compromises and handed us a big plate full of crap today. Shortly before I got into work, one of our departments started spewing email and fairly swiftly whacked into their rate limits. I was foolishly trusting of their competence and aware of their occasional need to send large mailshots, so I tried to let the mail through - but it rapidly became clear that tens of thousands of messages was way beyond normal and their arrival rate of hundreds per second was going to cause us problems even if it was legit. Silly me for not looking at a sample message. I got in touch with the department's techies and it rapidly became apparent there was a compromise, and we went into lock-down and clean-up mode. Between us we had to delete about 100,000 queued messages, with the added joy that we had to avoid deleting the couple of dozen legitimate messages that had the same sender address as the spam. Sadly my error meant that 20,000 messages escaped (plus 6000 with invalid recipient addresses) whereas it should have been only two or three thousand :-(

As I said yesterday, there are two ways to ensure that a web form sends email safely. The first is to fix the recipient address, for example in "send feedback to the webmaster" forms, or mail to registered users of your site. The second is to fix the contents of the message, for example in signup or purchase confirmation messages. Either of these is enough to make the form useless to spammers.

Today's excitement was caused by a "mail a copy of this story" form, which should have been implemented safely in the second style. However the form allowed remote users to add their own comments above the story, and gave them plenty of space to do so - enough for a typically lengthy 419 scam.

It's fair to say this has been a learning experience, more painful for some than for others :-) I should remember that technical cockups are one thing, but even if you are technically competent political requirements can still screw the pooch.

| Leave a comment | Share

Comments {3}

from: Dave Holland [org.uk]
date: 10th Jun 2009 16:06 (UTC)

How do you notice machines exceeding their rate limit? A script tail-ing the Exim log, or something more sophisticated?

Reply | Thread

Tony Finch

from: fanf
date: 10th Jun 2009 16:34 (UTC)

We have a script which monitors various things and is run from cron every 10 minutes. It avoids grepping logs because we produce about 300MB on each machine each day. (Note that at the moment DNSBL rejections have gone down by a factor of 3 because of the 3FN takedown, so the logs are smaller by a factor of 1.5 - 2.5 than they were.)

The check script cats the panic log and deletes it. This should be a no-op :-)

It uses exim -bp to look for frozen messages, which we use to quarantine floods of email from MUAs and any other machine that we do not know to retry correctly. It tells us if the number of frozen messages changed and who sent them.

The ratelimit part is complicated. We have an auxiliary script for managing the ratelimit database. It's a wrapper around exim_fixdb which allows us to list a user's or host's recorded rates, and we can delete the entries when we have cleaned up a mess so that normal usage can resume.

The checker loops through our configuration table which contains hosts with special limits. (Their mail is deferred instead of frozen.) It calculates the 90% warning thresholds, pulls the host's record out of the database, and prints a line if the entry in the database is greater.

One additional complication is that we actually have four limits, hourly and daily limits for messages and recipients, which lets us have lower backstop limits. (The mail/rcpt distinction is much less useful than the hourly/daily distinction.) So the ratelimit commands have to loop over the four limits.

See http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/conf/exim/sbin/

Reply | Parent | Thread

from: Dave Holland [org.uk]
date: 10th Jun 2009 16:38 (UTC)

Thanks. Very useful.

Reply | Parent | Thread