?

Log in

No account? Create an account

fanf

In which IPv6 day turns out to be unexpectedly exciting

« previous entry | next entry »
8th Jun 2011 | 17:21

As part of IPv6 Day we have enabled IPv6 on the important parts of our mail service: mx.cam (incoming mail), ppsw.cam (outgoing), and smtp/imap/pop/webmail.hermes. Most of the University is IPv4-only, apart from the Computing Service's staff network, the Institute of Astronomy, the CUSU/SRCF network, and a smattering elsewhere. Today has been mostly unexciting for us, which is cheerful. Mostly.

This afternoon I got an alarming email and phone call from a departmental sysadmin whose mail was all of a sudden being rejected by our outgoing relay. (This turned out to be one of the very few occasions where my special "accept all mail to postmaster regardless of ACLs" rule actually got used by someone to work around a problem!)

Looking at the logs I immediately saw that the rejected mail was arriving over IPv6 from a 6to4 address; our ACLs do not treat 6to4 addresses as being inside the University, so the mail was being rejected.

The quickest way to fix mail delivery was to add disable_ipv6 to the Exim configuration on the sender.

The breakage suddenly started happening at 15:24, which immediately made me suspect a bogus IPv6 router had appeared on the server's LAN at that time. Malc has a good description of how Windows Internet Connection Sharing breaks connectivity by issuing rogue router advertisements.

It is possible to identify the machine responsible for this kind of braindamage. The unwanted 6to4 address was of the form 2002:836f:... where 2002::/16 is the 6to4 prefix and the two following hexadectets are the IPv4 address of the tunnel endpoint. 836f:xxxx is hex for 131.111.ddd.ddd.

I trawled my logs for other 6to4 lossage and found a couple of colleges that had a few messages from their web servers rejected. One of them had at least four machines issuing rogue RAs.

It is going to be, um, interesting dealing with this in the future...

| Leave a comment | Share

Comments {6}

Pete

from: pjc50
date: 8th Jun 2011 16:27 (UTC)

That router annoucement bug is a disaster. Presumably people/malware could also start issuing malicious router advertisements?

Reply | Thread

Malc

from: mas90
date: 8th Jun 2011 16:29 (UTC)

Yes, but in that regard the only difference to traditional ARP spoofing is that RA spoofing has started happening by accident before anyone even bothered to do it maliciously...

Reply | Parent | Thread

frithonthehills

from: frithonthehills
date: 8th Jun 2011 16:27 (UTC)

Interesting, true, but I think part of the problem here is that we've never seen such high-profile use of IPv6 before. Having the UCS services listening on IPv6 will make the COs somewhat more likely to actually go and track down the rogue RAs.

Furthermore, the rogue RAs are much less of a problem once you have real IPv6 connectivity, as that will be preferred.

Reply | Thread

Tony Finch

from: fanf
date: 8th Jun 2011 16:49 (UTC)

Indeed. It is really helpful, though, to get an idea of the size of the latent problem, and it has made us a lot more aware of how much we should worry about IP-based access control during the IPv6 rollout. Maybe we also need to consider running our own 6to4 relay if that would allow us to treat the 6to4 translations of our IPv4 address space as local. At the moment some "internal" 6to4 traffic goes via HEAnet in Dublin...

Reply | Parent | Thread

6to4 relays

from: syscomet
date: 8th Jun 2011 20:29 (UTC)

It's generally useful to have your own 6to4 relays yourself anyway, even if you never plan to use 6to4, to reduce the pain and debugging for those who do have 6to4 enabled; less time wasted on debugging that, more time available to migrate to proper native IPv6.

By running your own relay, return traffic to external 6to4 addresses will travel over your IPv4 routes to their 6to4 box, removing one unknown gateway from the equation. This way, as long as someone's able to use 6to4 at all, it's likely to work well for them when they visit your sites, instead of breaking randomly because of, oh, some major backbone provider announcing the 6to4 prefix but filtering it inside their network.

And as long as you OSPF announce the prefix from the box yourself, you can have the machine fail and not lead to major breakage, as packets will revert to using a gateway outside your control but still probably working. So it really can be one commodity-class hardware box.

Reply | Parent | Thread

from: Dave Holland [org.uk]
date: 8th Jun 2011 20:31 (UTC)

Similarly unexciting here (sanger.ac.uk) - partly as a result of having gradually enabled IPv6 on various services over the last few weeks, rather than saving it for a big bang. Cheating, I know. ;-)

The only problem we found involved an off-site user on a dual-stack network connecting to our (stubbornly IPv4-only) VPN; they had a proxy explicitly named in the browser (against our recommendation); and found that they were unable to connect to our internal web caches, which are IPv6-enabled for clients as of today. This is ultimately fixable by the VPN vendor, or we can work around it with grotty DNS views or something. But we'll probably just turn off IPv6 client service for the web caches, and leave them able to make outgoing IPv6 requests.

Reply | Thread