# Friends

## Wednesday 18th December 2013

### daily_wtf CodeSOD: Don't Wear Out Your Methods

http://thedailywtf.com/Articles/Dont-Wear-Out-Your-Methods.aspx

Each year, whenever summer weather faded and the air started to turn brisk, Craig knew that it could mean only one thing - a new round of interns would be arriving in the department. Sure, some were...

### againstmonopoly Corruption: The New Hampshire Rebellion

http://www.againstmonopoly.org/index.php?perm=805808000000000657

Larry Lessig is says we should ask politicians just one question "What will you do to end this corruption?" I'm not sure I will trust their answers, but it is a good question. You find the details and help him here.

### cam_noir The Dean’s getaway vehicle was notable both for its sheer...

http://cambridgenoir.tumblr.com/post/70376290953

The Dean’s getaway vehicle was notable both for its sheer horsepower and its couch-like comfort …

### fixmystreet_cam Gwydir Street Road Surface, 18th December

http://www.fixmystreet.com/report/412397

The road surface down Gwydir street is very bad. This is an extremely popular cycling route and the surface makes it potentially dangerous to cyclists having to swerve to avoid on such a narrow road with many parked cars. There is a particularly bad pothole outside number 160 Gwydir Street (see photo).

Nearest road to the pin placed on the map (automatically generated by Bing Maps): 151 Gwydir Street, Cambridge
Report on FixMyStreet

### ucam_news Cells from the eye are inkjet-printed for the first time

http://www.cam.ac.uk/research/news/cells-from-the-eye-are-inkjet-printed-for-the-first-time

The breakthrough, which has been detailed in a paper published today, 18 December, in IOP Publishing’s journal Biofabrication, could lead to the production of artificial tissue grafts made from the variety of cells found in the human retina and may aid in the search to cure blindness.

At the moment the results are preliminary and provide proof-of-principle that an inkjet printer can be used to print two types of cells from the retina of adult rats―ganglion cells and glial cells. This is the first time the technology has been used successfully to print mature central nervous system cells and the results showed that printed cells remained healthy and retained their ability to survive and grow in culture.

Co-authors of the study Professor Keith Martin and Dr Barbara Lorber, from the John van Geest Centre for Brain Repair, University of Cambridge, said: “The loss of nerve cells in the retina is a feature of many blinding eye diseases. The retina is an exquisitely organised structure where the precise arrangement of cells in relation to one another is critical for effective visual function”.

“Our study has shown, for the first time, that cells derived from the mature central nervous system, the eye, can be printed using a piezoelectric inkjet printer. Although our results are preliminary and much more work is still required, the aim is to develop this technology for use in retinal repair in the future.”

The ability to arrange cells into highly defined patterns and structures has recently elevated the use of 3D printing in the biomedical sciences to create cell-based structures for use in regenerative medicine.

In their study, the researchers used a piezoelectric inkjet printer device that ejected the cells through a sub-millimetre diameter nozzle when a specific electrical pulse was applied. They also used high speed video technology to record the printing process with high resolution and optimised their procedures accordingly.

“In order for a fluid to print well from an inkjet print head, its properties, such as viscosity and surface tension, need to conform to a fairly narrow range of values. Adding cells to the fluid complicates its properties significantly,” commented Dr Wen-Kai Hsiao, another member of the team based at the Inkjet Research Centre in Cambridge.

Once printed, a number of tests were performed on each type of cell to see how many of the cells survived the process and how it affected their ability to survive and grow.

The cells derived from the retina of the rats were retinal ganglion cells, which transmit information from the eye to certain parts of the brain, and glial cells, which provide support and protection for neurons.

“We plan to extend this study to print other cells of the retina and to investigate if light-sensitive photoreceptors can be successfully printed using inkjet technology. In addition, we would like to further develop our printing process to be suitable for commercial, multi-nozzle print heads,” Professor Martin concluded.

The research was undertaken by Dr. Barbara Lorber, also at the John van Geest Centre for Brain Repair, in collaboration with Dr. Wen-Kai Hsiao and Prof. Ian Hutchings from the Inkjet Research Centre, University of Cambridge. The work was funded by Fight for Sight, the van Geest Foundation and the EPSRC.

Inset image: close-up of retinal cells in a jet

A group of researchers from the UK have used inkjet printing technology to successfully print cells taken from the eye for the very first time.

Our study has shown, for the first time, that cells derived from the mature central nervous system, the eye, can be printed using a piezoelectric inkjet printer
Professor Keith Martin and Dr Barbara Lorber
Father of the Eye - HDR

Yes
News type:

### ucam_news Flights of fancy: the evolution of plumage patterns in male and female birds

http://www.cam.ac.uk/research/news/flights-of-fancy-the-evolution-of-plumage-patterns-in-male-and-female-birds

Ducks, geese and swans are waterfowl, an order known to scientists as Anseriformes. Hens, pheasants, partridges and turkeys are game-birds (Galliformes). Both orders are famous not just for their flesh but also for their striking and elaborate plumages which are sought after as decorative flourishes.  Some members of these orders show marked differences in appearance between the sexes: a phenomenon known as sexual dimorphism. Male and female mallards look so different that for many years they were thought to be different species. In other members of the same orders, there is little apparent difference in the plumage of males and females.

Research by Cambridge PhD candidate Thanh-Lan Gluckman, published today in the Biological Journal of the Linnean Society, looks afresh at similarities and differences in plumage in almost 300 members of the Anseriformes and Galiformes orders – and focuses on patterning between male and female birds rather than colour. She said: “The colour of plumage has attracted much research interest, but the exquisite patterns of bird plumage, such as the spots of the guinea fowl and the barred patterns of ducks and turkeys, to just name a few, have received much less attention.”

Since the 1980s, differences in the appearances of male and female birds have been seen through a prism of genetic correlation. In other words, it was thought that female birds may have evolved similar patterning to males due to common genes but that female patterns would be subsequently lost as it is not beneficial.

“It was argued that male birds developed their spectacular colours and elaborate patterning as a result of their mating patterns – they used their plumage to compete for and attract females. On the other hand, female birds needed to blend into their surroundings in order to nest safely and protect their young – so they became drab and dull to protect themselves and their young from predators,” said Gluckman.

“My research looked at the plumage patterns of male and female birds on a separate and equal basis – and then went on to identify similarities and differences between them. By tracing the evolutionary pathways in the dimorphism of 288 species of waterfowl and gamebirds, I reconstructed the evolutionary history of plumage pattern sexual dimorphism, which allowed me to demonstrate that plumage patterns in females are not a result of genetic correlation. Essentially, what I found was that plumage patterning is remarkably labile – both male and female birds have the capacity to change between different types of patterns.  What’s interesting is to consider what are the forces driving these changes in male and female plumage patterns – whether they have an environmental basis and/or whether they have a signalling function between birds of different sexes or within the same sex.”

As early as 1780, the Philosophical Transactions of the Royal Society of London published a paper by John Hunter proposing that plumage differences between the sexes were driven by sexual selection. Ever since, the prevailing view of sexual dimorphism has been one of showy males strutting their stuff to win over demure females. The predominant explanation put forward to explain how differences in dimorphism evolved hinges on mating habits; males of polygamous species (those with more than one mate) had developed spectacular plumage in order to attract a maximum number of females, while monogamous species (those with one mate) retained similar plumage.

Gluckman said: “Previous research has shown that the traditional argument that differences in plumage between the sexes stem from differences in breeding systems doesn’t always hold up. In many putatively monogamous species, the plumage of the males is significantly different to that of females and, likewise, males and females in many polygamous species have the same type of plumage. This suggests that plumage is not exclusively an outcome of breeding habits – but is a matter of function in a highly complex way.”

In her study of patterning, Gluckman looked at the variations between the sexes of the same species and across species in order to build a picture of the pathways to similarity and differences between male and female bird plumage patterns. She used a classification of four broad types of patterning: mottled, scaled, barred and spotted. Birds exhibit a fabulous number of variations and combinations of these visual patterns in females as well as males.

“By emphasising similarities as well as differences in plumage patterns between male and female birds, rather than whether one sex is the same as the other, I found that sexual dimorphism in the plumage pattern of birds is highly nuanced and that there can be multiple types of sexual dimorphism. In expanding the definition of sexual dimorphism, and reconstructing evolutionary history, I found that changes in sexual dimorphism could be due to changes in males and/or females. In addition, the plumage patterns of birds seem to transition easily between different types of dimorphism, which is congruent with adaptation to fluctuating social and environmental conditions,” said Gluckman.

Thanh-Lan Gluckman is a PhD candidate in the Evolutionary genetics group at the Department of Zoology at the University of Cambridge. She carried out this research during her MPhil in the Department of Zoology, University of Melbourne, Australia.

Research published today looks at the evolutionary pathways to differences in bird plumage patterns between males and females – and concludes that birds are able to adapt their appearance with remarkable ease.

In expanding the definition of sexual dimorphism, and reconstructing evolutionary history, I found that changes in sexual dimorphism could be due to changes in males and/or females.
Thanh-Lan Gluckman
Images via Wikimedia
Waterfowl and gamebirds

Yes
News type:

### markdominus More about ZF's asymmetry between union and intersection

http://blog.plover.com/2012/08/24#topology-closed-2

In an article earlier this week, I explored some oddities of defining a toplogy in terms of closed sets rather than open sets, mostly as a result of analogous asymmetry in the ZF set theory axioms.

Let's review those briefly. The relevant axioms concern the operations by which sets can be constructed. There are two that are important. First is the axiom of union, which says that if is a family of sets, then we can form , which is the union of all the sets in the family.

The other is actually a family of axioms, the specification axiom schema. It says that for any one-place predicate and any set we can construct the subset of for which holds:

Both of these are required. The axiom of union is for making bigger sets out of smaller ones, and the specification schema is for extracting smaller sets from bigger ones. (Also important is the axiom of pairing, which says that if and are sets, then so is the two-element set ; with pairing and union we can construct all the finite sets. But we won't need it in this article.)

Conspicuously absent is an axiom of intersection. If you have a family of sets, and you want a set of every element that is in some member of , that is easy; it is what the axiom of union gets you. But if you want a set of every element that is in every member of , you have to use specification.

Let's begin by defining this compact notation:

for this longer formula:

This is our intersection of the members of , taken "relative to ", as we say in the biz. It gives us all the elements of that are in every member of . The is mandatory in , because ZF makes it mandatory when you construct a set by specification. If you leave it out, you get the Russell paradox.

Most of the time, though, the is not very important. When is nonempty, we can choose some element , and consider , which is the "normal" intersection of . We can easily show that

for any whatever, and this immediately implies that

for any two elements of , so when contains an element , we can omit the subscript and just write

for the usual intersection of members of .

Even the usually troublesome case of an empty family is no problem. In this case we have no to use for , but we can still take some other set and talk about , which is just .

Now, let's return to topology. I suggested that we should consider the following definition of a topology, in terms of closed sets, but without an a priori notion of the underlying space:

A co-topology is a family of sets, called "closed" sets, such that:

1. The union of any two elements of is again in , and
2. The intersection of any subfamily of is again in .
Item 2 begs the question of which intersection we are talking about here. But now that we have nailed down the concept of intersections, we can say briefly and clearly what we want: It is the intersection relative to . This set contains anything that is in any of the closed sets, and so , which I will henceforth call , is effectively a universe of discourse. It is certainly big enough that intersections relative to it will contain everything we want them to; remember that intersections of subfamilies of have a maximum size, so there is no way to make too big.

It now immediately follows that itself is a closed set, since it is the intersection of the empty subfamily of .

If itself is empty, then so is , and , so that is all right. From here on we will assume that is nonempty, and therefore that , with no relativization, is well-defined.

We still cannot prove that the empty set is closed; indeed, it might not be, because even might not be empty. But as David Turner pointed out to me in email, the elements of play a role dual to the extratoplogical points of a topological space that has been defined in terms of open sets. There might be points that are not in any open set anywhere, but we may as well ignore them, because they are topologically featureless, and just consider the space to be the union of the open sets. Analogously and dually, we can ignore the points of , which are topologically featureless in the same way. Rather than considering , we should consider , whose members are the members of , but with subtracted from each one:

So we may as well assume that this has been done behind the scenes and so that is empty. If we have done this, then the empty set is closed.

Now we move on to open sets. An open set is defined to be the complement of a closed set, but we have to be a bit careful, because ZF does not have a global notion of the complement of a set. Instead, it has only relative complements, or differences. is defined as:

Here we say that the complement of is taken relative to .

For the definition of open sets, we will say that the complement is taken relative to the universe of discourse , and a set is open if it has the form for some closed set .

Anatoly Karp pointed out on Twitter that we know that the empty set is open, because it is the relative complement of , which we already know is closed. And if we ensure that is empty, as in the previous paragraph, then since the empty set is closed, is open, and we have recovered all the original properties of a topology.

 Order General Topology with kickback no kickback
But gosh, what a pain it was; in contrast recovering the missing axioms from the corresponding open-set definition of a topology was painless. (John Armstrong said it was bizarre, and probably several other people were thinking that too. But I did not invent this bizarre idea; I got it from the opening paragraph of John L. Kelley's famous book General Topology, which has been in print since 1955.

Here Kelley deals with the empty set and the universe in two sentences, and never worries about them again. In contrast, doing the same thing for closed sets was fraught with technical difficulties, mostly arising from ZF. (The exception was the need to repair the nonemptiness of the minimal closed set , which was not ZF's fault.)

 Order On Numbers and Games with kickback no kickback
I don't think I have much of a conclusion here, except that whatever the advantages of ZF as a millieu for doing set theory, it is overrated as an underlying formalism for actually doing mathematics. (Another view on this is laid out by J.H. Conway in the Appendix to Part Zero of On Numbers and Games (Academic Press, 1976).) None of the problems we encountered were technically illuminating, and nothing was clarified by examining them in detail.

On the other hand, perhaps this conclusion is knocking down a straw man. I think working mathematicians probably don't concern themselves much with whether their stuff works in ZF, much less with what silly contortions are required to make it work in ZF. I think day-to-day mathematical work, to the extent that it needs to deal with set theory at all, handles it in a fairly naïve way, depending on a sort of folk theory in which there is some reasonably but not absurdly big universe of discourse in which one can take complements and intersections, and without worrying about this sort of technical detail.

[ MathJax doesn't work in Atom or RSS syndication feeds, and can't be made to work, so if you are reading a syndicated version of this article, such as you would in Google Reader, or on Planet Haskell or PhillyLinux, you are seeing inlined images provided by the Google Charts API. The MathJax looks much better, and if you would like to compare, please visit my blog's home site. ]

### markdominus How I got four errors into a one-line program

http://blog.plover.com/2012/12/16#git-commit-hook

At my current job, each task is assigned a ticket number of the form e12345. The git history is extremely convoluted, and it's been observed that it's easier to find things if you include the ticket number at the front of the commit message. I got tired of inserting it manually, and thought I would write a prepare-commit-message hook to insert it automatically.

A prepare-commit-message hook is a program that you stick in the file .git/hooks/prepare-commit-hook. When you run git-commit, git first writes the commit message to a file, then invokes the prepare-commit-message program on file; the program can modify the contents of the message, or abort the commit if it wants to. Then git runs the editor on the message, if it was going to do that, and creates the commit with the edited message.

The hook I wrote was basically a one-liner, and the reason I am posting this note is because I found three significant programming errors in it in the first day of use.

Here's the first cut:

case $2 in message) perl -i -lpe "s/^(e\d+:\s+)?/$(cs -): /" $1 ;; esac  This is a shell script, but the main purpose is to run the perl one-liner. The shell script gets two arguments:$1 is the path to the file that contains the proposed commit message. The $2 argument is a tag which describes the commit's context; it's merge if the commit is a merge commit, for example; it's template if the commit message is supplied from a template via -t on the command line or the commit.template configuration option. The default is the empty string, and message, which I have here, means that the message was supplied with the -m command-line option. The Perl script edits the commit message file, named in$1, in-place, looking for something like e12345: at the beginning of a line, and replacing it with the output of the cs - command, which is a little program I wrote to print the current ticket number.

(cs is run by the shell, and its output is inserted into the Perl script before perl is run, so that the program that Perl sees is something like s/^(e\d+:\s+)?/e12345: /.) Simple enough.

There is already an error here, although it's a design error, not an implementation error: the Perl one-liner is only invoked when $2 is message. For some reason I decided that I would want it only when I supplied git-commit with the -m message option. This belief lasted exactly until the first time I ran git-commit in default mode it popped up the editor to edit the commit message, and I had to insert the ticket number manually. So the first change was to let the hook run in the default case as well as the message case: case$2 in
""|message)
perl -i -lpe "s/^(e\d+:\s+)?/$(cs -): /"$1
;;
esac

This was wrong because it inserts the ticket number at the start of each line; I wanted it only at the start of the first line. So that was programming error number 1:

case $2 in ""|message) perl -i -lpe "$. == 1 && s/^(e\d+:\s+)?/$(cs -): /"$1
;;
esac

So far, so good.

Bug #2 appeared the first time I tried a rebase. The cs command infers the ticket number from the name of the current branch. If it fails, it issues a warning and emits the string eXXXXX instead. During a rebase, the head is detached and there is no current branch. So the four commits I rebased all had their formerly-correct ticket numbers replaced with the string eXXXXX.

There are several ways to fix this. The best way would be to make sure that the current ticket number was stashed somewhere that cs could always get it. Instead, I changed the Perl script to recognize when the commit message already began with a ticket number, and to leave it alone if so:

case $2 in ""|message) perl -i -lpe "\$. == 1 && !/^e\d+:\s+/ && s/^/$(cs -): /"$1
;;
esac

It probably would have been a good idea to leave an escape hatch, and have cs emit the value of $ENV{TICKET_NUMBER} if that is set, to allow invocations like TICKER_NUMBER=e71828 git commit -m …, but I didn't do it, yet. The third bug appeared when I did git commit --fixup for the first time. With --fixup you tell it which commit you are trying to fix up, and it writes the commit message in a special form that tells a subsequent git-rebase --interactive that this new commit should be handled specially. (It should be applied immediately after that other one, and should be marked as a "fixup", which means that it is squashed into the other one and that its log message is discarded in favor of the other one.) If you are fixing up a commit whose message was Frobulate the veeblefetzers, the fixup commit's message is automatically generated as fixup! Frobulate the veeblefetzers. Or it would have been, if you were not using my prepare-commit-message hook, which would rewrite it to e12345: fixup! Frobulate the veeblefetzers. This is not in the right form, so it's not recognized by git-rebase --interactive for special handling. So the hook became: case$2 in
""|message)
perl -i -lpe "\$. == 1 && !/^(squash|fixup)! / && !/^e\d+:\s+/ && s/^/$(cs -): /" $1 ;; esac  (The exception for squash is similar to the one for fixup. I never use squash, but it seemed foolish not to put it in while I was thinking of it.) This is starting to look a little gross, but in a program this small I can tolerate a little grossness. I thought it was remarkable that such a small program broke in so many different ways. Much of that is because it must interact with git, which is very large and complicated, and partly it is that it must interact with git, which is in many places not very well designed. The first bug, where the ticket number was appended to each line instead of just the first, is not git's fault. It was fallout from my initial bad design decision to apply the script only to messages supplied with -m, which are typically one-liners, so that's what I was thinking of when I wrote the Perl script. But the other two errors would have been avoided had the interface to the hook been more uniform. There seems to be no reason that rebasing (or cherry-picking) and git-commit --fixup contexts couldn't have been communicated to the hook via the same$2 argument that communicates other contexts. Had this been done in a more uniform way, my program would have worked more correctly. But it wasn't done, and it's probably too late to change it now, since such a change risks breaking many existing prepare-commit-message hooks. ("The enemy of software is software.) A well-written hook will of course have a catchall:

case $2 in ""|message) perl -i -lpe "\$. == 1 && !/^(squash|fixup)! / && !/^e\d+:\s+/ && s/^/$(cs -): /"$1
;;

merge|template|squash|commit)
# do nothing
;;
*)      # wat
echo "prepare-message-hook: unknown context '$2'" 1>&2 exit 1; ;; esac  But mine doesn't and I bet a lot of others don't either. ### wes_felter Ars Technica: Microsoft joins FIDO group hoping to replace passwords with public key cryptography http://blog.felter.org/post/70347398804 Ars Technica: Microsoft joins FIDO group hoping to replace passwords with public key cryptography: Hopefully this can merge with Persona since it has already made significant progress. ### wes_felter David Andersen: Briefly profitable alt-coin mining on Amazon through better code ### dotaturls Common markup for Markdown and reStructuredText. https://gist.github.com/dupuy/1855764 http://dotat.at/:/B8ZGG.html Common markup for Markdown and reStructuredText. ### eaglespath lbcd 3.4.2 http://www.eyrie.org/~eagle/journal/2013-12/004.html lbcd is the daemon run on individual systems that participate in DNS-based load-balanced pools using lbnamed. This is a portability release that (finally) switches the default API for user login information over to getutxent from getutent (required for Mac OS X) and enables building on FreeBSD and Debian GNU/kFreeBSD systems. Note that lbcd will only work on FreeBSD systems if the Linux-compatible /proc file system is mounted, but this appears to be a common configuration. You can get the latest release from the lbcd distribution page. ### markdominus Moonpig: a billing system that doesn't suck http://blog.plover.com/2013/12/17#Moonpig I'm in Amsterdam now, because Booking.com brought me out to tell them about Moonpig, the billing and accounting system that Rik Signes and I wrote. The talk was mostly a rehash of one I gave a Pittsburgh Perl Workshop a couple of months ago, but I think it's of general interest. The assumption behind the talk is that nobody wants to hear about how the billing system actually works, because most people either have their own billing system already or else don't need one at all. I think I could do a good three-hour talk about the internals of Moonpig, and it would be very interesting to the right group of people, but it would be a small group. So instead I have this talk, which lasts less than an hour. The takeaway from this talk is a list of several basic design decisions that Rik and I made while building Moonpig which weren't obviously good ideas at the time, but which turned out well in hindsight. That part I think everyone can learn from. You may not ever need to write a billing system, but chances are at some point you'll consider using an ORM, and it might be useful to have a voice in your head that says “Dominus says it might be better to do something completely different instead. I wonder if this is one of those times?” So because I think the talk was pretty good, and it's fresh in my mind right now, I'm going to try to write it down. The talk slides are here if you want to see them. The talk is mostly structured around a long list of things that suck, and how we tried to design Moonpig to eliminate, avoid, or at least mitigate these things. Moonpig, however, does not suck. Sometimes I see other people fuck up a project over and over, and I say “I could do that better”, and then I get a chance to try, and I discover it was a lot harder than I thought, I realize that those people who tried before are not as stupid as as I believed. That did not happen this time. Moonpig is a really good billing system. It is not that hard to get right. Those other guys really were as stupid as I thought they were. ### Brief explanation of IC Group When I tell people I was working for IC Group, they frown; they haven't heard of it. But quite often I say that IC Group runs pobox.com, and those same people say “Oh, pobox!”. ICG is a first wave dot-com. In the late nineties, people would often have email through their employer or their school, and then they would switch jobs or graduate and their email address would go away. The basic idea of pobox was that for a small fee, something like$15 per year, you could get a pobox.com address that would forward all your mail to your real email address. Then when you changed jobs or schools you could just tell pobox to change the forwarding record, and your friends would continue to send email to the same pobox.com address as before. Later, ICG offered mail storage, web mail, and, through listbox.com, mailing list management and bulk email delivery.

Moonpig was named years and years before the project to write it was started. ICG had a billing and accounting system already, a terrible one. ICG employees would sometimes talk about the hypothetical future accounting system that would solve all the problems of the current one. This accounting system was called Moonpig because it seemed clear that it would never actually be written, until pigs could fly.

And in fact Moonpig wouldn't have been written, except that the existing system severely constrained the sort of pricing structures and deals that could actually be executed, and so had to go. Even then the first choice was to outsource the billing and accounting functions to some company that specialized in such things. The Moonpig project was only started as a last resort after ICG's president had tried for 18 months to find someone to take over the billing and collecting. She was unsuccessful. A billing provider would seem perfect and then turn out to have some bizarre shortcoming that rendered it unsuitable for ICG's needs. The one I remember was the one that did everything we wanted, except it would not handle checks. “Don't worry,” they said. “It's 2010. Nobody pays by check any more.”

Well, as it happened, many of our customers, including some of the largest institutional ones, had not gotten this memo, and did in fact pay by check.

So with some reluctance, she gave up and asked Rik and me to write a replacement billing and accounting system.

As I mentioned, I had always wanted to do this. I had very clear ideas, dating back many years, about mistakes I would not make, were I ever called upon to write a billing system.

For example, I have many times received a threatening notice of this sort:

Your account is currently past due! Pay the outstanding balance of $0 . 00 or we will be forced to refer your account for collection. What I believe happened here is: some idiot programmer knows that money amounts are formatted with decimal points, so decides to denominate the money with floats. The amount I paid rounds off a little differently than the amount I actually owed, and the result after subtraction is all roundoff error, and leaves me with a nominal debt on the order of dollars. So I have said to myself many times “If I'm ever asked to write a billing system, it's not going to use any fucking floats.” And at the meeting at which the CEO told me and Rik that we would write it, those were nearly the first words out of my mouth: No fucking floats. ### Moonpig conceptual architecture I will try to keep this as short as possible, including only as much as is absolutely required to understand the more interesting and generally applicable material later. #### Pobox and Listbox accounts ICG has two basic use cases. One is Pobox addresses and mailboxes, where the customer pays us a certain amount of money to forward (or store) their mail for a certain amount of time, typically a year. The other is Listbox mailing lists, where the customer pays us a certain amount to attempt a certain number of bulk email deliveries on their behalf. #### The basic model is simple… The life cycle for a typical service looks like this: The customer pays us some money: a flat fee for a Pobox account, or a larger or smaller pile for Listbox bulk mailing services, depending on how much mail they need us to send. We deliver service for a while. At some point the funds in the customer's account start to run low. That's when we send them an invoice for an extension of the service. If they pay, we go back and continue to provide service and the process repeats; if not, we stop providing the service. #### …just like all basic models But on top of this basic model there are about 10,019 special cases: • Customers might cancel their service early. • Pobox has a long-standing deal where you get a sixth year free if you pay for five years of service up front. • Sometimes a customer with only email forwarding ($20 per year) wants to upgrade their account to one that does storage and provides webmail access ($50 per year), or vice-versa, in the middle of a year. What to do in this case? Business rules dictate that they can apply their current balance to the new service, and it should be properly pro-rated. So if I have 64 days of$50-per-year service remaining, and I downgrade to the $20-per-year service, I now have 160 days of service left. Well, that wasn't too bad, except that we should let the customer know the new expiration date. And also, if their service will now expire sooner than it would have, we should give them a chance to pay to extend the service back to the old date, and deal properly with their payment or nonpayment. Also something has to be done about any 6th free year that I might have had. We don't want someone to sign up for 5 years of$50-per-year service, get the sixth year free, then downgrade their account and either get a full free year of $50-per-year service or get a full free year of$20-per-year service after only of five full years.

• Sometimes customers do get refunds.

• Sometimes we screw up and give people a credit for free service, as an apology. Unlike regular credits, these are not refundable!

• Some customers get gratis accounts. The other cofounder of ICG used to hand these out at parties.

• There are a number of cases for coupons and discounts. For example, if you refer a friend who signs up, you get some sort of credit. Non-profit institutions get some sort of discount off the regular rates. Customers who pay for many accounts get some sort of bulk discount. I forget the details.

• Most customers get their service cut off if they don't pay. Certain large and longstanding customers should not be treated so peremptorily, and are allowed to run a deficit.

• And so to infinity and beyond.

#### Ledgers and Consumers

The Moonpig data store is mostly organized as a huge pile of ledgers. Each represents a single customer or account. It contains some contact information, a record of all the transactions associated with that customer, a history of all the invoices ever sent to that customer, and so forth.

It also contains some consumer objects. Each consumer represents some service that we have promised to perform in exchange for money. The consumer has methods in it that you can call to say “I just performed a certain amount of service; please charge accordingly”. It has methods for calculating how much money has been allotted to it, how much it has left, how fast it is consuming its funds, how long it expects to last, and when it expects to run out of money. And it has methods for constructing its own replacement and for handing over control to that replacement when necessary.

#### Heartbeats

Every day, a cron job sends a heartbeat event to each ledger. The ledger doesn't do anything with the heartbeat itself; its job is to propagate the event to all of its sub-components. Most of those, in turn, ignore the heartbeat event entirely.

But consumers do handle heartbeats. The consumer will wake up and calculate how much longer it expects to live. (For Pobox consumers, this is simple arithmetic; for mailing-list consumers, it guesses based on how much mail has been sent recently.) If it notices that it is going to run out of money soon, it creates a successor that can take over when it is gone. The successor immediately sends the customer an invoice: “Hey, your service is running out, do you want to renew?”

Eventually the consumer does run out of money. At that time it hands over responsibility to its replacement. If it has no replacement, it will expire, and the last thing it does before it expires is terminate the service.

### Things that suck: manual repairs

Somewhere is a machine that runs a daily cron job to heartbeat each ledger. What if one day, that machine is down, as they sometimes are, and the cron job never runs?

Or what if that the machine crashes while the cron job is running, and the cron job only has time to heartbeat 3,672 of the 10,981 ledgers in the system?

In a perfect world, every component would be able to depend on exactly one heartbeat arriving every day. We don't live in that world. So it was an ironclad rule in Moonpig development that anything that handles heartbeat events must be prepared to deal with missing heartbeats, duplicate heartbeats, or anything else that could screw up.

When a consumer gets a heartbeat, it must not cheerfully say "Oh, it's the dawn of a new day! I'll charge for a day's worth of service!". It must look at the current date and at its own charge record and decide on that basis whether it's time to charge for a day's worth of service.

Now the answers to those questions of a few paragraphs earlier are quite simple. What if the machine is down and the cron job never runs? What to do?

A perfectly acceptable response here is: Do nothing. The job will run the next day, and at that time everything will be up to date. Some customers whose service should have been terminated today will have it terminated tomorrow instead; they will have received a free day of service. This is an acceptable loss. Some customers who should have received invoices today will receive them tomorrow. The invoices, although generated and sent a day late, will nevertheless show the right dates and amounts. This is also an acceptable outcome.

What if the cron job crashes after heartbeating 3,672 of 10,981 ledgers? Again, an acceptable response is to do nothing. The next day's heartbeat will bring the remaining 7,309 ledgers up to date, after which everything will be as it should. And an even better response is available: simply rerun the job. 3,672 of the ledgers will receive the same event twice, and will ignore it the second time.

Contrast this with the world in which heartbeats were (mistakenly) assumed to be reliable. In this world, the programming staff must determine precisely which ledgers received the event before the crash, either by trawling through the log files or by grovelling over the ledger data. Then someone has to hack up a program to send the heartbeats to just the 7,309 ledgers that still need it. And there is a stiff deadline: they have to get it done before tomorrow's heartbeat issues!

Making everything robust in the face of heartbeat failure is a little more work up front, but that cost is recouped the first time something goes wrong with the heartbeat process, when instead of panicking you smile and open another beer. Let N be the number of failures and manual repairs that are required before someone has had enough and makes the heartbeat handling code robust. I hypothesize that you can tell a lot about an organization from the value of N.

Here's an example of the sort of code that is required. The non-robust version of the code would look something like this:

        sub charge {
my ($self,$event) = @_;
$self->charge_one_day($next);
}

The code, implemented by a role called Moonpig::Role::Consumer::ChargesPeriodically, actually looks something like this:

        has last_charge_date => ( … );

sub charge {
my ($self,$event) = @_;

my $now =$event->timestamp;

CHARGE: until ($self->next_charge_date->follows($now)) {
my $next =$self->next_charge_date;
$self->charge_one_day($next);
$self->last_charge_date($next);
if ($self->is_expired) {$self->replacement->handle_event($event) if$self->replacement;
last CHARGE;
}
}
}

The last_charge_date member records the last time the consumer actually issued a charge. The next_charge_date method consults this value and returns the next day on which the consumer should issue a charge—not necessarily the following day, since the consumer might issue weekly or monthly charges. The consumer will issue charge after charge until the next_charge_date is the future, when it will stop. It runs the until loop, using charge_one_day to issue another charge each time through, and updating last_charge_date each time, until the next_charge_date is in the future.

The one tricky part here the if block. This is because the consumer might run out of money before the loop completes. In that case it passes the heartbeat event on to its successor (replacement) and quits the loop. The replacement will run its own loop for the remaining period.

### Things that suck: real-time testing

A customer pays us $20. This will cover their service for 365 days. The business rules say that they should receive their first invoice 30 days before the current service expires; that is, after 335 days. How are we going to test that the invoice is in fact sent precisely 335 days later? Well, put like that, the answer is obvious: Your testing system must somehow mock the time. But obvious as this is, I have seen many many tests that made some method call and then did sleep 60, waiting and hoping that the event they were looking for would have occurred by then, reporting a false positive if the system was slow, and making everyone that much less likely to actually run the tests. I've also seen a lot of tests that crossed their fingers and hoped that a certain block of code would execute between two ticks of the clock, and that failed nondeterministically when that didn't happen. So another ironclad law of Moonpig design was that no object is ever allowed to call the time() function to find out what time it actually is. Instead, to get the current time, the object must call Moonpig->env->now. The tests run in a test environment. In the test environment, Moonpig->env returns a Moonpig::Env::Test object, which contains a fake clock. It has a stop_clock method that stops the clock, and an elapse_time method that forces the clock forward a certain amount. If you need to check that something happens after 40 days, you can call Moonpig->env->elapse_time(86_400 * 40), or, more likely:  for (1..40) { Moonpig->env->elapse_time(86_400);</tt>$test_ledger->heartbeat;
}

In the production environment, the environment object still has a now method, but one that returns the true current time from the system clock. Trying to stop the clock in the production environment is a fatal error.

Similarly, no Moonpig object ever interacts directly with the database; instead it must always go through the mediator returned by Moonpig->env->storage. In tests, this can be a fake storage object or whatever is needed. It's shocking how many tests I've seen that begin by allocating a new MySQL instance and executing a huge pile of DDL. Folks, this is not how you write a test.

Again, no Moonpig object ever posts email. It asks Moonpig->env->email_sender to post the email on its behalf. In tests, this uses the CPAN Email::Sender::Transport suite, and the test code can interrogate the email_sender to see exactly what emails would have been sent.

We never did anything that required filesystem access, but if we had, there would have been a Moonpig->env->fs for opening and writing files.

The Moonpig->env object makes this easy to get right, and hard to screw up. Any code that acts on the outside world becomes a red flag: Why isn't this going through the environment object? How are we going to test it?

### Things that suck: floating-point numbers

I've already complained about how I loathe floating-point numbers. I just want to add that although there are probably use cases for floating-point arithmetic, I don't actually know what they are. I've had a pretty long and varied programming career so far, and legitimate uses for floating point numbers seem very few. They are really complicated, and fraught with traps; I say this as a mathematical expert with a much stronger mathematical background than most programmers.

The law we adopted for Moonpig was that all money amounts are integers. Each money amount is an integral number of “millicents”, abbreviated “m¢”, worth of a cent, which in turn is of a U.S. dollar. Fractional millicents are not allowed. Division must be rounded to the appropriate number of millicents, usually in the customer's favor, although in practice it doesn't matter much, because the amounts are so small.

For example, a $20-per-year Pobox account actually bills m¢ each day. (5464 in leap years.) Since you don't want to clutter up the test code with a bunch of numbers like 1000000 ($10), there are two utterly trivial utility subroutines:

        sub cents   { $_[0] * 1000 } sub dollars {$_[0] * 1000 * 100 }

Now $10 can be written dollars(10). Had we dealt with floating-point numbers, it would have been tempting to write test code that looked like this:  cmp_ok(abs($actual_amount - $expected_amount), "<",$EPSILON, …);

That's because with floats, it's so hard to be sure that you won't end up with a leftover or something, so you write all the tests to ignore small discrepancies. This can lead to overlooking certain real errors that happen to result in small discrepancies. With integer amounts, these discrepancies have nowhere to hide. It sometimes happened that we would write some test and the money amount at the end would be wrong by 2m¢. Had we been using floats, we might have shrugged and attributed this to incomprehensible roundoff error. But with integers, that is a difference of 2, and you cannot shrug it off. There is no incomprehensible roundoff error. All the calculations are exact, and if some integer is off by 2 it is for a reason. These tiny discrepancies usually pointed to serious design or implementation errors. (In contrast, when a test would show a gigantic discrepancy of a million or more m¢, the bug was always quite easy to find and fix.)

There are still roundoff errors; they are unavoidable. For example, a consumer for a $20-per-year Pobox account bills only 365·5479m¢ = 1999835m¢ per year, an error in the customer's favor of 165m¢ per account; after 12 million years the customer will have accumulated enough error to pay for an extra year of service. For a business of ICG's size, this loss was deemed acceptable. For a larger business, it could be significant. (Imagine 6,000,000 customers times 165m¢ each; that's$9,900.) In such a case I would keep the same approach but denominate everything in micro-cents instead.

Happily, Moonpig did not have to deal with multiple currencies. That would have added tremendous complexity to the financial calculations, and I am not confident that Rik and I could have gotten it right in the time available.

### Things that suck: dates and times

Dates and times are terribly complicated, partly because the astronomical motions they model are complicated, and mostly because the world's bureaucrats keep putting their fingers in. It's been suggested recently that you can identify whether someone is a programmer by asking if they have an opinion on time zones. A programmer will get very red in the face and pound their fist on the table.

After I wrote that sentence, I then wrote 1,056 words about the right way to think about date and time calculations, which I'll spare you, for now. I'm going to try to keep this from turning into an article about all the ways people screw up date and time calculations, by skipping the arguments and just stating the main points:

1. Date-time values are a kind of number, and should be considered as such. In particular:
1. Date-time values inside a program should be immutable
2. There should be a single canonical representation of date-time values in the program, and it should be chosen for ease of calculation.
2. If the program does have to deal with date-time values in some other representation, it should convert them to the canonical representation as soon as possible, or from the canonical representation as late as possible, and in any event should avoid letting non-canonical values percolate around the program.
The canonical representation we chose was DateTime objects in UTC time. Requiring that the program deal only with UTC eliminates many stupid questions about time zones and DST corrections, and simplifies all the rest as much as they can be simplified. It also avoids DateTime's unnecessarily convoluted handling of time zones.

We held our noses when we chose to use DateTime. It has my grudging approval, with a large side helping of qualifications. The internal parts of it are okay, but the methods it provides are almost never what you actually want to use. For example, it provides a set of mutators. But, as per item 1 above, date-time values are numbers and ought to be immutable. Rik has a good story about a horrible bug that was caused when he accidentally called the ->subtract method on some widely-shared DateTime value and so mutated it, causing an unexpected change in the behavior of widely-separated parts of the program that consulted it afterward.

So instead of using raw DateTime, we wrapped it in a derived class called Moonpig::DateTime. This removed the mutators and also made a couple of other convenient changes that I will shortly describe.

#### Things that really really suck: DateTime::Duration

If you have a pair of DateTime objects and you want to know how much time separates the two instants that they represent, you have several choices, most of which will return a DateTime::Duration object. All those choices are wrong, because DateTime::Duration objects are useless. They are a kind of Roach Motel for date and time information: Data checks into them, but doesn't check out. I am not going to discuss that here, because if I did it would take over the article, but I will show the simple example I showed in the talk:

        my $then = DateTime->new( month => 4, day => 2, year => 1969, hour => 0, minute => 0, second => 0); my$now = DateTime->now();
my $elapsed =$now - $then; print$elapsed->in_units('seconds'), "\n";

You might think, from looking at this code, that it might print the number of seconds that elapsed between 1969-04-02 00:00:00 (in some unspecified time zone!) and the current moment. You would be mistaken; you have failed to reckon with the $elapsed object, which is a DateTime::Duration. Computing this object seems reasonable, but as far as I know once you have it there is nothing to do but throw it away and start over, because there is no way to extract from it the elapsed amount of time, or indeed anything else of value. In any event, the print here does not print the correct number of seconds. Instead it prints ME CAGO EN LA LECHE, which I have discovered is Spanish for “I shit in the milk”. So much for DateTime::Duration. When a and b are Moonpig::DateTime objects, a-b returns the number of seconds that have elapsed between the two times; it is that simple. You can divide it by 86,400 to get the number of days. Other arithmetic is similarly overloaded: If i is a number, then a+i and a-i are the times obtained by adding or subtracting i seconds to a, respectively. (C programmers should note the analogy with pointer arithmetic; C's pointers, and date-time values—also temperatures—are examples of a mathematical structure called an affine space, and study of the theory of affine spaces tells you just what rules these objects should obey. I hope to discuss this at length another time.) Going along with this arithmetic are a family of trivial convenience functions, such as:  sub hours {$_[0] * 3600  }
sub days  { $_[0] * 86400 }  so that you can use$a + days(7) to find the time 7 days after $a. Programmers at the Amsterdam talk were worried about this: what about leap seconds? And they are correct: the name days is not quite honest, because it promises, but does not deliver, exactly 7 days. It can't, because the definition of the day varies widely from place to place and time to time, and not only can't you know how long 7 days unless you know where it is, but it doesn't even make sense to ask. That is all right. You just have to be aware, when you add days(7), the the resulting time might not be the same time of day 7 days later. (Indeed, if the local date and time laws are sufficiently bizarre, it could in principle be completely wrong. But since Moonpig::DateTime objects are always reckoned in UTC, it is never more than one second wrong.) Anyway, I was afraid that Moonpig::DateTime would turn out to be a leaky abstraction, producing pleasantly easy and correct results thirty times out of thirty-one, and annoyingly wrong or bizarre results the other time. But I was surprised: it never caused a problem, or at least none has come to light. I am working on releasing this module to CPAN, under the name DateTime::Moonpig. (A draft version is already available, but I don't recommend that you use it.) ### Things that suck: mutable data I left this out of the talk, by mistake, but this is a good place to mention it: mutable data is often a bad idea. In the billing system we wanted to avoid it for accountability reasons: We never wanted the customer service agent to be in the position of being unable to explain to the customer why we thought they owed us$28.39 instead of the $28.37 they claimed they owed; we never wanted ourselves to be in the position of trying to track down a billing system bug only to find that the trail had been erased. One of the maxims Rik and I repeated freqently was that the moving finger writes, and, having writ, moves on. Moonpig is full of methods with names like is_expired, is_superseded, is_canceled, is_closed, is_obsolete, is_abandoned and so forth, representing entities that have been replaced by other entities but which are retained as part of the historical record. For example, a consumer has a successor, to which it will hand off responsibility when its own funds are exhausted; if the customer changes their mind about their future service, this successor might be replaced with a different one, or replaced with none. This doesn't delete or destroy the old successor. Instead it marks the old successor as "superseded", simultaneously recording the supersession time, and pushes the new successor (or undef, if none) onto the end of the target consumer's replacement_history array. When you ask for the current successor, you are getting the final element of this array. This pattern appeared in several places. In a particularly simple example, a ledger was required to contain a Contact object with contact information for the customer to which it pertained. But the Contact wasn't simply this:  has contact => ( is => 'rw', isa => role_type( 'Moonpig::Role::Contact' ), required => 1, );  Instead, it was an array; "replacing" the contact actually pushed the new contact onto the end of the array, from which the contact accessor returned the final element:  has contact_history => ( is => 'ro', isa => ArrayRef[ role_type( 'Moonpig::Role::Contact' ) ], required => 1, traits => [ 'Array' ], handles => { contact => [ get => -1 ], replace_contact => 'push', }, );  ### Things that suck: relational databases Why do we use relational databases, anyway? Is it because they cleanly and clearly model the data we want to store? No, it's because they are lightning fast. When your data truly is relational, a nice flat rectangle of records, each with all the same fields, RDBs are terrific. But Moonpig doesn't have much relational data. It basic datum is the Ledger, which has a bunch of disparate subcomponents, principally a heterogeneous collection of Consumer objects. And I would guess that most programs don't deal in relational data; Like Moonpig, they deal in some sort of object network. Nevertheless we try to represent this data relationally, because we have a relational database, and when you have a hammer, you go around hammering everything with it, whether or not that thing needs hammering. When the object model is mature and locked down, modeling the objects relationally can be made to work. But when the object model is evolving, it is a disaster. Your relational database schema changes every time the object model changes, and then you have to find some way to migrate the existing data forward from the old schema. Or worse, and more likely, you become reluctant to let the object model evolve, because reflecting that evolution in the RDB is so painful. The RDB becomes a ball and chain locked to your program's ankle, preventing it from going where it needs to go. Every change is difficult and painful, so you avoid change. This is the opposite of the way to design a good program. A program should be light and airy, its object model like a string of pearls. In theory the mapping between the RDB and the objects is transparent, and is taken care of seamlessly by an ORM layer. That would be an awesome world to live in, but we don't live in it and we may never. #### Things that really really suck: ORM software Right now the principal value of ORM software seems to be if your program is too fast and you need it to be slower; the ORM is really good at that. Since speed was the only benefit the RDB was providing in the first place, you have just attached two large, complex, inflexible systems to your program and gotten nothing in return. Watching the ORM try to model the objects is somewhere between hilariously pathetic and crushingly miserable. Perl's DBIx::Class, to the extent it succeeds, succeeds because it doesn't even try to model the objects in the database. Instead it presents you with objects that represent database rows. This isn't because a row needs to be modeled as an object—database rows have no interesting behavior to speak of—but because the object is an access point for methods that generate SQL. DBIx::Class is not for modeling objects, but for generating SQL. I only realized this recently, and angrily shouted it at the DBIx::Class experts, expecting my denunciation to be met with rage and denial. But they just smiled with amusement. “Yes,” said the DBIx::Class experts on more than one occasion, “that is exactly correct.” Well then. So Rik and I believe that for most (or maybe all) projects, trying to store the objects in an RDB, with an ORM layer mediating between the program and the RDB, is a bad, bad move. We determined to do something else. We eventually brewed our own object store, and this is the part of the project of which I'm least proud, because I believe we probably made every possible mistake that could be made, even the ones that everyone writing an object store should already know not to make. For example, the object store has a method, retrieve_ledger, which takes a ledger's ID number, reads the saved ledger data from the disk, and returns a live Ledger object. But it must make sure that every such call returns not just a Ledger object with the right data, but the same object. Otherwise two parts of the program will have different objects to represent the same data, one part will modify its object, and the other part, looking at a different object, will not see the change it should see. It took us a while to figure out problems like this; we really did not know what we were doing. What we should have done, instead of building our own object store, was use someone else's object store. KiokuDB is frequently mentioned in this context. After I first gave this talk people asked “But why didn't you use KiokuDB?” or, on hearing what we did do, said “That sounds a lot like KiokuDB”. I had to get Rik to remind me why we didn't use KiokuDB. We had considered it, and decided to do our own not for technical but for political reasons. The CEO, having made the unpleasant decision to have me and Rik write a new billing system, wanted to see some progress. If she had asked us after the first week what we had accomplished, and we had said “Well, we spent a week figuring out KiokuDB,” her head might have exploded. Instead, we were able to say “We got the object store about three-quarters finished”. In the long run it was probably more expensive to do it ourselves, and the result was certainly not as good. But in the short run it kept the customer happy, and that is the most important thing; I say this entirely in earnest, without either sarcasm or bitterness. (On the other hand, when I ran this article by Rik, he pointed out that KiokuDB had later become essentially unmaintained, and that had we used it he would have had to become the principal maintainer of a large, complex system which which he did not help design or implement. The Moonpig object store may be technically inferior, but Rik was with it from the beginning and understands it thoroughly.) #### Our object store All that said, here is how our object store worked. The bottom layer was an ordinary relational database with a single table. During the test phase this database was SQLite, and in production it was IC Group's pre-existing MySQL instance. The table had two fields: a GUID (globally-unique identifier) on one side, and on the other side a copy of the corresponding Ledger object, serialized with Perl's Storable module. To retrieve a ledger, you look it up in the table by GUID. To retrieve a list of all the ledgers, you just query the GUID field. That covers the two main use-cases, which are customer service looking up a customer's account history, and running the daily heartbeat job. A subsidiary table mapped IC Group's customer account numbers to ledger GUIDs, so that the storage engine could look up a particular customer's ledger starting from their account number. (Account numbers are actually associated with Consumers, but once you had the right ledger a simple method call to the ledger would retrieve the consumer object. But finding the right ledger required a table.) There were a couple of other tables of that sort, but overall it was a small thing. There are some fine points to consider. For example, you can choose whether to store just the object data, or the code as well. The choice is clear: you must store only the data, not the code. Otherwise, you would have to update all the objects every time you make a code change such as a bug fix. It should be clear that this would discourage bug fixes, and that had we gone this way the project would have ended as a pile of smoking rubble. Since the code is not stored in the database, the object store must be responsible, whenever it loads an object, for making sure that the correct class for that object actually exists. The solution for this was that along with every object is stored a list of all the roles that it must perform. At object load time, if the object's class doesn't exist yet, the object store retrieves this list of roles (stored in a third column, parallel to the object data) and uses the MooseX::ClassCompositor module to create a new class that does those roles. MooseX::ClassCompositor was something Rik wrote for the purpose, but it seems generally useful for such applications. Every once in a while you may make an upward-incompatible change to the object format. Renaming an object field is such a change, since the field must be renamed in all existing objects, but adding a new field isn't, unless the field is mandatory. When this happened—much less often than you might expect—we wrote a little job to update all the stored objects. This occurred only seven times over the life of the project; the update programs are all very short. We did also make some changes to the way the objects themselves were stored: Booking.Com's Sereal module was released while the project was going on, and we switched to use it in place of Storable. Also one customer's Ledger object grew too big to store in the database field, which could have been a serious problem, but we were able to defer dealing with the problem by using gzip to compress the serialized data before storing it. #### The relational database provides transactions The use of the RDB engine for the underlying storage got us MySQL's implementation of transactions and atomicity guarantees, which we trusted. This gave us a firm foundation on which to build the higher functions; without those guarantees you have nothing, and it is impossible to build a reliable system. But since they are there, we could build a higher-level transactional system on top of them. For example, we used an opportunistic locking scheme to prevent race conditions while updating a single ledger. For performance reasons you typically don't want to force all updates to be done through a single process (although it can be made to work; see Rochkind's Advanced Unix Programming). In an optimistic locking scheme, you store a version number with each record. Suppose you are the low-level storage manager and you get a request to update a ledger with a certain ID. Instead of doing this:  update ledger set serialized_data = … where ledger_id = 789  You do this:  update ledger set serialized_data = … , version = 4 where ledger_id = 789 and version = 3  and you check the return value from the SQL to see how many records were actually updated. The answer must be 0 or 1. If it is 1, all is well and you report the successful update back to your caller. But if it is 0, that means that some other process got there first and updated the same ledger, changing its version number from the 3 you were expecting to something bigger. Your changes are now in limbo; they were applied to a version of the object that is no longer current, so you throw an exception. But is the exception safe? What if the caller had previously made changes to the database that should have been rolled back when the ledger failed to save? No problem! We had exposed the RDB transactions to the caller, so when the caller requested that a transaction be begun, we propagated that request into the RDB layer. When the exception aborted the caller's transaction, all the previous work we had done on its behalf was aborted back to the start of the RDB transaction, just as one wanted. The caller even had the option to catch the exception without allowing it to abort the RDB transaction, and to retry the failed operation. #### Drawbacks of the object store The major drawback of the object store was that it was very difficult to aggregate data across ledgers: to do it you have to thaw each ledger, one at a time, and traverse its object structure looking for the data you want to aggregate. We planned that when this became important, we could have a method on the Ledger or its sub-objects which, when called, would store relevant numeric data into the right place in a conventional RDB table, where it would then be available for the usual SELECT and GROUP BY operations. The storage engine would call this whenever it wrote a modified Ledger back to the object store. The RDB tables would then be a read-only view of the parts of the data that were needed for building reports. A related problem is some kinds of data really are relational and to store them in object form is extremely inefficient. The RDB has a terrible impedance mismatch for most kinds of object-oriented programming, but not for all kinds. The main example that comes to mind is that every ledger contains a transaction log of every transaction it has ever performed: when a consumer deducts its 5479 m¢, that's a transaction, and every day each consumer adds one to the ledger. The transaction log for a large ledger with many consumers can grow rapidly. We planned from the first that this transaction data would someday move out of the ledger entirely into a single table in the RDB, access to which would be mediated by a separate object, called an Accountant. At present, the Accountant is there, but it stores the transaction data inside itself instead of in an external table. The design of the object store was greatly simplified by the fact that all the data was divided into disjoint ledgers, and that only ledgers could be stored or retrieved. A minor limitation of this design was that there was no way for an object to contain a pointer to a Ledger object, either its own or some other one. Such a pointer would have spoiled Perl's lousy garbage collection, so we weren't going to do it anyway. In practice, the few places in the code that needed to refer to another ledger just store the ledger's GUID instead and looked it up when it was needed. In fact every significant object was given its own GUID, which was then used as needed. I was surprised to find how often it was useful to have a simple, reliable identifier for every object, and how much time I had formerly spent on programming problems that would have been trivially solved if objects had had GUIDs. #### The object store was a success In all, I think the object store technique worked well and was a smart choice that went strongly against prevailing practice. I would recommend the technique for similar projects, except for the part where we wrote the object store ourselves instead of using one that had been written already. Had we tried to use an ORM backed by a relational database, I think the project would have taken at least a third longer; had we tried to use an RDB without any ORM, I think we would not have finished at all. ### Things that suck: multiple inheritance After I had been using Moose for a couple of years, including the Moonpig project, Rik asked me what I thought of it. I was lukewarm. It introduces a lot of convenience for common operations, but also hides a lot of complexity under the hood, and the complexity does not always stay well-hidden. It is very big and very slow to start up. On the whole, I said, I could take it or leave it. “Oh,” I added. “Except for Roles. Roles are awesome.” I had a long section in the talk about what is good about Roles, but I moved it out to a separate talk, so I am going to take that as a hint about what I should do here. As with my theory of dates and times, I will present only the thesis, and save the arguments for another post: 1. Object-oriented programming is centered around objects, which are encapsulated groups of related data, and around methods, which are opaque functions for operating on particular kinds of objects. 2. OOP does not mandate any particular theory of inheritance, either single or multiple, class-based or prototype based, etc., and indeed, while all OOP systems have objects and methods that are pretty much the same, each has an inheritance system all its own. 3. Over the past 30 years of OOP, many theories of inheritance have been tried, and all of them have had serious problems. 4. If there were no alternative to inheritance, we would have to struggle on with inheritance. However, Roles are a good alternative to inheritance: • Every problem solved by inheritance is solved at least as well by Roles. • Many problems not solved at all by inheritance are solved by Roles. • Many problems introduced by inheritance do not arise when using Roles. • Roles introduce some of their own problems, but none of them are as bad as the problems introduced by inheritance. 5. It's time to give up on inheritance. It was worth a try; we tried it as hard as we could for thirty years or more. It didn't work. 6. I'm going to repeat that: Inheritance doesn't work. It's time to give up on it. Moonpig doesn't use any inheritance (except that Moonpig::DateTime inherits from DateTime, which we didn't control). Every class in Moonpig is composed from Roles. This wasn't because it was our policy to avoid inheritance. It's because Roles did everything we needed, usually in simple and straightforward ways. I plan to write more extensively on this later on. This section is the end of the things I want to excoriate. Note the transition from multiple inheritance, which was a tremendous waste of everyone's time, to Roles, which in my opinion are a tremendous success, the Right Thing, and gosh if only Smalltalk-80 had gotten this right in the first place look how much trouble we all would have saved. ### Things that are GOOD: web RPC APIs Moonpig has a web API. Moonpig applications, such as the customer service dashboard, or the heartbeat job, invoke Moonpig functions through the API. The API is built using a system, developed in parallel with Moonpig, called Stick. (It was so-called because IC Group had tried before to develop a simple web API system, but none had been good enough to stick. This one, we hoped, would stick.) The basic principle of Stick is distributed routing, which allows an object to have a URI, and to delegate control of the URIs underneath it to other objects. To participate in the web API, an object must compose the Stick::Role::Routable role, which requires that it provide a _subroute method. The method is called with an array containing the path components of a URI. The _subroute method examines the array, or at least the first few elements, and decides whether it will handle the route. To refuse, it can throw an exception, or just return an undefined value, which will turn into a 404 error in the web protocol. If it does handle the path, it removes the part it handled from the array, and returns another object that will handle the rest, or, if there is nothing left, a public resource of some sort. In the former case the routing process continues, with the remaining route components passed to the _subroute method of the next object. If the route is used up, the last object in the chain is checked to make sure it composes the Stick::Role::PublicResource role. This is to prevent accidentally exposing an object in the web API when it should be private. Stick then invokes one final method on the public resource, either resource_get, resource_post, or similar. Stick collects the return value from this method, serializes it and sends it over the network as the response. So for example, suppose a ledger wants to provide access to its consumers. It might implement _subroute like this:  sub _subroute { my ($self, $route) = @_; if ($route->[0] eq "consumer") {
shift @$route; my$consumer_id = shift @$route; return$self->find_consumer( id => $consumer_id ); } else { return; # 404 } }  Then if /path/to/ledger is any URI that leads to a certain ledger, /path/to/ledger/consumer/12435 will be a valid URI for the specified ledger's consumer with ID 12345. A request to /path/to/ledger/FOOP/de/DOOP will yield a 404 error, as will a request to /path/to/ledger/consumer/98765 whenever find_consumer(id => 98765) returns undefined. A common pattern is to have a path that invokes a method on the target object. For example, suppose the ledger objects are already addressable at certain URIs, and one would like to expose in the API the ability to tell a ledger to handle a heartbeat event. In Stick, this is incredibly easy to implement:  publish heartbeat => { -http_method => 'post' } => sub { my ($self) = @_;
$self->handle_event( event('heartbeat') ); };  This creates an ordinary method, called heartbeat, which can be called in the usual way, but which is also invoked whenever an HTTP POST request arrives at the appropriate URI, the appropriate URI being anything of the form /path/to/ledger/heartbeat. The default case for publish is that the method is expected to be GET; in this case one can omit mentioning it:  publish amount_due => sub { my ($self) = @_;
…
return abs($due -$avail);
};

More complicated published methods may receive arguments; Stick takes care of deserializing them, and checking that their types are correct, before invoking the published method. This is the ledger's method for updating its contact information:

        publish _replace_contact => {
-path        => 'contact',
-http_method => 'put',
attributes   => HashRef,
} => sub {
my ($self,$arg) = @_;
my $contact = class('Contact')->new($arg->{attributes});
$self->replace_contact($contact);

return $contact; };  Although the method is named _replace_contact, is is available in the web API via a PUT request to /path/to/ledger/contact, rather than one to /path/to/ledger/_replace_contact. If the contact information supplied in the HTTP request data is accepted by class('Contact')->new, the ledger's contact is updated. (class('Contact') is a utility method that returns the name of the class that represents a contact. This is probably just the string Moonpig::Class::Contact.) In some cases the ledger has an entire family of sub-objects. For example, a ledger may have many consumers. In this case it's also equipped with a "collection" object that manages the consumers. The ledger can use the collection object as a convenient way to look up its consumers when it needs them, but the collection object also provides routing: If the ledger gets a request for a route that begins /consumers, it strips off /consumers and returns its consumer collection object, which handles further paths such as /guid/XXXX and /xid/1234 by locating and returning the appropriate consumer. The collection object is a repository for all sorts of convenient behavior. For example, if one composes the Stick::Role::Collection::Mutable role onto it, it gains support for POST requests to …/consumers/add, handled appropriately. Adding a new API method to any object is trivial, just a matter of adding a new published method. Unpublished methods are not accessible through the web API. After I wrote this talk I wished I had written a talk about Stick instead. I'm still hoping to write one and present it at YAPC in Orlando this summer. ### Things that are GOOD: Object-oriented testing Unit tests often have a lot of repeated code, to set up test instances or run the same set of checks under several different conditions. Rik's Test::Routine makes a test program into a class. The class is instantiated, and the tests are methods that are run on the test object instance. Test methods can invoke one another. The test object's attributes are available to the test methods, so they're a good place to put test data. The object's initializer can set up the required test data. Tests can easily load and run other tests, all in the usual ways. If you like OO-style programming, you'll like all the same things about building tests with Test::Routine. ### Things that are GOOD: Free software All this stuff is available for free under open licenses: (This has been a really long article. Thanks for sticking with me. Headers in the article all have named anchors, in case you want to refer someone to a particular section.) (I suppose there is a fair chance that this will wind up on Hacker News, and I know how much the kids at Hacker News love to dress up and play CEO and Scary Corporate Lawyer, and will enjoy posting dire tut-tuttings about whether my disclosure of ICG's secrets is actionable, and how reluctant they would be to hire anyone who tells such stories about his previous employers. So I may as well spoil their fun by mentioning that I received the approval of ICG's CEO before I posted this.) ### markdominus PIGS in SPACE!!! http://blog.plover.com/2013/10/05#moonpig Today I was at the Pittsburgh Perl Workshop and I gave a talk because that's my favorite part of a workshop, getting to give a talk. The talk is about Moonpig, the billing system that Rik Signes and I wrote in Perl. Actually it's about Moonpig as little as possible because I didn't think the audience would be interested in the details of the billing system. (They are very interesting, and someone who is actually interested in a billing system, or in a case study of a medium-sized software system, would enjoy a three-hour talk about the financial architecture of Moonpig. But I wasn't sure that person would be at the workshop.) Instead the talk is mostly about the interesting technical underpinnings of Moonpig. Here's the description: Moonpig is an innovative billing and accounting system that Rik Signes and I worked on between 2010 and 2012, totaling about twenty thousand lines of Perl. It was a success, and that is because Rik and I made a number of very smart decisions along the way, many of which weren't obviously smart at the time. You don't want to hear about the billing and accounting end of Moonpig, so I will discuss that as little as possible, to establish a context for the clever technical designs we made. The rest of the talk is organized as a series of horrible problems and how we avoided, parried, or mitigated them: • Times and time zones suck • Floating-point arithmetic sucks • It sucks to fix your mangled data after an automated process fails • Testing a yearlong sequence of events sucks • It sucks to have your automated test accidentally send a bunch of bogus invoices to the customers • Rounding errors suck • Relational databases usually suck • Modeling objects in the RDB really really sucks • Perl's garbage collection sucks • OO inheritance sucks Moonpig, however, does not suck. Some of the things I'll talk about will include the design of our web API server and how it played an integral role in the system, our testing strategies, and our idiotically simple (but not simply idiotic) persistent storage solution. An extended digression on our pervasive use of Moose roles will be deferred to the lightning talks session on Sunday. Much of the design is reusable, and is encapsulated in modules that have been released to CPAN or that are available on GitHub, or both. I went to sleep too late the night before, slept badly, and woke up at 6:30 and couldn't go back to sleep. I spent an hour wandering around Oakdale looking for a place that served breakfast before 8 AM, and didn't find one. Then I was in a terrible mood. But for this talk, that was just right. I snarled and cursed at all the horrible problems and by the end of the talk I felt pretty good. Slides are available here. Video may be forthcoming. Share and enjoy. [ Addendum 20131217: I wrote up the talk contents in detail and posted them here. ] [ Addendum 20131217: The video of the talk is available on YouTube, and has been since October, but I forgot to mention it. Unfortunately, the sound quality is poor; I tend to wander around a lot when I talk, and that confuses the microphone. Many thanks to Dan Wright for the video and especially for putting on PPW. ] ### hackernewsdaily Daily Hacker News for 2013-12-17 ## Tuesday 17th December 2013 ### dotaturls UK Internet archive will not be accessible via the Internet, because copyright. http://www.techdirt.com/articles/20131217/07042225586/copyright-strikes-again-no-online-access-to-uk-internet-archive.shtml http://dotat.at/:/YCJA2.html UK Internet archive will not be accessible via the Internet, because copyright. ### waxyo The Gold Factory http://gamehelp16.github.io/thegoldfactory/ http://www.waxy.org/links/archive/2013/12/index.shtml#081279 another frustratingly addictive incremental game like Candy Box ### waxyo Traffic reorganized by car color ### fixmystreet_cam Diversion signs, 17th December http://www.fixmystreet.com/report/412328 Saxon Street/Brookside/Pemberton Terrace - abandoned diversion signs, cones and sandbags littering the pavements and verges. Please remove! Nearest road to the pin placed on the map (automatically generated by Bing Maps): Pemberton Terrace, Cambridge Report on FixMyStreet ### dotaturls It's the Macroeconomy, Stupid. Neoliberal policy targets the structure not the level of wealth. http://www.dmytri.info/its-the-macroeconomy-stupid/ http://dotat.at/:/5V4R3.html It's the Macroeconomy, Stupid. Neoliberal policy targets the structure not the level of wealth. ### andyduckerlinks Pope replaces conservatives on Vatican committee that recommends bishops ### andyduckerlinks Headline of the day: Children evacuated from swimming pool after prosthetic leg mistaken for paedoph ### andyduckerlinks The social mores that prevent women from networking. ### andyduckerlinks A book I supported is getting rather good reviews on Amazon ### andyduckerlinks You’re going to make mistakes. That’s okay. Just make up for them. ### andyduckerlinks I didn't know Facebook profile cosplay/photoshop was a thing. And now I love it. ### valerie_aurora Here’s my favorite operating systems war story, what’s yours? http://blog.valerieaurora.org/2013/12/17/heres-my-favorite-operating-systems-war-story-whats-yours/ http://blog.valerieaurora.org/?p=1085 Val in her Dr Pepper days When I was working on my operating systems project in university, I stayed up all night for weeks feverishly rebooting my test machine, hoping that THIS time my interrupt handler changes worked. I survived on a diet of raisin bagels, baby carrots, and Dr Pepper, and only left the computer lab to shower and sleep for a few hours before stumbling back in and opening up EMACS again. I loved it. Julia Evans‘ blog posts about writing an operating system in Rust at Hacker School are making me miss the days when I thought everything about operating systems was magical. Julia is (a) hilarious, (b) totally honest, (c) incredibly enthusiastic about learning systems programming. (See “What does the Linux kernel even do?“, “After 5 days, my OS doesn’t crash when I press a key“, and “After 6 days, I have problems I don’t understand at all.”) I’m sure somewhere on Hacker News there is a thread getting upvoted about how Julia is (a) faking it, (b) a bad programmer, (c) really a man, but here in the real world she’s making me and a lot of other folks nostalgic for our systems programming days. Yesterday’s post about something mysteriously zeroing out everything about 12K in her binary reminded me of one of my favorite OS debugging stories. Since I’m stuck at home recovering from surgery, I can’t tell anyone it unless I write a blog post about it. VME crate (CC-BY-SA Sergio.ballestrero at en.wikipedia) In 2001, I got a job maintaining the Linux kernel for the (now defunct) Gemini subarchitecture of the PowerPC. The Gemini was an “embedded” SMP board in a giant grey metal VME cage with a custom BIOS. Getting the board in and out of the chassis required brute strength, profanity, and a certain amount of blood loss. The thing was a beast – loud and power hungry, intended for military planes and tanks where no one noticed a few extra dozen decibels. The Gemini subarchitecture had not had a maintainer or even been booted in about 6 months of kernel releases. This did not stop a particularly enthusiastic PowerPC developer from tinkering extensively with the Gemini-specific specific code, which was totally untestable without the Gemini hardware. With sinking heart, I compiled the latest kernel, tftp’d it to the VME board, and told the BIOS to boot it. It booted! Wow! What are the chances? Flushed with success, I made some minor cosmetic change and rebooted with the new kernel. Nothing, no happy printk’s scrolling down the serial console. Okay, somehow my trivial patch broke something. I booted the old binary. Still nothing. I thought for a while, made some random change, and booted again. It worked! Okay, this time I will reboot right away to make sure it is not a fluke. Reboot. Nothing. I guess it was a fluke. A few dozen reboots later, I went to lunch, came back, and tried again. Success! Reboot. Failure. Great, a non-deterministic bug – my favorite. Eventually I noticed that the longer the machine had been powered down before I tried to boot, the more likely it was to boot correctly. (I turned the VME cage off whenever possible because of the noise from the fans and the hard disks, which were those old SCSI drives that made a high-pitched whining noise that bored straight through your brain.) I used the BIOS to dump the DRAM (memory) on the machine and noticed that each time I dumped the memory, more and more bits were zeroes instead of ones. Of course I knew intellectually that DRAM loses data when you turned the power off (duh) but I never followed it through to the realization that the memory would gradually turn to zeroes as the electrons trickled out of their tiny holding pens. So I used the BIOS to zero out the section of memory where I loaded the kernel, and it booted – every time! After that, it didn’t take long to figure out that the part of the bootloader code that was supposed to zero out the kernel’s BSS section had been broken by our enthusiastic PowerPC developer. The BSS is the part of the binary that contains variables that are initialized to zero at the beginning of the program. To save space, the BSS is not usually stored as a string of zeroes in the binary image, but initialized to zero after the program is loaded but before it starts running. Obviously, it causes problems when variables that are supposed to be zero are something other than zero. I fixed the BSS zeroing code and went on to the next problem. This bug is an example of what I love about operating systems work. There’s no step-by-step algorithm to figure out what’s wrong; you can’t just turn on the debugger and step through till you see the bug. You have to understand the computer software and hardware from top to bottom to figure out what’s going wrong and fix it (and sometimes you need to understand quite a bit of electrical engineering and mathematical logic, too). If you have a favorite operating system debugging story to share, please leave a comment! Tagged: hacker school, kernel, linux, programming ### waxyo Space Jam in three.js ### waxyo Shia LaBeouf plagiarizes Dan Clowes for short film http://www.buzzfeed.com/jordanzakarin/shia-labeouf-rip-off-daniel-clowes-howard-cantour http://www.waxy.org/links/archive/2013/12/index.shtml#081277 then plagiarized his apology from Yahoo Answers ### circleid_all Name Collisions II - A Call for Research http://www.circleid.com/posts/20131217_name_collisions_ii_a_call_for_research/ This post is a heads up to all uber-geeks about a terrific research initiative to try to figure out causes and mitigation of name-collision risk. There's a$50,000 prize for the first-place paper, a $25,000 prize for the second place paper and up to five$10,000 prizes for third-place papers. That kind of money could buy a lot of toys, my peepul. And the presentation of those papers will be in London — my favorite town for curry this side of India. Interested? Read on. Here's a link to the research program — you can skip the rest of this post and get right to the Real Deal by clicking here:

www.namecollisions.net

Background and refresher course: what is the DNS name-collision problem?

Key points:

• Even now, after months of research and public discussion, I still don't know what's going to happen
• I still don't know what the impact is going to be, but in some cases it could be severe
• Others claim to know both of those things but I'm still not convinced by their arguments right now
• Thus, I still think the best thing to do is learn more
• That's why I'm so keen on this research project.

Do note that there is a strong argument raging in the DNS community about all this. There are some (myself included) who never met or even heard of the DNS purists who currently maintain that this whole problem is our fault and that none of this would have happened if we'd all configured our private networks with fully-qualified domain names right from the start.

Where were those folks in 1995 when I opened my first shrink-wrapped box of Windows NT and created the name that would ultimately become the root of a huge Active Directory network with thousands of nodes? Do you know how hard it was to get a domain name back then? The term "registrar" hadn't been invented yet. All we were trying to do is set up a shared file, print and mail server for crying out loud. The point is that there are lots of legacy networks that look like the one depicted below, some of them are going to be very hard and expensive to rename, and some of them are likely to break (perhaps catastrophically) when second level names in new gTLDs hit the root. m'Kay?

Here's my depiction the difference between a private network (with all kinds of domain names that don't route on the wider Internet) and the public Internet (with the top level names you're familiar with) back in the good old days before the arrival of 1400 new gTLDs.

Private networks, the way they may look AFTER 1400 new gTLDs get dropped into the root

The next picture shows the namespace collision problem that the research efforts should be aimed at addressing. This depiction is still endorsed by nobody, your mileage may vary, etc. etc. But you see what's happening. At some random point in the future, when a second-level name matching the name of your highly-trusted resource get delegated, there's the possibility that traffic which has consistently been going to the right place in your internal network will suddenly be routed to an unknown, untrusted destination on the worldwide Internet.

The new TLDs may unexpectedly cause traffic that you're expecting to go to your trusted internal networks (or your customer's networks) to suddenly start being routed to an untrusted external network, one that you didn't anticipate. Donald Rumsfeld might call those external networks "unknown unknowns" — something untrusted that you don't know about in advance.

Think of all the interesting and creative ways your old network could fail. Awesome to contemplate, no? But wait…

What if the person who bought that matching second-level name in a new gTLD is a bad-actor? What if they surveyed the error traffic arriving at that new gTLD and bought that second-level name ON PURPOSE, so that they could harvest that error traffic with the intention of doing harm? But wait…

What if you have old old old applications that are hard-coded to count on a consistent NXDOMAIN response from a root server. Suppose that the application gets a new response when the new gTLD gets delegated (and thus the response from the root changes from the expected NXDOMAIN to an unexpected pointer to the registry). What if the person that wrote that old old old application is long gone and the documentation is… um… sketchy? But wait…

To top it all off, with this rascal, problems may look like a gentle random rain of breakage over the next decade or so as 2nd-level names get sold. It's not going to happen on gTLD-delegation day, it's going to happen one domain at a time. Nice isolated random events sprinkled evenly across the world. Hot damn. But wait…

On the other end of the pipe, imagine the surprise when some poor unsuspecting domain-registrant lights up their shiny new domain and is greeted by a flood of email from network operators who are cranky because their networks just broke. What are THEY going to be able to do about those problems? Don't think it can happen? Check out my www.corp.com home page — those cats are BUSY. That domain gets 2,000,000 error hits A DAY. Almost all of it from Microsoft Active Directory sites.

So argue all you want. From my perch here on the sidelines it looks like life's going to get interesting when those new gTLDs start rolling into the root. And that, dear reader, is an introduction to the Name Collision problem.

Mitigation approaches.

Once upon a time, 3 or 4 months ago when I was young and stupid, I thought this might be a good way to approach this problem. I'm going to put it in this post as well, but then I'm going to tell you why it won't work. Another explanation of why we need this research and we need it now.

Start here:

If you have private networks that use new gTLDs (look on this list) best start planning for a future when those names (and any internal certificates using those names) may stop working right.

In essence, I thought the key to this puzzler was to take control of when the new gTLDs become visible to your internal network. It's still not a terrible idea, but I've added a few reasons why it won't work down at the end. Here's the scheme that I cooked up way back then.

By becoming authoritative for new gTLDs in your DNS servers now, before ICANN has delegated them, you get to watch the NXD error traffic right now rather than having to wait for messages from new registries. Here's a list of the new gTLDs to use in constructing your router configuration.

This is the part where you look at the NXD traffic and find the trouble spots. Then, with a mere wave of my hand and one single bullet point, I encourage you to fix all your networks. Maybe you've got a few hundred nodes of a distributed system all over the world that you need to touch? Shouldn't be a problem, right?

This is the Good Guy part of this approach. Of course, because we all subscribe to the One World, One Internet, Everybody Can Reach Everything credo, we will of course remember to remove the preventative blocking from our routers just as soon as possible. Right? Right?

The reasons why this won't work:

The first thing that blows my idea out of the water is that you probably don't have complete control over the DNS provider your customers use. I still think this is a pretty good idea in tightly-run corporate shops that don't permit end users to modify the configuration of their machines. But in this Bring Your Own Device world we live in, there's going to be a large population of people who configure their machines to point at DNS providers who aren't blocking the names that conflict with your private network space.

Let's assume for a minute that everything is fine in the internal network, and the corporate DNS resolver is blocking the offending names while repairs are being made (hopefully cheaply). Suppose a road warrior goes out to Starbucks and start using a laptop that's configured to point at Google's 8.8.8.8 DNS resolver. In the old days before new gTLDs, the person would fire up their computer, go to the private name, the query would fail and they would be reminded to fire up the VPN to get to those resources. Tomorrow, with a conflicting new gTLD in the root, that query might succeed, but they wouldn't be going to the right place.

Here's the second problem. My tra-la-la scheme above assumes that most mitigation will be easy, and successful. But what it it's not? What if you have a giant Active Directory tree which, by all accounts, is virtually impossible to rename without downtime? What if you have to "touch" a LOT of firmware in machines that are hard-wired to use new gTLDs. What if vendors haven't prepared fixes for the devices that are on your network looking at a new gTLD with the presumption that it won't route to the Internet (yet now it does)? Or the nightmare scenario — something breaks that has to be diagnosed and repaired in minutes?

The research project

See why we need you to look hard at this problem? Like, right now?? ICANN is already delegating these domains into the root. Here's a page that lists the ones that have already been delegated.

http://newgtlds.icann.org/en/program-status/delegated-strings

If you see one of your private network names on THIS list, you're already in the game. Hooyah! So this is moving FAST. This research should have been done years ago, long before we got to this stage. But here we are. We, the vast galaxy of network operators and administrators who don't even know this is coming, need your help. Please take a look at the NameCollisions.net site and see if you can come up with some cool ideas. I hope you win — because you'll help the rest of us a lot. I'll buy you a curry.

Written by Mike O'Connor, Curmudgeon Emeritus (formerly foo@bar.com)

More under: DNS, Domain Names, ICANN, Top-Level Domains

### circleid_all Rewiring Internet Governance: U.S. Government Will Likely Be Forced to Jettison ICANN (Part 3)

http://www.circleid.com/posts/20131217_rewiring_internet_governance_us_likely_be_forced_to_jettison_icann/

The survival thesis mentioned in Part 2 goes like this. ICANN's imaginary mandate is global. But the mind set is provincial. The latter is defensive; focused on keeping power and therefore control over internet policy. But the evidence points to policy actions that contradict policy rhetoric. Discrepancies disclose the delusion. Here's ICANN "core value" from Article 1, Section 2, Paragraph 6 (amended April, 2013): "Introducing and promoting competition in the registration of domain names where practicable and beneficial in the public interest."

On 3 January 2012 the U.S. Government (NTIA) sent the following language to ICANN in connection with new gTLDs. "How ICANN handles the new gTLD program will, for many, be a litmus test of the viability of this [multi-stakeholder] approach." Almost a year later, on 13 November 2013, ICANN published the following text on its web site: "The bottom-up, multistakeholder model of governance has addressed burgeoning policy issues in a remarkably agile and successful manner. As the Internet has taken on a larger role in the economy and society, some governments have grown increasingly uncomfortable with their perceived lack of control. [ICANN] recognizes that the single, open, global, interoperable Internet is under threat of failing due to emerging pressures on multistakeholder governance. [Help us] prevent the fragmentation of the global Internet."

Is ICANN really "remarkably agile and successful? Is Internet actually under threat from some governments? Here's reality: If Internet is tormented, the source of the threat is ICANN itself. The latter ignored warnings from four separate senate and house committees, plus the Federal Trade Commission, and NTIA, to go slow; look out for the public interest; protect the Internet; cushion small business; try a "small pilot project" for subsequent evaluation before a large scale rollout. Instead of listening to Congress ICANN took policy decisions that generated unwarranted complexity, semantic gymnastics, related legal headaches, forthcoming land rushes, inevitable cybersquatting, unnecessary trademark controversies, and avoidable predatory pricing, the latter which amounts to what economists call "asymmetrically dominated alternatives”(Ok et al, 2011). In other words, economic domination by private interests that encourage less than rational buying, selling, trading, reselling and auctioning of domain names which are ethereal goods without a physical presence and where there is no U.S. legal consensus as to whether domains names are even real property to begin with.

What should reasonable people make of this? ICANN resides in a Janus-like dream world where its delusional agility and success in dealing with "burgeoning policy issues" actually amounts to the biggest policy boondoggle in the history of the Internet; and where real world evidence shows itself in the double-talking promotions that are published on the web sites of ICANN's corporate beneficiaries.

Policy clearly reduces to Lowi's "who gets what"; but the problem runs deeper. The real politik behind ICANN's predicament stems from shifting political assessments about whether the rest of the world concurs with ICANN's ideology of the empire of business which blends into, and therefore corrupts, policy making. If, as ICANN claims, a global and interoperable Internet is failing because of "emerging pressures", the stressors were created by ICANN itself.

Arguably the source stems from ICANN's "cognitive toolkit" (Kurbalija, 5th edition, Diplo Foundation) which in governance terms defies common sense. ICANN has done some things reasonably well, meaning security and steadiness of the root zone which in a technical sense supports internet governance. However, in ICANN's case stability and security are subordinate to its driving ambition to be king of the world insofar as "ruling the root" The result is "bad equilibrium"(Fukuyama, 2013) and policy perversity.

The practical effect of subordinating policy decisions to internal ideology is that bottom-up governance, however well meaning, contradicts political reality. When push comes to shove politics is about power. ICANN's narrative coupled with policy blunders that ignore political sensibility anticipate ICANN's sooner rather than later demise. I will explain why that's a distinct possibility.

The ICANN Narrative

The Affirmation of Commitments by U.S. Commerce Department and ICANN on 30 September 2009 enfeebled U.S. Government control over ICANN. The government handed ICANN a long leash with two probable outcomes: ICANN could hang itself or conform to the letter of the agreement by devising progressive policies that met ICANN commitments on the one hand, but not at the expense of the public interest as distinct from the interests of economic elites.

To date ICANN has not gone to the gallows, but perverse policy choices have dragged Internet into a zero sum twilight guided entirely by laissez faire mania. This calls into question the competence of ICANN's managers. Its theory of governance, if it ever had one, stems from political artlessness and historical amnesia. ICANN seems blissfully unaware that there is nothing of substance in the 2009 accord that severs it fully and finally from its master.

Indeed, the U.S. Government's DNS "issues for congress" report (13 November 2013) states the following: "Congress has an impact on the issue of Internet governance, both via its oversight of NTIA and the DNS, and through its actions in other and more specific areas of Internet policymaking. For example, Congress continues to oversee and evaluate NTIA's strategy of supporting ICANN's multistakeholder model while opposing arguments for increased intergovernmental control".

The text also states: "One of NTIA's arguments for increasing government influence over ICANN policymaking (via the GAC) is that if governments feel their interests are not adequately addressed within the ICANN process, this perception will give support to the argument that the DNS and the Internet should be governed through a more formal intergovernmental mechanism." In other words, for the moment NTIA is the congressional proxy for oversight. But the oversight mandate will likely change because ICANN's policy model flatly contradicts the government's political objectives vis-á-vis Internet.

Blunders and Blinders

Froomkin (2011) points out that when government ceded partial control over ICANN, the commitments affirmation introduced a variety of control variables that ICANN could introduce, anyone of which could alter the political calculus. But, severance from NTIA oversight wasn't one of them. Granted, ICANN has exercised "more authority and more independence" (Froomkin). But its authority and perceived legitimacy has been undermined everywhere.

Froomkin notes that the 2009 accord between DOC and thus NTIA and ICANN allowed the latter to independently devise and impose by contract new rules and regulations. But such did not lessen the government's "unique influence" over ICANN. Specifically, section 9.3 of the pact specified that ICANN would promote competition, consumer trust and consumer choice, and that ICANN's contemplation of an expanded top-level (gTLD) name space also meant that in addition to security, resiliency and stability, ICANN (via GAC) would address malicious abuse issues, sovereignty concerns, and consumer rights protections.

That ICANN's policy actions contradict its implied and specific promises-of-performance is readily discerned. Consider the following: 1) ICANN's version of the "public interest" amounts to a world awash with privately run gTLDs, many of them from brand name entities that signal information silos controlled by big corporations. 2) ICANN's cognitive toolkit connected to an appetite for money that obviated applications from not-for-profits focused, for example, on world-wide hunger, in-country genocide, unspeakable poverty, rampant disease, or child mortality. 3) ICANN's clueless oblivion to the perceived cultural bias which obtains when the internet platform delivers technical virtuosity at the expensive of foreign political and cultural sensitivities.

All this speaks to the internet fragmentation that worries ICANN's board, and with good reason. Its own policy blunders beg the larger question of the probable impacts of over extended ambition when politics on a large canvas confronts U.S. Congress and thus NTIA over root zone stability seriously impaired by incompetence. Sooner rather than later peer institutions may begin avoiding discredited ICANN. Foreign governments will likely begin demanding accountability. Congress might be compelled to discard ICANN which was precisely the outcome it sought to avoid. What is it about ICANN that warrants this?

The Global Narrative

An example of ICANN's cognitive deficit in the global sand box derives from Walter Russell Mead who edits The American Interest blog. Mead clearly believes that U.S. hegemony, meaning the extent to which others are willing to accommodate U.S. policy interests, is in decline. The reason is provided by Francis Fukuyama, who is Mead's online collaborator.

The "End of History and the Last Man" thesis broadly defines Fukuyama. But, in a recent (8 December 2013) essay, Fukuyama speaks to "the decay of American political institutions". His explanation resonates in connection with the oversight narratives of both ICANN and NTIA. According to Fukuyama, governance methods in the U.S. subordinate accountability. This clearly runs contrary to U.S. policy intentions. There are reasons for this, one of which I previously referred to as a "dysfunctional governance predicament that somehow skews oversight." Fukuyama also mentions Lowi's "interest group liberalism" meaning "who gets what" as a "pluralist" political reckoning derived from mid-20th century thinking which has somehow transformed the narrative.

Fukuyama states: "The [theory was that] a cacophony of interest groups would collectively interact to produce a public interest, just as competition in a free market would provide public benefit through individuals following their narrow self-interest. Further, there was no justification for government to regulate this process, since there was no higher ground that could define a "public interest" standing above the narrow concerns of interest groups.... Alas, 'interest groups' and 'private associations' are two ways of describing the same basic phenomenon. So how do we reconcile these diametrically opposed narratives, the first claiming that interest groups corrupt democracy and harm economic growth, and the second asserting that they are necessary for healthy democracy?"

The above simplifies Fukuyama. But the main point is this. "The United States is trapped in a bad equilibrium. Because Americans historically distrust the government, they aren't typically willing to delegate authority to it. Instead, as we have seen, Congress mandates complex rules that reduce government autonomy and render decisions slow and expensive. The government then performs poorly, which perversely confirms the original distrust of government."

As mentioned above, ICANN's policy obstinance is the reverse mirror image of what the DOC, NTIA and Congress intend to elude. Fukuyama's "disequilibrium" thesis reflects ICANN. Shapiro's "delegation dilemma" about oversight quarrels pinpoints the source. Bred in the bone ideology that reflects a distrust of government ironically diminishes internet autonomy through wrong-headed policies that invite information oligopolies in the name of "openness". ICANN's public persona is mindful of A.M. Klein's "mad poet who stares at a mirror all day long, as if to recognize himself".

The evolution of Internet as a technical and social phenomenon emerged from antecedent occurrences, among them technical know-how and of course research and development money via DARPA and others. But the nexus that joins engineering dexterity to stability has not translated into competent governance. If it were otherwise, ICANN would have reckoned with political reality. As Kahler (2009) points out, contemporary guidance is about "networked politics" and other "new forms of governance" where frameworks like bottom-up stake holding hitherto regarded as consensual (and thus sacrosanct) are disclosed for what they are: political vehicles that subordinate policy to conflicts over hierarchy and status. The above 'intimations', large and small, demonstrate that a mature institution with competent management would have avoided the "asymmetrically dominated alternatives" that have eradicated ICANN's credibility and eroded confidence that the U.S. has a policy handle on reasonable internet supervision.

Summary

In Part 3 I have outlined the broader political environment which ought to have led ICANN's cognitive toolkit in the direction of adding new gTLDs carefully, gradually and mindful of perceived bias. In their stead, ICANN chose a radical, unwarranted and unprecedented expansion trajectory which leads to serious governance disequilibrium.

In Part 2 the argument exposed ICANN's trajectory which also reflects economic fabulism. As critics have feared, internet governance has become a smoke screen that reminds one of what McChesney (2013) refers to as "the great conflict between openness and a closed system of corporate profitability, the forces of capital have triumphed whenever an issue matter to them. The Internet has been subjected to the capital-accumulation process, which has a clear logic of its own, inimical to much of the democratic potential of digital communication."

The "great conflict" incorporates ICANN's provincialism and thus its defensive posture. For all its imagined power ICANN governance amounts to weasel worded rhetoric coupled with never ending discussions, expensive conferences, and meetings whether required or not, nearly everywhere in the world, all of which in the end game dwindle to kowtowing to big money interests. Those "emerging pressures on multistakeholder governance" lamented by ICANN's board are real. The board created them.

In Part 4 I will continue looking at ICANN's policy actions to try and uncover the roots of the conflict between that which professes openness, and ICANN's governance model which amounts to private sector hegemony instead of reasonable economic policies that serve the entire internet community.

Written by Frederick Harris

More under: ICANN, Internet Governance

The GMC are focused mainly on the narrow issue of an individual doctor’s competence when seeing individual patients. But there are broader issues that have an equally important impact on patient care and public trust: failure to publish clinical trials, failure to participate in research, and imperfectly declared conflicts of interest, for example. The Health […]

### bad_science The ethical standards of the “Ethical Standards in Health and Life Sciences Group”

Still catching up on posting things from this year. Here’s a piece I wrote in the BMJ with medical student colleagues about an extraordinary, influential, and rather depressing organisation called the “Ethical Standards in Health and Life Sciences Group”. This was a committee of the great and good in medicine, co-chaired by Sir Richard Thompson […]

### andyduckerlinks How Important Is A Bee?

http://what-if.xkcd.com/76/

At what point in human history were there too many (English) books to be able to read them all in one lifetime?

Gregory Willmot

This is a complicated question. Getting accurate counts of the number of extant books at different times in history is very hard bordering on impossible. For example, when the Library of Alexandria burned, a lot of writing was lost,[1]On the other hand, a lot of Egyptian readers were probably excited to get out of overdue book fines. but how much writing was lost is hard to pin down. Some estimates range from 40,000 books to 532,800 scrolls, and other writers say that all those numbers are implausible.

Researchers Eltjo Buringh and Jan Luiten van Zanden used historical book catalogs to put together statistics on the number of books (or manuscripts) published annually per region. By their figures, the rate of publication in the British Isles probably passed one manuscript per day in around the year 1075 CE.

Most of the manuscripts published in 1075 weren't in English, or even the variants of English common at the time. In 1075, literature was typically written in some form of Latin or French, even in areas where Old English was commonly spoken on the street.

The Canterbury Tales (written in the late 1300s) were part of a move toward vernacular English as a literary language, although they're not exactly readable to a modern eye:

Wepyng and waylyng, care and oother sorwe
I knowe ynogh, on even and a-morwe,'
Quod the Marchant, 'and so doon oother mo
That wedded been.

Even if we know how many manuscripts were published per year, in order to answer Gregory's question, we still need to know how long it takes to read a manuscript.

Rather than trying to figure out how long all the lost books and codices are, we can step back and take a longer view of things.

Writing speed

Tolkien wrote Lord of the Rings in 11 years, which means that he wrote at an average pace of 125 words per day, or less than 0.085 word per minute. Harper Lee wrote the 100,000-word To Kill a Mockingbird in two and a half years, for an average of 100 words per day, or 0.075 words per minute. Since To Kill a Mockingbird is her only published book, her lifetime average is 0.002 words per minute, or about three words per day.

Some writers are substantially faster. In the preface to Opus 200, the prolific writer Isaac Asimov estimated that he had published about 15,000,000 words between age 30 and 50. His average over his writing career might have been around 1 word per minute, and at times he was averaging writing several thousand words per day. (Over his entire life, his average dips as low as 0.5 words per minute.) Some pulp writers have even higher averages.

It's reasonable to assume historical writers had a similar range of speeds. You might point out that typing on a keyboard is more than twice as fast as writing a manuscript in longhand. But typing speed isn't a writer's bottleneck. After all, at a typing speed of 70 words per minute, it should only take 24 hours to type out To Kill a Mockingbird.

The reason typing and writing speeds are so different is that the bottleneck isn't typing speed—it's how quickly our brains can organize, produce, and edit stories. This "storytelling speed" has probably changed much less than our physical writing speed has.

This gives us a much better way to estimate when the number of books became too large to read.

The average person can read at 200-300 words per minute. If the average living writer, over their entire lifetime, falls somewhere between Isaac Asimov and Harper Lee, they might produce 0.05 words per minute over their entire lifetime.

If you were to read for 16 hours a day at 300 words per minute,[4]For an average of 200 words per minute. you could keep up with a world containing an average population of 100,000 living Harper Lees or 400 living Isaac Asimovs.

If we estimate that during their active periods, writers are producing somewhere between 0.1 and 1 word per minute, then one dedicated reader might be able to keep up with a population of about 500 or 1,000 active writers. The answer to Gregory's question—the date at which there were too many English books to read in a lifetime—happened sometime before the population of active English writers reached a few hundred. At that point, catching up became impossible.

The magazine Seed estimates that the total number of authors reached this point around the year 1500 and has continued rising rapidly ever since. The number of active English writers crossed this threshold shortly thereafter, around the time of Shakespeare, and the total number of books in English probably passed the lifetime reading limit sometime in the late 1500s.

On the other hand, how many of them would you want to read? If you go to goodreads.com/book/random, you can see a semi-random sample of what you'd be reading. Here's what came up for me:

• • School Decentralization in the Context of Globalizing Governance: International Comparison of Grassroots Responses, by Holger Daun
• • Powołanie (Dragon Age #2), by David Gaider
• • An Introduction to Vegetation Analysis: Principles, Practice and Interpretation, by David R. Causton
• • AACN Essentials of Critical-Care Nursing Pocket Handbook, by Marianne Chulay
• • National righteousness and national sin: the substance of a discourse delivered in the Presbyterian church of South Salem, Westchester co., N.Y., November 20, 1856, by Aaron Ladner Lindsley
• • Phantom of the Auditorium (Goosebumps #24), by R. L. Stine
• • High Court #153; Case Summaries on Debtors and Creditors-Keyed to Warren, by Dana L. Blatt
• • Suddenly No More Time, by Emil Gaverluk

So far, I've read ... the Goosebumps book.

To make it through the rest, I might need to recruit some help.

### ucsnews Wed 18 Dec, 17:30 - 17:40: Jackdaw Services interrupted

http://ucsnews.csx.cam.ac.uk/articles/2013/12/17/wed-17-nov-17-30-17-40-jackdaw-services-interrupted

Jackdaw Database Services (including IP registration database) will be interrupted between 17:30 and 17:40 on Wednesday 18th December for a system library patch.

### shape_of_code Variable naming based on lengths of existing variable names

http://shape-of-code.coding-guidelines.com/2013/12/17/variable-naming-based-on-lengths-of-existing-variable-names/

http://shape-of-code.coding-guidelines.com/?p=1561

Over the years I have spent a lot of time studying variable names and I sometimes encounter significant disbelief when explaining the more unusual developer variable name selection algorithms.

The following explanation from Rasmus Lerdorf, of PHP fame, provides a useful citable source for a variant on a common theme (i.e., name length).

“… Back when PHP had less than 100 functions and the function hashing mechanism was strlen(). In order to get a nice hash distribution of function names across the various function name lengths names were picked specifically to make them fit into a specific length bucket. This was circa late 1994 when PHP was a tool just for my own personal use and I wasn’t too worried about not being able to remember the few function names.”

Pointers to other admissions of youthful folly welcome.

### dotaturls We need to talk about TED.

We need to talk about TED.

### metoffice_news River flooding could increase under climate change

http://www.metoffice.gov.uk/news/releases/archive/2013/river-flooding-increase

17 December 2013 - River floods are likely to occur more frequently in many parts of the world as climate change intensifies heavy rainfall, according to a new study.

### bruce_schneier Security Vulnerabilities of Legacy Code

https://www.schneier.com/blog/archives/2013/12/security_vulner_3.html

An interesting research paper documents a "honeymoon effect" when it comes to software and vulnerabilities: attackers are more likely to find vulnerabilities in older and more familiar code. It's a few years old, but I haven't seen it before now. The paper is by Sandy Clark, Stefan Frei, Matt Blaze, and Jonathan Smith: "Familiarity Breeds Contempt: The Honeymoon Effect and the Role of Legacy Code in Zero-Day Vulnerabilities," Annual Computer Security Applications Conference 2010.

Abstract: Work on security vulnerabilities in software has primarily focused on three points in the software life-cycle: (1) finding and removing software defects, (2) patching or hardening software after vulnerabilities have been discovered, and (3) measuring the rate of vulnerability exploitation. This paper examines an earlier period in the software vulnerability life-cycle, starting from the release date of a version through to the disclosure of the fourth vulnerability, with a particular focus on the time from release until the very first disclosed vulnerability.

Analysis of software vulnerability data, including up to a decade of data for several versions of the most popular operating systems, server applications and user applications (both open and closed source), shows that properties extrinsic to the software play a much greater role in the rate of vulnerability discovery than do intrinsic properties such as software quality. This leads us to the observation that (at least in the first phase of a product's existence), software vulnerabilities have different properties from software defects.

We show that the length of the period after the release of a software product (or version) and before the discovery of the first vulnerability (the 'Honeymoon' period) is primarily a function of familiarity with the system. In addition, we demonstrate that legacy code resulting from code re-use is a major contributor to both the rate of vulnerability discovery and the numbers of vulnerabilities found; this has significant implications for software engineering principles and practice.

Previous 50