Recently I was learning ANTLR, trying to decide whether this was a right technology to pursue a particular project which involved parsing a preexisting message format. After a week of a spike, we decided that it was worth pursuing and started on earnest on the grammar for our project. However a week into this process, I had an epiphany … I was doing some things wrong with the ANTLR grammar which were now slowing progress in adding the new characteristics it needed to be complete. Many developers know this feeling; the features of my grammar that I had built over the first week were naive and now hampering it from expanding into the new requirements. I took it on myself to kill the entire grammar and start again. It took less than a day and half to replicate that week’s worth of work (i.e. pass the test suite which had built up around it).  I’ve done this before; scrap the first attempt at building a domain and try again. Here my domain was the same (it was after all defined in both a standards specification and in the many hundreds of thousands of sample messages we captured from an existing system), but its implementation needed refinement.

So I think that the rule about throwing away spikes can in fact be made a general axiom of programming:

When you are learning a new technology, make sure you throw away the first thing you build that works – to avoid accumulating your mistakes.

Thanks to Robert for the important qualifier “that works”.

N.B. my views about system rewrites have not changed regardless.

The Project Cartoon - how the customer explained it, what we built them & what the customer actually needed

It’s from a site called The Project Cartoon. What I didn’t know was the site allows you to use the little cartoons to make your own version. You can caption and re-order the panels. For your amusement, and perhaps your edification, I present “The Rewrite Will Be Ready Shortly”:

The existing software is ugly and poorly written. We will have to rewrite it if you want any new features.

[Certain reformists say] The country has this one chance, the idealist maintains, to sweep away our inhumane, wasteful patchwork system and replace it with something new and more rational. So we should prepare for a bold overhaul, just as every other Western democracy has. True reform requires transformation at a stroke. But is this really the way it has occurred in other countries? The answer is no. And the reality of how health reform has come about elsewhere is both surprising and instructive.

He goes at lengths to explain how the health care that the rest of the developed world (e.g. Australia) has found itself with, are the results of ad-hoc changes to existing systems, not large-scale rewrites. As a result, every country has a “path-dependent” health care system. For example, Britain’s NHS evolved in 1945 from wartime requirements and infrastructure. After the way, France’s infrastructure was largely destroyed, so they used (that is, modified and extended) an existing system of union organised health care insurance paid for through payroll taxes. And so on. Everyone’s system is different because their initial conditions where different, and solutions have to respect what’s there, and not just blithely sweep it all aside.

This possibly could give us a general principle with regards to all types of reform - you have to work with what you’ve got. That includes software.

We’ve known for some time that the big rewrite nearly always ends in failure (although not everyone agrees). Especially when the specification for the new system is “make it do what the old system does”. Why are you re-writing it then?! (Chad Fowler has an excellent article about this).

It’s an excellent general principle that software systems should to evolve  incrementally over time. Has anyone else worked in an environment where they’ve spent 12, 18, 24 months squirrelled away somewhere vigorously rewriting an entire system while the existing system is still being maintained and improved? Not so long ago I worked on a system where we spent 4 months re-writing just a small section of a piece of software – delivered in two steps – and it was bad enough as it was. But a year? It just has to be disastrous.

What’s important is your initial set of conditions, which shape the scope of what is actually possible to do – and importantly, how you’d go about doing it. Your future system is “path dependent” on the existing one, and that was dependant on what was there before.  Software developers frequently chafe at these limits, and instead form a desire to just chuck it out and start over. But you are dooming yourself to failure if you do that.

Part of our professional practices  should be the delivery of incremental design to systems. Instead of that 4 month re-write of a small section of functionality could we have delivered it in even smaller chunks? My feeling is quite probably, yes – and it would have been a lot less stressful at the time too. Always work with what you’ve got, and change -and improve – it one small piece at a time.