Recently I was learning ANTLR, trying to decide whether this was a right technology to pursue a particular project which involved parsing a preexisting message format. After a week of a spike, we decided that it was worth pursuing and started on earnest on the grammar for our project. However a week into this process, I had an epiphany … I was doing some things wrong with the ANTLR grammar which were now slowing progress in adding the new characteristics it needed to be complete. Many developers know this feeling; the features of my grammar that I had built over the first week were naive and now hampering it from expanding into the new requirements. I took it on myself to kill the entire grammar and start again. It took less than a day and half to replicate that week’s worth of work (i.e. pass the test suite which had built up around it).  I’ve done this before; scrap the first attempt at building a domain and try again. Here my domain was the same (it was after all defined in both a standards specification and in the many hundreds of thousands of sample messages we captured from an existing system), but its implementation needed refinement.

So I think that the rule about throwing away spikes can in fact be made a general axiom of programming:

When you are learning a new technology, make sure you throw away the first thing you build that works – to avoid accumulating your mistakes.

Thanks to Robert for the important qualifier “that works”.

N.B. my views about system rewrites have not changed regardless.

First up, the easy way out – take a day off, go for a walk in the park, have a nice dinner, get a good nights sleep. You should always work to “sustainable progress” and yes this means every hour you work over 40 hours a week means your quality is steadily decreasing until some point where every extra hour you put will take two to fix it later (in other words, negative impact). Even just pair programming will help prevent this in the first place as long at least one of you isn’t operating on half the sleep they need. But ultimately, all of this just blames the individual, when really poor code quality and technical debt are much more a function of the management system, methodology, total sum of the choices by every on the team, etc. (For a start, why are you up late in the first place? Was that your choice, or an outside imposed deadline?).

As James Adam and “iSteve” point out in the comments – technical debt is different from just “buggy software” and in my experience it’s usually function of management or other types of systemic choices (and “systemic” includes you, the programmer as well everyone else). What it really does is trade easy change now for even harder change later. It wants that sweet sweet sugar hit straight away but forgets about the “sugar coma” that comes a few minutes afterwards. It defers “paying the piper” until tomorrow for that lovely tune today. James Adam rightly says in his comment this can be a valid choice, but all of the team have to be fully aware of the consequences of that choice. One of which is typically that the longer  you defer the payment, the more debt you’ll accumulate, until all you are doing is paying off the interest and not getting to the principal.

As “iSteve” says, management (and maybe “management” might mean just you telling yourself this in your head) always promises to “fix it later” and never does allow that. “Just make the hack” and gloss over the problem is a mantra that we’ve all heard before. Where it gets painful is where every change is now taking twice as long as it should because of the accumulation of hacks. Every single one one of us has been there before, and are probably in various levels of it right now.

In my view this is the much more insidious circumstance because it is systemic.  After all the solution to a bad day’s work because I was too tired to think properly is to simply get a good night’s sleep, roll back yesterday’s coding, and do it again today. All I’ve done is lost a day’s work.  However, an organisation is likely to have accumulated months if not years of organisational cruft that makes changing their context that much harder than just rolling back yesterday’s poor code and trying again – it requires commitment to change from everyone who counts in the first instance.

On a technical mailing list I’m on, a developer asked for help in regard to a class he had to modify. There’s no need for the full gory detail here; suffice to say it’s main feature was that it had a very long if-then-else method that had nearly 20 steps testing multiple boolean statuses strung out after each other. The question was very simply; “how to refactor this?”. The suggestions were all good; ranging from using polymorphism to implementing a rules engine.

It’s not the solutions, or even the code that I want to discuss. Near the end of the thread, the original poster came back and said he’d discovered that (to paraphrase) ‘the feature change needs to go to the QA process today’ and that the refactoring couldn’t be done: the two-line hack would have to suffice.

I feel for the poor bugger, I really do, I’ve been in that situation myself. Maybe it’s just my bombastic, iconoclastic nature that makes me push back at these situations and tell people what fools they are for not valuing the craftsmanship of the very craftsmen they’ve hired specifically to build their software solutions. We need to start to insist that craftsmanship is valued and not just dismissed with an airy wave of the hand by management anymore. I’m fully aware this frequently makes me unpopular amongst those who don’t like to told the hard, brutal, truth. But I’ve given up caring. The truth it is.

In this spirit, I’m going to put on my “I’m as mad as hell and I’m not gonna take this anymore” hat and say the following. The reason that class in that state is because “changes to this feature are going to need to be in a build for QA today”. What needs to be done about these situations?

Just Say No.

Developers – those aspiring to be craftsmen anyway, need to be able to say it just is not possible for this feature to be changed today. Tell the management they have maxed out the credit card and the bailiffs are banging at the door and taking the silverware away as payment.

For a start, who decided this feature had to be in the QA build at some arbitrary cut-off date anyway? Someone who had never seen that class, that’s who. That “chicken” committed developer – the “pig” – to deliver the feature by some arbitrary cut off date and didn’t know the details. This is estimation by fiat. In any half-way decent process that respects it’s workers as people and not jut bit-flippers, no-one has the right to do this; it’s massively broken process. Chickens are not allowed to decide on estimates: only pigs are. The only way these things change if someone stands up and says “NO” to such broken process.

I don’t mean to say the developers can just go off with the fairies and gold-plate every piece of code they touch. But it has to be at least well-made. Obviously this particular process isn’t an agile environment – as it includes a “QA” stage with arbitrary cut off date at which point hte code “gets thrown over the wall”. And the company has management (I assume it’s management) running around and setting arbitrary delivery dates for unknown work.

Seriously, process improvement has to start somewhere. Be the change you seek. Stop tolerating poor code. If the process you work in demands you to tolerate it, change the process as best you can.

Link: http://www.viddler.com/explore/sergiopereira/videos/7/.

[Certain reformists say] The country has this one chance, the idealist maintains, to sweep away our inhumane, wasteful patchwork system and replace it with something new and more rational. So we should prepare for a bold overhaul, just as every other Western democracy has. True reform requires transformation at a stroke. But is this really the way it has occurred in other countries? The answer is no. And the reality of how health reform has come about elsewhere is both surprising and instructive.

He goes at lengths to explain how the health care that the rest of the developed world (e.g. Australia) has found itself with, are the results of ad-hoc changes to existing systems, not large-scale rewrites. As a result, every country has a “path-dependent” health care system. For example, Britain’s NHS evolved in 1945 from wartime requirements and infrastructure. After the way, France’s infrastructure was largely destroyed, so they used (that is, modified and extended) an existing system of union organised health care insurance paid for through payroll taxes. And so on. Everyone’s system is different because their initial conditions where different, and solutions have to respect what’s there, and not just blithely sweep it all aside.

This possibly could give us a general principle with regards to all types of reform - you have to work with what you’ve got. That includes software.

We’ve known for some time that the big rewrite nearly always ends in failure (although not everyone agrees). Especially when the specification for the new system is “make it do what the old system does”. Why are you re-writing it then?! (Chad Fowler has an excellent article about this).

It’s an excellent general principle that software systems should to evolve  incrementally over time. Has anyone else worked in an environment where they’ve spent 12, 18, 24 months squirrelled away somewhere vigorously rewriting an entire system while the existing system is still being maintained and improved? Not so long ago I worked on a system where we spent 4 months re-writing just a small section of a piece of software – delivered in two steps – and it was bad enough as it was. But a year? It just has to be disastrous.

What’s important is your initial set of conditions, which shape the scope of what is actually possible to do – and importantly, how you’d go about doing it. Your future system is “path dependent” on the existing one, and that was dependant on what was there before.  Software developers frequently chafe at these limits, and instead form a desire to just chuck it out and start over. But you are dooming yourself to failure if you do that.

Part of our professional practices  should be the delivery of incremental design to systems. Instead of that 4 month re-write of a small section of functionality could we have delivered it in even smaller chunks? My feeling is quite probably, yes – and it would have been a lot less stressful at the time too. Always work with what you’ve got, and change -and improve – it one small piece at a time.

I have found too many code bases over the years, with bolted-on, after the fact tests, plagued by spaghetti code and strongly coupled components to be that naive about unit tests. Sure people pay lip-service to testing, but I think many programmers haven’t yet found the value of test driven design in their professional practice. I know enough that I come across this situation frequently. “Oh I’ll do the tests later”, “I don’t have time to write tests, I have to write code!”, etc. Heck, I am sometimes still guilty of it myself. Here’s some rolled-gold reasons Bain gives to pay particular attention to the discipline of unit testing in a test-first scenario:

• It reduces bugging time.
• It makes the development process more predictable – finding bugs often exhibits non-linear unpredictability.
• It encourages appropriate coupling, ensures cohesiveness, and reduces redundancy.
• It makes you think about your code in more depth.

I must say, even when confronted with an existing code base that has little to no unit testing, it’s always advantageous to start writing the tests before you start fooling around too much with the code base. It will help you discover the design weaknesses.

Another point I have to make about unit testing, is that whenever I need to evaluate a new technology,  I now use the ability to utilise test-driven methodologies as an important criteria. Especially if that technology is supposed to be a productivity-booster for programmers. If I can’t write the tests first, then I just know the tool, or technology, is going to produce less than optimum results – how can it make any programmer more productive if it removes the ability to write code until the tests pass? And if I can’t write any tests – that is it has no test framework either built-in or freely available, then I just know the tool sucks, and is to be actively avoided and recommended against. I’m especially suspicious of technology such BPEL frameworks and other frameworks designed to provide “programmer-less programming” (but that will have to wait for another day).

Emergent Design: The Evolutionary Nature of Professional Software Development by Scott L. Bain

Amazon

[E]mergent design works by refactoring and enhancing code, due to the changes, bugs, and extensions that have to accommodate, while paying close attention to these principles of coding. (152)

In order to deal with the idea idea of evolutionary software development with a lean or agile methodology you have to first grasp what software development is and isn’t. As a consequence of this, the first part of the book deals with the idea of “Software as a profession”. A lot of this is very thought-provoking; I’ve never really thought about what a ‘profession’ actually is before.

Software development has not been around for very long, in the grand scheme of things. Other, similar complex endeavors (medicine, the law, architecture and so on) have been around for hundreds, even thousands, of years, and in that time a whole set of standards, practices, and general wisdom has been captured and handed down from generation to generation. This has helped to increase the rate of predictable success for each new batch of doctors, lawyers, and builders, and in each case has led to the formation of an organism we call the profession.

Although he admits that the choice of development as profession is somewhat arbitrary, it does serve to highlight that what we do as developers is like no other activity. In other words, software development is a unique activity, and there’s no point in organising it using the methods of electrical or mechanical engineering, research science, medicine, or house building. Each of these professions is different from software development and different from each other. Software developers need our own ways of organising our field of knowledge, some of these are currently absent.

Features that need to be covered in software development as a profession:

• Specialised language. This would be at a high level, not a low abstraction level like at the code itself. Bain maintains that the design patterns ‘movement’ is a move towards the development of a high-level specialised language that matures our field towards professionalism.

• A clear entry path. Something that’s currently clearly missing, although I’d contend that perhaps a computer science or information technology degree and employment as a graduate programmer, at least in this country (Australia), is something that somewhat fills that role.

• Peer-review. Peer review is fairly ad-hoc and chaotic, despite attempts at things like user groups and websites.

• Standards and practices. Unfortunately there is little agreement amongst the broad software development community about ‘what is right’. However, I would argue that the Agile movement is at least a move in the right direction.

The sorts of things that need to be discussed and understood by a profession of evolutionary software development might number:

• Qualities. How do we know when something is good? How do we choose the best thing? What is meant by best?

• Principles. The fundamentals of good software. “Principles promise better results in terms of the qualities we will emphasize”.

• Practices. The things you do in your daily coding activities. Practices must be easily taught because they “are truly valuable when they are shared and promoted to all the developers on a team”.

• Disciplines. These are similar to practices, but on a larger scale. Things you should do that have a basic and profound value to what you do as a developer. For example, unit testing, refactoring, test-driven development.

• Patterns. What worked before; “the set of interrelated points of wisdom that reflect what [the profession knows] about certain situations … that we find ourselves in again and again”. I would add, and the common, standarised jargon that we use to describe and discuss these patterns.

• Processes. How does development work? How do we know what to build? How do we know when we’ve built it? How do we correct the situation when we discover something’s wrong? How we we discover that something is wrong? Et cetera.

In this day and age I can still find developers, usually working in crippled environments, who won’t even agree on the simple principles of maintaining ‘good’ code practices, for any version of good. Whatever’s there, they’ll accept. Even when faced with seriously broken design, and a mandate to effect change, a cascading series of ever-fragile hacks will be accepted, even if it takes more time to produce, or modify, the hack. Code will never be deleted, even long after obsolescence. It is left remaining, “just in case”, and as if there was no version control system.

As a consultant, I have no real choice but to work with these programmers. I can only try to encourage them to follow a right path. I was recently told I was a perfectionist, simply for demanding any level of unit testing greater than 0.