Our system has generic components which “pass off” specific knowledge of specific data sets to specific processing components, where there are not a fixed set of these specific components available to all the generic components. In classic OO design this is achieved with interfaces and many design patterns like the Visitor Pattern. In our design we’ve replaced much of the complexity of the interface with REST-based architecture. Our application has become far more modular (despite its run-time complexity). More importantly, we are at the the point where we can easily start delivering the advanced features that the business has now specified for us without introducing a massive bloat-o-burger one-size-fits-all set of “common data definitions” that are the union of all possible sub-system functionalities.  Each sub-system can have its specific features required for that data set and still satisfy the handful of generic actions it is required to support. Each of these specific features can be developed in isolation. It’s a big win for us.

In Eclipse’s defence, Intellij IDEA and Netbeans also exhibit identical broken functionality.

Recently I was learning ANTLR, trying to decide whether this was a right technology to pursue a particular project which involved parsing a preexisting message format. After a week of a spike, we decided that it was worth pursuing and started on earnest on the grammar for our project. However a week into this process, I had an epiphany … I was doing some things wrong with the ANTLR grammar which were now slowing progress in adding the new characteristics it needed to be complete. Many developers know this feeling; the features of my grammar that I had built over the first week were naive and now hampering it from expanding into the new requirements. I took it on myself to kill the entire grammar and start again. It took less than a day and half to replicate that week’s worth of work (i.e. pass the test suite which had built up around it).  I’ve done this before; scrap the first attempt at building a domain and try again. Here my domain was the same (it was after all defined in both a standards specification and in the many hundreds of thousands of sample messages we captured from an existing system), but its implementation needed refinement.

So I think that the rule about throwing away spikes can in fact be made a general axiom of programming:

When you are learning a new technology, make sure you throw away the first thing you build that works – to avoid accumulating your mistakes.

Thanks to Robert for the important qualifier “that works”.

N.B. my views about system rewrites have not changed regardless.

First up, the easy way out – take a day off, go for a walk in the park, have a nice dinner, get a good nights sleep. You should always work to “sustainable progress” and yes this means every hour you work over 40 hours a week means your quality is steadily decreasing until some point where every extra hour you put will take two to fix it later (in other words, negative impact). Even just pair programming will help prevent this in the first place as long at least one of you isn’t operating on half the sleep they need. But ultimately, all of this just blames the individual, when really poor code quality and technical debt are much more a function of the management system, methodology, total sum of the choices by every on the team, etc. (For a start, why are you up late in the first place? Was that your choice, or an outside imposed deadline?).

As James Adam and “iSteve” point out in the comments – technical debt is different from just “buggy software” and in my experience it’s usually function of management or other types of systemic choices (and “systemic” includes you, the programmer as well everyone else). What it really does is trade easy change now for even harder change later. It wants that sweet sweet sugar hit straight away but forgets about the “sugar coma” that comes a few minutes afterwards. It defers “paying the piper” until tomorrow for that lovely tune today. James Adam rightly says in his comment this can be a valid choice, but all of the team have to be fully aware of the consequences of that choice. One of which is typically that the longer  you defer the payment, the more debt you’ll accumulate, until all you are doing is paying off the interest and not getting to the principal.

As “iSteve” says, management (and maybe “management” might mean just you telling yourself this in your head) always promises to “fix it later” and never does allow that. “Just make the hack” and gloss over the problem is a mantra that we’ve all heard before. Where it gets painful is where every change is now taking twice as long as it should because of the accumulation of hacks. Every single one one of us has been there before, and are probably in various levels of it right now.

In my view this is the much more insidious circumstance because it is systemic.  After all the solution to a bad day’s work because I was too tired to think properly is to simply get a good night’s sleep, roll back yesterday’s coding, and do it again today. All I’ve done is lost a day’s work.  However, an organisation is likely to have accumulated months if not years of organisational cruft that makes changing their context that much harder than just rolling back yesterday’s poor code and trying again – it requires commitment to change from everyone who counts in the first instance.

First, I no where made any “assertion that if you are putting ‘//TODO’ markers in your code, you’re somehow not properly following the Scrum process” – I said if you find yourself using ‘//TODO’ markers in your code you are being prevented from getting to done-done and that you should “Identify the problem, propose a counter-measure and fix the process“. I can’t emphasise that enough. “Fix the process” does not mean “follow scrum” but rather to use your team’s collective brain to work out what stops you from fixing the code then and there and propose a solution to the problem.

I think ‘//TODO’ should either be in the current story (I will concede that you might want to use a ‘//TODO’ to remind yourself about tasks in your current story, i.e. “come back here tomorrow when I add the validation”) or in the “backlog”. I put “backlog” in quotes there to point out I mean “backlog” in the broadest sense, if you using Scrum or XP that’s actually where the work-to-be-done is physically kept, but if you have some other process then your “backlog” is kept somewhere else, and what is preventing you from adding to it?

I will ask a question of the anonymous poster – at what point exactly  is a ‘//TODO’ comment left decaying in the codebase without a story for days or weeks a good idea? Or is it that you just do not want to make what it is being done in the code base transparent to those who are paying your bills?

I will contend; ‘//TODO’ never gets done because the product owners never see them. They act as an opaque barrier to open communication. If you find a ‘//TODO’, lets say a big major refactoring will be needed to support a story you already see in the backlog, what’s the problem with having an open and honest communication with the team about the story’s estimate? If it’s a future story, i.e. feature related, it should be in the “backlog”; if it is code or design related which impacts you now, either it should be done then and there (refactor mercilessly) or it’s a bug which you either fix or raise a bug report for, and if it’s just some nebulous “improvement” either do it now or YAGNI. If you’ve got great tooling like Mylyn for Eclipse you can drop a “task” at the code point and move on. The point there is the “TODO” isn’t obscured away in the code but rather visible to all because the ‘//TODO’ is now either in your Wiki, your task-tracking tooling via it’s Mylyn integration, or is included as part of a story card on the story board.

If you can’t refactor mercilessly – what the post is really about – all the usual excuses apply, i.e. “we’re not confident about the code base not breaking in unknown ways” (i.e. no tests), “it has to be done by tomorrow,” et cetera, and all of these are symptoms of dysfunction in your organisation.

People love their little comfort zones and they don’t like being pushed out of them, that’s for sure, programmers included (yeah, and that’s a me too). Habit is not an excuse for avoiding improvement.

Here’s a great imaginary conversation from the C2 wiki entry about YAGNI that I think applies here:

“But Ron, I know how to do it right now, and later I might not.”

“So, Sam, you’re telling me that this class you’re writing is so complex that even you won’t be able to maintain it?”

Today I just want to meditate on a code artifact that often manifests itself in such environments. I speak of the comment //TODO seen frequently in such code. The idea that future “refactoring” goes on the “todo” list in such an environment is completely broken. An environment that allows that to happen means it will never get done. Do it now, or don’t do it. TODO is considered harmful.

I have come around to the view that ‘//TODO’ comments should make a build fail. Either:

1. It is a future story yet to be implemented and placed into the backlog,
2. It is work in progress (and by definition doesn’t need a TODO because everything still needs to be ‘done’), or,
3. It is done, as in DONE-done, and therefore the story is complete and there’s nothing left “TO DO”.

In other words, “//TODO” is a indicator of a process that’s not letting developers get to “done”. Identify the problem, propose a counter-measure and fix the process that stopping you from getting to completion.

On a technical mailing list I’m on, a developer asked for help in regard to a class he had to modify. There’s no need for the full gory detail here; suffice to say it’s main feature was that it had a very long if-then-else method that had nearly 20 steps testing multiple boolean statuses strung out after each other. The question was very simply; “how to refactor this?”. The suggestions were all good; ranging from using polymorphism to implementing a rules engine.

It’s not the solutions, or even the code that I want to discuss. Near the end of the thread, the original poster came back and said he’d discovered that (to paraphrase) ‘the feature change needs to go to the QA process today’ and that the refactoring couldn’t be done: the two-line hack would have to suffice.

I feel for the poor bugger, I really do, I’ve been in that situation myself. Maybe it’s just my bombastic, iconoclastic nature that makes me push back at these situations and tell people what fools they are for not valuing the craftsmanship of the very craftsmen they’ve hired specifically to build their software solutions. We need to start to insist that craftsmanship is valued and not just dismissed with an airy wave of the hand by management anymore. I’m fully aware this frequently makes me unpopular amongst those who don’t like to told the hard, brutal, truth. But I’ve given up caring. The truth it is.

In this spirit, I’m going to put on my “I’m as mad as hell and I’m not gonna take this anymore” hat and say the following. The reason that class in that state is because “changes to this feature are going to need to be in a build for QA today”. What needs to be done about these situations?

Just Say No.

Developers – those aspiring to be craftsmen anyway, need to be able to say it just is not possible for this feature to be changed today. Tell the management they have maxed out the credit card and the bailiffs are banging at the door and taking the silverware away as payment.

For a start, who decided this feature had to be in the QA build at some arbitrary cut-off date anyway? Someone who had never seen that class, that’s who. That “chicken” committed developer – the “pig” – to deliver the feature by some arbitrary cut off date and didn’t know the details. This is estimation by fiat. In any half-way decent process that respects it’s workers as people and not jut bit-flippers, no-one has the right to do this; it’s massively broken process. Chickens are not allowed to decide on estimates: only pigs are. The only way these things change if someone stands up and says “NO” to such broken process.

I don’t mean to say the developers can just go off with the fairies and gold-plate every piece of code they touch. But it has to be at least well-made. Obviously this particular process isn’t an agile environment – as it includes a “QA” stage with arbitrary cut off date at which point hte code “gets thrown over the wall”. And the company has management (I assume it’s management) running around and setting arbitrary delivery dates for unknown work.

Seriously, process improvement has to start somewhere. Be the change you seek. Stop tolerating poor code. If the process you work in demands you to tolerate it, change the process as best you can.

Link: http://www.viddler.com/explore/sergiopereira/videos/7/.

The Project Cartoon - how the customer explained it, what we built them & what the customer actually needed

It’s from a site called The Project Cartoon. What I didn’t know was the site allows you to use the little cartoons to make your own version. You can caption and re-order the panels. For your amusement, and perhaps your edification, I present “The Rewrite Will Be Ready Shortly”:

The existing software is ugly and poorly written. We will have to rewrite it if you want any new features.

[Certain reformists say] The country has this one chance, the idealist maintains, to sweep away our inhumane, wasteful patchwork system and replace it with something new and more rational. So we should prepare for a bold overhaul, just as every other Western democracy has. True reform requires transformation at a stroke. But is this really the way it has occurred in other countries? The answer is no. And the reality of how health reform has come about elsewhere is both surprising and instructive.

He goes at lengths to explain how the health care that the rest of the developed world (e.g. Australia) has found itself with, are the results of ad-hoc changes to existing systems, not large-scale rewrites. As a result, every country has a “path-dependent” health care system. For example, Britain’s NHS evolved in 1945 from wartime requirements and infrastructure. After the way, France’s infrastructure was largely destroyed, so they used (that is, modified and extended) an existing system of union organised health care insurance paid for through payroll taxes. And so on. Everyone’s system is different because their initial conditions where different, and solutions have to respect what’s there, and not just blithely sweep it all aside.

This possibly could give us a general principle with regards to all types of reform - you have to work with what you’ve got. That includes software.

We’ve known for some time that the big rewrite nearly always ends in failure (although not everyone agrees). Especially when the specification for the new system is “make it do what the old system does”. Why are you re-writing it then?! (Chad Fowler has an excellent article about this).

It’s an excellent general principle that software systems should to evolve  incrementally over time. Has anyone else worked in an environment where they’ve spent 12, 18, 24 months squirrelled away somewhere vigorously rewriting an entire system while the existing system is still being maintained and improved? Not so long ago I worked on a system where we spent 4 months re-writing just a small section of a piece of software – delivered in two steps – and it was bad enough as it was. But a year? It just has to be disastrous.

What’s important is your initial set of conditions, which shape the scope of what is actually possible to do – and importantly, how you’d go about doing it. Your future system is “path dependent” on the existing one, and that was dependant on what was there before.  Software developers frequently chafe at these limits, and instead form a desire to just chuck it out and start over. But you are dooming yourself to failure if you do that.

Part of our professional practices  should be the delivery of incremental design to systems. Instead of that 4 month re-write of a small section of functionality could we have delivered it in even smaller chunks? My feeling is quite probably, yes – and it would have been a lot less stressful at the time too. Always work with what you’ve got, and change -and improve – it one small piece at a time.