• is afraid of change
• not interested in improvement
• has a rigid top-down process development style
• doesn’t care what you think

I think the greatest problem is the organisation basically doesn’t trust it’s employees to know what they are doing. It doesn’t matter that you may know something better or that you care about improving the company’s performance.  I find when I hear this sort of phrase from in companies I’m consulting at, you soon discover all manner of other issues, idiotic decision making processes, strange convoluted internal processes, inflexible management styles, complete reliance on reporting and long meetings for project visibility, and completely rigid thinking.

What the upshot of all this is, is usually a company that then is left with workers that accept their situation rather then improving it. Ultimately, when such companies confront the reality of their situation and require change to avoid failure, they fail because their employees don’t want change – they’ve had it ground down out of them over the years.

Stasis as a corporate strategy only works for so long.

DAVID DUNNING: Well, my specialty is decision-making. How well do people make the decisions they have to make in life? And I became very interested in judgments about the self, simply because, well, people tend to say things, whether it be in everyday life or in the lab, that just couldn’t possibly be true. And I became fascinated with that. Not just that people said these positive things about themselves, but they really, really believed them. Which led to my observation: if you’re incompetent, you can’t know you’re incompetent.

ERROL MORRIS: Why not?

DAVID DUNNING: If you knew it, you’d say, “Wait a minute. The decision I just made does not make much sense. I had better go and get some independent advice.” But when you’re incompetent, the skills you need to produce a right answer are exactly the skills you need to recognize what a right answer is. In logical reasoning, in parenting, in management, problem solving, the skills you use to produce the right answer are exactly the same skills you use to evaluate the answer. And so we went on to see if this could possibly be true in many other areas. And to our astonishment, it was very, very true.

Recently I was learning ANTLR, trying to decide whether this was a right technology to pursue a particular project which involved parsing a preexisting message format. After a week of a spike, we decided that it was worth pursuing and started on earnest on the grammar for our project. However a week into this process, I had an epiphany … I was doing some things wrong with the ANTLR grammar which were now slowing progress in adding the new characteristics it needed to be complete. Many developers know this feeling; the features of my grammar that I had built over the first week were naive and now hampering it from expanding into the new requirements. I took it on myself to kill the entire grammar and start again. It took less than a day and half to replicate that week’s worth of work (i.e. pass the test suite which had built up around it).  I’ve done this before; scrap the first attempt at building a domain and try again. Here my domain was the same (it was after all defined in both a standards specification and in the many hundreds of thousands of sample messages we captured from an existing system), but its implementation needed refinement.

So I think that the rule about throwing away spikes can in fact be made a general axiom of programming:

When you are learning a new technology, make sure you throw away the first thing you build that works – to avoid accumulating your mistakes.

Thanks to Robert for the important qualifier “that works”.

N.B. my views about system rewrites have not changed regardless.

I don’t hold that software makes an existing business process “efficient” at all. Rather I think software makes possible a new process, which should be more “efficient” in terms of money gained less dollars spent – but its a new process, not the old one. In effect, new software creates new business opportunities. New software will only make an existing, unchanged process, less efficient, if a new business process is not designed along with the new software. If the business just wants new software without changing “what they do” they are wasting their money, IMHO.

Of course there is the possibility (probability?) the business doesn’t actually understand what it is they actually do anyway. This is not an uncommon position for many businesses that are happy to cruise along in neutral making some marginal profit on some marginal activity. Usually these businesses are also found to be beating their workers with sticks (usually only metaphorical ones unless they ‘offshore’ their operation to countries where killing your workers is just a part of ‘Business as Usual’. Typically they hold that marginal process can be made ‘better’ simply with just more exhortation (or threats) to greater and greater efforts at a totally demoralized (if not actively hostile) workforce, but I suspect that’s a story for another day!

Lost in the shuffle is something basic: Programmer productivity. Friends who are hands-on with such matters tell me the available SOA development environments are less than half as productive as products like PowerBuilder and Delphi were, back when they were viable.

Putting aside the Powerbuilder and Delphi love for just one minute, this is something I’ve been banging on about now for the past year … the programming tooling that is foisted onto programmers by the choice of the deployment architecture. It’s just all wrong.

In my view, what makes a programming language really productive is notepad. Or vi, or emacs, or gvim, or textmate, take your pick. What I mean is … the programming language has to be able to be programmed with a simple editor. Yes, an advanced IDE will make things more productive, but the basics must also apply. Now a lot of SOA environments are simply not programmable without the specific IDE tied to it. Even worse, the IDEs are often completely custom jobs that require a developer to be re-trained … losing years and years of productive speed with muscle-memory style automatic ability to navigate the programmer’s usual editing tool. Seriously. This stuff is whack. A program language or an environment needs to be IDE-neutral. If you got a plumber around, would you insist that he only use the tools you supply from your home handyman kit? Or would you expect the plumber to have mastered a set of his own tools already? And making matters worse, it’s rarely programmers that choose these tools which are foisted on them. The server/deployment environment and the language used to implement need to be decoupled from the tools used to build it.

But an even worse failing of many of these SOA tool suites, is that they all strongly and irrevocably coupled to the deployment/runtime environment. Generally they totally lack the ability to keep up with modern programming practice. Like for instance, automated testing. Or even unit tests, let alone advanced and productive techniques such as Test-First approaches or Test Driven Design. The tools often lack refactoring support. All of these things are in my opinion, and in the opinion of many leading developers, absolutely essential to quality engineering practice and agile development outcomes like “delivery of working software”. Both the “delivering” and the “working” part means the whole process needs to be repeatable. That’s why automated integration testing, to name just one thing, is essential in modern development. But often the fancy custom development tooling is a complete barrier to achieving this.

But you won’t hear any of this from the big vendors. One big vendor recently announced their new version of their middleware product suite had a ‘focus on testability’, but you ask any of their presales guys to demonstrate this in an actual development environment. Ask them about continuous integration support, for example. Witness their blank looks. Their development product is completely orientated to “one button push from the IDE to production” modes of thinking the idea of continuous integration builds is almost totally antithetical to the very concepts of operation the product is organised around. They think that finally adding support for Subversion version control system, at least five years too late, is a wondrous achievement.

They are aiming for ‘programmerless programming’: of course in process just creating a new type of programmer. Every new generation of programmers simply have to learn the same hard-fought lessons of software engineering over and over again because each generation of tooling apparently scraps the paradigm over and over again in a vain attempt to create push-button, wizard-driven programming models. They nearly all suffer from ‘hello world’ programming – the simple examples that sell them to IT management are trivial to conquer using the wizards, but more complex problems (i.e. real world ones) are flat-out impossible. Thus these tools are always mirages which look great at a huge distance on the horizon but are flat lifeless salt pans of bleached skulls and bones on closer examination (or, maybe they are more like tar pits that look like a nice waterhole but one step into it and you are sucked down  to your doom).

As you might be able to tell, I am utterly contemptuous of many of these SOA tool paradigms. I have nothing against SOA itself, per se. But there is nothing more productive than a programmer who understands the importance of simple and repeatable build and deployment automation using command line tools and who knows his programming editor inside out after ten years of use. Give that programmer a better language by all means, add incremental features to that IDE, allow the programmers to continuously improve their techniques, promote professional craftsmanship, yes, yes and a thousand times yes. But no amount of drag and drop wizards, push-button deployments, and “object inspector” property editors will ever usurp that deep knowledge a good programmer brings to both his language and his personal tooling choices.

The repetative (sic) steps that a single worker performs on an assembly line is a good example of a simple task

This is a terribly Fordist/Taylorist view to take of assembly work. That most beloved of industrial examples, Toyota, takes weeks to train its process-line workers to be able not only to do the simple repetitive steps of assembly, but also to be able to identify and rectify problems with the assembly-line. As Steven Spear, author of Chasing the Rabbit (my review here) might say, anyone who thinks that “assembly line” work is simple task probably doesn’t understand the nature of their process enough to keep up with their more innovate competitors, let alone outdistance them. Understanding your process, well more importantly, your organisational goals and how your process does or does not fulfil those goals, is critical to improving your organisation, and it leads me onto the idea of “evangelism”.

I both agree and disagree with Coder Friendly’s position:

Don’t become an evangelist yourself : look at your project and decide which methodologies is the most adapted. Sometimes, Agile is not the right one but if you see that your requirements are far from agreement or certainty, Agile approaches are probably a good choice.

I agree that “evangelism” can be politically off-putting.People don’t react well to being told their old kool-aid contains poison, and to uncritically drink this new kool-aid instead. However the main problem I have with a statement like this is that non-Agile companies use this “adaptation” argument and approach, in order to keep their poor contexts alive. This is the fallacy that’s found in ‘scrum butt’: “Oh we do Scrum, but …”. The clauses after the ‘but’ are typically critical. The organisation concerned is usually trying to keep some essential organisational dysfunction alive, because of politics, resistance to change, simple misunderstanding, lack of understanding about their own process, or maybe even simple laziness. I’ve got no problem with an organisation that is mature in its use of an agile method adapting it to their local circumstance – in fact if you are doing it properly, with its emphasis on empowering teams to own their process and carrying out regular ‘inspect and adapt’ at the end of every iteration, after six or twelve months of this the process will most certainly be adapted to the local conditions.

However, the real danger is at the start of the agile adoption process, when you have people hand-waving about “can’t do that here”, for whatever value of that there is. I’ve seen it said about: pairing; story boards; acceptance criteria; automated testing; retrospectives; daily stand-up; adaptive and emergent design; and just about every other aspect of XP or Scrum process. The reason is, I think, that these organisations rarely understand what the real organisational goals actually are, and they understand even less about their existing process, to the point they just “do” the process that “is”. “It is what it is”, is a great term, and if anyone says this to you it’s just a fatalistic attempt to accept as unchanging something you can change. Accept the bad. Don’t think it could be better.  Attempts to meddle in the process, i.e. change it, induce fear, and with fear often comes timidity.

This is where you need a real agile evangelist. This is not someone who just thinks that Scrum or XP or whatever other process should be blindly substituted for the existing one, but someone who understands that process is about competitive advantage, eliminating waste, and that you have to start from a known position and continuously work towards improvement. Change is not just good, but necessary, and you need those change agents. Passion for doing things better is bad? Only in a change-averse company. So learn your existing process, thoroughly, and then learn the proposed agile process, thoroughly, and only then will you be able to ‘synthesise’ a replacement. Only I think you’ll find that replacement will look a lot more like the plain old agile process than your existing unagile process. After all, if your existing process already has adequate inspect-and-adapt cycles within it (not lip-service ones, real ones), probably you are already doing and agile or lean process even if you don’t call it that, and you’re already number one in your marketplace. If not, perhaps you require the passion of an evangelist.

First up, the easy way out – take a day off, go for a walk in the park, have a nice dinner, get a good nights sleep. You should always work to “sustainable progress” and yes this means every hour you work over 40 hours a week means your quality is steadily decreasing until some point where every extra hour you put will take two to fix it later (in other words, negative impact). Even just pair programming will help prevent this in the first place as long at least one of you isn’t operating on half the sleep they need. But ultimately, all of this just blames the individual, when really poor code quality and technical debt are much more a function of the management system, methodology, total sum of the choices by every on the team, etc. (For a start, why are you up late in the first place? Was that your choice, or an outside imposed deadline?).

As James Adam and “iSteve” point out in the comments – technical debt is different from just “buggy software” and in my experience it’s usually function of management or other types of systemic choices (and “systemic” includes you, the programmer as well everyone else). What it really does is trade easy change now for even harder change later. It wants that sweet sweet sugar hit straight away but forgets about the “sugar coma” that comes a few minutes afterwards. It defers “paying the piper” until tomorrow for that lovely tune today. James Adam rightly says in his comment this can be a valid choice, but all of the team have to be fully aware of the consequences of that choice. One of which is typically that the longer  you defer the payment, the more debt you’ll accumulate, until all you are doing is paying off the interest and not getting to the principal.

As “iSteve” says, management (and maybe “management” might mean just you telling yourself this in your head) always promises to “fix it later” and never does allow that. “Just make the hack” and gloss over the problem is a mantra that we’ve all heard before. Where it gets painful is where every change is now taking twice as long as it should because of the accumulation of hacks. Every single one one of us has been there before, and are probably in various levels of it right now.

In my view this is the much more insidious circumstance because it is systemic.  After all the solution to a bad day’s work because I was too tired to think properly is to simply get a good night’s sleep, roll back yesterday’s coding, and do it again today. All I’ve done is lost a day’s work.  However, an organisation is likely to have accumulated months if not years of organisational cruft that makes changing their context that much harder than just rolling back yesterday’s poor code and trying again – it requires commitment to change from everyone who counts in the first instance.

Agile requires the discipline to move projects through teams. Multitasking is nuts in agile. Moving team members around to have the “best” specialist available for a particular team is nuts. Performance reviews for individuals is nuts. Managers have to change everything they do, if they want to move the organization to agile. Managers need to see the problems exposed by any given project’s transition to agile and work to remove those obstacles before transitioning to agile.

It’s too easy to think that just because you can get one of your teams to use an agile approach, that your organisation is now Agile. Too many managers “see the problems exposed by given project’s transition to agile” and decide to blame the agile process for the problems thus exposed.In far too many organisations I’ve seen have extremely capable teams delivering quality software in an agile process but remain – in some cases years later – completely unsupported by their senior management (and in some cases with very little support even from their CIO level).

That’s a very sad situation as it is and takes all your selling skills to get the management to “get with the program” so to speak. But this situation can be even more compounded when the management have been convinced that Agile is some sort of silver bullet that will cure all their issues. Whenever you are trying to sell agile to management, never waver from the fact that agile is hard (as everything worth doing is nearly always difficult anyway!). Agile takes discipline.

Johanna’s right to say you can’t adopt agile across an entire organisation in a big bang approach. But you can’t also adopt it without skill and commitment to doing so from managers and executives.

But the real kicker was a presentation on Agile Integration, by one of the vendor’s partners (and in the interests of disclosure, a competitor to my own employer). A few slides in and there’s the inevitable “what is agile” slide with a standard dictionary definition and some lip-service to previous history. Now the thing here of course, is that the word Agile in software development parlance is by now a well-entrenched piece of jargon – it has a specific meaning to most people that is fairly precise and different to just the garden-variety common English usage. This is the purpose of jargon in a discipline: to be precise in communication. And the agile manifesto, nearly 10 years old, just means in effect, if you’re in the software development game you just can’t redefine the term to mean something else.

Except of course, you can, if you do, and you’re big enough and enough people listen to your incredibly mangled marketoid-speak. But implementing a SOA solution is not Agile, no matter how many times you make an incredibly weak case that it is or how many times you repeat any other platform-orientated mantra. Yes, you read it right, according to the vendor’s partner at least, SOA is Agile.

This banality made me so angry, it was very lucky that straight after there was the morning coffee break because I was seething and it took all my self-discipline not to launch into an attack during the questions (easy picking, because there were no other questions asked!). I had to vent at my poor colleagues in the break.

Anyway, on my admittedly pessimistic reading it means that Agile Is Dead, as pretty soon we are going to find clients telling people, “Yeah we do agile, we bought the million-dollar SOA package off that vendor”, and then they’ll be saying the reason their integration projects are still failing is because they haven’t installed the very latest patchset, or upgraded to all-new LEANsuite 17.4.15.8 release B with the optional drag-n-drop Super-Agility add-in for Portals, Release 16f (the “f” is for “fail”).

At sushi train lunch afterwards, we made the observation that pretty much the vendor wants to be all the food groups all at once. Kind of like this bit of gluggy sushi I mistakenly lifted off the train:

All the food groups. Nasty.

Now I’m not saying that you can’t do “SOA” with an agile process, if that’s your preferred architectural style, then if you are committed you can certainly develop it in an Agile fashion. You can’t afford however, to confuse the two things. If you do, that picture above is  what your enterprise will look like too if you mistake the buzzword bingo offered by vendors for actual insight into agile development process. Talk to an expert instead.

Sorry to be so negative, the whole morning pretty much sapped my will to live.

First, I no where made any “assertion that if you are putting ‘//TODO’ markers in your code, you’re somehow not properly following the Scrum process” – I said if you find yourself using ‘//TODO’ markers in your code you are being prevented from getting to done-done and that you should “Identify the problem, propose a counter-measure and fix the process“. I can’t emphasise that enough. “Fix the process” does not mean “follow scrum” but rather to use your team’s collective brain to work out what stops you from fixing the code then and there and propose a solution to the problem.

I think ‘//TODO’ should either be in the current story (I will concede that you might want to use a ‘//TODO’ to remind yourself about tasks in your current story, i.e. “come back here tomorrow when I add the validation”) or in the “backlog”. I put “backlog” in quotes there to point out I mean “backlog” in the broadest sense, if you using Scrum or XP that’s actually where the work-to-be-done is physically kept, but if you have some other process then your “backlog” is kept somewhere else, and what is preventing you from adding to it?

I will ask a question of the anonymous poster – at what point exactly  is a ‘//TODO’ comment left decaying in the codebase without a story for days or weeks a good idea? Or is it that you just do not want to make what it is being done in the code base transparent to those who are paying your bills?

I will contend; ‘//TODO’ never gets done because the product owners never see them. They act as an opaque barrier to open communication. If you find a ‘//TODO’, lets say a big major refactoring will be needed to support a story you already see in the backlog, what’s the problem with having an open and honest communication with the team about the story’s estimate? If it’s a future story, i.e. feature related, it should be in the “backlog”; if it is code or design related which impacts you now, either it should be done then and there (refactor mercilessly) or it’s a bug which you either fix or raise a bug report for, and if it’s just some nebulous “improvement” either do it now or YAGNI. If you’ve got great tooling like Mylyn for Eclipse you can drop a “task” at the code point and move on. The point there is the “TODO” isn’t obscured away in the code but rather visible to all because the ‘//TODO’ is now either in your Wiki, your task-tracking tooling via it’s Mylyn integration, or is included as part of a story card on the story board.

If you can’t refactor mercilessly – what the post is really about – all the usual excuses apply, i.e. “we’re not confident about the code base not breaking in unknown ways” (i.e. no tests), “it has to be done by tomorrow,” et cetera, and all of these are symptoms of dysfunction in your organisation.

People love their little comfort zones and they don’t like being pushed out of them, that’s for sure, programmers included (yeah, and that’s a me too). Habit is not an excuse for avoiding improvement.

Here’s a great imaginary conversation from the C2 wiki entry about YAGNI that I think applies here:

“But Ron, I know how to do it right now, and later I might not.”

“So, Sam, you’re telling me that this class you’re writing is so complex that even you won’t be able to maintain it?”