• is afraid of change
• not interested in improvement
• has a rigid top-down process development style
• doesn’t care what you think

I think the greatest problem is the organisation basically doesn’t trust it’s employees to know what they are doing. It doesn’t matter that you may know something better or that you care about improving the company’s performance.  I find when I hear this sort of phrase from in companies I’m consulting at, you soon discover all manner of other issues, idiotic decision making processes, strange convoluted internal processes, inflexible management styles, complete reliance on reporting and long meetings for project visibility, and completely rigid thinking.

What the upshot of all this is, is usually a company that then is left with workers that accept their situation rather then improving it. Ultimately, when such companies confront the reality of their situation and require change to avoid failure, they fail because their employees don’t want change – they’ve had it ground down out of them over the years.

Stasis as a corporate strategy only works for so long.

The repetative (sic) steps that a single worker performs on an assembly line is a good example of a simple task

This is a terribly Fordist/Taylorist view to take of assembly work. That most beloved of industrial examples, Toyota, takes weeks to train its process-line workers to be able not only to do the simple repetitive steps of assembly, but also to be able to identify and rectify problems with the assembly-line. As Steven Spear, author of Chasing the Rabbit (my review here) might say, anyone who thinks that “assembly line” work is simple task probably doesn’t understand the nature of their process enough to keep up with their more innovate competitors, let alone outdistance them. Understanding your process, well more importantly, your organisational goals and how your process does or does not fulfil those goals, is critical to improving your organisation, and it leads me onto the idea of “evangelism”.

I both agree and disagree with Coder Friendly’s position:

Don’t become an evangelist yourself : look at your project and decide which methodologies is the most adapted. Sometimes, Agile is not the right one but if you see that your requirements are far from agreement or certainty, Agile approaches are probably a good choice.

I agree that “evangelism” can be politically off-putting.People don’t react well to being told their old kool-aid contains poison, and to uncritically drink this new kool-aid instead. However the main problem I have with a statement like this is that non-Agile companies use this “adaptation” argument and approach, in order to keep their poor contexts alive. This is the fallacy that’s found in ‘scrum butt’: “Oh we do Scrum, but …”. The clauses after the ‘but’ are typically critical. The organisation concerned is usually trying to keep some essential organisational dysfunction alive, because of politics, resistance to change, simple misunderstanding, lack of understanding about their own process, or maybe even simple laziness. I’ve got no problem with an organisation that is mature in its use of an agile method adapting it to their local circumstance – in fact if you are doing it properly, with its emphasis on empowering teams to own their process and carrying out regular ‘inspect and adapt’ at the end of every iteration, after six or twelve months of this the process will most certainly be adapted to the local conditions.

However, the real danger is at the start of the agile adoption process, when you have people hand-waving about “can’t do that here”, for whatever value of that there is. I’ve seen it said about: pairing; story boards; acceptance criteria; automated testing; retrospectives; daily stand-up; adaptive and emergent design; and just about every other aspect of XP or Scrum process. The reason is, I think, that these organisations rarely understand what the real organisational goals actually are, and they understand even less about their existing process, to the point they just “do” the process that “is”. “It is what it is”, is a great term, and if anyone says this to you it’s just a fatalistic attempt to accept as unchanging something you can change. Accept the bad. Don’t think it could be better.  Attempts to meddle in the process, i.e. change it, induce fear, and with fear often comes timidity.

This is where you need a real agile evangelist. This is not someone who just thinks that Scrum or XP or whatever other process should be blindly substituted for the existing one, but someone who understands that process is about competitive advantage, eliminating waste, and that you have to start from a known position and continuously work towards improvement. Change is not just good, but necessary, and you need those change agents. Passion for doing things better is bad? Only in a change-averse company. So learn your existing process, thoroughly, and then learn the proposed agile process, thoroughly, and only then will you be able to ‘synthesise’ a replacement. Only I think you’ll find that replacement will look a lot more like the plain old agile process than your existing unagile process. After all, if your existing process already has adequate inspect-and-adapt cycles within it (not lip-service ones, real ones), probably you are already doing and agile or lean process even if you don’t call it that, and you’re already number one in your marketplace. If not, perhaps you require the passion of an evangelist.

First up, the easy way out – take a day off, go for a walk in the park, have a nice dinner, get a good nights sleep. You should always work to “sustainable progress” and yes this means every hour you work over 40 hours a week means your quality is steadily decreasing until some point where every extra hour you put will take two to fix it later (in other words, negative impact). Even just pair programming will help prevent this in the first place as long at least one of you isn’t operating on half the sleep they need. But ultimately, all of this just blames the individual, when really poor code quality and technical debt are much more a function of the management system, methodology, total sum of the choices by every on the team, etc. (For a start, why are you up late in the first place? Was that your choice, or an outside imposed deadline?).

As James Adam and “iSteve” point out in the comments – technical debt is different from just “buggy software” and in my experience it’s usually function of management or other types of systemic choices (and “systemic” includes you, the programmer as well everyone else). What it really does is trade easy change now for even harder change later. It wants that sweet sweet sugar hit straight away but forgets about the “sugar coma” that comes a few minutes afterwards. It defers “paying the piper” until tomorrow for that lovely tune today. James Adam rightly says in his comment this can be a valid choice, but all of the team have to be fully aware of the consequences of that choice. One of which is typically that the longer  you defer the payment, the more debt you’ll accumulate, until all you are doing is paying off the interest and not getting to the principal.

As “iSteve” says, management (and maybe “management” might mean just you telling yourself this in your head) always promises to “fix it later” and never does allow that. “Just make the hack” and gloss over the problem is a mantra that we’ve all heard before. Where it gets painful is where every change is now taking twice as long as it should because of the accumulation of hacks. Every single one one of us has been there before, and are probably in various levels of it right now.

In my view this is the much more insidious circumstance because it is systemic.  After all the solution to a bad day’s work because I was too tired to think properly is to simply get a good night’s sleep, roll back yesterday’s coding, and do it again today. All I’ve done is lost a day’s work.  However, an organisation is likely to have accumulated months if not years of organisational cruft that makes changing their context that much harder than just rolling back yesterday’s poor code and trying again – it requires commitment to change from everyone who counts in the first instance.

Agile requires the discipline to move projects through teams. Multitasking is nuts in agile. Moving team members around to have the “best” specialist available for a particular team is nuts. Performance reviews for individuals is nuts. Managers have to change everything they do, if they want to move the organization to agile. Managers need to see the problems exposed by any given project’s transition to agile and work to remove those obstacles before transitioning to agile.

It’s too easy to think that just because you can get one of your teams to use an agile approach, that your organisation is now Agile. Too many managers “see the problems exposed by given project’s transition to agile” and decide to blame the agile process for the problems thus exposed.In far too many organisations I’ve seen have extremely capable teams delivering quality software in an agile process but remain – in some cases years later – completely unsupported by their senior management (and in some cases with very little support even from their CIO level).

That’s a very sad situation as it is and takes all your selling skills to get the management to “get with the program” so to speak. But this situation can be even more compounded when the management have been convinced that Agile is some sort of silver bullet that will cure all their issues. Whenever you are trying to sell agile to management, never waver from the fact that agile is hard (as everything worth doing is nearly always difficult anyway!). Agile takes discipline.

Johanna’s right to say you can’t adopt agile across an entire organisation in a big bang approach. But you can’t also adopt it without skill and commitment to doing so from managers and executives.

But the real kicker was a presentation on Agile Integration, by one of the vendor’s partners (and in the interests of disclosure, a competitor to my own employer). A few slides in and there’s the inevitable “what is agile” slide with a standard dictionary definition and some lip-service to previous history. Now the thing here of course, is that the word Agile in software development parlance is by now a well-entrenched piece of jargon – it has a specific meaning to most people that is fairly precise and different to just the garden-variety common English usage. This is the purpose of jargon in a discipline: to be precise in communication. And the agile manifesto, nearly 10 years old, just means in effect, if you’re in the software development game you just can’t redefine the term to mean something else.

Except of course, you can, if you do, and you’re big enough and enough people listen to your incredibly mangled marketoid-speak. But implementing a SOA solution is not Agile, no matter how many times you make an incredibly weak case that it is or how many times you repeat any other platform-orientated mantra. Yes, you read it right, according to the vendor’s partner at least, SOA is Agile.

This banality made me so angry, it was very lucky that straight after there was the morning coffee break because I was seething and it took all my self-discipline not to launch into an attack during the questions (easy picking, because there were no other questions asked!). I had to vent at my poor colleagues in the break.

Anyway, on my admittedly pessimistic reading it means that Agile Is Dead, as pretty soon we are going to find clients telling people, “Yeah we do agile, we bought the million-dollar SOA package off that vendor”, and then they’ll be saying the reason their integration projects are still failing is because they haven’t installed the very latest patchset, or upgraded to all-new LEANsuite 17.4.15.8 release B with the optional drag-n-drop Super-Agility add-in for Portals, Release 16f (the “f” is for “fail”).

At sushi train lunch afterwards, we made the observation that pretty much the vendor wants to be all the food groups all at once. Kind of like this bit of gluggy sushi I mistakenly lifted off the train:

All the food groups. Nasty.

Now I’m not saying that you can’t do “SOA” with an agile process, if that’s your preferred architectural style, then if you are committed you can certainly develop it in an Agile fashion. You can’t afford however, to confuse the two things. If you do, that picture above is  what your enterprise will look like too if you mistake the buzzword bingo offered by vendors for actual insight into agile development process. Talk to an expert instead.

Sorry to be so negative, the whole morning pretty much sapped my will to live.

First, I no where made any “assertion that if you are putting ‘//TODO’ markers in your code, you’re somehow not properly following the Scrum process” – I said if you find yourself using ‘//TODO’ markers in your code you are being prevented from getting to done-done and that you should “Identify the problem, propose a counter-measure and fix the process“. I can’t emphasise that enough. “Fix the process” does not mean “follow scrum” but rather to use your team’s collective brain to work out what stops you from fixing the code then and there and propose a solution to the problem.

I think ‘//TODO’ should either be in the current story (I will concede that you might want to use a ‘//TODO’ to remind yourself about tasks in your current story, i.e. “come back here tomorrow when I add the validation”) or in the “backlog”. I put “backlog” in quotes there to point out I mean “backlog” in the broadest sense, if you using Scrum or XP that’s actually where the work-to-be-done is physically kept, but if you have some other process then your “backlog” is kept somewhere else, and what is preventing you from adding to it?

I will ask a question of the anonymous poster – at what point exactly  is a ‘//TODO’ comment left decaying in the codebase without a story for days or weeks a good idea? Or is it that you just do not want to make what it is being done in the code base transparent to those who are paying your bills?

I will contend; ‘//TODO’ never gets done because the product owners never see them. They act as an opaque barrier to open communication. If you find a ‘//TODO’, lets say a big major refactoring will be needed to support a story you already see in the backlog, what’s the problem with having an open and honest communication with the team about the story’s estimate? If it’s a future story, i.e. feature related, it should be in the “backlog”; if it is code or design related which impacts you now, either it should be done then and there (refactor mercilessly) or it’s a bug which you either fix or raise a bug report for, and if it’s just some nebulous “improvement” either do it now or YAGNI. If you’ve got great tooling like Mylyn for Eclipse you can drop a “task” at the code point and move on. The point there is the “TODO” isn’t obscured away in the code but rather visible to all because the ‘//TODO’ is now either in your Wiki, your task-tracking tooling via it’s Mylyn integration, or is included as part of a story card on the story board.

If you can’t refactor mercilessly – what the post is really about – all the usual excuses apply, i.e. “we’re not confident about the code base not breaking in unknown ways” (i.e. no tests), “it has to be done by tomorrow,” et cetera, and all of these are symptoms of dysfunction in your organisation.

People love their little comfort zones and they don’t like being pushed out of them, that’s for sure, programmers included (yeah, and that’s a me too). Habit is not an excuse for avoiding improvement.

Here’s a great imaginary conversation from the C2 wiki entry about YAGNI that I think applies here:

“But Ron, I know how to do it right now, and later I might not.”

“So, Sam, you’re telling me that this class you’re writing is so complex that even you won’t be able to maintain it?”

Today I just want to meditate on a code artifact that often manifests itself in such environments. I speak of the comment //TODO seen frequently in such code. The idea that future “refactoring” goes on the “todo” list in such an environment is completely broken. An environment that allows that to happen means it will never get done. Do it now, or don’t do it. TODO is considered harmful.

I have come around to the view that ‘//TODO’ comments should make a build fail. Either:

1. It is a future story yet to be implemented and placed into the backlog,
2. It is work in progress (and by definition doesn’t need a TODO because everything still needs to be ‘done’), or,
3. It is done, as in DONE-done, and therefore the story is complete and there’s nothing left “TO DO”.

In other words, “//TODO” is a indicator of a process that’s not letting developers get to “done”. Identify the problem, propose a counter-measure and fix the process that stopping you from getting to completion.

On a technical mailing list I’m on, a developer asked for help in regard to a class he had to modify. There’s no need for the full gory detail here; suffice to say it’s main feature was that it had a very long if-then-else method that had nearly 20 steps testing multiple boolean statuses strung out after each other. The question was very simply; “how to refactor this?”. The suggestions were all good; ranging from using polymorphism to implementing a rules engine.

It’s not the solutions, or even the code that I want to discuss. Near the end of the thread, the original poster came back and said he’d discovered that (to paraphrase) ‘the feature change needs to go to the QA process today’ and that the refactoring couldn’t be done: the two-line hack would have to suffice.

I feel for the poor bugger, I really do, I’ve been in that situation myself. Maybe it’s just my bombastic, iconoclastic nature that makes me push back at these situations and tell people what fools they are for not valuing the craftsmanship of the very craftsmen they’ve hired specifically to build their software solutions. We need to start to insist that craftsmanship is valued and not just dismissed with an airy wave of the hand by management anymore. I’m fully aware this frequently makes me unpopular amongst those who don’t like to told the hard, brutal, truth. But I’ve given up caring. The truth it is.

In this spirit, I’m going to put on my “I’m as mad as hell and I’m not gonna take this anymore” hat and say the following. The reason that class in that state is because “changes to this feature are going to need to be in a build for QA today”. What needs to be done about these situations?

Just Say No.

Developers – those aspiring to be craftsmen anyway, need to be able to say it just is not possible for this feature to be changed today. Tell the management they have maxed out the credit card and the bailiffs are banging at the door and taking the silverware away as payment.

For a start, who decided this feature had to be in the QA build at some arbitrary cut-off date anyway? Someone who had never seen that class, that’s who. That “chicken” committed developer – the “pig” – to deliver the feature by some arbitrary cut off date and didn’t know the details. This is estimation by fiat. In any half-way decent process that respects it’s workers as people and not jut bit-flippers, no-one has the right to do this; it’s massively broken process. Chickens are not allowed to decide on estimates: only pigs are. The only way these things change if someone stands up and says “NO” to such broken process.

I don’t mean to say the developers can just go off with the fairies and gold-plate every piece of code they touch. But it has to be at least well-made. Obviously this particular process isn’t an agile environment – as it includes a “QA” stage with arbitrary cut off date at which point hte code “gets thrown over the wall”. And the company has management (I assume it’s management) running around and setting arbitrary delivery dates for unknown work.

Seriously, process improvement has to start somewhere. Be the change you seek. Stop tolerating poor code. If the process you work in demands you to tolerate it, change the process as best you can.

Firstly, I found the second day much better than the first. I did see Michael Nygard’s two talks on Monday morning and got a lot a lot of useful information from those. Titled ‘Failure comes in flavours’ the first was a great overview of the types of failure that applications running in production encounter, the second ways to avoid those failure modes. Basically it’s how many of the non-functional requirements can eat your app (and even the entire integration if you let the failure propagate easily) if you don’t give careful consideration to them. He’s got a book so I’m going to check that out next.

The first day for me was severely spoilt by a poor vendor presentation (hint: one that is not Oracle, not that they would have necessarily done any better), which I should have known better to avoid. I already swim in the BPEL/BPM/SOA soup, I fully understand just how broken it is as an actual development concept (e.g. anti-test-first to name just one). I was fooled by the title to thinking maybe it had some insights to avoid the worst of these busted concepts, but no, the vendor’s tooling is all the rage if you drink the poor-tasting conceptual kool-aid. This stuff is marketed at managers and non-coding architects who choose to buy this stuff then dump it from a very great height into the laps of the poor developers who have to mangle it into a deliverable system (half the time with a development budget just a tenth the cost of the licence). Anyway back to the good stuff.

Mike Cannon-Brookes from Atlassian (which makes great development tools, highly recommended) gave a mostly great closing speech on the first day which made me think about a couple of things. It wasn’t until the next morning however that it crystallised in my mind. It’s a bit of a minor criticism.

Mike threw out a minor line, an aside really, talking about Atlassian’s agile development methodology, and basically that said, people who say ‘you must do this’ are wrong, because they miss your context. Here ‘this’ is some type of agile process artefact like maybe, pairing, or estimating in story points, or whatever. Well, it’s wrong and on two counts. The first count I realised just after the talk but couldn’t articulate it correctly in semi-drunken conversation. That first count is very simple. Mike’s timeline on his Atlassian experience has him, if I recall correctly, starting his company and creating the great tool Jira, in his early twenties. Now as he dissed “old people” in his talk I’m going to give some back. In your twenties you’ve got no idea what works, in a general sense. You might know (or think you know) what works for you, but that’s a completely different kettle of fish to what works generally. Why not? The word is experience. Its not until you reach … oh … I’ll say around 35 (and so maybe lose the capacity to build technical innovation, perhaps, as Mike asserted) that you can have real insights into process and people. It’s just because at this point, you just have not got enough practice at it (see Malcolm Gladwell’s recent book for some basic data about the power of practice).

But the bigger point I think is the word context. The phrase ‘my context is different’, is this classic phrase which really means ‘that won’t work here’ and that really means, I don’t want to change. As a consultant , you hear this stuff all the time: … “We’re unique, our company is not like those other companies. We can’t do that here. Detailed estimates are really the most important part of the development process. The budget and the features must be fixed. We have a reporting process that allows the budget and features to be fixed. Pair programming doubles the time it takes to deliver the features. Quality doesn’t matter. Technical quality is a business decision. The business are too busy to talk to you.” … And a whole other other bunch of epic agile fail. Someone’s context is exactly why they aren’t as agile as their competitors and are looking at using ‘agile development’ to help save them … but as long as they insist on my context trumps all, they will only get a fractional improvement and not an order-of-magnitude one. But Atlassian make great products, so it doesn’t mean you have to have ‘scrum’, or ‘XP’ to do that, the greatness I guess, is orthogonal. However if you are looking to ‘do agile’, then do agile. Pick a method like Scrum or XP then do it all. How do you know pair programming will fail in your organisation if you don’t try it first? You don’t. Do it like it says to do it (e.g. the Kent Beck books) and after a few months experience for your team, use the lessons from the retrospectives that the team has discovered to improve the process. Do not cripple the process with your ‘context’ before you even try.

And right on cue, on the second day along comes Steve Hayes with a great presentation on exactly this topic: “How your choices affect your agility”. Steve is a fantastic speaker, entertaining and pretty insightful too I thought. I rated his talk the best talk of the conference. He said he got lots of red cards (participants could rate each talk as green==good, yellow==ok, red==bad) in his Sydney session – what is the matter with your people? It was a great talk. I loved the part about naivety. To propose a solution from the most naive perspective possible, and what an epic win it is when a customer says “ok”! It means I understand the problem! I even tried this today with my client, I didn’t get the instant “OK” but working from the basis of that incredibly naive solution we did manage to eventually envisage the simplest thing that could possibly work. And I love that. So thanks tons for that insight, Steve!

Another great talk on the second day was Linda Rising’s ‘Deception and Estimation’ talk. Turns out (via psychology and evolutionary biology) we engage not in rational decision making but in emotional choices all the time. All choice is emotional. So we engage ourselves in self-deception all the time. And none of this is bad … it’s good. People who don’t engage in this constant self-deception are generally clinically insane. So get over it, and yourself, and when you’re estimating (and from the sounds of it, doing any sort of major-impact decision making) and get as many perspectives as you can from the most diverse range of people you can manage (to avoid groupthink, which is I guess, the situation where everyone’s self-deceptions all align). Joshua Bloch’s second-day keynote was also highly worthy of praise. Good insight and excellent effective code examples (yes, in a keynote). Also worth the greatly discounted entry ticket price was the many interesting conversations I had with past colleagues and new friends on many different development issues.

In short, JAOO was generally of very high quality and I hope is keeps coming to Brisbane! Will definitely be going next year. Many thanks to all the organisers, presenters and sponsors.

Consider the following scenario.

1. You’ve got two parts of the code base. One part is beautiful code, written TDD, used continuous integration, pair programming, well written stories were used to produce it, clear acceptance tests, all the good stuff. The other part is horrific, hardly any tests, when the build failed, you disabled the tests that failed, the stories were written as technical tasks, acceptance tests vague, etc.
2. You are asked to deliver two medium-value stories in the next iteration. One story is most done in the good part of the code base. The other story in the bad part.
3. Correspondingly, your estimates will vary according to the complexity of the code base – the story in the “good” code might be 5 points, the story in the “bad” code is 15. But the value to the business delivered is the same!

The point is, the amount of effort that you do (20 points) to deliver 2 medium value units, is actually not a measure of your progress. It’s a measure of your waste. The next iteration you might be able to deliver 4 medium value points, if they are all in the “good code” base! Same effort, twice the return on value. Software developers know this about well designed code bases combined with good engineering practice, of course. The point is about directly exposing the effects of bad engineering as it manifests itself as a barrier to progress.

What if you told the business instead, you’ll deliver x business value points instead? That is, your burn down and burn up charts don’t show story points of effort expended, but directly indicated value delivered. What impetus does this lead us to? Well, now we’ve got a direct measurement of story points over business value, which is the proportion of waste to value. The ultimate story is infinite in value and takes zero points to deliver!

I think this is not all; because it directly establishes the ratio of waste to value, it gives you an impetus to eliminate waste, that is, to practice good, professional, software craftsmanship of the type that Robert C. Martin talks about in the video I linked to earlier tonight. Software that allows you to change efficiently to meet new needs now becomes not an abstract principle that you must kick back at your managers every time the demand to see “more features” and less engineering, but becomes a direcly correlated measurement of productive output.

It’s not all plain sailing, for the really hard nut has to be cracked. How the hell do you measure business value in units like “points”? I don’t have a cheap and easy answer right for it right now. But I don’t think this very hard question should go unanswered. For me, it’s the missing piece of the agile software delivery puzzle. I’ve been asked countless times, to estimate effort on whole swathes of stories where I think it’s obvious that the business value is actually highly dubious – or at least cannot be quantified by the product owner. On one level, that’s fine, they pay the bills and get to set the priorities, but on another level, I think that having to produce a quantifier for the value will help the product owner focus their minds onto the outcome in a much better way.

And it enables them to directly see the correlation between bad engineering practices and the inefficiencies they engender. Those bad practices are certainly our fault for putting them in place, don’t get me wrong, I’m not shirking our personal and professional responsibilities in that regard. However reducing outside pressures will certainly also help by giving us a little breathing space to demonstrate with hard numbers the value of taking care with our craftsmanship. And it reports the thing the business are most interested in – the value you deliver.

If my post is a little unclear, forgive me it’s late at night as I write this. Better blog entry than code in the tired state I’m in!