• is afraid of change
• not interested in improvement
• has a rigid top-down process development style
• doesn’t care what you think

I think the greatest problem is the organisation basically doesn’t trust it’s employees to know what they are doing. It doesn’t matter that you may know something better or that you care about improving the company’s performance.  I find when I hear this sort of phrase from in companies I’m consulting at, you soon discover all manner of other issues, idiotic decision making processes, strange convoluted internal processes, inflexible management styles, complete reliance on reporting and long meetings for project visibility, and completely rigid thinking.

What the upshot of all this is, is usually a company that then is left with workers that accept their situation rather then improving it. Ultimately, when such companies confront the reality of their situation and require change to avoid failure, they fail because their employees don’t want change – they’ve had it ground down out of them over the years.

Stasis as a corporate strategy only works for so long.

DAVID DUNNING: Well, my specialty is decision-making. How well do people make the decisions they have to make in life? And I became very interested in judgments about the self, simply because, well, people tend to say things, whether it be in everyday life or in the lab, that just couldn’t possibly be true. And I became fascinated with that. Not just that people said these positive things about themselves, but they really, really believed them. Which led to my observation: if you’re incompetent, you can’t know you’re incompetent.

ERROL MORRIS: Why not?

DAVID DUNNING: If you knew it, you’d say, “Wait a minute. The decision I just made does not make much sense. I had better go and get some independent advice.” But when you’re incompetent, the skills you need to produce a right answer are exactly the skills you need to recognize what a right answer is. In logical reasoning, in parenting, in management, problem solving, the skills you use to produce the right answer are exactly the same skills you use to evaluate the answer. And so we went on to see if this could possibly be true in many other areas. And to our astonishment, it was very, very true.

I don’t hold that software makes an existing business process “efficient” at all. Rather I think software makes possible a new process, which should be more “efficient” in terms of money gained less dollars spent – but its a new process, not the old one. In effect, new software creates new business opportunities. New software will only make an existing, unchanged process, less efficient, if a new business process is not designed along with the new software. If the business just wants new software without changing “what they do” they are wasting their money, IMHO.

Of course there is the possibility (probability?) the business doesn’t actually understand what it is they actually do anyway. This is not an uncommon position for many businesses that are happy to cruise along in neutral making some marginal profit on some marginal activity. Usually these businesses are also found to be beating their workers with sticks (usually only metaphorical ones unless they ‘offshore’ their operation to countries where killing your workers is just a part of ‘Business as Usual’. Typically they hold that marginal process can be made ‘better’ simply with just more exhortation (or threats) to greater and greater efforts at a totally demoralized (if not actively hostile) workforce, but I suspect that’s a story for another day!

In recent times I’ve seen this often enough that I think it’s really a failing of the IT industry in general, and we need to educate business IT users about the various scenarios and categories of software.

The simplest analogy I can think of is to say the website or portal is the letter, and the platform is the word processor.  Regardless if you use Word, Wordperfect, Pages 09, your email program or just plain old ‘notepad.exe’, at the end of the day the time to write the letter is pretty much the same effort and therefore the task is basically identical. If you said “I want you to help me to write a letter to my member of parliament”, should  I ask you whether you’re using the new letter wizard in Word? Have you seen this great “clippy” feature? Have you considered the new upgrade to Office 2007? Tell you to buy a Mac? Or would I be better off asking who is your member of parliament and what’s the matter about? When we get these sorts of naive clients we need to concentrate their minds on what their actual problem is and the best way we can solve it, when they’ve got their head in the sand thinking about that great drag-and-drop wizard feature the vendor showed them they totally thinking about the completely wrong thing.

The repetative (sic) steps that a single worker performs on an assembly line is a good example of a simple task

This is a terribly Fordist/Taylorist view to take of assembly work. That most beloved of industrial examples, Toyota, takes weeks to train its process-line workers to be able not only to do the simple repetitive steps of assembly, but also to be able to identify and rectify problems with the assembly-line. As Steven Spear, author of Chasing the Rabbit (my review here) might say, anyone who thinks that “assembly line” work is simple task probably doesn’t understand the nature of their process enough to keep up with their more innovate competitors, let alone outdistance them. Understanding your process, well more importantly, your organisational goals and how your process does or does not fulfil those goals, is critical to improving your organisation, and it leads me onto the idea of “evangelism”.

I both agree and disagree with Coder Friendly’s position:

Don’t become an evangelist yourself : look at your project and decide which methodologies is the most adapted. Sometimes, Agile is not the right one but if you see that your requirements are far from agreement or certainty, Agile approaches are probably a good choice.

I agree that “evangelism” can be politically off-putting.People don’t react well to being told their old kool-aid contains poison, and to uncritically drink this new kool-aid instead. However the main problem I have with a statement like this is that non-Agile companies use this “adaptation” argument and approach, in order to keep their poor contexts alive. This is the fallacy that’s found in ‘scrum butt’: “Oh we do Scrum, but …”. The clauses after the ‘but’ are typically critical. The organisation concerned is usually trying to keep some essential organisational dysfunction alive, because of politics, resistance to change, simple misunderstanding, lack of understanding about their own process, or maybe even simple laziness. I’ve got no problem with an organisation that is mature in its use of an agile method adapting it to their local circumstance – in fact if you are doing it properly, with its emphasis on empowering teams to own their process and carrying out regular ‘inspect and adapt’ at the end of every iteration, after six or twelve months of this the process will most certainly be adapted to the local conditions.

However, the real danger is at the start of the agile adoption process, when you have people hand-waving about “can’t do that here”, for whatever value of that there is. I’ve seen it said about: pairing; story boards; acceptance criteria; automated testing; retrospectives; daily stand-up; adaptive and emergent design; and just about every other aspect of XP or Scrum process. The reason is, I think, that these organisations rarely understand what the real organisational goals actually are, and they understand even less about their existing process, to the point they just “do” the process that “is”. “It is what it is”, is a great term, and if anyone says this to you it’s just a fatalistic attempt to accept as unchanging something you can change. Accept the bad. Don’t think it could be better.  Attempts to meddle in the process, i.e. change it, induce fear, and with fear often comes timidity.

This is where you need a real agile evangelist. This is not someone who just thinks that Scrum or XP or whatever other process should be blindly substituted for the existing one, but someone who understands that process is about competitive advantage, eliminating waste, and that you have to start from a known position and continuously work towards improvement. Change is not just good, but necessary, and you need those change agents. Passion for doing things better is bad? Only in a change-averse company. So learn your existing process, thoroughly, and then learn the proposed agile process, thoroughly, and only then will you be able to ‘synthesise’ a replacement. Only I think you’ll find that replacement will look a lot more like the plain old agile process than your existing unagile process. After all, if your existing process already has adequate inspect-and-adapt cycles within it (not lip-service ones, real ones), probably you are already doing and agile or lean process even if you don’t call it that, and you’re already number one in your marketplace. If not, perhaps you require the passion of an evangelist.

Agile requires the discipline to move projects through teams. Multitasking is nuts in agile. Moving team members around to have the “best” specialist available for a particular team is nuts. Performance reviews for individuals is nuts. Managers have to change everything they do, if they want to move the organization to agile. Managers need to see the problems exposed by any given project’s transition to agile and work to remove those obstacles before transitioning to agile.

It’s too easy to think that just because you can get one of your teams to use an agile approach, that your organisation is now Agile. Too many managers “see the problems exposed by given project’s transition to agile” and decide to blame the agile process for the problems thus exposed.In far too many organisations I’ve seen have extremely capable teams delivering quality software in an agile process but remain – in some cases years later – completely unsupported by their senior management (and in some cases with very little support even from their CIO level).

That’s a very sad situation as it is and takes all your selling skills to get the management to “get with the program” so to speak. But this situation can be even more compounded when the management have been convinced that Agile is some sort of silver bullet that will cure all their issues. Whenever you are trying to sell agile to management, never waver from the fact that agile is hard (as everything worth doing is nearly always difficult anyway!). Agile takes discipline.

Johanna’s right to say you can’t adopt agile across an entire organisation in a big bang approach. But you can’t also adopt it without skill and commitment to doing so from managers and executives.

On the other hand I have also worked with lower-level product owners who simply could not quantify the ROI for any particular feature. When they asked for (effectively inessential) features which doubled the size of the effort required they would have a meltdown (i.e. emotional pressure to get us to ‘magically’ reduce our estimates) rather than admitting that perhaps we could delivery 95% of the business value and nearly 100% of the projected project revenue  with only 50% of the effort.

We once proposed a software feature that would vastly increase the efficiency of a business’s payment handling, eliminating several costly manual steps. We thought we were doing good. But … Our Bad!!! Our Very Bad. The very inefficiency of the process meant that the business in question could hang onto the revenue – 90% of which was passed from consumer to ultimate supplier – for a much longer period. It added days if not weeks. This is very very good from the CFO’s point of view. Much more money earnt on interest earnings and inflating the cash-at-hand than cost-base savings that could be made making a process “efficient”. Leave it alone and don’t ask about it again was the general message!

In a similar view, many years ago I was a programmer at the technology arm of a futures broker; rather than making most of its money from brokerage or its own trading activities (or software technology), in fact the best and most consistent revenue came from the interest earnt on aggregating the client’s trading accounts (which had a minimum deposit level, and of course, did not pay interest to the client).

Sometimes you work for a business, and their business isn’t really what you think it is.

A few people seem to be sweating about MySQL. It would not be stressing about MySQL too much. It could get spun off, who knows. It’s even possible, as some commenters on Techcrunch say, that Oracle might use the free MySQL offering to hammer Microsoft’s database market from the bottom-up

It’s more touch-and-go in the app-server market (what Oracle likes to call ‘middleware’) which is already suffering a little from Oracle’s transition over to BEA Weblogic from its older products. Sun has excellent products in that area, i.e. Glassfish. I have used all of these products (plus Websphere and JBoss) and Glassfish is easily the nicest (in fact I would say it’s the best app-server I’ve used, apart from plain old Tomcat).

And what will happen to Java? Of course, Oracle wants Java, that’s part of the reason they are buying Sun in the first place (as well as their hardware business). But will IBM play along now it’s most critical competitor owns Java (and IBM has previously bet its software integration farm on the Java stack)? And what of the JCP?

Even more worrying for some, is what will happen in the IDE space. Of course, I’m a confirmed Eclipse man, but it is always a worry when competition is reduced. What will happen to Netbeans? And dare I say it … JDeveloper is fairly horrible compared to Netbeans but will that save either of them? Oracle’s got a lot invested into it’s tooling, which all runs on JDeveloper. As much as I prefer to just write code, that use wizards, Oracle does seem to have at least some customers in that area. Oracle’s development model tends to focus a little too much on magic wizards. From the IDE-JDeveloper-to-the-app-server single click-to-deploy, drag-and-drop to create-the-control, easy-peasy wizardry, which I hate, because I think it gets in the way of craftmanship. It’s the Bunnings Warehouse of software development. Show me the command-line Oracle! you should remember, that Sun is most definitely a company built around the Unix shell prompt. Even with Oracle middleware, it’s definitely possible, I’ve just been working on command-line deployment automation for their older Orion-based app-server, they just don’t like to promote it.

I’m hoping that Sun can reverse-ferret them on that score and teach them about Ant, Maven, /bin/sh and the goodness of installations that are as simple as tar xcf in the spot you want to install it.

Consider the following scenario.

1. You’ve got two parts of the code base. One part is beautiful code, written TDD, used continuous integration, pair programming, well written stories were used to produce it, clear acceptance tests, all the good stuff. The other part is horrific, hardly any tests, when the build failed, you disabled the tests that failed, the stories were written as technical tasks, acceptance tests vague, etc.
2. You are asked to deliver two medium-value stories in the next iteration. One story is most done in the good part of the code base. The other story in the bad part.
3. Correspondingly, your estimates will vary according to the complexity of the code base – the story in the “good” code might be 5 points, the story in the “bad” code is 15. But the value to the business delivered is the same!

The point is, the amount of effort that you do (20 points) to deliver 2 medium value units, is actually not a measure of your progress. It’s a measure of your waste. The next iteration you might be able to deliver 4 medium value points, if they are all in the “good code” base! Same effort, twice the return on value. Software developers know this about well designed code bases combined with good engineering practice, of course. The point is about directly exposing the effects of bad engineering as it manifests itself as a barrier to progress.

What if you told the business instead, you’ll deliver x business value points instead? That is, your burn down and burn up charts don’t show story points of effort expended, but directly indicated value delivered. What impetus does this lead us to? Well, now we’ve got a direct measurement of story points over business value, which is the proportion of waste to value. The ultimate story is infinite in value and takes zero points to deliver!

I think this is not all; because it directly establishes the ratio of waste to value, it gives you an impetus to eliminate waste, that is, to practice good, professional, software craftsmanship of the type that Robert C. Martin talks about in the video I linked to earlier tonight. Software that allows you to change efficiently to meet new needs now becomes not an abstract principle that you must kick back at your managers every time the demand to see “more features” and less engineering, but becomes a direcly correlated measurement of productive output.

It’s not all plain sailing, for the really hard nut has to be cracked. How the hell do you measure business value in units like “points”? I don’t have a cheap and easy answer right for it right now. But I don’t think this very hard question should go unanswered. For me, it’s the missing piece of the agile software delivery puzzle. I’ve been asked countless times, to estimate effort on whole swathes of stories where I think it’s obvious that the business value is actually highly dubious – or at least cannot be quantified by the product owner. On one level, that’s fine, they pay the bills and get to set the priorities, but on another level, I think that having to produce a quantifier for the value will help the product owner focus their minds onto the outcome in a much better way.

And it enables them to directly see the correlation between bad engineering practices and the inefficiencies they engender. Those bad practices are certainly our fault for putting them in place, don’t get me wrong, I’m not shirking our personal and professional responsibilities in that regard. However reducing outside pressures will certainly also help by giving us a little breathing space to demonstrate with hard numbers the value of taking care with our craftsmanship. And it reports the thing the business are most interested in – the value you deliver.

If my post is a little unclear, forgive me it’s late at night as I write this. Better blog entry than code in the tired state I’m in!

Sprint to the lead in your industry – and stay there!

So says the back of the dust cover on Chasing The Rabbit: How market leaders outdistance the competition and how great companies can catch up and win by Steven J Spear (McGraw Hill, New York, 2009). I referenced this book last week in my post ‘Let it fail then learn to succeed’ which is about using failure as a signal for process improvement.

It’s a great book. Obviously it is not addressed directly to software development, and as I said, Spear covers its application in a range of different settings, including engineering and health-care as well as manufacturing. There’s a distinct lack of focus on the buzzwords kanban, kaizen, and just-in-time, all of which Spear says are merely artifacts of a deeper underlying approach that he proceeds to illustrate extremely well in the course of the book.

The approach might be illustrated by the asking of some fairly simple questions: ‘What did you do?’, ‘What happened then?’, ‘What did you expect to see?’, ‘Was it successful?’, ‘What did you do to correct it?’, ‘What did you learn?’ (my list is not exhaustive). Spears characterises it as:

… a systematic approach to designing and operating systems, a simple set of rules for problem solving and improvement, a clear way to share learning, and a well-defined role as a manager. (p 358)

More formally, the organisation exhibiting these qualities shows two characteristics and four capabilities.

Two characteristics:

1. Managing the functions as parts of the process. A high-velocity organisation orientates its processes away from functional ‘silos’ (without ignoring the necessary competency of individual functions), and towards “the way the work of individuals, teams and technologies will contribute to (or impede) the process of which they are part” (p 20). For example, Spear describes his own experience while at Toyota: he is asked what a particular factory he is being trained at makes. At first he interviews the management of the factory, and returns with a list, only to be told he doesn’t really know. Then he looks at invoices of the shipped items, which is still said to be wrong. In response to this, he examines what the receiving factory actually paid for, but he is still told he doesn’t know what the factory makes. Finally, he stands at the shipping dock, as as each shipment was sent out, ignoring the shipping label, took a note of the part number stamped onto the part. Then he checked that the factory actually possessed the die for stamping the part number. In response, he was told; “Well, that’s probably not wrong. But I have another question: How are these parts made?” (p 280).
2. Continually improve the pieces and the process. High-velocity organisations are always learning about all the work they do. Workarounds, tolerating problems, putting out fires, and heroic efforts are simply not tolerated. Every time you have to put out a fire, or devise a workaround, or engage in a heroic measure to get work done, you’ve found a process problem that should be eliminated. Your highest priority isn’t engaging in the workaround, or performing heroic efforts of late-night last-minute production delivery effort. Your highest priority, and that of your management, is swarming the problem to fix the process that led to the situation. Doug South described it to me as “not tolerating things”. Instead you should fix their causes rather than developing a workaround to the immediate problem.

Four capabilities:

1. Specifying design to capture existing knowledge and building in tests to reveal problems. Make explicit the most effective approach currently known to you to successfully achieve the task. Make sure that the approach has the capacity to detect failure as soon as it occurs. You need to be sure what outcomes you expect, not from a “perverse Taylorism” (p 23) but as an investment. By specifying expectations, you can more easily capture when something unexpected happens – that is, what the gaps in your current knowledge are. Go out of your way to build in these tests. Alcoa for example, focused exclusively on improving safety – with the goal of zero injuries, because this “meant perfect processes based on perfect knowledge of how to do work” (p 94). The test therefore, was the occurrence of workplace accidents. When an accident occurred, it highlighted a dangerous ignorance, and had to be rectified.
2. Swarming and solving problems to build new knowledge. When you encounter problems, at the time and place of the encounter, the problem should be swarmed. Contain the problem to prevent it from spreading. Diagnose and treat the cause so the problem cannot reoccur. Problem-solving skills are built by solving actual problems, and those doing the work are responsible for improving the process by which the work is done. Doing this will allow you to “build ever-deeper knowledge about how to manage the systems for doing [your] work, converting inevitable up-front ignorance into knowledge” (p 24). For example, at Alcoa, it was made a rule that business-unit presidents had to not only inform the Alcoa CEO within 24 hours of an injury or injury near-miss, “but within two days they had to report what the initial investigation had revealed about its causes and what was being done to prevent the problem from recurring” (p 96). This ensured a discipline, a warp-speed cycle involving real-time problem recognition, diagnosis and treatment.
3. Sharing new knowledge throughout the organisation. It’s not just that you learn. It’s that the entire organisation learns. Toyota did this for a joint-venture plant it created with General Motors. Toyota’s approach was a one-variable-at-a-time tuning. The American plant would make an existing model and not an all-new car. Rather than creating an entirely new plant, or swarming the old GM plant with Toyota managers, it took newly hired American managers and coached them in the Toyota way. They didn’t just drop them into the plant. They first made similar products (Corollas) at a Toyota plant. Then they went back to the American plant and Toyota gave the new managers continuous support from Japanese experts. The joint-venture plant was far more successful than the old General Motors plant it replaced. In 2003, in order to be abale to accelerate organisational learning, Toyota created the Global Production Center, which was staffed with experienced cadres of trainers in areas identified as critical to production success. They developed standardised practice equipment and had other groups test the new equipment to identify problems with it. And because these practices are orientated around the discovery of solutions via practical experimentation, and rapid cycles of concepts and validaty testing, they can “smoothly converge on solutions that were agreeable across disciplines” (p 253). Having done all this, it shared it’s newfound knowledge with representatives from factories around the world, and trained managers and group-leaders who were setting up a similar centre for Toyota’s North American operations. Just as important as creating new knowledge and sharing information about particular capabilities, it’s just as critical to spread problem-solving capability throughout the organisation.
4. Leading by developing the first three capabilities. Managers are not there to measure, berate, and exhort their workers in a command-and-control environment. The role of management is to ensure their organisation becomes “ever more self-diagnosing and self-improving, skilled at detecting problems, solving them, and multiplying the effect by making the solutions available throughout the organisation” (p 26). The process of finding out what the plant made which I outlined above was part of learning this fourth capability by observing a system at four levels – output, pathways, connections and activities.

At the end of the book is an illuminating conversation about golf. Now personally I can’t stand golf but I suppose lots of middle-aged businessmen do, and the conversation was about the simplicity of the game. Despite protestations about bunkers, traps, and hazards, roughs and fast greens, and how difficult it really is, in the end it’s actually a simple game with very simple rules. Keep hitting the ball with a club until you get it into the hole. Simple rules that require a lot of practice to do it right. And like golf, making your organisation high-velocity might run with a fairly simple set of rules, but they require a great deal of practice at the various skills needed to make it happen.

That gives a general overview of the content of the book. It’s written in a very accessible style which communicates the essential concepts very well without trivialising them. If you are a software developer who is interested in thinking about development process, and general business process, then I definitely recommend this book. Sometime in the next few weeks I will try to produce a further post relating some of these specific concepts that Spear has identified and apply them to software development process. In the meantime, buy and read this book.