Recently I was learning ANTLR, trying to decide whether this was a right technology to pursue a particular project which involved parsing a preexisting message format. After a week of a spike, we decided that it was worth pursuing and started on earnest on the grammar for our project. However a week into this process, I had an epiphany … I was doing some things wrong with the ANTLR grammar which were now slowing progress in adding the new characteristics it needed to be complete. Many developers know this feeling; the features of my grammar that I had built over the first week were naive and now hampering it from expanding into the new requirements. I took it on myself to kill the entire grammar and start again. It took less than a day and half to replicate that week’s worth of work (i.e. pass the test suite which had built up around it).  I’ve done this before; scrap the first attempt at building a domain and try again. Here my domain was the same (it was after all defined in both a standards specification and in the many hundreds of thousands of sample messages we captured from an existing system), but its implementation needed refinement.

So I think that the rule about throwing away spikes can in fact be made a general axiom of programming:

When you are learning a new technology, make sure you throw away the first thing you build that works – to avoid accumulating your mistakes.

Thanks to Robert for the important qualifier “that works”.

N.B. my views about system rewrites have not changed regardless.

Lost in the shuffle is something basic: Programmer productivity. Friends who are hands-on with such matters tell me the available SOA development environments are less than half as productive as products like PowerBuilder and Delphi were, back when they were viable.

Putting aside the Powerbuilder and Delphi love for just one minute, this is something I’ve been banging on about now for the past year … the programming tooling that is foisted onto programmers by the choice of the deployment architecture. It’s just all wrong.

In my view, what makes a programming language really productive is notepad. Or vi, or emacs, or gvim, or textmate, take your pick. What I mean is … the programming language has to be able to be programmed with a simple editor. Yes, an advanced IDE will make things more productive, but the basics must also apply. Now a lot of SOA environments are simply not programmable without the specific IDE tied to it. Even worse, the IDEs are often completely custom jobs that require a developer to be re-trained … losing years and years of productive speed with muscle-memory style automatic ability to navigate the programmer’s usual editing tool. Seriously. This stuff is whack. A program language or an environment needs to be IDE-neutral. If you got a plumber around, would you insist that he only use the tools you supply from your home handyman kit? Or would you expect the plumber to have mastered a set of his own tools already? And making matters worse, it’s rarely programmers that choose these tools which are foisted on them. The server/deployment environment and the language used to implement need to be decoupled from the tools used to build it.

But an even worse failing of many of these SOA tool suites, is that they all strongly and irrevocably coupled to the deployment/runtime environment. Generally they totally lack the ability to keep up with modern programming practice. Like for instance, automated testing. Or even unit tests, let alone advanced and productive techniques such as Test-First approaches or Test Driven Design. The tools often lack refactoring support. All of these things are in my opinion, and in the opinion of many leading developers, absolutely essential to quality engineering practice and agile development outcomes like “delivery of working software”. Both the “delivering” and the “working” part means the whole process needs to be repeatable. That’s why automated integration testing, to name just one thing, is essential in modern development. But often the fancy custom development tooling is a complete barrier to achieving this.

But you won’t hear any of this from the big vendors. One big vendor recently announced their new version of their middleware product suite had a ‘focus on testability’, but you ask any of their presales guys to demonstrate this in an actual development environment. Ask them about continuous integration support, for example. Witness their blank looks. Their development product is completely orientated to “one button push from the IDE to production” modes of thinking the idea of continuous integration builds is almost totally antithetical to the very concepts of operation the product is organised around. They think that finally adding support for Subversion version control system, at least five years too late, is a wondrous achievement.

They are aiming for ‘programmerless programming’: of course in process just creating a new type of programmer. Every new generation of programmers simply have to learn the same hard-fought lessons of software engineering over and over again because each generation of tooling apparently scraps the paradigm over and over again in a vain attempt to create push-button, wizard-driven programming models. They nearly all suffer from ‘hello world’ programming – the simple examples that sell them to IT management are trivial to conquer using the wizards, but more complex problems (i.e. real world ones) are flat-out impossible. Thus these tools are always mirages which look great at a huge distance on the horizon but are flat lifeless salt pans of bleached skulls and bones on closer examination (or, maybe they are more like tar pits that look like a nice waterhole but one step into it and you are sucked down  to your doom).

As you might be able to tell, I am utterly contemptuous of many of these SOA tool paradigms. I have nothing against SOA itself, per se. But there is nothing more productive than a programmer who understands the importance of simple and repeatable build and deployment automation using command line tools and who knows his programming editor inside out after ten years of use. Give that programmer a better language by all means, add incremental features to that IDE, allow the programmers to continuously improve their techniques, promote professional craftsmanship, yes, yes and a thousand times yes. But no amount of drag and drop wizards, push-button deployments, and “object inspector” property editors will ever usurp that deep knowledge a good programmer brings to both his language and his personal tooling choices.

Link: http://www.viddler.com/explore/sergiopereira/videos/7/.

Sprint to the lead in your industry – and stay there!

So says the back of the dust cover on Chasing The Rabbit: How market leaders outdistance the competition and how great companies can catch up and win by Steven J Spear (McGraw Hill, New York, 2009). I referenced this book last week in my post ‘Let it fail then learn to succeed’ which is about using failure as a signal for process improvement.

It’s a great book. Obviously it is not addressed directly to software development, and as I said, Spear covers its application in a range of different settings, including engineering and health-care as well as manufacturing. There’s a distinct lack of focus on the buzzwords kanban, kaizen, and just-in-time, all of which Spear says are merely artifacts of a deeper underlying approach that he proceeds to illustrate extremely well in the course of the book.

The approach might be illustrated by the asking of some fairly simple questions: ‘What did you do?’, ‘What happened then?’, ‘What did you expect to see?’, ‘Was it successful?’, ‘What did you do to correct it?’, ‘What did you learn?’ (my list is not exhaustive). Spears characterises it as:

… a systematic approach to designing and operating systems, a simple set of rules for problem solving and improvement, a clear way to share learning, and a well-defined role as a manager. (p 358)

More formally, the organisation exhibiting these qualities shows two characteristics and four capabilities.

Two characteristics:

1. Managing the functions as parts of the process. A high-velocity organisation orientates its processes away from functional ‘silos’ (without ignoring the necessary competency of individual functions), and towards “the way the work of individuals, teams and technologies will contribute to (or impede) the process of which they are part” (p 20). For example, Spear describes his own experience while at Toyota: he is asked what a particular factory he is being trained at makes. At first he interviews the management of the factory, and returns with a list, only to be told he doesn’t really know. Then he looks at invoices of the shipped items, which is still said to be wrong. In response to this, he examines what the receiving factory actually paid for, but he is still told he doesn’t know what the factory makes. Finally, he stands at the shipping dock, as as each shipment was sent out, ignoring the shipping label, took a note of the part number stamped onto the part. Then he checked that the factory actually possessed the die for stamping the part number. In response, he was told; “Well, that’s probably not wrong. But I have another question: How are these parts made?” (p 280).
2. Continually improve the pieces and the process. High-velocity organisations are always learning about all the work they do. Workarounds, tolerating problems, putting out fires, and heroic efforts are simply not tolerated. Every time you have to put out a fire, or devise a workaround, or engage in a heroic measure to get work done, you’ve found a process problem that should be eliminated. Your highest priority isn’t engaging in the workaround, or performing heroic efforts of late-night last-minute production delivery effort. Your highest priority, and that of your management, is swarming the problem to fix the process that led to the situation. Doug South described it to me as “not tolerating things”. Instead you should fix their causes rather than developing a workaround to the immediate problem.

Four capabilities:

1. Specifying design to capture existing knowledge and building in tests to reveal problems. Make explicit the most effective approach currently known to you to successfully achieve the task. Make sure that the approach has the capacity to detect failure as soon as it occurs. You need to be sure what outcomes you expect, not from a “perverse Taylorism” (p 23) but as an investment. By specifying expectations, you can more easily capture when something unexpected happens – that is, what the gaps in your current knowledge are. Go out of your way to build in these tests. Alcoa for example, focused exclusively on improving safety – with the goal of zero injuries, because this “meant perfect processes based on perfect knowledge of how to do work” (p 94). The test therefore, was the occurrence of workplace accidents. When an accident occurred, it highlighted a dangerous ignorance, and had to be rectified.
2. Swarming and solving problems to build new knowledge. When you encounter problems, at the time and place of the encounter, the problem should be swarmed. Contain the problem to prevent it from spreading. Diagnose and treat the cause so the problem cannot reoccur. Problem-solving skills are built by solving actual problems, and those doing the work are responsible for improving the process by which the work is done. Doing this will allow you to “build ever-deeper knowledge about how to manage the systems for doing [your] work, converting inevitable up-front ignorance into knowledge” (p 24). For example, at Alcoa, it was made a rule that business-unit presidents had to not only inform the Alcoa CEO within 24 hours of an injury or injury near-miss, “but within two days they had to report what the initial investigation had revealed about its causes and what was being done to prevent the problem from recurring” (p 96). This ensured a discipline, a warp-speed cycle involving real-time problem recognition, diagnosis and treatment.
3. Sharing new knowledge throughout the organisation. It’s not just that you learn. It’s that the entire organisation learns. Toyota did this for a joint-venture plant it created with General Motors. Toyota’s approach was a one-variable-at-a-time tuning. The American plant would make an existing model and not an all-new car. Rather than creating an entirely new plant, or swarming the old GM plant with Toyota managers, it took newly hired American managers and coached them in the Toyota way. They didn’t just drop them into the plant. They first made similar products (Corollas) at a Toyota plant. Then they went back to the American plant and Toyota gave the new managers continuous support from Japanese experts. The joint-venture plant was far more successful than the old General Motors plant it replaced. In 2003, in order to be abale to accelerate organisational learning, Toyota created the Global Production Center, which was staffed with experienced cadres of trainers in areas identified as critical to production success. They developed standardised practice equipment and had other groups test the new equipment to identify problems with it. And because these practices are orientated around the discovery of solutions via practical experimentation, and rapid cycles of concepts and validaty testing, they can “smoothly converge on solutions that were agreeable across disciplines” (p 253). Having done all this, it shared it’s newfound knowledge with representatives from factories around the world, and trained managers and group-leaders who were setting up a similar centre for Toyota’s North American operations. Just as important as creating new knowledge and sharing information about particular capabilities, it’s just as critical to spread problem-solving capability throughout the organisation.
4. Leading by developing the first three capabilities. Managers are not there to measure, berate, and exhort their workers in a command-and-control environment. The role of management is to ensure their organisation becomes “ever more self-diagnosing and self-improving, skilled at detecting problems, solving them, and multiplying the effect by making the solutions available throughout the organisation” (p 26). The process of finding out what the plant made which I outlined above was part of learning this fourth capability by observing a system at four levels – output, pathways, connections and activities.

At the end of the book is an illuminating conversation about golf. Now personally I can’t stand golf but I suppose lots of middle-aged businessmen do, and the conversation was about the simplicity of the game. Despite protestations about bunkers, traps, and hazards, roughs and fast greens, and how difficult it really is, in the end it’s actually a simple game with very simple rules. Keep hitting the ball with a club until you get it into the hole. Simple rules that require a lot of practice to do it right. And like golf, making your organisation high-velocity might run with a fairly simple set of rules, but they require a great deal of practice at the various skills needed to make it happen.

That gives a general overview of the content of the book. It’s written in a very accessible style which communicates the essential concepts very well without trivialising them. If you are a software developer who is interested in thinking about development process, and general business process, then I definitely recommend this book. Sometime in the next few weeks I will try to produce a further post relating some of these specific concepts that Spear has identified and apply them to software development process. In the meantime, buy and read this book.

“Oh, my God! You’re going to let a sprint fail, just so you can determine the optimum length?” Yes.

In other words, failure lets you learn your limits. But more importantly, as suggested here, is that failure can be deliberately introduced in order to improve the process – to decrease your limitations. I would say, for an organisation committed to process optimisation, you should shorten the sprint until it fails, but rather than just accepting that limit, learn why it fails and then experiment with optimising it until it succeeds. Let me explain.

Courtesy of a tip from Doug South at the Brisbane XP user’s group, I’m nearly finished reading a book at the moment, Chasing the Rabbit by Steven J. Spear of MIT and formerly of Harvard. The book’s full title is Chasing The Rabbit: How market leaders outdistance the competition and how great companies can catch up and win. It’s a business book that describes the essential feature of successful companies such as Toyota and the Toyota Production System.

Before you roll your eyes and think I’m just another one of these lean software guys who’s about to rave over kanban, kaizen, and just-in-time, and applying manufacturing production-line process, possibly inappropriately, to the professional practice of software development, let me assure you I’m not. Neither does Spear. Basing his book on years of hands-on research into these high-velocity organisations (including time spent actually working on real production lines) he characterises kanban and friends as merely an artifact of a deeper philosophy towards building a successful company. In some ways its breathtaking simple, although hard to practice. The theme of the book is discovery.

The factory was not only a place to to produce physical products, it was also a place to learn how to produce those products and – most important of all – it was a place to keep learning how to produce those products. In fact, this is exactly what so much of the early research about Japanese management had revealed – that learning and discovery were intrinsic to success. But that idea had gotten lost as people focused on the particular tools and artifacts used in the workplace at the expense of understanding the principles of how those systems where managed. (p 15).

These companies don’t succeed or fail. They succeed or learn to succeed.

Now it is of course slightly more complex that just that. There are two characteristics and four capabilities that high-velocity organisations exhibit. The four capabilities are:

1. Specifying design to capture existing knowledge and building in tests to reveal problems.
2. Swarming and solving problems to build new knowledge.
3. Sharing new knowledge throughout the organisation.
4. Leading by developing the first three capabilities.

Now these capabilities can be developed – in different manifestations – in any sort of process, not just production-line processes. Spear shows it in action in health-care, in nuclear engineering, in aluminium production, and of course automotive production. I’ll into further detail perhaps in a later post. Here I’m going to discuss a simple example illustrating not just capability 1, in which a process not only captures knowledge but has built-in tests, and also capability 2, involving problem solving and process improvement, where the process is tested to failure, in order to reveal what knowledge is missing about the process.

In the book he describes a mattress factory, Aisin, that, after a long capability improvement process ends up with two production lines each capable of making (e.g.) 100 units a day to custom order. On some given day, imagine they have a requirement to make say 180 units. Rather instead of running one line at 100 the other at 80, or both at 90 units, they will instead deliberately overload one of the lines (e.g. ramp it to 110 or 120 units) in order to discover the process failure modes (that is, the process’s bottlenecks) so as to to improve them. They already know how to make 100 units a day. They can learn more about their process if they try to make 120. The reason they do this on days of lighter loading, is that the can use the unstressed production line as a back up, if the test was too stressful for the line under test.

Experienced software engineers might recognize this stress testing from their performance testing toolkit. Loading a software system to the point of failure is one quick way to learn where a system’s bottlenecks are, and what to do about eliminating them, and thus making your software architecture more efficient, and discovering new knowledge about how your architecture performs and what needs to change.

The point is of course, I’m not talking about stress testing software. I’m talking about stress testing software development methodology so you can discover what you don’t already know about your process, and therefore improve it. I don’t mean, just arbitrarily place more demands on your software teams to produce more with less. That’s just the most retarded type of management-by-objective.

Think about all the moving parts in your development process. What do you know about them? What do you assume? What do you just take for granted? How much do your team members know about the process? Can they identify problems and propose solutions? Do they need to learn that capability? Where are your failure modes? Do you just accept them, and perform a work around, or do you understand them as signals to elicit further discovery about the things you don’t understand about the process? Embrace failure. In fact, actively seek it out. Use it an opportunity to discover unknown information about your business and to improve your team’s skills.

Chasing The Rabbit is a great book, highly recommended that all software development practioners read it, and special thanks to Doug for bringing it to my attention.

Anyway it turns out that Kent is writing a product called JUnit Max which looks super interesting for those using the discipline of TDD to write better code. Look at the Flash presentation on this page: http://www.threeriversinstitute.org/junitmax/subscribe.html … Also it brings together another thread we had on the mailing list which was about Eclipse’s incremental compilation features. This is incremental test running!

A pity it’s not free but I guess Kent’s got to eat. I might be tempted to buy it in the near future.

I might agree that for simple enough systems, it could end up being that in fact the service layers end up with PersistenceManager logic directly in their methods. But then I might say a language and framework like Ruby on Rails could suit your application better in that case. For any sufficiently complex Java-based enterprise system, a POJO-style data persistence layer naturally evolves, in order to reduce as far as possible the responsibilities the service layer must carry out.

As mentioned by commentators on his blog, the discipline of test-driven design is usually  a strong driver of having a separated set of components that deal with persistence and other data concerns. Not just from the issue over mocking the PersistenceManager, which is certainly a strong driver, but also from design principles.

Service layers marshall requests from clients, and decompose them to call various component layers. Data access layers typically form part of these component layers. A TDD discipline means these data access layers usually end up with more than just CRUD operations.

Furthermore, as a matter of discipline, in any sufficiently complex service layer I prefer not to work directly with database abstractions (i.e. entities that are modelled on database tables and their relationships) but on first-class objects designed to fit directly with the modelled domain. I want those objects to have rich behaviour – the service layer is actually part of their behaviour. I want to start the design of the system by directly modelling those actions, e.g.

passenger.at(origin);
passenger.orderJourney(dispatch, destination);
Vehicle vehicle = dispatch.sendVehicle(origin);
vehicle.pickUp(passenger);
vehicle.arrives(destination).dropsOff(passenger);
vehicle.notify(dispatch);

I do not want these objects to know about database semantics. That is a responsibility of a separated persistence layer, which we may for convienience sake, called the DAO. Even if internally we end up with;

class Passenger {
  at(Location origin) {
    this.origin=origin;
  }
  call(DispatchCentre dispatch, Location destination) {
    dispatch.passengerOrder(this, destination);
  }
}

DispatchCentre at this level, is a service, but we will model it as part of the domain of passengers, locations, and vehicles. Behind that service, it will have a database layer with the semantics as to how it saves and loads the data concerning passengers, locations, and vehicles. I don’t necessarily want the client objects (e.g. the web action) to have to deal with JPA annotated objects. They just deal with objects as much as possible. And I don’t see there’s anything served with developing possibly complex persistence logic directly in the DispatchCentre. In fact, taking a top-down approach, I know pretty much I won’t do that.

Only after I’ve done this initial part of the modelling, will I be concerned as to which of these components can be directly JPA or Hibernate annotated, which are Spring-injected services, which are Session EJBs, and the like. I think starting your modelling from the point of view of having a persistence framework like JPA or Session EJBs or any other ‘archtectural artifact’ leads to an unnatural design.

I was riding my exercise bike, listening to Stack Overflow #38 when I heard Jeff Atwood and Joel Spolsky say “Quality just doesn’t matter that much.” I nearly fell off my bike.

(There’s a transcript of part of the podcast on Joel’s blog “Joel On Software” if you don’t want to listen to it).

Of course I’m going to say here that I completely agree with Uncle Bob. The more extraordinary parts of the actual podcast come along when Joel attacks the idea of TDD – test-driven design (although he refers to test-driven development):

But the real problem with unit tests as I’ve discovered is that the type of changes that you tend to make as code evolves tend to break a constant percentage of your unit tests. Sometimes you will make a change to your code that, somehow, breaks 10% of your unit tests. Intentionally. [ ... ] And you have to be able to go in and recreate those tests to reflect the new reality of the code.

This stuff is just a sheer obtuse misunderstanding about the very nature of TDD. Uncle Bob takes him to task for it I’m not going to repeat them here other to say I utterly agree with everything Mr. Martin has to say. Go and read the link above, it’s very effective and thorough pulling apart of Joel’s position on the matter.

I just want to talk about some other things, about the nature of software development, as against some of the things that Joel says is the best way to organise development and development teams. Joel just doesn’t seem to get agile engineering practices, and never has, but even  worse, he willfully misrepresents its disciplines and practices in order to demolish a straw man of his own making. The stuff above about TDD is clearly the prosecution’s exhibit A. He’s a respected development guru, and a lot of people trust what he says about code and the business of running a development sales organisation, which I admit, it’s reasonably sensible most of the time. However when he starts to talk about development practices and team organisation, while he says some things that are highly insightful, some of the others are full of horse manure. Or even worse than that, it’s 50% good advice, and 25% outright harmful advice.  There’s plenty of manure spread around by everybody else of course, me included, but due to his position as an exhalted spokesperson, and due to the wrongful nature of the things he says,  I find that his is particularly dangerous horse manure.

For example, all that stuff regarding “developers get offices of their own”. The best teams I’ve worked in are all in the one room, clustered on desks together and as noisy as hell. It’s usually best to put the whole team in a large office space of their own – so they don’t annoy the hell out of other teams. Joel maintains that individual developer offices are the only way – the illusion of the developer as a single genius toiling away in silence – rather than a member of a multifunctional, communicating, effective team. I disagree, and vehemently. Sure a big ‘TEAMWORK’ banner is just self-deception, but effective teams aren’t make effective through slogans and exhortations. Effective teams are made through actual action. If you have offices, they should be large team-based spaces where whole teams are co-located. And small temporary, hot-desk’d offices, if for good reasons, a developer (or better, a pair) need some quiet time to concentrate on solving some particularly hard problem. But mostly, delivering effective software is about team communication, not developer concentration.

Joel’s totally focused on the notion of the hero-developer.

There’s a bell curve of developer ability (like all such talent) which guarantees the vast majority of developers are “average”. Focusing your effort on finding only the top 5th percentile for your team is futile. Especially when you’ve already got a team you have to work with. Are you going to fire them all if they don’t come up to standard? Or just fire maybe the incompetant ones and work to improve the team. Sure, when you find you’ve got one of these “rock star developers” on your team, maybe you do have to treat him (or her) a little different to the others, to keep him (or her) motivated and interested at the tasks you offer. Nonetheless, 70% of your developers are going to fall within the standard deviation, almost by definition. An additional consideration is, you can get an absolute genius developer who is a complete social misfit and can’t work in a team very effectively. We all know (and maybe some of us are) this guy. At the end of the day sometimes these people are destructive to teams – reducing the teams productive capacity, if they can’t play nice with others. I’m not advocating some sort of we-are-all-mediocre approach and, yes, everyone’s an individual – but we do generally have to work in teams in order to achieve productive software development outcomes.

It’s the agile approach to value ‘people and interactions’ over ‘processes and tools’ but also notice it’s people, in the plural, and not individuals, and interactions implies work is multiplied when those people exchange information effectively and work together as a team. Information exchange bandwidth is greatest face-to-face, or better yet across entire teams. Private conversations that can’t be interrupted are only useful in the context of personal, and therefore truly private, information. Not team interactions. Any team member should be able to ‘stick their oar in’ on any matter being handled by the team – that’s how you get the best ideas! The idea of hierarchicalised, compartmentalised, information exchanges in the context of a private office sounds to me like an MI6 operation, not a software development practice.

Joel might know heaps about the micro-algorithms of manipulating data structures and carrying out instructions efficiently but ultimately it’s the macro-algorithms as to how software is effectively developed and how teams can deliver the best and quickest value to the customer that are the really hard problems. These are also the very problems that IT the whole world over have with the greater business world – how to prove to them we can organise, manage and deliver development effectively. Wearing a pointy hat and talking obtusely about pointer arithmetic and demanding expensive offices for all the prima-donna developers you’re going to employ to toil away in silence won’t convince the suits in charge of the actual money of anything other than you’re a crazy nerd who can’t be trusted with a budget.

Every few years a vendor, or a group of vendors comes out with some bright idea to allow some class of end-user, or business analyst, or what have you to build their own systems, in recent years usually by some type of graphic language (draw your system’s program flow right on the screen! bypass those pesky, and expensive, programmers who just ask confusing questions and force you to clarify your thinking!  press the magic ‘deployment’ button and your half-baked idea is in production in seconds!). In the past, these systems used “natural” language or “plain English”  languages such as say Director Lingo.  Every few years it moves onto some new paradigm or expression of the old one. And it always just ends up simply with a whole new class of programmers being created.

Do not trust any product – or standard – that promises to deliver “programmerless programming”. Plenty of 4GLs like SQL and the like are originally meant for non-programmers to use … imagine giving a business user just a SQLPLUS prompt now! Included in that mix there was CASE … a recent example from the past few years I can think of might be BPEL and ESB style tools, or for another example a number of attempts to define architectural languages that generate the system components for you off the UML or similar.

A big issue I see with these systems is that they tend to be highly resistant to modern programming principles and practices. For example, test-driven design – or even simple unit testing. Sure, if you’re building a web app with these things, maybe you could use a web-testing framework to build functional tests and build-to-pass. But most of these tools are very resistant to a test-first approach.

Take Oracle’s BPEL tooling. For a start, you’re forced to used JDeveloper, which is a crap IDE you’d never otherwise inflict on yourself (it is so bad, on start up it displays the most meaningless product slogan I’ve seen seen on such a tool – “Productivity through Choice”). But more than poor tooling, the very paradigm is completely broken. Forget about a test-first approach (although I’m sure there’s some seriously weird way to achieve it), just try setting the damn tools to have the traditional development, testing and production environments. When ever you need to call a service end-point, you are bound to the particular host and port you specify – it’s embedded in XML document that represents the BPEL design. The workarounds? Well if you are super organised you could arrange for your different environments to contain identical setups and  use DNS hacks (or host files) to ensure that each service looks the same when you move from development to integration testing to production. Or you can write Ant scripts (or sed and awk if you really want) to search-and-replace the offending hostnames and ports and replace them with the correct values for the target environment. Clearly, once you use it, you see that the tool’s paradigm is, in reality, to just hook up the IDE to the production server, draw your pretty flow chart modelling your business process, then publish it straight back into production. In other words, there’s a whole new class of programmer out there now – BPEL programmers – and life must be a nightmare for them. Its fundamental paradigm is broken. Given me any language with a unit test framework with a half-decent programmer on any day of the week, and you will beat the pants off this system in productivity and choice. It’s living a lie – and a big one.

Ultimately, as each new generation of “programmerless” program-construction tooling comes along, it allows developers to do more powerful things (and sometimes, less powerful things in a more retarded way). And the scale of the task always increases to the point where development specialists are still needed. In my view, there is a definite on-going role for for the talented programmer/analyst who knows how to take a vaguely-worded, incomplete, and contradictory bunch of requirements expressed in common verbal language and then ask the right questions in order to clarify them and decompose the requirements to a series of simple logical components – and then develop and deliver those components in the form of usable systems. You need to continuously learn new languages and systems, but the basic skills stay the same.

I have found too many code bases over the years, with bolted-on, after the fact tests, plagued by spaghetti code and strongly coupled components to be that naive about unit tests. Sure people pay lip-service to testing, but I think many programmers haven’t yet found the value of test driven design in their professional practice. I know enough that I come across this situation frequently. “Oh I’ll do the tests later”, “I don’t have time to write tests, I have to write code!”, etc. Heck, I am sometimes still guilty of it myself. Here’s some rolled-gold reasons Bain gives to pay particular attention to the discipline of unit testing in a test-first scenario:

• It reduces bugging time.
• It makes the development process more predictable – finding bugs often exhibits non-linear unpredictability.
• It encourages appropriate coupling, ensures cohesiveness, and reduces redundancy.
• It makes you think about your code in more depth.

I must say, even when confronted with an existing code base that has little to no unit testing, it’s always advantageous to start writing the tests before you start fooling around too much with the code base. It will help you discover the design weaknesses.

Another point I have to make about unit testing, is that whenever I need to evaluate a new technology,  I now use the ability to utilise test-driven methodologies as an important criteria. Especially if that technology is supposed to be a productivity-booster for programmers. If I can’t write the tests first, then I just know the tool, or technology, is going to produce less than optimum results – how can it make any programmer more productive if it removes the ability to write code until the tests pass? And if I can’t write any tests – that is it has no test framework either built-in or freely available, then I just know the tool sucks, and is to be actively avoided and recommended against. I’m especially suspicious of technology such BPEL frameworks and other frameworks designed to provide “programmer-less programming” (but that will have to wait for another day).