The Frustrated Architect:

Software architecture plays a pivotal role in the delivery of successful software yet it’s frustratingly neglected by many teams. Whether performed by one person or shared amongst the team, the architecture role exists on even the most agile of teams yet the balance of up front and evolutionary thinking often reflects aspiration rather than reality.

(Via coding the architecture.)

Update – if you go to this page you can get a video of the presentation; it’s about an hour long. A word of warning: I couldn’t make the video play on the site with Chrome, I had to use Safari.

The technology world has just lost another giant, though one without the towering public persona of Steve Jobs.

If you’re not actually a programmer, you’ve probably never heard of Dennis Ritchie. But the vast majority of software you use was built using a tool that he originally designed, and the rest by tools that very liberally sample from his.

The “native language” that the central processor in a computer understands is an ornery beast. For one thing, back in the 1970s every two-bit computer company (if you’ll pardon the techy pun) had its own native language; these days, there still remain two very common ones, and dozens of less common examples out there. More importantly, it’s almost incomprehensible, even to most programmers. Take this little snippet, part of the preliminaries to a very simple program that just prints the message “hello, world” to the screen:

_start:
        mov    eax, 4
        mov    ebx, 1
        mov    ecx, str
        mov    edx, str_len
        int    80h

Writing long and complicated bits of software with such unhelpful notation is extremely slow and error-prone.

“High-level” languages, that allowed the logic of software to be expressed in more compact and readable notation, had existed since the 1950s; Grace Hopper was responsible for one of the first. Over time, more and more of the scientific and business software run on the large computers of the era was written in FORTRAN, COBOL, and other high-level langages. However, the “operating systems”, the software plumbing that joined those applications to the hardware, was invariably written in the machine language of specific systems.

In the late 1960s, Ritchie, working with Ken Thompson at Bell Laboratories, hacked together their own little operating system for an obsolete computer nobody was making use of. It was small, but it worked, and was one of the first practical systems to support “timesharing” – the ability for multiple users to run multiple programs simultaneously and interactively. Fairly early on, they had another brainwave; they would rewrite as much of the system – which became known as Unix – as possible in a high-level programming language, to speed development. But first, they needed a suitable high-level language. The resulting language, an evolution of earlier languages entitled BCPL and B, was called “C”.

Both C and Unix were raging successes, partly because of their inherent strengths. The use of C allowed Unix to be “ported” to many different computer systems, a process that continues today as its spiritual successor Linux, written in C, runs on everything from IBM mainframes (and the amphormous “Googleplex” of Google’s servers which, reportedly, draw 240 megawatts of power), to virtually every smartphone on the planet (the iPhone’s operating system is not Linux, but it is also a derivation of Unix and substantial parts are written in C). They also had the good fortune, as with the Internet and World Wide Web which owes so much to both, that they gradually escaped the crush of intellectual property law to become part of the intellectual commons of the field.

Almost as important was the elegance and economy that Ritchie, along with Brian Kernighan, brought to teaching the language. Their textbook The C Programming Language remains the best programming language textbook ever written, in my view, and the one that I still strongly recommend to my students.

Much of the Windows operating system, and Mac OS X, are implemented in C. Those parts that aren’t, are implemented in computer languages directly derived from it – C++ and Objective-C. Most of the software that runs on those systems is also written in C or its successor languages. And perhaps the most pervasive “new” high-level language of the last 20 years – Java – retains so much of C’s “look and feel” that it often takes a second glance to tell which language a piece of code is written in.

Neither C nor Unix were by any means perfect. While some of its design faults have been eliminated in its successors others remain, and will likely continue to bamboozle neophyte (and, all too often, experienced) programmers, for generations to come. But there was so much he and his colleagues got right. Fate did play its part, but there are very good reasons that generations of software developers not yet born will express themselves in notations largely based on Ritchie’s.

RIP, dmr.

In Eclipse’s defence, Intellij IDEA and Netbeans also exhibit identical broken functionality.

First the technologies and the development methodologies.

So the first company uses a very Waterfall process and the integration platform is SOA. We’ve managed to build, in the middle of this, a small and focussed Java component that uses JMS in and out to avoid building horrible horrible BPEL or BPMN etc. But at either end, there’s some SOA bits to manage the integration with the “legacy”. So we’ve got this great bit of software, does amazing things, delivers real value to the business, which they want to put into production asap, but at every turn we’re hampered by the organisation or the technology platform. Its not so much the technology platform but the waterfall process the company has put around it. I’ve been told an internal wiki article detailing all the JMS  configuration is not  acceptable and the detail had to be in a Word document attached to an email requesting a not-production server configuration change. Word documents attached to emails are apparently far more “controllable” than a wiki with strong authentication protocols and history details in this world-view. Naturally I cut and pasted my wiki configuration detail into a Word document, spent a morning formatting it, and even added a link to the wiki before attaching it to an email. Acceptable process; configuration delivered. SOA and waterfall go together to make software development hell.

The second project has its many issues but one thing it does not have (for the components I have designed at least) is any SOA. It is all REST all the way down. There are multiple server-side-only components that all communicate to each other with REST over HTTP. Some of the data passed between servers is (or soon will be) JSON. The user interface is about to be delivered in two parts – one through actual REST-inspired web page requests to get the HTML and the other, via JQuery running on the web pages to actual JSON over REST services. These services will also call the REST services on the other components of the system (no database at the UI level). In fact many of the same JSON documents will be used to communicate from server to server as from ui to server. We are using Jersey for the REST. The process is Agile. The organisation hasn’t delivered such a large project with Agile before, and while I’ve been away on the other project there’s been some loss of focus. But despite some critical moments we had last week there’s been a renewed commitment to improving the engineering and project processes to achieve high velocity and good success. Its not perfect but it feels like the senior managers want the agile process to succeed. So, REST and Agile go together to make developer success.

Now, to the organisations. See if you can match the organisational description to the project style.

One of these two projects is a really fascinating piece of software in the transport industry (I can’t say more than that) which is integrating a number of “old” software solutions and creating some really sexy new features that the customer wants and loves. It is across what Eric Evans calls the “core domain” of the business. Not only is it a relatively short piece of work, but it will deliver high value to the business. It also enables a bunch of even higher value business services in the future. Things that directly expand their capabilities they can offer to their customers (which will be many of my readers) and do other related sexy things core to the business. The business salivate over it. It buffs their shine. They want it.

The other project is a little less flashy. The product is a rather niche product but it’s purpose is around the environmental effects of carbon emissions so at it’s core is the mission “save the planet from CO2″ — a pretty high value mission you’d think. However the actual functions of the software product are fairly mundane – based around helping large organisations capture and control data about their carbon emissions, because many of them have been required now to report these numbers to government agencies for a number of years. It has customers, all of then big companies or government agencies, and all the future prospects are similar sorts of organisations.

Which one is which? Which one uses REST/Agile, and which one is the SOA/Waterfall project? Do you think you can guess?

Would you be surprised if I said that the “sexy” transport project is the first one (SOA/Waterfall) and the second one is the REST/Agile project?

Recently I was learning ANTLR, trying to decide whether this was a right technology to pursue a particular project which involved parsing a preexisting message format. After a week of a spike, we decided that it was worth pursuing and started on earnest on the grammar for our project. However a week into this process, I had an epiphany … I was doing some things wrong with the ANTLR grammar which were now slowing progress in adding the new characteristics it needed to be complete. Many developers know this feeling; the features of my grammar that I had built over the first week were naive and now hampering it from expanding into the new requirements. I took it on myself to kill the entire grammar and start again. It took less than a day and half to replicate that week’s worth of work (i.e. pass the test suite which had built up around it).  I’ve done this before; scrap the first attempt at building a domain and try again. Here my domain was the same (it was after all defined in both a standards specification and in the many hundreds of thousands of sample messages we captured from an existing system), but its implementation needed refinement.

So I think that the rule about throwing away spikes can in fact be made a general axiom of programming:

When you are learning a new technology, make sure you throw away the first thing you build that works – to avoid accumulating your mistakes.

Thanks to Robert for the important qualifier “that works”.

N.B. my views about system rewrites have not changed regardless.

The science of the architect depends upon many disciplines and various apprenticeships which are carried out in other arts. His work consists in craftsmanship and technology. Craftsmanship is continued and familiar practice, which is carried out in the hands in such material as is necessary for the purpose of a design. Technology sets  forth and explains things wrought in accordance with technical skills and method.

So architects who without culture aim at manual skill cannot gain a prestige corresponding to their labours, while those who trust to theory and literature obviously chase a shadow and not reality. But those who have mastered both, like men equipped in full armour, soon acquire influence and attain their purpose.

M. Vitruvius Pollio,   De Architectura Libri Decem, 1.1.1-2. (approx 31 to 27 B.C.)

Link: http://www.viddler.com/explore/sergiopereira/videos/7/.