Recently I was learning ANTLR, trying to decide whether this was a right technology to pursue a particular project which involved parsing a preexisting message format. After a week of a spike, we decided that it was worth pursuing and started on earnest on the grammar for our project. However a week into this process, I had an epiphany … I was doing some things wrong with the ANTLR grammar which were now slowing progress in adding the new characteristics it needed to be complete. Many developers know this feeling; the features of my grammar that I had built over the first week were naive and now hampering it from expanding into the new requirements. I took it on myself to kill the entire grammar and start again. It took less than a day and half to replicate that week’s worth of work (i.e. pass the test suite which had built up around it).  I’ve done this before; scrap the first attempt at building a domain and try again. Here my domain was the same (it was after all defined in both a standards specification and in the many hundreds of thousands of sample messages we captured from an existing system), but its implementation needed refinement.

So I think that the rule about throwing away spikes can in fact be made a general axiom of programming:

When you are learning a new technology, make sure you throw away the first thing you build that works – to avoid accumulating your mistakes.

Thanks to Robert for the important qualifier “that works”.

N.B. my views about system rewrites have not changed regardless.

There is a web service inside a component, let’s call that component a ‘Node’, that receives something like (but not identical to!) an Event on its interface. Inside the Event is some data that the Node does not particularly care about (and actually has no access to – it’s just a byte[] as far as the Node can tell). However, the Node contains a Registry which enables components, lets call them Event Handlers, to register themselves with the Node, as available to process certain Events (i.e. decode that byte[] and do something with it) according to criteria which the EventHandler injects into the Node’s Registry.  EventHandler is an interface with a handful of simple methods related to handling the Event, and also registration with the Registry.

So, the process flow looks something like this: The Node first records the reception of the Event at the interface in a log. Then it tells the Registry about the Event, and the Registry produces the EventHandler(s) it needs to use.  The Registry gives the Node back the instances of the EventHandler interface. The Node then hands off the Event to the EventHandler, which does whatever it does unbeknownst to the Node, and returns a fairly simple EventResponse object. The Node records the EventResponse object in its log (i.e. a database) and returns it as the response to the web service call.

Consider the Node service method as looking something like this (you’ll have to excuse the Java 1.4-ness of this code, the WordPress code highlighting plugin apparently hates Java 5) :

  public List receive(Event event) {
    List responses = new ArrayList();
    for (EventHandler handler : this.registry.lookup(event.getMetadata())) {
      Response response = handler.handle(event);
      saveResponse(response, event, handler);
      responses.add(response);
    }
    return responses;
  }

To enable the wiring, the Registry has a method:

  void register(EventMetadata metadata, EventHandler handler)

and among other methods the EventHandler interface defines:

  void setRegistry(Registry registry)

Currently the WAR file imports the Node.JAR and a group of EventHandler.JAR files which are specific implementations for handling different kinds of Events. We configure it in Spring currently so that the specific EventHandler is injected with the Registry object (from the Node.JAR). The EventHandler implementation then registers itself with the Registry in a call-back operation, telling it what sort of Events it will handle.

This all works just fine at the moment. The problem with what we have is that it is all currently statically compiled into the WAR file.The WAR file specifies a Spring application context XML file which in turn loads the Spring configuration for the Node and Registry component, and every Spring application context for each Handler JAR included inside the WAR file’s WEB-INF/lib directory.

Now, we now don’t want every deployed instance of every Node to handle every possible Event. Currently we’ve got a small .properties file that actually tells another Spring component which EventHandlers are be to be instantiated or not. This is working fine when we only desire some Nodes to handle maybe one or two of a larger group of related Events. That is, where we currently don’t mind that the WAR file is identical in every respect on every Node — it’s just that each node contains a .properties file in its classpath that tells it which EventHandlers it is allowed to load and use (and therefore what Events it is capable of receiving, bearing in mind that when I say ‘Events’ there is really only one concrete type of Event, I mean the encrypted data which is held within the Event which is actually consumed by the Handler).

However, we are now in a situation where we want to use this same architecture for a completely different group of Events. We definitely don’t want to have to compile and assemble a new WAR file for different Nodes based on the Event cluster. We want to deploy a standardized Node.WAR which has available on its classpath a dynamic set of XxxEventHandler.JAR which can dynamically register themselves with the Node’s registry. There may be also a requirement for some related custom extension points in the future.

Initially we considered OSGi as the technology to enable this. After some discussion yesterday with people who know better about OSGi, this approach was rejected as impractical for the moment. Therefore last night I set out tooling about with the Spring 2.5.6 ApplicationContext and its related objects to see what could be done to enable dynamically-loaded JAR files within a parent Spring application context. Here is what I’ve discovered we can do, with only some small restrictions on developers writing the individual EventHandler implementations.

The first issue is, we need to locate a set of Spring application context XML file which should be on the classpath but not yet instantiated into any live Spring context.

Assuming we’re in a bean that’s ContextAware or otherwise has access to the Spring application context which is loading it, after the parent context has loaded and initialized (there are method hooks for this sort of thing) we can use the org.springframework.core.io.support.PathMatchingResourcePatternResolver class to search the classpath for a resource:

  PathMatchingResourcePatternResolver pmrl = new PathMatchingResourcePatternResolver(context.getClassLoader());
  Resource[] resources = pmrl.getResources(
    "classpath*:/net/crazymcphee/dynamiccontext/*/Crazy*DynamicContext.xml"
  );

Looking at the Spring 2.5.6 source code we found that the String passed to the getResources() method isn’t a proper regular expression, which is a pity. So you can’t do something like look for **/Crazy*DynamicContext.xml and expect to match a file Crazy*DynamicContext.xml in any package. Also we found that it had to prefixed with that classpath*: … yes, the asterix, literally … else it wouldn’t search the classpath, as opposed to the file path. So we’re restricted in the above example to a file called Crazy<something>DynamicContext.xml in a package exactly one deep from net.crazymcphee.dynamiccontext … e.g. net.crazymcphee.dynamiccontext.package.CrazyMcpheeDynamicPackage.xml matches but net.crazymcphee.dynamiccontext.package.sub.CrazySubDynamicPackage.xml and net.crazymcphee.dynamiccontext.CrazySuperDynamicPackage.xml do not. Therefore we will have a restriction on what package the dynamic context can be in and what it’s name will be. I don’t think that’s too onerous on our developers – we just have to pick a sensible standard.

The next part of the problem is that we have to load the ‘Resource’ thus found into a Spring context. This is pretty easy:

 for (Resource r : resources) {
   GenericApplicationContext createdContext = new GenericApplicationContext(context);
   XmlBeanDefinitionReader reader = new XmlBeanDefinitionReader(createdContext);
   int i = reader.loadBeanDefinitions(r);
 }

The int i will be set to the number of beans found in the createdContext. The createdContext will have the original context as its parent context, so it can gain access to any beans defined there (and also, although we are yet to test this (!), it should also be intercepted by the AOP-based transaction interceptors in the parent, and so forth).

The only other part of the puzzle may be to query the createdContext to see if it has any target beans within it, luckily for us an ApplicationContext has a method getBeansOfType(Class clazz) which will load all the beans of a particular type:

  for (Resource r : resources) {
    GenericApplicationContext createdContext = new GenericApplicationContext(context);
    XmlBeanDefinitionReader reader = new XmlBeanDefinitionReader(createdContext);
    int i = reader.loadBeanDefinitions(r);
    Map map = createdContext.getBeansOfType(EventHandler.class);
    for (Object o : map.keySet()) {
      EventHandler handler = (EventHandler) createdContext.getBean((String) o);
      // do some programmatic manipulation with the EventHandler that we found here
    }
  }

Using these three Spring features will enable us to be able to place a JAR file containing an interface implementation, and a Spring context XML file matching a particular pattern, into the classpath of our WAR, and on restart, we can dynamically pick up the newly inserted features into our application installation. I’ll report back when we get an actual production prototype together that can do this. Hopefully it will be OK to put the code in the blog too (if we make it generic enough).

If you have any further ideas or refinements to this idea, please leave them in the comments!

But the bigger headache was the Java libraries for handling WebDAV. There is the now-abandoned Apache Slide, the based-on-Slide webdavclient4j, and the way-more-complex-than-it-needs-to-be Jackrabbit. Slide and webdavclient4J have the appearance and feel of crusty, molten code from 2004. Jackrabbit is a full-featured server product … to quote, it is a “hierarchical content store with support for structured and unstructured content, full text search, versioning, transactions, observation” … and I’m sure that’s real nice if that’s what you want, but I just need to automate push a file onto an Apache HTTPD instance and tell another service what URL it now lives at. It’s not rocket science. Jackrabbit has too many features, and too little simple documentation, to be truly useful for what I need.

Enter the easy to use WebDAV client, Sardine. Here is the author’s post announcing it.  It’s very simple to use:

Sardine sardine = SardineFactory.begin();
List<DavResource> resources = sardine.getResources("http://yourdavserver.com/adirectory/");
for (DavResource res : resources)
{
     System.out.println(res);
}

Anyway, simplicity … just the sort of thing I was looking for. Recommended.

To include it into your Maven POM you’ll have to add my repository to a profile in your settings.xml:

<repositories>
  <repository>
    <id>crazy-mcphee</id>
    <url>http://modular.autonomous.org:80/artifactory-2.0.5/libs-releases-local</url>
    <snapshots>
      <enabled>false</enabled>
    </snapshots>
    <releases>
      <enabled>true</enabled>
    </releases>
  </repository>
</repositories>

At which point you can include the dependency in your pom.xml for your project:

<dependency>
  <groupId>org.autonomous</groupId>
  <artifactId>money-is-money</artifactId>
  <version>0.17</version>
</dependency>

If you want to check the source code out with SVN the release is at http://crazymcphee.net/svn/money/tags/money-is-money-0.17 and the latest trunk at http://crazymcphee.net/svn/money/trunk

Lost in the shuffle is something basic: Programmer productivity. Friends who are hands-on with such matters tell me the available SOA development environments are less than half as productive as products like PowerBuilder and Delphi were, back when they were viable.

Putting aside the Powerbuilder and Delphi love for just one minute, this is something I’ve been banging on about now for the past year … the programming tooling that is foisted onto programmers by the choice of the deployment architecture. It’s just all wrong.

In my view, what makes a programming language really productive is notepad. Or vi, or emacs, or gvim, or textmate, take your pick. What I mean is … the programming language has to be able to be programmed with a simple editor. Yes, an advanced IDE will make things more productive, but the basics must also apply. Now a lot of SOA environments are simply not programmable without the specific IDE tied to it. Even worse, the IDEs are often completely custom jobs that require a developer to be re-trained … losing years and years of productive speed with muscle-memory style automatic ability to navigate the programmer’s usual editing tool. Seriously. This stuff is whack. A program language or an environment needs to be IDE-neutral. If you got a plumber around, would you insist that he only use the tools you supply from your home handyman kit? Or would you expect the plumber to have mastered a set of his own tools already? And making matters worse, it’s rarely programmers that choose these tools which are foisted on them. The server/deployment environment and the language used to implement need to be decoupled from the tools used to build it.

But an even worse failing of many of these SOA tool suites, is that they all strongly and irrevocably coupled to the deployment/runtime environment. Generally they totally lack the ability to keep up with modern programming practice. Like for instance, automated testing. Or even unit tests, let alone advanced and productive techniques such as Test-First approaches or Test Driven Design. The tools often lack refactoring support. All of these things are in my opinion, and in the opinion of many leading developers, absolutely essential to quality engineering practice and agile development outcomes like “delivery of working software”. Both the “delivering” and the “working” part means the whole process needs to be repeatable. That’s why automated integration testing, to name just one thing, is essential in modern development. But often the fancy custom development tooling is a complete barrier to achieving this.

But you won’t hear any of this from the big vendors. One big vendor recently announced their new version of their middleware product suite had a ‘focus on testability’, but you ask any of their presales guys to demonstrate this in an actual development environment. Ask them about continuous integration support, for example. Witness their blank looks. Their development product is completely orientated to “one button push from the IDE to production” modes of thinking the idea of continuous integration builds is almost totally antithetical to the very concepts of operation the product is organised around. They think that finally adding support for Subversion version control system, at least five years too late, is a wondrous achievement.

They are aiming for ‘programmerless programming’: of course in process just creating a new type of programmer. Every new generation of programmers simply have to learn the same hard-fought lessons of software engineering over and over again because each generation of tooling apparently scraps the paradigm over and over again in a vain attempt to create push-button, wizard-driven programming models. They nearly all suffer from ‘hello world’ programming – the simple examples that sell them to IT management are trivial to conquer using the wizards, but more complex problems (i.e. real world ones) are flat-out impossible. Thus these tools are always mirages which look great at a huge distance on the horizon but are flat lifeless salt pans of bleached skulls and bones on closer examination (or, maybe they are more like tar pits that look like a nice waterhole but one step into it and you are sucked down  to your doom).

As you might be able to tell, I am utterly contemptuous of many of these SOA tool paradigms. I have nothing against SOA itself, per se. But there is nothing more productive than a programmer who understands the importance of simple and repeatable build and deployment automation using command line tools and who knows his programming editor inside out after ten years of use. Give that programmer a better language by all means, add incremental features to that IDE, allow the programmers to continuously improve their techniques, promote professional craftsmanship, yes, yes and a thousand times yes. But no amount of drag and drop wizards, push-button deployments, and “object inspector” property editors will ever usurp that deep knowledge a good programmer brings to both his language and his personal tooling choices.

Collection mapping is one of those “hello world” problems. (The “hello world” example in the doco looks totally trivial and completely ideal [which is the problem], but suck-in-the-galaxy-greet-it-and-then-map-all-the-stars problem, which is more like what your real app looks like, is far less than trivial and not so easily achieved with the demonstrated toolkit). In my experience, collection mapping is the cause of all sorts of pain and performance perfidy.

Here’s a further situation where I find that they really get in the way. Consider integration with existing systems that use highly-normalised database schemas (designed by advanced SQL programmers who know what they are doing in that language).

This is not an uncommon scenario – we have this system ‘A’ which (let’s say) has all our customer shipping, product, warehousing data in it. We don’t want to replace it. But we need to extend it so that e.g. available to web systems. Or maybe it needs to be integrated into the big fancy off-the-shelf CRM we’re installing. Something like that. We need to build some sort of service layer around this system ‘A’ so that other systems can access its data and/or functions. It’s common to think that an ORM around that big complex database schema is going to help, but in my experience, it doesn’t for many of the reasons Stephan and Aldo list. That sort of highly-normal database schema in my experiences completely kills ORM object representations stone cold. You end up with so many LazyInitialisationExceptions and various other problems it sucks productivity out the developers and performance out of the system.

Also, I get a bit annoyed about ORM abstractions leaking into the web tier, this is especially prevalent with using annotations rather than ORM mapping files.

Different sorts of approaches are sometimes needed to be applied with careful thought in that scenario. Perhaps you need to model the “intermediate” domain you need directly, and then use, for example, simple DAO layers which operate directly on the domain you’ve defined. These DAO layers then might call functions (stored procs) in the database that do the problematic mapping in an efficent relational language (after all it’s the direct representation of the target data format).

I’m not saying that you have to do the above in every case, or that an ORM is always wrong, but anyway the point is, ORM isn’t a magic bullet like any other technology you have to consider carefully. Your application doesn’t always need one by default.

First up, the easy way out – take a day off, go for a walk in the park, have a nice dinner, get a good nights sleep. You should always work to “sustainable progress” and yes this means every hour you work over 40 hours a week means your quality is steadily decreasing until some point where every extra hour you put will take two to fix it later (in other words, negative impact). Even just pair programming will help prevent this in the first place as long at least one of you isn’t operating on half the sleep they need. But ultimately, all of this just blames the individual, when really poor code quality and technical debt are much more a function of the management system, methodology, total sum of the choices by every on the team, etc. (For a start, why are you up late in the first place? Was that your choice, or an outside imposed deadline?).

As James Adam and “iSteve” point out in the comments – technical debt is different from just “buggy software” and in my experience it’s usually function of management or other types of systemic choices (and “systemic” includes you, the programmer as well everyone else). What it really does is trade easy change now for even harder change later. It wants that sweet sweet sugar hit straight away but forgets about the “sugar coma” that comes a few minutes afterwards. It defers “paying the piper” until tomorrow for that lovely tune today. James Adam rightly says in his comment this can be a valid choice, but all of the team have to be fully aware of the consequences of that choice. One of which is typically that the longer  you defer the payment, the more debt you’ll accumulate, until all you are doing is paying off the interest and not getting to the principal.

As “iSteve” says, management (and maybe “management” might mean just you telling yourself this in your head) always promises to “fix it later” and never does allow that. “Just make the hack” and gloss over the problem is a mantra that we’ve all heard before. Where it gets painful is where every change is now taking twice as long as it should because of the accumulation of hacks. Every single one one of us has been there before, and are probably in various levels of it right now.

In my view this is the much more insidious circumstance because it is systemic.  After all the solution to a bad day’s work because I was too tired to think properly is to simply get a good night’s sleep, roll back yesterday’s coding, and do it again today. All I’ve done is lost a day’s work.  However, an organisation is likely to have accumulated months if not years of organisational cruft that makes changing their context that much harder than just rolling back yesterday’s poor code and trying again – it requires commitment to change from everyone who counts in the first instance.

The basics of Tapestry is that it is a component-based web framework. Just about everything, including web pages, are components. Components may contain other components. The way it works is very simple and quite elegant, once you get used to it and weaned off the big-XML-file style of configuring a web application.

When you start, you have two main Java packages that are created for you (if you use the maven archetype, otherwise you will create these packages yourself). If your package root is say “net.crazymcphee.webapp”, then your two packages are “net.crazymcphee.webapp.pages” and “net.crazymcphee.webapp.components”. To configure a Tapestry 5 project with maven use this command and answer the prompts:

mvn archetype:generate \

    -DarchetypeCatalog=http://tapestry.formos.com/maven-snapshot-repository

Don’t use the one in the tutorial as it will not work! This is an excellent illustration of the first and most serious problem that Tapestry 5 has: the documentation not only has massive lacunas, it is also sometimes wrong and not updated.

Now, any class that you create in the “pages” package will automatically become a page in your application. But these classes need not be very complex at all. For example:

package net.crazymcphee.webapp.pages;
 
import java.util.Collections;
import java.util.List;
 
import org.apache.tapestry5.annotations.Persist;
import org.apache.tapestry5.annotations.Property;
import org.apache.tapestry5.ioc.annotations.Inject;
 
import net.crazymcphee.webapp.model.Person;
import net.crazymcphee.webapp.services.PersonService;
 
public class Persons {
 
    @Inject
    private PersonService personService;
    @Property
    @Persist
    private List persons;
    @Property
    @Persist
    private String searchTerm;
 
    Object onSubmitFromSearch() {
        persons = personService.findPersons(searchTerm);
        return this;
    }
 
    Object onSubmitFromClear() {
        persons = Collections.EMPTY_LIST;
        return this;
    }
 
}

You can see that this class doesn’t extend any infrastructure classes and has quite a simple structure. This is a fully functional page with two actions – one populates a list, the other clears it.

Now, to explain just a little here; PersonService is injected by Tapestry, but you need to configure in your AppModule which implementation you want to use. You can also use it with Spring as your IOC container but the one that comes with Tapestry is perfectly good enough for most applications.

The List, persons, and the searchTerm parameters are marked as @Property (so we don’t need to add getters and setters) and also @Persist so that the variables are preserved from request to request.

The methods “onSubmitFromSearch” and “onSubmitFromClear” are using a convention – “onSubmit” will also work, but assuming (as is true in this case) that we may have multiple forms on the one page, each method will only be fired from the “search” form in one instance, and the “clear” form in the other. These names are not special, it’s just (as you’ll see below) the forms will have these two names. They could be “Bill” and “Ben” in which case the methods would be “onSubmitFromBill” and “onSubmitFromBen”.

You will also note that these methods return the same page instance, which tells Tapestry to re-render the same page, but if you wanted to forward onto another page, you would add a instance variable, mark it with an @InjectPage annotation, and return that instance variable (after initialising it in your “onSubmit” method) instead of just returning “this”.

This page Persons, is available at {application-context-path}/persons. But there is one other part of the puzzle – the actual view. As mentioned, Tapestry uses convention over configuration and in this case, the convention is that the Page markup must be named the same: Persons.tml. Here is the matching tml file for the class above;

<html t:type=“layout” title=“Peoples I Might Know”

t:sidebarTitle=“Current Time”

xmlns:t=“http://tapestry.apache.org/schema/tapestry_5_1_0.xsd”

xmlns:p=“tapestry:parameter”>

<!– Most of the page content, including <head>, <body>, etc. tags, comes from Layout.tml –>

<t:form t:id=“search”>

<t:textfield t:id=“searchTerm” validate=“required” size=“20″/>

<t:submit t:id=“searchPeople” value=“search”/>

</t:form>

<t:grid source=“persons”/>

<t:form t:id=“clear”>

<t:submit t:id=“clearPeople” value=“clear”/>

</t:form>

</html>

Yes, it’s that bloody simple. The first form fires the “onSubmitFromSearch” method (with a little bit of validation, done in javascript) and the second method clears the list. In-between, there is this <t:grid source=“persons”/> business, which, if the ‘persons’ variable in the page class is populated, will show a list of its contents!

To test our app, we can use the command-line ‘mvn clean jetty:run’. When Jetty has run up, then we can point our browser at the web app: http://localhost:8080/sample-webapp/Persons, and with any luck we will see the following:

Now, by default, Tapestry creates the web app to use the template design as shown above. Of course, you can make it look completely some other way – or even use a totally different template if you want. The page template is just a component that’s included.

So if you enter in a search term and click the search button, you’d expect to see a result like this:

As you can see, the list is automatically populated with the details returned from the service method (in this particular instance, this are just a canned response, but normally of course they’d be the result of a database or a web service call of some type).

When you click the clear button, the list is cleared as you’d expect:

Now, here’s a really powerful feature I find in Tapestry. Let’s say our automated acceptance tests assert that when the ‘Clear’ button is pressed, the Search box as well as the persons list is cleared. What Tapestry allows us to to do, is keep Jetty running, edit the files in the IDE, and re-run the tests against the running app without restarting! We edit our method:

    Object onSubmitFromClear() {
        persons = Collections.EMPTY_LIST;
        searchTerm="";
        return this;
    }

In this case I’ve also added a bit of space around the clear button in the template as well:

And now our failing test will pass.

Editing files like this in the running web app only works for Pages and I think Components. If I had to change the service or model classes I’d have to restart, but “mvn jetty:run” isn’t very a heavyweight process.

As I said above, Tapestry’s not perfect: it’s major flaw is the poor documentation. Convention-over-configuration is easy to grok – if you know the convention. If you don’t and the documentation doesn’t tell you, and you can’t find a sample of what you need, it can be very frustrating. There is an excellent user list though.

Its other major flaw may be the stability of the API. Tapestry 5 is different (and incompatible) from Tapestry 4 is different from Tapestry 3. But so far I’ve used it on a couple of projects and I’m really enjoying it. I really hate the oodles of XML boilerplate and massive amounts of configuration found in Spring and I find myself somewhat reluctant to use it unless I really have to. Tapestry solves for me a range of different problems and mostly it presents a very elegant way to create a componentised web application. You might like to give it a try.

Attached is the sample code used above. It took me about 10 minutes to write (far shorter than it took me to write this blog entry!) - sample-webapp.tar bundle

inessential.com: Anatomy of a feature.

Today I just want to meditate on a code artifact that often manifests itself in such environments. I speak of the comment //TODO seen frequently in such code. The idea that future “refactoring” goes on the “todo” list in such an environment is completely broken. An environment that allows that to happen means it will never get done. Do it now, or don’t do it. TODO is considered harmful.

I have come around to the view that ‘//TODO’ comments should make a build fail. Either:

1. It is a future story yet to be implemented and placed into the backlog,
2. It is work in progress (and by definition doesn’t need a TODO because everything still needs to be ‘done’), or,
3. It is done, as in DONE-done, and therefore the story is complete and there’s nothing left “TO DO”.

In other words, “//TODO” is a indicator of a process that’s not letting developers get to “done”. Identify the problem, propose a counter-measure and fix the process that stopping you from getting to completion.