In Eclipse’s defence, Intellij IDEA and Netbeans also exhibit identical broken functionality.

For a start, the documentation is merely a series of outlines of brief hints. One measly section.The laughably short Spring 3 doco section 13.5.1.4 “Dealing with multiple persistence units” conveniently omits  the transaction manager configuration from the example. The problem appears to be that the JPA transaction manager only (and compulsorily) deals with one entity manager factory. And an entity manager factory only deals with one persistence unit. So therefore you have to have two JPA transaction managers. Which means the two persistence unit transactions won’t co-operate properly even though they may share the underlying datasource and are configured through the single persistence unit manager bean.

The above scenario is totally trivial in an EJB 3 container backed with Hibernate, or Eclipselink, as the provider. About one-quarter of the configuration. If I have to use a JTA transaction manager obtained by JNDI lookup from the container, to run a transaction across two JPA persistence units which share the same underlying datasource, why the hell am I using Spring in the first place?

All I wanted was to isolate one set of db transactions from another, use JPA persistence units so the two sets of tables could live easily in two different schemas, and use the annotations to kill the fragile AOP regex-like-but-not-like-regex class and method pattern-matching jiggery pokery from the Spring configuration.

My god, this is such a stonkingly non-trivial forest of horrible configuration and a trial and error morass of filthy swamp miasma … an equivalent EJB3 backed by Hibernate JPA set up is, by comparison, an almost effortless task. I can produce a unit-tested all singing all dancing multiple persistence unit JPA app running in an EJB 3 container with XA datasources using Hibernate as the JPA implementation in almost no time at all. Getting the same thing with Spring (oh, and running some tests with Jetty inside Maven) is like having all your teeth pulled while you’re coming down from a three day methamphetamine binge. Getting Spring to run with multiple JPA persistence units on the same JDBC connection (without XA, across the same database connection) with a transaction propagation of REQUIRES_NEW on the entry point of one of the units is a hair-pulling, beard-greying, head-desk banging, co-worker punching, drunken ranting, blood-pressure raising experience of pure horror. Which apparently doesn’t end.

Die, Spring, die. Can’t come soon enough as far as I am concerned. It’s more evil than Oracle. At least you know with Oracle you’re in for an un-lubed hard and fast backdoor job from Ellison with no reach-around before you even unpack the box. Spring is like a beautiful young sexy soft-porn film that turns into some relentlessly horrific succubus-filled horror film half way through.

Seriously, its enough to make one pine for bloody Weblogic. Spring is the new EJB 2.1.

First the technologies and the development methodologies.

So the first company uses a very Waterfall process and the integration platform is SOA. We’ve managed to build, in the middle of this, a small and focussed Java component that uses JMS in and out to avoid building horrible horrible BPEL or BPMN etc. But at either end, there’s some SOA bits to manage the integration with the “legacy”. So we’ve got this great bit of software, does amazing things, delivers real value to the business, which they want to put into production asap, but at every turn we’re hampered by the organisation or the technology platform. Its not so much the technology platform but the waterfall process the company has put around it. I’ve been told an internal wiki article detailing all the JMS  configuration is not  acceptable and the detail had to be in a Word document attached to an email requesting a not-production server configuration change. Word documents attached to emails are apparently far more “controllable” than a wiki with strong authentication protocols and history details in this world-view. Naturally I cut and pasted my wiki configuration detail into a Word document, spent a morning formatting it, and even added a link to the wiki before attaching it to an email. Acceptable process; configuration delivered. SOA and waterfall go together to make software development hell.

The second project has its many issues but one thing it does not have (for the components I have designed at least) is any SOA. It is all REST all the way down. There are multiple server-side-only components that all communicate to each other with REST over HTTP. Some of the data passed between servers is (or soon will be) JSON. The user interface is about to be delivered in two parts – one through actual REST-inspired web page requests to get the HTML and the other, via JQuery running on the web pages to actual JSON over REST services. These services will also call the REST services on the other components of the system (no database at the UI level). In fact many of the same JSON documents will be used to communicate from server to server as from ui to server. We are using Jersey for the REST. The process is Agile. The organisation hasn’t delivered such a large project with Agile before, and while I’ve been away on the other project there’s been some loss of focus. But despite some critical moments we had last week there’s been a renewed commitment to improving the engineering and project processes to achieve high velocity and good success. Its not perfect but it feels like the senior managers want the agile process to succeed. So, REST and Agile go together to make developer success.

Now, to the organisations. See if you can match the organisational description to the project style.

One of these two projects is a really fascinating piece of software in the transport industry (I can’t say more than that) which is integrating a number of “old” software solutions and creating some really sexy new features that the customer wants and loves. It is across what Eric Evans calls the “core domain” of the business. Not only is it a relatively short piece of work, but it will deliver high value to the business. It also enables a bunch of even higher value business services in the future. Things that directly expand their capabilities they can offer to their customers (which will be many of my readers) and do other related sexy things core to the business. The business salivate over it. It buffs their shine. They want it.

The other project is a little less flashy. The product is a rather niche product but it’s purpose is around the environmental effects of carbon emissions so at it’s core is the mission “save the planet from CO2″ — a pretty high value mission you’d think. However the actual functions of the software product are fairly mundane – based around helping large organisations capture and control data about their carbon emissions, because many of them have been required now to report these numbers to government agencies for a number of years. It has customers, all of then big companies or government agencies, and all the future prospects are similar sorts of organisations.

Which one is which? Which one uses REST/Agile, and which one is the SOA/Waterfall project? Do you think you can guess?

Would you be surprised if I said that the “sexy” transport project is the first one (SOA/Waterfall) and the second one is the REST/Agile project?

Well, nope.

Especially nope if you hope to use clustered broadcast messaging (i.e. Topics)  to multiple connected clustered application receivers. Unless you cope with multiple versions of the same message in your application code.

Funny thing is, they already have this product that can do clustered broadcast messaging well proper, it’s called Coherence. But if you need the JMS API, not so suitable. However they could use this product to underpin their crappy 10 year old clustered JMS implementation in Weblogic. Do you think they will do that? Anyone want to open a book on it?

The original post is here - http://www.crazymcphee.net/x/2009/02/13/mechanical-pencil-lust/

The Pentel graph gear was my original favorite, I gave the Faber-Castell 2011 straight away to Lisa, it was inadequate and plasticly for my needs, but perfect for Lisa’s, she destroys those things and already used 0.7mm lead. The Graph Gear eventually couldn’t go the distance. The the pocket clip, which is also the spring clip that holds the pencil tip out, finally gave away after about a year. Also its knobbly grip texture made writing with it for a long period actually painful on the fingers. Even though it was by far the most stylish looking pencil of the three.

Pentel Graph Gear 1000 0.5mm

My firm favorite soon switched to and has stayed with the TK Fine Vario L. The one drawback with the TK Fine Vario L is that the tip sleeve doesn’t retract, so it’s not strictly pocket safe. This is OK for me most of the time as usually it travels in one of my laptop backpack’s pen/pencil slots.However it’s sometimes annoying, although it’s adjustable soft-hard feature is very nice. It also balances extremely well in the hand and the metal barrel grip is very comfortable. With its dark green plastic, hexagonal section, upper half, it does look very old fashioned. However this is offset by its excellent writing comfort and long retractable rubber length (the Pentel’s short rubber had quickly worn out, and the cap had to be taken off to use it).

Faber-Castell TK Fine Vario L 0.5mm

So, I’ll continue to use the TK Fine Vario unless one of these just-purchased beauties can knock it off its perch (images link to the product page on Cult Pens):

OHTO Super Promecha Pencil PM-1500P

OHTO Super Promecha Pencil PM-1500P

The OHTO Super Promecha one looks like a superb technical pen but we’ll see how comfortable that the “fine diamond-cut knurled pattern” grip really is, especially with sustained use taking research notes. It has a lot of interesting features.

Uni Kuru Toga Pencil 0.5mm M5-450

Uni Kuru Toga Pencil 0.5mm M5-450

The Uni Kuru Toga claims to be a “self-sharpening” mechanical pencil. Every time the lead touches the paper it rotates the lead, resulting in a conical shape on the end of the lead. It’s pretty cheap, so I got it to find out if this miraculous claim is true or not. I ordered the orange one.

rotring Tikky 3 Pencil 0.5mm Colour Barrel

rotring Tikky 3 Pencil 0.5mm Colour Barrel

A cheap and cheerful pencil, it’s always handy to have some of those for emergencies, in the car, people wanting to borrow them, etc. I got the red one.

That’s it – total price GBP 33.93 including GBP 10 in shipping. The Ohto is naturally more than half of that total cost, but given the strength of the AUD at the moment I though it a good time to shop for an item like this.

Recently I was learning ANTLR, trying to decide whether this was a right technology to pursue a particular project which involved parsing a preexisting message format. After a week of a spike, we decided that it was worth pursuing and started on earnest on the grammar for our project. However a week into this process, I had an epiphany … I was doing some things wrong with the ANTLR grammar which were now slowing progress in adding the new characteristics it needed to be complete. Many developers know this feeling; the features of my grammar that I had built over the first week were naive and now hampering it from expanding into the new requirements. I took it on myself to kill the entire grammar and start again. It took less than a day and half to replicate that week’s worth of work (i.e. pass the test suite which had built up around it).  I’ve done this before; scrap the first attempt at building a domain and try again. Here my domain was the same (it was after all defined in both a standards specification and in the many hundreds of thousands of sample messages we captured from an existing system), but its implementation needed refinement.

So I think that the rule about throwing away spikes can in fact be made a general axiom of programming:

When you are learning a new technology, make sure you throw away the first thing you build that works – to avoid accumulating your mistakes.

Thanks to Robert for the important qualifier “that works”.

N.B. my views about system rewrites have not changed regardless.

There is a web service inside a component, let’s call that component a ‘Node’, that receives something like (but not identical to!) an Event on its interface. Inside the Event is some data that the Node does not particularly care about (and actually has no access to – it’s just a byte[] as far as the Node can tell). However, the Node contains a Registry which enables components, lets call them Event Handlers, to register themselves with the Node, as available to process certain Events (i.e. decode that byte[] and do something with it) according to criteria which the EventHandler injects into the Node’s Registry.  EventHandler is an interface with a handful of simple methods related to handling the Event, and also registration with the Registry.

So, the process flow looks something like this: The Node first records the reception of the Event at the interface in a log. Then it tells the Registry about the Event, and the Registry produces the EventHandler(s) it needs to use.  The Registry gives the Node back the instances of the EventHandler interface. The Node then hands off the Event to the EventHandler, which does whatever it does unbeknownst to the Node, and returns a fairly simple EventResponse object. The Node records the EventResponse object in its log (i.e. a database) and returns it as the response to the web service call.

Consider the Node service method as looking something like this (you’ll have to excuse the Java 1.4-ness of this code, the WordPress code highlighting plugin apparently hates Java 5) :

  public List receive(Event event) {
    List responses = new ArrayList();
    for (EventHandler handler : this.registry.lookup(event.getMetadata())) {
      Response response = handler.handle(event);
      saveResponse(response, event, handler);
      responses.add(response);
    }
    return responses;
  }

To enable the wiring, the Registry has a method:

  void register(EventMetadata metadata, EventHandler handler)

and among other methods the EventHandler interface defines:

  void setRegistry(Registry registry)

Currently the WAR file imports the Node.JAR and a group of EventHandler.JAR files which are specific implementations for handling different kinds of Events. We configure it in Spring currently so that the specific EventHandler is injected with the Registry object (from the Node.JAR). The EventHandler implementation then registers itself with the Registry in a call-back operation, telling it what sort of Events it will handle.

This all works just fine at the moment. The problem with what we have is that it is all currently statically compiled into the WAR file.The WAR file specifies a Spring application context XML file which in turn loads the Spring configuration for the Node and Registry component, and every Spring application context for each Handler JAR included inside the WAR file’s WEB-INF/lib directory.

Now, we now don’t want every deployed instance of every Node to handle every possible Event. Currently we’ve got a small .properties file that actually tells another Spring component which EventHandlers are be to be instantiated or not. This is working fine when we only desire some Nodes to handle maybe one or two of a larger group of related Events. That is, where we currently don’t mind that the WAR file is identical in every respect on every Node — it’s just that each node contains a .properties file in its classpath that tells it which EventHandlers it is allowed to load and use (and therefore what Events it is capable of receiving, bearing in mind that when I say ‘Events’ there is really only one concrete type of Event, I mean the encrypted data which is held within the Event which is actually consumed by the Handler).

However, we are now in a situation where we want to use this same architecture for a completely different group of Events. We definitely don’t want to have to compile and assemble a new WAR file for different Nodes based on the Event cluster. We want to deploy a standardized Node.WAR which has available on its classpath a dynamic set of XxxEventHandler.JAR which can dynamically register themselves with the Node’s registry. There may be also a requirement for some related custom extension points in the future.

Initially we considered OSGi as the technology to enable this. After some discussion yesterday with people who know better about OSGi, this approach was rejected as impractical for the moment. Therefore last night I set out tooling about with the Spring 2.5.6 ApplicationContext and its related objects to see what could be done to enable dynamically-loaded JAR files within a parent Spring application context. Here is what I’ve discovered we can do, with only some small restrictions on developers writing the individual EventHandler implementations.

The first issue is, we need to locate a set of Spring application context XML file which should be on the classpath but not yet instantiated into any live Spring context.

Assuming we’re in a bean that’s ContextAware or otherwise has access to the Spring application context which is loading it, after the parent context has loaded and initialized (there are method hooks for this sort of thing) we can use the org.springframework.core.io.support.PathMatchingResourcePatternResolver class to search the classpath for a resource:

  PathMatchingResourcePatternResolver pmrl = new PathMatchingResourcePatternResolver(context.getClassLoader());
  Resource[] resources = pmrl.getResources(
    "classpath*:/net/crazymcphee/dynamiccontext/*/Crazy*DynamicContext.xml"
  );

Looking at the Spring 2.5.6 source code we found that the String passed to the getResources() method isn’t a proper regular expression, which is a pity. So you can’t do something like look for **/Crazy*DynamicContext.xml and expect to match a file Crazy*DynamicContext.xml in any package. Also we found that it had to prefixed with that classpath*: … yes, the asterix, literally … else it wouldn’t search the classpath, as opposed to the file path. So we’re restricted in the above example to a file called Crazy<something>DynamicContext.xml in a package exactly one deep from net.crazymcphee.dynamiccontext … e.g. net.crazymcphee.dynamiccontext.package.CrazyMcpheeDynamicPackage.xml matches but net.crazymcphee.dynamiccontext.package.sub.CrazySubDynamicPackage.xml and net.crazymcphee.dynamiccontext.CrazySuperDynamicPackage.xml do not. Therefore we will have a restriction on what package the dynamic context can be in and what it’s name will be. I don’t think that’s too onerous on our developers – we just have to pick a sensible standard.

The next part of the problem is that we have to load the ‘Resource’ thus found into a Spring context. This is pretty easy:

 for (Resource r : resources) {
   GenericApplicationContext createdContext = new GenericApplicationContext(context);
   XmlBeanDefinitionReader reader = new XmlBeanDefinitionReader(createdContext);
   int i = reader.loadBeanDefinitions(r);
 }

The int i will be set to the number of beans found in the createdContext. The createdContext will have the original context as its parent context, so it can gain access to any beans defined there (and also, although we are yet to test this (!), it should also be intercepted by the AOP-based transaction interceptors in the parent, and so forth).

The only other part of the puzzle may be to query the createdContext to see if it has any target beans within it, luckily for us an ApplicationContext has a method getBeansOfType(Class clazz) which will load all the beans of a particular type:

  for (Resource r : resources) {
    GenericApplicationContext createdContext = new GenericApplicationContext(context);
    XmlBeanDefinitionReader reader = new XmlBeanDefinitionReader(createdContext);
    int i = reader.loadBeanDefinitions(r);
    Map map = createdContext.getBeansOfType(EventHandler.class);
    for (Object o : map.keySet()) {
      EventHandler handler = (EventHandler) createdContext.getBean((String) o);
      // do some programmatic manipulation with the EventHandler that we found here
    }
  }

Using these three Spring features will enable us to be able to place a JAR file containing an interface implementation, and a Spring context XML file matching a particular pattern, into the classpath of our WAR, and on restart, we can dynamically pick up the newly inserted features into our application installation. I’ll report back when we get an actual production prototype together that can do this. Hopefully it will be OK to put the code in the blog too (if we make it generic enough).

If you have any further ideas or refinements to this idea, please leave them in the comments!

But the bigger headache was the Java libraries for handling WebDAV. There is the now-abandoned Apache Slide, the based-on-Slide webdavclient4j, and the way-more-complex-than-it-needs-to-be Jackrabbit. Slide and webdavclient4J have the appearance and feel of crusty, molten code from 2004. Jackrabbit is a full-featured server product … to quote, it is a “hierarchical content store with support for structured and unstructured content, full text search, versioning, transactions, observation” … and I’m sure that’s real nice if that’s what you want, but I just need to automate push a file onto an Apache HTTPD instance and tell another service what URL it now lives at. It’s not rocket science. Jackrabbit has too many features, and too little simple documentation, to be truly useful for what I need.

Enter the easy to use WebDAV client, Sardine. Here is the author’s post announcing it.  It’s very simple to use:

Sardine sardine = SardineFactory.begin();
List<DavResource> resources = sardine.getResources("http://yourdavserver.com/adirectory/");
for (DavResource res : resources)
{
     System.out.println(res);
}

Anyway, simplicity … just the sort of thing I was looking for. Recommended.

Lost in the shuffle is something basic: Programmer productivity. Friends who are hands-on with such matters tell me the available SOA development environments are less than half as productive as products like PowerBuilder and Delphi were, back when they were viable.

Putting aside the Powerbuilder and Delphi love for just one minute, this is something I’ve been banging on about now for the past year … the programming tooling that is foisted onto programmers by the choice of the deployment architecture. It’s just all wrong.

In my view, what makes a programming language really productive is notepad. Or vi, or emacs, or gvim, or textmate, take your pick. What I mean is … the programming language has to be able to be programmed with a simple editor. Yes, an advanced IDE will make things more productive, but the basics must also apply. Now a lot of SOA environments are simply not programmable without the specific IDE tied to it. Even worse, the IDEs are often completely custom jobs that require a developer to be re-trained … losing years and years of productive speed with muscle-memory style automatic ability to navigate the programmer’s usual editing tool. Seriously. This stuff is whack. A program language or an environment needs to be IDE-neutral. If you got a plumber around, would you insist that he only use the tools you supply from your home handyman kit? Or would you expect the plumber to have mastered a set of his own tools already? And making matters worse, it’s rarely programmers that choose these tools which are foisted on them. The server/deployment environment and the language used to implement need to be decoupled from the tools used to build it.

But an even worse failing of many of these SOA tool suites, is that they all strongly and irrevocably coupled to the deployment/runtime environment. Generally they totally lack the ability to keep up with modern programming practice. Like for instance, automated testing. Or even unit tests, let alone advanced and productive techniques such as Test-First approaches or Test Driven Design. The tools often lack refactoring support. All of these things are in my opinion, and in the opinion of many leading developers, absolutely essential to quality engineering practice and agile development outcomes like “delivery of working software”. Both the “delivering” and the “working” part means the whole process needs to be repeatable. That’s why automated integration testing, to name just one thing, is essential in modern development. But often the fancy custom development tooling is a complete barrier to achieving this.

But you won’t hear any of this from the big vendors. One big vendor recently announced their new version of their middleware product suite had a ‘focus on testability’, but you ask any of their presales guys to demonstrate this in an actual development environment. Ask them about continuous integration support, for example. Witness their blank looks. Their development product is completely orientated to “one button push from the IDE to production” modes of thinking the idea of continuous integration builds is almost totally antithetical to the very concepts of operation the product is organised around. They think that finally adding support for Subversion version control system, at least five years too late, is a wondrous achievement.

They are aiming for ‘programmerless programming’: of course in process just creating a new type of programmer. Every new generation of programmers simply have to learn the same hard-fought lessons of software engineering over and over again because each generation of tooling apparently scraps the paradigm over and over again in a vain attempt to create push-button, wizard-driven programming models. They nearly all suffer from ‘hello world’ programming – the simple examples that sell them to IT management are trivial to conquer using the wizards, but more complex problems (i.e. real world ones) are flat-out impossible. Thus these tools are always mirages which look great at a huge distance on the horizon but are flat lifeless salt pans of bleached skulls and bones on closer examination (or, maybe they are more like tar pits that look like a nice waterhole but one step into it and you are sucked down  to your doom).

As you might be able to tell, I am utterly contemptuous of many of these SOA tool paradigms. I have nothing against SOA itself, per se. But there is nothing more productive than a programmer who understands the importance of simple and repeatable build and deployment automation using command line tools and who knows his programming editor inside out after ten years of use. Give that programmer a better language by all means, add incremental features to that IDE, allow the programmers to continuously improve their techniques, promote professional craftsmanship, yes, yes and a thousand times yes. But no amount of drag and drop wizards, push-button deployments, and “object inspector” property editors will ever usurp that deep knowledge a good programmer brings to both his language and his personal tooling choices.

Collection mapping is one of those “hello world” problems. (The “hello world” example in the doco looks totally trivial and completely ideal [which is the problem], but suck-in-the-galaxy-greet-it-and-then-map-all-the-stars problem, which is more like what your real app looks like, is far less than trivial and not so easily achieved with the demonstrated toolkit). In my experience, collection mapping is the cause of all sorts of pain and performance perfidy.

Here’s a further situation where I find that they really get in the way. Consider integration with existing systems that use highly-normalised database schemas (designed by advanced SQL programmers who know what they are doing in that language).

This is not an uncommon scenario – we have this system ‘A’ which (let’s say) has all our customer shipping, product, warehousing data in it. We don’t want to replace it. But we need to extend it so that e.g. available to web systems. Or maybe it needs to be integrated into the big fancy off-the-shelf CRM we’re installing. Something like that. We need to build some sort of service layer around this system ‘A’ so that other systems can access its data and/or functions. It’s common to think that an ORM around that big complex database schema is going to help, but in my experience, it doesn’t for many of the reasons Stephan and Aldo list. That sort of highly-normal database schema in my experiences completely kills ORM object representations stone cold. You end up with so many LazyInitialisationExceptions and various other problems it sucks productivity out the developers and performance out of the system.

Also, I get a bit annoyed about ORM abstractions leaking into the web tier, this is especially prevalent with using annotations rather than ORM mapping files.

Different sorts of approaches are sometimes needed to be applied with careful thought in that scenario. Perhaps you need to model the “intermediate” domain you need directly, and then use, for example, simple DAO layers which operate directly on the domain you’ve defined. These DAO layers then might call functions (stored procs) in the database that do the problematic mapping in an efficent relational language (after all it’s the direct representation of the target data format).

I’m not saying that you have to do the above in every case, or that an ORM is always wrong, but anyway the point is, ORM isn’t a magic bullet like any other technology you have to consider carefully. Your application doesn’t always need one by default.