It all started when I needed to start the Transaction out in the view, i.e. as soon as the resource is opened on the HTTP side (rather than when the database service layer is called). I’m using JPA for a number of reasons;

1. I need to access multiple databases each with a different schema (this means different connections)
2. They need to have XA transactions.
3. I’d like the transactions managed by the container (JTA).

JPA provides all these things easily without all the complex Hibernate.xbm.xml mapping files and what-have-you.

The trick to starting the transaction with the web session is to use spring’s OpenEntityManagerInViewFilter. Unfortunately I was using  PersistenceAnnotationBeanPostProcessor to manage my multiple persistence units. The filter wants to know what EntityManagerFactory it should bind to, and fair enough. But with my minimal configuration, there was no addressable EntityManagerFactory!

The solution was to stop the PersistenceAnnotationBeanPostProcessor from doing the JNDI look ups and bind each PersistenceUnit into the Spring context with a manual JNDI lookup;

  <tx:annotation-driven />
  <tx:jta-transaction-manager />

  <bean id="pabpp" 
   class="org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor" />

  <jee:jndi-lookup id="onePU" jndi-name="persistence/onePU" />
  <jee:jndi-lookup id="twoPU" jndi-name="persistence/twoPU" />
  <jee:jndi-lookup id="threePU" jndi-name="persistence/threePU" />

then, in the web.xml the ViewFilter could be bound explicitly to the required persistence unit, in this case “onePU”. A fuller explanation can be found on my spring source forum post here; http://forum.springsource.org/showthread.php?115844-OpenEntityManagerInViewFilter-with-JPA-PersistenceAnnotationBeanPostProcessor

Overall, this is now quite an elegant solution.

UPDATE. More documentation of the solution and the various configurations at the Spring forum here: http://forum.springsource.org/showthread.php?115587-Example-for-using-two-databases-w-Spring-amp-JTA-transaction-manager&p=383432#post383432

Our system has generic components which “pass off” specific knowledge of specific data sets to specific processing components, where there are not a fixed set of these specific components available to all the generic components. In classic OO design this is achieved with interfaces and many design patterns like the Visitor Pattern. In our design we’ve replaced much of the complexity of the interface with REST-based architecture. Our application has become far more modular (despite its run-time complexity). More importantly, we are at the the point where we can easily start delivering the advanced features that the business has now specified for us without introducing a massive bloat-o-burger one-size-fits-all set of “common data definitions” that are the union of all possible sub-system functionalities.  Each sub-system can have its specific features required for that data set and still satisfy the handful of generic actions it is required to support. Each of these specific features can be developed in isolation. It’s a big win for us.

First the technologies and the development methodologies.

So the first company uses a very Waterfall process and the integration platform is SOA. We’ve managed to build, in the middle of this, a small and focussed Java component that uses JMS in and out to avoid building horrible horrible BPEL or BPMN etc. But at either end, there’s some SOA bits to manage the integration with the “legacy”. So we’ve got this great bit of software, does amazing things, delivers real value to the business, which they want to put into production asap, but at every turn we’re hampered by the organisation or the technology platform. Its not so much the technology platform but the waterfall process the company has put around it. I’ve been told an internal wiki article detailing all the JMS  configuration is not  acceptable and the detail had to be in a Word document attached to an email requesting a not-production server configuration change. Word documents attached to emails are apparently far more “controllable” than a wiki with strong authentication protocols and history details in this world-view. Naturally I cut and pasted my wiki configuration detail into a Word document, spent a morning formatting it, and even added a link to the wiki before attaching it to an email. Acceptable process; configuration delivered. SOA and waterfall go together to make software development hell.

The second project has its many issues but one thing it does not have (for the components I have designed at least) is any SOA. It is all REST all the way down. There are multiple server-side-only components that all communicate to each other with REST over HTTP. Some of the data passed between servers is (or soon will be) JSON. The user interface is about to be delivered in two parts – one through actual REST-inspired web page requests to get the HTML and the other, via JQuery running on the web pages to actual JSON over REST services. These services will also call the REST services on the other components of the system (no database at the UI level). In fact many of the same JSON documents will be used to communicate from server to server as from ui to server. We are using Jersey for the REST. The process is Agile. The organisation hasn’t delivered such a large project with Agile before, and while I’ve been away on the other project there’s been some loss of focus. But despite some critical moments we had last week there’s been a renewed commitment to improving the engineering and project processes to achieve high velocity and good success. Its not perfect but it feels like the senior managers want the agile process to succeed. So, REST and Agile go together to make developer success.

Now, to the organisations. See if you can match the organisational description to the project style.

One of these two projects is a really fascinating piece of software in the transport industry (I can’t say more than that) which is integrating a number of “old” software solutions and creating some really sexy new features that the customer wants and loves. It is across what Eric Evans calls the “core domain” of the business. Not only is it a relatively short piece of work, but it will deliver high value to the business. It also enables a bunch of even higher value business services in the future. Things that directly expand their capabilities they can offer to their customers (which will be many of my readers) and do other related sexy things core to the business. The business salivate over it. It buffs their shine. They want it.

The other project is a little less flashy. The product is a rather niche product but it’s purpose is around the environmental effects of carbon emissions so at it’s core is the mission “save the planet from CO2″ — a pretty high value mission you’d think. However the actual functions of the software product are fairly mundane – based around helping large organisations capture and control data about their carbon emissions, because many of them have been required now to report these numbers to government agencies for a number of years. It has customers, all of then big companies or government agencies, and all the future prospects are similar sorts of organisations.

Which one is which? Which one uses REST/Agile, and which one is the SOA/Waterfall project? Do you think you can guess?

Would you be surprised if I said that the “sexy” transport project is the first one (SOA/Waterfall) and the second one is the REST/Agile project?

The basics of Tapestry is that it is a component-based web framework. Just about everything, including web pages, are components. Components may contain other components. The way it works is very simple and quite elegant, once you get used to it and weaned off the big-XML-file style of configuring a web application.

When you start, you have two main Java packages that are created for you (if you use the maven archetype, otherwise you will create these packages yourself). If your package root is say “net.crazymcphee.webapp”, then your two packages are “net.crazymcphee.webapp.pages” and “net.crazymcphee.webapp.components”. To configure a Tapestry 5 project with maven use this command and answer the prompts:

mvn archetype:generate \

    -DarchetypeCatalog=http://tapestry.formos.com/maven-snapshot-repository

Don’t use the one in the tutorial as it will not work! This is an excellent illustration of the first and most serious problem that Tapestry 5 has: the documentation not only has massive lacunas, it is also sometimes wrong and not updated.

Now, any class that you create in the “pages” package will automatically become a page in your application. But these classes need not be very complex at all. For example:

package net.crazymcphee.webapp.pages;
 
import java.util.Collections;
import java.util.List;
 
import org.apache.tapestry5.annotations.Persist;
import org.apache.tapestry5.annotations.Property;
import org.apache.tapestry5.ioc.annotations.Inject;
 
import net.crazymcphee.webapp.model.Person;
import net.crazymcphee.webapp.services.PersonService;
 
public class Persons {
 
    @Inject
    private PersonService personService;
    @Property
    @Persist
    private List persons;
    @Property
    @Persist
    private String searchTerm;
 
    Object onSubmitFromSearch() {
        persons = personService.findPersons(searchTerm);
        return this;
    }
 
    Object onSubmitFromClear() {
        persons = Collections.EMPTY_LIST;
        return this;
    }
 
}

You can see that this class doesn’t extend any infrastructure classes and has quite a simple structure. This is a fully functional page with two actions – one populates a list, the other clears it.

Now, to explain just a little here; PersonService is injected by Tapestry, but you need to configure in your AppModule which implementation you want to use. You can also use it with Spring as your IOC container but the one that comes with Tapestry is perfectly good enough for most applications.

The List, persons, and the searchTerm parameters are marked as @Property (so we don’t need to add getters and setters) and also @Persist so that the variables are preserved from request to request.

The methods “onSubmitFromSearch” and “onSubmitFromClear” are using a convention – “onSubmit” will also work, but assuming (as is true in this case) that we may have multiple forms on the one page, each method will only be fired from the “search” form in one instance, and the “clear” form in the other. These names are not special, it’s just (as you’ll see below) the forms will have these two names. They could be “Bill” and “Ben” in which case the methods would be “onSubmitFromBill” and “onSubmitFromBen”.

You will also note that these methods return the same page instance, which tells Tapestry to re-render the same page, but if you wanted to forward onto another page, you would add a instance variable, mark it with an @InjectPage annotation, and return that instance variable (after initialising it in your “onSubmit” method) instead of just returning “this”.

This page Persons, is available at {application-context-path}/persons. But there is one other part of the puzzle – the actual view. As mentioned, Tapestry uses convention over configuration and in this case, the convention is that the Page markup must be named the same: Persons.tml. Here is the matching tml file for the class above;

<html t:type=“layout” title=“Peoples I Might Know”

t:sidebarTitle=“Current Time”

xmlns:t=“http://tapestry.apache.org/schema/tapestry_5_1_0.xsd”

xmlns:p=“tapestry:parameter”>

<!– Most of the page content, including <head>, <body>, etc. tags, comes from Layout.tml –>

<t:form t:id=“search”>

<t:textfield t:id=“searchTerm” validate=“required” size=“20″/>

<t:submit t:id=“searchPeople” value=“search”/>

</t:form>

<t:grid source=“persons”/>

<t:form t:id=“clear”>

<t:submit t:id=“clearPeople” value=“clear”/>

</t:form>

</html>

Yes, it’s that bloody simple. The first form fires the “onSubmitFromSearch” method (with a little bit of validation, done in javascript) and the second method clears the list. In-between, there is this <t:grid source=“persons”/> business, which, if the ‘persons’ variable in the page class is populated, will show a list of its contents!

To test our app, we can use the command-line ‘mvn clean jetty:run’. When Jetty has run up, then we can point our browser at the web app: http://localhost:8080/sample-webapp/Persons, and with any luck we will see the following:

Now, by default, Tapestry creates the web app to use the template design as shown above. Of course, you can make it look completely some other way – or even use a totally different template if you want. The page template is just a component that’s included.

So if you enter in a search term and click the search button, you’d expect to see a result like this:

As you can see, the list is automatically populated with the details returned from the service method (in this particular instance, this are just a canned response, but normally of course they’d be the result of a database or a web service call of some type).

When you click the clear button, the list is cleared as you’d expect:

Now, here’s a really powerful feature I find in Tapestry. Let’s say our automated acceptance tests assert that when the ‘Clear’ button is pressed, the Search box as well as the persons list is cleared. What Tapestry allows us to to do, is keep Jetty running, edit the files in the IDE, and re-run the tests against the running app without restarting! We edit our method:

    Object onSubmitFromClear() {
        persons = Collections.EMPTY_LIST;
        searchTerm="";
        return this;
    }

In this case I’ve also added a bit of space around the clear button in the template as well:

And now our failing test will pass.

Editing files like this in the running web app only works for Pages and I think Components. If I had to change the service or model classes I’d have to restart, but “mvn jetty:run” isn’t very a heavyweight process.

As I said above, Tapestry’s not perfect: it’s major flaw is the poor documentation. Convention-over-configuration is easy to grok – if you know the convention. If you don’t and the documentation doesn’t tell you, and you can’t find a sample of what you need, it can be very frustrating. There is an excellent user list though.

Its other major flaw may be the stability of the API. Tapestry 5 is different (and incompatible) from Tapestry 4 is different from Tapestry 3. But so far I’ve used it on a couple of projects and I’m really enjoying it. I really hate the oodles of XML boilerplate and massive amounts of configuration found in Spring and I find myself somewhat reluctant to use it unless I really have to. Tapestry solves for me a range of different problems and mostly it presents a very elegant way to create a componentised web application. You might like to give it a try.

Attached is the sample code used above. It took me about 10 minutes to write (far shorter than it took me to write this blog entry!) - sample-webapp.tar bundle

anon 9:04PM
[about that code]

crazymcphee 9:05 PM
well, it WAS perfect … CRAZY perfect

anon 9:06 PM
lol… no doubt it even spans a 2yr time frame and 2 organisational puchases later, with no impact

crazymcphee 9:07 PM
exactly. all future and past and alternative universe combinations taken care of forever. no changes necessary.

anon 9:08 PM
sweeet… it should be a product

crazymcphee 9:08 PM
oh it already is just not in this instantiation of the multiverse

anon 9:08 PM
this instance has finished run level 3 yet

crazymcphee 9:08 PM
but in those OTHER multiverses, Oracle already bought it for $500mil

anon 9:09 PM
hahaha, i can see you partying with ellison and his geisha girls ;D

crazymcphee 9:09 PM
tried a ‘sudo shutdown’ but something’s threadlocked the kernel

anon 9:10 PM
then I stepped in a core dump

crazymcphee 9:10 PM
that’s why i’m hard at work building a mutiverse portal so i get me a slice of sweet ellison geisha-girl action

and here you are thinking about some OTHER sort of portal when i said i was working on an ‘Oracle 10g Portal Implementation’

anon 9:11 PM
don’t forget to add some proprietary and intrusive components that spread like a virus and grind all the other appservers to dust

crazymcphee 9:12 PM
well, that’s what at least half of those 10^100 multiverses full of new Indian IT grads are working on

I just send them a 500 page spec each month and they will get it to me at sometime before the death of this multiverse

anon 9:13 PM
that means it should be about ready by now, well as in it probably compiles in at least one of those multiverses, will be fully cmm lvl 5, yet not actually do what you asked

crazymcphee 9:13 PM
(oh by ‘spec’ i mean, a drunken rant shouted into my mobile phone on the walk home)

well, yes, but i’m fully expecting it will meet oracle’s stringent marketing requirements

anon 9:14 PM
well shit, they have to do some work, what do they expect, 4 u to write the code as well

yes marketing tickbox on the packaging is the only requirements they really need

crazymcphee 9:15 PM
yeah, i mean, hell, we pay at least $5.50 a day per developer …

oh we don’t tell the DEVELOPERS about the marketing tickbox requirements! they are super-secret. we just slap those on the box at the end. when i say ‘end’ i mean end of the box design process which has been ready for about 8 months now.

My case-in-point du jour is Oracle’s new Rich Enterprise Applications site and the associated technology. On that site you will find a flash presentation of their technology around delivering and I quote:

multi-channel capable applications that deliver desktop quality, highly interactive user experiences that are pre-integrated to enterprise class server technology.

That sounds pretty good, might be of interest to me. Lets investigate further. Now, a lot this technology is built on top of standards, like JSF, and other commonly used technology like AJAX. The earlier version of the ADF component technology was donated to Apache where it lives as the Apache Myfaces Trinidad project. The new components are built on top of these open sourced components. So far, so much is OK.

But, the kick in the tail can be seen in even just in the quoted sentence above: “pre-integrated to enterprise class server technology” – they certainly don’t mean Tomcat, right? The point of the presentation is induce lock in. However I’m still willing to cut them some slack on this particular point. Of course, if you are thinking about using Oracle’s JSF technology, you probably are already committed to Oracle’s middleware technology anyway – and Weblogic is a perfectly good application server, if you need such advanced capabilities (and whether you actually do is another question for another day).

But what absolutely does my head in, is the insistence the only way to develop for this technology is to write it using their proprietary, non-standard developer technology, aka JDeveloper. This is the point where I have to say, I’m sorry Oracle, but … FAIL. I actually don’t even care if your IDE is even the world’s best! Or even if it comes as an Eclipse plugin. My beef with this sort of IDE integration is the total mind-set that it embodies. The IDE might have all the drag-and-drop interface-building mojo in all the world, but the code it produces will never, ever, stand up to a hand coded model driven off the target domain, using modern agile development techniques that are designed to ensure the most cost-effective, best-quality development delivering timely business process transformation (aka value) that the enterprise prioritises and chooses. Just look at the ADF FAQ on the Oracle Wiki. Half the questions are related to the IDE. Here’s a tutorial style introduction that just assumes JDeveloper as the default way to get access to the Oracle JDF components.

No, no a thousand times NO!

The biggest insult is the way that developers are treated as mere commodities in this whole process. They are just the warm bodies occupying the chair driving the interface. Much like the continually failed dream of developerless development, their primary skill is seen as not “being a programmer” but “being a driver of a software package”. If I said, my financial skills and experience include 10 years Excel and 5 years MYOB – I can haz CFO jobz now? NO?! Why not? It’s insulting, that’s why not.

Software development is a profession, or at the very least a craft, and software developers should be able to choose their own tools. Obviously, some agreement has to be made organisationally, or at least across a single team or a whole project, to use a certain common toolset. Here’s one possible checklist:

We’re writing a Java app, check. Deploying onto Weblogic, check. Database is Oracle 11g, check. We have Scrum as our process framework, check. TDD is in our list of favoured engineering disciplines, check. Maven is our chosen build tool, check. Using Subversion for version control, check. Hudson for a continuous integration server, check. Writing the interface with Oracle ADF, check. Using Crossfire to deliver our web services, check. Spring for application wiring, check. Hibernate for persistence, check.

Each of these things has to be agreed upon at an architectural and organisational level (and obviously I don’t mean to say that each of the above is the only choice for each type of item, I’m just giving examples).

You should note one very deliberate omission from that list, the IDE. Architects might impose a certain application server, a particular approach to application layering, and so on. The developers – at least the senior ones – have to choose and agree, for example, on whether to use Ant or Maven, CVS or SVN, Cruisecontrol or Hudson. But what they do not do, is force developers to favour Eclipse or IntelliJ or Netbeans or JDeveloper. What is the point of taking someone who is super-productive with a particular IDE’s keystrokes and then forcing them to spend six months learning a whole new set, a new environment? That’s not productive! A developer needs to have the IDE get out of the way, become second nature – they need to be thinking about code, not where the menu is for the rename-method refactoring because they are unfamiliar with the keystrokes of the unfamiliar IDE. Keystrokes are muscle memory. You just think, rename this method and your fingers are already on the keys you need to press, without breaking your all-important flow. That’s how a developer is as productive as they can be. A productive development team should have a mix of IDEs (I’d propose that this would be statistically in proportion to the usage patterns of the common IDEs). Then we have the specific example of JDeveloper. I won’t even go into it except to say I’ve never heard a developer do anything but curse it, or maybe laugh at it. But it’s the laughter of the condemned man.

It’s something that vendors really need to get serious about. Rather than presenting me with a seamlessly integrated vision, from developer’s screen, through application server, to user’s screen, their technology presentations ought to think discrete. Each piece of technology ought to live and die on it’s own merits, something I’m sure Oracle’s ADF is perfectly capable of – without JDeveloper. If it’s not capable of being developed productively without JDeveloper, it’s already in the reject pile as far as I am concerned.

This essential information needs to be right up front. Don’t bury the information a thousand web sites deep in your developer portal. Put it right there in the first presentation! Separate the selling of the developer tooling from the selling of the user presentation frameworks and the server technology. Show me how I integrate Oracle ADF into TDD, into continuous integration, into the IDE of the developer’s choice. Because if you don’t, you’ve already lost me as a potential customer. This is the real future of software development. Look at where the major thought leaders of software development are taking development – open source frameworks, inversion of control, TDD, Domain Driven Design, Domain Specific Languages, Agile Methods, delivering real process transformation and business capability as early as possible – not drag and drop with wizards. FFS.

I might agree that for simple enough systems, it could end up being that in fact the service layers end up with PersistenceManager logic directly in their methods. But then I might say a language and framework like Ruby on Rails could suit your application better in that case. For any sufficiently complex Java-based enterprise system, a POJO-style data persistence layer naturally evolves, in order to reduce as far as possible the responsibilities the service layer must carry out.

As mentioned by commentators on his blog, the discipline of test-driven design is usually  a strong driver of having a separated set of components that deal with persistence and other data concerns. Not just from the issue over mocking the PersistenceManager, which is certainly a strong driver, but also from design principles.

Service layers marshall requests from clients, and decompose them to call various component layers. Data access layers typically form part of these component layers. A TDD discipline means these data access layers usually end up with more than just CRUD operations.

Furthermore, as a matter of discipline, in any sufficiently complex service layer I prefer not to work directly with database abstractions (i.e. entities that are modelled on database tables and their relationships) but on first-class objects designed to fit directly with the modelled domain. I want those objects to have rich behaviour – the service layer is actually part of their behaviour. I want to start the design of the system by directly modelling those actions, e.g.

passenger.at(origin);
passenger.orderJourney(dispatch, destination);
Vehicle vehicle = dispatch.sendVehicle(origin);
vehicle.pickUp(passenger);
vehicle.arrives(destination).dropsOff(passenger);
vehicle.notify(dispatch);

I do not want these objects to know about database semantics. That is a responsibility of a separated persistence layer, which we may for convienience sake, called the DAO. Even if internally we end up with;

class Passenger {
  at(Location origin) {
    this.origin=origin;
  }
  call(DispatchCentre dispatch, Location destination) {
    dispatch.passengerOrder(this, destination);
  }
}

DispatchCentre at this level, is a service, but we will model it as part of the domain of passengers, locations, and vehicles. Behind that service, it will have a database layer with the semantics as to how it saves and loads the data concerning passengers, locations, and vehicles. I don’t necessarily want the client objects (e.g. the web action) to have to deal with JPA annotated objects. They just deal with objects as much as possible. And I don’t see there’s anything served with developing possibly complex persistence logic directly in the DispatchCentre. In fact, taking a top-down approach, I know pretty much I won’t do that.

Only after I’ve done this initial part of the modelling, will I be concerned as to which of these components can be directly JPA or Hibernate annotated, which are Spring-injected services, which are Session EJBs, and the like. I think starting your modelling from the point of view of having a persistence framework like JPA or Session EJBs or any other ‘archtectural artifact’ leads to an unnatural design.

I am now a contractor for a large government organisation and as part of my role there, I get to participate in these workshops. Trust me, a fruitless exercise. People bickering about their favourite frameworks, why to use Axis over Spring-WS, why everyone should use maven style dependency management, maven is the devil, etc, etc.

Well, this has been a problem in Java circles for a long time now. I think we are learning to live with it. But what I don’t clearly see is its full relevance to Java “architecture”.

In my view, architecture is an independent set of constraints on the selection of the requisite frameworks; but the selection of the frameworks isn’t itself  an archiecture, nor necessarily an architectural task. In my view, achitecture is the view that says, “this is a web service” or maybe in more detail: “these are five components ‘C’ {Ca, Cb, Cc, Cd, Ce} having responsibilities ‘R’ {Ra, Rb, Rc, Rd, Rd} and they communicate via an asynchronous messaging bus in message formats ‘M’ {M1, M2, M3}”. The actual selection of the frameworks to accomplish this just should not be charged to a committee – ideally the developers should do it, with appropriate guidance from the relevant architect (who should be on the development team anyway).

Of course this proliferation of frameworks and standards to get this stuff done can lead such committees up the proverbial garden path. But, and without being privy to the full scope of the particular organisation Simon refers to, the essential problem I see here is the function of the committee – it’s just compounded by the additional issue of choice, and for some, those choices have inevitably become religious, making the said committee even more dysfunctional. Some would argue the proliferation of frameworks is something that makes Java stronger. There’s a marketplace in frameworks. But we all know about design by committee, don’t we?

That said, a lot ‘standards’ (even the JSR standards) are indeed irrelevant and out of date, as I pointed out only a few days ago.

In closing his blog entry, Simon leaves us with a question:

In closing, do you think JEE can be saved from becoming the next dinosaur?

Well, that, of course, is the million dollar question for a Java programmer isn’t it? Myself, I just prefer to get down and dirty and into the writing of the code – the business objects – that encapsulate the domain at hand. If we can’t just naturally select an uncontroversial framework to enable some segment of the desired infrastructure, then the best thing I find is to use your process, or more appropriately, your design, to just route around the controversy. But the best position is just not to be in that position anyway. Unless the chosen framework absolutely prevents you from doing something you absolutely need to do, you’re not quite in a pile of stinking do-do just yet.

And if your architecture committee is bogged down in arguing about framework selection, then I’ll tell you your real problem is is your committee, not the frameworks.

I’ve worked on a successful (JSR168) portal implementation, just last year. I truly don’t see what the concept of ‘portal’ brings to a web application beyond what could be simply programmed into a customised web application. We used Spring MVC Portal running under Websphere Portal and calling back-end services deployed on a BEA Weblogic server using Spring remoting (note: not WSRP which would have probably massively increased complexity). The reason this mix of platforms (Websphere and Weblogic) was used was political, not technical. I believe the whole use of portals in general tends to follow organisation’s political and not technical needs.

Despite all that, overall the custom portlets and the project as a whole was quite successful and the customer satisfied with the result. However I felt the most difficult aspects of the project were the parts of the deployment that were left to the portal container to provide. These are the things that are touted as the ‘advantage’ using a portal container over a simple custom-built web application. Content management features caused the team members delivering these enormous headaches (and late nights) and the same can be said of the authentication and authorisation mechanisms employed. I felt we could have easily delivered these components using known JEE development APIs with all the customisation required by the customer in at least an equal amount of time.

As a contrast, we used Tapestry 5 to develop (actually, completely rewrite from an earlier phase one version in dire need of improvement) an administration component as a straight web application deployed into the Weblogic server. This took a pair of us less than two weeks to write using a functional-test-driven approach and delivered a very nice web app.

The issue of portal containers and standards being a political rather than a technical concern is particularly pertinent to me at the moment because it looks likely in the near future I’ll be working on another portal implementation for another client. I can’t be more specific regarding this but this particular client had already bought the portal server from a vendor and then approached consultants in order to get an actual implementation completed. Of course, no requirements gathering or analysis was done to determine if the portal solution was indeed the right one – the vendor’s sales team effectively provided the assurances of product suitability. Because the money is already spent and reputations already staked on the portal platform there’s no possible way a better solution can be suggested by any technical expert at this stage.

In my view, that’s the “relevance” of portal servers in the Java world. Vendor sales teams love to sell them.

Update

Jakub Holý makes an excellent observation about validity of the concept of a portal server:

It’s a pity that portals and portlets aren’t easier to use and haven’t made it into the mainstream, the abilities of content aggregation, personalization, uniform security etc. are promising and as we can see in the rise of personal mashups like iGoogle, the fundamental ideas behind portals are still valid and extremely attractive. I hope that some mashup technology will soon provide a viable, light-weight alternative to portals.

That’s an important point: iGoogle is the most widely used example of a “portal” architecture. The modern trend towards the “mashup” (especially in social software) also shows the idea to aggregate different web applications under a common view, with security and other features is a valid one: it’s just the bloated, slow specification process, and predatory vendor sales departments that are stopping real innovation in this space in the Java area. Hopefully some light weight technology will soon emerge from among something like Spring, Tapestry, and GWT to produce a “portal” server that delivers actual value, and ease of development, deployment and usage, in a corporate context.

Every few years a vendor, or a group of vendors comes out with some bright idea to allow some class of end-user, or business analyst, or what have you to build their own systems, in recent years usually by some type of graphic language (draw your system’s program flow right on the screen! bypass those pesky, and expensive, programmers who just ask confusing questions and force you to clarify your thinking!  press the magic ‘deployment’ button and your half-baked idea is in production in seconds!). In the past, these systems used “natural” language or “plain English”  languages such as say Director Lingo.  Every few years it moves onto some new paradigm or expression of the old one. And it always just ends up simply with a whole new class of programmers being created.

Do not trust any product – or standard – that promises to deliver “programmerless programming”. Plenty of 4GLs like SQL and the like are originally meant for non-programmers to use … imagine giving a business user just a SQLPLUS prompt now! Included in that mix there was CASE … a recent example from the past few years I can think of might be BPEL and ESB style tools, or for another example a number of attempts to define architectural languages that generate the system components for you off the UML or similar.

A big issue I see with these systems is that they tend to be highly resistant to modern programming principles and practices. For example, test-driven design – or even simple unit testing. Sure, if you’re building a web app with these things, maybe you could use a web-testing framework to build functional tests and build-to-pass. But most of these tools are very resistant to a test-first approach.

Take Oracle’s BPEL tooling. For a start, you’re forced to used JDeveloper, which is a crap IDE you’d never otherwise inflict on yourself (it is so bad, on start up it displays the most meaningless product slogan I’ve seen seen on such a tool – “Productivity through Choice”). But more than poor tooling, the very paradigm is completely broken. Forget about a test-first approach (although I’m sure there’s some seriously weird way to achieve it), just try setting the damn tools to have the traditional development, testing and production environments. When ever you need to call a service end-point, you are bound to the particular host and port you specify – it’s embedded in XML document that represents the BPEL design. The workarounds? Well if you are super organised you could arrange for your different environments to contain identical setups and  use DNS hacks (or host files) to ensure that each service looks the same when you move from development to integration testing to production. Or you can write Ant scripts (or sed and awk if you really want) to search-and-replace the offending hostnames and ports and replace them with the correct values for the target environment. Clearly, once you use it, you see that the tool’s paradigm is, in reality, to just hook up the IDE to the production server, draw your pretty flow chart modelling your business process, then publish it straight back into production. In other words, there’s a whole new class of programmer out there now – BPEL programmers – and life must be a nightmare for them. Its fundamental paradigm is broken. Given me any language with a unit test framework with a half-decent programmer on any day of the week, and you will beat the pants off this system in productivity and choice. It’s living a lie – and a big one.

Ultimately, as each new generation of “programmerless” program-construction tooling comes along, it allows developers to do more powerful things (and sometimes, less powerful things in a more retarded way). And the scale of the task always increases to the point where development specialists are still needed. In my view, there is a definite on-going role for for the talented programmer/analyst who knows how to take a vaguely-worded, incomplete, and contradictory bunch of requirements expressed in common verbal language and then ask the right questions in order to clarify them and decompose the requirements to a series of simple logical components – and then develop and deliver those components in the form of usable systems. You need to continuously learn new languages and systems, but the basic skills stay the same.