Monday, July 29, 2013

Thanks H2!

After struggling with Java persistency I was getting a tad desperate. It is not that hard to get any of the tutorials to work but virtually all tutorial are creating unwanted couplings and often have reams of dependencies (often hidden behind inconspicuous looking maven poms). In an OSGi environment you need a persistent solution that works well with components. This means that in general components should not be bound to a specific brand of database because this would make these components not very reusable, they would work only in environments that had the same database brand. I was getting a bit desperate because it was so hard to find a working combination that did it the OSGi way.

The OSGi way is that the database is provided as a service. By providing it as a service you allow the deployer (the person that provides your application a place to run, which is often yourself but it makes sense to separate these roles) to pick a database brand at runtime by deploying the proper bundle and configuring it. That is, as a user of the database this is the code in the OSGi way:

public class DatabaseUser {
   DataSource ds;

   void setDataSource( DataSource ds) {
     this.ds = ds;

This model is very elegant, it uncouples a lot of details from the component. The component does not have to worry about database configuration, transaction model, passwords, pooling, and not even its brand. Obviously, a component can have special dependencies because it uses proprietary features of certain databases (SQL is treated by vendors more like a recommendation than a specification), these can always be reflected by Require-Capability headers. That said, careful components can restrict themselves to a lowest common denominator so that they can work with any or most  SQL databases.

This leaves us the problem how to get the DataSource object without specifying its brand (or password!) in our code/xml? In a perfect OSGi world we would have JDBC drivers that use Configuration Admin to specify the desired Data Sources; the driver bundle would just register these configured Data Sources. This requires a trivial amount of code since JDBC drivers already configure themselves with properties, a good match to Configuration Admin.

We did not choose this model during the OSGi Enterprise specification because we thought this might be too much work for the driver vendors. I guess that was the correct assessment since the even easier to implement DataSourceFactory service that we did specify is still rare. The Data Source Factory service provides methods to create configured ConnectionPoolDataSource, DataSource, Driver, and XADataSource objects. With this interface it is fortunately not very hard to make a generic component that registers our desired Data Source services:

public class ConfiguredDataSource extends DataSource{
   DataSourceFactory dsf;
   DataSource ds;

   void activate(Map map) throws Exception {
     Properties p = new Properties();
     ds = dsf.createDataSource(p);

   public Connection getConnection() throws SQLException {
     return ds.getConnection();
   public Connection getConnection(String username, String password) 
    throws SQLException {
    return ds.getConnection(username, password);

   void setDataSourceFactory( DataSourceFactory ds) {
     this.dsf = dsf;

Realize hat this code is much more flexible than it looks at first sight because it leverages Declarative Services's features. Through Configuration Admin you can bind it to different brands of databases, you can set passwords, configuration options, and instantiate multiple DataSource services. However, the hard part, as is sadly still too often the case with OSGi, is finding a bundle that implements this specification. Without much hope I looked at the H2 database that I was already using in my prototypes. It is a fine example of a Java library. Rarely have seen a product with so many features, so small (1Mb), and so few dependencies (0). It is delivered out of the box as an OSGi bundle. And it actually registers a Data Source Factory service! I also got H2 to work with OpenJPA the OSGi way since the OSGi Specifications mandates JPA to use the OSGi way for JDBC, though this did not work out of the box yet. If only more code was delivered like this ...

To conclude, Java persistence is still a mess in my personal opinion; it will require a lot of work to make it all work the OSGi way. However, at least there is light at the end of the tunnel. Thanks H2 guys for this good out of the box component experience, hope you set a widely followed precedent. You definitely gave me hope.

Peter Kriens @pkriens

P.S. If you want to experiment with this approach, look at OPS4j Pax JADBC, the created adapters for laggards in this area.

Monday, July 22, 2013

Accidental Complexity

I am currently writing an RFP for the increasing adoption work I will do for the OSGi Alliance. To understand the landscape, I interviewed a number of software companies. The first interview was a Spring/Maven shop, representing mainstream Java development. However, I've seen that PHP is very  popular in the lower segments of the web applications. Since I am a strong believer in the powers of Java, I always wondered why those people 'struggled' with PHP so I decided to interview a small PHP shop to find out why they used PHP.

The story they told me over lunch was quite interesting. Early 2000 they had some people go on a course at IBM to learn about Java EE. After many thousands of dollars and some weeks later they all had become utterly confused. In their eyes, Java was just way too complicated for what they wanted to do. PHP gave them a simple platform that was more than sufficient for their needs. And not only was it much easier to learn, it was also much easier to find libraries and snippets that they could use in their daily work.

When I tried to defend Java (neutrality is not my stronghold) I found that most of those really good things in Java fell on deaf ears. For example, refactoring is one of my top features in Java but they had no idea what it entailed. So I found myself desperately trying to explain refactoring in terms of a type safe search and replace but it was clear that their eyes glazed over, wondering what the big deal was (ok, the rosé might not have helped). It was clear to me that much of the accidental complexity we have in Java was not understood to have sound software engineering reasons, for them it was just painful and overly complex. PHP was so much easier to get started with. I must admit that when a friend, who is a non-technical sociology professor, built a professionally looking PHP website, including large forms and payments, I was duly impressed.

After some more Rosé my lunch partners did actually come up with problems. Problems that I think could be addressed but the problem was that they saw these 'problem' as facts of life, not something that could be addressed.

Therefore, the primary problem that we should try to address is how to cross the steep threshold that Java puts between an uninitiated developer and a web app. I think having a framework for web apps that provides a skeleton for other applications is the start point that will make a difference. However, to also attract non-Java developers we will need to minimize the accidental complexity of Java and OSGi. With DS annotations and bndtools I think OSGi's accidental complexity is ok for the powers it offers. However, after researching how to handle persistency in Java I find it hard to apply the Java EE APIs in OSGi so that they provide an out of the box experience like PHP. At the same time, I find some other libraries that are not Java EE but seem a lot easier to use. It will be interesting to find out how we will weigh the compatibility requirements against the simplicity requirements.
Peter Kriens (@pkriens)

Tuesday, July 16, 2013

Real Men Don't Use DS

In general watching Stackoverflow is a pleasant way to spent some spare time. Helping other people out feels good and it is nice to see how the number of OSGi questions, and thus adoption, is increasing. However, it can also hurt looking at how people are struggling because of their own choices. There is this idea in our industry that real men start with the command line and Bundle Activators. Bare metal! Somehow this is considered to be a better way for real men to learn a technology than using a sissy IDE. Just like real men do not learn how to drive a car until they can clean and tune the carburator! The fact that  carburators have been absent in cars since the late seventies seems irrelevant to them.
I've written forewords for numerous books about OSGi and virtually all made the same mistake that to understand the technology you should first proof your worth by struggling with a Bundle Activator and a Service Tracker. Just like cars today have electronic injection so does OSGi have dependency injection. Use it. My strong recommendation is that if you want to learn OSGi, use bndtools with a Declarative Service (DS) example project. Make sure you understand the life cycle of declarative services and their immensely powerful integration with Configuration Admin. Realize that any class can be made a service with a simple annotation and any service can be made a dependency with a another annotation. If you need something to initialize, add an activate or start method to your service. Check your imports in the content pane and adjust imports them with drag and drop to make them cohesive and simple to use modules. Visualize what and how is running with Apache Felix Webconsole and Xray. Enjoy the sub-second edit-debug cycle. That is what you should learn and fall in love with because that is what makes OSGi so powerful.
Only after your mastered this initial level start looking under the hood and find that all the goodies that real men need are really there: Continuous integration, command line stuff, low level service access, weaving, proxying, bundle activators, bundle trackers, service trackers, whatever. It is wonderful to know it is all there when you need it but it is not what OSGi is all about.

Monday, July 15, 2013

The Trouble with Objects

When I discovered objects in the early '80s my mind was blown away. This was a technology that felt right, it gave me a way to think about my software that I utterly missed in structured programming. My career in the 90's centered around helping companies getting started with objects. Objects had clearly won, it was a run race. Lately, however, I have been getting more and more doubts.
The unique selling points of objects is that the data can change while the behavior remains the same (or at least backward compatible) via its public interface. If the world is the process then this is a great model. However, with 10Gb ethernets and larger, larger problems, cheaper computers, the world of a program is no longer a process: You are building distributed systems.
Distributed systems exchange information. In Java an attempt was made to preserve the object oriented semantics by not only communicating the data, but also sending the classes. However, ensuring that each system would use the correct class version and maintaing security turned quickly into a quagmire except for the simplest of systems. And even if it does work, it excludes any non-Java participant from collaborating. It is not that the OO community did not get any warning signals. The object oriented impedance mismatch with relational databases shows, at least in retrospect, that hiding your data while persisting it is not an option. At the time, we thought this was a problem that would solve itself when object oriented databases were ready (any day now!). Alas, they largely ran into the same problem that ensuring that the classes that read/wrote the data were of the same or compatible version was hard, if not impossible in many cases.
Looking at Java (and non-Java) persistence solutions I was ran into JOOQ, a library that declares peace between Java and SQL. Instead of abstracting it, it provides a type safe, fluid builder, way to use relational databases. It gave up on abstracting the database but it seems to have won a lot in simplicity and exploiting the powerful relational model. I cannot but wonder if all those abstractions to make the life of the application developer easier are in the end worth the increased complexity. Would it be worth to create a small, pet clinic like, OSGi application that would use all these different persistence models?

Monday, July 8, 2013


My assignment for the OSGi Alliance is to increase adoption by making it easier to get started with OSGi. So I am currently writing a Request For Proposal (RFP), the standards requirements document in the OSGi Alliance. One of the primary parts is the Application Domain. In this section you neutrally describe the current practices, it is basically used to scope the problem domain area and provide a vocabulary for the subsequent sections and documents.
So last week I started the section persistence. It is an area that I have rather little experience with so I welcomed the chance to work with them during my sabbatical. I picked the document oriented database Mongodb because it felt much easier to work with in an object oriented environment than relational databases, and I must admit that choice has made me quite happy (except for the lack of transactions). However, it is clear that relational databases are the bread and butter of web applications. I therefore had to look deeper into what's happening in this area.
I then stumbled upon the debate around Java Data Objects (JDO) and Java Persistence Architecture (JPA). I had the privilege to work with Mike Keith (Oracle, JPA spec lead) on the OSGi JPA specifications so I knew something about JPA. However, so far I'd never seriously looked at JDO, and actually thought JPA was replacing JDO. Unfortunately, life seemed to be not so simple.
Both JDO and JPA define metadata to store normal Java objects in a database. JDO focuses on any type of persistent store (even S3 seems to be supported) while JPA is only used for relational databases. JDO seems to be a real working horse that got a new lift when Google selected it for its Google App engine. Though JPA is the new kid on the block its decision to limit itself to relational databases seems a severe limitation with the increasing popularity of NoSQL databases like Mongodb, Cassandra, Neo4j, etc. So far, it also looks like JDO is more portable and flexible while JPA seems faster. However, lots of experiences seem to come from toy projects or evaluation projects. Now lets not start a flamewar but I would love to hear (neutral) experiences of the use of these technologies in real life sized projects. Obviously I am extremely interested in how they work in OSGi.

Peter Kriens

Monday, July 1, 2013

The Architecture

Last year when I started with jpm4j I spent quite a bit of time exploring the state-of the art technologies. Even before that start point I had decided that the world had changed, most Java web frameworks looked very outdated with the advent of the upcoming HTML5. Now, with 55 years I guess I am an old geezer so I remember time sharing (punch cards, ASR/Teletypes, ending in intelligent terminals), the swing to the PC applications that gave us 'fat clients' in the eighties and early nineties, then the pendulum going back to 'thin' browser based clients from 1995 and onwards; and now the back swing to fat Javascript based clients. For many Java developers that started in the late nineties, no longer being a rebel but being treated as an incumbent will come as a nasty surprise since most Java web frameworks like JSP, JSF, Vaadin, Struts, Spring MVC, Wicket, and hundreds of others will be relegated to the dustbin of history over the next decade.

What changed? What has changed is HTML5 combined with Moore's Law. For the first time, there is a software platform that allows you to develop a single application code base that will run on virtually any user facing machine. From smartphones to high-end workstations. Though the rudiments have been available for a long time it was not until the browser vendors/organizations kicked w3c's lead and decided to go on their own in the WHATWG workgroup that we got a pretty decent, fairly wide, specification for web applications that was quickly supported by virtually all browsers. We are back running our application code on the client but this time we have no deployment problem nor having to worry about the client's Operating System.

This is of course a game changer for Java web frameworks. In a Java web framework the UI is managed in the server. This model requires a round-trip from the browser to the server when the GUI needs an update; a slow process that not only places a heavy demand on the server but it can also feel sluggish. With HTML5, the server can focus on the data and becomes oblivious of the GUI. The client worries about the graphics and interaction. The biggest advantage is scaling, no longer is the server required to maintain the often large state of the GUI; instead it provides a REST interface or JSON RPC interface . So not only is the footprint reduced, distributing the workload is even easier since the  protocols can easily be made stateless. Another advantage is simplicity, in my experience the server code becomes considerably smaller and therefore much simpler.

So why am I so excited about HTML5 since it seems irrelevant to OSGi? As  Emanual Rahm said: "Never let a serious crisis go to waste". I believe that HTML5 will be a disruptive change, causing a lot of web applications to be rewritten since customers will demand the responsiveness of local applications. This offers an opportunity for OSGi; an environment that in almost all aspects seems to have been designed for this paradigm shift.

Peter Kriens