Multiple Bundles per Project

I just got back from a very successful JAX London, these are always very well organized conferences and this conference was no exception. Yesterday we had an OSGi day during which Neil Bartlett succeeded in writing a Vaadin Web application without having to restart the application OSGi framework. OSGi is well on its way to provide a development process with much shorter turn around time than any other environment I know. All because of its dynamic model, there is just no need to restart the framework even if you make very big changes in your code, even your bundle layout can change or the run environment.

During this presentation someone in the audience asked why bndtools supported multiple bundles per project because all other environments (like Maven, PDE, etc) limit a project to a single artifact. So why does bndtools have multiple bundles per project? There are several reasons and I think it is interesting to highlight them.

The first reason is practical. In the OSGi build for the Reference Implementation and Compliance Tests has over 1300 bundles, mostly test bundles. Having to maintain that many projects would be very painful. In reality we now have only around 130 projects. I shudder what it would mean to have 1300 projects in your Package Explorer ...

The second reason is cohesion. Sometimes your deliverable consists of multiple bundles that are actually quite cohesive. For example, assume there is a bundle that is useful on its own but someone wants to use it in conjunction with Spring. To support that use case, your bundle then requires an import of a Spring package, which would transitively drag in many other packages. You could make this import optional but that is awkward. The best solution would be to create two bundles, one with,  and one without Spring. You could even make them a main bundle and a fragment for the Spring dependency. Being forced to put these two bundles in different projects creates an artificial separation that unnecessarily complicates development.

Last but not least, when we begin, we rarely know where the module boundaries are optimal. In bndtools an extra bundle has no overhead. It is easy to move packages between bundles or even allow them to live in multiple bundles. Experimenting with splitting a bundle or combining, or even just creating a test bundle, has no associated costs. In my experience, this changes the way you think about bundles, the process of developing them becomes more fluid. I rarely have projects that produces a single bundle nowadays. However, if you prefer a single bundle per project then that is your choice, 1:1 is the easy case.

That said, I actually never understood why popular build tools are constrained to a single output per project. Isn't that a little bit too much convention over configuration?

Peter Kriens


P.S. Ok, ok, Neil's Eclipse instance crashed because it ran out of heap space killing the application framework as well. So he almost reached perfection. :-)

OSGi lite

Yesterday we had a rather heated discussion during the CPEG meeting here in Austin.We discussed OSGi lite for which I had submitted an RFP (the requirements document). Key issue was the positioning, there was a general concern that an OSGi lite detracts from the core message: strong modularity. It would encourage you to work more modular but lacking enforcement of the module boundaries would not help solving many of the problems that plague our software industry today: we too often build on fuzzy foundations.
In my previous post I stated that OSGi lite would provide 80% of the benefits and this was seen as too high by some. I do agree that the 80% was not very scientific, it is a very personal estimate. I picked 80% because I do believe that the µservice layer  is the most important software innovation since OO came around in the eighties. Building systems out of µservices automatically makes systems more modular because they minimize  coupling.

That said, the modularity layer also does provide lots of advantages, especially when used within larger groups. Strong modularity can enforce rules across the development life cycle that are vital for large projects. As John Wells (from BEA at the time, now Oracle) once said: "If you think you're working modular but you're not using OSGi, then you're not working modular." Unfortunately, most people are not working modular and find the path to OSGi's strong modularity too painful. OSGi lite is the on-ramp to OSGi because it provides immediate benefits in any WAR or class path based application without any pain. Once you moved your code base to µservices you can then more easily transition to OSGi to reap its additional benefits.

So the question of conscience is, what will I use? If I think I get 80% of the benefits, why bother with the real thing? 

After working with OSGi lite last week I felt, a bit to my surprise, that the development cycle with OSGi is much faster and smoother than with OSGi lite. The difference is the dynamic update of OSGi. The Eclipse debugger can replace code in memory but it often has to restart because certain changes cannot be applied in a running process. Using bndtools this is never a problem because every time you save the source file it rebuilds all artifacts and deploys them on the running framework immediately. It works so smooth that that the deployment step becomes largely invisible. So with OSGi you can work for days with the same framework happily humming along. In contrast, working with OSGi lite forced me to use the standard Java Application run mode and the GWT Servlet runner that were not even remotely as smooth. And not to forget, the framework finds many hidden errors like missing packages, invalid versions, etc. before your code stumbles over them.

Last but not least, with OSGi I know it will deliver when I run into hard problems because it is mature. I might today not have many of the problems OGSi provides solutions to, but when tomorrow comes that situation might differ. If you would develop on OSGi lite you're bound to have a serious porting problem when you would not test/develop your bundles on OSGi regularly. After all, its strength that allows all these class loader hacks is also its greatest weakness.

Peter Kriens

P.S. All opinions in this blog are mine not necessarily the opinion of the OSGi Alliance. As should be clear in this case, OSGi lite is not without controversy inside the OSGi Alliance.

OSGi Lite

A few years ago I wrote a blog about OSXA, an OSGi framework without the module layer. Over the years, this idea kept intriguing me and discussed it with many but never found the time to work on it. Unfortunately, OSXA seems to be no longer active (the web site appears to be in zombie state) but thanks to Karl Pauls this idea has given a new lease. Last week, at the (fantastic as usual) EclipseCon/OSGi DevCon, he surprised me with an implementation of OSGi Lite. OSGi Lite is OSGi without the module layer; he based his implementation on Apache Felix.

OSGi without modules is so interesting because it significantly simplifies the migration to the service layer from existing (web) applications. Services are for me the key innovation in OSGi; the module layer was only a necessity to make that service layer work. Playing with OSGi Lite I realized that in a way we've been trying to convince the market to go OSGi without a good transition model. We promised all those goodies but then adopters had to basically refactor their whole code base before they could benefit from these goodies. Not particularly good marketing.

The primary transition problem is that today class loader hacks are prevalent in Java code. These hacks make it appear that "modules" are decoupled from each other.  For example, the JVM's Factories pervasively use a multitude of class loading hacks to desperately find an implementation for an interface; the Service Loader model supposed to improve the class loading hacks even hard codes the class loader in its API. Though an extension mechanism might be based on class loaders a cohesive design should not expose this implementation detail in the API as the Service Loader does.

All these class loading hacks are the antithesis of modularity because they mandate the visibility of implementation classes across module boundaries. Your POJO might look superficially  like a POJO but in classic Java someone in your code base must violate module boundaries to get an implementation. You can try to hide in some Spring XML or  Guice module but it is inevitable that someone needs global visibility. This is the exact problem addressed by the OSGi service layer. In OSGi, the only one knowing the implementation class for a service is the module that owns it, as it is supposed to be. It is even better, the module can register instances with context instead of having to revert to statics to provide context.

However, to be able to reap the wonderful benefits of the service layer one has to remove all the class loading hacks and that is usually a big change. Big changes in an active source base are usually not such a brilliant idea. Therefore: meet OSGi Lite.

Karl's OSGi Lite implementation is based on Apache Felix, ensuring good fidelity with a real OSGi framework.  His implementation runs in any standard Java environment: class path, WAR files, wherever your current Java is running because it never touches a class loader. Running an OSGi application is as easy as running the main method in PojoSR while the class path contains the "bundles". PojoSR will scan the JARs on the class path and treat each JAR as a bundle regardless if it is a bundle or not. If the JAR is a bundle and contains a Bundle Activator then this is activated. If you want to own the main method, you can initialize the OSGi Lite subsystem yourself using the META-INF/services model.

It is quite surprising how many bundles actually work out of the box: Declarative Services, the shell, etc. all work well with a bit of twiddling. With many bundles the only problem is the Bundle-ClassPath, this obviously does not work. However, it is trivial with bnd to unroll the bundle class path with the @ operator. For Felix's declarative services I had to do unroll the Bundle Class Path because it included the KXML parser.

To demo OSGi Lite, I wanted to port an existing application to OSGi to show how easy it has become.

After "one of those days" where every open source project I tried to build failed horribly(SpringSource Pet Store, Sun's Pet Store, some GWT Pet Store, and others). After half a day of horror I ended up with the Google Web Tookit  (GWT) standard sample application. GWT provides an Ajax based GUI environment combined with a Remote Procedure Call (RPC) model. The demo application sends a message to a GreetingService (!) and displays the result in a dialog.

The standard GWT GreetingService is definitely not a POJO because it must extend the RemoteServlet class. From a modularity point of view this is bad, really bad. Now we have OSGi Lite we can create a Dispatching Servlet bundle that decouples the Greeting Service POJO from the web environment. The Dispatching servlet creates the OSGi Lite runtime with Karl's code when it is first constructed. It then receives all the GWT requests and then uses the service name, which is embedded in the request, to find the service in the OSGi registry. It therefore is not coupled to any service code at all, it only knows the standard GWT classes. The next stage was to create a bundle that provided the Greeting Service. Well, with bnd's annotations and bndtools that was a snap. These two bundles then had to be placed in the WEB-INF/lib directory to become part of the WAR's class path.

To test this I used the GWT Eclipse plugin, this provides an environment based on Jetty to run a directory that looks like a WAR. This was a bit painful because I had to copy the bundles to the lib directory whenever I made changes and for some reason all the XML and hard coded path names make this stuff incredibly fragile and error prone. I really missed the standard dynamic run environment from bndtools! But I got it to work. I then created a war file and ran it on AWS Beanstalk, which worked like a charm (might be gone by the time you read this because it costs money to run there). You can find the workspace for this code at github, see the aQute.gwt.war project.

What did we achieve? From a modularity point of view the code has been significantly cleaned up. We could turn the Greeting Service implementation into a real POJO that no external dependencies while the old one was heavily coupled to the servlet API and GWT's implementation. Now, the service is deployed as a separate bundle and can easily be substituted. Also in this case the service registry demonstrates the shining power of the µservice model!

Conclusion
OSGi Lite provides 80% of the power of OSGi in a completely non-intrusive way. It allows any application build in Java to reap the benefits of increased modularity without first having to rid an existing code base of any of its class loader hacks; all these hacks just keep working as they used to work because everything runs in a single class loader.

The drawback is is that OSGi Lite does not enforce module boundaries and obviously does not allow multiple versions of the same package, nor does it support the Bundle-ClassPath. However, with OSGi Lite applications can move to the µservice model with virtually no complexity. The µservice model can then be used to get rid of the class loading hacks over time, after which it will be really easy to move to OSGi and get side by side versioning and real module boundaries.

It would be interesting if the OSGi Alliance would standardize this OSGi Lite model. So from these experiences I've written an OSGi Request for Proposal (RFP) and submitted for discussion next week in Austin. Though a standard is not absolutely  needed to start playing with OSGi Lite now Karl has implemented it, it would be better if the model was officially condoned by the OSGi Alliance. It could be used without coupling to a particular implementation.

I think this could be one of the biggest enablers for the OSGi model ever. If you're interested, check out the code and provide feedback!

  Peter Kriens

Karl Pauls' code: http://code.google.com/p/pojosr/
The GWT project: https://github.com/bnd/aQute

P.S. All opinions in this blog are mine and obviously not necessarily shared by the OSGi Alliance.