OSGi BOF at OSGi Community Event

The OSGi Alliance will be hosting an OSGi BOF at the OSGi Community Event next week.  The BOF is scheduled for 19.00hrs on Weds 30 Oct in the Silchersaal room at the conference which is located at Forum am Schlosspark in Ludwigsburg.

Peter Kriens will provide a short introduction to "OSGi en Route" a new initiative from the Alliance which is intended to make it easier for developers to get started with using OSGi.  Peter will be keen to answer any questions and get your feedback and suggestions on this initiative.

As with any BOF the goal is for everyone to participate, so if you have any burning questions, problems, or 'itches you need to scratch' then please join us for the BOF to share them with us.  These can be anything you want to discuss, from a specific problem you are facing with OSGi development, through to more general community or OSGi futures questions.

We had a great BOF at the Community Event last year with fantastic participation and we are looking forward to repeating it again this year.

If you unfortunately aren't able to join us but have something you would like raised feel free to submit a comment to this post or email me and we will try and cover this during the session and will feed back to you.

bndtools 2.2.2!

There is a new release out for bndtools! What is it? Well, bndtools is an Eclipse plugin that provides some very nice tools to develop OSGi bundles, it is also used by the OSGi Alliance itself to build the RI's and run the test suites. Since it is based on bndlib, it provides lots of functions that work well together with the maven bundle plugin, ant, sbt, gradle, and many other build environments. The bndtools plugin provides a friendly environment to develop bundles that automatically can then also be built in a continuous integration setup, using one of the aforementioned plugins.

The 2.10 release was a major release in which we added baselining. This release has many smaller features but definitely did improve a lot of things.

The easiest way to get started (or update) with this Eclipse plugin is through the Eclipse Marketplace (In the rather unexpected Help menu). You can also install it manually from its update site: http://bndtools-updates.s3.amazonaws.com

Enjoy, and let the team know how they are doing ...

   Peter Kriens @pkriens

Are you coming to Ludwigsburg next week?

The OSGi Community Event 2013 is taking place next week in Ludwigsburg, Germany between Tues 29 and Thurs 31 Oct.

We have a schedule that is overflowing with OSGi goodness.  The conference kicks off with a "Mastering OSGi with Ease" tutorial on Tuesday morning. From Tues afternoon to close on Thursday there are 26 OSGi talks ranging from embedded, to enterprise, to cloud and covering topics from development hints, tips and best practices, to case studies on how OSGi is being in the world. In addition Thursday morning sees an OSGi Keynote from Ian Robinson of IBM called Travelling Light for the Long Haul.

The full schedule is available online.

There will also be an OSGi BOF on one of the evenings offering the opportunity to find out more about some of the current OSGi Alliance initiatives and to discuss OSGi opportunities and challenges that you see.  Further details on the BOF will be announced soon.

The Community Event is co-located with EclipseCon Europe and there is still time to register to secure your place.  All attendees also get full access to the EclipseCon Europe conference. Details on registration, location and hotels are available on the Community Event homepage.

There are of course plenty of social activities and opportunities to network with your friends and peers, and of course drink and eat plenty of German beer and food!  Tuesday evening sees a Stammtisch and Cirque d'Eclipse; and a reception will be held on Wednesday evening.

It you are joining us for the Community Event you may also be interested in the popular full day Code Camp that the OSGi Users' Forum Germany are running the day before the Community Event (Monday 28 Oct) - Building OSGi based HTML5 Web Application with Peter Kriens and Neil Bartlett. Full details of the agenda and how to register for this are available here.

If you have any questions please email the OSGi Community Event Program Committee.

We hope you can join us and look forward to seeing you next week.

Breaking Bad

In my quest to get OSGi and JPA working together purely through services I ran head on to an API that breaking my code badly: java(x).sql. After making things work on Java 6, BJ Hargrave (IBM) setup a continuous integration build on Cloudbees. Since Java 6 is end of life, he rightly picked the Java 7 JDK. Unfortunately, this broke my program since I was proxying the DataSource and Connection class from java.sql and javax.sql respectively. After inspecting it turned out that Java 7 had added a number of methods to javax.sql.DataSource and java.sql.Connection, effectively killing every driver and delegating proxy in existence since they did not implement these methods in their implementation classes.

The problem is not breaking backward compatibility. JDBC moves on and if you want those features drivers will have to upgrade, fair enough. The problem is that decision to upgrade your database drivers is now tightly connected with the choice of the VM. If you want to run on Java 7, JDBC 4.1 is forced upon you since every previous driver will fail to run on Java 7.

It is illustrating to look at the hoops a database vendor has to jump through to allow its drivers to run on Java 6 and Java 7. It must compile on Java 7 to see the new methods but generate Java 1.6 byte codes to make the code run on Java 6. However, the vendor must be extremely careful not to pick up any methods or classes in Java 7 that are not available on Java 7. An awful illustration of how painful a type safe language becomes when you do it wrong.

In OSGi this would all be no problem^*. There would be a semantically versioned javax.sql package that contains the SQL API. Consumers (in general you) of this package would get a lot of backward compatibility and providers of this package (the database vendors) will have to provide new releases for each new API. Since in runtime multiple releases can coexist in the same VM, the choice for which VM will not unnecessarily constrain the choices for a database vendor. It is kind of odd that Oracle, a database vendor, makes such a mess in the API to their databases ...

The java(x).sql disarray is a fine illustration of how aggregation creates constrains between its constituents. It is at the heart of the OSGi package dependency model. The siren song of one huge library that contains everything one could ever need (and much more) should be resisted in lieu of modularity since over the long run, super aggregation creates more problems than it solves.

   Peter Kriens (@pkriens)

^* If it was not for the nasty detail that javax.sql is badly intertwined with java.sql and java.sql can only be loaded from the VM's boot classpath because it is a java package. Sigh.

Baselining, Semantic Versioning Made Easy

Versioning is one of those things where everybody has a general idea but few really understand it well, resulting in many different and sometimes bizarre practices. The semantic versioning movement put a more solid footing on the version syntax, creating a version Domain Specific Language (DSL) to signal backward compatibility. It uses a 3-part version, where the first part (MAJOR) signals the breaking changes, the second part (MINOR) signals backward compatible changes, and the third part (MICRO/PATCH) signal bug fixes not visible in the public API. For example, an artifact with version 1.2.3 has the same API as 1.2.4, will be backward compatible with 1.3.0, and will break with 2.0.0. By using semantic versioning you pledge that in the future you will use this DSL to signal backward compatibility so that tools can point out breakage or select compatible components. Semantic versions are a big step in software engineering.

So any decent software engineer will agree that semantic versioning is good; being able to watch Maven central close up, I can also see that it has become widely used over the past 2 years. That said, how much work is there for a developer to maintain these versions? Developers are rightly lazy people and versions can be quite error prone and are complicated to maintain without tool support. To minimize the work, I've used the OSGi semantic version rules extensively in bnd. If you compile against an API then you are bound to a range of versions. For example, if you compile against an API with version 1.2.3 then bnd will calculate the corresponding import range: [1.2.3,2). (Actually it is a bit more subtle, see the OSGi semantic version paper.)

Though bnd has the tools to maintain your semantic versions and therefore pledged how things would be updated, it never checked if those pledges were actually kept. If you forgot to change a version after a code change then all bets were off. Since humans are really bad at versions and developers rarely know all the compatibility rules there were many errors.

Meet baselining. 

When you enable baselining, bnd will baseline the new bundle against the the last released non-snapshot bundle, a.k.a. the baseline. That is, it compares the public exported API of the new bundle with the baseline  If there are any changes it will use the OSGi semantic version rules to calculate the minimum new version. If the new bundle has a lower version, a number of errors are generated. 

The first error is on the place where the version is defined. The other errors are on the API changes that caused the version change. Since bndtools runs bnd continuously you have the uncanny effect that adding a method to an interface suddenly generates errors in different places, pointing out that you are trying to make an incompatible change. Quick fixes are then available to bump the version or to remove the offending API change. Detecting errors earlier is the hallmark of Eclipse and is a great boon to productivity. We all know how much time it saves when you find these bugs while they are being made.

Baselining teaches the actual developers a lot about backward compatibility. After enabling baselining on bnd this weekend I was actually shocked to find that some of the (expected to be) tiny changes I had made in the last three weeks since we froze 2.2 were not as compatible as I thought. (This is another way of saying I had not bumped the appropriate versions.) They were not just bug fixes but actually had API repercussions I had not foreseen, humbling.

Peter Kriens @pkriens

The Magic of Modularity

Anybody that has done some computer classes over the last 30 years has learned about modularity and should know it is good. However, to describe why it is good is always a bit harder. Maybe it is not that hard to understand the benefits of encapsulation because we all have been in situations where we could not change something because it was exposed. However, for me the magic actually appears during design, when you pick the modules and decide about their responsibilities. This is reflected in the seminal paper of David Parnas [1971] called "On the criteria to be used in decomposing systems into modules".

Last week I was designing a function for bnd and there I ran into an example that illustrated very nicely why the decomposition is so important. The problem was the communication to the remote repositories. Obviously, one uses a URL for this purpose since it is extremely well supported in Java, it supports many protocols, and in OSGi it can be easily extended by registering a Stream Handler service. However, security and other details often require some pre-processing of the connection request. For example, basic Http authentication requires a request header with the encoded user id and password. Anybody that has ever touched security standards knows this is a vast area that cannot be covered out of the box, it requires some plugin model. This was the reason we already had a very convenient URLConnector interface in bnd that could translate a URL into an Input Stream:

   public interface URLConnector {
      InputStream connect( URL url) throws Exception;
   }

Even more convenient, there were already several implementations, one that disabled Https certificate verification and one for basic authentication. Always so nice when you find you can reuse something.

However, after starting to use this abstraction I found that I was repeating a lot of code in different URL Connector implementations. I first solved this problem with a base class, but then it required extra parameters to select which of the options should be used. And the basic design did not support output (you know you can even send a mail with just a URL?). So after some struggling I decided to change the design and leverage the URLConnection class instead. Though the common use for a URL is to call openStream(), you can actually first get a URLConnection, parameterize it, and the actually open the connection. So instead of a URLConnector interface I devised a URLConnectionHandler interface. This interface had a single method:

   public interface URLConnectionHandler {
      void handle( URLConnection connection) throws Exception;
   }

Since this interface now specifies a transformation it can be called multiple times, unlike the URLConnector interface. This enabled me to write a number of tiny adapters that only did one thing and were therefore much simpler and actually more powerful. The user can now specify a number of  URLConnectionHandler for a matching URL. For example, Basic Authentication should in general not be used without HTTPS since it shows the user id and password in clear text. Instead of building this verification in the Basic Authentication plugin it can now just be selected by the user so that for another URL it can be used with a different combination. 

After porting the existing functionality of the URLConnector implementations I ended up with significantly less code and much more power., only because the structure was different. That is what I call the magic of modularity.

Peter Kriens @pkriens

P.S. Registered for the OSGi Community Event in Ludwigsburg? I will be giving a talk about Developing Web Apps with OSGi. For Germans, there is also an OSGi bootcamp from the OSGi User's Forum Germany. Advance registration ends Oct 1.

OSGi's Popularity in Numbers

Just some interesting statistics that I found out by scanning Maven Central. I've got all the metadata in a Mongo database so it is easy to analyze. The current database consists of over 426 thousand JARs organized in more than 46 thousand projects. I have been scanning Maven central since last year and these numbers seems to have almost doubled, which is a scary though if this continues at that exponential rate (especially for Sonatype who seems to pay the bandwidth and storage costs of Maven Central).

Almost 10% of the 46000 projects in Maven Central today are OSGi bundles. The most surprising part for me was that the the official OSGi Core JAR actually comes in at #36. For the official OSGi Core JAR, there are more than 24 thousand transitively inbound projects. It is more popular than dom4j (#43) or Apache commons collections (#45).

So what does this ranking number mean? I uses an algorithm similar to Google Pagerank, a project is more important when it has more inbound maven dependencies (compile and runtime scope) based on the latest revision of a project. A staggering more than half of the 46 thousand projects have a transient dependency on the OSGi JAR. Staggering because it says probably more about the infectious nature of the maven dependency model than OSGi's popularity.

There are over 50 projects that contain the package org.osgi.framework. In overall ranking Eclipse Equinox at #78 and Apache Felix at #216. That said, Apache Felix also provides a compile only JAR with the OSGi packages that comes in at #100. When looking at this list it turns out that the OSGi Jars appear in several incarnations. I even found the release 3 JARs: 300k for core and compendium combined. The compendium, a JAR that is only used by projects that are really OSGi, comes in at #50.

The numbers look very good for OSGi and I think it indicates that we will see more and more projects providing OSGi metadata. As David Bosschaert wrote earlier this year, if you need help adding this metadata then let us know.

So which project is #1? I know you've been dying to ask. Well, the top 5 is:

1. org.hamcrest : hamcrest-core
2. junit : junit
3. javax.activation : activation
4. javax.mail : mail
5. org.apache.geronimo.genesis.config : logging-config

Peter Kriens @pkriens