OSGi Bundle Repository Indexer Open Sourced

The OSGi Bundle Repository (OBR) is one of the most popular pages on the OSGi developer web site (www2.osgi.org), it receives almost as many hits as its main page. Surprising, because OBR is hardly advertised.

We developed OBR last year to provide a single place where OSGi developers can find bundles for their projects. One of the key drivers of the OSGi Alliance is to provide a universal middleware platform, reuse is therefore crucial. To reuse, one has to find appropriate components. OBR was developed to address this need.

How does it work technically? Well, the current OBR was derived from Richard Hall's Oscar Bundle Repository, and it is based on the same idea of a federated repository. A repository is a web server that allows access to the meta data and location of bundles. This information can be used to install bundles on a framework directly. For example, Richard developed a shell extension for Oscar/Felix that downloads a bundle from OBR, including its dependencies. Last year, I developed a web interface that allows for browsing the repository interactively.

At the bottom of OBR you find the repository.xml file. This file contains the meta data of the bundles and can link to other repositories as well. OBR is therefore a federated repository. This was an important aspect because we needed to make it as easy as possible for people to provide their own bundles to the repository. In the OBR model, any company or person can maintain its own repository and link it to others. This also enables repositories to be partly private on an Intranet while it still provides access to external repositories. In this model, the point where OBR is accessed can provide a trust model.

The information you find in the repository.xml is equivalent but not identical, to the information in the bundle. The reason it is not identical is that it has become more clear to us in the last few years that the OSGi technology needs a more generic dependency model. Over time, we have added more and more dependencies like Import-Package, Require-Bundle, Bundle-RequiredExecutionEnvironment, Bundle-NativeCode, etc. These dependencies had their own header and their own intricacies. Though this has some advantages, I am afraid that over time it will become too complex. For this reason, we used a generic model for OBR consisting of Capabilities and Requirements. This model resembles JSR 124 but we made it more powerful by making requirements OSGi filters instead of simple properties. Filter based requirements make it possible to create complex expressions and check magnitudes. For example, a requirement can be that there is a screen with a size of more than 100x100 pixels. JSR 124 can only handle exact matches.

An OSGi Environment, at a certain moment in time, has a set of capabilities. Each bundle (or hardware) requires a set of capabilities but can also provide a set of new capabilities when installed (and its requirements are met). This is a wonderfully elegant model. For example, if you have a mobile phone and insert a headset, the phone will be extended with a headset capability. This enables the installation of a bundle that requires a headset, which in its turn provides new capabilities. This model worked so well, that Richard Hall changed his framework resolver in Apache Felix (the program that wires bundles before they can be started) to work in this generic mode. Though we found a few nasty cases related to constraints that are currently not modeled, the idea performed very well.

This model is described in a document available from our website. Please note that this document does not mean that the OSGi Alliance will develop this concept further, we are testing the waters only!

Maintaining the repository.xml by hand is not an option. Eclipse alone provides over 1100 bundles. Fortunately, the OSGi specifications define lots of manifest headers that provide meta data. We therefore developed a program that reads a set of bundles and creates the repository.xml. Last month we got permission of the OSGi Alliance Board to open source this program. This makes it possible for organizations to include a bundle indexer. The sources are hosted on a Subversion server at http://www2.osgi.org/svn/public/trunk/org.osgi.impl.bundle.bindex/.

So how can we make OBR a success. Well, only if you help us do we have a chance. In my work I see so many companies that develop lots of bundles that are not strategic for their company. If we want to make the idea of a universal middleware work, we must share these non-strategic bundles with each others so that we do not get an umpteenth version of a serial port bundle.

If you have those bundles that you want to share, then setup your own repository. It is quite easy. Absolutely crucial is the meta data in the bundles. It is sad, but there are way to many bundles (and JARs) out there that have no name, no description, no license information, no nothing. It takes 5 minutes to add this information to a manifest and it will make bundles more reusable. So you have no excuse to skip this! Then if you have documented bundles, you just create a directory on your company's webserver and save the bundles there. Then once a day you run bindex on this directory. Bindex creates the repository.xml. The only thing left to do is the URL to me. I will link it into the main OSGi Bundle Repository. So, do not ask what bundle you can get from OBR, ask what you can mean for OBR!

Looking forward to seeing all your repositories,

Peter Kriens

Deja Vu

Obviously I am closely following the 277 list, though with pain in my heart that Sun does not allow me to post. So I can only use this blog to try to influence the discussions. If you follow the JSR 277 Modularization mailing list, you should have noticed the struggle that is taking place with initialization. The early draft specified a module initializer what would help link the module to other modules. However, the EG now also finds there is a need to initialize the user part of a module, a.k.a. the Bundle Activator. In this blog, let me take you back to a place long ago (well, only about 9 years).

Once upon a time there was a brilliant product from Sun called Java Embedded Server devised by Anselm Baird. JES was the predecessor of the OSGi specifications. Anselm got a lot of things right, but not everything. One of the things he had got wrong were the wizards. There was a DependenciesWizard, an InstallWizard, an UninstallWizard, etc. These wizards were identified by a class names and loaded from the bundle at the appropriate time, and given control. This is similar to the proposal in JSR 277 Modularization to let the module help in linking its dependencies, I recall this was the DependenciesWizard (Unfortunately, there seem to be no traces of JES left on the Internet, I only found some old sources from 1999 where I played with JES).

Going back through the mail discussions of that time, I was surprised how little I had learned in ten year because my opposition to these wizards was fierce. The primary reason was that these wizards were running in limbo. This is the place between heaven and hell where you go when things are a tad fuzzy around your soul. Classloader limbo is when any class you use can blow up because it depends on some other module not yet initialized. It is a bit like walking in the dark in an unfamiliar room, blindfolded, and your hands tied behind your back.

In January 1999, we had a meeting in Cupertino. Coming back to Ericsson I could report the following to Ericsson management:

The simplification of the wizards was quite successful: They were all removed. This was already proposed on the mail. We also decided to simplify the dependencies. Some elements of ServiceSpace required knowledge about the dependencies. This requirement made certain aspects of ServiceSpace rather inflexible. After these were removed a simple model remained where only one class reflects the Bundle for ServiceSpace, and this class is named in the manifest. In my opinion, these changes made the implementation of a ServiceSpace significantly simpler.

Even Anselm, after a lot of bashing from the other participants became convinced that a declarative approach was better. He proposed some of the API changes that became part of the OSGi:

Here's an API proposal to simplify the way wizards work in ServiceSpace. The essential motivation for that change is to move wizards from programmatic objects into declarative objects, as much as possible (the ultimate plan being to be able to provide tools that work on bundles).

My original drive for removing the wizards was to not have code run in an undefined environment. Looking back at Anselm's mail I see that he actually foresaw how important tools would become. Such tools can only work when the dependencies are declared. The problem to know what a wizard will do when you call it is an insoluable problem. So we succeeded to convince Sun in 1999, if we could only do the same in 2007 ...

Peter Kriens

Best Practice with Depressed Bundles

After an active life, some bundles feel that it has been enough and just want it to end. They carefully get their Bundle object and slowly calls uninstall... Can this be done in an OSGi Framework?

The issue is that the stop() method returns to a method of a bundle that has just been stopped. This means that its Bundle Context is no longer valid, its services have been unregistered and its Class Loader released. Any attempt to use any of the OSGi mechanisms will therefore result in an exception. For this reason, stopping oneself has always been frowned upon in the OSGi inner circles. However, being able to end itself is useful sometimes. We briefly touched this subject in R4 but did not take the proper time so we deferred it to the next release. Well, now is the time.

The proposed solution was to allow a bundle call an asynchronous stop/uninstall/update method. However after a heavy and sometimes violent discussion we basically concluded that stopping oneself or stopping another bundle both have exactly the same issue. If a bundle is stopped by another bundle, there is always a possibility that the stopped bundle is not stopping correctly and continue to execute after being stopped. From a class loading point of view, this can not do much harm, class loaders are only garbage collected when they are no longer used. As long as a class is on the run stack of some thread, it can not be garbage collected. It is only when OSGi mechanisms are used that additional problems occur. Any programmer that stops its bundles, should (obviously) not continue to execute code because its stop method has been called, that is the contract.

We therefore decided to not add an additional mechanism and leave the spec as is.

However, what is the best practice in this area? The best practice is to execute the stop/update/uninstall method in a separate thread:

void stopSafe(final Bundle bundle ) {
new Thread() {
  public void run() {
    bundle.stop();
  }
}.start();
}

Or do you have other ideas?

Peter Kriens

P.S. I have added a snippet as well that shows how it all fits together

Can Someone Tell Sun About OSGi?

Someone just sent me a mail linking JSR 316. A JSR that will specify Java Enterprise Edition 6, the successor of Java EE 5, which was defined in JSR 244.

Now before we take a look, lets just investigate some recent trends in the Enterprise computing world. Hmm, BEA moved their micro Service Architecture on top of OSGi, IBM Websphere 6.1 seems to have chosen OSGi, Jonas is an EE framework build on OSGi from day one, and the JBoss Microcontainer is modified to support OSGi. On top of that, we have one product that made many people re-think Java EE: Spring. Now this product fell in love with OSGi. The market clearly says one size does not fit all. One would expect that these trends have some influence on Java EE 6? Sigh, think again.

To address the problem that one size does not fit all, Sun proposes to create a few more sizes, called profiles. Surely that should fit all? Well, profiles have been tried in J2ME (for Sun people working in JEE, this is the other leg of the JCP standards department focused on constraining environments) and they failed in my opinion. Profiles require specifications and specifications are expensive to produce. Therefore, there will always be a push to minimize the number of profiles. However, in real life, no profile will be perfect for a specific application because the demands differ so much. So profiles cost a lot but still make most people unhappy because it is likely not a perfect fit.

There is a very simple solution to the problem of a limited set of profiles: componentization. Instead of gluing a number of Java APIs together and calling it a profile, we need to allow the developers to exactly choose the components they need. Let the developer make their own profiles! Wow, isn't that a great concept?

What would we need for this? Well, we would need a component framework that allows the management of these components. Preferably dynamic so that we do not have to restart the server all the time while developing or running business critical applications. We would also need a model where the components could collaborate without requiring singletons with their nasty side effects. Does such a component framework exists? Yes, it is called OSGi/JSR 291. Does JSR 316 refer to OSGi/JSR 291?

Nope.

JSR 316 only mentions JSR 277 in the specification request, in a rather indirect way. They basically say that 277 is coming and they'll defer any decisions until it is there. Seems fair because 277 is older and more mature than 291 looking at their numbers? Well, except for the fact that 277 is still in draft review and 291 is already final because 291 could be based on the very mature specification that originated in 1999 and has gone through 4 major revisions. So there must be another reason why JSR 291 is not mentioned? Maybe 277 will provide features that JSR 291 does not have? Hmm, not really. Looking at the JSR 277 specification request one can not claim that the ambition level is high in comparison to JSR 291/OSGi: no dynamics, no class space consistency, no unloading, no package imports, etc. Their early draft was still on a rather simplistic level. The only real addition over 291 is the repository and there are many loose ends. JSR 277 recently opened up the mailing list, taking a look at the discussions it seems that they are still struggling with some of the basics of modularity. However, fortunately, the JSR 277 expert group has promised to make the 277 module system interoperate with JSR 291/OSGi. That makes choosing OSGi risk free as well as providing the additional benefit of running today on a wide range of VMs (from 1.2 to 6, also in embedded devices), unlike JSR 277 that will only run on Java 7 when it comes out late 2008. So why should JSR 316 wait? The perfect solution exists today and is promised by JSR 277 to be compatible with tomorrow?

In a context free society basing JSR 316 on OSGi/JSR 291 would be an absolute no-brainer. I can not think of one rational argument why JSR 316 would not choose OSGi today so we can get the advantage in Jave EE 6. Can you?

Peter Kriens

To Declare Or Not to Declare

A few weeks ago I described the “hibernate” problem, how to let multiple bundles independently use a Hibernate Session Factory without requiring a priori knowledge of each other. These postings are not just finding practical solutions to today’s problems with today’s specifications. These are my research subjects; I am trying to keep future generations of the OSGi specification clean and in the right spirit so I am playing with options. Next time I will try to be more pragmatic.

The solution I found to share the Session Factory was service based, it allowed bundles to contribute Hibernate domain classes, and it allowed bundles to use a collective of domain classes. However, in this solution there still lurked a class loading problem caused by a peculiar habit of Hibernate. In the solution this problem was solved with a simple Require-Bundle. However, though this solution works, it is a hack that can easily fall apart if the required classes are refactored into another bundle. So for today, this solution is good enough, for tomorrow we need to understand the root problems and come up with solutions to this root problem. So lets dive deeper into the problem and analyze what is going on.

When Hibernate creates a Session Factory, it actually creates a dynamic class proxy to its implementation class. This in itself is not a problem was it not for the fact that it uses the client’s class loader to create the proxy. The client therefore must have visibility to classes that it does not use. That is, Hibernate assumes all the classes itself sees are visible to the client. In a modular system, this is obviously not the case. A simple fix is to let the client bundle use Require-Bundle for the Hibernate bundle, however, this Require-Bundle has its own set of problems, see the OSGi R4.1 specification.

It is easy to take the high ground and tell Hibernate to get its act together. However, this is not a very productive stance for most of us. Despite the increasing popularity of OSGi Technology, the majorities of JARs out there are not prepared for proper modularity.

These type of class loading problem are endemic because so many developers have (ab)used class loaders for home grown extension mechanisms, due to a lack of a proper extension mechanisms in Java á la the OSGi service registry. All of these home grown mechanisms just fall apart when it becomes necessary to share the VM with multiple versions of the same package. They also can not provide class space consistency, which is important for applications that want to work reliably. Therefore, the best solution would be if the OSGi specifications addressed these issues directly.

The OSGi CPEG asked me to come up with an OSGi RFP about this problem so I Skyped my favorite OSGi Invited Researcher: Richard S. Hall. We did some brainstorming and in the past few weeks we did some prototyping to better understand the problems. Richard modified Apache Felix to support our experiments and I used these modifications with a simple web based notes taking application that uses Hibernate.

The first approach we took was the declarative “implicit-wire” directive. We came to this solution because the deep structure of the problem is that when you import a package, the consequence of this import is that you should import additional packages. That is, in the Hibernate example, when you import org.hibernate.cfg, you should also import net.sf.cglib.*, even if you have no direct dependency on these packages. Richard therefore created an x-implicitwire directive on an Export-Package clause. The value of this directive is a list of packages that should be added to the import of any bundle importing the package. For example:

Export-Package: org.hibernate.cfg;x-implicitwire:=”net.sf.cglib.proxy,net.
sf.cglib.core,net.sf.cglib.reflect, org.hibernate.proxy”

If Apache Felix has to wire to the org.hibernate.cfg package, it will now automatically add wires to the packages from the x-implicitwire directive as well, thereby ensuring that the importing bundle can see the proper packages when Hibernate attempts to create a proxy for the Session Factory.

The declarative approach is quite simple and it did solve the problem. The prototype code, a simple web based notes program, worked fine. The only change that was necessary was the bnd file that created the hibernate bundle. However, we were not convinced that this solution could solve world hunger; there are many cases where the declarative approach with x-implicitwire is not feasible because the set of required implicit wires is not known ahead of time. A crucial example is generic Aspect Oriented Programming, where the set of additional classes is completely open-ended. That is, an AspectJ program can weave any possible class in another class.
The only solution to this problem is allowing another bundle to add imports during runtime. After some discussions Richard found the courage, and time, to add an addRequire function to the Bundle Context object. Imports added this way are treated as DynamicImport-Package clauses. That is, they are consulted last and always until the import is found.

I created a simple extender bundle that inspects all bundles when they become resolved. I chose the import of the org.hibernate.cfg package as trigger. That is, when a bundle imported this package, it would automatically get the net.sf.cglib.* packages. As can be expected, also this approach worked fine and did not require anything special from the client bundle, no changes were necessary. However, unfortunately there is a race condition to which we do no have a good solution. If the extender bundle gets started after the client bundle, we have a problem. The resolved client bundle could get started and create a Session Factory before the extender has a change to add the imports. Alternatively, the additional imports could be added persistent, in that case the install event could trigger the adding of the imports. In that case, the window for the race condition is much smaller. This problem is prevented with the declarative approach of x-implicitwire.

So what is the best solution? I am really not sure. I like the simplicity of the declarative approach. It allows a clean and very simple solution for problems where the implicit imports are known ahead of time. A very important advantage of this approach is that it does not have race conditions. However, when these imports are dynamic, the problem is no longer usable. The procedural approach of dynamically adding an import in runtime has the required flexibility but has the problem of the race condition. It will require deployers to use the start level to ensure that the extender bundle is started before the client bundles are installed.

What do you think?

Peter Kriens