Can Someone Tell Sun About OSGi?

Someone just sent me a mail linking JSR 316. A JSR that will specify Java Enterprise Edition 6, the successor of Java EE 5, which was defined in JSR 244.

Now before we take a look, lets just investigate some recent trends in the Enterprise computing world. Hmm, BEA moved their micro Service Architecture on top of OSGi, IBM Websphere 6.1 seems to have chosen OSGi, Jonas is an EE framework build on OSGi from day one, and the JBoss Microcontainer is modified to support OSGi. On top of that, we have one product that made many people re-think Java EE: Spring. Now this product fell in love with OSGi. The market clearly says one size does not fit all. One would expect that these trends have some influence on Java EE 6? Sigh, think again.

To address the problem that one size does not fit all, Sun proposes to create a few more sizes, called profiles. Surely that should fit all? Well, profiles have been tried in J2ME (for Sun people working in JEE, this is the other leg of the JCP standards department focused on constraining environments) and they failed in my opinion. Profiles require specifications and specifications are expensive to produce. Therefore, there will always be a push to minimize the number of profiles. However, in real life, no profile will be perfect for a specific application because the demands differ so much. So profiles cost a lot but still make most people unhappy because it is likely not a perfect fit.

There is a very simple solution to the problem of a limited set of profiles: componentization. Instead of gluing a number of Java APIs together and calling it a profile, we need to allow the developers to exactly choose the components they need. Let the developer make their own profiles! Wow, isn't that a great concept?

What would we need for this? Well, we would need a component framework that allows the management of these components. Preferably dynamic so that we do not have to restart the server all the time while developing or running business critical applications. We would also need a model where the components could collaborate without requiring singletons with their nasty side effects. Does such a component framework exists? Yes, it is called OSGi/JSR 291. Does JSR 316 refer to OSGi/JSR 291?

Nope.

JSR 316 only mentions JSR 277 in the specification request, in a rather indirect way. They basically say that 277 is coming and they'll defer any decisions until it is there. Seems fair because 277 is older and more mature than 291 looking at their numbers? Well, except for the fact that 277 is still in draft review and 291 is already final because 291 could be based on the very mature specification that originated in 1999 and has gone through 4 major revisions. So there must be another reason why JSR 291 is not mentioned? Maybe 277 will provide features that JSR 291 does not have? Hmm, not really. Looking at the JSR 277 specification request one can not claim that the ambition level is high in comparison to JSR 291/OSGi: no dynamics, no class space consistency, no unloading, no package imports, etc. Their early draft was still on a rather simplistic level. The only real addition over 291 is the repository and there are many loose ends. JSR 277 recently opened up the mailing list, taking a look at the discussions it seems that they are still struggling with some of the basics of modularity. However, fortunately, the JSR 277 expert group has promised to make the 277 module system interoperate with JSR 291/OSGi. That makes choosing OSGi risk free as well as providing the additional benefit of running today on a wide range of VMs (from 1.2 to 6, also in embedded devices), unlike JSR 277 that will only run on Java 7 when it comes out late 2008. So why should JSR 316 wait? The perfect solution exists today and is promised by JSR 277 to be compatible with tomorrow?

In a context free society basing JSR 316 on OSGi/JSR 291 would be an absolute no-brainer. I can not think of one rational argument why JSR 316 would not choose OSGi today so we can get the advantage in Jave EE 6. Can you?

Peter Kriens

To Declare Or Not to Declare

A few weeks ago I described the “hibernate” problem, how to let multiple bundles independently use a Hibernate Session Factory without requiring a priori knowledge of each other. These postings are not just finding practical solutions to today’s problems with today’s specifications. These are my research subjects; I am trying to keep future generations of the OSGi specification clean and in the right spirit so I am playing with options. Next time I will try to be more pragmatic.

The solution I found to share the Session Factory was service based, it allowed bundles to contribute Hibernate domain classes, and it allowed bundles to use a collective of domain classes. However, in this solution there still lurked a class loading problem caused by a peculiar habit of Hibernate. In the solution this problem was solved with a simple Require-Bundle. However, though this solution works, it is a hack that can easily fall apart if the required classes are refactored into another bundle. So for today, this solution is good enough, for tomorrow we need to understand the root problems and come up with solutions to this root problem. So lets dive deeper into the problem and analyze what is going on.

When Hibernate creates a Session Factory, it actually creates a dynamic class proxy to its implementation class. This in itself is not a problem was it not for the fact that it uses the client’s class loader to create the proxy. The client therefore must have visibility to classes that it does not use. That is, Hibernate assumes all the classes itself sees are visible to the client. In a modular system, this is obviously not the case. A simple fix is to let the client bundle use Require-Bundle for the Hibernate bundle, however, this Require-Bundle has its own set of problems, see the OSGi R4.1 specification.

It is easy to take the high ground and tell Hibernate to get its act together. However, this is not a very productive stance for most of us. Despite the increasing popularity of OSGi Technology, the majorities of JARs out there are not prepared for proper modularity.

These type of class loading problem are endemic because so many developers have (ab)used class loaders for home grown extension mechanisms, due to a lack of a proper extension mechanisms in Java á la the OSGi service registry. All of these home grown mechanisms just fall apart when it becomes necessary to share the VM with multiple versions of the same package. They also can not provide class space consistency, which is important for applications that want to work reliably. Therefore, the best solution would be if the OSGi specifications addressed these issues directly.

The OSGi CPEG asked me to come up with an OSGi RFP about this problem so I Skyped my favorite OSGi Invited Researcher: Richard S. Hall. We did some brainstorming and in the past few weeks we did some prototyping to better understand the problems. Richard modified Apache Felix to support our experiments and I used these modifications with a simple web based notes taking application that uses Hibernate.

The first approach we took was the declarative “implicit-wire” directive. We came to this solution because the deep structure of the problem is that when you import a package, the consequence of this import is that you should import additional packages. That is, in the Hibernate example, when you import org.hibernate.cfg, you should also import net.sf.cglib.*, even if you have no direct dependency on these packages. Richard therefore created an x-implicitwire directive on an Export-Package clause. The value of this directive is a list of packages that should be added to the import of any bundle importing the package. For example:

Export-Package: org.hibernate.cfg;x-implicitwire:=”net.sf.cglib.proxy,net.
sf.cglib.core,net.sf.cglib.reflect, org.hibernate.proxy”

If Apache Felix has to wire to the org.hibernate.cfg package, it will now automatically add wires to the packages from the x-implicitwire directive as well, thereby ensuring that the importing bundle can see the proper packages when Hibernate attempts to create a proxy for the Session Factory.

The declarative approach is quite simple and it did solve the problem. The prototype code, a simple web based notes program, worked fine. The only change that was necessary was the bnd file that created the hibernate bundle. However, we were not convinced that this solution could solve world hunger; there are many cases where the declarative approach with x-implicitwire is not feasible because the set of required implicit wires is not known ahead of time. A crucial example is generic Aspect Oriented Programming, where the set of additional classes is completely open-ended. That is, an AspectJ program can weave any possible class in another class.
The only solution to this problem is allowing another bundle to add imports during runtime. After some discussions Richard found the courage, and time, to add an addRequire function to the Bundle Context object. Imports added this way are treated as DynamicImport-Package clauses. That is, they are consulted last and always until the import is found.

I created a simple extender bundle that inspects all bundles when they become resolved. I chose the import of the org.hibernate.cfg package as trigger. That is, when a bundle imported this package, it would automatically get the net.sf.cglib.* packages. As can be expected, also this approach worked fine and did not require anything special from the client bundle, no changes were necessary. However, unfortunately there is a race condition to which we do no have a good solution. If the extender bundle gets started after the client bundle, we have a problem. The resolved client bundle could get started and create a Session Factory before the extender has a change to add the imports. Alternatively, the additional imports could be added persistent, in that case the install event could trigger the adding of the imports. In that case, the window for the race condition is much smaller. This problem is prevented with the declarative approach of x-implicitwire.

So what is the best solution? I am really not sure. I like the simplicity of the declarative approach. It allows a clean and very simple solution for problems where the implicit imports are known ahead of time. A very important advantage of this approach is that it does not have race conditions. However, when these imports are dynamic, the problem is no longer usable. The procedural approach of dynamically adding an import in runtime has the required flexibility but has the problem of the race condition. It will require deployers to use the start level to ensure that the extender bundle is started before the client bundles are installed.

What do you think?

Peter Kriens

Call To Arms: OSGi Community Event

The OSGi Alliance has a community event in Munich on June 26th - 27th, 2007 that you must attend. The program is turning out to be quite interesting! Many of the key players in the OSGi eco system will be around. This is clearly the place to be if you work with OSGi Technology.

Obviously we will have a lot of attention for Enterprise software. Manfred Hutt, (Siemens), and Nicole Wengatz (Siemens) will demonstrate and talk about their use of OSGi in creating large Enterprise applications for business phone exchanges. BEA will show their microService Architecture based on OSGi and how it benefits their customers. Gunnar Wagenknecht (Truition), and Jochen Hiller (Business Operation Systems) will talk about server side Eclipse. Pierre Dubois (Requea) will talk about how to modularize Web Applications using OSGi Technology. The chief architect of the DB4Objects, Carl Rosenberger, will discuss OSGi Technology and persistence. A crucial subject for Enterprise applications.

One of the most exciting OSGi based developments is JSR-232 and the adoption by Sprint. Brandon Annan, their Lead Chief Java Architect, will explain what the plans are with OSGi. For any OSGi developer, these plans are very interesting, even if you never thought about programming for mobiles. The mobile phone is finally becoming a viable application platform for more than games. Just think what you could do if you could easily integrate mobile phone based solutions in your applications?

In the home automation markets we will have an experience reports from Miguel García Longarón (Telefónica). He shows real applications being used in the field. Andre Bottaro, Software Engineer, France Telecom, Sylvain Marie, Software Engineer, Schneider Electric will show how UPnP+OSGi can be used to finally make the home plug and play dream come true.

Abdallah Bushnaq from Ricoh will give an overview of how OSGi is used in printers. Interestingly, there are several printer manufacturers that have OSGi inside, for example Canon has an OSGi runtime that can be used to download print applications. I am curious how Ricoh will use this technology.

Industrial automation is one of my favorite application areas for OSGi. I think it is an almost perfect fit, especially because this market is less cost sensitive than consumer products. Daniel Grossmann, Research Assistant, Information Technology in Mechanical Engineering, will give a very interesting presentation of how OSGi technology can be used in the automation of very large plants like refineries. Roman Roelofsen (IONA) will discuss how Service Oriented Archictures also fit embedded systems, and OSGi Technology makes this straightforward.

Cross industry, there is the need for remote management. Joachim Ritter, ProSyst Software, will give a presentation how you can use OSGi Technology to manage large number of runtimes. A key component of remote management is the software repository. It is nice to have a framework that resolves bundles, but where do those bundles come from? Richard S. Hall will give a presentation about the OSGi Bundle Repository under the intriguing title: "The Bundle Dilemma."

And of course we have lots of sessions for the people that are busy with the technology itself. First we will give an overview of the state of the OSGi specifications. The chairs of the expert groups will present their progress: BJ Hargrave (IBM), Kai Hackbarth (Prosyst), and Eric Newcomer (IONA). There is lots of progress in the Enterprise group and the Requirements group at the moment. If you plan to use OSGi technology in the future, you better be there.

We have some very interesting lightning talks for OSGi developers. BJ Hargrave will elucidate the changes for v4.1. Maven, the Apache build tool, has been extended with OSGi support. Stuart McCulloch (Jayway Malaysia), will show how these tools can be used to create OSGi bundles. Ales Justin (JBoss) will tell about the work that JBoss is doing adapting their microkernel to support OSGi.

Services, extensions? Neil Bartlett, famous from the popular beginner tutorials. will give a talk comparing the Eclipse Extension Registry (which is a service!) with the OSGi Service Registry. These two concepts are quite close and still wildly different. Puzzled? Neil will make it crystal clear. And if you have decided for services, then you should not miss the talk of Clément Escoffier about iPOJO. He worked with Richard Hall to develop a next generation component model for OSGi. This project can be seen as the successor of Service Binder that was the basis for OSGi Declarative Services. Obviously, if you program for OSGi Technology you should not miss the OSGi best practices talk that BJ Hargrave and I are giving!

The day after the community event, Thursday the 28th, we will have EG meetings. The Enterprise Expert Group meeting will be for members only but there will also be a Remote Management and Automotive requirements gathering meeting presided by Kai Hackbarth. These two meetings are also open for non-members.

Really, if you are involved with OSGi technology you have very little excuse to not show up. There will be plenty of time to talk to the speakers and other participants. I really hope I will meet you there! If you register before the 12th, you still get the early bird discount.

Peter Kriens

OSGi and Hibernate

A long time Dutch friend of mine, Petr van Blokland and me are working on a yet another web framework. I will not go into the reasons for YAWFW, except that his background is graphic design which means aesthetics play a large role in this framework. However, this focus does not exclude us from having to handle mundane details like web requests, models, views, and databases. I am participating because it is a good way to get some enterprise systems experience. Obviously the whole infrastructure is based around OSGi.

To get the picture, a short sketch of the architecture. Incoming requests are dispatched to a servlet by the HOST header to a service. This makes it possible to handle multiple sites in the same OSGi framework. We call this servlet the site servlet.The servlet we dispatch to can be anything, but so far we are using a hand made servlet that uses Groovy classes as pages. The first part of the path is the class name, the second part is the method name and the remainder are parameters. That is, a request like http://www.acme.com/home/index is a call to the class index() method in the home class.

The Groovy class is a normal class that does not have to extend or implement anything. When we create it, we assign it a special meta class which we call the Builder. It is a bit different from a Groovy builder, but tries to achieve the same goal: merge code and html. The Builder is setup by the site servlet. Each site can have its own specialized builder. The builder can provide methods to the page class, this makes it easy to provide high level tags to the page. For example, you can make your own tag for a navigation bar.

Builders in Groovy are a marvelous concept, they merge the power of a full blown language with the power of declarative programming à la HTML. The achieve this goals using closures. Closures are blocks of code that have access to their surrounding context but that can be manipulated as objects.

def x = { print "hello $it" }
x()  // Call the block

As a Smalltalker, the absence of closures in Java is one of the most painful experiences in my professional life. Anyway, closures allow you to parametrize functions. In Groovy, closures allow you to write code like:

def index() {
  def title = "Hello World"
  html {
      head {
         title { title }
      }
      body {
         h1 { title }
      }
  }
}

This is actually quite an interesting approach but it is not the purpose of this blog. We also needed a database for our web framework ... Groovy provides some very interesting SQL capabilities and I was really tempted. However, Groovy uses some weird trick to recompile the source code in runtime to prevent one from having to write SQL statements (see findAll). A lofty goal, but in this case not worth the expense. The code generated weird errors when the source code was not available in run time. So the only alternative seemed to be Hibernate.

Hibernate is an Object Relational Mapper (ORM). It allows you to write simple Java objects that are persisted in a database. The objects are free of any markup, required super classes or implemented interfaces. All the mapping information is detailed in xml files that are adjacent to the class files. That is, when hibernate must map a class to a relational database, it uses the class name to create a resource path and loads that resource path as its mapping file. However, such a mapping file must be read before the database connection is created.

Hibernate manipulates the classpath, and programs like that usually do not work well together with OSGi based systems. The reason is that in many systems the class visibility between modules is more or less unrestricted. In OSGi frameworks, the classpath is well defined and restricted. This gives us a lot of good features but it also gives us pain when we want to use a library that has aspirations to become a classloader when it grows up.

We needed a database so in the past few weeks I created a bundle that mediates between bundles with domain objects, connections to the database, and bundles that want to use a database. It was actually quite tricky to get it all to work and I am not sure I found the best solution. Lets take a look what I did and let me know what you think.



To work with Hibernate, you need a Session object. You can get a Session object from a SessionFactory. To get the the SessionFactory, you need to create it with a Configuration object. The Configuration object is created from a configuration XML file. By default, Hibernate loads this from the root of your JAR file, however, you can add classes manually to the configuration if so desired.

This gave us a problem. We had several bundles that needed to use the database but could not a priori decide which bundles would be available at any given moment. They could all have used their own Hibernate Session Factory but that would significantly complicate the configuration management and it would cost performance.

This looked like a clear extender problem. Bundles should be able to declare the classes they contribute to a Hibernate session.

For this model I architected the following manifest header:

Hibernate-Contribution ::=  default; \
          classes="xierpa.impl.pw.User,xierpa.impl.pw.Role"

The name field in the header (default) is the name of the database contribution. The classes attribute contains a comma separated list of classes. These classes are loaded from the bundle and added to the configuration that uses this contribution. Simple. Any bundle can now provide domain classes to Hibernate by just declaring this header.

However, where does the Session Factory Configuration come from? I decided to use Configuration Admin for this purpose. For each configuration, one can create a factory configuration. The properties for this configuration describe the connection parameters. The configuration was so large, that I had to extend FileInstall to not only install bundles but also handle configuration files. The normal interaction with the command line was too basic.

So, the Hibernate extender receives the configurations and tracks started bundles. With this information it matches up the contributions from any bundles to configurations for Session Factories.

The bundle that wants to get a Hibernate SessionFactory object just gets the HibernateDomain service and requests a session any time it needs a database transaction. This session comes from a factory that is automatically refreshed whenever one of the contribution changes or the configuration parameters changes.

Overall this model turns out to work very well. It is easy for bundles to provide contributions and it is easy for bundles to use a fresh Hibernate Session, without having to track the configuration and contributions.

I of course also had to run into one nasty class loader problem. Hibernate created a proxy using classes from the Hibernate bundle. Unfortunately, it asked the class loader of the domain objects for this Hibernate specific class. "Oh, what tangled webs we weave when first practice to classload ..."

Obviously I always use bnd and bnd only inserts Import-Package statements for the packages that the code really uses. One of the great advantages of Hibernate is that it allows you to use objects not coupled to Hibernate at all. Obviously, bnd can therefore not insert any references.

The short term solution to this problem was Require-Bundle, which is of course not a good solution as readers of this blog can testify. It solves the problem in this case but it creates many other problems in the long run. In the next release of the OSGi specification we must find a solution to this problem. It must be possible to create uncluttered bundles that only import what they need but can still load classes that other bundles make them require. This problem has already been raised in CPEG and it has high attention because the same patterns appears for other libraries as well.

I am very interested in how other people solve the problem of using libraries like Hibernate in an OSGi Service Platform. Please provide feedback.

Peter Kriens

Where Will JSR 277 End Up?

Sun has opened up the mailing lists of JSR 277 for observers. As I am denied membership, I now can at least have the privilege of hearing the conversation in this group. And interesting it is. Last week there was a discussion about importing classes, modules, or packages. As I argued vehemently (too) many times, importing packages is better than importing modules because it decreases coupling to a minimum. Some JSR 277 members seem to feel similar and are pushing for a similar solution. However, I think Glyn Normington gave the right answer:

Of course "import package" is superior to "require bundle" in many situations, but I think it would be a vast waste of time for JSR 277 to play "catch-up" with JSR 291.

A better approach would be for the Java 7 platform to provide first class support for JSR 291. This boils down to standardising the experimental class loader deadlock fix ([1]) and enabling JSR 291 to exploit JSR 277's repositories and JSR 294's superpackages.

I like this idea! JSR 277's unique aspect is the repository. Repositories have a clear value and Java does not have one at the moment. Instead of trying to invent a vastly inferior modularity layer that will take years to mature, it would be so much better to use the existing OSGi technology standardized in JSR 291 instead. Modularity from 291, repository from 277, and a new keyword from 294, I would almost start looking forward to Java 7!

Which brings me to another point. Why actually needs a Java 7? I have just started using Java 5 in my daily work, though most of my customers force me to use 1.4 bytecodes. This week I installed Java 6 but I have not found a single item that interests me so far. I understand that in a non-modular system one has to rev the whole system, even if the updates to the parts of the system are minor. Major new VM releases are painful and the steady release of new Java environments are increasing the distance between J2ME and J2SE with every release. Making it harder and harder to run on all platforms.

The industry needs modular Java technology so that we can go back to achieving the goal of "write once, run anywhere". If Java 7 can bring us closer to that goal, I guess it is worth it. If not, then why bother?

Peter Kriens

JSR 291 (OSGi 4.1) Is Accepted by the Executive Committee

As of yesterday May 21, 2007, JSR 291 has been approved. The Executive Committee has approved the OSGi R4.1 specification! I'd like to congratulate Glyn Normington for running this JSR in an exemplary way. It is really good to see that the JCP, in the end, can accept outside technology and bring it under the Java technology umbrella.

Unfortunately, not all EC members voted in favor. Only Sun and Hani Suleiman voted against for an apparent allergy to rubber (stamping). Let us take a look at their comments:

On 2007-05-15 Sun Microsystems, Inc. voted No with the following comment:
Sun's vote is not a negative assessment of the OSGi specification or technology, but rather on the purpose of this JSR and expert group when the specification exists and is being actively developed elsewhere.

The JSR 291 specification is simply a reference to the OSGi Core spec version r4. No other specification work has been created in the context of the JCP JSR 291 expert group. Rather the work to update the r4 spec occurred in the OSGi working group.

This manner of working contradicts the goals of the JCP to create expert groups empowered to evolve Java technology with the freedom to make design choices and technical decisions based on the needs of the Java Community. Sun has consistently expressed these concerns, as have other community members, throughout the lifetime of the JSR as our previous voting comments show.

Since these concerns been not addressed by the spec lead, either at the inception ballot or at the public review ballot, we are reluctantly voting no.

I can now rant about that fact that the OSGi Alliance updated the specification to version 4.1 based upon the JSR 291 EG requirements, and that these rubber stamping issues have been addressed by the specification lead ad nauseum on the mailing list, without receiving any reply from Sun. However, that is completely besides the point. The crucial question that arises from their comments is: Why on earth should rubber stamping be wrong?

To me, the integration of an external, mature standard in the Java world sounds like a perfect time & money saver. And I do not mean the hours saved by the JSR 291 Expert Group by having a solid document and existing implementations. Those hours are minor. I mean the hours saved by the end users because they do not have to waste endless hours trying to use immature specifications that not infrequently have to be overhauled. Any serious Java user can surely recall several JSRs where version 1 and 2 were not really that usable in practice. J2EE comes to mind? Obviously, Java users can only gain when the JCP adopts mature technology. Several key Executive Committee members voiced this sentiment in their comments as well.

Sun's statement shows a troubling disconnect between them and the interest of the Java community. We, as a community, need standards to minimize our work and maximize the quality of our products. We need standards so that for a given problem there is a shared industry wide API so we can be assured that our investments in these APIs are secure. We need standards so we can choose between different implementations, which will reduce the price and increase quality. We do not need standards to keep Expert Groups on remedial therapy. The advantage of a adopting a pre-existing standard is so blatantly obvious that one can only wonder what Sun's motives are. They are clearly not in the interest of our industry.

This disconnect is bad for our industry, and it is obviously upsetting for the OSGi members, adopters, and myself. Our attitude to standards has always been very much to adopt instead of develop. The IO Connector Service, the URL Stream Handler Service, the XML Parsing Service, the OMA DM management interface, and even the whole security model have been designed to work seamlessly with Java and anything that comes from the JCP. For R3, there was a strong push to design a brand new messaging service but after ample considerations we developed the IO Connection service that relies on the javax.microedition.io API. It has been our clear philosophy to only add a standard that fulfilled an unfulfilled need. And, if we did, provide a high quality specification. The standards landscape is already polluted enough as it is. Nobody is waiting for new standards unless it provides significant new value.

To conclude: JSR 291 has been approved and that is good. Sun's comments are upsetting but fortunately not serious because the other EC members voted yes. However, down the pipeline we see the Modularity JSR 277 coming, which is almost a full subset of OSGi technology, and thus JSRs 291 and 232. Is this good for our industry or is this another case where Sun is confused about the needs of our industry?

Peter Kriens

OSGi @ JavaOne so far...

Wow! I am very pleasantly surprised at the number of OSGi mentions at
JavaOne this year. OSGi was mentioned in the opening keynote by Sun when
discussion module systems for Java 7. It was also mentioned in the JSRs 277
and 294 sessions as well.

I attended a Java SE Embedded session where it was mentioned that OSGi was
being requested by a number of Java SE Embedded customers. Both Equinox and
ProSyst were specifically mentioned.

Peter Kriens and I just finished our OSGi Best Practices session which went
very well. We had 150+ attendees and loads of really good questions at the
end of the session. I hope to make the charts available online for you next
week.

Off to another session...