Why OSGi Technology is Strategic

The recent interest in OSGi Alliance from the server folks has been very nice. We recently welcomed several new companies that are major players in this market. Big players like JBoss/Redhat, IONA, BEA, and others. We also had some companies join that are smaller but highly innovative like Paremus, Jayway, Interface21, and Luminis. Other companies, already long time members, like IBM and Oracle have put extra people on the specification work. The EEG meeting after JavaOne at Oracle in Redwood looks to be very interesting. So overall, things are looking pretty good.

However, I recently noticed the difference between the original membership and how the membership is shaping up now. The newcomers are quite different from the people that originally founded the OSGi Alliance in 1999. At that time, there was a clear vision about a potentially huge home automation market. Technical people like me were dragged in by business people to create the required technology. Member meetings at that time were mostly crowded with business developers and account managers. We attracted a lot of companies that were hoping to sell products to the operators and service providers. Similar effects happened when Nokia and Motorola initiated the OSGi Mobile Specification and BMW and Acunia the Vehicle Specification.

That picture has quite dramatically changed over the last few years. Today the OSGi Alliance is mostly focused on providing a dynamic module system to Java, with the vision to create a universal middleware standard. The need that we found in the home automation market in 1999 turned out to have a very generic solution, useful for many other industries.

However happy I am about reaching the developers today, and their enthusiasm for the specifications, I think we still fail to convey the importance of the OSGi Alliance to the strategic management of companies that invest significantly in software, or almost any larger company. As a very technically focused club we have won the minds of many developers but we fail to bring the message to CIOs and CEOs. And we need to reach them, their companies are bound to gain most when OSGi technologies are widely adopted.

I understand the lack of their attention because it is a complex world out there. From a high level many specifications, products, and open source projects, look very similar and are in general: overwhelming. Even I, as an expert in the field and strong desire to know what is going on, feels lost lately with the immense amount of components where one can choose from. Maven repositories are said to hold over 20.000 artifacts! And I am already restricting myself to Java related solutions, it is a vast field.

So why does the OSGi Alliance require any different treatment than the Apache Foundation, Microsoft, Codehaus, Objectweb, IBM, or any other of the hundreds of thousands of software component producers? Why specific attention to the OSGi Alliance over the sea of other products? Why should developers choose the technologies based on technical merits, but should OSGi technology demand strategic attention?

The answer is simple, the OSGi Alliance is about managing this overwhelming sea of components and their interaction. It is about being in control of the thousands of dependencies that software has to internal and external artifacts. It is a framework that addresses the heart of the software development process. It is about managing the software development, deployment and the, most expensive one, the maintenance process.

Not by prescribing an agile or heavy development process, but by defining a solid specification for the artifacts that are being produced, deployed, and maintained. Anybody that is familiar with a software development process is intricately aware of the myriad of problems that occur due to the lack well defined dependencies and countless hours are wasted solving problems that could have been prevented. The OSGi technology addresses this complicated and messy problem. Even companies that think that they manage their software dependencies well, often find out to their surprise they did not after starting to use OSGi technologies.

Superficially, the Java platform looks like it is providing this kind of control. However, though the Java platform clearly provides a tremendous value to our industry, there are two big flaws: one technical and one commercial.

First the technical flaw. The Java platform does not standardize a dependency and deployment model. There have been several attempts but non of these attempts is as comprehensive and complete as what the OSGi specifications offer today. The result is that many problems in a development process show up in the final stages, the time when they are the most expensive. There is no roadmap in the Java platform to create a modular system where robust applications can be reliably and quickly build from internal and external components. Integrating external components is difficult and hard work, often creating interactions that are unwanted. It is hard to overestimate the costs associated with this lack of modularity.

The OSGi specifications clearly address this area. The solid specifications define the format of the artifact and their dependencies. These artifacts can be used by OSGi frameworks to manage the components in a robust, secure, and reliable way. In conjunction with a repository, missing dependencies can be downloaded on demand. There is a tremendous amount of technical detail necessary to create such a platform, but once it is there, it is easy to use for developers and very suited to build applications with small, large, and very large teams. Obviously the most visible use case is the Eclipse IDE, based on the OSGi specifications. Eclipse with its thousands of components clearly proves that the specifications are solid. The success of OSGi technology in the mobile phone and embedded market show it is applicable for small systems as well. This is a standard that is clearly cross industry, trying to create a universal middleware layer.

The second flaw in the Java world is not technical, it is commercial. Sun has developed the Java language with the help of many, many companies. Around 2000 they started the Java Community Process (JCP) to formalize the specification process of the Java languages and its libraries. Some very solid work has been performed under this umbrella. However, the process is not neutral because there is no separate organization: the JCP is Sun, there is no separation. For example, participating in the JCP requires one to sign a contract with Sun. In the end, Sun has the final say. This is not a healthy process; the sad story of two OSGi technology related specifications (JSR 277 and JSR 294) show that despite quite heavy industry pressure, Sun can decide to develop a new technology that already has a de facto industry standard solution.

The OSGi Alliance and its specifications have strategic importance because it corrects the aforementioned flaws. The specifications provide a solid foundation to integrate the myriad of external components that are being used in software development today. The organization provides a neutral platform for standardizing components, both cross industry and specific for vertical markets.

The potential gains of using OSGi technology in the development process are large. Most organizations that produce and use software can have tremendous savings if they adopt the OSGi specifications. However, we are not there yet because the value of a standard like OSGi is largely based on the network effect. We need more members, not only to help us expand the specifications to foster an industry wide component market, but we also need new members to make the specifications more attractive for others. The grass-root adoption by the developers that we are seeing today proves that the technical foundation is solid. Any CIO or CEO that recognizes the potential can be sure that the technology works.

Today we are at a stage that you can ask what the specifications can do for you and get a welcome answer. However, we are also at a stage that you should ask what you can do for the OSGi Alliance to increase the rewards for our complete industry. Why not join and make it happen?

Peter Kriens

P.S. Hope to see you on JavaOne or the OSGi community event in Munich June 26-27

The Importance of Exporting ánd Importing

[changed jan 2010]
Though most of what is described here is correct, there is a subtlety. There are two rules where one should not import an exported package:

1) An exported package refers to an private package
2) There is no private package referring to the exported package

Rule 1 (which is really bad practice btw) makes an export not substitutable. Rule 2 is logical if you think about, if nobody uses the exported package internally, why bother? To simplify your life, bnd's since 1.15.0 will automatically apply those rules.
[end change]



Richard Hall started yelling at us telling that we had "done a terrible job of explaining the importance of the collaboration model in OSGi and how importing AND exporting the same package fits into it..." Fortunately he included himself. Anyway, I am afraid he is right.

If you carefully read the specification you will find that all the details are there, succinctly described. However, the specifications are a lot to digest and many details are not that relevant for the bundle programmers, just for framework implementers.

So what is Richard talking about?

An OSGi bundle is a JAR file that contains classes and resources. Meta data in the manifest header describes which packages are exported and which packages are imported. Not exported packages are private to the bundle; other bundles can not import them. This is a simple model that a (clever) child can understand. First time users therefore clearly separate their packages into exported, private, and imported.

Ok? Well, no.

The problem is that the OSGi specifications allow multiple exporters for the same package. In this case, the OSGi framework selects one of the exported packages to be the actual importer for a bundle and ignores the other export definitions.

Before OSGi R4 the framework would automatically turn an export clause into an import if not selected to export. However, the R4 expert group felt that it was more flexible to allow a developer to only export a package but never import it. Export only guarantees the bundle developer that he would always see his own classes and never import those classes from other bundles. If he is not selected as the exporter, he happily loads those classes from its own JAR. I have never felt the need for this feature but I am sure someone will tell now comment how he cannot live without this export only option. Since R4, bundles must specifically import their exported packages if they desire substitutability of those packages.

For example, lets assume we have two bundles A and B, both exporting package q and a third bundle importing package q:

 Bundle-SymbolicName: A
 Export-Package:      q

 Bundle-SymbolicName: B
 Export-Package:      q

 Bundle-SymbolicName: C
 Import-Package:      q

The framework must now pick an exporter, lets say it picks bundle A. Therefore, the import C.q is bound to A.q.

So what is wrong with that? Well, because B does not import package q, the framework must divide your installed bundles in different class spaces. A class space is a consistent set of classes. In the previous example we have divided the world; bundle B lives in a separate class space than bundle A and C. This is not a problem as long as they do not have to communicate. Bundle B is perfectly happy to use its own variant of package q from its own JAR file. Bundles A and C happily share the
version from A.

However, problems occur when B wants to communicate with bundle A or C: it no longer can. If A would create a service from package q then this service is bound to A's class space. C also belongs to this class space so if bundle C would get this service the object would be compatible with the class that it is bound to. However, if bundle B would get this service its definition of the class would come from another class loader than the service object's class loader (A). The result would be a very nasty class loader exception. Nasty because the assigned object has actually the same class name as the class it is assigned to, the only difference is the class loader. You can stare surprisingly long at a line like:

 X x = (X) ctxt.getService(ref)

When the object you get back from the getService method is actually implementing X! The first time it happens to you, you start doubting the compiler, the VM, and in the end your sanity.

The OSGi framework therefore keeps services from bundles that live in different class spaces separately. This means that if you list the services in the service registry with the getServiceReferences method, you can only see the services that are compatible with your bundle's class space. There is another method, getAllServiceReferences, that does not filter the services. However when you use that method you are on your own with respect to ClassCastException.

The solution is fairly simple, just import all the packages you export. If you use my bnd tool (which is included in the osgi plugin for maven then you do not have to worry because exported packages are automatically imported. This R3 like model is a very sane default in my opinion.

Importing and exporting enables substitutability. Maybe in this example it does not sound as a big deal. However, not providing it erodes the roots of the OSGi model. The OSGi specifications create quite a bit of overhead to enable sharing. Bundles that only export packages are obviously not good citizens in this model because they are unwilling to use the packages from other bundles. Using export only bundles quickly creates a lot of standalone bundles that can no longer collaborate, voiding a significant use of the OSGi specifications.

So when you export, do not forget to import. Be a good OSGi citizen!

Peter Kriens