Where Will JSR 277 End Up?

Sun has opened up the mailing lists of JSR 277 for observers. As I am denied membership, I now can at least have the privilege of hearing the conversation in this group. And interesting it is. Last week there was a discussion about importing classes, modules, or packages. As I argued vehemently (too) many times, importing packages is better than importing modules because it decreases coupling to a minimum. Some JSR 277 members seem to feel similar and are pushing for a similar solution. However, I think Glyn Normington gave the right answer:

Of course "import package" is superior to "require bundle" in many situations, but I think it would be a vast waste of time for JSR 277 to play "catch-up" with JSR 291.

A better approach would be for the Java 7 platform to provide first class support for JSR 291. This boils down to standardising the experimental class loader deadlock fix ([1]) and enabling JSR 291 to exploit JSR 277's repositories and JSR 294's superpackages.

I like this idea! JSR 277's unique aspect is the repository. Repositories have a clear value and Java does not have one at the moment. Instead of trying to invent a vastly inferior modularity layer that will take years to mature, it would be so much better to use the existing OSGi technology standardized in JSR 291 instead. Modularity from 291, repository from 277, and a new keyword from 294, I would almost start looking forward to Java 7!

Which brings me to another point. Why actually needs a Java 7? I have just started using Java 5 in my daily work, though most of my customers force me to use 1.4 bytecodes. This week I installed Java 6 but I have not found a single item that interests me so far. I understand that in a non-modular system one has to rev the whole system, even if the updates to the parts of the system are minor. Major new VM releases are painful and the steady release of new Java environments are increasing the distance between J2ME and J2SE with every release. Making it harder and harder to run on all platforms.

The industry needs modular Java technology so that we can go back to achieving the goal of "write once, run anywhere". If Java 7 can bring us closer to that goal, I guess it is worth it. If not, then why bother?

Peter Kriens

JSR 291 (OSGi 4.1) Is Accepted by the Executive Committee

As of yesterday May 21, 2007, JSR 291 has been approved. The Executive Committee has approved the OSGi R4.1 specification! I'd like to congratulate Glyn Normington for running this JSR in an exemplary way. It is really good to see that the JCP, in the end, can accept outside technology and bring it under the Java technology umbrella.

Unfortunately, not all EC members voted in favor. Only Sun and Hani Suleiman voted against for an apparent allergy to rubber (stamping). Let us take a look at their comments:

On 2007-05-15 Sun Microsystems, Inc. voted No with the following comment:
Sun's vote is not a negative assessment of the OSGi specification or technology, but rather on the purpose of this JSR and expert group when the specification exists and is being actively developed elsewhere.

The JSR 291 specification is simply a reference to the OSGi Core spec version r4. No other specification work has been created in the context of the JCP JSR 291 expert group. Rather the work to update the r4 spec occurred in the OSGi working group.

This manner of working contradicts the goals of the JCP to create expert groups empowered to evolve Java technology with the freedom to make design choices and technical decisions based on the needs of the Java Community. Sun has consistently expressed these concerns, as have other community members, throughout the lifetime of the JSR as our previous voting comments show.

Since these concerns been not addressed by the spec lead, either at the inception ballot or at the public review ballot, we are reluctantly voting no.

I can now rant about that fact that the OSGi Alliance updated the specification to version 4.1 based upon the JSR 291 EG requirements, and that these rubber stamping issues have been addressed by the specification lead ad nauseum on the mailing list, without receiving any reply from Sun. However, that is completely besides the point. The crucial question that arises from their comments is: Why on earth should rubber stamping be wrong?

To me, the integration of an external, mature standard in the Java world sounds like a perfect time & money saver. And I do not mean the hours saved by the JSR 291 Expert Group by having a solid document and existing implementations. Those hours are minor. I mean the hours saved by the end users because they do not have to waste endless hours trying to use immature specifications that not infrequently have to be overhauled. Any serious Java user can surely recall several JSRs where version 1 and 2 were not really that usable in practice. J2EE comes to mind? Obviously, Java users can only gain when the JCP adopts mature technology. Several key Executive Committee members voiced this sentiment in their comments as well.

Sun's statement shows a troubling disconnect between them and the interest of the Java community. We, as a community, need standards to minimize our work and maximize the quality of our products. We need standards so that for a given problem there is a shared industry wide API so we can be assured that our investments in these APIs are secure. We need standards so we can choose between different implementations, which will reduce the price and increase quality. We do not need standards to keep Expert Groups on remedial therapy. The advantage of a adopting a pre-existing standard is so blatantly obvious that one can only wonder what Sun's motives are. They are clearly not in the interest of our industry.

This disconnect is bad for our industry, and it is obviously upsetting for the OSGi members, adopters, and myself. Our attitude to standards has always been very much to adopt instead of develop. The IO Connector Service, the URL Stream Handler Service, the XML Parsing Service, the OMA DM management interface, and even the whole security model have been designed to work seamlessly with Java and anything that comes from the JCP. For R3, there was a strong push to design a brand new messaging service but after ample considerations we developed the IO Connection service that relies on the javax.microedition.io API. It has been our clear philosophy to only add a standard that fulfilled an unfulfilled need. And, if we did, provide a high quality specification. The standards landscape is already polluted enough as it is. Nobody is waiting for new standards unless it provides significant new value.

To conclude: JSR 291 has been approved and that is good. Sun's comments are upsetting but fortunately not serious because the other EC members voted yes. However, down the pipeline we see the Modularity JSR 277 coming, which is almost a full subset of OSGi technology, and thus JSRs 291 and 232. Is this good for our industry or is this another case where Sun is confused about the needs of our industry?

Peter Kriens

OSGi @ JavaOne so far...

Wow! I am very pleasantly surprised at the number of OSGi mentions at
JavaOne this year. OSGi was mentioned in the opening keynote by Sun when
discussion module systems for Java 7. It was also mentioned in the JSRs 277
and 294 sessions as well.

I attended a Java SE Embedded session where it was mentioned that OSGi was
being requested by a number of Java SE Embedded customers. Both Equinox and
ProSyst were specifically mentioned.

Peter Kriens and I just finished our OSGi Best Practices session which went
very well. We had 150+ attendees and loads of really good questions at the
end of the session. I hope to make the charts available online for you next
week.

Off to another session...

Why OSGi Technology is Strategic

The recent interest in OSGi Alliance from the server folks has been very nice. We recently welcomed several new companies that are major players in this market. Big players like JBoss/Redhat, IONA, BEA, and others. We also had some companies join that are smaller but highly innovative like Paremus, Jayway, Interface21, and Luminis. Other companies, already long time members, like IBM and Oracle have put extra people on the specification work. The EEG meeting after JavaOne at Oracle in Redwood looks to be very interesting. So overall, things are looking pretty good.

However, I recently noticed the difference between the original membership and how the membership is shaping up now. The newcomers are quite different from the people that originally founded the OSGi Alliance in 1999. At that time, there was a clear vision about a potentially huge home automation market. Technical people like me were dragged in by business people to create the required technology. Member meetings at that time were mostly crowded with business developers and account managers. We attracted a lot of companies that were hoping to sell products to the operators and service providers. Similar effects happened when Nokia and Motorola initiated the OSGi Mobile Specification and BMW and Acunia the Vehicle Specification.

That picture has quite dramatically changed over the last few years. Today the OSGi Alliance is mostly focused on providing a dynamic module system to Java, with the vision to create a universal middleware standard. The need that we found in the home automation market in 1999 turned out to have a very generic solution, useful for many other industries.

However happy I am about reaching the developers today, and their enthusiasm for the specifications, I think we still fail to convey the importance of the OSGi Alliance to the strategic management of companies that invest significantly in software, or almost any larger company. As a very technically focused club we have won the minds of many developers but we fail to bring the message to CIOs and CEOs. And we need to reach them, their companies are bound to gain most when OSGi technologies are widely adopted.

I understand the lack of their attention because it is a complex world out there. From a high level many specifications, products, and open source projects, look very similar and are in general: overwhelming. Even I, as an expert in the field and strong desire to know what is going on, feels lost lately with the immense amount of components where one can choose from. Maven repositories are said to hold over 20.000 artifacts! And I am already restricting myself to Java related solutions, it is a vast field.

So why does the OSGi Alliance require any different treatment than the Apache Foundation, Microsoft, Codehaus, Objectweb, IBM, or any other of the hundreds of thousands of software component producers? Why specific attention to the OSGi Alliance over the sea of other products? Why should developers choose the technologies based on technical merits, but should OSGi technology demand strategic attention?

The answer is simple, the OSGi Alliance is about managing this overwhelming sea of components and their interaction. It is about being in control of the thousands of dependencies that software has to internal and external artifacts. It is a framework that addresses the heart of the software development process. It is about managing the software development, deployment and the, most expensive one, the maintenance process.

Not by prescribing an agile or heavy development process, but by defining a solid specification for the artifacts that are being produced, deployed, and maintained. Anybody that is familiar with a software development process is intricately aware of the myriad of problems that occur due to the lack well defined dependencies and countless hours are wasted solving problems that could have been prevented. The OSGi technology addresses this complicated and messy problem. Even companies that think that they manage their software dependencies well, often find out to their surprise they did not after starting to use OSGi technologies.

Superficially, the Java platform looks like it is providing this kind of control. However, though the Java platform clearly provides a tremendous value to our industry, there are two big flaws: one technical and one commercial.

First the technical flaw. The Java platform does not standardize a dependency and deployment model. There have been several attempts but non of these attempts is as comprehensive and complete as what the OSGi specifications offer today. The result is that many problems in a development process show up in the final stages, the time when they are the most expensive. There is no roadmap in the Java platform to create a modular system where robust applications can be reliably and quickly build from internal and external components. Integrating external components is difficult and hard work, often creating interactions that are unwanted. It is hard to overestimate the costs associated with this lack of modularity.

The OSGi specifications clearly address this area. The solid specifications define the format of the artifact and their dependencies. These artifacts can be used by OSGi frameworks to manage the components in a robust, secure, and reliable way. In conjunction with a repository, missing dependencies can be downloaded on demand. There is a tremendous amount of technical detail necessary to create such a platform, but once it is there, it is easy to use for developers and very suited to build applications with small, large, and very large teams. Obviously the most visible use case is the Eclipse IDE, based on the OSGi specifications. Eclipse with its thousands of components clearly proves that the specifications are solid. The success of OSGi technology in the mobile phone and embedded market show it is applicable for small systems as well. This is a standard that is clearly cross industry, trying to create a universal middleware layer.

The second flaw in the Java world is not technical, it is commercial. Sun has developed the Java language with the help of many, many companies. Around 2000 they started the Java Community Process (JCP) to formalize the specification process of the Java languages and its libraries. Some very solid work has been performed under this umbrella. However, the process is not neutral because there is no separate organization: the JCP is Sun, there is no separation. For example, participating in the JCP requires one to sign a contract with Sun. In the end, Sun has the final say. This is not a healthy process; the sad story of two OSGi technology related specifications (JSR 277 and JSR 294) show that despite quite heavy industry pressure, Sun can decide to develop a new technology that already has a de facto industry standard solution.

The OSGi Alliance and its specifications have strategic importance because it corrects the aforementioned flaws. The specifications provide a solid foundation to integrate the myriad of external components that are being used in software development today. The organization provides a neutral platform for standardizing components, both cross industry and specific for vertical markets.

The potential gains of using OSGi technology in the development process are large. Most organizations that produce and use software can have tremendous savings if they adopt the OSGi specifications. However, we are not there yet because the value of a standard like OSGi is largely based on the network effect. We need more members, not only to help us expand the specifications to foster an industry wide component market, but we also need new members to make the specifications more attractive for others. The grass-root adoption by the developers that we are seeing today proves that the technical foundation is solid. Any CIO or CEO that recognizes the potential can be sure that the technology works.

Today we are at a stage that you can ask what the specifications can do for you and get a welcome answer. However, we are also at a stage that you should ask what you can do for the OSGi Alliance to increase the rewards for our complete industry. Why not join and make it happen?

Peter Kriens

P.S. Hope to see you on JavaOne or the OSGi community event in Munich June 26-27

The Importance of Exporting ánd Importing

[changed jan 2010]
Though most of what is described here is correct, there is a subtlety. There are two rules where one should not import an exported package:

1) An exported package refers to an private package
2) There is no private package referring to the exported package

Rule 1 (which is really bad practice btw) makes an export not substitutable. Rule 2 is logical if you think about, if nobody uses the exported package internally, why bother? To simplify your life, bnd's since 1.15.0 will automatically apply those rules.
[end change]



Richard Hall started yelling at us telling that we had "done a terrible job of explaining the importance of the collaboration model in OSGi and how importing AND exporting the same package fits into it..." Fortunately he included himself. Anyway, I am afraid he is right.

If you carefully read the specification you will find that all the details are there, succinctly described. However, the specifications are a lot to digest and many details are not that relevant for the bundle programmers, just for framework implementers.

So what is Richard talking about?

An OSGi bundle is a JAR file that contains classes and resources. Meta data in the manifest header describes which packages are exported and which packages are imported. Not exported packages are private to the bundle; other bundles can not import them. This is a simple model that a (clever) child can understand. First time users therefore clearly separate their packages into exported, private, and imported.

Ok? Well, no.

The problem is that the OSGi specifications allow multiple exporters for the same package. In this case, the OSGi framework selects one of the exported packages to be the actual importer for a bundle and ignores the other export definitions.

Before OSGi R4 the framework would automatically turn an export clause into an import if not selected to export. However, the R4 expert group felt that it was more flexible to allow a developer to only export a package but never import it. Export only guarantees the bundle developer that he would always see his own classes and never import those classes from other bundles. If he is not selected as the exporter, he happily loads those classes from its own JAR. I have never felt the need for this feature but I am sure someone will tell now comment how he cannot live without this export only option. Since R4, bundles must specifically import their exported packages if they desire substitutability of those packages.

For example, lets assume we have two bundles A and B, both exporting package q and a third bundle importing package q:

 Bundle-SymbolicName: A
 Export-Package:      q

 Bundle-SymbolicName: B
 Export-Package:      q

 Bundle-SymbolicName: C
 Import-Package:      q

The framework must now pick an exporter, lets say it picks bundle A. Therefore, the import C.q is bound to A.q.

So what is wrong with that? Well, because B does not import package q, the framework must divide your installed bundles in different class spaces. A class space is a consistent set of classes. In the previous example we have divided the world; bundle B lives in a separate class space than bundle A and C. This is not a problem as long as they do not have to communicate. Bundle B is perfectly happy to use its own variant of package q from its own JAR file. Bundles A and C happily share the
version from A.

However, problems occur when B wants to communicate with bundle A or C: it no longer can. If A would create a service from package q then this service is bound to A's class space. C also belongs to this class space so if bundle C would get this service the object would be compatible with the class that it is bound to. However, if bundle B would get this service its definition of the class would come from another class loader than the service object's class loader (A). The result would be a very nasty class loader exception. Nasty because the assigned object has actually the same class name as the class it is assigned to, the only difference is the class loader. You can stare surprisingly long at a line like:

 X x = (X) ctxt.getService(ref)

When the object you get back from the getService method is actually implementing X! The first time it happens to you, you start doubting the compiler, the VM, and in the end your sanity.

The OSGi framework therefore keeps services from bundles that live in different class spaces separately. This means that if you list the services in the service registry with the getServiceReferences method, you can only see the services that are compatible with your bundle's class space. There is another method, getAllServiceReferences, that does not filter the services. However when you use that method you are on your own with respect to ClassCastException.

The solution is fairly simple, just import all the packages you export. If you use my bnd tool (which is included in the osgi plugin for maven then you do not have to worry because exported packages are automatically imported. This R3 like model is a very sane default in my opinion.

Importing and exporting enables substitutability. Maybe in this example it does not sound as a big deal. However, not providing it erodes the roots of the OSGi model. The OSGi specifications create quite a bit of overhead to enable sharing. Bundles that only export packages are obviously not good citizens in this model because they are unwilling to use the packages from other bundles. Using export only bundles quickly creates a lot of standalone bundles that can no longer collaborate, voiding a significant use of the OSGi specifications.

So when you export, do not forget to import. Be a good OSGi citizen!

Peter Kriens

QCon: Cold Turkey after EclipseCon

Last week at EclipseCon I had the feeling that my evangelist days were numbered. At that conference it was too easy to think that we conquered the world and could claim victory. In comparison, QCon is a bit going cold turkey. There seems to be something called .NET from an obscure company in Redmond that is also being used to build systems. I even met a woman from IBM focusing on open source software that was not really aware of OSGi. I guess that I have to keep on running around in my Ask me about OSGi shirts a bit longer.

Though the news is not all bad, Rod Johnson (he seem to be an even more frequent conference speaker than I) from Interface21 (the Spring people), gave a very interesting presentation in which the OSGi technology played a major role. He even followed the OSGi marketing missives as a new member, explaining that the OSGi Alliance denies they have anything to do with gateways. Of course the effect is that he is educating most of the the audience about the fact that we once had something to do with gateways! Anyway, it was a good presentation that helped to alleviate the first withdrawal symptoms.

Rod also gave a presentation about Aspect Oriented Programming that was quite interesting. I was at the OOPSLA 97 conference where AOP was first made public and have followed it over time, now and then playing a bit with it. However, I always shied away of using it in real products. It is interesting how Spring allows you to use many of the AspectJ features without really requiring any AspectJ compiler. The integration facilities of Spring are quite powerful, just the fact that they use XML is a bit off-putting.

This dislike of XML is the reason that I read the Grails book lastnight (the party was way too noisy and too crowded, so I walked back from Soho after 2 minutes and 12 seconds in the bar). Grails is supposed to be Java's answer to Ruby on Rails (RoR). Key feature of Ruby on Rails is that it does not use XML! Last year at Oredev Niclas Nilsson of Activa gave a cool presentation of RoR on Öredev in Malmö. Anything that actively avoids XML configuration files has my undivided attention.

Unfortunately, Ruby has its own world disjoint from Java. JRuby is supposed to correct this shortcoming but in the meantime Grails is trying to clone and surpass RoR.

Grails consists of Spring (configuration, bean handling, MVC model), Hibernate (persistence), Quartz (scheduling), and Canoo (testing), all bound together using Groovy. I had eyed the Groovy language but never really looked into it. I must admit from the Grails book's explanation I was quite impressed. I am not completely clear how they do all their magic but it looks impressive. This seems a really nice dynamic language that is well integrated into Java. Recently I spent some flights trying out Scala but that syntax was a bit harder to digest (though it is a static typed language while Groovy has typing as an option). Key thing is that Groovy supports closures and syntax for lists and maps. The lack of closures in Java is an unforgivable oversight and the lack of special syntax for lists and maps is a major inconvenience.

Key thing of Grails is that it uses a lot of conventions to minimize the coding one has to do. A code generator uses conventions to create most of the boiler plate code that is so prevalent in enterprise software. I have read the book but not played with it yet so I might be disappointed when I really start to play with it, we'll see. But so far it looks promising.

I am not sure if it is my advanced age or the industry is accelerating but I never felt so behind in catching up with emerging technologies. I guess I have to take some weeks off and learn to work with all those libraries and frameworks that seem to become mainstream. Spring, Hibernate, Grails, Linq, Scala, Groovy, AOP, JRuby, Jython, .NET 3.0, Struts, the list seems endless.

A lot of these technologies have become mainstream and through the Enterprise Expert Group they have become key technologies that should run on OSGi service platforms. Grails seems to contain most of theses hot technologies so creating an application with it and making it run on
an OSGi framework would teach me a lot. A cool project would be to port the OBR repository to Grails and run it on an OSGi framework. OBR v2 seems a very suitable application for it. Then again, time is always in short supply somehow. Which reminds me, I better fine tune my slides for my presentation.

Peter Kriens

Reflecting on EclipseCon 2007

If you followed this blog you have noticed how BJ Hargrave practiced real time blogging from his Blackberry. If you were not on the conference, you must have realized what you have missed a lot. For any OSGi aficionado, this was the conference to be.

So what were the top 3 observations?

Number one by far is just the sheer number of people that talked about OSGi success stories. I can only conclude that the OSGi technology must work. JBoss (who has just become an OSGi member), BEA, IBM, Siemens, Compuware, Oracle, and others all indicate that modularity really helps. Companies seem to benefit tremendously by having modularity. John Wells from BEA made an interesting observation about this. They had always thought they worked in a modular fashion until they tried to use the OSGi technology. To their utter surprise they found out when they ported their system to an OSGi platform that very little existing code was actually modular. I think this is common for all of us; if you do not verify the modularity of the systems you build, it is highly unlikely that you will have it. The only way to get the benefits of modularity is by enforcing the rules, and the OSGi specifications are one of the few standards in this area.

The second observation is that the OSGi service model is becoming more and more important. Interesting, because a few years ago we had serious discussions inside the OSGi Alliance if the service model was relevant. We even broke up the specification in different layers to allow the specifications to be advocated in parts: the execution environment, the module layer, the life cycle layer, and the service layer. Several parties in the industry told us that the modularity layer was cool but they had no use for the service layer. Some even said that did not want to use OSGi just for the service layer. Even in JSR 291 we considered removing the service layer. In the end we decided to keep this layer because it felt too weird to create a novel mechanism to provide the optional services like Package Admin service and Start Level service. It is now hard to believe that we ever considered this! Most projects presented on the EclipseCon were heavily using the service registry. There were even complaints about the support for making service usage simpler, a clear task for R5. A good example of the power of the service model was shown by Jan S. Rellermeyer from ETH. He made the service registry distributed, allowing applications to use each other’s services even if they ran on different machines. Distribution is a cornerstone for the Enterprise Expert Group so Jan’s presentation was very interesting in this light.

The last, but probably most important, observation is that we really need a book. I think that every presentation that I sat in started with complaining about the lack of documentation for beginners. Almost unanimously they like the quality of the specification but indicate that the specification is too concise and detailed for programmers that want to use the specification. It really is time to start writing this book about OSGi ... I am pretty sure others are working on this as well but I guess I can provide a lot of background information that others cannot. Sigh, if I only had more time.

We will clearly do something again on EclipseCon 2008. It was heartwarming to see how the people from the Eclipse Foundation have helped us this year: Ian Skerret, Mike Milinkovich, and Bjorn Freeman-Benson. They sure radiated a lot of enthusiasm for the OSGi technologies (I actually had to blush during the press conference). Thanks! I can assure you that the experience has been very good for us too!

Peter Kriens

P.S. I am speaking on Thursday at QCon in London.

P.P.S. Next week we have a Vehicle Requirements meeting in Eindhoven, the Netherlands, which is open to non-OSGi members. Read more …