JSR 277 Review

The JSR 277 Java Module System is an important JSR for the OSGi Alliance and me. Though we have been working on Java modularization since 1998 I did not get a chair on the table; the table was already full with 14 seats. Last week, the current 20 man Expert Group has recently dispatched their Early Draft. So this is a personal account of my reading of the Early Draft.

JSR 277 defines a static module system that is similar to the OSGi Require-Bundle approach. Modules define their metadata in the MODULE-INF/METADATA.module resource. Dependencies are pecified in the import clause with the module’s symbolic name and version range. Other clauses in the module’s metadata list the resources and classes that are visible to other modules. When a JSR 277 application gets started, it should have a main class with a static main method. However, a module must be resolved, as well as its required modules, before the main method is called.

Module definitions can be obtained from repositories. Repositories contain module definitions that provide metadata to guide the resolving process. Name and version ranges are supported from the metadata. However, an import policy class can be used to do more complex resolving strategies. Repositories can be file or URL based, allowing for remote repositories. Repositories are hierarchically linked; if a repository can not find a module definition, it will delegate to its parent. At the top of the chain is a system repository and above that a boot repository. Module definitions are instantiated before they can be used; this associates a class loader with a module. When all the modules are resolved and instantiated, the main class is called and the program can execute. There is no API for unloading a module.

The ambition level of JSR 277 is significantly lower than where the OSGi specifications are today, even way lower than we set out in 1998. In a way, for me the JSR 277 proposal feels rather toyish. The Expert Group took a simplistic module loading model and ignored many of the lessons that we learned over the past 8 years. No built in consistency, no unloading, no package based sharing. Maybe we wasted a lot of time or solved unimportant problems, but I really do not think so. It would be ok if they had found a much cleaner model, well supported by the VM, but it is clearly not. Their model is based on delegated class loaders, just like the OSGi framework minus 8 years of experience. Ok, I am biased, so let us take a closer look. Though just a warning, the following text is utterly boring unless you are handicapped with a deep interested in Java class loaders. Much of the work over the past 8 years was making bundle developers worry about application functionality and let the OSGi framework worry about class loaders. This stuff should be under the covers. Alas.

The key disappointment in JSR 277 is the lack of dynamics. There is no dynamic loading and unloading of modules/bundles, any change will require a reboot of the VM. The model is geared to the traditional application model of starting an application, running, and then killing the VM. Fortunately, JSR 291 Dynamic Component Support addresses dynamics and more comprehensive module loading, so let’s hope that will become the Java standard.

The most surprising feature of JSR 277 is a total lack of consistency checking. Modules are loaded by name and version only; there is no verification that the graph of class loaders form a consistent class space. Take the example in paragraph 8.3.3.3. There are four modules: A, B, C, and D (-> means import).

A -> B v1.0, C v1.0
B -> D v1.1
C -> D v1.0

Modules resolve independently, implying that module B will resolve to module D v1.1 and module C will resolve to module D v1.0. The result is that module A can see the same class coming from module D v1.0 and module D v1.1, likely resulting in class cast exceptions. Or more concrete, assume module A is your application and module B and module C implement some web framework and module D is javax.servlet 2.1 and 2.4. If you use a Servlet, do you get it through module B or C? Errors resulting from this inconsistency can be very hard to trace. There exists a method in the Module class called deepValidate() that uses a brute force method by loading all the classes in all modules to see if class cast pr class not found exceptions occur. Obviously this is very expensive and not even bound to find most consistency problems, just loading problems!

A key problem with the Require-Bundle/Module approach is ordering. Though the EDR implies that classes are searched in a specific order, this still depends on what other modules do. Assume we have:

  A -> B, C, D
  B -> D

The order for module A is not B, C, D but instead B, D, C because module B also imports module <D (assuming classes we re-exported). This is one of the reasons I do not like Require-Bundle, though practically we have much less of a problem than JSR 277 will have. The issue is split packages.

Split packages are packages that come partly from one module and partly from another module. Split packages are nasty because the package security access does not work and it is also easy to unexpectedly shadow classes and resources.

Security problems occur with split packages when you load class com.p.A from module A, and com.p.B from module B. The different class loaders will make it impossible for A to see package private methods in B despite the fact they reside in the same package. This is not a theoretical problem especially if multiple versions of the same module are supported. It is easy to pickup a new class from a new version but then load the auxiliary classes from a module that is listed earlier.

Worst of all, none of this is under the control of the importer. The importer just imports a name and version, and then receives whatever the exporter decides to export. Obviously, this is a brittle connection. For example, module A contains packages a and b. One day module A is refactored and package b is moved to module B. This is not uncommon because big modules are not easy to work with, more about that later. Unfortunately, all clients of module A must now be changed to add module B to their import lists even though nothing has changed except the packaging. Why is this not a problem with OSGi imports? Well, OSGi imports packages, in the previous example package b will just be obtained from module B without any change. It might not seem like a big deal, but for large systems this can amount to major work.

Another problem with modules is their fan out. Typical projects create JARs that are useful in a number of situations. For example, a program like bnd (creates bundle manifest headers from the class files) can be used as an Ant task, an Eclipse plugin, from the command line, and a Maven plugin. It will therefore have to import packages from Ant, Eclipse, and Maven. Those dependencies are needed and are therefore more or less ok (though it is nice to make them optional, a feature missing from JSR 277). However, Ant will import another (large) set of modules. Similarly for maven and Eclipse, ad nauseum. The bnd program is a terrifying example, but the problem is that the usual fan out of a module is large and the fact that dependencies are transitive can worsen this effect. If you want to see the effect of module like imports, check out maven. A simple hello world program can drag in hundreds of modules due to the transitive dependencies. Modules (and Require Bundle) as well are a typical example of creating a constructs to solve a problem but simultaneously creating problems on the next level, problems which are usually ignored in the first version only to bite the early adopters.

A puzzling aspect of 277 is the dependency on super packages from JSR 294. Super packages are shrouded in a veil of mystery. The only public information so far came from Gilad Bracha’s blog. JSR 277 unveils a bit more but many questions are left unanswered. Super packages list all their member classes and a class can only be member of a single module. In 277, it is implied that this module file maps closely to the module metadata. If this is true, than this is a huge constraint. It means that the deployment format is rigidly bound to the development time modules. Each development module must be deployed separately.

I’ve found that managing the packaging is a powerful tool for the deployer. I have written many bundles that mix and match packages from different projects. This flexibility is needed because there are two counter acting forces. On one side you want to simplify deployment by deploying the minimum number of bundles. Anybody that had to chase dependencies knows how annoying the need for more and more bundles. On the other hand, bigger bundles are likely to create more dependencies because they contain more (sometimes unnecessary) code. So you like to minimize the number of external dependencies. Sometimes the best solution is to include the code of other JARs in a deployment JAR. The proposed super packages will put a stop to that model: it will be impossible to manage the deployment packaging because the programmer has decided the packaging a priori; bad idea.

The biggest regret the EG members will have within 2 years is the import policy. Why? Isn’t it a nice idea that the programmer can participate in the resolving? Well, in Java the solution to those problems looks deceptively simple: use a class to abstract the required functionality. An import policy is a class in your module that gets called during the resolving of the module. It can inspect the module metadata, query the repositories and bind other modules.

In the OSGi Alliance, we inherited a similar idea from Java Embedded Server (JES), the archetypical OSGi framework developed by SUN. After countless hours talking and testing we decided that it was a bad idea because of 2 key reasons:

1. By definition, you do not have a valid Java environment before you resolved all the required modules. Code executed in a module that is not resolved is in limbo and is bound to run into problems. There are also several security related issues.


2. Having a procedural solution will prevent management systems from predicting what module will resolve to what module. Within the OSGi specifications, we spent countless hours to make the systems predictable for this reason. This is necessary because in the future more and more systems will be managed (I can’t wait for that day!). Inserting a user class in to the resolution process leaves the management system in the blind. Interestingly, Java made the same mistake with security permissions. Abstracting the permissions as a class is good OO practice, but it kills any possible optimization. A declarative approach could have significantly reduced the 20%-30% overhead of Java security while the flexibility that the model offers is rarely used.

Wasn’t there anything I liked? Well, I like the idea of the repositories. The OSGi framework maintains the repository internally with its installed bundles, it left external repositories outside the specification by standardizing the installation API. Repositories allow the framework to download or find modules just before activation. This is an interesting model, pursued by Maven and the OSGi Alliance with OBR.

However, I think the JSR 277 repository model is too simplistic. It codifies the current maven repository model, which is still immature and will likely change over time. For example, the current model takes only two extra constraints into account: OS, and platform. Unfortunately, life is seriously more complicated. OS’s have their own versioning scheme, processors have compatibility matrices (i.e. an x86 runs on a i586, win32 is compatible with WinXP but not vice versa), the OS is often a variation of OS and window system. Encoding these constraints in the file name is obviously bound to collide with reality one day. In contrast, the OSGi OBR uses a generic requirement-capability model derived from JSR 124 that is much better suited for finding the right module.

Well, the paper document I have reviewed is heavily covered with my marker and I could continue for more pages. However, it is too much for this blog. Well, ok, last complaint: the syntax for version ranges, it is too tempting to leave it alone (though it is not that important). The industry more or less has standardized on versions with a major, micro, minor number and a string qualifier. JSR 277 adds an update number that is between minor and qualifier (Maybe one day someone can explain to me why we need all those number while the only thing that is signaled is a change that is backward compatible or not, but developers seem to like to have all those numbers). However, I have no problem adding another number beneath minor. I do have a problem with a version range syntax that is obtuse and non-standard.

Intervals have a mathematical notation that is easy to understand. Parentheses are non-inclusive and brackets are inclusive. The interval 1 < x <= 5 is therefore represented as (1,5]. [2.3,5.1) indicates any version that is more or equal to 2.0 and less than 5.1. Simple, elegant, has been around since Pythagoras. Choosing this notation for the OSGi specifications was a no-brainer.

Now, let us take a look at what JSR 277 brewed. They use the same terminals as regular expressions, but they have a very different meaning. A partial version can be suffixed with a + (anything later) or a * (anything with the same prefix). So if you say 1+ you mean [1,∞) or 1. 1* is [1,2). The JSR uses square brackets to fix the first positions while floating the last. For example, 1.[2.3+] is [1.2.3, ∞) and 1.[2.3*] is [1.2.3,1.2.4). The JSR 277 syntax has no easy formulation for the not uncommon case [1,5). The only way to express this is with concatenation of version ranges: 1*;2*;3*;4*. Ok, enough about this strange syntax.

Conclusion. JSR 277 takes a simplistic view of the world, ignoring many real life problems: consistency, optionality, split packages, etc. The only way this EG can pass its final review is lack of attention for detail of the Executive Committee or alternatively, serious muscle power of SUN. JSR 277 is a missed opportunity for Java. I do not doubt that this specification will end up in Java 7, but it will further fragment the Java world for no technical reason. Not only is this specification impossible to implement on J2ME any time soon, it will also leave the many OSGi adopters out in the cold. Why?

Peter Kriens

Alternative OSGi Styles

When you spent most of your daily life worrying about extremely detailed Java class loader peculiarities then you tend to start thinking that what you’re doing is really important. This feeling is amplified when you see all those important people taking it really serious. And then you suddenly get confronted by a group of people that completely violate our specification but still do something very interesting.

I am talking about osxa. They looked at our specification and decided that the goal was lofty but that it contained too much unnecessary cruft for them. They just took the interesting 80% (well for them interesting) and ignored the rest. I almost took it personal, but they just put all bundles on the class path! Ok, it meant that they did not need class loaders anymore and that opened a larger range of deployment scenarios. For example, they can deploy their applications in a JNLP web start without the need for signing.

Obviously this implies they have no proper isolation between bundles; bundles can conflict in their packages. However, this is isolation is not always needed, especially when the set of bundles that gets deployed together is fixed. These conflicts can then be resolved ahead of time.

And this is the different, surprising, perspective that they took. They love the service registry and do not care about the module layer. They needed a model where they could use many different components, bunch them up together, and deploy them as a whole in a JNLP deployment or inside web app server. What me worry about bundle life cycles?

Actually, when I realized how much they liked the service registry, I could not stop myself from warming up to them. I am often told that the real important stuff of the OSGi is the module layer. The service registry is just a hanger on. In JSR 291 we actually tried to remove the service registry to simplify the specification. We did not because the module layer and the service registry were very hard to separate. However, I always liked the services most; for me, the module layer was the necessity but the services were interesting.

Obviously the osxa people trample over our carefully crafted specifications. However, I think their perspective is interesting. I think it demonstrates the power of the specification that you can write bundles that will run in a compliant framework but also in something that is wildly different. I have always felt that bundles are the important parts; as long as bundles can write against a good specification, then framework developers should be allowed a lot of leeway. It is interesting to see how they use quite a few equinox bundles.

Anyway, the osxa will not receive compliance anytime soon because they completely discard the module layer and take a lot of liberty with other aspects. They can therefore obviously not call themselves an OSGi framework implementation and developers should be very careful not to target any of the peculiarities of this framework. Still it is interesting to see that the success of the OSGi specifications is enabling these initiatives. Initiatives that can have value in certain circumstances and clearly prove the viability of the OSGi model.

Now, if they only could get their demo working again …

     Peter Kriens

OSGi UIs and the Web

The OSGi user interface has been a problematic aspect from the beginning, despite the fact that there are so many to chose from: awt, swt, lcdui, swing, thinlet, flash, svg, p3ml, proprietary, and html based. We tried many, many times to come up with a model that all could live with, so far to no avail.

However, one model that seems to emerge is web based UIs. The advent of DHTML, Javascript, and SVG provide an extremely rich environment for application development. Unfortunately, Javascript is not a nice language to handle large and complex problems. Many companies solve this by using webservices to distribute the work between the graphic user interface (GUI) and a server. Web services are remote procedure calls done over the HTTP(S) protocol. This model is reasonably well supported by Javascript using the XMLHttpRequest object.

Th web services model maps very well to an OSGi service platform. In a way, you just need a way to easily call the services in the OSGi service registry from Javascript. So how would such an application look like?

Let us first layout the bundles I like to think in bundles because it is nicely concrete and down to earth. We obviously need a web server, which is easy to fulfill with the OSGi Http Service. We then require a bundle to handle the grunts of the remote procedure calling and another bundle to prove that it works.



The rpc calling bundle has the responsibility to handle the remote procedure calls from Javascript and map them to calls on an OSGi service. I’ve called this bundle webrpc. The second bundle is the demo application. This summer I emotionally blackmailed my son Thomas to learn how to write a program; he caved in and wrote a working Sudoku game in Java after a false start in C++. This exercise gave me more experience with the Sudoku domain than I ever wanted to have so the demo will be a sudoku bundle.

The webrpc bundle should not indiscriminately export services. There are obviously security issues but also not all services are well suited to be exported. However, we’d like to keep the service as simple as possible. The choice was made to look for the public property on the service registration. This is easy to set with declarative services and it means we can use a POJO as our service; the service itself is not involved with the remote procedure calling. The value of the property is the name under which the service should be known to the outside world.

I could have used a complete web services stack, there are an amazing amount of those applications available, almost all written in Java. However, I wanted to run the demo on my Nokia E70 phone that runs an OSGi service platform. Memory is only abundant on PCs and services, so a complete web services stack can add up. It also adds up on the Javascript side because a compliant web services stack is non-trivial. I therefore decided to optimize space and implement a very lite model of RPC over HTTP. Requests are done using the HTTP protocol. The path provides the service and its method, while the query parameters provide the arguments. I used 0 for the first argument, 1 for the next, and so on. For example:

     http://localhost/rpc/sudoku/newGame?0=simple

This request will invoke the newGame method on the service that is registered with the property public set to “sudoku”.

The webrpc bundle depends on the Http Service. When there is an Http Service present, it registers itself under the /rpc alias. The servlet will get called for all requests that start with /rpc. For such a request, the webrpc bundle looks up the service name (in the example sudoku) in the service registry using a simple query string with the OSGi filter language to do this. The next part is calling the right method in this service; this is non-trivial because it is necessary to coerce all the arguments from strings into the method argument types. I wrote a separate Invoker class to handle this coercion. Read it and weep. Most of the work is handling Java’s silly primitives.

When the method has been called it returns an Object. This object must be shipped back to the Javascript in the browser. Now the gut reaction of most programmers is to think XML. Not me this time, there is another standard called JSON that is so much easier for Javascript. It is a representation of strings, numbers, arrays, and maps that is compatible with the Javascript syntax. Converting an object into JSON is quite straightforward for values like numbers, strings, maps, and arrays; references to objects are a bit harder but that is a nice subject for another article. The JSON value is then returned as the body of the HTTP response. The Javascript will receive this text body and compile it to Javascript arrays, values, and dictionaries. From the Javascript point of view it can not be simpler.

The webrpc bundle maps requests that do not look like a method name to a getResource() call on the service object’s class. A service implementer can place all its web resources in the www directory and they are automatically mapped. That is, if someone requests /rpc/sudoku/prototype.js, it will load the resource from aQute/sudoko/www/prototype.js. Assuming the implementation class is in the aQute.sudoku package.

All this effort in the webrpc bundle has makes the sudoku bundle almost trivial. Well, I guess it was trivial in the first place because the server only calculates the board and this is not rocket science. The Soduko service only implements a newGame method. This method returns an array of 81 integers (remember how 9x9 was?). Negative integers mark positions that can be shown from the beginning; positive integers are to be guessed by the player. For the HTML side I decided to make a splash screen in SVG. Hey! We now have those fantastic tools, so let us use them. Though splash screen sounds more dramatic than it turned out, I admit, I am a lousy artist.

For the handling of the HTML and http requests calls I am using prototype.js. The Javascript and HTML is actually quite small. The Sudoku board is completely drawn in HTML and formatted using XML. When the Javascript starts up, it requests a new game from the Sodoku service. When this information (asynchronously) arrives, it formats the board with the received information. Rather cool in this model is the use of Cascading Style Sheets (CSS). CSS makes the mundane detail of formatting all the little details nicely manageable; however, remember I am not an artist so the looks can be improved.

Both the webrpc and sudoku bundles used declarative services. The webrpc bundle was hard to test standalone because it relies heavily on the OSGi service registry. The Sudoku service was however a POJO and could therefore easily be tested.

Try it out! If you think it looks too big, just make the window smaller. Notice how nicely it all scales because CSS and SVG are used to adjust all parameters to the window size.

You can download the bundles from the OSGi Bundle Repository (OBR). The JAR’s contain the source code of the bundles in the OPT-INF/src directory. See how extremely little code is required to achieve the goals. Enjoy!

     Peter Kriens

Enterprise Workshop

I am typing this on my way back home from a very successful enterprise workshop. Over 30 people from many different companies had taken the trouble to show up for this requirements gathering session; many of them OSGi members but also a number of new companies that are interested in using the OSGi service platform in less resource constrained environments than home gateways, cars, or mobile phones. After this busy day I can safely say that there is sufficient interest to start working on enterprise OSGi! We will create the charter for the EG in the coming weeks.

How do you gather the requirements from 30 people at the same time? Not an easy task. We decided to first get a short introduction from all participatns and then let volunteers present their position statement. We got a few interesting presentations from Siemens, SAP, and others about where they see the opportunities for the OSGi specifications to play a role. We then spent some time doing a round table discussion to capture the ideas and requirements from the different participants. These ideas were categorized and then sorted by voting. After several failed voting system attempts we decided to vote with post-it notes glued to the projector screen, which reminded Jon Bork from Intel of “hanging chads”. Fortunately, we did not need the supreme court to intervene.

“Stay modular and small”. This was one of the first things Hal Hildebrand from Oracle said, and nobody disagreed. People like the OSGi technology because it was created for a constrained environment, which has given it a lean appearance. Unanimously, we liked to keep it that way. The new specifications required for the enterprise group should not fragment the world like J2ME and J2SE and neither should they bloat like J2EE. New features should be added in a modular fashion so that the basic framework is still usable in embedded environments.

A key requirement we identified was distribution. The strength of the OSGi specifications are that they specify a very efficient model for multiple applications to share a single VM in a single process. However, when moving to the enterprise it becomes crucial to provide mechanisms to scale to multiple processes, multiple VMs, multiple machines, and multiple languages. So far, we have not addressed outside VM issues because inter-process communications requires very CPU intensive mechanisms that are just no viable in resource constrained environments. It is, however, clear that the enterprise world is more than willing to pay the extra price for the increased scaling, availability, reliability, and access to applications written in other languages than Java that a distributed model can provide. As was remarked, most new code is just the fur on a giant hairball of legacy applications. From an implementation point of view, the OSGi service registry seems the perfect candidate to implement these requirements.

Another requirement was more focus on the whole life cycle of applications. The OSGi Alliance provides good specifications for the service platform but development, debugging, deployment, management, auditing, monitoring, patching, education, and certifications of applications have so far been out of scope. These were considered crucial aspects for “universal middleware” standard to succeed in the enterprises.

There was also a very interesting discussion about more comprehensive dependency management. Can we model the aspects that are outside the VM? More extensive metadata can go a long way to make deployments more accurate and offers possibilities in grid computing like models.

A number of new services were proposed. An interesting one was the network awareness service. I remember discussing such a service at least 5 years ago. Time flies. Also, people would like to see even more security mechanisms, an enhanced configuration admin, data synchronization, web services stacks, and several more.

There were many more topics raised and all of them deserve further discussions. However, the previously sketched areas seem to cover the most urgent needs. The next steps we have to take is chart a charter for the expert group, select a chair (if you are interested, you should propose yourself as a candidate), get board approval and then have our kick off meeting so we can start working. Most companies volunteered people to work on different aspects of the specifications..

The general scope of the EG is clear: We want to make OSGi feasible in the more distributed enterprise world and address the non-Java oriented aspects of enterprise applications. If your company is working in this scope, I would seriously consider joining. Looking at the current adoption rate of OSGi in his area and the participating companies, I would say that we will probably develop some pretty important standards in the coming time; specifications that will impact the work of many. Though we have a sufficient number of companies to get started, we can always use more!

     Peter Kriens
.

OSGi and Spring

Update: JSR 232 is approved in the final ballot!

Last week I visited the BEA offices for a meeting with the Spring-OSGi group. This project started a few months ago when Interface21 (the shepherds of Spring), specifically Adrian Colyer, contacted us to see if we could make the Spring- and OSGi technology compatible. At first I had to look a bit deeper into Spring because I had only heard of the technology. I quickly found out that it is highly popular in a diverse range of Java applications. So what does Spring do?

The core responsibility of Spring is configuration. One of the key software lessons our industry learned in the last decade is that you want your software modules to be as cohesive and uncoupled as possible. The more uncoupled software is, the easier it is to reuse it. However, in the end you need some place where the modules are wired together and their options set. It is a like squeezing a balloon, you can never get rid of the coupling, you can only concentrate it in as few places as possible. This is a similar model that is promoted for OSGi applications; your bundle activator should configure a number of POJOs that finally do the real work. Spring's reason for being is based on this software model.

The Spring core interprets an XML file that describes how a set of beans must be wired together. A bean is a POJO that is configured with get and set methods. Beans are accessed using reflection but can optionally participate by implementing a number of Spring interfaces. The Spring XML looks a lot like a script language (something XML is extremely ill suited for, but that is another story, and yes, I know there is an alternative) but actually creates a graph and can therefore correctly handle dependencies and ordering.

On top of this configuration code, the Spring library has a large number of nicely decoupled libraries that can be used to decorate the beans. Do you need for JTA transaction support, but your POJO did not know anything about JTA? No problem, with Spring you can provide the required transaction calls around your methods without changing the source code. Spring uses two forms of aspect oriented programming to achieve the decoration. You can use events around the method calls defined in an interface, or you can use full blown AspectJ programming. There is also a library that provides an shell around JDBC, removing any implementation intricacies. And there is of course much, much, more. Overall pretty cool stuff.

Oracle and BEA were also very interested in providing Spring capabilities on top of the OSGi Service Platform. Hal Hildebrand (Oracle) and Andy Piper (BEA) were part of the team doing the work. On the mailing list, Richard Hall and Jeff McAffer also tuned in. With these people, you can be guaranteed of lots of interesting discussions on the Spring-OSGi mailing list; I learned a lot about features required to integrate larger enterprise applications. There is a lot of messy code out there! It is amazing how many home grown mechanisms application programmers have invented to get some kind of extensibility. Many of these mechanisms have peculiar ideas about class loading that do not always mesh well with the OSGi rather strict rules. One of the key focus of OSGi R5 will likely be addressing how this legacy code will run unmodified on OSGi service platforms.

A couple of weeks ago I decided to plummet in the deep and see what all the fuzz was all about so I wrote a simple application using the Spring-OSGi prototype. Adrian had written a very nice draft of a spec so it was not hard to get started. After a couple of hours the application was running and I seemed to be able to use many of the facilities. From an OSGi perspective, I had some qualms. I thought some of the solutions were to cumbersome and there was too much choice in certain areas. I wrote a review report from my experiences, which I had to redo because it got lost in the mail ... I think the project was going very well but I felt we needed to do a hack session together. I did not have enough Spring experience to change their code, and I think they could use some OSGi experience to improve their code. We therefore decided to meet in London. Olivier Gruber and BJ Hargrave from IBM also attended because of their
OSGi expertise and interest in Spring.

The group had become a bit too big to do any real hacking but that turned out to be ok. We spent most of the first day going through Adrian's excellent specification text. We were incredibly productive that day, we killed so many features! With lots of intense discussions we could significantly simplify the model. It turned out that the OSGi service registry and the Spring application context could really work together very well. The tricky part is of course the dynamics, the OSGi specifications seem to remain unique in their focus on dynamicity. Interestingly, the end result looks like a specification that can take over or merge with our Declarative Services in the long run because it is now a super-set. Another item to discuss for OSGi R5.

Overall, this is a very interesting project that has a lot of industry attention: two very cool technologies that are being married. I hope that the group can turn the prototype into the mainstream Spring code. Adrian told me they were thinking of Spring 2.1. As a bonus, Spring 2.1 will deliver all its libraries as bundles, that is, all libraries will have the correct OSGi headers.

I am convinced that the OSGi Enterprise workshop will discuss this project, for example, should OSGi standardize the OSGi layer of Spring?

Peter Kriens

P.S. You did RSVP did you? Realize that we will have limited seating based on the people that have RSVP'ed. Hope to see you there!

OSGi Enterprise Workshop

It is rare that a technology developed for embedded devices ends up at the core of enterprise applications. The constraints and optimization axes of these two worlds are, ehhh, worlds apart. Still, it looks like the OSGi Alliance is succeeding in this remarkable feat as demonstrated by the increasing interest of enterprise software vendors in OSGi technology. Initially, some vendors voiced concerns about potential patents held by the founding members, but the by now famous patent pledge made those worries go away.

Enterprise software is not a well defined area. We know it is usually big, requires lots of people to work on it, and it is invariably expensive. Web servers are of course a big aspect of Enterprise software as well as tight integration with databases. Java made big inroads in this world because it provided a uniform environment for applications. Its type safety reduced the number of errors during the integration phases and this made it smoother to integrate different applications.

J2EE was SUN’s attempt to capture the communality of Enterprise applications in a framework. The idea behind a framework is that it reduces the application specific code and the resulting application more robust. It is a wonderful appealing idea and object oriented theory gave birth to too many frameworks in the nineties. I even build one for Ericsson in 94! Frameworks are incredibly appealing but they are so hard to get right. They key problem is the natural focus on the advantages of the framework and total disregard for the increased complexity and learning curve that a framework will cause. For example, if we add a function for authentication, we needlessly increase the complexity for people that do not need authentication at all. Just the increase in the API and manual is negative. The battle field of software is therefore littered with failed frameworks (alas, including mine). Only frameworks that can restrict themselves to core functionality and fanatical focus on decoupling stand a chance to succeed.

I do not think SUN and compatriots heeded those lessons from the nineties when they started with J2EE. From the start, J2EE became a monster in deployment complexity and intrusiveness. 5 years ago I bought a book about J2EE and spent a week developing applications, just to understand what the hype was. Well, I utterly disliked the intrusiveness of the code and the many, seemingly unnecessary, restrictions. This dislike happened ever before I found out about the XML situps that one needs to perform before you get your “Hello World” running. Look at the funny comparison between web application development environments from JPL to demonstrate my dislike. I understand that the EJB 3 JSR is providing more focus on simplicity, but I will hold my judgment until I have played with it.

The success of Spring is caused by the intrusiveness of J2EE into applications and its inclusion of many technologies that are not always necessary, or in other words, the classic Framework syndrome. Developers love frameworks but they are not very faithful; this is where the POJOs are coming from. POJOs are successful because the coupling between the units of design are minimized, but more important, they are coupled in the right direction. Let the framework call you, not you calling the framework. The result is that the developers can focus on their domain area, not on learning Yet Another Framework.

The OSGi Framework is successful because it was designed to decouple, which provides many benefits during development and deployment. The OSGi Framework could not care less about what your components do and how they do it. It only provides the primitives to deploy your components and to let them collaborate in a dynamic way. Collaboration is done through service interfaces, which makes the components easy to test without complicated harnesses. Components can be build on top of POJOs, very little code has to be coupled to the OSGi Framework. Standardizing this component layer will create universal middleware. Middleware developed in different industries that can be deployed on any platform. In the end, application development will be mixing and matching universal middleware.

The OSGi development model is so attractive to the Enterprise world because it is sometimes amazing how often I see the wheels being reinvented. Often just because the existing wheels were connected to axes that the developer had no use for. Though the OSGi Framework is quite mature for the embedded, vehicle, home automation, and mobile markets, we need to work on the issues that are crucial for the enterprise markets. What functionality is missing in R4 that is important for enterprise applications?

Next month I will lead an OSGi workshop in the bay area where we will discuss these issues with a number of experts. This is an open workshop but seats are limited. If your company is into making Enterprise applications, please attend (register ASAP with rranck@inventures.com, we have limited number of seats). The signs are that OSGi technologies will play a major role in server side applications over the coming years. If you can not attend the workshop, do not hesitate to add comments to this blog or just email me.

Peter Kriens

Why is Software So Brittle?

The past few days I acted as a software user instead of software programmer: I needed a wiki for an OSGi project. How hard can that be? After a bit of Googling I found Ward Cunningham’s site (he is the Wiki inventor) and he is hosting a wiki about wikis (This text will have a weird number of w words). Well, that site contained a very long list of wikis! I guess a wiki is on the sweet spot of being relative easy to make but with a big visual impact.

How should one choose from so many wikis? Silly enough, the implementation language seemed to be the most logical selection criteria. As you might have guessed, Java is one of my favorite languages and was therefore an obvious choice. Eagerly checking the list for OSGi based implementations turned out, however, to be a disappointment. Many wikis point out how extensible they are but none had chosen the OSGi service platform to implement their plugins; they all developed plugins in a proprietary way. There is a lot of evangelizing left to do.

Anyway, after looking at some Java implementations I found a wiki that looked very promising: FitNesse. This is not a simple wiki, it also contains an extensive testing framework. However, the wiki looked really good (not a common feature for a lot of wikis) and the code looked interesting. When I inspected the code, my hands itched … if I had been a wise men, I would have slapped them and continued. The code looked so clean and so easy to bundlefy that I could not resist. Turning it into a bundle was indeed trivial; the makers of FitNesse provided a very clean configuration and start/stop interface. I only had a slight problem with resource loading. They used the system class loader for resource loading instead of their own class loading, can’t see why, but if that was all I could fix it. Still optimistic at that moment in time.

We have an OSGi framework running that hosts the Bundle Repository, so it was attractive to install this wiki bundle on the same machine, sharing the same VM for these applications is a serious memory saver. The FitNesse bundle ran its own web server (Oh, why don’t they use OSGi technology?). This required me to map their web server through our standard web server so that we could handle authentication in one place. The standard OBR web server runs behind a firewall but is proxied through our normal web server. After trying this same trick for the wiki, it turned out that the FitNesse wiki code assumed it had the whole web server on its own; it could not handle a prefix path. I was getting this sinking feeling that I might have taken the wrong path. As a solution, I could of course run the web server directly out in the open. Unfortunately, this would require me to write the authentication code. The sinking feeling changed into hitting rock bottom. Maybe it was time to turn around and see if there were easier ways of getting this done. If I only had been a wise man …

I talked to BJ Hargrave and he recommended the MediaWiki, the wiki that powers the WikiPedia. This wiki is PHP based and run by hundreds of sites. How hard can it be to install this? Well, hard. It turned out we were running PHP 4 on our server and MediaWiki runs on PHP 5. Sigh. Well, we have rpm installed on our Linux system so we just download PHP 5 and we are on our way. Well, that is the theory. The sinking feeling returned. PHP 5 required a long list of additional uploads. Carefully trying to install a few dependent packages quickly showed that the dependency tree closely resembled the big bang. BJ had tole me there was also a PHP 4 version of the MediaWiki so I felt myself floating to the surface again, only to sink at record speed again after installing this version. Yes, it requires version 4, which happened to be a different version 4 than our version 4. The dependency fan out was not as enormous as the PHP 5, but still impressive enough for me to give up.

Simple and small became my main criteria now. So I finally found a simple and very small PmWiki based on PHP that actually had the functionality I needed, and nothing more. It was a breeze to install and did not have any nasty dependencies.

So what did I learn? Well, besides not trying to be clever, I learned that we are an industry with a problem. The last couple of years we have spent a tremendous amount of energy in formalizing the dependencies of our software artifacts; assuming that this would solve the problems of unreliable software. However, it looks like we have created another problem which is the dependency fan out. Unfortunately, virtually all known dependency mechanisms are based on versioned artifacts. Artifacts are usually a collection of functions that are not always needed. Often there is a core of highly coupled packages and a periphery of utilities and mappers to other artifacts. For example, many tool vendors today provide a mapper to run their tools under ant or maven; implicitly coupling their artifact to maven and ant.

We need to make the use of software less painful. Though we can extend our current way of working with big-bang releases (integrating many different products in a single build), in the end we need to find a way that minimizes dependencies between artifacts. The OSGi service registry with its service oriented architecture is clearly in the right direction. However, we need more. Tools that can visualize the dependencies and that can help us reduce the dependencies. Yes, with today’s tools we can actually manage the dependencies, but that does not imply that dependencies have become a positive thing. I think the OSGi is on the right way, but a lot of work is left to do.