Simplicity, Size, and Seniority

It must have been 1996 when I explained to Ralph Johnson that I liked Java over Smalltalk because it was so small and simple. He looked at me warily and said: "wait a couple of years, it will grow". Ralph is a wise man and after a few years when Java grew from a handful of packages to hundreds of packages I understood what he had meant.

I was reminded of this conversation when I read a blog that compared HK2 with OSGi. HK2 was simpler because it had only 50 classes (!) and only a few pages of documentation. To be honest, I had not yet heard of HK2 so I investigated. This turns out to be the Glassfish module system developed by Sun. From the blog you get the false impression that OSGi is the Goliath spec of thousands of classes and HK2 is this new little David that has a more human size.

Interestingly, it is more the other way around, despite their immaturity. Their module/life cycle layer is comprised of 30 classes, their service/component layer is 20 classes. Security is absent. The only mandatory package in OSGi is org.osgi.framework, which is comprised of only 25 classes and contains the security, module, and service layer. Realistically one should add Package Admin, which adds another 3 classes. Still significantly smaller than the 50 classes of HK2, and they have only just started while we have been around for 9 years and had to address all the hard problems that happen when you build real life applications for constrained environments. All the other classes in the OSGi specifications are optional, when you have a specific problem we might have a specification that addresses that problem, when you do not care, just stick to the framework.

HK2 stands for Hundred Kilobytes Kernel. I guess this is something similar as the 1 dollar stores that over time become extremely creative or outright deceit because this (artificial) limit is not maintainable. I am not sure where they are today, but I am pretty sure that HK2 will require more than 100K within 2 years if they want to do it right, the real world tends to be quite messy.

Anyway, for a specification the code size of the implementations is a bit different. OSGi R3 can be implemented in less than 100k, look at Concierge. However, R4 likely requires a larger implementation because it addresses so many more use cases. Apache Felix and Knopflerfish are around 250K and Eclipse Equinox is around 900K. Why is Equinox so big in comparison? Well, their non-functional requirements state that they must be able to handle 5000 bundles. To make such a gigantic system work, they need to revert to many optimizations and caching techniques which add complexity. This difference is one of the best illustrations why the OSGi specifications are important. Depending on your needs you can choose to use Felix, Knopflerfish, Equinox, or a commercial implementation. Each implementation has made trade offs that you can match against your requirements. However, they all over the same basic OSGi environments and most bundles run on all systems.

Comparing the documentation of HK2 and OSGi it is the other way around: OSGi is bigger. The core OSGi framework specification with the optional system services is 288 pages. I agree it is a lot easier to read the 3 web pages and the Javadoc of HK2, but does it help you when there is a problem? It is interesting how the size of the documentation can be used against you. Yesterday, on the Python list someone asked if there could be an OSGi for Python and the reply was a joke about how "impressed" he was with the 1000 pages of the specification. To set the record straight, all the OSGi specifications combined comprise a 1000 pages but the core is less than 300 pages including the Javadoc. This is still a lot of text but we really tried very hard to make them as readable and usable as possible. So what do you prefer? A handful of web pages or a detailed specification. I can only defer to another wise man: "Things should be as simple as possible, but not simpler".

Peter Kriens

OSGi Summer School

Sometimes being an evangelist has its perks. Last week I was a guest at a very beautiful spot in Brittany at the École été OSGi. The only thing I have to do is talk about OSGi! Ok, the travel was a tad long for the distance, the hotel where we are staying took the car, a plane, a bus, a train, and another bus to get to the path that floods twice a day. At times of flood, one has to use a simple ferry. However, the location definitely has charm.

To my surprise, there were 55 students registered, to my even bigger surprise they had to deny many people that registered late because there was only limited place. It really looks like the OSGi technology is starting to make inroads. Especially the fact that the business people outnumber the academics. For a long time, the academics (or at least research people) outnumbered the business people. I love academics, but of course we need people from the industry to move OSGi into the next phase of universal middleware.

This morning I gave the opening speech. Instead of trying to explain the technology, I gave a more emotional story about the way the work on the OSGi got started, how we struggled, what failed, and what finally worked. I also spent a good amount of time on lingering about why service oriented programming is so important. As I wrote last Monday, they key is that service oriented programming allows us to reason about systems that have a much coarser granularity than objects. I tried to hard to reach the French audience, however I betted that their English was better than my lousy French. Hope they got the message!

This afternoon everybody worked hard on a prepared course. It was really cool to see all those people struggling to sell and buy service on an Apache Felix framework. well, it is time for dinner so the only thing I can say, wish you were here!

Peter Kriens

SOA & OSGi

Nine years ago, when we started working on the OSGi specifications, we used the word service to describe the object that acted as a conduit between bundles. Today, Service Oriented Architectures (SOA) are hot and every software vendor seems intent on confusing a muddy picture even further by bringing their products under this wide umbrella. The result is that many people file OSGi under webservices, as the most popular exponent of SOA, and then conveniently ignore any further documentation because they already know what web services are. As an example, a friend of mine consistently gets his papers rejected because many of the reviewers make this very false assumption.

A bit of history. In 1985, I was developing systems for the newspaper market and we just started our Magnus Opus: a newspaper page makeup terminal (this was the time that a 25 Mhz 286 was state of the art). This product was of strategic importance for our company and I did due diligence on a lof of languages to select the best one for this project. After discovering Smalltalk/V 286, I fell head over heels in love. I thought object oriented (OO) software was the best thing since sliced bread! I never had been comfortable with the common design methodologies at that time called Structured Design. Discovering OO felt like coming home after a long journey. After grasping the technology I made my second worst advice ever (the worst was when I told a programmer friend in 1994 that building an e-commerce shopping cart application was a silly idea, sorry Matts). I told my manager that if we started to build our systems using objects we would be able to get fantastic reuse after a year or so. Obviously under the condition that he allowed me to go to all these cool conferences like the OOPSLA. I assured him, new applications could be build in days instead of years, just using objects ...

Well, as most of us know it did not turn out this way. Yes, we got a lot of advantages using OO technology but reuse turned out a lot harder than we thought. Sadly, I think the key cause of the lack of reuse was the violation of the two sacred rules of Structured Design: low coupling, high cohesion. It was very much like a couple of girls on the beach trying desperately not to expose themselves to the boys in front of them while changing into their swimming gear; not realizing that there were also some boys sitting behind them. We were so busy hiding our data in instance variables writing get and set methods that we never realized that we were exposing the object structure to anybody using our classes. That is, if you do new Foo() from class Bar, you just coupled the classes Foo and Bar for eternity. Trying to reuse Bar immediately drags in Foo. Now if Foo and Bar had been cohesive (meaning they are really parts of the same function) this would not be such a problem. However, because we were so busy writing get/set methods we mostly ended up with systems that had transitive dependencies that basically included all classes in your company's repository. If you want to see an example, write a "Hello world" program with an empty local Maven repository. Not only made this coupling reuse very hard, the coupling made making changes also very complicated because many changes were far reaching.

At the end of the previous millennium our industry learned many hacks to reduce this structural coupling; the most (in)famous is the Factory pattern. This pattern allowed you to manage the coupling outside the original program by moving the coupling to a runtime decision. However, it is a hack. OO languages have no built-in answer to this problem, just look at the large number of attempts one can find in Java to solve this coupling problem.

Also, if you look at one of the great insights of the last decade, Inversion of Control (IoC), you see that this is really about decoupling. IoC (or the Hollywood principle of "do not call us, we will call you") takes an object and provides it with its dependencies instead of this object trying to create its dependencies. This simple idea allows for POJOs, objects that are minimally coupled.

Services are trying to solve the same problem: minimize complexity by decoupling. The key insight that lead to the service model is that the OO has an important role in modularization on the small scale but we need something of a bigger granularity on the larger scale. A modularity that minimizes coupling to well defined conduits between coarse components. This was exactly the problem we tried to solve since that 1998 meeting in the Research Triangle Park in Raleigh where the technical OSGi work got started. Our key assignment was to let software from many different service providers collaborate in a single residential gateway. Trying to standardize large class libraries with highly coupled frameworks would have taken forever and it would have done a terrible job. By focusing on the coarse components (the bundles) that would collaborate through objects named by their interface that were found in a registry (the services) we created a model that had minimal, well defined, and documented coupling between bundles.

The key advantage of SOA is therefore the reduced coupling between the blocks of a system. The model became wide known with Webservices because Microsoft and IBM decided to put a lot of marketing dollars in that direction, making SOA almost synonymous with XML, SOAP, HTTP, some other acronyms, and associated distributed computing problems. It is not. Service Oriented Architectures reduces the complexity caused by not managing structural coupling by providing well defined conduits between components. Webservices achieve this using a myriad of acronyms, OSGi technology achieves this by using an extremely low overhead in-VM service publish/find/bind model.

So the key question is if the confusion between Webservices and OSGi is doing us harm? I think it does, many people fail to look further because they assume they understand. For example, I am flabbergasted that leading software magazines like ACM Communications and IEEE Software never had an in-depth article about OSGi technology because I strongly believe that the ramifications of this technology for our industry are profound.

Peter Kriens

Diversity is Good

A couple of weeks ago there was a discussion on the OSGi developer list about a support library for OSGi programmers that is intended to make the life of an OSGi programmer easier. This library was developed on, well lets say, framework E. A developer in Asia tried this library on framework K and found out that the library did not work on this framework. After a couple of messages back and forth the library developer basically said that it was probably a bug in framework K and that the best solution was to let the framework K developers figure out where the bug was.

I made a lot of noise on the list because I think we should develop against the OSGi specifications, not against a particular framework implementation. The fact that it runs on framework E does not mean by definition that it is good, even though I know that the developers of framework E are really good.

Now I have been in this industry for 30 years and I still have not learned to sit still and wait until everybody has forgotten the issue. This stupidity obviously resulted the problem ending up in my lap. Today I decided to run the faulty code on frameworks K, E, and F. This is a lot easier to write down than to do it, actually. Each framework has its own peculiar way of starting and it took me quite some time to get the whole setup running. To be able to switch quickly, I used the aQute File Install bundle. This is a simple bundle that watches a directory and installs/uninstall bundles depending on their presence in this folder. As a bonus, it can also manage configuration data in a similar vein. I then created Eclipse launchers for each of the frameworks.

After this hard labor, finding the bug was not too hard. It turned out there was a misinterpretation of the spec that should be easy to fix. However, it was almost a shame to see how much preparation I had to do to be able to run bundles on multiple frameworks. The setup I had created I thought was quite fancy, it was easy to test all kinds of bundles this way.

I therefore decided to use the altruism of Google and use some of their abundant disk space as a Google project. Wonder if all those Google investors know how their money is spent to support open source? Anyway, I created the multi-osgi project. Within Eclipse you can check this out with SVN (you need Subclipse, Subversive, or some other SVN plugin. The URL to the project is http://code.google.com/p/multi-osgi/source.

The project works quite simple. In Eclipse, you can just find an Equinox, Knopflerfish, and Felix launch in the Run/Debug Dialog. From a shell, you can just switch to the Felix, Knopflerfish, or Eclipse directory and do run (in DOS) or ./run.sh in Unix.

Once you got one or more frameworks running, you can easily install the bundles by just dropping them in the load directory. A good way to find bundles is to go to the OBR site. The input box allows you to type search criteria. For example, you want to install a shell. Just look for "Shell". This gives two bundles from the Apache Felix project. Just click on the download icon and save the file in the multi-osgi/load directory. You should be able to use this shell from the console. Equinox already starts up with a console, so you will get some interaction. Try it!

I will keep maintaining this project for the foreseeable future. However, I would welcome anybody that is willing to add other frameworks into the mix, or write scripts for special environments. Obviously, the committers of the different frameworks or commercial vendors are more then welcome.

Peter Kriens

Help Wanted

Last week I was in Boston at IONA's Waltham offices for an OSGi Enterprise Expert Group meeting. This was a very interesting meeting and well attended by OSGi members. It was pretty intensive because we had so many different subjects to cover. SCA, how to handle WARs and EARs, distribution, database access, making security usable, a new OSGi Component model, hyper-packages (super packages that cross bundles), JNDI, and much more. Looking at the actors around the table and the technical progress it seems clear that this work will have major impact on the enterprise software industry. All the key players were there: Oracle, BEA, IBM, IONA, Siemens, RedHat, and others.

During this meeting it suddenly hit me that we only had vendors around the table. I understand that vendors often take the lead in these efforts because it will enable them to see different products. However, lesson one in software development is that you'd like to have the customers included so that you can focus on the real needs and not the perceived needs. Fortune 500 corporations spend hundreds of millions each year on software development, where are they in standardization efforts? In the end, these are the actors that benefit most when the standards are solid. Look at how easy J2EE turns into a vendor lock-in model because the standard leaves crucial aspects undefined or fuzzy. Good solid specifications make the vendor switching cost low, forcing vendors to compete on quality and performance instead of maximum lock-in.

So how do we get more people from the trenches involved?

Peter Kriens

OSGi Bundle Repository Indexer Open Sourced

The OSGi Bundle Repository (OBR) is one of the most popular pages on the OSGi developer web site (www2.osgi.org), it receives almost as many hits as its main page. Surprising, because OBR is hardly advertised.

We developed OBR last year to provide a single place where OSGi developers can find bundles for their projects. One of the key drivers of the OSGi Alliance is to provide a universal middleware platform, reuse is therefore crucial. To reuse, one has to find appropriate components. OBR was developed to address this need.

How does it work technically? Well, the current OBR was derived from Richard Hall's Oscar Bundle Repository, and it is based on the same idea of a federated repository. A repository is a web server that allows access to the meta data and location of bundles. This information can be used to install bundles on a framework directly. For example, Richard developed a shell extension for Oscar/Felix that downloads a bundle from OBR, including its dependencies. Last year, I developed a web interface that allows for browsing the repository interactively.

At the bottom of OBR you find the repository.xml file. This file contains the meta data of the bundles and can link to other repositories as well. OBR is therefore a federated repository. This was an important aspect because we needed to make it as easy as possible for people to provide their own bundles to the repository. In the OBR model, any company or person can maintain its own repository and link it to others. This also enables repositories to be partly private on an Intranet while it still provides access to external repositories. In this model, the point where OBR is accessed can provide a trust model.

The information you find in the repository.xml is equivalent but not identical, to the information in the bundle. The reason it is not identical is that it has become more clear to us in the last few years that the OSGi technology needs a more generic dependency model. Over time, we have added more and more dependencies like Import-Package, Require-Bundle, Bundle-RequiredExecutionEnvironment, Bundle-NativeCode, etc. These dependencies had their own header and their own intricacies. Though this has some advantages, I am afraid that over time it will become too complex. For this reason, we used a generic model for OBR consisting of Capabilities and Requirements. This model resembles JSR 124 but we made it more powerful by making requirements OSGi filters instead of simple properties. Filter based requirements make it possible to create complex expressions and check magnitudes. For example, a requirement can be that there is a screen with a size of more than 100x100 pixels. JSR 124 can only handle exact matches.

An OSGi Environment, at a certain moment in time, has a set of capabilities. Each bundle (or hardware) requires a set of capabilities but can also provide a set of new capabilities when installed (and its requirements are met). This is a wonderfully elegant model. For example, if you have a mobile phone and insert a headset, the phone will be extended with a headset capability. This enables the installation of a bundle that requires a headset, which in its turn provides new capabilities. This model worked so well, that Richard Hall changed his framework resolver in Apache Felix (the program that wires bundles before they can be started) to work in this generic mode. Though we found a few nasty cases related to constraints that are currently not modeled, the idea performed very well.

This model is described in a document available from our website. Please note that this document does not mean that the OSGi Alliance will develop this concept further, we are testing the waters only!

Maintaining the repository.xml by hand is not an option. Eclipse alone provides over 1100 bundles. Fortunately, the OSGi specifications define lots of manifest headers that provide meta data. We therefore developed a program that reads a set of bundles and creates the repository.xml. Last month we got permission of the OSGi Alliance Board to open source this program. This makes it possible for organizations to include a bundle indexer. The sources are hosted on a Subversion server at http://www2.osgi.org/svn/public/trunk/org.osgi.impl.bundle.bindex/.

So how can we make OBR a success. Well, only if you help us do we have a chance. In my work I see so many companies that develop lots of bundles that are not strategic for their company. If we want to make the idea of a universal middleware work, we must share these non-strategic bundles with each others so that we do not get an umpteenth version of a serial port bundle.

If you have those bundles that you want to share, then setup your own repository. It is quite easy. Absolutely crucial is the meta data in the bundles. It is sad, but there are way to many bundles (and JARs) out there that have no name, no description, no license information, no nothing. It takes 5 minutes to add this information to a manifest and it will make bundles more reusable. So you have no excuse to skip this! Then if you have documented bundles, you just create a directory on your company's webserver and save the bundles there. Then once a day you run bindex on this directory. Bindex creates the repository.xml. The only thing left to do is the URL to me. I will link it into the main OSGi Bundle Repository. So, do not ask what bundle you can get from OBR, ask what you can mean for OBR!

Looking forward to seeing all your repositories,

Peter Kriens

Deja Vu

Obviously I am closely following the 277 list, though with pain in my heart that Sun does not allow me to post. So I can only use this blog to try to influence the discussions. If you follow the JSR 277 Modularization mailing list, you should have noticed the struggle that is taking place with initialization. The early draft specified a module initializer what would help link the module to other modules. However, the EG now also finds there is a need to initialize the user part of a module, a.k.a. the Bundle Activator. In this blog, let me take you back to a place long ago (well, only about 9 years).

Once upon a time there was a brilliant product from Sun called Java Embedded Server devised by Anselm Baird. JES was the predecessor of the OSGi specifications. Anselm got a lot of things right, but not everything. One of the things he had got wrong were the wizards. There was a DependenciesWizard, an InstallWizard, an UninstallWizard, etc. These wizards were identified by a class names and loaded from the bundle at the appropriate time, and given control. This is similar to the proposal in JSR 277 Modularization to let the module help in linking its dependencies, I recall this was the DependenciesWizard (Unfortunately, there seem to be no traces of JES left on the Internet, I only found some old sources from 1999 where I played with JES).

Going back through the mail discussions of that time, I was surprised how little I had learned in ten year because my opposition to these wizards was fierce. The primary reason was that these wizards were running in limbo. This is the place between heaven and hell where you go when things are a tad fuzzy around your soul. Classloader limbo is when any class you use can blow up because it depends on some other module not yet initialized. It is a bit like walking in the dark in an unfamiliar room, blindfolded, and your hands tied behind your back.

In January 1999, we had a meeting in Cupertino. Coming back to Ericsson I could report the following to Ericsson management:

The simplification of the wizards was quite successful: They were all removed. This was already proposed on the mail. We also decided to simplify the dependencies. Some elements of ServiceSpace required knowledge about the dependencies. This requirement made certain aspects of ServiceSpace rather inflexible. After these were removed a simple model remained where only one class reflects the Bundle for ServiceSpace, and this class is named in the manifest. In my opinion, these changes made the implementation of a ServiceSpace significantly simpler.

Even Anselm, after a lot of bashing from the other participants became convinced that a declarative approach was better. He proposed some of the API changes that became part of the OSGi:

Here's an API proposal to simplify the way wizards work in ServiceSpace. The essential motivation for that change is to move wizards from programmatic objects into declarative objects, as much as possible (the ultimate plan being to be able to provide tools that work on bundles).

My original drive for removing the wizards was to not have code run in an undefined environment. Looking back at Anselm's mail I see that he actually foresaw how important tools would become. Such tools can only work when the dependencies are declared. The problem to know what a wizard will do when you call it is an insoluable problem. So we succeeded to convince Sun in 1999, if we could only do the same in 2007 ...

Peter Kriens