Just a reminder, it is only two weeks to OSGi DevCon! We just had OSGi DevCon London and that was a great success. The OSGi DevCon London was organized by JAX and as always it was superbly organized in an excellent hotel. I always like it when the hotel and the conference are together, it increases the chance to get to talk to people. Tuesday night I did not get to my room until 1.30 AM. These on- and offline talks are crucially important to better understand where the OSGi eco system is moving to. Obviously it is interesting to hear what the usual suspects are thinking but these conferences also allow me to hear from the trenches. Sometimes you can correct invalid understandings but often you can learn a lot of the problems people face in software. This year's OSGi DevCon in Santa Clara will be held in the Hyatt hotel, I am already looking forward to have lots of discussions with people visiting the conference.
That said, one should not underestimate the program, we've got a very nice OSGi program this year. On Monday night we'll be introducing OSGi Enterprise R4.2. Tim Diekmann, the Enterprise Expert Group co-chair, will be introducing this seminal piece of work. We'll raffle the first copy of the Enterprise release book, signed by the co-chairs, might one day be a collectors item.
On Thursday we will also have a very interesting workshop on Cloud Computing that is already heavily oversubscribed, looks like a hot topic and I am expecting a very interesting day.
As the Michelin Guide would say: "Worth the trip!" So if you're not registered yet, do it now. Note the discount for OSGi members, just use your OSGi member mailing address.
Hoping to see many of you in Santa Clara!
Peter Kriens
Friday, February 26, 2010
Monday, February 8, 2010
OSGi & Cloud Computing
The Eclipse Foundation and the OSGi Alliance are holding a Cloud workshop during the OSGi DevCon/EclipseCon developer conference in Santa Clara, Thursday March 25.
They key question we want to answer in this workshop: what role can OSGi play in the cloud? Offerings like the ones from Amazon (aws.amazon.com) are agnostic of any application model and OSGi can play in their EC2 offering like anybody else because it is based on generic x86 machines. However, a model like the Google App Engine so severely knee-capped Java that it is doubt full that OSGi can run on it. Many cloud computing providers have free plans to get you started, or at least make the cost trivial. However, the costly part is your own investment in the software you develop for the cloud. On the desktop and on the server we've had a lot of advantage of standards that abstracted us from the vendors. This portability allows us to move our code to different app servers (well mostly). Though most of the lessons we learned in the past still apply to the cloud, the current vendors of cloud computing have very specific offerings that easily create portability problems. How to access the storage? How to discover and handle multiple instances of the application in the cloud? How to handle storage? How to share domain specific services? Standardizing interfaces for these aspects of cloud computing could provide a lot of portability. And portability is not only in the interest of the clients, also vendors gain by having a much larger market.
Perusing the different offerings for cloud computing I can clearly see that the OSGi bundle model would work very well in this area. Applications can easily be managed remotely because remote management is inside OSGi's genes. This always has made OSGi easy to use in clusters and much of those benefits apply equally to cloud computing. However, the advantages of the OSGi service model seems to be even more clear. A cloud computing environment is by definition a dynamic environment. Adding instances, removing instances, and instances that fail will likely influence the other instances. This means that the application will need to handle the dynamicity of the services that these computing instances provide. There will be also be dependencies that must be managed. OSGi services shine in these areas, making it relatively simple to correctly model these dynamic dependencies.
So overall the combination of cloud computing and OSGi is clearly an interesting one. With the workshop we want to bring together cloud people and OSGi people and see where there are areas where OSGi standards could help. This first workshop is by invitation only because for this first time we want to learn; we need people with experience in the area of cloud computing and that see OSGi as a potential standards player in this area; creating a discussion between cloud experts and OSGi experts. So if you're heavily into cloud computing and you want to attend, send me or Ian Skerret from the Eclipse Foundation a mail. Amazon? Google? Microsoft? You?
Peter Kriens
They key question we want to answer in this workshop: what role can OSGi play in the cloud? Offerings like the ones from Amazon (aws.amazon.com) are agnostic of any application model and OSGi can play in their EC2 offering like anybody else because it is based on generic x86 machines. However, a model like the Google App Engine so severely knee-capped Java that it is doubt full that OSGi can run on it. Many cloud computing providers have free plans to get you started, or at least make the cost trivial. However, the costly part is your own investment in the software you develop for the cloud. On the desktop and on the server we've had a lot of advantage of standards that abstracted us from the vendors. This portability allows us to move our code to different app servers (well mostly). Though most of the lessons we learned in the past still apply to the cloud, the current vendors of cloud computing have very specific offerings that easily create portability problems. How to access the storage? How to discover and handle multiple instances of the application in the cloud? How to handle storage? How to share domain specific services? Standardizing interfaces for these aspects of cloud computing could provide a lot of portability. And portability is not only in the interest of the clients, also vendors gain by having a much larger market.
Perusing the different offerings for cloud computing I can clearly see that the OSGi bundle model would work very well in this area. Applications can easily be managed remotely because remote management is inside OSGi's genes. This always has made OSGi easy to use in clusters and much of those benefits apply equally to cloud computing. However, the advantages of the OSGi service model seems to be even more clear. A cloud computing environment is by definition a dynamic environment. Adding instances, removing instances, and instances that fail will likely influence the other instances. This means that the application will need to handle the dynamicity of the services that these computing instances provide. There will be also be dependencies that must be managed. OSGi services shine in these areas, making it relatively simple to correctly model these dynamic dependencies.
So overall the combination of cloud computing and OSGi is clearly an interesting one. With the workshop we want to bring together cloud people and OSGi people and see where there are areas where OSGi standards could help. This first workshop is by invitation only because for this first time we want to learn; we need people with experience in the area of cloud computing and that see OSGi as a potential standards player in this area; creating a discussion between cloud experts and OSGi experts. So if you're heavily into cloud computing and you want to attend, send me or Ian Skerret from the Eclipse Foundation a mail. Amazon? Google? Microsoft? You?
Peter Kriens
Tuesday, February 2, 2010
OSGi DevCon 2010!
Time flies, it is more than 3 years ago that Bjorn Freeman-Benson, BJ Hargrave, and me sat down after the 2006 conference to discuss the possibilities to organize an OSGi DevCon in conjunction with EclipseCon. Today I am proud to announce the 4th OSGi DevCon in Santa Clara, March 22-25. The program is, as usual, staggering. It always impresses me how many people are willing to contribute to EclipseCon/OSGi DevCon. Overall there were more than 350 submissions and about 60 of those were for OSGi DevCon. Picking the most interesting program was even harder than previous years because there is less space; we therefore have less time for OSGi DevCon. However, the resulting program is probably of even higher quality.
First I would like to draw your attention to the fact that we will officially publish the OSGi Enterprise Specification during EclipseCon. The OSGi Alliance will host a BOF on Monday night. One of the co-chairs of the OSGi Enterprise Expert Group, Tim Diekmann, will give a presentation during this BOF of what is in this specification and why it is ground breaking.
We have three tutorials. The first tutorial is from the people that wrote the OSGi and Equinox: Creating Highly Modular Java Systems book. You will get a feel for Toast telematics! See Working with OSGi: The stuff you need to know.
The next tutorial is from Kirk Knoernschild and Neil Bartlett, both very experienced developers and excellent writers and presenters. This tutorial was actually chosen in the EclipseCon Program Commitee top 5. The subject is a very hot topic at the moment: modularity. We all learned the lessons about coupling and cohesion. However, applying those lessons in large developments is still hard. This tutorial will give you theoretical as well as practical insight in modularity and using OSGi to achieve it. See Modular Architecture from Top to Bottom.
The last tutorial is from Karl Pauls and Marcel Offermans. They are the lead developers of the Apache ACE project and have been developing with OSGi forever. Their subject is absolutely core for OSGi although not always that visible. OSGi is not a "Hello World" technology, such examples only work well when the scope is small. The scope of OSGi is, however, large scale technology. Size does matter for OSGi. A consequence of the scale is that systems have a large number of bundles. This number becomes so large that handling these bundles requires automation because it is just too much to do by hand. Karl and Marcel will teach how to manage installations that reach these problems. See Become a Certified Bundle Manager today.
The first long talk is a must for anyone using OSGi. One of the most exciting pieces of work inside the OSGi is the nested framework RFC. Nested frameworks bring back the initial philosophy of OSGi: the bundles are your application. Enterprise servers based on OSGi starting to deploy many applications inside a single framework. In such a constellation, your peer bundles and peer services might no longer be yours. Nested frameworks returns to this model, an application will be installed in a child framework, also called composite bundles. The lead developers of Eclipse Equinox as well as Apache Felix will present the proposed architecture and discuss merits, pitfalls, and problems that still need to be solved. So do not miss Composite Bundles - Isolating Applications in a Collaborative OSGi World.
OSGi is like a sharp knife. When used well, it is extremely useful, when used wrongly it hurts. Chris Aniszczyk, Jeff McAffer, Martin Lippert, and Paul Vanderlei have been working with OSGi for the better part of the noughties and therefore have lots of experiences and the bruises and cuts to prove it. Between them they cover almost any computing aspect that can be used in conjunction with OSGi. Jeff was the driver behind Eclipse's adoption of OSGi, Chris is the lead developer of PDE, Martin has worked on Aspect Oriented Programming in Eclipse including the weaving issues and is an aficionado of OSGi as well, and Paul brings the experience from the embedded world. A must for anybody that wants to adopt OSGi. See OSGi Best and Worst Practices.
OSGi is at the foundation of RCP, obviously. However, you can use RCP and not see much of OSGi. David Orme has been contracting for J.P. Morgan where they created an internal platform based on RCP. In the last few years they re-architected this platform to take more advantage of OSGi. This is a very good experience report for anybody that has to develop software to be used inside large organizations. See OneBench Reloaded - Pushing the (OSGI) Modularity Story in an Enterprise-wide Rich Client Stack.
Looking at the size of this blog, I do not think I should loose more readers going through each of the 25 mins talks, even though I think they're more than worth it. I therefore list them here as bullets:
I am looking forward to see you again in this 4th OSGi DevCon, lets hope it will be the best ever!
Peter Kriens
First I would like to draw your attention to the fact that we will officially publish the OSGi Enterprise Specification during EclipseCon. The OSGi Alliance will host a BOF on Monday night. One of the co-chairs of the OSGi Enterprise Expert Group, Tim Diekmann, will give a presentation during this BOF of what is in this specification and why it is ground breaking.
We have three tutorials. The first tutorial is from the people that wrote the OSGi and Equinox: Creating Highly Modular Java Systems book. You will get a feel for Toast telematics! See Working with OSGi: The stuff you need to know.
The next tutorial is from Kirk Knoernschild and Neil Bartlett, both very experienced developers and excellent writers and presenters. This tutorial was actually chosen in the EclipseCon Program Commitee top 5. The subject is a very hot topic at the moment: modularity. We all learned the lessons about coupling and cohesion. However, applying those lessons in large developments is still hard. This tutorial will give you theoretical as well as practical insight in modularity and using OSGi to achieve it. See Modular Architecture from Top to Bottom.
The last tutorial is from Karl Pauls and Marcel Offermans. They are the lead developers of the Apache ACE project and have been developing with OSGi forever. Their subject is absolutely core for OSGi although not always that visible. OSGi is not a "Hello World" technology, such examples only work well when the scope is small. The scope of OSGi is, however, large scale technology. Size does matter for OSGi. A consequence of the scale is that systems have a large number of bundles. This number becomes so large that handling these bundles requires automation because it is just too much to do by hand. Karl and Marcel will teach how to manage installations that reach these problems. See Become a Certified Bundle Manager today.
The first long talk is a must for anyone using OSGi. One of the most exciting pieces of work inside the OSGi is the nested framework RFC. Nested frameworks bring back the initial philosophy of OSGi: the bundles are your application. Enterprise servers based on OSGi starting to deploy many applications inside a single framework. In such a constellation, your peer bundles and peer services might no longer be yours. Nested frameworks returns to this model, an application will be installed in a child framework, also called composite bundles. The lead developers of Eclipse Equinox as well as Apache Felix will present the proposed architecture and discuss merits, pitfalls, and problems that still need to be solved. So do not miss Composite Bundles - Isolating Applications in a Collaborative OSGi World.
OSGi is like a sharp knife. When used well, it is extremely useful, when used wrongly it hurts. Chris Aniszczyk, Jeff McAffer, Martin Lippert, and Paul Vanderlei have been working with OSGi for the better part of the noughties and therefore have lots of experiences and the bruises and cuts to prove it. Between them they cover almost any computing aspect that can be used in conjunction with OSGi. Jeff was the driver behind Eclipse's adoption of OSGi, Chris is the lead developer of PDE, Martin has worked on Aspect Oriented Programming in Eclipse including the weaving issues and is an aficionado of OSGi as well, and Paul brings the experience from the embedded world. A must for anybody that wants to adopt OSGi. See OSGi Best and Worst Practices.
OSGi is at the foundation of RCP, obviously. However, you can use RCP and not see much of OSGi. David Orme has been contracting for J.P. Morgan where they created an internal platform based on RCP. In the last few years they re-architected this platform to take more advantage of OSGi. This is a very good experience report for anybody that has to develop software to be used inside large organizations. See OneBench Reloaded - Pushing the (OSGI) Modularity Story in an Enterprise-wide Rich Client Stack.
Looking at the size of this blog, I do not think I should loose more readers going through each of the 25 mins talks, even though I think they're more than worth it. I therefore list them here as bullets:
- Apache Aries: Enterprise OSGi in Action - A report from a new open source project that will bring us lots of enterprise components for OSGi. Graham Charters from IBM will present.
- My Unmanned System is Eclipse Powered - Next time you see an unmanned vehicle, OSGi might be behind the wheel. Talk about cool OSGi apps! Tankut Koray will show you the role OSGi plays in their architecture.
- Next Generation OSGi Shells - Traditionally shells run inside the OSGi framework, however, this shell works as launching tool, interacting with a Paremus' Nimble to find the necessary bundles. Robert Dunne will tell you about these shells and show you how easy it is to deploy applications consisting of many bundles.
- OSamI Tools for OSGi Application Developers - OSamI is a very large cross-european project to develop common technology for ambient intelligence, all based on OSGi. Naci Dai and Murat Yener from eteration A.S. will tell you more.
- Managing OBR Repositories with Nexus - Maven is moving to OSGi and there is more and more collaboration. Sonatype has adopted OBR in their Nexus repository, allowing it to play with the advanced resolvers that appearing in the market. Jason van Zyl, the man behind Maven, will tell you about their strategy.
- Using JPA in OSGi - Mike Keith and Timothy Ward are the lead authors of the OSGi JPA adaption, a part of the OSGi Enterprise Specification. See how you can simplify using persistence in OSGi bundles.
- OSGi Enterprise for Java EE Developers - How do you go from Java EE to OSGi? Many patterns that are necessary in Java EE do not work well in a very modular environment. Timothy DeBoer will show you how to use Eclipse tools to ease the transition.
- OSGi & Java EE in GlassFish - When Glassfish adopted OSGi a few years ago I was very excited to see how Java EE and OSGi can co-exist, each providing their strengths. Since then, the Glassfish team has more and more adopted OSGi, they even hired Richard Hall, the lead Apache Felix developer. Sahoo and Jerome Donchez are the lead architects and will report to you about the new cool features.
- Realistic Remote Management of OSGi-based Residential Boxes - OSGi was made to be managed remotely. However, managing thousands of devices running OSGi somewhere out there remains a complex area. Dimiar Valtchev from ProSyst has a very long experience with this problem and will elucidate you on the issues and solutions.
- Overcoming sticker shock: addressing the unexpected costs of moving to OSGi in the enterprise - Eric Johnson from TIBCO will explain you what you can expect when you move from a Java EE environment to OSGi, the rules and patterns that work are quite different. This will be an experience report but will also focus on how the community can work to ease this migration.
- Making Dependency Injection work for you - Joep Rottinghuis and Parag Raval from eBay tell you how to use Spring DM to use Dependency Injection in bundles.
- Logging in OSGi Enterprise Application - As a non-enterprise programmer I am always in awe when I see the avalanche of logging information coming out of enterprise programs. However, it seems important and OSGi puts some unique challenges in the way of traditional loggers because they often require global visibility and of course the OSGi Log Service. Ekkehard Gentz provides an overview and a demo of OSGi logging.
- ScalaModules: OSGi the Easy Way with a Scala DSL - The last months I've tried to use Scala because it has features I know from my Smalltalk days and daily miss when using Java. Though any new programing language is painfull to learn (what takes you seconds in Java initially takes you minutes in Scala because you have to figure out how), Scala really looks very interesting. Roman Roelofsen and Neil Bartlett will report to you about Scala Modules, a way to bring modularity to the Scala Language.
I am looking forward to see you again in this 4th OSGi DevCon, lets hope it will be the best ever!
Peter Kriens
Tuesday, January 26, 2010
Backward Compatibility
In our day to day work we often use the term backward compatible. We use this term as if it is a binary: something is backward compatible or it is not backward compatible. And yes, this is true if a client directly works with a provider. If the provider can work with clients that were compiled against a previous version then the provider can be said to be backward compatible with that previous version.
So is this always binary? Nope. The reason is the design by contract rule that we all follow, or should follow. In Java, we have this rule embodied in the interface, in C++ we used abstract classes. The primary advantage of design by contract is that now the client depends on the contract and the implementation depends on the contract but the client no longer depends on the implementation. Not only allows this model to have multiple implementations for the same contract, it also makes the dependencies smaller, more concise, and most important of all explicit. This model is depicted in the next figure.

In the OSGi service specifications clients and implementers are bundles. The contract is defined in a package that is imported by both the client and the implementer. The implementation is (normally) registered as a service under the interface defined in the contract package.
Instead of having two parties, where the backward compatibility was binary, we now have three parties making the situation a tad more convoluted. The compatibility is now expressed against the contract package because client and implementer have no longer any dependency on each other. What kind of changes can we make that do not affect the client? Well, these are the same changes we could make to the implementer in the simple case. Adding members to classes and adding fields to interfaces is harmless for clients of these classes and interfaces, they never know the difference. Even semantic changes are ok as long as we specify the evolution rules in the new contract.
However, the situation is different for an implementer of a contract; an implementer is semantically much closer bound to the contract than the client. A client compiled against version 1 of the contract can be bound to a backward compatible implementer that is bound to version 2 of the contract. However, a client compiled against version 2 of a contract must never be bound to a version 1 implementation because such an implementer has no knowledge of the changes in the contract and can therefore not faithfully implement it.
Interestingly, some of these incompatibility semantics show up in the way Java works. Implementers usually implement a number of Java interfaces; not implementing all the methods in such an interface will throw a No Such Method Error when called, clearly a violation of the new contract. In this article I talk about implementing the contract, however. There are many OSGi specifications where the client is also required to implement interfaces for callbacks but they are still considered clients. For example, in the Event Admin specification the client must implement the Event Listener service. These interfaces are called client interfaces and any change in them is incompatible for a client.
Using the contract model, we must take this asymmetric situation between clients and implementers into account when discussing backward compatibility. Almost any change in the contract will require the implementer to be aware of this change. However, there are a few cases where you can change the contract without requiring the implementer's awareness. We had such an instance in the upcoming enterprise release. In the previous release, the Blueprint API had no generics, in this release the generic signatures are added. Generics are erased in runtime, therefore existing Blueprint implementations cannot detect the difference in API and there are no additional responsibilities. Such a change is backward compatible for implementers.
I hope it is clear that backward compatibility has 2 dimensions: clients and implementers. When we make a change to the contract we must ask ourselves if this change is compatible with clients and implementers. Theoretically there are four cases, however, in practice any client backward incompatible change is very likely to be implementation incompatible as well, so there are only three cases left. The remaining question is now how to handle these three cases in OSGi. Obviously, the version attribute is the most applicable place to start.
The only party that knows about the change is the person changing contract. This person must somehow must convey its backward compatibility rules to the client and to the implementer. Surprisingly (well not really), these three cases map very well to the three parts of the OSGi version scheme:
Using OSGi version ranges, implementers can import all versions where the major and minor part is fixed and ignore micro changes. For example, when the package that is compiled against has version 2.3.6, then the implementer should import [2.3,2.4). Clients can import all versions where the major part is fixed. For example: [2.3,3). I call this model of importing different ranges based on the version that is compiled against the version policy. There is an implementation policy and a client policy.
This model works very well but it has one huge disadvantage: it requires that exporters follow the OSGi version semantics and not just the syntax. Unfortunately, we punted on the semantics when we had to specify the version attribute. We did recommend a good strategy but we did not mandate it nor was it complete. In practice, this means that people are not carefully versioning their packages (if at all!). It is always tempting to put the specification version on the package because this makes it clear which version of the specification you're getting when you have to select a package. However, this is the so called marketing version. Netscape Navigator came out as version 4.0 because it had to compete with Internet Explorer 3.0, there never was a version 3.0. In OSGi, we are currently at release 4.2 but if you look at the framework package version you'll find we're at 1.5.1, telling you it had 5 client backward compatible changes and since then one implementation backward compatible change. In contrast Wireadmin is still at 1.0. There are valid reasons for marketing versions but they unfortunately do not encode the evolution path the package has taken. It means that clients and implementers can no longer use a version policy to specify their compatibility requirements and must treat the version as an opaque identifier. The dire consequence of this model is that you basically have to rebuild all dependencies for any tiny change because clients and implementers can no longer reason about backward compatibility.
One solution that I proposed many years ago is to export a package under multiple versions. The exporter knows much more about its compatibility with prior versions, being able to specify the compatibility saves the importer from having to make assumptions. However, exporting a package under multiple versions only supports 2 cases for backward compatibility. If it is listed, it is backward compatible, if not, it is not compatible. As I hope this blog has demonstrated, treating backward compatibility as black and white is not sufficient.
I therefore hope it is clear that the exporter must provide different bits of information for the implementers and the clients. This could be a new version like attribute or it could use something like exporting three independent numbers:
The author of the contract package would maintain these numbers and incrementing them when the corresponding compatibility broke. This model seems to combine the best of both worlds. It exposes the different compatibilities without any required knowledge on the importer's side. However, my personal position is that the current version policy works today if people are willing to follow the rules. Anything else will require spec changes. The OSGi has been accurately versioning their packages correctly since 1998. The thousand dollar question is, will others follow these semantics?
Peter Kriens
P.S. In the past year I've done some experimenting with automatically generating import ranges based on the exported version in bnd and an implementation and client version policy. You can read about these experiments here.
So is this always binary? Nope. The reason is the design by contract rule that we all follow, or should follow. In Java, we have this rule embodied in the interface, in C++ we used abstract classes. The primary advantage of design by contract is that now the client depends on the contract and the implementation depends on the contract but the client no longer depends on the implementation. Not only allows this model to have multiple implementations for the same contract, it also makes the dependencies smaller, more concise, and most important of all explicit. This model is depicted in the next figure.
Instead of having two parties, where the backward compatibility was binary, we now have three parties making the situation a tad more convoluted. The compatibility is now expressed against the contract package because client and implementer have no longer any dependency on each other. What kind of changes can we make that do not affect the client? Well, these are the same changes we could make to the implementer in the simple case. Adding members to classes and adding fields to interfaces is harmless for clients of these classes and interfaces, they never know the difference. Even semantic changes are ok as long as we specify the evolution rules in the new contract.
However, the situation is different for an implementer of a contract; an implementer is semantically much closer bound to the contract than the client. A client compiled against version 1 of the contract can be bound to a backward compatible implementer that is bound to version 2 of the contract. However, a client compiled against version 2 of a contract must never be bound to a version 1 implementation because such an implementer has no knowledge of the changes in the contract and can therefore not faithfully implement it.
Interestingly, some of these incompatibility semantics show up in the way Java works. Implementers usually implement a number of Java interfaces; not implementing all the methods in such an interface will throw a No Such Method Error when called, clearly a violation of the new contract. In this article I talk about implementing the contract, however. There are many OSGi specifications where the client is also required to implement interfaces for callbacks but they are still considered clients. For example, in the Event Admin specification the client must implement the Event Listener service. These interfaces are called client interfaces and any change in them is incompatible for a client.
Using the contract model, we must take this asymmetric situation between clients and implementers into account when discussing backward compatibility. Almost any change in the contract will require the implementer to be aware of this change. However, there are a few cases where you can change the contract without requiring the implementer's awareness. We had such an instance in the upcoming enterprise release. In the previous release, the Blueprint API had no generics, in this release the generic signatures are added. Generics are erased in runtime, therefore existing Blueprint implementations cannot detect the difference in API and there are no additional responsibilities. Such a change is backward compatible for implementers.
I hope it is clear that backward compatibility has 2 dimensions: clients and implementers. When we make a change to the contract we must ask ourselves if this change is compatible with clients and implementers. Theoretically there are four cases, however, in practice any client backward incompatible change is very likely to be implementation incompatible as well, so there are only three cases left. The remaining question is now how to handle these three cases in OSGi. Obviously, the version attribute is the most applicable place to start.
The only party that knows about the change is the person changing contract. This person must somehow must convey its backward compatibility rules to the client and to the implementer. Surprisingly (well not really), these three cases map very well to the three parts of the OSGi version scheme:
- major change - Incompatible for implementers and clients
- minor change - Incompatible for implementers, compatible for clients.
- micro change - Compatible for implementers and clients
Using OSGi version ranges, implementers can import all versions where the major and minor part is fixed and ignore micro changes. For example, when the package that is compiled against has version 2.3.6, then the implementer should import [2.3,2.4). Clients can import all versions where the major part is fixed. For example: [2.3,3). I call this model of importing different ranges based on the version that is compiled against the version policy. There is an implementation policy and a client policy.
This model works very well but it has one huge disadvantage: it requires that exporters follow the OSGi version semantics and not just the syntax. Unfortunately, we punted on the semantics when we had to specify the version attribute. We did recommend a good strategy but we did not mandate it nor was it complete. In practice, this means that people are not carefully versioning their packages (if at all!). It is always tempting to put the specification version on the package because this makes it clear which version of the specification you're getting when you have to select a package. However, this is the so called marketing version. Netscape Navigator came out as version 4.0 because it had to compete with Internet Explorer 3.0, there never was a version 3.0. In OSGi, we are currently at release 4.2 but if you look at the framework package version you'll find we're at 1.5.1, telling you it had 5 client backward compatible changes and since then one implementation backward compatible change. In contrast Wireadmin is still at 1.0. There are valid reasons for marketing versions but they unfortunately do not encode the evolution path the package has taken. It means that clients and implementers can no longer use a version policy to specify their compatibility requirements and must treat the version as an opaque identifier. The dire consequence of this model is that you basically have to rebuild all dependencies for any tiny change because clients and implementers can no longer reason about backward compatibility.
One solution that I proposed many years ago is to export a package under multiple versions. The exporter knows much more about its compatibility with prior versions, being able to specify the compatibility saves the importer from having to make assumptions. However, exporting a package under multiple versions only supports 2 cases for backward compatibility. If it is listed, it is backward compatible, if not, it is not compatible. As I hope this blog has demonstrated, treating backward compatibility as black and white is not sufficient.
I therefore hope it is clear that the exporter must provide different bits of information for the implementers and the clients. This could be a new version like attribute or it could use something like exporting three independent numbers:
- An implementation compatibility number
- A client compatibility number
- A revision number
The author of the contract package would maintain these numbers and incrementing them when the corresponding compatibility broke. This model seems to combine the best of both worlds. It exposes the different compatibilities without any required knowledge on the importer's side. However, my personal position is that the current version policy works today if people are willing to follow the rules. Anything else will require spec changes. The OSGi has been accurately versioning their packages correctly since 1998. The thousand dollar question is, will others follow these semantics?
Peter Kriens
P.S. In the past year I've done some experimenting with automatically generating import ranges based on the exported version in bnd and an implementation and client version policy. You can read about these experiments here.
Wednesday, January 20, 2010
OSGi on the Gateway
The reason for this Berlin meeting is that is a flurry of activity around the home gateway. The HGI is looking at adding an execution environment, the Broadband Forum is working on a management model for execution environments on gateways, and the UPnP Forum has been working on managing an execution environment through UPnP as well (UPnP DM). And OSGi, well, we've been doing things in this area since the last millennium.
It is really good to have these different groups around the table. It turns out our maturity is a huge advantage. Though all the planned execution environment plans on the table clearly do not restrict themselves to OSGi, OSGi is in all cases a primary candidate. This seems to work out very well for all parties. OSGi is quite mature and is based on a lot of real world experience. This is a good test case for the different models and it of course ensures that the intricacies on OSGi are well supported.
It is a good sign that these different organizations are collaborating this much, lots of energy in the room!
Peter Kriens
Sunday, January 17, 2010
Nested Frameworks
I do not think we had application servers in mind when we had our first OSGi discussions in 1998. On the contrary, the OSGi service platform was the application; the idea of running multiple applications inside the same OSGi Service Platform did not cross our minds. Would somebody have explained the concept at that time, it would probably have been considered an anathema to us. Bundles were not meant to be applications, they were meant to be the constituents; the application was what appeared when you ran different bundles together in a service platform.
History played an interesting trick with this model. The class loading model of OSGi was more or less an afterthought, necessary to support the service oriented programming model. This detail caught the eye of many developers struggling with the class loaders in Java. However, instead of seeing a collaborative programming model, most saw class loaders on steroids. With release 4 we provided the zenith of all class loaders: allowing multiple versions to reside in the same framework. This was by many seen as the grail that solved the pesky problem of using multiple libraries that had conflicting version requirements. Though OSGi indeed could solve this problem better than anyone else it is illustrative that multiple versions was completely unnecessary for the original OSGi service oriented programming model. Actually, Richard Hall and BJ Hargrave had to explain the paradoxically sounding idea of exporting implementation classes many times to me.
With release 4, OSGi became salon fähig for the Application Server space. JONAS was the first but over time BEA WebLogic, IBM Websphere, JBoss followed. Even SAP and Oracle promised to follow this lead. The advantages of OSGi on this level are crystal clear but having the OSGi framework so close to the applications creates the temptation to allow applications to also use OSGi. Many applications are bursting out of their WAR format seams. The modularization provided by OSGi seems very attractive to remove the duplication that is embedded in the WAR architecture. It was Spring Source that made this module first available for enterprise developers with their Spring Source server. This lead is now followed by Oracle, IBM, and JBoss in a diverse set of open source projects.

The first automobiles in 1889 looked very much like the technology they replaced: horse carts. Over time the unique capabilities and requirements of an internal combustion engine resulted in the designs we see today. Is there an analogy with Application Servers and OSGi? Is OSGi just like an Application Server's programming model or should it be treated differently?
Though the OSGi indeed has class loaders on steroids, these class loaders can only alleviate the dependency problems caused by today's application models. I do not believe that the Application Server model where a number of WAR files are running in the same VM should be followed by an OSGi model where these WAR files are running in an OSGi framework only sharing their dependencies. Though it is good to manage a dependency (as OSGi does), it is best to not have a dependency in the first place. It is good to handle multiple versions of the same library but the problems caused by multiple versions quickly cascades to umanageable proportions.
I therefore believe that the original model promoted by OSGi, where the application emerges from the installed bundles, is still by far the best model we have. In this model bundles adapt to each other and collaborate with each other through services; exporting implementation classes is an anathema in this model. Any services in the service registry are available to all bundles because the framework is the application.
This model raises the question how to handle multiple independent applications. Do they all require their own framework? The answer is an unequivocal yes! The cost of this model with the current OSGi frameworks could be quite high. Fortunately, the OSGi CPEG is currently working on nested frameworks. Nested frameworks allow the creation of a composite bundle. This is a bundle that acts as a wormhole to another framework. Nested frameworks will have low overhead and can easily share services and packages among each other.
Nested frameworks will likely appear in a future OSGi specification. It will definitely take time to work out all the special cases but it looks very promising so far. Thomas Watson from IBM has been experimenting and the latest versions of Equinox already provide some of the behavior in experimental form. For me, this is the most exciting RFC in progress. The nested model will enable an application programming model very close to the original OSGi service oriented programming model we envisioned a decade ago. However, with this simple recursive rule of nested frameworks we do not limit it to small embedded devices but enable the same model for (very) large enterprise applications running in large application servers as well.
Peter Kriens
History played an interesting trick with this model. The class loading model of OSGi was more or less an afterthought, necessary to support the service oriented programming model. This detail caught the eye of many developers struggling with the class loaders in Java. However, instead of seeing a collaborative programming model, most saw class loaders on steroids. With release 4 we provided the zenith of all class loaders: allowing multiple versions to reside in the same framework. This was by many seen as the grail that solved the pesky problem of using multiple libraries that had conflicting version requirements. Though OSGi indeed could solve this problem better than anyone else it is illustrative that multiple versions was completely unnecessary for the original OSGi service oriented programming model. Actually, Richard Hall and BJ Hargrave had to explain the paradoxically sounding idea of exporting implementation classes many times to me.
With release 4, OSGi became salon fähig for the Application Server space. JONAS was the first but over time BEA WebLogic, IBM Websphere, JBoss followed. Even SAP and Oracle promised to follow this lead. The advantages of OSGi on this level are crystal clear but having the OSGi framework so close to the applications creates the temptation to allow applications to also use OSGi. Many applications are bursting out of their WAR format seams. The modularization provided by OSGi seems very attractive to remove the duplication that is embedded in the WAR architecture. It was Spring Source that made this module first available for enterprise developers with their Spring Source server. This lead is now followed by Oracle, IBM, and JBoss in a diverse set of open source projects.

Though the OSGi indeed has class loaders on steroids, these class loaders can only alleviate the dependency problems caused by today's application models. I do not believe that the Application Server model where a number of WAR files are running in the same VM should be followed by an OSGi model where these WAR files are running in an OSGi framework only sharing their dependencies. Though it is good to manage a dependency (as OSGi does), it is best to not have a dependency in the first place. It is good to handle multiple versions of the same library but the problems caused by multiple versions quickly cascades to umanageable proportions.
I therefore believe that the original model promoted by OSGi, where the application emerges from the installed bundles, is still by far the best model we have. In this model bundles adapt to each other and collaborate with each other through services; exporting implementation classes is an anathema in this model. Any services in the service registry are available to all bundles because the framework is the application.
This model raises the question how to handle multiple independent applications. Do they all require their own framework? The answer is an unequivocal yes! The cost of this model with the current OSGi frameworks could be quite high. Fortunately, the OSGi CPEG is currently working on nested frameworks. Nested frameworks allow the creation of a composite bundle. This is a bundle that acts as a wormhole to another framework. Nested frameworks will have low overhead and can easily share services and packages among each other.
Nested frameworks will likely appear in a future OSGi specification. It will definitely take time to work out all the special cases but it looks very promising so far. Thomas Watson from IBM has been experimenting and the latest versions of Equinox already provide some of the behavior in experimental form. For me, this is the most exciting RFC in progress. The nested model will enable an application programming model very close to the original OSGi service oriented programming model we envisioned a decade ago. However, with this simple recursive rule of nested frameworks we do not limit it to small embedded devices but enable the same model for (very) large enterprise applications running in large application servers as well.
Peter Kriens
Wednesday, January 13, 2010
Enterprise Specifications
I am currently in Southampton where we are finalizing the specifications for the 4.2 Enterprise release in an EEG meeting. It looks like we will be able to actually release this work during EclipseCon in Santa Clara, March 22-26. It is a sizable body of work, again. The last 8 weeks have been spent editing, editing, editing, and editing for 10-12 hours a day. Yesterday we did an (almost) final review so a few more days and then I am going to take some time off, trying to get my shoulder muscles back in a state where they can move again.
So what is likely going to be in that specification? Though I cannot give any hard guarantees, the following paragraphs give an overview of the intended content.
We had a few errata in Blueprint and some other specs, but all very minor. So most new specifications for the Enterprise release are about Java EE APIs and make them usable in OSGi. These specifications work very hard to preserve the semantics of the Java EE APIs but map them to OSGi services and defines life cycle issues. At the same time, we're trying very hard to be as compatible as possible with existing JEE code.
The hardest problem being that Java has nothing like a service registry; unfortunately most Java EE specs use the factory pattern with class loading tricks in slightly different ways. Working on the OSGi specifications has taught me a lot of things I never wanted to know about these Java EE specs.
The most "Java EEish" spec in this aspect is I guess JNDI. JNDI is a registry that has become part of Java SE. It is a surprisingly complex design for a rather simple problem (registry). This was the first design ever where I ran into a factory of factories, and JNDI even has two: InitialContextFactoryBuilder and ObjectFactoryBuilder. JNDI had to be supported because a lot of Java EE code is using JNDI to link to other subsystems and get configuration information. Key problem was that people tend to use new InitialContext(). The constructor of the InitialContext class then has to work surprisingly hard to find some Context implementation that was intended by the caller or the container.
The Initial Context design is a variation of a static factory pattern, something that does not work well with OSGi because of the dynamic life cycle. Static factories ignore dependencies, putting a huge burden on the deployer to get the ordering right. We did map the different factories to OSGi services and allow the usage of the new InitialContext(). If JNDI is really needed then depending on the JNDI Context Manager service will keep deployments simple because dependencies are managed.
For JDBC we created a Data Source Factory service that can be registered by database drivers. This service can then provide a surprisingly large number of Data Source variations: Data Source, Pooled Connection, XA Data Source, etc.
The OSGi JPA specification leverages this Data Source Factory service. JPA has become a very popular specification very quickly. The core access point for this specification the Entity Manager Factory
Transactions have been added by registering the Java Transaction Architecture (JTA) interfaces as services and allowing them to be used by applications and containers. The enlistment strategy of the resources is up to the container or by the applications themselves.
The WAR specification introduces WABs: Web Application Bundles. These bundles have the same purpose as WARs: serving web content. Part of the specification is a URL Handler that can install a WAR; the URL Handler will convert a WAR to a WAB by adding the appropriate manifest headers for OSGi. This will likely be a very popular way to get started with OSGi for developers that have a lot of experience with WARs.
This release will also support JMX (Java Management Extension). We've defined a number of MBeans that allow management of the OSGi framework and a number of the key services.
Last but definitely not least. My favorite specification is Remote Service Admin. In the previous release we specified a number of properties that enable the distribution of OSGi services. Services are very well suited to distribution; not because they were designed for distribution, but because dynamics are an intrinsic part of their design. The Remote Service Admin specification uses the distribution properties and allows a topology manager (another bundle) to manage a distributed topology of services. Topology managers can be simple like managing a number of key services in a cluster or implement fail-over or load balancing. Included in the specification is an abstraction to discover services from many different sources. This specification is leveraging OSGi to the hilt and by doing so it enables applications to be largely unaware of their distribution. This specification might be the "killer spec" for OSGi.
A key concept in OSGi remote services is configuration types. A configuration type specifies how to configure the remoting of a service to create a communications endpoint. For example, you could have a configuration type for web services. In the coming release, you will likely find an SCA configuration type. SCA being the Service Component Architecture, a specification at OASIS. SCA is a comprehensive specification to create distributed applications in many different technology. The SCA configuration type allows OSGi frameworks to play a role in SCA.
So lots of new stuff! I really hope to see a lot of people at EclipseCon when this specification will be introduced.
Peter Kriens
So what is likely going to be in that specification? Though I cannot give any hard guarantees, the following paragraphs give an overview of the intended content.
We had a few errata in Blueprint and some other specs, but all very minor. So most new specifications for the Enterprise release are about Java EE APIs and make them usable in OSGi. These specifications work very hard to preserve the semantics of the Java EE APIs but map them to OSGi services and defines life cycle issues. At the same time, we're trying very hard to be as compatible as possible with existing JEE code.
The hardest problem being that Java has nothing like a service registry; unfortunately most Java EE specs use the factory pattern with class loading tricks in slightly different ways. Working on the OSGi specifications has taught me a lot of things I never wanted to know about these Java EE specs.
The most "Java EEish" spec in this aspect is I guess JNDI. JNDI is a registry that has become part of Java SE. It is a surprisingly complex design for a rather simple problem (registry). This was the first design ever where I ran into a factory of factories, and JNDI even has two: InitialContextFactoryBuilder and ObjectFactoryBuilder. JNDI had to be supported because a lot of Java EE code is using JNDI to link to other subsystems and get configuration information. Key problem was that people tend to use new InitialContext(). The constructor of the InitialContext class then has to work surprisingly hard to find some Context implementation that was intended by the caller or the container.
The Initial Context design is a variation of a static factory pattern, something that does not work well with OSGi because of the dynamic life cycle. Static factories ignore dependencies, putting a huge burden on the deployer to get the ordering right. We did map the different factories to OSGi services and allow the usage of the new InitialContext(). If JNDI is really needed then depending on the JNDI Context Manager service will keep deployments simple because dependencies are managed.
For JDBC we created a Data Source Factory service that can be registered by database drivers. This service can then provide a surprisingly large number of Data Source variations: Data Source, Pooled Connection, XA Data Source, etc.
The OSGi JPA specification leverages this Data Source Factory service. JPA has become a very popular specification very quickly. The core access point for this specification the Entity Manager Factory
Transactions have been added by registering the Java Transaction Architecture (JTA) interfaces as services and allowing them to be used by applications and containers. The enlistment strategy of the resources is up to the container or by the applications themselves.
The WAR specification introduces WABs: Web Application Bundles. These bundles have the same purpose as WARs: serving web content. Part of the specification is a URL Handler that can install a WAR; the URL Handler will convert a WAR to a WAB by adding the appropriate manifest headers for OSGi. This will likely be a very popular way to get started with OSGi for developers that have a lot of experience with WARs.
This release will also support JMX (Java Management Extension). We've defined a number of MBeans that allow management of the OSGi framework and a number of the key services.
Last but definitely not least. My favorite specification is Remote Service Admin. In the previous release we specified a number of properties that enable the distribution of OSGi services. Services are very well suited to distribution; not because they were designed for distribution, but because dynamics are an intrinsic part of their design. The Remote Service Admin specification uses the distribution properties and allows a topology manager (another bundle) to manage a distributed topology of services. Topology managers can be simple like managing a number of key services in a cluster or implement fail-over or load balancing. Included in the specification is an abstraction to discover services from many different sources. This specification is leveraging OSGi to the hilt and by doing so it enables applications to be largely unaware of their distribution. This specification might be the "killer spec" for OSGi.
A key concept in OSGi remote services is configuration types. A configuration type specifies how to configure the remoting of a service to create a communications endpoint. For example, you could have a configuration type for web services. In the coming release, you will likely find an SCA configuration type. SCA being the Service Component Architecture, a specification at OASIS. SCA is a comprehensive specification to create distributed applications in many different technology. The SCA configuration type allows OSGi frameworks to play a role in SCA.
So lots of new stuff! I really hope to see a lot of people at EclipseCon when this specification will be introduced.
Peter Kriens
Subscribe to:
Posts (Atom)
