It took sometime but yesterday it was official: Sun has hired Richard Hall in the Glassfish team! This is good news for Richard, who was looking for just a position like that, Sun, who can clearly use the expertise on OSGi that Richard as one of the foremost OSGi experts provides, Apache Felix because this means that Richard will be able to spend more time on this project, and also good news for the OSGi Alliance that will have one of its key expert group members represent Sun. Rarely is a change positive in so many ways.
I look forward to continue working with Richard in his new role. I hope that Richard's role will mean that Glassfish will become one of the key players in the next OSGi release, they clearly have the experience to match. Though it will be interesting to find out what happens in JSR 277 because Richard is currently an independent member. Will he continue to participate (though participate is a big word with the activity in the last year) or will he be able to actively work in that group (and get the long overdue EDR2 out)? That is still a question mark.
I wish Sun and Richard a very happy and fruitful cooperation and hope to be able to work with Richard for a long time.
Peter Kriens
OSGi Blog
Monday, August 4, 2008
Wednesday, July 30, 2008
Thursday, July 17, 2008
OSGi Application Grouping
Several people in and outside the OSGi want application grouping. They want to be able to deploy a set of bundles as an "application". These "applications" should have some scoping on class loading and service issues. I am opposed to this but, as Eric Newcomer pointed out to me last week in Dublin, I never really made my case and shown a solution to some of the valid requirements. So, here it is ...
Back to basics. In the beginning, OSGi was about providing functionality to a device. Adding bundles would add more functions and removing bundles would get rid of those functions. Key was that these bundles would collaborate, they would not just be started and running in some silo container; ignorant of their siblings around them. However, these bundles would not have a priori knowledge of each other because the business idea was that different service providers would supply them independently.
The only way you can make this work is if you have a communication model that allows you to very loosely couple the bundles. This was the service model. The service model is very simple: the producer registers some object under an interface and the consumer gets it through the interface. The implementation is safely hidden, the consumer has no clue. Unfortunately, this service model required rules about class loading because those service interfaces involve classes and you better make sure that the producer and the consumer both share the same class loader for that interface/class. It was never the original intention to share implementation classes between bundles because bundles were supposed to communicate through services with a minimal interface to promote loose coupling. Sharing implementation classes was more or less an unintended side effect. Prior to R4 we could not even support multiple versions of the same exported package because that need is not very acute in a service oriented world program.
In the meantime at plain Java, the poor man's extension model became Class.forName. Where the OSGi model offers the services to the consumer, in the poor man's model the consumer creates the services through dynamic class loading, albeit with the indirection of some XML or other text file. This, alas, requires implementation visibility into the provider bundle.
The popularity of this Class.forName extension model makes the OSGi class loaders on steroids the most attractive aspect of OSGi for many people. However, having to export implementation classes from a bundle obviously opens huge holes in the modularity story, suddenly you are exporting implementation classes to the world. This makes the coupling a lot less loose.
My personal feeling is that application models are only trying to patch up these modularity holes with an emergency bandage. This is like drinking heavily and then taking aspirine against the headache. Maybe less drinking would be healthier?
This said, there is of course a concept of applicationness, otherwise not so many people would demand it. However, I do feel that the bundle is the right granularity for application granularity. A good bundle provides one or more functions and expresses dependencies on other bundles through services. That is all, no other primitives are needed (the services imply one or more Java packages). If I want to run an application I pick a bundle that provides me with the right functionality. Systems like OBR can then be used to find bundles that provide the needed services.
Dynamic dependency resolving is needed to create flexible systems. One of the key problems with an application model is that it fixes the application to a specific set of bundles; making it applicable to only one environment and thereby limiting severely the overall reusability.
One of the key issues is of course testing. Many companies want to be able to send a set of bundles to QA and be sure that that is exactly the set that will be deployed on the target system. I do understand this issue. However, I think that the best solution to this requirement is to run these bundles in a framework, or even a VM, of their own. In certain cases, it is even wise to just wrap up all the bundles into one so you are sure they can not be separated. From a modularity perspective, if these bundles are so closely coupled together, why separate them?
Fortunately, the EGs have come up with a model that seems to fit the needs to scope a set of bundles: nested frameworks. They are currently investigating a solution where you can create new frameworks inside an existing framework with relatively little cost. This solution seems to be very unintrusive to the spec because it adds a feature but does not (I hope) influence the existing features, at least not in a major way. Nested framework allow a managing framework to create nested frameworks for each "application". Services can be shared between frameworks by using the existing notification mechanisms and registering the service in alternative frameworks. Even packages could be shared with the
To conclude. I think the demand for an application model is largely driven by problems that are caused by the Class.forName extension model and tight coupling that caused developers to share implementation classes. Even after ten years I think that the bundle-service model of OSGi is the cleanest solutions to software development that I know. I do hope that we will not dilute this model by patching OSGi to support the lesser implementation class sharing model. Then again, with nested frameworks we all can seem to get what we want!
Peter Kriens
Back to basics. In the beginning, OSGi was about providing functionality to a device. Adding bundles would add more functions and removing bundles would get rid of those functions. Key was that these bundles would collaborate, they would not just be started and running in some silo container; ignorant of their siblings around them. However, these bundles would not have a priori knowledge of each other because the business idea was that different service providers would supply them independently.
The only way you can make this work is if you have a communication model that allows you to very loosely couple the bundles. This was the service model. The service model is very simple: the producer registers some object under an interface and the consumer gets it through the interface. The implementation is safely hidden, the consumer has no clue. Unfortunately, this service model required rules about class loading because those service interfaces involve classes and you better make sure that the producer and the consumer both share the same class loader for that interface/class. It was never the original intention to share implementation classes between bundles because bundles were supposed to communicate through services with a minimal interface to promote loose coupling. Sharing implementation classes was more or less an unintended side effect. Prior to R4 we could not even support multiple versions of the same exported package because that need is not very acute in a service oriented world program.
In the meantime at plain Java, the poor man's extension model became Class.forName. Where the OSGi model offers the services to the consumer, in the poor man's model the consumer creates the services through dynamic class loading, albeit with the indirection of some XML or other text file. This, alas, requires implementation visibility into the provider bundle.
The popularity of this Class.forName extension model makes the OSGi class loaders on steroids the most attractive aspect of OSGi for many people. However, having to export implementation classes from a bundle obviously opens huge holes in the modularity story, suddenly you are exporting implementation classes to the world. This makes the coupling a lot less loose.
My personal feeling is that application models are only trying to patch up these modularity holes with an emergency bandage. This is like drinking heavily and then taking aspirine against the headache. Maybe less drinking would be healthier?
This said, there is of course a concept of applicationness, otherwise not so many people would demand it. However, I do feel that the bundle is the right granularity for application granularity. A good bundle provides one or more functions and expresses dependencies on other bundles through services. That is all, no other primitives are needed (the services imply one or more Java packages). If I want to run an application I pick a bundle that provides me with the right functionality. Systems like OBR can then be used to find bundles that provide the needed services.
Dynamic dependency resolving is needed to create flexible systems. One of the key problems with an application model is that it fixes the application to a specific set of bundles; making it applicable to only one environment and thereby limiting severely the overall reusability.
One of the key issues is of course testing. Many companies want to be able to send a set of bundles to QA and be sure that that is exactly the set that will be deployed on the target system. I do understand this issue. However, I think that the best solution to this requirement is to run these bundles in a framework, or even a VM, of their own. In certain cases, it is even wise to just wrap up all the bundles into one so you are sure they can not be separated. From a modularity perspective, if these bundles are so closely coupled together, why separate them?
Fortunately, the EGs have come up with a model that seems to fit the needs to scope a set of bundles: nested frameworks. They are currently investigating a solution where you can create new frameworks inside an existing framework with relatively little cost. This solution seems to be very unintrusive to the spec because it adds a feature but does not (I hope) influence the existing features, at least not in a major way. Nested framework allow a managing framework to create nested frameworks for each "application". Services can be shared between frameworks by using the existing notification mechanisms and registering the service in alternative frameworks. Even packages could be shared with the
To conclude. I think the demand for an application model is largely driven by problems that are caused by the Class.forName extension model and tight coupling that caused developers to share implementation classes. Even after ten years I think that the bundle-service model of OSGi is the cleanest solutions to software development that I know. I do hope that we will not dilute this model by patching OSGi to support the lesser implementation class sharing model. Then again, with nested frameworks we all can seem to get what we want!
Peter Kriens
Wednesday, July 2, 2008
QVO VADIMVS?
We are seeing more and more outlines appearing for the next OSGi release. One of the major issues is legacy code. Not only inside the OSGi, but if you go to the web you see a lot of people struggling to get old code to work inside OSGi frameworks. Obviously, we want to mitigate the issues around legacy code as much as possible, the more people that use OSGi the better. However, lately I have some (personal, this absolutely is not an OSGi standpoint!) musings about how to attack the issue of legacy code.
A short story to illustrate my musings. In the eighties, I worked on a page-makeup terminal for the newspaper industry. Petr van Blokland, a graphic designer turned computer specialist, introduced me to the layout grid. This grid had columns and between the columns there was a small gutter. Text and pictures were placed on this grid, usually encompassing multiple columns and gutters. Like:
The OSGi always reminds me of this grid. Why? Because they both restrict you severely but in return they provide simplicity. Instead of having infinite freedom to do whatever you feel like, you must obey some pretty basic rules, which some people find upsetting. But what you get back is that the elements work together as a whole, instead of fighting with each other.
Layouts done with this grid almost invariably look good with no effort (try working with the average layout manager in Swing or SWT!). The advantage is that elements always line up and there is always the same space between elements. Without a grid, it is very hard to avoid unwanted visual effects.
Genuine OSGi bundles almost invariably collaborate with each other without much effort (anybody saw the combination Eclipse and Spring coming?) because modules are self-contained and can only export packages and communicate via services instead of the myriad of ways people have devised in Java.
Interestingly, both are achieved by restricting ones freedom, the opposite of providing more features. But neither OSGi nor this grid is simplistic. A simplistic grid would be a square 8x8 grid, and they just do not work. A simplistic OSGi would be some Class.forName based system without handling versions and dependencies. Both OSGi and the grid seem to be in a sweet spot: simple but not simplistic, providing maximum bang for the buck.
However, legacy code seems to be forcing us to add more and more mechanisms to the OSGi specification. Unfortunately, these mechanisms are often also then used for new OSGi applications because the legacy concepts they represent feel familiar to people. See how many people use Require-Bundle and fragments.
If we add all these freedoms to the next generation, will we not pollute the original model and become in the end much less attractive? Or, if we do not make it easier to use legacy code, will people turn away because they feel affronted that their direct needs are not addressed? Should we leave these issues to framework implementations making legacy code not really portable?
The current popularity of OSGi seems to allow the OSGi to make a stand. What do you think?
Peter Kriens
A short story to illustrate my musings. In the eighties, I worked on a page-makeup terminal for the newspaper industry. Petr van Blokland, a graphic designer turned computer specialist, introduced me to the layout grid. This grid had columns and between the columns there was a small gutter. Text and pictures were placed on this grid, usually encompassing multiple columns and gutters. Like:
The OSGi always reminds me of this grid. Why? Because they both restrict you severely but in return they provide simplicity. Instead of having infinite freedom to do whatever you feel like, you must obey some pretty basic rules, which some people find upsetting. But what you get back is that the elements work together as a whole, instead of fighting with each other.
Layouts done with this grid almost invariably look good with no effort (try working with the average layout manager in Swing or SWT!). The advantage is that elements always line up and there is always the same space between elements. Without a grid, it is very hard to avoid unwanted visual effects.
Genuine OSGi bundles almost invariably collaborate with each other without much effort (anybody saw the combination Eclipse and Spring coming?) because modules are self-contained and can only export packages and communicate via services instead of the myriad of ways people have devised in Java.
Interestingly, both are achieved by restricting ones freedom, the opposite of providing more features. But neither OSGi nor this grid is simplistic. A simplistic grid would be a square 8x8 grid, and they just do not work. A simplistic OSGi would be some Class.forName based system without handling versions and dependencies. Both OSGi and the grid seem to be in a sweet spot: simple but not simplistic, providing maximum bang for the buck.
However, legacy code seems to be forcing us to add more and more mechanisms to the OSGi specification. Unfortunately, these mechanisms are often also then used for new OSGi applications because the legacy concepts they represent feel familiar to people. See how many people use Require-Bundle and fragments.
If we add all these freedoms to the next generation, will we not pollute the original model and become in the end much less attractive? Or, if we do not make it easier to use legacy code, will people turn away because they feel affronted that their direct needs are not addressed? Should we leave these issues to framework implementations making legacy code not really portable?
The current popularity of OSGi seems to allow the OSGi to make a stand. What do you think?
Peter Kriens
Tuesday, June 17, 2008
JSR 277 and Import-Package
It looks like Sun is not convinced that Import-Package is a good idea. On the JSR 277 mailing list there was an interesting discussion about the dependency model: Does Import-Package have value over Require-Bundle (to use the OSGi terms)?Require-Bundle has an intuitive appeal that is hard to deny. These are the artifacts on the class path during compile time and there is a clear advantage to have these same artifacts during deployment. Import-Package is cumbersome to maintain by hand and basically requires tools like bnd. Besides, the whole world seems to use the Require-Bundle model (maven, ivy, netbeans, ...). So, who would ever want to use Import-Package?
Well, have you ever had chewing gum in your hair?The harder you try to remove it, the more entangled it gets? This is what Require-Bundle gives you. Import-Package is more like Play-Doh. It is still better not to have it in your hair, but you can ask any kindergarten teacher what she prefers!
So, now we have the right mental image, what is the technical reason for this stickiness? I see two reasons:
- Require-Bundle gives you much more than you actually use because your class files just do not refer to it, a key advantage of a static typed language. Now before you jump to the conclusion you do not care because it is free, realize that there is rent to pay for these unused parts. These unused parts have their own dependencies that need to be satisfied; so they've has just become your dependencies. Visualize the chewing gum in that strand of hair? Dependencies are annoying, and unnecessary dependencies are worse, but these kinds of tied dependencies can be a death knell if they turn out not to be compatible with yours when your code is combined with other bundles to create a system.
- You depend on the whims of the bundle's author. He usually has no clue what parts of his bundle are used and is already busy working on the next version. He will make decisions based on his needs, not yours. It is therefore likely that the constituents in the bundle will change over time. How can you be sure that this migration happens in a way that is compatible with your bundles? Not only can the next version bring quite unnecessary and unexpected new dependencies, it might actually remove the actual packages you depend on. Unfortunately, your bundle resolves because the bundle you require is present, but its changes were just not compatible with your expectations so you get a Class Not Found Exception in the middle of shutting down this nuclear reactor.
The underlying reason for this problem is lack of cohesion. Cohesion is the measure of how much the code in a JAR is related. For example, utility packages or bundles often have quite low cohesion. For example, the Apache Derby SQL database contains a very interesting file system abstraction that is highly usable without Derby. That is, the cohesion between the core SQL database code and this abstraction is very low. The core Derby uses it, but it is not pertinent to being a SQL database. This is common practice because we programmers really dislike to get lots of puny little artifacts.
It is a fact of life that the cohesion in most JARs is quite low while the cohesion in packages is very high. As an example, the Apache Derby file abstraction is nicely packaged in a specification package and an implementation package. Why not extract it one day? Well, I am fine with this improvement and it would not affect any of my bundles at all. Import-Package does not care what bundle provides the package. If I had not warned them so often, I would feel sorry for all my colleagues that just had their bundles broken by a lone programmer somewhere in the world ...
Two very practical use cases that illustrate these problems. Eclipse, who heavily relies on Require-Bundle, needed to support SWT (the graphics library) on the embedded platform and on the SE platform. The embedded platform needed to be a subset of the SE platform. Unfortunately, all users of SWT had been using Require-Bundle. This made the simple solution, refactoring the SWT bundle in two bundles, impossible because it would break each and every Eclipse plugin. Bundles that used Import-Package would have been oblivious of this change.
The other use case is described in a recent blog: Catch-22 Logging with OSGI Frameworks . It is a rather long story but it boils down to that he could not combine two libraries due to unnecessary constraints caused by Require-Bundle with respect to logging. If you have the masochistic desire for the same sensation as having chewing gum in your hair, I recommend to read this blog.
Peter Kriens
Thursday, June 5, 2008
Community Event
You did not forget to register yet, did you?
See you next week in Berlin!
Peter Kriens
P.S. As an extra bonus we will host an SE Radio Interview! Shouldn't miss it if I were you!
See you next week in Berlin!
Peter Kriens
P.S. As an extra bonus we will host an SE Radio Interview! Shouldn't miss it if I were you!
Friday, May 30, 2008
Is 9903520300447984150353281023 Too Small?
JSR 277's Stanley Ho published a rationale why (in the so far unpublished EDR2) JAva Modules (JAM) invent a brand new version scheme. A rationale that needs a lot of text. I could go in painstaking detail, but I think the rationale derails in one of the first paragraphs where the requirements are (implicitly) described. I highlighted the part that describes in what kind of situations you use the major, minor, micro, update, and qualifier version parts:
Can you spot the difference between minor and micro? I can't, and that is the reason that OSGi proposes the following convention: incompatible (major), backward compatible (minor), no API change/bugfix (micro), and builds or variations (qualifier). That is, our micro is Sun's update because Sun's minor and micro appear to have an identical purpose.
And ever thought about the concept of largely backward compatible? Isn't that something like a little bit pregnant?
There are always reason to improve on existing schemes, but any improvement should be balanced against the interests of the existing and future audiences. I am not stating that the OSGi version is the mother of all version schemes, we are as fallible as all of us. Maybe we were too rationalistic; we looked at a lot of schemes and saw how people were also looking for more room at the low-end of the version scheme and hardly ever incremented the first numbers. If you standardize there is always a tension between allowing as much freedom as possible but also minimizing the complexity of the implementations that are depending on the variations you offer. We chose simplicity.
Is the OSGi scheme usable? Well, we have no outstanding requirements or bugs in this area, nor were any proposed in JSR 291, while at the same time it is heavily used. Jason van Zyl, Maven, told me they were thinking of adopting the scheme as well. SpringSource converted almost 400 open source projects to bundles and they did not complain. Seems there is a lot of practical usage out there.
Wouldn't it be a lot more productive for all of us if Sun would just adopt the OSGi scheme? There is lots of work to do on the module API, why not reuse existing specifications where you can? And if the OSGi scheme has burning issues, why not report that as a bug or change request, after all Sun is a distinguished OSGi member.
With an infinite number of builds or variations, 9903520300447984150353281023 possible bug fixes (80.000 fixes per microsecond until the Sun implodes), 4611686014132420609 backward compatible changes, and 2147483647 incompatible releases, the OSGi spec seems to have enough room to breathe for mere mortals.
Peter Kriens
P.S. You did register for the OSGi Community Event in Berlin, June 10-11? Please do so, we have a very strong program prepared with lots of cool demos. Hope to see you there!
- Major version number should be incremented for making changes that are not backward-compatible. The minor and the micro numbers should then be reset to zero and the update number omitted.
- Minor version number should be incremented for making medium or minor changes where the software remains largely backward-compatible (although minor incompatibilities might be possible); the micro number should then be reset to zero and the update number omitted.
- Micro version number should be incremented for changing implementation details where the software remains largely backward compatible (although minor incompatibilities might be possible); the update number should then be omitted.
- Update version number should be incremented for adding bug fixes or performance improvements in a highly compatible fashion.
- Qualifier should be changed when the build number or milestone is changed.
Can you spot the difference between minor and micro? I can't, and that is the reason that OSGi proposes the following convention: incompatible (major), backward compatible (minor), no API change/bugfix (micro), and builds or variations (qualifier). That is, our micro is Sun's update because Sun's minor and micro appear to have an identical purpose.
And ever thought about the concept of largely backward compatible? Isn't that something like a little bit pregnant?
There are always reason to improve on existing schemes, but any improvement should be balanced against the interests of the existing and future audiences. I am not stating that the OSGi version is the mother of all version schemes, we are as fallible as all of us. Maybe we were too rationalistic; we looked at a lot of schemes and saw how people were also looking for more room at the low-end of the version scheme and hardly ever incremented the first numbers. If you standardize there is always a tension between allowing as much freedom as possible but also minimizing the complexity of the implementations that are depending on the variations you offer. We chose simplicity.
Is the OSGi scheme usable? Well, we have no outstanding requirements or bugs in this area, nor were any proposed in JSR 291, while at the same time it is heavily used. Jason van Zyl, Maven, told me they were thinking of adopting the scheme as well. SpringSource converted almost 400 open source projects to bundles and they did not complain. Seems there is a lot of practical usage out there.
Wouldn't it be a lot more productive for all of us if Sun would just adopt the OSGi scheme? There is lots of work to do on the module API, why not reuse existing specifications where you can? And if the OSGi scheme has burning issues, why not report that as a bug or change request, after all Sun is a distinguished OSGi member.
With an infinite number of builds or variations, 9903520300447984150353281023 possible bug fixes (80.000 fixes per microsecond until the Sun implodes), 4611686014132420609 backward compatible changes, and 2147483647 incompatible releases, the OSGi spec seems to have enough room to breathe for mere mortals.
Peter Kriens
P.S. You did register for the OSGi Community Event in Berlin, June 10-11? Please do so, we have a very strong program prepared with lots of cool demos. Hope to see you there!
Subscribe to:
Posts (Atom)