After some slow (hot!) summer weeks there is suddenly a lot of activity. The best news so far is the patent pledge that the OSGi Alliance has made this week. Five key members of the OSGi Alliance have promised not to sue anybody that implements the release 4 spec for patent infringement as long as the patent is necessary for the implementation. Patents are a necessity but software patents can be bizarre. With over 6 million approved patents, no programmer can claim that he is writing code that does not infringe on some patents. If you are a really big company the problem is not that serious. Once Microsoft knocks on the door of Sun Microsystems it usually ends with an exchange of patents and a deal not to sue each other. As long as you are small, the costs of suing you are less than any potential royalties so small companies are also relatively safe. However, when you grow you can suddenly find some lawyers knocking on your door, potentially eating away a large part of your well deserved fortune. Look at Research in Motion (RIM) of the Blackberry that was forced to pay 600 million dollar for a patent that was suspect at the least.
The patent pledge that was made by the 5 OSGi members has largely removed this potential booby trap when you implement OSGi specifications. And in my opinion, that is exactly the way standards organizations should work. Participants in the OSGi ecosystem should all gain by having more adoption of OSGi technology because it grows the market; something we all can take advantage of. It never was the intention of the OSGi members to become rich on royalties; this pledge has made that crystal clear.
Less positive this week was the vote for JSR 298 by the J2ME Executive Committee. They decided to approve a JSR that is right in OSGi Alliance’s backyard: Telematics. I must not have paid attention because I thought this JSR was stillborn. In May it was voted down by the EC, I have missed the reconsideration ballot of this week.
Why is this JSR 298 bad? Lets take a look. First it states that “OSGi could be too heavy”. Sigh. First, OSGi technology is extremely lean for what it does. Second, it was designed to build applications out of managed middleware. This model allows applications to be lean and mean because they rely on provided middleware. MIDP and other J2ME models have no concept of shared libraries and therefore require a choice between installing it on the platform via an undefined way (growing t the platform) or including it in your application (growing your application). Due to the sharing model, moderately complex OSGi applications are usually much smaller than their brethren on other J2ME application environments.
Now let us step back for a second. Flash memory costs 2 cts a megabyte today in large quantities. What are we talking about? An OSGi R4 framework can be implemented in 250K. CPU speed is not an issue because the OSGi framework lets components communicate with only a little bit of setup overhead. I dare to say that the price/performance ratio of an OSGi framework is very hard to beat.
And despite rumors of the contrary, OSGi technology does run on CLDC environments! Not out of the box because the class loaders of CLDC are veiled, but most CLDC VMs can run an OSGi framework with a bit of work. All the OSGi APIs limit themselves to a subset of CLDC and have therefore no probem on CLDC. However, also in this area one should stay realistic. CLDC is a cramped environment that maybe once was necessary because CDC was too big. Since then, processors have become magnitudes more powerful, flash memory has become a gift with cereals in the form of a USB memory stick, and internal memory is also dropping in price. And this trend is bound to continue for the coming years.
There are clearly applications that require the Bill Of Material (BOM) to be squeezed to the last cent. However, one can ask if these are the applications that require standardized Java APIs. The advantage of standardized Java APIs is that software from different parties can run together and collaborate to achieve the desired functionality. Creating a cramped environment is likely to make this goal a lot harder to achieve. I have seen many examples where the penny wise choice of a limited environment turned out to be pound foolish.
Last but not least: security. The OSGi specifications have an extensive and very powerful security model that is almost absent in MIDP; the security in MIDP is very difficult to extend to a complex area as telematics. Software that erroneously sends out a few unwanted short messages is not good but neither is it a disaster. Controlling real world devices like cars is a different story. The JSR is completely silent about security. How come?
I hope this elucidation has put the OSGi technology is heavy argument finally out of the way! Then again, if this is out of the way, the whole reason for JSR 298 goes away. Much of the work that the JSR plans do is already done in the OSGi Vehicle Expert Group. Especially the upcoming VEG release will handle full control of the vehicle using standard protocols like OMA DM as well as application control. And we’ll likely also have an interesting navigation model!
The second ballot for JSR 298 succeeded. Ok, I was not paying attention but if they got a second chance to passthe JSR, couldwe get a second chance to vote it down?
Peter Kriens
OSGi Blog
Wednesday, July 26, 2006
Monday, July 17, 2006
To Include Or Not To Include
I am always curious if other engineering disciplines can vehemently argue so much about fundamentals as we do? Do bridge engineers have heavy discussions about how much concrete is needed for a bridge pillar? Can they get into heated arguments if a skyscraper needs I reinforcement or not? In the information industry we can differ about the most fundamental approaches, everybody is an expert. Where are our universities testing the approaches and telling is what works and what does not? How could we more easily test ideas and decide about their value?
Why do I muse about this? Last week I had an argument with some experts that we could not decide. I lost because the owner of the problem decided to go another way while I strongly believed he was wrong. I’ll explain the problem and then you can figure out how to decide what the better approach should have been.
A couple of weeks ago I had to give a tutorial at ApacheCon Europe 2006. Obviously, I had to adapt my tutorial to use Apache Felix instead of Eclipse Equinox so I would not insult my hosts (or undergo the wrath of Richard Hall). Alas, Apache Felix does not yet have an implementation of declarative services. I therefore looked at several open source implementations and picked one to port. Declarative services require a framework specific implementation because it needs the Bundle Context of other bundles and there is no public function to do this in the specification. Bad, but we could not figure out in R4 how to do this in a secure way without using security. Now we know that you can not do secure things when security is not on, but that is another story.
So porting the declarative services was straightforward except for one snatch. To parse the Service-Component manifest header, the implementation used a utility function, which turned out to be implemented in the framework JAR. Not good. Looking at the parser, I decided that the easy way out was to write a simple replacement parser; the syntax for the Service-Component header is quite simple. The replacement code added about 6 lines to the implementation, making the declarative services bundle run on Felix without requiring any other supporting bundles.
After the tutorial I decided to submit my change as a patch, trying to be a good citizen. Great was my surprise when I started getting pushback. First, they felt that the parser was too simplistic; it ignored empty paths and did not detect syntax errors in the attributes. Personally that is fine by me, parsers should be lenient (though not creating erroneous values) and generators should be strict. However, this is again another story, many people feel more comfortable with rigid parsers for diagnostic reasons.
So I then redid the parser, discovering that the Service-Component header did not even support attributes! I threw the right exceptions on empty paths and discovered that Service-Component allowed quotes! Interestingly, the original manifest parser was thus not suited for this header at all.
After submitting my second patch, my expectation to be the hero now was again not honored. They still did not like it. Why? There was now a redundancy in the system; there were now two manifest parsers (despite that the original was wrong). They were therefore not willing to accept my patch. Instead, they will update their central manifest parser to support the Service-Component header quotes and reject attributes. Why? They did not like the redundancy.
This obviously did not fix my coupling problem whatsoever. I am a strong believer in least amount of coupling. In the OSGi build system I have addressed exactly this problem by copying the class files of utilities from the class path into the JAR; by making the packages private I prevent any version clashes. This gives me a single source without lots of utility bundles. The disadvantage is of course that if you find a bad bug, you must update all the bundles that contain that code. In my experience this is rarely much different from a shared bundle. State of the art version handling is so brittle that it is likely that all dependent bundles require an update to make the system resolve with the new utility bundle. And even if they require an update, the difference between updating one bundle or several bundles is not that big. It can even be questioned if you want to update the dependent bundles, often they do not really need the bug fix.
However, in the manifest parser case, I would personally gladly have accepted the source code redundancy; the new “improved” parser was only 15 lines. Yes, it is redundant, but the chance that you have an error in this code is pretty minute after testing and review.
And this is the point I want to discuss after this long introduction. We must balance redundancy (bad) versus coupling between bundles (bad). Redundancy is bad because it means we sometimes have to fix bugs or make improvements in multiple places. Coupling is bad because it makes the deployment situation more complex. The fact that today we are starting to handle dependencies does not mean dependencies have become benign. They can still bite you unexpectedly and mean. So how do we balance two bads? How do we decide which is the least bad?
Peter Kriens
Why do I muse about this? Last week I had an argument with some experts that we could not decide. I lost because the owner of the problem decided to go another way while I strongly believed he was wrong. I’ll explain the problem and then you can figure out how to decide what the better approach should have been.
A couple of weeks ago I had to give a tutorial at ApacheCon Europe 2006. Obviously, I had to adapt my tutorial to use Apache Felix instead of Eclipse Equinox so I would not insult my hosts (or undergo the wrath of Richard Hall). Alas, Apache Felix does not yet have an implementation of declarative services. I therefore looked at several open source implementations and picked one to port. Declarative services require a framework specific implementation because it needs the Bundle Context of other bundles and there is no public function to do this in the specification. Bad, but we could not figure out in R4 how to do this in a secure way without using security. Now we know that you can not do secure things when security is not on, but that is another story.
So porting the declarative services was straightforward except for one snatch. To parse the Service-Component manifest header, the implementation used a utility function, which turned out to be implemented in the framework JAR. Not good. Looking at the parser, I decided that the easy way out was to write a simple replacement parser; the syntax for the Service-Component header is quite simple. The replacement code added about 6 lines to the implementation, making the declarative services bundle run on Felix without requiring any other supporting bundles.
After the tutorial I decided to submit my change as a patch, trying to be a good citizen. Great was my surprise when I started getting pushback. First, they felt that the parser was too simplistic; it ignored empty paths and did not detect syntax errors in the attributes. Personally that is fine by me, parsers should be lenient (though not creating erroneous values) and generators should be strict. However, this is again another story, many people feel more comfortable with rigid parsers for diagnostic reasons.
So I then redid the parser, discovering that the Service-Component header did not even support attributes! I threw the right exceptions on empty paths and discovered that Service-Component allowed quotes! Interestingly, the original manifest parser was thus not suited for this header at all.
After submitting my second patch, my expectation to be the hero now was again not honored. They still did not like it. Why? There was now a redundancy in the system; there were now two manifest parsers (despite that the original was wrong). They were therefore not willing to accept my patch. Instead, they will update their central manifest parser to support the Service-Component header quotes and reject attributes. Why? They did not like the redundancy.
This obviously did not fix my coupling problem whatsoever. I am a strong believer in least amount of coupling. In the OSGi build system I have addressed exactly this problem by copying the class files of utilities from the class path into the JAR; by making the packages private I prevent any version clashes. This gives me a single source without lots of utility bundles. The disadvantage is of course that if you find a bad bug, you must update all the bundles that contain that code. In my experience this is rarely much different from a shared bundle. State of the art version handling is so brittle that it is likely that all dependent bundles require an update to make the system resolve with the new utility bundle. And even if they require an update, the difference between updating one bundle or several bundles is not that big. It can even be questioned if you want to update the dependent bundles, often they do not really need the bug fix.
However, in the manifest parser case, I would personally gladly have accepted the source code redundancy; the new “improved” parser was only 15 lines. Yes, it is redundant, but the chance that you have an error in this code is pretty minute after testing and review.
And this is the point I want to discuss after this long introduction. We must balance redundancy (bad) versus coupling between bundles (bad). Redundancy is bad because it means we sometimes have to fix bugs or make improvements in multiple places. Coupling is bad because it makes the deployment situation more complex. The fact that today we are starting to handle dependencies does not mean dependencies have become benign. They can still bite you unexpectedly and mean. So how do we balance two bads? How do we decide which is the least bad?
Peter Kriens
Tuesday, July 11, 2006
Eclipse PDE versus JDE/ant
Last week there was a discussion about Eclipse Plugin Development Environment (PDE) on the equinox developers list. As you likely know, not only is Eclipse based on the OSGi Service Platform, it also provides support for developing bundles. They call them plugins, but they are really supposed to be bundles. I use the Eclipse Java Development Environment (JDE) for all my work, including the OSGi build with specification, reference implementations, and test suites. The build contains a total of more than 300 bundles files, most of them bundles, in more than 130 projects. There are bigger builds, but this is a significant project by my metrics.
The PDE discussion evolved around the flexibility of the environment. It got started by a request from the maven people to make the location of the manifest flexible. This touched a nerve with me because it is right at the heart why I am not using the PDE. I unfortunately still use the normal JDE because the PDE does not cut it for me. That is, my bundles are still bundles and have not moved up the evolution chain to become plugins. I have tried it many times but they always seem to revert to my home brewn ant/btool solution. Honestly, I tried.
The reason why I am not using the PDE architecture is because it contains a number of harsh constraints. My key problems are:
The reason for these (serious) architectural constraints is that Eclipse must be able to run your code at a whiff at any time. “No build” is the mantra. If you click Debug, Eclipse starts the OSGi framework, and runs your bundle from the project directory, without wrapping the content in a JAR. Well almost, the bin directory is mapped on the class path despite the fact that it is not really a member of the bundle class path, but hey it works! Then again, I think this quick edit-debug cycle can also be achieved with less draconian constraints.
Despite my complaints, rants, or other kinds of obnoxious behavior with the Eclipse guys, I did not get one iota further. I have been going through Eclipse 3.0, 3.1, and 3.2 without making even a dent in this model. Interestingly, I thought my dislike was generally shared by many but in the Equinox mailing list discussion there was a response that intrigued me:
We seem to have the same dislike of spaghetti, but we completely differed in the approach to avoid it. The red thread in my professional career has been decoupling, decoupling, and decoupling. Last week I stumbled on a course I had given in the early nineties and I was surprised how little I had moved forward in this area. I still strongly feel that the coupling of deliverables (bundles or other JARs) is an architectural decision, and not an ad-hoc design choice or afterthought. Today we have lots of tools to manage dependencies (maven, OSGi, etc.), but no-dependency is still much better than a managed dependency. This implies that adding a dependency like a library is a decision that needs to be taken by the architect of the project, not any developer. I fail to see a problem with “Organize Imports”, as long as the scope is defined by the project. Incidentally, this is the way the JDE works. You need to specifically add a JAR to the build path. Once it is on the build path, developers can (and should) use it as much as possible. Just like a pregnancy, there is no such thing is a little bit of coupling; it is a yes/no decision. If you use it once, you might as well touch it often if it saves you time. During the development, developers should not have to be concerned about what they use of the libraries that are made available to them.
However, next comes the packaging. The PDE can only have one JAR file and the content is defined by my project directory layout (sort of). I do not know how other people are building bundles, but most bundles I am responsible for require some packaging tricks.
For example, sometimes I use a package providing some util function. I really do not want to create a dependency on an evil util bundle. I also do not want to copy the source code. So in quite a few projects I copy the byte codes into my bundle and make the package bundle private. This way I combine a single source without adding additional dependencies.
Another example is the OSGi test case. An OSGi test case consists of a set of inner bundles that are loaded in the target and then perform their testing. This requires me to wrap up bundles in bundles. A similar structure is the deliverable for all the reference implementations. This is a single bundle that installs a set of other bundles, which it contains. I also often need to deliver the same code in different forms, for example in a midlet and a bundle.
I also find that I use the same information items in lots of different places. It is surprising how often you need the bundle symbolic name, or a name derived from it. I use properties to ensure that I have a single definition. I can easily preprocess the manifest, the declarative services XML file, or a readme.txt file. All these tricks are impossible with the PDE, they require some kind of processing before you run the bundle which is unfortunately not available.
And most important, in this phase you can finally see what your real dependencies are. In the OSGi build I calculate the imported packages and regularly inspect the outcome. Quite often, I am surprised what I see and then find out why a specific dependency was introduced. They are usually easy to solve.
So in contrast to the mailing list response, I think I have carefully thought about the dependency issue. Despite reflecting on his remarks, I still think that the current model I use is better than the Eclipse PDE model. Now I only have to find a way so that they listen to me!
Peter Kriens
The PDE discussion evolved around the flexibility of the environment. It got started by a request from the maven people to make the location of the manifest flexible. This touched a nerve with me because it is right at the heart why I am not using the PDE. I unfortunately still use the normal JDE because the PDE does not cut it for me. That is, my bundles are still bundles and have not moved up the evolution chain to become plugins. I have tried it many times but they always seem to revert to my home brewn ant/btool solution. Honestly, I tried.
The reason why I am not using the PDE architecture is because it contains a number of harsh constraints. My key problems are:
- A project = a bundle. It is impossible to deliver multiple artifacts from the same code base with the provided tools. For the OSGi build it would mean 300++ projects.
- The project directory layout must look like the root of the bundle, well eh, sort of. The PDE does perform some magic under the covers before it runs your code or packages the bundle, but the idea is that there is a 1:1 relation.
- The manifest must be up to date at all time, forcing the developer to enter the dependency information in literal form. There is no possibility to generate information for the manifest during build time.
The reason for these (serious) architectural constraints is that Eclipse must be able to run your code at a whiff at any time. “No build” is the mantra. If you click Debug, Eclipse starts the OSGi framework, and runs your bundle from the project directory, without wrapping the content in a JAR. Well almost, the bin directory is mapped on the class path despite the fact that it is not really a member of the bundle class path, but hey it works! Then again, I think this quick edit-debug cycle can also be achieved with less draconian constraints.
Despite my complaints, rants, or other kinds of obnoxious behavior with the Eclipse guys, I did not get one iota further. I have been going through Eclipse 3.0, 3.1, and 3.2 without making even a dent in this model. Interestingly, I thought my dislike was generally shared by many but in the Equinox mailing list discussion there was a response that intrigued me:
I've generally fine with 1 project = 1 bundle, and I like being forced to maintain manifests as I develop my bundles. Your model of pooling all the source together and then sorting out the bundles later sounds messy to me. With no constraints on importing packages, developers just hit "Organise Imports" to pull in dependencies from all over the place, resulting in spaghetti code.
We seem to have the same dislike of spaghetti, but we completely differed in the approach to avoid it. The red thread in my professional career has been decoupling, decoupling, and decoupling. Last week I stumbled on a course I had given in the early nineties and I was surprised how little I had moved forward in this area. I still strongly feel that the coupling of deliverables (bundles or other JARs) is an architectural decision, and not an ad-hoc design choice or afterthought. Today we have lots of tools to manage dependencies (maven, OSGi, etc.), but no-dependency is still much better than a managed dependency. This implies that adding a dependency like a library is a decision that needs to be taken by the architect of the project, not any developer. I fail to see a problem with “Organize Imports”, as long as the scope is defined by the project. Incidentally, this is the way the JDE works. You need to specifically add a JAR to the build path. Once it is on the build path, developers can (and should) use it as much as possible. Just like a pregnancy, there is no such thing is a little bit of coupling; it is a yes/no decision. If you use it once, you might as well touch it often if it saves you time. During the development, developers should not have to be concerned about what they use of the libraries that are made available to them.
However, next comes the packaging. The PDE can only have one JAR file and the content is defined by my project directory layout (sort of). I do not know how other people are building bundles, but most bundles I am responsible for require some packaging tricks.
For example, sometimes I use a package providing some util function. I really do not want to create a dependency on an evil util bundle. I also do not want to copy the source code. So in quite a few projects I copy the byte codes into my bundle and make the package bundle private. This way I combine a single source without adding additional dependencies.
Another example is the OSGi test case. An OSGi test case consists of a set of inner bundles that are loaded in the target and then perform their testing. This requires me to wrap up bundles in bundles. A similar structure is the deliverable for all the reference implementations. This is a single bundle that installs a set of other bundles, which it contains. I also often need to deliver the same code in different forms, for example in a midlet and a bundle.
I also find that I use the same information items in lots of different places. It is surprising how often you need the bundle symbolic name, or a name derived from it. I use properties to ensure that I have a single definition. I can easily preprocess the manifest, the declarative services XML file, or a readme.txt file. All these tricks are impossible with the PDE, they require some kind of processing before you run the bundle which is unfortunately not available.
And most important, in this phase you can finally see what your real dependencies are. In the OSGi build I calculate the imported packages and regularly inspect the outcome. Quite often, I am surprised what I see and then find out why a specific dependency was introduced. They are usually easy to solve.
So in contrast to the mailing list response, I think I have carefully thought about the dependency issue. Despite reflecting on his remarks, I still think that the current model I use is better than the Eclipse PDE model. Now I only have to find a way so that they listen to me!
Peter Kriens
Thursday, June 29, 2006
News from ApacheCon 2006 Europe
This week I attend the ApacheCon Europe in Dublin. Oh boy, what a difference with last year in San Diego at ApacheCon US! Not only is Dublin the opposite of San Diego in many ways, but the visibility and acceptance of OSGi technology by the Apache crowd makes a world of difference. There were several people that were complimenting OSGi technology so much that I’d to blush; some even told me they regularly read this blog!
I primarily attended ApacheCon to give a tutorial. This was an adapted version of the Eclipse tutorial. I had tried to replace Equinox with Apache Felix in this tutorial but had to unfortunately give up late Saturday afternoon because the Declarative Services did not work yet on Felix. When I realized this (again) on Friday, I had tried to port the Eclipse Declarative Services implementation to Felix. Initially this looked very simple; there was a small dependency that was easy to get rid of. Unfortunately, after installing it failed to initialize my component. At that time it had become Saturday afternoon and I had to revert to Equinox or spent yet another Sunday away from the family. However, yesterday evening Richard Hall and I sat together and fixed it. So now Felix can soon use the declarative services from Equinox. Isn’t OSGi technology wonderful?
The tutorial went very well, though 3 hours is very short. Especially because I’d like to tell too much about the background. The big surprise, however, was yesterday. The whole afternoon in the Leinster room was dedicated to OSGi subjects. First Richard gave an excellent, albeit speedy, overview of OSGi, JSR 277, JSR 291, and JSR 294. Man, that guy can speak fast! The next subject was a panel about OSGi. Geir Magnusson (Apache’s Jakarta project chair) was the moderator and on the panel you could find Tim Ellison (IBM, Apache Harmony), Richard Hall (Apache Felix), Enrique Rodriguez (Apache Directory server), Jason van Zyl (Maven 2), and me. After having to answer the customary grilling about licensing, patents, and the possibility to contribute to the spec (yes, we are working on all those issues) we had an interesting discussion.
A major subject is of course how to get Apache projects to adopt OSGi technology. The Apache Foundation does not have a technology roadmap. Technology is developed bottom up and projects are completely free to choose their own technology. To get OSGi adopted by Apache it is therefore necessary to convince each project of its merits. A key strategy to get Apache to use OSGi seems to be the Maven route. Maven is a build tool that more and more projects start to use. The Apache Felix project is using Maven. A couple of weeks ago I had helped Richard to update the OSGi plugin for Maven so that it was easier to use. During the panel this plugin seems a route into Maven. The OSGi headers are non-intrusive. A JAR can be made in such a way that it runs on the Classpath as well as a bundle in an OSGi framework. Maybe this plugin should become part of Maven so that the appropriate headers become part of all Apache JARs. Should be investigated!
The last session was from Marcel Offermans (luminis) about OSGi and best practices. Marcel is an active committer on the Apache Felix project. luminis has adopted OSGi technology 4 years ago and has lots of experience at different customers. This experience showed in the slides. I loved his slides that showed an overview of how to document package dependencies and service dependencies. The service dependencies were based on my triangle/sphere model. We really need a graphic tool that visualizes these dependencies based on a set of JARs or metadata. Where are the students that can create all these goodies?
This afternoon I will fly back home but I already feel very happy about this conference. What a difference with JavaOne and even ApacheCon in San Diego! Most people now have a basic idea what OSGi is and several Apache projects are using it. This makes evangelizing OSGi technology a lot more fun!
I primarily attended ApacheCon to give a tutorial. This was an adapted version of the Eclipse tutorial. I had tried to replace Equinox with Apache Felix in this tutorial but had to unfortunately give up late Saturday afternoon because the Declarative Services did not work yet on Felix. When I realized this (again) on Friday, I had tried to port the Eclipse Declarative Services implementation to Felix. Initially this looked very simple; there was a small dependency that was easy to get rid of. Unfortunately, after installing it failed to initialize my component. At that time it had become Saturday afternoon and I had to revert to Equinox or spent yet another Sunday away from the family. However, yesterday evening Richard Hall and I sat together and fixed it. So now Felix can soon use the declarative services from Equinox. Isn’t OSGi technology wonderful?
The tutorial went very well, though 3 hours is very short. Especially because I’d like to tell too much about the background. The big surprise, however, was yesterday. The whole afternoon in the Leinster room was dedicated to OSGi subjects. First Richard gave an excellent, albeit speedy, overview of OSGi, JSR 277, JSR 291, and JSR 294. Man, that guy can speak fast! The next subject was a panel about OSGi. Geir Magnusson (Apache’s Jakarta project chair) was the moderator and on the panel you could find Tim Ellison (IBM, Apache Harmony), Richard Hall (Apache Felix), Enrique Rodriguez (Apache Directory server), Jason van Zyl (Maven 2), and me. After having to answer the customary grilling about licensing, patents, and the possibility to contribute to the spec (yes, we are working on all those issues) we had an interesting discussion.
A major subject is of course how to get Apache projects to adopt OSGi technology. The Apache Foundation does not have a technology roadmap. Technology is developed bottom up and projects are completely free to choose their own technology. To get OSGi adopted by Apache it is therefore necessary to convince each project of its merits. A key strategy to get Apache to use OSGi seems to be the Maven route. Maven is a build tool that more and more projects start to use. The Apache Felix project is using Maven. A couple of weeks ago I had helped Richard to update the OSGi plugin for Maven so that it was easier to use. During the panel this plugin seems a route into Maven. The OSGi headers are non-intrusive. A JAR can be made in such a way that it runs on the Classpath as well as a bundle in an OSGi framework. Maybe this plugin should become part of Maven so that the appropriate headers become part of all Apache JARs. Should be investigated!
The last session was from Marcel Offermans (luminis) about OSGi and best practices. Marcel is an active committer on the Apache Felix project. luminis has adopted OSGi technology 4 years ago and has lots of experience at different customers. This experience showed in the slides. I loved his slides that showed an overview of how to document package dependencies and service dependencies. The service dependencies were based on my triangle/sphere model. We really need a graphic tool that visualizes these dependencies based on a set of JARs or metadata. Where are the students that can create all these goodies?
This afternoon I will fly back home but I already feel very happy about this conference. What a difference with JavaOne and even ApacheCon in San Diego! Most people now have a basic idea what OSGi is and several Apache projects are using it. This makes evangelizing OSGi technology a lot more fun!
Friday, June 23, 2006
Signature Analysis for Mobile OSGi
This week I received a mail about the requirements for the JSR 232 Test Case Kit (TCK). We had all the parts except for one. SUN requires a signature verification of the API packages. These are always the fun parts: how to verify the API signatures with the least amount of work. Let me first give you a short explanation of the purpose of this test.
Sun is rightly paranoid about Java fragmentation. That they sometimes cause fragmentation by being paranoia about it is another story, the need for consistency between implementations of an API is clear. During the early Java days Microsoft improved Java by adding extra, windows specific, methods to the java.lang packages. What is wrong with improvements one can ask? Well, developers started to use these features because they were really, eh, improvements. Unfortunately, the use of the improvements made it impossible to run that code on any other environment, like, lets say Solaris. This is a bit painful if one realizes how much cost is involved to provide the Java portability. After this experience, Sun hired a large amount of lawyers to add clauses to all their contracts that APIs could be implemented but not extended. Today, they therefore require that all TCKs verify that the API they test is not improved. So you are warned, it is likely that one day you have promised at least once to Sun never to improve their APIs; I have many times.
Not improving does not mean that the classes in the specification and implementation are identical. It is perfectly legal to re-implement them and add extra (package) private methods and classes. The re-implementation may not change anything in the visible classes, methods or fields. This includes the type, the name, staticness, finalness, or abstractness. Visible means it is public or protected.
So how does one go about checking the equality? The naïve solution is to write one humongous test case where you hard code a test for each field, method, etc. This is of course my nightmare solution because it is horror to maintain. My attention span for repetitive tasks is unfortunately measured in nanoseconds. Writing such a test case would likely make me loose my hair and take ages, neither of which is an attractive option.
I could generate an XML file from the API jar files, write an XML parser and verify the structure of the API based on the information in the XML file. I already had the code to parse the JAR file and turn it into XML. However, frequent readers know my great love for XML solutions, so this was not a desirable option either. What next?
When making a prototype for declarative services ++ (which was the embryo for iPojo), I had been playing with ASM. ASM is a fantastically small library (base is 20k) that makes it very easy to modify class files. If you can modify it, you should also be able to read class files. ASM uses the visitor pattern, which is very powerful but not that well known; probably because it is non trivial to understand. With the visitor pattern, you provide separate an object structure from its algorithms. Instead of the algorithm traversing the object structure, the structure visits each member and calls appropriate methods in the algorithm object. The visitor patterns allows you to be oblivious of the types and structure of your target object but learn about it in a controlled and type safe way.
ASM uses the visitor pattern to traverse the bytecodes of a class. The Class Reader class takes the class bytes and a visitor object. This visitor object is called for all the class information, fields, methods, etc. There is a base class for these Class Visitor objects so you only have to implement the methods you really need. Normally, you also need a Class Writer, which is a visitor itself so you can do a 1:1 transformation by giving the Class Reader a Class Writer.
ASM looked interesting for the signature analysis. So the next step was to get the API. I am proud to say that we have a very clean build in the OSGi build system. We have a JAR file for the Core, Compendium, and MEG specifications. I just copied this JAR to the signature verification bundles (one for each specification). The rest was peanuts, in the OSGi test case I read the JAR and parse all the classes using the Class Visitor. For each found class, I then get the Class object from the OSGi Framework using Class.forName. The rest is just work, I compare each method and field for visibility, presence, or absence. Any discrepancy in the visible aspects of the class are flagged as errors.
The whole bundle is about 300 lines and the extra build lines were 3! Quite satisfactory. However, before I get too snug, I was stuck with one problem: I could not detect extra classes in the implementation. Java unfortunately does not have a method to iterate over the contents of a package. On the PC you can use the java.classpath property but that is not an option in the embedded world; there is no guarantee that the code is stored as directories or JARs. So I gained a battle, but not yet the war. If you can tell me a solution to finding the extra classes (or tell me with authority that Sun is not testing extra classes), then I would really appreciate this!
Sun is rightly paranoid about Java fragmentation. That they sometimes cause fragmentation by being paranoia about it is another story, the need for consistency between implementations of an API is clear. During the early Java days Microsoft improved Java by adding extra, windows specific, methods to the java.lang packages. What is wrong with improvements one can ask? Well, developers started to use these features because they were really, eh, improvements. Unfortunately, the use of the improvements made it impossible to run that code on any other environment, like, lets say Solaris. This is a bit painful if one realizes how much cost is involved to provide the Java portability. After this experience, Sun hired a large amount of lawyers to add clauses to all their contracts that APIs could be implemented but not extended. Today, they therefore require that all TCKs verify that the API they test is not improved. So you are warned, it is likely that one day you have promised at least once to Sun never to improve their APIs; I have many times.
Not improving does not mean that the classes in the specification and implementation are identical. It is perfectly legal to re-implement them and add extra (package) private methods and classes. The re-implementation may not change anything in the visible classes, methods or fields. This includes the type, the name, staticness, finalness, or abstractness. Visible means it is public or protected.
So how does one go about checking the equality? The naïve solution is to write one humongous test case where you hard code a test for each field, method, etc. This is of course my nightmare solution because it is horror to maintain. My attention span for repetitive tasks is unfortunately measured in nanoseconds. Writing such a test case would likely make me loose my hair and take ages, neither of which is an attractive option.
I could generate an XML file from the API jar files, write an XML parser and verify the structure of the API based on the information in the XML file. I already had the code to parse the JAR file and turn it into XML. However, frequent readers know my great love for XML solutions, so this was not a desirable option either. What next?
When making a prototype for declarative services ++ (which was the embryo for iPojo), I had been playing with ASM. ASM is a fantastically small library (base is 20k) that makes it very easy to modify class files. If you can modify it, you should also be able to read class files. ASM uses the visitor pattern, which is very powerful but not that well known; probably because it is non trivial to understand. With the visitor pattern, you provide separate an object structure from its algorithms. Instead of the algorithm traversing the object structure, the structure visits each member and calls appropriate methods in the algorithm object. The visitor patterns allows you to be oblivious of the types and structure of your target object but learn about it in a controlled and type safe way.
ASM uses the visitor pattern to traverse the bytecodes of a class. The Class Reader class takes the class bytes and a visitor object. This visitor object is called for all the class information, fields, methods, etc. There is a base class for these Class Visitor objects so you only have to implement the methods you really need. Normally, you also need a Class Writer, which is a visitor itself so you can do a 1:1 transformation by giving the Class Reader a Class Writer.
ASM looked interesting for the signature analysis. So the next step was to get the API. I am proud to say that we have a very clean build in the OSGi build system. We have a JAR file for the Core, Compendium, and MEG specifications. I just copied this JAR to the signature verification bundles (one for each specification). The rest was peanuts, in the OSGi test case I read the JAR and parse all the classes using the Class Visitor. For each found class, I then get the Class object from the OSGi Framework using Class.forName. The rest is just work, I compare each method and field for visibility, presence, or absence. Any discrepancy in the visible aspects of the class are flagged as errors.
The whole bundle is about 300 lines and the extra build lines were 3! Quite satisfactory. However, before I get too snug, I was stuck with one problem: I could not detect extra classes in the implementation. Java unfortunately does not have a method to iterate over the contents of a package. On the PC you can use the java.classpath property but that is not an option in the embedded world; there is no guarantee that the code is stored as directories or JARs. So I gained a battle, but not yet the war. If you can tell me a solution to finding the extra classes (or tell me with authority that Sun is not testing extra classes), then I would really appreciate this!
Tuesday, June 13, 2006
JSR 232 Passed the Public Review Ballot
I just heard that the JSR 232 Expert Group has passed the public review ballot by the Executive Committee. The vote was overwhelmingly positive: In Favor 10, Against 0, Abstain 3, No Vote 3.
Congratulations to the JSR 232 EG and the OSGi MEG! Now, when do we get the JSR 232 phones? I want one!
Interestingly, Philips abstained with a comment that showed a mistaken understanding of the OSGi specifications:
I heard this story now several times as an argument against the OSGi so some elucidation is in place. The OSGi service model is very well suited for a multiple VM model but does not enforce distribution model on all applications because the performance overhead is pretty steep.
When we started with the OSGi Framework in 1999 we developed a prototype that was based on full isolation. The bundles communicated through communication pipes that required a specific protocol. After trying to start our third VM on the 486/32Mb machine we decided that scaling might be a tiny little blocking problem with this architecture. After lots of discussions, we decided that we wanted to isolate the bundles as much as possible but still run them in the same VM. The bundle communications were taking place through services. In OSGi speak, a service is a plain old Java object registered under an interface with the Framework.
So the OSGi model is very simple, if you want to communicate to any peers, you must obtain a service from the Framework or register a service with the Framework. All your communication therefore has to pass through a set of well known objects, objects which are tracked by the Framework.
In the current implementations, these objects are usually untouched by the Framework. This has the advantage of maximum speed and zero overhead, quite important in an embedded world. People have been telling me since 1981 that computers get cheaper and faster, but somehow, we the software people have always taken a significant bite out of that gain with easier programming models that are a lot less efficient. And as long as the average web site that has pages ending in .jsp feel slower than others, this efficiency gain seems not unimportant to the big guys as well.
However, there is nothing in the specification that forbids a Framework vendor to take advantage of multi VMs, as long as the methods in the service interface can be marshalled (serialized to transfer the parameters and return values to the other VM). Marshalling is not an OSGi specific mechanism, any multi-VM model will require it. Marshalling makes services implementations considerably harder and slower. A factor of 1000 to 10000 times slower is not uncommon. There is absolutely no free lunch here.
Within the OSGi model you have a choice. You can architect your system for maximum performance by running all bundles in the same VM. For most applications this is the best solution because the isolation that the OSGi model provides is sufficient, despite it is not 100%. Bundles that require a higher level of isolation can be moved to different VMs but suffer a performance bottleneck. So far, the isolation that the OSGi provides using the class loader model has proven to be sufficient, and for many applications the overhead of moving to a distributed model with higher isolation is just not worth it.
I hope that the Philips representative on the JCP EC will reconsider its judgment. Not only is better the enemy of good, in this case the OSGi Service Platform does not restrict the better in any way.
Congratulations to the JSR 232 EG and the OSGi MEG! Now, when do we get the JSR 232 phones? I want one!
Interestingly, Philips abstained with a comment that showed a mistaken understanding of the OSGi specifications:
On 2006-06-12 Philips Electronics UK Ltd voted Abstain with the following comment:
I am unable to vote in favour of this JSR due to technical concerns about OSGi. Specifically the requirement for all service bundles to run in the same VM instance. To freeze this in an architecture just as techniques for efficiently supporting multiple VM instances are appearing in products seems wrong.
I heard this story now several times as an argument against the OSGi so some elucidation is in place. The OSGi service model is very well suited for a multiple VM model but does not enforce distribution model on all applications because the performance overhead is pretty steep.
When we started with the OSGi Framework in 1999 we developed a prototype that was based on full isolation. The bundles communicated through communication pipes that required a specific protocol. After trying to start our third VM on the 486/32Mb machine we decided that scaling might be a tiny little blocking problem with this architecture. After lots of discussions, we decided that we wanted to isolate the bundles as much as possible but still run them in the same VM. The bundle communications were taking place through services. In OSGi speak, a service is a plain old Java object registered under an interface with the Framework.
So the OSGi model is very simple, if you want to communicate to any peers, you must obtain a service from the Framework or register a service with the Framework. All your communication therefore has to pass through a set of well known objects, objects which are tracked by the Framework.
In the current implementations, these objects are usually untouched by the Framework. This has the advantage of maximum speed and zero overhead, quite important in an embedded world. People have been telling me since 1981 that computers get cheaper and faster, but somehow, we the software people have always taken a significant bite out of that gain with easier programming models that are a lot less efficient. And as long as the average web site that has pages ending in .jsp feel slower than others, this efficiency gain seems not unimportant to the big guys as well.
However, there is nothing in the specification that forbids a Framework vendor to take advantage of multi VMs, as long as the methods in the service interface can be marshalled (serialized to transfer the parameters and return values to the other VM). Marshalling is not an OSGi specific mechanism, any multi-VM model will require it. Marshalling makes services implementations considerably harder and slower. A factor of 1000 to 10000 times slower is not uncommon. There is absolutely no free lunch here.
Within the OSGi model you have a choice. You can architect your system for maximum performance by running all bundles in the same VM. For most applications this is the best solution because the isolation that the OSGi model provides is sufficient, despite it is not 100%. Bundles that require a higher level of isolation can be moved to different VMs but suffer a performance bottleneck. So far, the isolation that the OSGi provides using the class loader model has proven to be sufficient, and for many applications the overhead of moving to a distributed model with higher isolation is just not worth it.
I hope that the Philips representative on the JCP EC will reconsider its judgment. Not only is better the enemy of good, in this case the OSGi Service Platform does not restrict the better in any way.
Monday, June 12, 2006
The PHP Blues
This weekend I had to make a web site for my wife. She is a teacher and after trying for other work she decided to start a Dutch Language school for children in the Languedoc-Roussillon area (France). These type of spousal activities usually add sizably to the workload of being a loving but technically oriented husband. I am sure many of my readers will recognize this situation and sympathize. So this weekend, I really had to get the site up and running because there was an open day Saturday afternoon.
I could not find a cheap hosting service for Java and besides, a friend had recently send me a movie about Web application development. Though this movie is very long it is a funny account of where technology can go wrong. It compares different web development technologies like J2EE, Zope, Ruby on Rails, etc. J2EE comes out rock bottom on all measured scales because of the highly complex configuration and the many XML “situps”. The movie had made it abundantly clear that J2EE was not the technology to go for a tiny school’s web site.
The most popular technique available is likely PHP. PHP is a language that gets executed on the server side. The processor there parses a file that ends with “.php”. These files consist partly of HTML and partly of PHP code. PHP code is signaled with a “<?php” instruction and ends with “?>”. PHP makes it easy to do simple things, unlike some other environments we all know. I found an interesting program on the web that contained a PHP web server and MySQL lite. When started, it serves on port 800 a set of pages from the directory where it started from. That made it easy to test the web site on my own computer. I just need a web site where the navigation bar is maintained in a single file so that changes do not require an update of all other pages. This is easy to do with PHP using includes. The language is supported on every hosting service and there are lots of code snippets on the net that can be “borrowed”.
Sounds all good … Well, it means you have to learn a new language and that is always excruciatingly painful. The syntax is not such an issue nowadays, most languages look similar on that level. However, the libraries are where the pain is. You constantly have to search for code snippets to find out how the functions you are looking for are called. Something I can do in 5 minutes in Java takes me half an hour in PHP because I cannot use my Java experience. Despite these obstacles, it was surprisingly easy to get the site up and running. Developing, and thus debugging, the site was surprisingly easy. You never have to compile and build. Just change the text in Notepad, save and refresh your browser. Though this style does not work well for large systems, it works surprisingly well for simple applications. It is so simple, that my wife can now maintain most of the web site because the text pages have only a boilerplate PHP.
Now, as a husband I was properly rewarded with an impressed wife. As the OSGi evangelist I was on edge. I have made lots of small web sites with OSGi because the Http Service provides excellent Servlet support. I have tried most libraries but never found a library that really made it as easy (and as pleasant) as PHP did. Now OSGi is an open environment so what are the changes of hosting PHP inside an OSGi system? Then again, PHP works so well not because the language is great (it isn’t), but because it strikes the correct balance between HTML coding and programming. I then had to think of Beanshell. Beanshell is a Java interpreter that accepts a simpler syntax of Java as well that is on par with PHP. It does not need types, though it does accept them. The good news (for me) about Beanshell is that I can use the Java libraries. It blends seamlessly with the Java environment. So last night I decided to see if I could make this idea work. Use Beanshell as PHP. I decided to use the same structure as PHP. So a hello world would look like:
<html>
<body>
<?bsh print( “Hello world at ” + new Date() ); ?>
</body>
</html>
This turned out to be surprisingly easy. Partly because I already had a small bundle that served files from the file system. It uses configuration admin to match a directory to a path on the web site (can be found on “/web/admin” which is a bit too big a name for what it does). I changed this bundle to register a servlet that would parse any .bsh file, extract the Beanshell code, and merge the HTML and Beanshell execution into the Servlets output. Before the script is executed, I set the Bundle Context, the Http Servlet Request and Http Servlet Response. This gives you full access to the OSGi environment.
Currently the program reads the files from disk, however, there is no reason why it could not read the files from the JAR resources. You develop the pages quickly using the file system and then store them in the JAR before delivery.
I guess by spending the rest of the Sunday behind my computer I have consumed a significant part of the goodwill I had won with my wife … Then again, this was a cool exercise! It is really nice to be able to combine the easy programming model of a script language with the power of a full blown OSGi/Java system. No more PHP jealousy!
If you have too much time and want to play with the prototype, feel free to download it. It is a prototype, so do not expect too much. It should also be clear that this is a personal toy and has no official status. The JAR also contains the sources. There is a small admin console on the http://localhost:/web/admin where you can fill in the alias (for example /test) and the path (for example c:\tmp). Have fun!
Peter Kriens
I could not find a cheap hosting service for Java and besides, a friend had recently send me a movie about Web application development. Though this movie is very long it is a funny account of where technology can go wrong. It compares different web development technologies like J2EE, Zope, Ruby on Rails, etc. J2EE comes out rock bottom on all measured scales because of the highly complex configuration and the many XML “situps”. The movie had made it abundantly clear that J2EE was not the technology to go for a tiny school’s web site.
The most popular technique available is likely PHP. PHP is a language that gets executed on the server side. The processor there parses a file that ends with “.php”. These files consist partly of HTML and partly of PHP code. PHP code is signaled with a “<?php” instruction and ends with “?>”. PHP makes it easy to do simple things, unlike some other environments we all know. I found an interesting program on the web that contained a PHP web server and MySQL lite. When started, it serves on port 800 a set of pages from the directory where it started from. That made it easy to test the web site on my own computer. I just need a web site where the navigation bar is maintained in a single file so that changes do not require an update of all other pages. This is easy to do with PHP using includes. The language is supported on every hosting service and there are lots of code snippets on the net that can be “borrowed”.
Sounds all good … Well, it means you have to learn a new language and that is always excruciatingly painful. The syntax is not such an issue nowadays, most languages look similar on that level. However, the libraries are where the pain is. You constantly have to search for code snippets to find out how the functions you are looking for are called. Something I can do in 5 minutes in Java takes me half an hour in PHP because I cannot use my Java experience. Despite these obstacles, it was surprisingly easy to get the site up and running. Developing, and thus debugging, the site was surprisingly easy. You never have to compile and build. Just change the text in Notepad, save and refresh your browser. Though this style does not work well for large systems, it works surprisingly well for simple applications. It is so simple, that my wife can now maintain most of the web site because the text pages have only a boilerplate PHP.
Now, as a husband I was properly rewarded with an impressed wife. As the OSGi evangelist I was on edge. I have made lots of small web sites with OSGi because the Http Service provides excellent Servlet support. I have tried most libraries but never found a library that really made it as easy (and as pleasant) as PHP did. Now OSGi is an open environment so what are the changes of hosting PHP inside an OSGi system? Then again, PHP works so well not because the language is great (it isn’t), but because it strikes the correct balance between HTML coding and programming. I then had to think of Beanshell. Beanshell is a Java interpreter that accepts a simpler syntax of Java as well that is on par with PHP. It does not need types, though it does accept them. The good news (for me) about Beanshell is that I can use the Java libraries. It blends seamlessly with the Java environment. So last night I decided to see if I could make this idea work. Use Beanshell as PHP. I decided to use the same structure as PHP. So a hello world would look like:
<html>
<body>
<?bsh print( “Hello world at ” + new Date() ); ?>
</body>
</html>
This turned out to be surprisingly easy. Partly because I already had a small bundle that served files from the file system. It uses configuration admin to match a directory to a path on the web site (can be found on “/web/admin” which is a bit too big a name for what it does). I changed this bundle to register a servlet that would parse any .bsh file, extract the Beanshell code, and merge the HTML and Beanshell execution into the Servlets output. Before the script is executed, I set the Bundle Context, the Http Servlet Request and Http Servlet Response. This gives you full access to the OSGi environment.
Currently the program reads the files from disk, however, there is no reason why it could not read the files from the JAR resources. You develop the pages quickly using the file system and then store them in the JAR before delivery.
I guess by spending the rest of the Sunday behind my computer I have consumed a significant part of the goodwill I had won with my wife … Then again, this was a cool exercise! It is really nice to be able to combine the easy programming model of a script language with the power of a full blown OSGi/Java system. No more PHP jealousy!
If you have too much time and want to play with the prototype, feel free to download it. It is a prototype, so do not expect too much. It should also be clear that this is a personal toy and has no official status. The JAR also contains the sources. There is a small admin console on the http://localhost:
Peter Kriens
Subscribe to:
Posts (Atom)