This week I attend the ApacheCon Europe in Dublin. Oh boy, what a difference with last year in San Diego at ApacheCon US! Not only is Dublin the opposite of San Diego in many ways, but the visibility and acceptance of OSGi technology by the Apache crowd makes a world of difference. There were several people that were complimenting OSGi technology so much that I’d to blush; some even told me they regularly read this blog!
I primarily attended ApacheCon to give a tutorial. This was an adapted version of the Eclipse tutorial. I had tried to replace Equinox with Apache Felix in this tutorial but had to unfortunately give up late Saturday afternoon because the Declarative Services did not work yet on Felix. When I realized this (again) on Friday, I had tried to port the Eclipse Declarative Services implementation to Felix. Initially this looked very simple; there was a small dependency that was easy to get rid of. Unfortunately, after installing it failed to initialize my component. At that time it had become Saturday afternoon and I had to revert to Equinox or spent yet another Sunday away from the family. However, yesterday evening Richard Hall and I sat together and fixed it. So now Felix can soon use the declarative services from Equinox. Isn’t OSGi technology wonderful?
The tutorial went very well, though 3 hours is very short. Especially because I’d like to tell too much about the background. The big surprise, however, was yesterday. The whole afternoon in the Leinster room was dedicated to OSGi subjects. First Richard gave an excellent, albeit speedy, overview of OSGi, JSR 277, JSR 291, and JSR 294. Man, that guy can speak fast! The next subject was a panel about OSGi. Geir Magnusson (Apache’s Jakarta project chair) was the moderator and on the panel you could find Tim Ellison (IBM, Apache Harmony), Richard Hall (Apache Felix), Enrique Rodriguez (Apache Directory server), Jason van Zyl (Maven 2), and me. After having to answer the customary grilling about licensing, patents, and the possibility to contribute to the spec (yes, we are working on all those issues) we had an interesting discussion.
A major subject is of course how to get Apache projects to adopt OSGi technology. The Apache Foundation does not have a technology roadmap. Technology is developed bottom up and projects are completely free to choose their own technology. To get OSGi adopted by Apache it is therefore necessary to convince each project of its merits. A key strategy to get Apache to use OSGi seems to be the Maven route. Maven is a build tool that more and more projects start to use. The Apache Felix project is using Maven. A couple of weeks ago I had helped Richard to update the OSGi plugin for Maven so that it was easier to use. During the panel this plugin seems a route into Maven. The OSGi headers are non-intrusive. A JAR can be made in such a way that it runs on the Classpath as well as a bundle in an OSGi framework. Maybe this plugin should become part of Maven so that the appropriate headers become part of all Apache JARs. Should be investigated!
The last session was from Marcel Offermans (luminis) about OSGi and best practices. Marcel is an active committer on the Apache Felix project. luminis has adopted OSGi technology 4 years ago and has lots of experience at different customers. This experience showed in the slides. I loved his slides that showed an overview of how to document package dependencies and service dependencies. The service dependencies were based on my triangle/sphere model. We really need a graphic tool that visualizes these dependencies based on a set of JARs or metadata. Where are the students that can create all these goodies?
This afternoon I will fly back home but I already feel very happy about this conference. What a difference with JavaOne and even ApacheCon in San Diego! Most people now have a basic idea what OSGi is and several Apache projects are using it. This makes evangelizing OSGi technology a lot more fun!
OSGi Blog
Thursday, June 29, 2006
Friday, June 23, 2006
Signature Analysis for Mobile OSGi
This week I received a mail about the requirements for the JSR 232 Test Case Kit (TCK). We had all the parts except for one. SUN requires a signature verification of the API packages. These are always the fun parts: how to verify the API signatures with the least amount of work. Let me first give you a short explanation of the purpose of this test.
Sun is rightly paranoid about Java fragmentation. That they sometimes cause fragmentation by being paranoia about it is another story, the need for consistency between implementations of an API is clear. During the early Java days Microsoft improved Java by adding extra, windows specific, methods to the java.lang packages. What is wrong with improvements one can ask? Well, developers started to use these features because they were really, eh, improvements. Unfortunately, the use of the improvements made it impossible to run that code on any other environment, like, lets say Solaris. This is a bit painful if one realizes how much cost is involved to provide the Java portability. After this experience, Sun hired a large amount of lawyers to add clauses to all their contracts that APIs could be implemented but not extended. Today, they therefore require that all TCKs verify that the API they test is not improved. So you are warned, it is likely that one day you have promised at least once to Sun never to improve their APIs; I have many times.
Not improving does not mean that the classes in the specification and implementation are identical. It is perfectly legal to re-implement them and add extra (package) private methods and classes. The re-implementation may not change anything in the visible classes, methods or fields. This includes the type, the name, staticness, finalness, or abstractness. Visible means it is public or protected.
So how does one go about checking the equality? The naïve solution is to write one humongous test case where you hard code a test for each field, method, etc. This is of course my nightmare solution because it is horror to maintain. My attention span for repetitive tasks is unfortunately measured in nanoseconds. Writing such a test case would likely make me loose my hair and take ages, neither of which is an attractive option.
I could generate an XML file from the API jar files, write an XML parser and verify the structure of the API based on the information in the XML file. I already had the code to parse the JAR file and turn it into XML. However, frequent readers know my great love for XML solutions, so this was not a desirable option either. What next?
When making a prototype for declarative services ++ (which was the embryo for iPojo), I had been playing with ASM. ASM is a fantastically small library (base is 20k) that makes it very easy to modify class files. If you can modify it, you should also be able to read class files. ASM uses the visitor pattern, which is very powerful but not that well known; probably because it is non trivial to understand. With the visitor pattern, you provide separate an object structure from its algorithms. Instead of the algorithm traversing the object structure, the structure visits each member and calls appropriate methods in the algorithm object. The visitor patterns allows you to be oblivious of the types and structure of your target object but learn about it in a controlled and type safe way.
ASM uses the visitor pattern to traverse the bytecodes of a class. The Class Reader class takes the class bytes and a visitor object. This visitor object is called for all the class information, fields, methods, etc. There is a base class for these Class Visitor objects so you only have to implement the methods you really need. Normally, you also need a Class Writer, which is a visitor itself so you can do a 1:1 transformation by giving the Class Reader a Class Writer.
ASM looked interesting for the signature analysis. So the next step was to get the API. I am proud to say that we have a very clean build in the OSGi build system. We have a JAR file for the Core, Compendium, and MEG specifications. I just copied this JAR to the signature verification bundles (one for each specification). The rest was peanuts, in the OSGi test case I read the JAR and parse all the classes using the Class Visitor. For each found class, I then get the Class object from the OSGi Framework using Class.forName. The rest is just work, I compare each method and field for visibility, presence, or absence. Any discrepancy in the visible aspects of the class are flagged as errors.
The whole bundle is about 300 lines and the extra build lines were 3! Quite satisfactory. However, before I get too snug, I was stuck with one problem: I could not detect extra classes in the implementation. Java unfortunately does not have a method to iterate over the contents of a package. On the PC you can use the java.classpath property but that is not an option in the embedded world; there is no guarantee that the code is stored as directories or JARs. So I gained a battle, but not yet the war. If you can tell me a solution to finding the extra classes (or tell me with authority that Sun is not testing extra classes), then I would really appreciate this!
Sun is rightly paranoid about Java fragmentation. That they sometimes cause fragmentation by being paranoia about it is another story, the need for consistency between implementations of an API is clear. During the early Java days Microsoft improved Java by adding extra, windows specific, methods to the java.lang packages. What is wrong with improvements one can ask? Well, developers started to use these features because they were really, eh, improvements. Unfortunately, the use of the improvements made it impossible to run that code on any other environment, like, lets say Solaris. This is a bit painful if one realizes how much cost is involved to provide the Java portability. After this experience, Sun hired a large amount of lawyers to add clauses to all their contracts that APIs could be implemented but not extended. Today, they therefore require that all TCKs verify that the API they test is not improved. So you are warned, it is likely that one day you have promised at least once to Sun never to improve their APIs; I have many times.
Not improving does not mean that the classes in the specification and implementation are identical. It is perfectly legal to re-implement them and add extra (package) private methods and classes. The re-implementation may not change anything in the visible classes, methods or fields. This includes the type, the name, staticness, finalness, or abstractness. Visible means it is public or protected.
So how does one go about checking the equality? The naïve solution is to write one humongous test case where you hard code a test for each field, method, etc. This is of course my nightmare solution because it is horror to maintain. My attention span for repetitive tasks is unfortunately measured in nanoseconds. Writing such a test case would likely make me loose my hair and take ages, neither of which is an attractive option.
I could generate an XML file from the API jar files, write an XML parser and verify the structure of the API based on the information in the XML file. I already had the code to parse the JAR file and turn it into XML. However, frequent readers know my great love for XML solutions, so this was not a desirable option either. What next?
When making a prototype for declarative services ++ (which was the embryo for iPojo), I had been playing with ASM. ASM is a fantastically small library (base is 20k) that makes it very easy to modify class files. If you can modify it, you should also be able to read class files. ASM uses the visitor pattern, which is very powerful but not that well known; probably because it is non trivial to understand. With the visitor pattern, you provide separate an object structure from its algorithms. Instead of the algorithm traversing the object structure, the structure visits each member and calls appropriate methods in the algorithm object. The visitor patterns allows you to be oblivious of the types and structure of your target object but learn about it in a controlled and type safe way.
ASM uses the visitor pattern to traverse the bytecodes of a class. The Class Reader class takes the class bytes and a visitor object. This visitor object is called for all the class information, fields, methods, etc. There is a base class for these Class Visitor objects so you only have to implement the methods you really need. Normally, you also need a Class Writer, which is a visitor itself so you can do a 1:1 transformation by giving the Class Reader a Class Writer.
ASM looked interesting for the signature analysis. So the next step was to get the API. I am proud to say that we have a very clean build in the OSGi build system. We have a JAR file for the Core, Compendium, and MEG specifications. I just copied this JAR to the signature verification bundles (one for each specification). The rest was peanuts, in the OSGi test case I read the JAR and parse all the classes using the Class Visitor. For each found class, I then get the Class object from the OSGi Framework using Class.forName. The rest is just work, I compare each method and field for visibility, presence, or absence. Any discrepancy in the visible aspects of the class are flagged as errors.
The whole bundle is about 300 lines and the extra build lines were 3! Quite satisfactory. However, before I get too snug, I was stuck with one problem: I could not detect extra classes in the implementation. Java unfortunately does not have a method to iterate over the contents of a package. On the PC you can use the java.classpath property but that is not an option in the embedded world; there is no guarantee that the code is stored as directories or JARs. So I gained a battle, but not yet the war. If you can tell me a solution to finding the extra classes (or tell me with authority that Sun is not testing extra classes), then I would really appreciate this!
Tuesday, June 13, 2006
JSR 232 Passed the Public Review Ballot
I just heard that the JSR 232 Expert Group has passed the public review ballot by the Executive Committee. The vote was overwhelmingly positive: In Favor 10, Against 0, Abstain 3, No Vote 3.
Congratulations to the JSR 232 EG and the OSGi MEG! Now, when do we get the JSR 232 phones? I want one!
Interestingly, Philips abstained with a comment that showed a mistaken understanding of the OSGi specifications:
I heard this story now several times as an argument against the OSGi so some elucidation is in place. The OSGi service model is very well suited for a multiple VM model but does not enforce distribution model on all applications because the performance overhead is pretty steep.
When we started with the OSGi Framework in 1999 we developed a prototype that was based on full isolation. The bundles communicated through communication pipes that required a specific protocol. After trying to start our third VM on the 486/32Mb machine we decided that scaling might be a tiny little blocking problem with this architecture. After lots of discussions, we decided that we wanted to isolate the bundles as much as possible but still run them in the same VM. The bundle communications were taking place through services. In OSGi speak, a service is a plain old Java object registered under an interface with the Framework.
So the OSGi model is very simple, if you want to communicate to any peers, you must obtain a service from the Framework or register a service with the Framework. All your communication therefore has to pass through a set of well known objects, objects which are tracked by the Framework.
In the current implementations, these objects are usually untouched by the Framework. This has the advantage of maximum speed and zero overhead, quite important in an embedded world. People have been telling me since 1981 that computers get cheaper and faster, but somehow, we the software people have always taken a significant bite out of that gain with easier programming models that are a lot less efficient. And as long as the average web site that has pages ending in .jsp feel slower than others, this efficiency gain seems not unimportant to the big guys as well.
However, there is nothing in the specification that forbids a Framework vendor to take advantage of multi VMs, as long as the methods in the service interface can be marshalled (serialized to transfer the parameters and return values to the other VM). Marshalling is not an OSGi specific mechanism, any multi-VM model will require it. Marshalling makes services implementations considerably harder and slower. A factor of 1000 to 10000 times slower is not uncommon. There is absolutely no free lunch here.
Within the OSGi model you have a choice. You can architect your system for maximum performance by running all bundles in the same VM. For most applications this is the best solution because the isolation that the OSGi model provides is sufficient, despite it is not 100%. Bundles that require a higher level of isolation can be moved to different VMs but suffer a performance bottleneck. So far, the isolation that the OSGi provides using the class loader model has proven to be sufficient, and for many applications the overhead of moving to a distributed model with higher isolation is just not worth it.
I hope that the Philips representative on the JCP EC will reconsider its judgment. Not only is better the enemy of good, in this case the OSGi Service Platform does not restrict the better in any way.
Congratulations to the JSR 232 EG and the OSGi MEG! Now, when do we get the JSR 232 phones? I want one!
Interestingly, Philips abstained with a comment that showed a mistaken understanding of the OSGi specifications:
On 2006-06-12 Philips Electronics UK Ltd voted Abstain with the following comment:
I am unable to vote in favour of this JSR due to technical concerns about OSGi. Specifically the requirement for all service bundles to run in the same VM instance. To freeze this in an architecture just as techniques for efficiently supporting multiple VM instances are appearing in products seems wrong.
I heard this story now several times as an argument against the OSGi so some elucidation is in place. The OSGi service model is very well suited for a multiple VM model but does not enforce distribution model on all applications because the performance overhead is pretty steep.
When we started with the OSGi Framework in 1999 we developed a prototype that was based on full isolation. The bundles communicated through communication pipes that required a specific protocol. After trying to start our third VM on the 486/32Mb machine we decided that scaling might be a tiny little blocking problem with this architecture. After lots of discussions, we decided that we wanted to isolate the bundles as much as possible but still run them in the same VM. The bundle communications were taking place through services. In OSGi speak, a service is a plain old Java object registered under an interface with the Framework.
So the OSGi model is very simple, if you want to communicate to any peers, you must obtain a service from the Framework or register a service with the Framework. All your communication therefore has to pass through a set of well known objects, objects which are tracked by the Framework.
In the current implementations, these objects are usually untouched by the Framework. This has the advantage of maximum speed and zero overhead, quite important in an embedded world. People have been telling me since 1981 that computers get cheaper and faster, but somehow, we the software people have always taken a significant bite out of that gain with easier programming models that are a lot less efficient. And as long as the average web site that has pages ending in .jsp feel slower than others, this efficiency gain seems not unimportant to the big guys as well.
However, there is nothing in the specification that forbids a Framework vendor to take advantage of multi VMs, as long as the methods in the service interface can be marshalled (serialized to transfer the parameters and return values to the other VM). Marshalling is not an OSGi specific mechanism, any multi-VM model will require it. Marshalling makes services implementations considerably harder and slower. A factor of 1000 to 10000 times slower is not uncommon. There is absolutely no free lunch here.
Within the OSGi model you have a choice. You can architect your system for maximum performance by running all bundles in the same VM. For most applications this is the best solution because the isolation that the OSGi model provides is sufficient, despite it is not 100%. Bundles that require a higher level of isolation can be moved to different VMs but suffer a performance bottleneck. So far, the isolation that the OSGi provides using the class loader model has proven to be sufficient, and for many applications the overhead of moving to a distributed model with higher isolation is just not worth it.
I hope that the Philips representative on the JCP EC will reconsider its judgment. Not only is better the enemy of good, in this case the OSGi Service Platform does not restrict the better in any way.
Monday, June 12, 2006
The PHP Blues
This weekend I had to make a web site for my wife. She is a teacher and after trying for other work she decided to start a Dutch Language school for children in the Languedoc-Roussillon area (France). These type of spousal activities usually add sizably to the workload of being a loving but technically oriented husband. I am sure many of my readers will recognize this situation and sympathize. So this weekend, I really had to get the site up and running because there was an open day Saturday afternoon.
I could not find a cheap hosting service for Java and besides, a friend had recently send me a movie about Web application development. Though this movie is very long it is a funny account of where technology can go wrong. It compares different web development technologies like J2EE, Zope, Ruby on Rails, etc. J2EE comes out rock bottom on all measured scales because of the highly complex configuration and the many XML “situps”. The movie had made it abundantly clear that J2EE was not the technology to go for a tiny school’s web site.
The most popular technique available is likely PHP. PHP is a language that gets executed on the server side. The processor there parses a file that ends with “.php”. These files consist partly of HTML and partly of PHP code. PHP code is signaled with a “<?php” instruction and ends with “?>”. PHP makes it easy to do simple things, unlike some other environments we all know. I found an interesting program on the web that contained a PHP web server and MySQL lite. When started, it serves on port 800 a set of pages from the directory where it started from. That made it easy to test the web site on my own computer. I just need a web site where the navigation bar is maintained in a single file so that changes do not require an update of all other pages. This is easy to do with PHP using includes. The language is supported on every hosting service and there are lots of code snippets on the net that can be “borrowed”.
Sounds all good … Well, it means you have to learn a new language and that is always excruciatingly painful. The syntax is not such an issue nowadays, most languages look similar on that level. However, the libraries are where the pain is. You constantly have to search for code snippets to find out how the functions you are looking for are called. Something I can do in 5 minutes in Java takes me half an hour in PHP because I cannot use my Java experience. Despite these obstacles, it was surprisingly easy to get the site up and running. Developing, and thus debugging, the site was surprisingly easy. You never have to compile and build. Just change the text in Notepad, save and refresh your browser. Though this style does not work well for large systems, it works surprisingly well for simple applications. It is so simple, that my wife can now maintain most of the web site because the text pages have only a boilerplate PHP.
Now, as a husband I was properly rewarded with an impressed wife. As the OSGi evangelist I was on edge. I have made lots of small web sites with OSGi because the Http Service provides excellent Servlet support. I have tried most libraries but never found a library that really made it as easy (and as pleasant) as PHP did. Now OSGi is an open environment so what are the changes of hosting PHP inside an OSGi system? Then again, PHP works so well not because the language is great (it isn’t), but because it strikes the correct balance between HTML coding and programming. I then had to think of Beanshell. Beanshell is a Java interpreter that accepts a simpler syntax of Java as well that is on par with PHP. It does not need types, though it does accept them. The good news (for me) about Beanshell is that I can use the Java libraries. It blends seamlessly with the Java environment. So last night I decided to see if I could make this idea work. Use Beanshell as PHP. I decided to use the same structure as PHP. So a hello world would look like:
<html>
<body>
<?bsh print( “Hello world at ” + new Date() ); ?>
</body>
</html>
This turned out to be surprisingly easy. Partly because I already had a small bundle that served files from the file system. It uses configuration admin to match a directory to a path on the web site (can be found on “/web/admin” which is a bit too big a name for what it does). I changed this bundle to register a servlet that would parse any .bsh file, extract the Beanshell code, and merge the HTML and Beanshell execution into the Servlets output. Before the script is executed, I set the Bundle Context, the Http Servlet Request and Http Servlet Response. This gives you full access to the OSGi environment.
Currently the program reads the files from disk, however, there is no reason why it could not read the files from the JAR resources. You develop the pages quickly using the file system and then store them in the JAR before delivery.
I guess by spending the rest of the Sunday behind my computer I have consumed a significant part of the goodwill I had won with my wife … Then again, this was a cool exercise! It is really nice to be able to combine the easy programming model of a script language with the power of a full blown OSGi/Java system. No more PHP jealousy!
If you have too much time and want to play with the prototype, feel free to download it. It is a prototype, so do not expect too much. It should also be clear that this is a personal toy and has no official status. The JAR also contains the sources. There is a small admin console on the http://localhost:/web/admin where you can fill in the alias (for example /test) and the path (for example c:\tmp). Have fun!
Peter Kriens
I could not find a cheap hosting service for Java and besides, a friend had recently send me a movie about Web application development. Though this movie is very long it is a funny account of where technology can go wrong. It compares different web development technologies like J2EE, Zope, Ruby on Rails, etc. J2EE comes out rock bottom on all measured scales because of the highly complex configuration and the many XML “situps”. The movie had made it abundantly clear that J2EE was not the technology to go for a tiny school’s web site.
The most popular technique available is likely PHP. PHP is a language that gets executed on the server side. The processor there parses a file that ends with “.php”. These files consist partly of HTML and partly of PHP code. PHP code is signaled with a “<?php” instruction and ends with “?>”. PHP makes it easy to do simple things, unlike some other environments we all know. I found an interesting program on the web that contained a PHP web server and MySQL lite. When started, it serves on port 800 a set of pages from the directory where it started from. That made it easy to test the web site on my own computer. I just need a web site where the navigation bar is maintained in a single file so that changes do not require an update of all other pages. This is easy to do with PHP using includes. The language is supported on every hosting service and there are lots of code snippets on the net that can be “borrowed”.
Sounds all good … Well, it means you have to learn a new language and that is always excruciatingly painful. The syntax is not such an issue nowadays, most languages look similar on that level. However, the libraries are where the pain is. You constantly have to search for code snippets to find out how the functions you are looking for are called. Something I can do in 5 minutes in Java takes me half an hour in PHP because I cannot use my Java experience. Despite these obstacles, it was surprisingly easy to get the site up and running. Developing, and thus debugging, the site was surprisingly easy. You never have to compile and build. Just change the text in Notepad, save and refresh your browser. Though this style does not work well for large systems, it works surprisingly well for simple applications. It is so simple, that my wife can now maintain most of the web site because the text pages have only a boilerplate PHP.
Now, as a husband I was properly rewarded with an impressed wife. As the OSGi evangelist I was on edge. I have made lots of small web sites with OSGi because the Http Service provides excellent Servlet support. I have tried most libraries but never found a library that really made it as easy (and as pleasant) as PHP did. Now OSGi is an open environment so what are the changes of hosting PHP inside an OSGi system? Then again, PHP works so well not because the language is great (it isn’t), but because it strikes the correct balance between HTML coding and programming. I then had to think of Beanshell. Beanshell is a Java interpreter that accepts a simpler syntax of Java as well that is on par with PHP. It does not need types, though it does accept them. The good news (for me) about Beanshell is that I can use the Java libraries. It blends seamlessly with the Java environment. So last night I decided to see if I could make this idea work. Use Beanshell as PHP. I decided to use the same structure as PHP. So a hello world would look like:
<html>
<body>
<?bsh print( “Hello world at ” + new Date() ); ?>
</body>
</html>
This turned out to be surprisingly easy. Partly because I already had a small bundle that served files from the file system. It uses configuration admin to match a directory to a path on the web site (can be found on “/web/admin” which is a bit too big a name for what it does). I changed this bundle to register a servlet that would parse any .bsh file, extract the Beanshell code, and merge the HTML and Beanshell execution into the Servlets output. Before the script is executed, I set the Bundle Context, the Http Servlet Request and Http Servlet Response. This gives you full access to the OSGi environment.
Currently the program reads the files from disk, however, there is no reason why it could not read the files from the JAR resources. You develop the pages quickly using the file system and then store them in the JAR before delivery.
I guess by spending the rest of the Sunday behind my computer I have consumed a significant part of the goodwill I had won with my wife … Then again, this was a cool exercise! It is really nice to be able to combine the easy programming model of a script language with the power of a full blown OSGi/Java system. No more PHP jealousy!
If you have too much time and want to play with the prototype, feel free to download it. It is a prototype, so do not expect too much. It should also be clear that this is a personal toy and has no official status. The JAR also contains the sources. There is a small admin console on the http://localhost:
Peter Kriens
Monday, June 5, 2006
OSGi for Industrial Devices
At the moment I am sitting in the train coming back from a consultancy job. A mid-sized company in Holland is developing a control system for their high end analytical devices. The devices are for a highly specialized market; the amount of knowledge that goes into these instruments is staggering. I was asked to review the architecture because their new architecture is based on OSGi technology.
At the end of the day I got a very enthusiastic tour of the company. I was shown all the different systems and the myriad of adapters that they can add to the core device. All these adapters must be supported by the software, using the standard OSGi Device Management. It is really cool to see all that stuff work in practice. Getting paid for learning so much new domain knowledge!
As you probably have noticed, I am not allowed to tell which manufacturer, or what he actually produces, because the company wants to keep the use of OSGi confidential. This is unfortunately true for many OSGi applications. It is always really hard to see these incredibly interesting applications of OSGi technology and then have to keep silent. I would love to tell you about the solutions they found to the hard problems of device control in high end technology.
Higher end devices are a perfect space for OSGi technology. We developed the specification for small embedded devices that are focused on consumer electronics. However, adoption of OSGi technology in this space is not high because of the eternal trade off between Bill Of Material (BOM) cost and development cost. When you intend to produce 100.000 devices, a single dollar saved in the BOM translates to 100.000 dollars in development. The extra requirements that the OSGi/Java puts upon the device (higher end CPU, more Flash, and more dynamic memory) are therefore often seen as prohibitive. I am not sure the arguments are still that valid, a Megabyte of Flash nowadays costs 2 dollar cents, but the consumer electronics is a very conservative industry. In these markets OSGi technology will not find large scale adoption until the end users start demanding it or providers like telcos see that they can generate money by selling electronic services.
However, as my visit showed, the level above consumer electronics thinks that the OSGi proposition is very interesting. In the higher end markets, the BOM is easily dwarfed by development and management cost. When your market is 600 devices a year and you have to feed a thousand mouths, the cost for a few megabytes and a faster processor is trivial. For these devices, using OSGi technology should almost be a no-brainer. High end embedded devices show a lot of diversity; in certain cases almost every configuration is unique. Any experienced person knows that this poses a nightmare for software maintenance. Trying to integrate this type of diversity in a single static image is just not workable. So far, only component systems can cope well with this problem and the OSGi Service Platform is the premier component system in the market. The OSGi technology for industrial devices can reduce development cost, simplify management, and opens up new business cases.
I do not think this goes unnoticed by many companies. During almost every trip I make I hear another story of a company that has been using OSGi for years. Most companies are proud about it but do not seek publicity. I think it is a character trait of embedded people to just get it done and not talk too much about it. There are also other reasons. For some companies it is such a strategic advantage that they do not want their competition to learn about their use of OSGi.
From my experience, I expect that in the next two years a lot of stories will surface where people tell how they successfully used OSGi technology to control many, many high end embedded devices. If you apply OSGi technology in an interesting environment, and you can talk about it, let me know. I’d love to write some articles about cool applications with OSGi technology.
Peter Kriens
At the end of the day I got a very enthusiastic tour of the company. I was shown all the different systems and the myriad of adapters that they can add to the core device. All these adapters must be supported by the software, using the standard OSGi Device Management. It is really cool to see all that stuff work in practice. Getting paid for learning so much new domain knowledge!
As you probably have noticed, I am not allowed to tell which manufacturer, or what he actually produces, because the company wants to keep the use of OSGi confidential. This is unfortunately true for many OSGi applications. It is always really hard to see these incredibly interesting applications of OSGi technology and then have to keep silent. I would love to tell you about the solutions they found to the hard problems of device control in high end technology.
Higher end devices are a perfect space for OSGi technology. We developed the specification for small embedded devices that are focused on consumer electronics. However, adoption of OSGi technology in this space is not high because of the eternal trade off between Bill Of Material (BOM) cost and development cost. When you intend to produce 100.000 devices, a single dollar saved in the BOM translates to 100.000 dollars in development. The extra requirements that the OSGi/Java puts upon the device (higher end CPU, more Flash, and more dynamic memory) are therefore often seen as prohibitive. I am not sure the arguments are still that valid, a Megabyte of Flash nowadays costs 2 dollar cents, but the consumer electronics is a very conservative industry. In these markets OSGi technology will not find large scale adoption until the end users start demanding it or providers like telcos see that they can generate money by selling electronic services.
However, as my visit showed, the level above consumer electronics thinks that the OSGi proposition is very interesting. In the higher end markets, the BOM is easily dwarfed by development and management cost. When your market is 600 devices a year and you have to feed a thousand mouths, the cost for a few megabytes and a faster processor is trivial. For these devices, using OSGi technology should almost be a no-brainer. High end embedded devices show a lot of diversity; in certain cases almost every configuration is unique. Any experienced person knows that this poses a nightmare for software maintenance. Trying to integrate this type of diversity in a single static image is just not workable. So far, only component systems can cope well with this problem and the OSGi Service Platform is the premier component system in the market. The OSGi technology for industrial devices can reduce development cost, simplify management, and opens up new business cases.
I do not think this goes unnoticed by many companies. During almost every trip I make I hear another story of a company that has been using OSGi for years. Most companies are proud about it but do not seek publicity. I think it is a character trait of embedded people to just get it done and not talk too much about it. There are also other reasons. For some companies it is such a strategic advantage that they do not want their competition to learn about their use of OSGi.
From my experience, I expect that in the next two years a lot of stories will surface where people tell how they successfully used OSGi technology to control many, many high end embedded devices. If you apply OSGi technology in an interesting environment, and you can talk about it, let me know. I’d love to write some articles about cool applications with OSGi technology.
Peter Kriens
Monday, May 22, 2006
OSGi Alliance @ JavaOne
This blog is terribly late due to a security issue at our provider that interfered with a spam filter on Gmail :-( Excuses.
The OSGi BOFs that we had organized were a complete success. The Thursday BOF attracted over a hundred people; this is incredibly successful considering that this BOF was at 8.30 PM while the “JavaOne After Dark Bash” was going on at the same time. The BOF started a bit scary because the organization had not setup the tables I had requested. Fortunately, we could steal some tables from the public space. We started with a presentation by BJ Hargrave. He was regularly interrupted with questions. Often he did not get a chance to answer the questions because someone in the audience did. This created a very interactive feeling, which worked well. After 20 minutes we asked the people to go to the demonstrations or get an OSGi T-Shirt. Though the T-Shirts won the popularity contest in the beginning, the demonstrations then won hands on. We had presentations from ProSyst, Siemens, Paremus, and Luminis. While all this was going on we presented a slide show of 240 slides (!) that consisted of presentations that I had received about OSGi related projects (Thanks all submitters!). In the end we had to be forcefully removed from the room for the next BOF.
The BOF on Wednesday, Component Programming was also quite successful. We attracted around 50 people, which at 10.30 PM and a million parties going on is amazing. I presented the introduction to the OSGi. Thomas Watson gave a presentation how Eclipse can be used to program components and Jeff McAffer talked about the Slug. We got lots of good questions and the feedback was very positive.
We also had lots of questions about the OSGi during the conference. I had made some nice shirts with “Ask me about OSGi” on the back and this generated the desired response. I also had printed stickers with this slogan and hand wrestled everybody that knew anything about the OSGi specifications to wear them. It was a bit awkward that there were actually a surprising number of people that had never heard of “OSGi”. Well, I guess there still is a lot of evangelizing to do.
Very positive are the reactions about OSGi and the enterprise. I have met many people that are extremely excited about OSGi technology being used with Spring and other enterprise technologies. I think the coming year will be very exciting with all the developments in this area. The OSGi Alliance really must organize a workshop on OSGi in the Enterprise soon.
From an OSGi Alliance point of view JavaOne was a great success!
The conference itself was, from my point of view, rather disappointing. It is big and it is monopolized by Sun; it was not easy to find a technical presentation by a non-Sun employee. I actually think I got a bit of a rash from too much sun exposure …
Peter Kriens
The OSGi BOFs that we had organized were a complete success. The Thursday BOF attracted over a hundred people; this is incredibly successful considering that this BOF was at 8.30 PM while the “JavaOne After Dark Bash” was going on at the same time. The BOF started a bit scary because the organization had not setup the tables I had requested. Fortunately, we could steal some tables from the public space. We started with a presentation by BJ Hargrave. He was regularly interrupted with questions. Often he did not get a chance to answer the questions because someone in the audience did. This created a very interactive feeling, which worked well. After 20 minutes we asked the people to go to the demonstrations or get an OSGi T-Shirt. Though the T-Shirts won the popularity contest in the beginning, the demonstrations then won hands on. We had presentations from ProSyst, Siemens, Paremus, and Luminis. While all this was going on we presented a slide show of 240 slides (!) that consisted of presentations that I had received about OSGi related projects (Thanks all submitters!). In the end we had to be forcefully removed from the room for the next BOF.
The BOF on Wednesday, Component Programming was also quite successful. We attracted around 50 people, which at 10.30 PM and a million parties going on is amazing. I presented the introduction to the OSGi. Thomas Watson gave a presentation how Eclipse can be used to program components and Jeff McAffer talked about the Slug. We got lots of good questions and the feedback was very positive.
We also had lots of questions about the OSGi during the conference. I had made some nice shirts with “Ask me about OSGi” on the back and this generated the desired response. I also had printed stickers with this slogan and hand wrestled everybody that knew anything about the OSGi specifications to wear them. It was a bit awkward that there were actually a surprising number of people that had never heard of “OSGi”. Well, I guess there still is a lot of evangelizing to do.
Very positive are the reactions about OSGi and the enterprise. I have met many people that are extremely excited about OSGi technology being used with Spring and other enterprise technologies. I think the coming year will be very exciting with all the developments in this area. The OSGi Alliance really must organize a workshop on OSGi in the Enterprise soon.
From an OSGi Alliance point of view JavaOne was a great success!
The conference itself was, from my point of view, rather disappointing. It is big and it is monopolized by Sun; it was not easy to find a technical presentation by a non-Sun employee. I actually think I got a bit of a rash from too much sun exposure …
Peter Kriens
Wednesday, May 10, 2006
Why Installers are Evil
About 7 years ago the OSGi Alliance forced me to switch from a Macintosh to a PC. Ok, they did not wrestle my arm but I needed access to the latest (and not so greatest) version of Java to do my work properly. With regret, I parted with my dear Macintosh and dived into the wonderful new world of PCs. Now, as a Mac owner you are bound to be a bit naïve about computing, you actually think it is not such a big deal. Well, you find out how wrong you are when you migrate to the PC.
The first time I stepped into the PC mess was when I wanted to clean up my disk. Excited about all the free downloadable goodies (and baddies) I had collected quite a set. I had installed these applications in different places on my hard disk and and decided to group them nicely in a folder. As you know, and I do know now, that is a distinctly stupid idea. It is useful, but still stupid because Windows does not want you to do that and in a fight between you and Windows, you find out why many call it “Win”.
Well, maybe that is too much credit; Windows has actually no clue that you moved the application file; the next time it needs the application it will blindly look in the old place and fail miserably with a cryptic message. After having too many of those cryptic message, I decided to reformat and reinstall Windows. I learned my lesson and after that occasion I never cleaned up anymore (My wife sometimes has a problem with this behavior, but fortunately she is significantly more benign than Windows).
So what is this rant doing in an OSGi related blog? Well, there is a lesson that we learned early in the OSGi: installers are evil. This is easy to say, but I will try to make it clear to you. Let us analyze why it became so painful to move these files around on a PC and not on a Mac. On the Mac, an application is a single file (on System X, it is actually a directory). This file is a simple database that consists of the code resources and other resources like configuration files, icons, html pages, properties, translations, etc. Anything that the program needs is stored in this single file. This file is introspective; this means that other programs can inspect the contents and react according to this content. We call these progams/utilities that react to this content extensions. Extensions are extremely useful.
Application files have a special type in the Mac, and this type is recognized by the disk operating system. Whenever you copy or move an application file the OS will track this in an application database. The OS actually looks in the application file's database and finds the information about the files it can open and other useful aspects. Ergo, you can move the applications around as much as you want; the OS just finds them when you need them. It has this genuine advantage that if you trash the application file, you really cleaned up all aspects of that application.
So why does Windows fail so miserably when you move an application to another directory? Well, Windows does not track the applications because it has installers! An installer is an ingenious piece of software that most PC users think is a necessity of nature. It is usually zipped and then compressed after which it is squeezed, requires a lot of thinking, asks you questions you have no clue how to answer or could not care less (fortunately there are defaults), shows lines that you cannot read because they are replaced too quick (which at least gives you the cozy feeling something is happening), splatters files all over your hard disk (sometimes in quite unexpected places), overwrites important DLLs, then spends minutes updating the registry, after which it asks you to read a README file that is incomprehensible until you need it because the application balks at you; unfortunately at that time you have no clue where to find it anymore. And if this all went well, you can run the application. The fact that this is all overkill is demonstrated by Eclipse that has no installer.
You would expect that for security reasons Windows would have a registry of these installed applications, and a way to forcefully uninstall an application. Well, you thought wrong. There is something of a voluntary registry that the installers can fill in if they feel like it. However, the information is not guaranteed by Windows. Worse, to uninstall an application Windows will ask the original installer to remove all the files, remove the registry changes, and generally clean up. The uninstaller sometimes fails because a file is missing or something else is wrong. At such a moment you have an orphan application that likely remains as a bit-squatter until you have to reformat your hard disk. Lots of these orphans can bring that time significantly closer.
When we designed the OSGi framework we were sure that the Windows Installer model was not where we wanted to go; the Mac model looked a lot more attractive. Fortunately, the JAR file model of Java is an even more powerful format than the Mac resource file. The ZIP file format is flexible enough to allow all kinds of application extension scenarios based on resource name and directory. However, due to the security risk attached to running applications we decided to strictly control all the aspects of the installation, update, and uninstallation; all these are primitive in the OSGi, there is no untrusted code involved. OSGi applications (bundles) can normally not move files to odd places and modify all kinds of settings. OSGi applications can declare information in their JAR files and other applications or utilities can inspect this information and act upon it. This is a clear example of Inversion of Control or IoC: “Don't call us, we'll call you!” For example, an application using Declarative Services contains an XML file with the proper setup. When the application is installed the Declarative Service implementation will find this file, the application does not have to register.
This is an incredibly robust model that prevents a lot of abuse and errors. As a simple example, assume an application fails all the time and you want to uninstall it. The fact that the application fails makes it very likely that the uninstaller will also fail. Leaving remnants in lots of places on your system. An OSGi application can always be uninstalled by the Framework. The uninstallation will notify any applications that performed some function on behalf of the uninstalled application. These other applications are in a good state and can therefore cleanup correctly, regardless of the state of the culprit.
The application model of the OSGi has learned from its predecessors and provides excellent characteristics. Most importantly: An overview what you have really installed and a guarantee that you can always throw anything away.No code can run without the Framework being aware of it.
Every time when new groups enter the OSGi Alliance we have the Installer discussion again. Every time people ask me how it can be wrong when Windows does it that way? 100 million users can't be wrong, can they? I leave that up to you to figure it out.
Peter Kriens
The first time I stepped into the PC mess was when I wanted to clean up my disk. Excited about all the free downloadable goodies (and baddies) I had collected quite a set. I had installed these applications in different places on my hard disk and and decided to group them nicely in a folder. As you know, and I do know now, that is a distinctly stupid idea. It is useful, but still stupid because Windows does not want you to do that and in a fight between you and Windows, you find out why many call it “Win”.
Well, maybe that is too much credit; Windows has actually no clue that you moved the application file; the next time it needs the application it will blindly look in the old place and fail miserably with a cryptic message. After having too many of those cryptic message, I decided to reformat and reinstall Windows. I learned my lesson and after that occasion I never cleaned up anymore (My wife sometimes has a problem with this behavior, but fortunately she is significantly more benign than Windows).
So what is this rant doing in an OSGi related blog? Well, there is a lesson that we learned early in the OSGi: installers are evil. This is easy to say, but I will try to make it clear to you. Let us analyze why it became so painful to move these files around on a PC and not on a Mac. On the Mac, an application is a single file (on System X, it is actually a directory). This file is a simple database that consists of the code resources and other resources like configuration files, icons, html pages, properties, translations, etc. Anything that the program needs is stored in this single file. This file is introspective; this means that other programs can inspect the contents and react according to this content. We call these progams/utilities that react to this content extensions. Extensions are extremely useful.
Application files have a special type in the Mac, and this type is recognized by the disk operating system. Whenever you copy or move an application file the OS will track this in an application database. The OS actually looks in the application file's database and finds the information about the files it can open and other useful aspects. Ergo, you can move the applications around as much as you want; the OS just finds them when you need them. It has this genuine advantage that if you trash the application file, you really cleaned up all aspects of that application.
So why does Windows fail so miserably when you move an application to another directory? Well, Windows does not track the applications because it has installers! An installer is an ingenious piece of software that most PC users think is a necessity of nature. It is usually zipped and then compressed after which it is squeezed, requires a lot of thinking, asks you questions you have no clue how to answer or could not care less (fortunately there are defaults), shows lines that you cannot read because they are replaced too quick (which at least gives you the cozy feeling something is happening), splatters files all over your hard disk (sometimes in quite unexpected places), overwrites important DLLs, then spends minutes updating the registry, after which it asks you to read a README file that is incomprehensible until you need it because the application balks at you; unfortunately at that time you have no clue where to find it anymore. And if this all went well, you can run the application. The fact that this is all overkill is demonstrated by Eclipse that has no installer.
You would expect that for security reasons Windows would have a registry of these installed applications, and a way to forcefully uninstall an application. Well, you thought wrong. There is something of a voluntary registry that the installers can fill in if they feel like it. However, the information is not guaranteed by Windows. Worse, to uninstall an application Windows will ask the original installer to remove all the files, remove the registry changes, and generally clean up. The uninstaller sometimes fails because a file is missing or something else is wrong. At such a moment you have an orphan application that likely remains as a bit-squatter until you have to reformat your hard disk. Lots of these orphans can bring that time significantly closer.
When we designed the OSGi framework we were sure that the Windows Installer model was not where we wanted to go; the Mac model looked a lot more attractive. Fortunately, the JAR file model of Java is an even more powerful format than the Mac resource file. The ZIP file format is flexible enough to allow all kinds of application extension scenarios based on resource name and directory. However, due to the security risk attached to running applications we decided to strictly control all the aspects of the installation, update, and uninstallation; all these are primitive in the OSGi, there is no untrusted code involved. OSGi applications (bundles) can normally not move files to odd places and modify all kinds of settings. OSGi applications can declare information in their JAR files and other applications or utilities can inspect this information and act upon it. This is a clear example of Inversion of Control or IoC: “Don't call us, we'll call you!” For example, an application using Declarative Services contains an XML file with the proper setup. When the application is installed the Declarative Service implementation will find this file, the application does not have to register.
This is an incredibly robust model that prevents a lot of abuse and errors. As a simple example, assume an application fails all the time and you want to uninstall it. The fact that the application fails makes it very likely that the uninstaller will also fail. Leaving remnants in lots of places on your system. An OSGi application can always be uninstalled by the Framework. The uninstallation will notify any applications that performed some function on behalf of the uninstalled application. These other applications are in a good state and can therefore cleanup correctly, regardless of the state of the culprit.
The application model of the OSGi has learned from its predecessors and provides excellent characteristics. Most importantly: An overview what you have really installed and a guarantee that you can always throw anything away.No code can run without the Framework being aware of it.
Every time when new groups enter the OSGi Alliance we have the Installer discussion again. Every time people ask me how it can be wrong when Windows does it that way? 100 million users can't be wrong, can they? I leave that up to you to figure it out.
Peter Kriens
Subscribe to:
Posts (Atom)