Update: JSR 232 is approved in the final ballot!
Last week I visited the BEA offices for a meeting with the Spring-OSGi group. This project started a few months ago when Interface21 (the shepherds of Spring), specifically Adrian Colyer, contacted us to see if we could make the Spring- and OSGi technology compatible. At first I had to look a bit deeper into Spring because I had only heard of the technology. I quickly found out that it is highly popular in a diverse range of Java applications. So what does Spring do?
The core responsibility of Spring is configuration. One of the key software lessons our industry learned in the last decade is that you want your software modules to be as cohesive and uncoupled as possible. The more uncoupled software is, the easier it is to reuse it. However, in the end you need some place where the modules are wired together and their options set. It is a like squeezing a balloon, you can never get rid of the coupling, you can only concentrate it in as few places as possible. This is a similar model that is promoted for OSGi applications; your bundle activator should configure a number of POJOs that finally do the real work. Spring's reason for being is based on this software model.
The Spring core interprets an XML file that describes how a set of beans must be wired together. A bean is a POJO that is configured with get and set methods. Beans are accessed using reflection but can optionally participate by implementing a number of Spring interfaces. The Spring XML looks a lot like a script language (something XML is extremely ill suited for, but that is another story, and yes, I know there is an alternative) but actually creates a graph and can therefore correctly handle dependencies and ordering.
On top of this configuration code, the Spring library has a large number of nicely decoupled libraries that can be used to decorate the beans. Do you need for JTA transaction support, but your POJO did not know anything about JTA? No problem, with Spring you can provide the required transaction calls around your methods without changing the source code. Spring uses two forms of aspect oriented programming to achieve the decoration. You can use events around the method calls defined in an interface, or you can use full blown AspectJ programming. There is also a library that provides an shell around JDBC, removing any implementation intricacies. And there is of course much, much, more. Overall pretty cool stuff.
Oracle and BEA were also very interested in providing Spring capabilities on top of the OSGi Service Platform. Hal Hildebrand (Oracle) and Andy Piper (BEA) were part of the team doing the work. On the mailing list, Richard Hall and Jeff McAffer also tuned in. With these people, you can be guaranteed of lots of interesting discussions on the Spring-OSGi mailing list; I learned a lot about features required to integrate larger enterprise applications. There is a lot of messy code out there! It is amazing how many home grown mechanisms application programmers have invented to get some kind of extensibility. Many of these mechanisms have peculiar ideas about class loading that do not always mesh well with the OSGi rather strict rules. One of the key focus of OSGi R5 will likely be addressing how this legacy code will run unmodified on OSGi service platforms.
A couple of weeks ago I decided to plummet in the deep and see what all the fuzz was all about so I wrote a simple application using the Spring-OSGi prototype. Adrian had written a very nice draft of a spec so it was not hard to get started. After a couple of hours the application was running and I seemed to be able to use many of the facilities. From an OSGi perspective, I had some qualms. I thought some of the solutions were to cumbersome and there was too much choice in certain areas. I wrote a review report from my experiences, which I had to redo because it got lost in the mail ... I think the project was going very well but I felt we needed to do a hack session together. I did not have enough Spring experience to change their code, and I think they could use some OSGi experience to improve their code. We therefore decided to meet in London. Olivier Gruber and BJ Hargrave from IBM also attended because of their
OSGi expertise and interest in Spring.
The group had become a bit too big to do any real hacking but that turned out to be ok. We spent most of the first day going through Adrian's excellent specification text. We were incredibly productive that day, we killed so many features! With lots of intense discussions we could significantly simplify the model. It turned out that the OSGi service registry and the Spring application context could really work together very well. The tricky part is of course the dynamics, the OSGi specifications seem to remain unique in their focus on dynamicity. Interestingly, the end result looks like a specification that can take over or merge with our Declarative Services in the long run because it is now a super-set. Another item to discuss for OSGi R5.
Overall, this is a very interesting project that has a lot of industry attention: two very cool technologies that are being married. I hope that the group can turn the prototype into the mainstream Spring code. Adrian told me they were thinking of Spring 2.1. As a bonus, Spring 2.1 will deliver all its libraries as bundles, that is, all libraries will have the correct OSGi headers.
I am convinced that the OSGi Enterprise workshop will discuss this project, for example, should OSGi standardize the OSGi layer of Spring?
Peter Kriens
P.S. You did RSVP did you? Realize that we will have limited seating based on the people that have RSVP'ed. Hope to see you there!
OSGi Blog
Tuesday, August 29, 2006
Wednesday, August 16, 2006
OSGi Enterprise Workshop
It is rare that a technology developed for embedded devices ends up at the core of enterprise applications. The constraints and optimization axes of these two worlds are, ehhh, worlds apart. Still, it looks like the OSGi Alliance is succeeding in this remarkable feat as demonstrated by the increasing interest of enterprise software vendors in OSGi technology. Initially, some vendors voiced concerns about potential patents held by the founding members, but the by now famous patent pledge made those worries go away.
Enterprise software is not a well defined area. We know it is usually big, requires lots of people to work on it, and it is invariably expensive. Web servers are of course a big aspect of Enterprise software as well as tight integration with databases. Java made big inroads in this world because it provided a uniform environment for applications. Its type safety reduced the number of errors during the integration phases and this made it smoother to integrate different applications.
J2EE was SUN’s attempt to capture the communality of Enterprise applications in a framework. The idea behind a framework is that it reduces the application specific code and the resulting application more robust. It is a wonderful appealing idea and object oriented theory gave birth to too many frameworks in the nineties. I even build one for Ericsson in 94! Frameworks are incredibly appealing but they are so hard to get right. They key problem is the natural focus on the advantages of the framework and total disregard for the increased complexity and learning curve that a framework will cause. For example, if we add a function for authentication, we needlessly increase the complexity for people that do not need authentication at all. Just the increase in the API and manual is negative. The battle field of software is therefore littered with failed frameworks (alas, including mine). Only frameworks that can restrict themselves to core functionality and fanatical focus on decoupling stand a chance to succeed.
I do not think SUN and compatriots heeded those lessons from the nineties when they started with J2EE. From the start, J2EE became a monster in deployment complexity and intrusiveness. 5 years ago I bought a book about J2EE and spent a week developing applications, just to understand what the hype was. Well, I utterly disliked the intrusiveness of the code and the many, seemingly unnecessary, restrictions. This dislike happened ever before I found out about the XML situps that one needs to perform before you get your “Hello World” running. Look at the funny comparison between web application development environments from JPL to demonstrate my dislike. I understand that the EJB 3 JSR is providing more focus on simplicity, but I will hold my judgment until I have played with it.
The success of Spring is caused by the intrusiveness of J2EE into applications and its inclusion of many technologies that are not always necessary, or in other words, the classic Framework syndrome. Developers love frameworks but they are not very faithful; this is where the POJOs are coming from. POJOs are successful because the coupling between the units of design are minimized, but more important, they are coupled in the right direction. Let the framework call you, not you calling the framework. The result is that the developers can focus on their domain area, not on learning Yet Another Framework.
The OSGi Framework is successful because it was designed to decouple, which provides many benefits during development and deployment. The OSGi Framework could not care less about what your components do and how they do it. It only provides the primitives to deploy your components and to let them collaborate in a dynamic way. Collaboration is done through service interfaces, which makes the components easy to test without complicated harnesses. Components can be build on top of POJOs, very little code has to be coupled to the OSGi Framework. Standardizing this component layer will create universal middleware. Middleware developed in different industries that can be deployed on any platform. In the end, application development will be mixing and matching universal middleware.
The OSGi development model is so attractive to the Enterprise world because it is sometimes amazing how often I see the wheels being reinvented. Often just because the existing wheels were connected to axes that the developer had no use for. Though the OSGi Framework is quite mature for the embedded, vehicle, home automation, and mobile markets, we need to work on the issues that are crucial for the enterprise markets. What functionality is missing in R4 that is important for enterprise applications?
Next month I will lead an OSGi workshop in the bay area where we will discuss these issues with a number of experts. This is an open workshop but seats are limited. If your company is into making Enterprise applications, please attend (register ASAP with rranck@inventures.com, we have limited number of seats). The signs are that OSGi technologies will play a major role in server side applications over the coming years. If you can not attend the workshop, do not hesitate to add comments to this blog or just email me.
Peter Kriens
Enterprise software is not a well defined area. We know it is usually big, requires lots of people to work on it, and it is invariably expensive. Web servers are of course a big aspect of Enterprise software as well as tight integration with databases. Java made big inroads in this world because it provided a uniform environment for applications. Its type safety reduced the number of errors during the integration phases and this made it smoother to integrate different applications.
J2EE was SUN’s attempt to capture the communality of Enterprise applications in a framework. The idea behind a framework is that it reduces the application specific code and the resulting application more robust. It is a wonderful appealing idea and object oriented theory gave birth to too many frameworks in the nineties. I even build one for Ericsson in 94! Frameworks are incredibly appealing but they are so hard to get right. They key problem is the natural focus on the advantages of the framework and total disregard for the increased complexity and learning curve that a framework will cause. For example, if we add a function for authentication, we needlessly increase the complexity for people that do not need authentication at all. Just the increase in the API and manual is negative. The battle field of software is therefore littered with failed frameworks (alas, including mine). Only frameworks that can restrict themselves to core functionality and fanatical focus on decoupling stand a chance to succeed.
I do not think SUN and compatriots heeded those lessons from the nineties when they started with J2EE. From the start, J2EE became a monster in deployment complexity and intrusiveness. 5 years ago I bought a book about J2EE and spent a week developing applications, just to understand what the hype was. Well, I utterly disliked the intrusiveness of the code and the many, seemingly unnecessary, restrictions. This dislike happened ever before I found out about the XML situps that one needs to perform before you get your “Hello World” running. Look at the funny comparison between web application development environments from JPL to demonstrate my dislike. I understand that the EJB 3 JSR is providing more focus on simplicity, but I will hold my judgment until I have played with it.
The success of Spring is caused by the intrusiveness of J2EE into applications and its inclusion of many technologies that are not always necessary, or in other words, the classic Framework syndrome. Developers love frameworks but they are not very faithful; this is where the POJOs are coming from. POJOs are successful because the coupling between the units of design are minimized, but more important, they are coupled in the right direction. Let the framework call you, not you calling the framework. The result is that the developers can focus on their domain area, not on learning Yet Another Framework.
The OSGi Framework is successful because it was designed to decouple, which provides many benefits during development and deployment. The OSGi Framework could not care less about what your components do and how they do it. It only provides the primitives to deploy your components and to let them collaborate in a dynamic way. Collaboration is done through service interfaces, which makes the components easy to test without complicated harnesses. Components can be build on top of POJOs, very little code has to be coupled to the OSGi Framework. Standardizing this component layer will create universal middleware. Middleware developed in different industries that can be deployed on any platform. In the end, application development will be mixing and matching universal middleware.
The OSGi development model is so attractive to the Enterprise world because it is sometimes amazing how often I see the wheels being reinvented. Often just because the existing wheels were connected to axes that the developer had no use for. Though the OSGi Framework is quite mature for the embedded, vehicle, home automation, and mobile markets, we need to work on the issues that are crucial for the enterprise markets. What functionality is missing in R4 that is important for enterprise applications?
Next month I will lead an OSGi workshop in the bay area where we will discuss these issues with a number of experts. This is an open workshop but seats are limited. If your company is into making Enterprise applications, please attend (register ASAP with rranck@inventures.com, we have limited number of seats). The signs are that OSGi technologies will play a major role in server side applications over the coming years. If you can not attend the workshop, do not hesitate to add comments to this blog or just email me.
Peter Kriens
Thursday, August 3, 2006
Why is Software So Brittle?
The past few days I acted as a software user instead of software programmer: I needed a wiki for an OSGi project. How hard can that be? After a bit of Googling I found Ward Cunningham’s site (he is the Wiki inventor) and he is hosting a wiki about wikis (This text will have a weird number of w words). Well, that site contained a very long list of wikis! I guess a wiki is on the sweet spot of being relative easy to make but with a big visual impact.
How should one choose from so many wikis? Silly enough, the implementation language seemed to be the most logical selection criteria. As you might have guessed, Java is one of my favorite languages and was therefore an obvious choice. Eagerly checking the list for OSGi based implementations turned out, however, to be a disappointment. Many wikis point out how extensible they are but none had chosen the OSGi service platform to implement their plugins; they all developed plugins in a proprietary way. There is a lot of evangelizing left to do.
Anyway, after looking at some Java implementations I found a wiki that looked very promising: FitNesse. This is not a simple wiki, it also contains an extensive testing framework. However, the wiki looked really good (not a common feature for a lot of wikis) and the code looked interesting. When I inspected the code, my hands itched … if I had been a wise men, I would have slapped them and continued. The code looked so clean and so easy to bundlefy that I could not resist. Turning it into a bundle was indeed trivial; the makers of FitNesse provided a very clean configuration and start/stop interface. I only had a slight problem with resource loading. They used the system class loader for resource loading instead of their own class loading, can’t see why, but if that was all I could fix it. Still optimistic at that moment in time.
We have an OSGi framework running that hosts the Bundle Repository, so it was attractive to install this wiki bundle on the same machine, sharing the same VM for these applications is a serious memory saver. The FitNesse bundle ran its own web server (Oh, why don’t they use OSGi technology?). This required me to map their web server through our standard web server so that we could handle authentication in one place. The standard OBR web server runs behind a firewall but is proxied through our normal web server. After trying this same trick for the wiki, it turned out that the FitNesse wiki code assumed it had the whole web server on its own; it could not handle a prefix path. I was getting this sinking feeling that I might have taken the wrong path. As a solution, I could of course run the web server directly out in the open. Unfortunately, this would require me to write the authentication code. The sinking feeling changed into hitting rock bottom. Maybe it was time to turn around and see if there were easier ways of getting this done. If I only had been a wise man …
I talked to BJ Hargrave and he recommended the MediaWiki, the wiki that powers the WikiPedia. This wiki is PHP based and run by hundreds of sites. How hard can it be to install this? Well, hard. It turned out we were running PHP 4 on our server and MediaWiki runs on PHP 5. Sigh. Well, we have rpm installed on our Linux system so we just download PHP 5 and we are on our way. Well, that is the theory. The sinking feeling returned. PHP 5 required a long list of additional uploads. Carefully trying to install a few dependent packages quickly showed that the dependency tree closely resembled the big bang. BJ had tole me there was also a PHP 4 version of the MediaWiki so I felt myself floating to the surface again, only to sink at record speed again after installing this version. Yes, it requires version 4, which happened to be a different version 4 than our version 4. The dependency fan out was not as enormous as the PHP 5, but still impressive enough for me to give up.
Simple and small became my main criteria now. So I finally found a simple and very small PmWiki based on PHP that actually had the functionality I needed, and nothing more. It was a breeze to install and did not have any nasty dependencies.
So what did I learn? Well, besides not trying to be clever, I learned that we are an industry with a problem. The last couple of years we have spent a tremendous amount of energy in formalizing the dependencies of our software artifacts; assuming that this would solve the problems of unreliable software. However, it looks like we have created another problem which is the dependency fan out. Unfortunately, virtually all known dependency mechanisms are based on versioned artifacts. Artifacts are usually a collection of functions that are not always needed. Often there is a core of highly coupled packages and a periphery of utilities and mappers to other artifacts. For example, many tool vendors today provide a mapper to run their tools under ant or maven; implicitly coupling their artifact to maven and ant.
We need to make the use of software less painful. Though we can extend our current way of working with big-bang releases (integrating many different products in a single build), in the end we need to find a way that minimizes dependencies between artifacts. The OSGi service registry with its service oriented architecture is clearly in the right direction. However, we need more. Tools that can visualize the dependencies and that can help us reduce the dependencies. Yes, with today’s tools we can actually manage the dependencies, but that does not imply that dependencies have become a positive thing. I think the OSGi is on the right way, but a lot of work is left to do.
How should one choose from so many wikis? Silly enough, the implementation language seemed to be the most logical selection criteria. As you might have guessed, Java is one of my favorite languages and was therefore an obvious choice. Eagerly checking the list for OSGi based implementations turned out, however, to be a disappointment. Many wikis point out how extensible they are but none had chosen the OSGi service platform to implement their plugins; they all developed plugins in a proprietary way. There is a lot of evangelizing left to do.
Anyway, after looking at some Java implementations I found a wiki that looked very promising: FitNesse. This is not a simple wiki, it also contains an extensive testing framework. However, the wiki looked really good (not a common feature for a lot of wikis) and the code looked interesting. When I inspected the code, my hands itched … if I had been a wise men, I would have slapped them and continued. The code looked so clean and so easy to bundlefy that I could not resist. Turning it into a bundle was indeed trivial; the makers of FitNesse provided a very clean configuration and start/stop interface. I only had a slight problem with resource loading. They used the system class loader for resource loading instead of their own class loading, can’t see why, but if that was all I could fix it. Still optimistic at that moment in time.
We have an OSGi framework running that hosts the Bundle Repository, so it was attractive to install this wiki bundle on the same machine, sharing the same VM for these applications is a serious memory saver. The FitNesse bundle ran its own web server (Oh, why don’t they use OSGi technology?). This required me to map their web server through our standard web server so that we could handle authentication in one place. The standard OBR web server runs behind a firewall but is proxied through our normal web server. After trying this same trick for the wiki, it turned out that the FitNesse wiki code assumed it had the whole web server on its own; it could not handle a prefix path. I was getting this sinking feeling that I might have taken the wrong path. As a solution, I could of course run the web server directly out in the open. Unfortunately, this would require me to write the authentication code. The sinking feeling changed into hitting rock bottom. Maybe it was time to turn around and see if there were easier ways of getting this done. If I only had been a wise man …
I talked to BJ Hargrave and he recommended the MediaWiki, the wiki that powers the WikiPedia. This wiki is PHP based and run by hundreds of sites. How hard can it be to install this? Well, hard. It turned out we were running PHP 4 on our server and MediaWiki runs on PHP 5. Sigh. Well, we have rpm installed on our Linux system so we just download PHP 5 and we are on our way. Well, that is the theory. The sinking feeling returned. PHP 5 required a long list of additional uploads. Carefully trying to install a few dependent packages quickly showed that the dependency tree closely resembled the big bang. BJ had tole me there was also a PHP 4 version of the MediaWiki so I felt myself floating to the surface again, only to sink at record speed again after installing this version. Yes, it requires version 4, which happened to be a different version 4 than our version 4. The dependency fan out was not as enormous as the PHP 5, but still impressive enough for me to give up.
Simple and small became my main criteria now. So I finally found a simple and very small PmWiki based on PHP that actually had the functionality I needed, and nothing more. It was a breeze to install and did not have any nasty dependencies.
So what did I learn? Well, besides not trying to be clever, I learned that we are an industry with a problem. The last couple of years we have spent a tremendous amount of energy in formalizing the dependencies of our software artifacts; assuming that this would solve the problems of unreliable software. However, it looks like we have created another problem which is the dependency fan out. Unfortunately, virtually all known dependency mechanisms are based on versioned artifacts. Artifacts are usually a collection of functions that are not always needed. Often there is a core of highly coupled packages and a periphery of utilities and mappers to other artifacts. For example, many tool vendors today provide a mapper to run their tools under ant or maven; implicitly coupling their artifact to maven and ant.
We need to make the use of software less painful. Though we can extend our current way of working with big-bang releases (integrating many different products in a single build), in the end we need to find a way that minimizes dependencies between artifacts. The OSGi service registry with its service oriented architecture is clearly in the right direction. However, we need more. Tools that can visualize the dependencies and that can help us reduce the dependencies. Yes, with today’s tools we can actually manage the dependencies, but that does not imply that dependencies have become a positive thing. I think the OSGi is on the right way, but a lot of work is left to do.
Wednesday, July 26, 2006
Interesting times
After some slow (hot!) summer weeks there is suddenly a lot of activity. The best news so far is the patent pledge that the OSGi Alliance has made this week. Five key members of the OSGi Alliance have promised not to sue anybody that implements the release 4 spec for patent infringement as long as the patent is necessary for the implementation. Patents are a necessity but software patents can be bizarre. With over 6 million approved patents, no programmer can claim that he is writing code that does not infringe on some patents. If you are a really big company the problem is not that serious. Once Microsoft knocks on the door of Sun Microsystems it usually ends with an exchange of patents and a deal not to sue each other. As long as you are small, the costs of suing you are less than any potential royalties so small companies are also relatively safe. However, when you grow you can suddenly find some lawyers knocking on your door, potentially eating away a large part of your well deserved fortune. Look at Research in Motion (RIM) of the Blackberry that was forced to pay 600 million dollar for a patent that was suspect at the least.
The patent pledge that was made by the 5 OSGi members has largely removed this potential booby trap when you implement OSGi specifications. And in my opinion, that is exactly the way standards organizations should work. Participants in the OSGi ecosystem should all gain by having more adoption of OSGi technology because it grows the market; something we all can take advantage of. It never was the intention of the OSGi members to become rich on royalties; this pledge has made that crystal clear.
Less positive this week was the vote for JSR 298 by the J2ME Executive Committee. They decided to approve a JSR that is right in OSGi Alliance’s backyard: Telematics. I must not have paid attention because I thought this JSR was stillborn. In May it was voted down by the EC, I have missed the reconsideration ballot of this week.
Why is this JSR 298 bad? Lets take a look. First it states that “OSGi could be too heavy”. Sigh. First, OSGi technology is extremely lean for what it does. Second, it was designed to build applications out of managed middleware. This model allows applications to be lean and mean because they rely on provided middleware. MIDP and other J2ME models have no concept of shared libraries and therefore require a choice between installing it on the platform via an undefined way (growing t the platform) or including it in your application (growing your application). Due to the sharing model, moderately complex OSGi applications are usually much smaller than their brethren on other J2ME application environments.
Now let us step back for a second. Flash memory costs 2 cts a megabyte today in large quantities. What are we talking about? An OSGi R4 framework can be implemented in 250K. CPU speed is not an issue because the OSGi framework lets components communicate with only a little bit of setup overhead. I dare to say that the price/performance ratio of an OSGi framework is very hard to beat.
And despite rumors of the contrary, OSGi technology does run on CLDC environments! Not out of the box because the class loaders of CLDC are veiled, but most CLDC VMs can run an OSGi framework with a bit of work. All the OSGi APIs limit themselves to a subset of CLDC and have therefore no probem on CLDC. However, also in this area one should stay realistic. CLDC is a cramped environment that maybe once was necessary because CDC was too big. Since then, processors have become magnitudes more powerful, flash memory has become a gift with cereals in the form of a USB memory stick, and internal memory is also dropping in price. And this trend is bound to continue for the coming years.
There are clearly applications that require the Bill Of Material (BOM) to be squeezed to the last cent. However, one can ask if these are the applications that require standardized Java APIs. The advantage of standardized Java APIs is that software from different parties can run together and collaborate to achieve the desired functionality. Creating a cramped environment is likely to make this goal a lot harder to achieve. I have seen many examples where the penny wise choice of a limited environment turned out to be pound foolish.
Last but not least: security. The OSGi specifications have an extensive and very powerful security model that is almost absent in MIDP; the security in MIDP is very difficult to extend to a complex area as telematics. Software that erroneously sends out a few unwanted short messages is not good but neither is it a disaster. Controlling real world devices like cars is a different story. The JSR is completely silent about security. How come?
I hope this elucidation has put the OSGi technology is heavy argument finally out of the way! Then again, if this is out of the way, the whole reason for JSR 298 goes away. Much of the work that the JSR plans do is already done in the OSGi Vehicle Expert Group. Especially the upcoming VEG release will handle full control of the vehicle using standard protocols like OMA DM as well as application control. And we’ll likely also have an interesting navigation model!
The second ballot for JSR 298 succeeded. Ok, I was not paying attention but if they got a second chance to passthe JSR, couldwe get a second chance to vote it down?
Peter Kriens
The patent pledge that was made by the 5 OSGi members has largely removed this potential booby trap when you implement OSGi specifications. And in my opinion, that is exactly the way standards organizations should work. Participants in the OSGi ecosystem should all gain by having more adoption of OSGi technology because it grows the market; something we all can take advantage of. It never was the intention of the OSGi members to become rich on royalties; this pledge has made that crystal clear.
Less positive this week was the vote for JSR 298 by the J2ME Executive Committee. They decided to approve a JSR that is right in OSGi Alliance’s backyard: Telematics. I must not have paid attention because I thought this JSR was stillborn. In May it was voted down by the EC, I have missed the reconsideration ballot of this week.
Why is this JSR 298 bad? Lets take a look. First it states that “OSGi could be too heavy”. Sigh. First, OSGi technology is extremely lean for what it does. Second, it was designed to build applications out of managed middleware. This model allows applications to be lean and mean because they rely on provided middleware. MIDP and other J2ME models have no concept of shared libraries and therefore require a choice between installing it on the platform via an undefined way (growing t the platform) or including it in your application (growing your application). Due to the sharing model, moderately complex OSGi applications are usually much smaller than their brethren on other J2ME application environments.
Now let us step back for a second. Flash memory costs 2 cts a megabyte today in large quantities. What are we talking about? An OSGi R4 framework can be implemented in 250K. CPU speed is not an issue because the OSGi framework lets components communicate with only a little bit of setup overhead. I dare to say that the price/performance ratio of an OSGi framework is very hard to beat.
And despite rumors of the contrary, OSGi technology does run on CLDC environments! Not out of the box because the class loaders of CLDC are veiled, but most CLDC VMs can run an OSGi framework with a bit of work. All the OSGi APIs limit themselves to a subset of CLDC and have therefore no probem on CLDC. However, also in this area one should stay realistic. CLDC is a cramped environment that maybe once was necessary because CDC was too big. Since then, processors have become magnitudes more powerful, flash memory has become a gift with cereals in the form of a USB memory stick, and internal memory is also dropping in price. And this trend is bound to continue for the coming years.
There are clearly applications that require the Bill Of Material (BOM) to be squeezed to the last cent. However, one can ask if these are the applications that require standardized Java APIs. The advantage of standardized Java APIs is that software from different parties can run together and collaborate to achieve the desired functionality. Creating a cramped environment is likely to make this goal a lot harder to achieve. I have seen many examples where the penny wise choice of a limited environment turned out to be pound foolish.
Last but not least: security. The OSGi specifications have an extensive and very powerful security model that is almost absent in MIDP; the security in MIDP is very difficult to extend to a complex area as telematics. Software that erroneously sends out a few unwanted short messages is not good but neither is it a disaster. Controlling real world devices like cars is a different story. The JSR is completely silent about security. How come?
I hope this elucidation has put the OSGi technology is heavy argument finally out of the way! Then again, if this is out of the way, the whole reason for JSR 298 goes away. Much of the work that the JSR plans do is already done in the OSGi Vehicle Expert Group. Especially the upcoming VEG release will handle full control of the vehicle using standard protocols like OMA DM as well as application control. And we’ll likely also have an interesting navigation model!
The second ballot for JSR 298 succeeded. Ok, I was not paying attention but if they got a second chance to passthe JSR, couldwe get a second chance to vote it down?
Peter Kriens
Monday, July 17, 2006
To Include Or Not To Include
I am always curious if other engineering disciplines can vehemently argue so much about fundamentals as we do? Do bridge engineers have heavy discussions about how much concrete is needed for a bridge pillar? Can they get into heated arguments if a skyscraper needs I reinforcement or not? In the information industry we can differ about the most fundamental approaches, everybody is an expert. Where are our universities testing the approaches and telling is what works and what does not? How could we more easily test ideas and decide about their value?
Why do I muse about this? Last week I had an argument with some experts that we could not decide. I lost because the owner of the problem decided to go another way while I strongly believed he was wrong. I’ll explain the problem and then you can figure out how to decide what the better approach should have been.
A couple of weeks ago I had to give a tutorial at ApacheCon Europe 2006. Obviously, I had to adapt my tutorial to use Apache Felix instead of Eclipse Equinox so I would not insult my hosts (or undergo the wrath of Richard Hall). Alas, Apache Felix does not yet have an implementation of declarative services. I therefore looked at several open source implementations and picked one to port. Declarative services require a framework specific implementation because it needs the Bundle Context of other bundles and there is no public function to do this in the specification. Bad, but we could not figure out in R4 how to do this in a secure way without using security. Now we know that you can not do secure things when security is not on, but that is another story.
So porting the declarative services was straightforward except for one snatch. To parse the Service-Component manifest header, the implementation used a utility function, which turned out to be implemented in the framework JAR. Not good. Looking at the parser, I decided that the easy way out was to write a simple replacement parser; the syntax for the Service-Component header is quite simple. The replacement code added about 6 lines to the implementation, making the declarative services bundle run on Felix without requiring any other supporting bundles.
After the tutorial I decided to submit my change as a patch, trying to be a good citizen. Great was my surprise when I started getting pushback. First, they felt that the parser was too simplistic; it ignored empty paths and did not detect syntax errors in the attributes. Personally that is fine by me, parsers should be lenient (though not creating erroneous values) and generators should be strict. However, this is again another story, many people feel more comfortable with rigid parsers for diagnostic reasons.
So I then redid the parser, discovering that the Service-Component header did not even support attributes! I threw the right exceptions on empty paths and discovered that Service-Component allowed quotes! Interestingly, the original manifest parser was thus not suited for this header at all.
After submitting my second patch, my expectation to be the hero now was again not honored. They still did not like it. Why? There was now a redundancy in the system; there were now two manifest parsers (despite that the original was wrong). They were therefore not willing to accept my patch. Instead, they will update their central manifest parser to support the Service-Component header quotes and reject attributes. Why? They did not like the redundancy.
This obviously did not fix my coupling problem whatsoever. I am a strong believer in least amount of coupling. In the OSGi build system I have addressed exactly this problem by copying the class files of utilities from the class path into the JAR; by making the packages private I prevent any version clashes. This gives me a single source without lots of utility bundles. The disadvantage is of course that if you find a bad bug, you must update all the bundles that contain that code. In my experience this is rarely much different from a shared bundle. State of the art version handling is so brittle that it is likely that all dependent bundles require an update to make the system resolve with the new utility bundle. And even if they require an update, the difference between updating one bundle or several bundles is not that big. It can even be questioned if you want to update the dependent bundles, often they do not really need the bug fix.
However, in the manifest parser case, I would personally gladly have accepted the source code redundancy; the new “improved” parser was only 15 lines. Yes, it is redundant, but the chance that you have an error in this code is pretty minute after testing and review.
And this is the point I want to discuss after this long introduction. We must balance redundancy (bad) versus coupling between bundles (bad). Redundancy is bad because it means we sometimes have to fix bugs or make improvements in multiple places. Coupling is bad because it makes the deployment situation more complex. The fact that today we are starting to handle dependencies does not mean dependencies have become benign. They can still bite you unexpectedly and mean. So how do we balance two bads? How do we decide which is the least bad?
Peter Kriens
Why do I muse about this? Last week I had an argument with some experts that we could not decide. I lost because the owner of the problem decided to go another way while I strongly believed he was wrong. I’ll explain the problem and then you can figure out how to decide what the better approach should have been.
A couple of weeks ago I had to give a tutorial at ApacheCon Europe 2006. Obviously, I had to adapt my tutorial to use Apache Felix instead of Eclipse Equinox so I would not insult my hosts (or undergo the wrath of Richard Hall). Alas, Apache Felix does not yet have an implementation of declarative services. I therefore looked at several open source implementations and picked one to port. Declarative services require a framework specific implementation because it needs the Bundle Context of other bundles and there is no public function to do this in the specification. Bad, but we could not figure out in R4 how to do this in a secure way without using security. Now we know that you can not do secure things when security is not on, but that is another story.
So porting the declarative services was straightforward except for one snatch. To parse the Service-Component manifest header, the implementation used a utility function, which turned out to be implemented in the framework JAR. Not good. Looking at the parser, I decided that the easy way out was to write a simple replacement parser; the syntax for the Service-Component header is quite simple. The replacement code added about 6 lines to the implementation, making the declarative services bundle run on Felix without requiring any other supporting bundles.
After the tutorial I decided to submit my change as a patch, trying to be a good citizen. Great was my surprise when I started getting pushback. First, they felt that the parser was too simplistic; it ignored empty paths and did not detect syntax errors in the attributes. Personally that is fine by me, parsers should be lenient (though not creating erroneous values) and generators should be strict. However, this is again another story, many people feel more comfortable with rigid parsers for diagnostic reasons.
So I then redid the parser, discovering that the Service-Component header did not even support attributes! I threw the right exceptions on empty paths and discovered that Service-Component allowed quotes! Interestingly, the original manifest parser was thus not suited for this header at all.
After submitting my second patch, my expectation to be the hero now was again not honored. They still did not like it. Why? There was now a redundancy in the system; there were now two manifest parsers (despite that the original was wrong). They were therefore not willing to accept my patch. Instead, they will update their central manifest parser to support the Service-Component header quotes and reject attributes. Why? They did not like the redundancy.
This obviously did not fix my coupling problem whatsoever. I am a strong believer in least amount of coupling. In the OSGi build system I have addressed exactly this problem by copying the class files of utilities from the class path into the JAR; by making the packages private I prevent any version clashes. This gives me a single source without lots of utility bundles. The disadvantage is of course that if you find a bad bug, you must update all the bundles that contain that code. In my experience this is rarely much different from a shared bundle. State of the art version handling is so brittle that it is likely that all dependent bundles require an update to make the system resolve with the new utility bundle. And even if they require an update, the difference between updating one bundle or several bundles is not that big. It can even be questioned if you want to update the dependent bundles, often they do not really need the bug fix.
However, in the manifest parser case, I would personally gladly have accepted the source code redundancy; the new “improved” parser was only 15 lines. Yes, it is redundant, but the chance that you have an error in this code is pretty minute after testing and review.
And this is the point I want to discuss after this long introduction. We must balance redundancy (bad) versus coupling between bundles (bad). Redundancy is bad because it means we sometimes have to fix bugs or make improvements in multiple places. Coupling is bad because it makes the deployment situation more complex. The fact that today we are starting to handle dependencies does not mean dependencies have become benign. They can still bite you unexpectedly and mean. So how do we balance two bads? How do we decide which is the least bad?
Peter Kriens
Tuesday, July 11, 2006
Eclipse PDE versus JDE/ant
Last week there was a discussion about Eclipse Plugin Development Environment (PDE) on the equinox developers list. As you likely know, not only is Eclipse based on the OSGi Service Platform, it also provides support for developing bundles. They call them plugins, but they are really supposed to be bundles. I use the Eclipse Java Development Environment (JDE) for all my work, including the OSGi build with specification, reference implementations, and test suites. The build contains a total of more than 300 bundles files, most of them bundles, in more than 130 projects. There are bigger builds, but this is a significant project by my metrics.
The PDE discussion evolved around the flexibility of the environment. It got started by a request from the maven people to make the location of the manifest flexible. This touched a nerve with me because it is right at the heart why I am not using the PDE. I unfortunately still use the normal JDE because the PDE does not cut it for me. That is, my bundles are still bundles and have not moved up the evolution chain to become plugins. I have tried it many times but they always seem to revert to my home brewn ant/btool solution. Honestly, I tried.
The reason why I am not using the PDE architecture is because it contains a number of harsh constraints. My key problems are:
The reason for these (serious) architectural constraints is that Eclipse must be able to run your code at a whiff at any time. “No build” is the mantra. If you click Debug, Eclipse starts the OSGi framework, and runs your bundle from the project directory, without wrapping the content in a JAR. Well almost, the bin directory is mapped on the class path despite the fact that it is not really a member of the bundle class path, but hey it works! Then again, I think this quick edit-debug cycle can also be achieved with less draconian constraints.
Despite my complaints, rants, or other kinds of obnoxious behavior with the Eclipse guys, I did not get one iota further. I have been going through Eclipse 3.0, 3.1, and 3.2 without making even a dent in this model. Interestingly, I thought my dislike was generally shared by many but in the Equinox mailing list discussion there was a response that intrigued me:
We seem to have the same dislike of spaghetti, but we completely differed in the approach to avoid it. The red thread in my professional career has been decoupling, decoupling, and decoupling. Last week I stumbled on a course I had given in the early nineties and I was surprised how little I had moved forward in this area. I still strongly feel that the coupling of deliverables (bundles or other JARs) is an architectural decision, and not an ad-hoc design choice or afterthought. Today we have lots of tools to manage dependencies (maven, OSGi, etc.), but no-dependency is still much better than a managed dependency. This implies that adding a dependency like a library is a decision that needs to be taken by the architect of the project, not any developer. I fail to see a problem with “Organize Imports”, as long as the scope is defined by the project. Incidentally, this is the way the JDE works. You need to specifically add a JAR to the build path. Once it is on the build path, developers can (and should) use it as much as possible. Just like a pregnancy, there is no such thing is a little bit of coupling; it is a yes/no decision. If you use it once, you might as well touch it often if it saves you time. During the development, developers should not have to be concerned about what they use of the libraries that are made available to them.
However, next comes the packaging. The PDE can only have one JAR file and the content is defined by my project directory layout (sort of). I do not know how other people are building bundles, but most bundles I am responsible for require some packaging tricks.
For example, sometimes I use a package providing some util function. I really do not want to create a dependency on an evil util bundle. I also do not want to copy the source code. So in quite a few projects I copy the byte codes into my bundle and make the package bundle private. This way I combine a single source without adding additional dependencies.
Another example is the OSGi test case. An OSGi test case consists of a set of inner bundles that are loaded in the target and then perform their testing. This requires me to wrap up bundles in bundles. A similar structure is the deliverable for all the reference implementations. This is a single bundle that installs a set of other bundles, which it contains. I also often need to deliver the same code in different forms, for example in a midlet and a bundle.
I also find that I use the same information items in lots of different places. It is surprising how often you need the bundle symbolic name, or a name derived from it. I use properties to ensure that I have a single definition. I can easily preprocess the manifest, the declarative services XML file, or a readme.txt file. All these tricks are impossible with the PDE, they require some kind of processing before you run the bundle which is unfortunately not available.
And most important, in this phase you can finally see what your real dependencies are. In the OSGi build I calculate the imported packages and regularly inspect the outcome. Quite often, I am surprised what I see and then find out why a specific dependency was introduced. They are usually easy to solve.
So in contrast to the mailing list response, I think I have carefully thought about the dependency issue. Despite reflecting on his remarks, I still think that the current model I use is better than the Eclipse PDE model. Now I only have to find a way so that they listen to me!
Peter Kriens
The PDE discussion evolved around the flexibility of the environment. It got started by a request from the maven people to make the location of the manifest flexible. This touched a nerve with me because it is right at the heart why I am not using the PDE. I unfortunately still use the normal JDE because the PDE does not cut it for me. That is, my bundles are still bundles and have not moved up the evolution chain to become plugins. I have tried it many times but they always seem to revert to my home brewn ant/btool solution. Honestly, I tried.
The reason why I am not using the PDE architecture is because it contains a number of harsh constraints. My key problems are:
- A project = a bundle. It is impossible to deliver multiple artifacts from the same code base with the provided tools. For the OSGi build it would mean 300++ projects.
- The project directory layout must look like the root of the bundle, well eh, sort of. The PDE does perform some magic under the covers before it runs your code or packages the bundle, but the idea is that there is a 1:1 relation.
- The manifest must be up to date at all time, forcing the developer to enter the dependency information in literal form. There is no possibility to generate information for the manifest during build time.
The reason for these (serious) architectural constraints is that Eclipse must be able to run your code at a whiff at any time. “No build” is the mantra. If you click Debug, Eclipse starts the OSGi framework, and runs your bundle from the project directory, without wrapping the content in a JAR. Well almost, the bin directory is mapped on the class path despite the fact that it is not really a member of the bundle class path, but hey it works! Then again, I think this quick edit-debug cycle can also be achieved with less draconian constraints.
Despite my complaints, rants, or other kinds of obnoxious behavior with the Eclipse guys, I did not get one iota further. I have been going through Eclipse 3.0, 3.1, and 3.2 without making even a dent in this model. Interestingly, I thought my dislike was generally shared by many but in the Equinox mailing list discussion there was a response that intrigued me:
I've generally fine with 1 project = 1 bundle, and I like being forced to maintain manifests as I develop my bundles. Your model of pooling all the source together and then sorting out the bundles later sounds messy to me. With no constraints on importing packages, developers just hit "Organise Imports" to pull in dependencies from all over the place, resulting in spaghetti code.
We seem to have the same dislike of spaghetti, but we completely differed in the approach to avoid it. The red thread in my professional career has been decoupling, decoupling, and decoupling. Last week I stumbled on a course I had given in the early nineties and I was surprised how little I had moved forward in this area. I still strongly feel that the coupling of deliverables (bundles or other JARs) is an architectural decision, and not an ad-hoc design choice or afterthought. Today we have lots of tools to manage dependencies (maven, OSGi, etc.), but no-dependency is still much better than a managed dependency. This implies that adding a dependency like a library is a decision that needs to be taken by the architect of the project, not any developer. I fail to see a problem with “Organize Imports”, as long as the scope is defined by the project. Incidentally, this is the way the JDE works. You need to specifically add a JAR to the build path. Once it is on the build path, developers can (and should) use it as much as possible. Just like a pregnancy, there is no such thing is a little bit of coupling; it is a yes/no decision. If you use it once, you might as well touch it often if it saves you time. During the development, developers should not have to be concerned about what they use of the libraries that are made available to them.
However, next comes the packaging. The PDE can only have one JAR file and the content is defined by my project directory layout (sort of). I do not know how other people are building bundles, but most bundles I am responsible for require some packaging tricks.
For example, sometimes I use a package providing some util function. I really do not want to create a dependency on an evil util bundle. I also do not want to copy the source code. So in quite a few projects I copy the byte codes into my bundle and make the package bundle private. This way I combine a single source without adding additional dependencies.
Another example is the OSGi test case. An OSGi test case consists of a set of inner bundles that are loaded in the target and then perform their testing. This requires me to wrap up bundles in bundles. A similar structure is the deliverable for all the reference implementations. This is a single bundle that installs a set of other bundles, which it contains. I also often need to deliver the same code in different forms, for example in a midlet and a bundle.
I also find that I use the same information items in lots of different places. It is surprising how often you need the bundle symbolic name, or a name derived from it. I use properties to ensure that I have a single definition. I can easily preprocess the manifest, the declarative services XML file, or a readme.txt file. All these tricks are impossible with the PDE, they require some kind of processing before you run the bundle which is unfortunately not available.
And most important, in this phase you can finally see what your real dependencies are. In the OSGi build I calculate the imported packages and regularly inspect the outcome. Quite often, I am surprised what I see and then find out why a specific dependency was introduced. They are usually easy to solve.
So in contrast to the mailing list response, I think I have carefully thought about the dependency issue. Despite reflecting on his remarks, I still think that the current model I use is better than the Eclipse PDE model. Now I only have to find a way so that they listen to me!
Peter Kriens
Thursday, June 29, 2006
News from ApacheCon 2006 Europe
This week I attend the ApacheCon Europe in Dublin. Oh boy, what a difference with last year in San Diego at ApacheCon US! Not only is Dublin the opposite of San Diego in many ways, but the visibility and acceptance of OSGi technology by the Apache crowd makes a world of difference. There were several people that were complimenting OSGi technology so much that I’d to blush; some even told me they regularly read this blog!
I primarily attended ApacheCon to give a tutorial. This was an adapted version of the Eclipse tutorial. I had tried to replace Equinox with Apache Felix in this tutorial but had to unfortunately give up late Saturday afternoon because the Declarative Services did not work yet on Felix. When I realized this (again) on Friday, I had tried to port the Eclipse Declarative Services implementation to Felix. Initially this looked very simple; there was a small dependency that was easy to get rid of. Unfortunately, after installing it failed to initialize my component. At that time it had become Saturday afternoon and I had to revert to Equinox or spent yet another Sunday away from the family. However, yesterday evening Richard Hall and I sat together and fixed it. So now Felix can soon use the declarative services from Equinox. Isn’t OSGi technology wonderful?
The tutorial went very well, though 3 hours is very short. Especially because I’d like to tell too much about the background. The big surprise, however, was yesterday. The whole afternoon in the Leinster room was dedicated to OSGi subjects. First Richard gave an excellent, albeit speedy, overview of OSGi, JSR 277, JSR 291, and JSR 294. Man, that guy can speak fast! The next subject was a panel about OSGi. Geir Magnusson (Apache’s Jakarta project chair) was the moderator and on the panel you could find Tim Ellison (IBM, Apache Harmony), Richard Hall (Apache Felix), Enrique Rodriguez (Apache Directory server), Jason van Zyl (Maven 2), and me. After having to answer the customary grilling about licensing, patents, and the possibility to contribute to the spec (yes, we are working on all those issues) we had an interesting discussion.
A major subject is of course how to get Apache projects to adopt OSGi technology. The Apache Foundation does not have a technology roadmap. Technology is developed bottom up and projects are completely free to choose their own technology. To get OSGi adopted by Apache it is therefore necessary to convince each project of its merits. A key strategy to get Apache to use OSGi seems to be the Maven route. Maven is a build tool that more and more projects start to use. The Apache Felix project is using Maven. A couple of weeks ago I had helped Richard to update the OSGi plugin for Maven so that it was easier to use. During the panel this plugin seems a route into Maven. The OSGi headers are non-intrusive. A JAR can be made in such a way that it runs on the Classpath as well as a bundle in an OSGi framework. Maybe this plugin should become part of Maven so that the appropriate headers become part of all Apache JARs. Should be investigated!
The last session was from Marcel Offermans (luminis) about OSGi and best practices. Marcel is an active committer on the Apache Felix project. luminis has adopted OSGi technology 4 years ago and has lots of experience at different customers. This experience showed in the slides. I loved his slides that showed an overview of how to document package dependencies and service dependencies. The service dependencies were based on my triangle/sphere model. We really need a graphic tool that visualizes these dependencies based on a set of JARs or metadata. Where are the students that can create all these goodies?
This afternoon I will fly back home but I already feel very happy about this conference. What a difference with JavaOne and even ApacheCon in San Diego! Most people now have a basic idea what OSGi is and several Apache projects are using it. This makes evangelizing OSGi technology a lot more fun!
I primarily attended ApacheCon to give a tutorial. This was an adapted version of the Eclipse tutorial. I had tried to replace Equinox with Apache Felix in this tutorial but had to unfortunately give up late Saturday afternoon because the Declarative Services did not work yet on Felix. When I realized this (again) on Friday, I had tried to port the Eclipse Declarative Services implementation to Felix. Initially this looked very simple; there was a small dependency that was easy to get rid of. Unfortunately, after installing it failed to initialize my component. At that time it had become Saturday afternoon and I had to revert to Equinox or spent yet another Sunday away from the family. However, yesterday evening Richard Hall and I sat together and fixed it. So now Felix can soon use the declarative services from Equinox. Isn’t OSGi technology wonderful?
The tutorial went very well, though 3 hours is very short. Especially because I’d like to tell too much about the background. The big surprise, however, was yesterday. The whole afternoon in the Leinster room was dedicated to OSGi subjects. First Richard gave an excellent, albeit speedy, overview of OSGi, JSR 277, JSR 291, and JSR 294. Man, that guy can speak fast! The next subject was a panel about OSGi. Geir Magnusson (Apache’s Jakarta project chair) was the moderator and on the panel you could find Tim Ellison (IBM, Apache Harmony), Richard Hall (Apache Felix), Enrique Rodriguez (Apache Directory server), Jason van Zyl (Maven 2), and me. After having to answer the customary grilling about licensing, patents, and the possibility to contribute to the spec (yes, we are working on all those issues) we had an interesting discussion.
A major subject is of course how to get Apache projects to adopt OSGi technology. The Apache Foundation does not have a technology roadmap. Technology is developed bottom up and projects are completely free to choose their own technology. To get OSGi adopted by Apache it is therefore necessary to convince each project of its merits. A key strategy to get Apache to use OSGi seems to be the Maven route. Maven is a build tool that more and more projects start to use. The Apache Felix project is using Maven. A couple of weeks ago I had helped Richard to update the OSGi plugin for Maven so that it was easier to use. During the panel this plugin seems a route into Maven. The OSGi headers are non-intrusive. A JAR can be made in such a way that it runs on the Classpath as well as a bundle in an OSGi framework. Maybe this plugin should become part of Maven so that the appropriate headers become part of all Apache JARs. Should be investigated!
The last session was from Marcel Offermans (luminis) about OSGi and best practices. Marcel is an active committer on the Apache Felix project. luminis has adopted OSGi technology 4 years ago and has lots of experience at different customers. This experience showed in the slides. I loved his slides that showed an overview of how to document package dependencies and service dependencies. The service dependencies were based on my triangle/sphere model. We really need a graphic tool that visualizes these dependencies based on a set of JARs or metadata. Where are the students that can create all these goodies?
This afternoon I will fly back home but I already feel very happy about this conference. What a difference with JavaOne and even ApacheCon in San Diego! Most people now have a basic idea what OSGi is and several Apache projects are using it. This makes evangelizing OSGi technology a lot more fun!
Subscribe to:
Posts (Atom)