The OSGi has teamed up with EclipseCon 2007, 5-8 March, Santa Clara California to organize the 2007 OSGi developer conference. This will be the premier conference for OSGi developers to attend in 2007.
We are currently looking for submissions! Long talks, short talks, tutorials, demos, and panels. Please submit a proposal to the Eclipse submissions site in the OSGi track. Proposals will be reviewed by the program committee, where we have a seat.
Some of the submission deadlines are near, please submit your proposal as soon as possible. The submissions must be registered at the EclipseCon Submissions site. Do not forget to register at the OSGi track.
I know there are a large number of interesting projects under way that are heavily using OSGi technologies. If you are working on one of those projects, do not hesitate to submit a proposal, we have quite a few slots.
We have a proposal on the table to add an OSGi user groups meeting at the
end of the conference. We will let you know more soon.
Peter Kriens
OSGi Blog
Friday, October 20, 2006
Thursday, October 19, 2006
JSR 277 Review
The JSR 277 Java Module System is an important JSR for the OSGi Alliance and me. Though we have been working on Java modularization since 1998 I did not get a chair on the table; the table was already full with 14 seats. Last week, the current 20 man Expert Group has recently dispatched their Early Draft. So this is a personal account of my reading of the Early Draft.
JSR 277 defines a static module system that is similar to the OSGi Require-Bundle approach. Modules define their metadata in the
Module definitions can be obtained from repositories. Repositories contain module definitions that provide metadata to guide the resolving process. Name and version ranges are supported from the metadata. However, an import policy class can be used to do more complex resolving strategies. Repositories can be file or URL based, allowing for remote repositories. Repositories are hierarchically linked; if a repository can not find a module definition, it will delegate to its parent. At the top of the chain is a system repository and above that a boot repository. Module definitions are instantiated before they can be used; this associates a class loader with a module. When all the modules are resolved and instantiated, the main class is called and the program can execute. There is no API for unloading a module.
The ambition level of JSR 277 is significantly lower than where the OSGi specifications are today, even way lower than we set out in 1998. In a way, for me the JSR 277 proposal feels rather toyish. The Expert Group took a simplistic module loading model and ignored many of the lessons that we learned over the past 8 years. No built in consistency, no unloading, no package based sharing. Maybe we wasted a lot of time or solved unimportant problems, but I really do not think so. It would be ok if they had found a much cleaner model, well supported by the VM, but it is clearly not. Their model is based on delegated class loaders, just like the OSGi framework minus 8 years of experience. Ok, I am biased, so let us take a closer look. Though just a warning, the following text is utterly boring unless you are handicapped with a deep interested in Java class loaders. Much of the work over the past 8 years was making bundle developers worry about application functionality and let the OSGi framework worry about class loaders. This stuff should be under the covers. Alas.
The key disappointment in JSR 277 is the lack of dynamics. There is no dynamic loading and unloading of modules/bundles, any change will require a reboot of the VM. The model is geared to the traditional application model of starting an application, running, and then killing the VM. Fortunately, JSR 291 Dynamic Component Support addresses dynamics and more comprehensive module loading, so let’s hope that will become the Java standard.
The most surprising feature of JSR 277 is a total lack of consistency checking. Modules are loaded by name and version only; there is no verification that the graph of class loaders form a consistent class space. Take the example in paragraph 8.3.3.3. There are four modules:
Modules resolve independently, implying that module
A key problem with the Require-Bundle/Module approach is ordering. Though the EDR implies that classes are searched in a specific order, this still depends on what other modules do. Assume we have:
The order for module
Split packages are packages that come partly from one module and partly from another module. Split packages are nasty because the package security access does not work and it is also easy to unexpectedly shadow classes and resources.
Security problems occur with split packages when you load class
Worst of all, none of this is under the control of the importer. The importer just imports a name and version, and then receives whatever the exporter decides to export. Obviously, this is a brittle connection. For example, module
Another problem with modules is their fan out. Typical projects create JARs that are useful in a number of situations. For example, a program like bnd (creates bundle manifest headers from the class files) can be used as an Ant task, an Eclipse plugin, from the command line, and a Maven plugin. It will therefore have to import packages from Ant, Eclipse, and Maven. Those dependencies are needed and are therefore more or less ok (though it is nice to make them optional, a feature missing from JSR 277). However, Ant will import another (large) set of modules. Similarly for maven and Eclipse, ad nauseum. The bnd program is a terrifying example, but the problem is that the usual fan out of a module is large and the fact that dependencies are transitive can worsen this effect. If you want to see the effect of module like imports, check out maven. A simple hello world program can drag in hundreds of modules due to the transitive dependencies. Modules (and Require Bundle) as well are a typical example of creating a constructs to solve a problem but simultaneously creating problems on the next level, problems which are usually ignored in the first version only to bite the early adopters.
A puzzling aspect of 277 is the dependency on super packages from JSR 294. Super packages are shrouded in a veil of mystery. The only public information so far came from Gilad Bracha’s blog. JSR 277 unveils a bit more but many questions are left unanswered. Super packages list all their member classes and a class can only be member of a single module. In 277, it is implied that this module file maps closely to the module metadata. If this is true, than this is a huge constraint. It means that the deployment format is rigidly bound to the development time modules. Each development module must be deployed separately.
I’ve found that managing the packaging is a powerful tool for the deployer. I have written many bundles that mix and match packages from different projects. This flexibility is needed because there are two counter acting forces. On one side you want to simplify deployment by deploying the minimum number of bundles. Anybody that had to chase dependencies knows how annoying the need for more and more bundles. On the other hand, bigger bundles are likely to create more dependencies because they contain more (sometimes unnecessary) code. So you like to minimize the number of external dependencies. Sometimes the best solution is to include the code of other JARs in a deployment JAR. The proposed super packages will put a stop to that model: it will be impossible to manage the deployment packaging because the programmer has decided the packaging a priori; bad idea.
The biggest regret the EG members will have within 2 years is the import policy. Why? Isn’t it a nice idea that the programmer can participate in the resolving? Well, in Java the solution to those problems looks deceptively simple: use a class to abstract the required functionality. An import policy is a class in your module that gets called during the resolving of the module. It can inspect the module metadata, query the repositories and bind other modules.
In the OSGi Alliance, we inherited a similar idea from Java Embedded Server (JES), the archetypical OSGi framework developed by SUN. After countless hours talking and testing we decided that it was a bad idea because of 2 key reasons:
Wasn’t there anything I liked? Well, I like the idea of the repositories. The OSGi framework maintains the repository internally with its installed bundles, it left external repositories outside the specification by standardizing the installation API. Repositories allow the framework to download or find modules just before activation. This is an interesting model, pursued by Maven and the OSGi Alliance with OBR.
However, I think the JSR 277 repository model is too simplistic. It codifies the current maven repository model, which is still immature and will likely change over time. For example, the current model takes only two extra constraints into account: OS, and platform. Unfortunately, life is seriously more complicated. OS’s have their own versioning scheme, processors have compatibility matrices (i.e. an x86 runs on a i586, win32 is compatible with WinXP but not vice versa), the OS is often a variation of OS and window system. Encoding these constraints in the file name is obviously bound to collide with reality one day. In contrast, the OSGi OBR uses a generic requirement-capability model derived from JSR 124 that is much better suited for finding the right module.
Well, the paper document I have reviewed is heavily covered with my marker and I could continue for more pages. However, it is too much for this blog. Well, ok, last complaint: the syntax for version ranges, it is too tempting to leave it alone (though it is not that important). The industry more or less has standardized on versions with a major, micro, minor number and a string qualifier. JSR 277 adds an update number that is between minor and qualifier (Maybe one day someone can explain to me why we need all those number while the only thing that is signaled is a change that is backward compatible or not, but developers seem to like to have all those numbers). However, I have no problem adding another number beneath minor. I do have a problem with a version range syntax that is obtuse and non-standard.
Intervals have a mathematical notation that is easy to understand. Parentheses are non-inclusive and brackets are inclusive. The interval 1 < x <= 5 is therefore represented as (1,5]. [2.3,5.1) indicates any version that is more or equal to 2.0 and less than 5.1. Simple, elegant, has been around since Pythagoras. Choosing this notation for the OSGi specifications was a no-brainer.
Now, let us take a look at what JSR 277 brewed. They use the same terminals as regular expressions, but they have a very different meaning. A partial version can be suffixed with a + (anything later) or a * (anything with the same prefix). So if you say 1+ you mean [1,∞) or 1. 1* is [1,2). The JSR uses square brackets to fix the first positions while floating the last. For example, 1.[2.3+] is [1.2.3, ∞) and 1.[2.3*] is [1.2.3,1.2.4). The JSR 277 syntax has no easy formulation for the not uncommon case [1,5). The only way to express this is with concatenation of version ranges: 1*;2*;3*;4*. Ok, enough about this strange syntax.
Conclusion. JSR 277 takes a simplistic view of the world, ignoring many real life problems: consistency, optionality, split packages, etc. The only way this EG can pass its final review is lack of attention for detail of the Executive Committee or alternatively, serious muscle power of SUN. JSR 277 is a missed opportunity for Java. I do not doubt that this specification will end up in Java 7, but it will further fragment the Java world for no technical reason. Not only is this specification impossible to implement on J2ME any time soon, it will also leave the many OSGi adopters out in the cold. Why?
Peter Kriens
JSR 277 defines a static module system that is similar to the OSGi Require-Bundle approach. Modules define their metadata in the
MODULE-INF/METADATA.module resource. Dependencies are pecified in the import clause with the module’s symbolic name and version range. Other clauses in the module’s metadata list the resources and classes that are visible to other modules. When a JSR 277 application gets started, it should have a main class with a static main method. However, a module must be resolved, as well as its required modules, before the main method is called.Module definitions can be obtained from repositories. Repositories contain module definitions that provide metadata to guide the resolving process. Name and version ranges are supported from the metadata. However, an import policy class can be used to do more complex resolving strategies. Repositories can be file or URL based, allowing for remote repositories. Repositories are hierarchically linked; if a repository can not find a module definition, it will delegate to its parent. At the top of the chain is a system repository and above that a boot repository. Module definitions are instantiated before they can be used; this associates a class loader with a module. When all the modules are resolved and instantiated, the main class is called and the program can execute. There is no API for unloading a module.
The ambition level of JSR 277 is significantly lower than where the OSGi specifications are today, even way lower than we set out in 1998. In a way, for me the JSR 277 proposal feels rather toyish. The Expert Group took a simplistic module loading model and ignored many of the lessons that we learned over the past 8 years. No built in consistency, no unloading, no package based sharing. Maybe we wasted a lot of time or solved unimportant problems, but I really do not think so. It would be ok if they had found a much cleaner model, well supported by the VM, but it is clearly not. Their model is based on delegated class loaders, just like the OSGi framework minus 8 years of experience. Ok, I am biased, so let us take a closer look. Though just a warning, the following text is utterly boring unless you are handicapped with a deep interested in Java class loaders. Much of the work over the past 8 years was making bundle developers worry about application functionality and let the OSGi framework worry about class loaders. This stuff should be under the covers. Alas.
The key disappointment in JSR 277 is the lack of dynamics. There is no dynamic loading and unloading of modules/bundles, any change will require a reboot of the VM. The model is geared to the traditional application model of starting an application, running, and then killing the VM. Fortunately, JSR 291 Dynamic Component Support addresses dynamics and more comprehensive module loading, so let’s hope that will become the Java standard.
The most surprising feature of JSR 277 is a total lack of consistency checking. Modules are loaded by name and version only; there is no verification that the graph of class loaders form a consistent class space. Take the example in paragraph 8.3.3.3. There are four modules:
A, B, C, and D (-> means import).
A -> B v1.0, C v1.0
B -> D v1.1
C -> D v1.0
Modules resolve independently, implying that module
B will resolve to module D v1.1 and module C will resolve to module D v1.0. The result is that module A can see the same class coming from module D v1.0 and module D v1.1, likely resulting in class cast exceptions. Or more concrete, assume module A is your application and module B and module C implement some web framework and module D is javax.servlet 2.1 and 2.4. If you use a Servlet, do you get it through module B or C? Errors resulting from this inconsistency can be very hard to trace. There exists a method in the Module class called deepValidate() that uses a brute force method by loading all the classes in all modules to see if class cast pr class not found exceptions occur. Obviously this is very expensive and not even bound to find most consistency problems, just loading problems!A key problem with the Require-Bundle/Module approach is ordering. Though the EDR implies that classes are searched in a specific order, this still depends on what other modules do. Assume we have:
A -> B, C, D
B -> D
The order for module
A is not B, C, D but instead B, D, C because module B also imports module <D (assuming classes we re-exported). This is one of the reasons I do not like Require-Bundle, though practically we have much less of a problem than JSR 277 will have. The issue is split packages.Split packages are packages that come partly from one module and partly from another module. Split packages are nasty because the package security access does not work and it is also easy to unexpectedly shadow classes and resources.
Security problems occur with split packages when you load class
com.p.A from module A, and com.p.B from module B. The different class loaders will make it impossible for A to see package private methods in B despite the fact they reside in the same package. This is not a theoretical problem especially if multiple versions of the same module are supported. It is easy to pickup a new class from a new version but then load the auxiliary classes from a module that is listed earlier. Worst of all, none of this is under the control of the importer. The importer just imports a name and version, and then receives whatever the exporter decides to export. Obviously, this is a brittle connection. For example, module
A contains packages a and b. One day module A is refactored and package b is moved to module B. This is not uncommon because big modules are not easy to work with, more about that later. Unfortunately, all clients of module A must now be changed to add module B to their import lists even though nothing has changed except the packaging. Why is this not a problem with OSGi imports? Well, OSGi imports packages, in the previous example package b will just be obtained from module B without any change. It might not seem like a big deal, but for large systems this can amount to major work.Another problem with modules is their fan out. Typical projects create JARs that are useful in a number of situations. For example, a program like bnd (creates bundle manifest headers from the class files) can be used as an Ant task, an Eclipse plugin, from the command line, and a Maven plugin. It will therefore have to import packages from Ant, Eclipse, and Maven. Those dependencies are needed and are therefore more or less ok (though it is nice to make them optional, a feature missing from JSR 277). However, Ant will import another (large) set of modules. Similarly for maven and Eclipse, ad nauseum. The bnd program is a terrifying example, but the problem is that the usual fan out of a module is large and the fact that dependencies are transitive can worsen this effect. If you want to see the effect of module like imports, check out maven. A simple hello world program can drag in hundreds of modules due to the transitive dependencies. Modules (and Require Bundle) as well are a typical example of creating a constructs to solve a problem but simultaneously creating problems on the next level, problems which are usually ignored in the first version only to bite the early adopters.
A puzzling aspect of 277 is the dependency on super packages from JSR 294. Super packages are shrouded in a veil of mystery. The only public information so far came from Gilad Bracha’s blog. JSR 277 unveils a bit more but many questions are left unanswered. Super packages list all their member classes and a class can only be member of a single module. In 277, it is implied that this module file maps closely to the module metadata. If this is true, than this is a huge constraint. It means that the deployment format is rigidly bound to the development time modules. Each development module must be deployed separately.
I’ve found that managing the packaging is a powerful tool for the deployer. I have written many bundles that mix and match packages from different projects. This flexibility is needed because there are two counter acting forces. On one side you want to simplify deployment by deploying the minimum number of bundles. Anybody that had to chase dependencies knows how annoying the need for more and more bundles. On the other hand, bigger bundles are likely to create more dependencies because they contain more (sometimes unnecessary) code. So you like to minimize the number of external dependencies. Sometimes the best solution is to include the code of other JARs in a deployment JAR. The proposed super packages will put a stop to that model: it will be impossible to manage the deployment packaging because the programmer has decided the packaging a priori; bad idea.
The biggest regret the EG members will have within 2 years is the import policy. Why? Isn’t it a nice idea that the programmer can participate in the resolving? Well, in Java the solution to those problems looks deceptively simple: use a class to abstract the required functionality. An import policy is a class in your module that gets called during the resolving of the module. It can inspect the module metadata, query the repositories and bind other modules.
In the OSGi Alliance, we inherited a similar idea from Java Embedded Server (JES), the archetypical OSGi framework developed by SUN. After countless hours talking and testing we decided that it was a bad idea because of 2 key reasons:
- By definition, you do not have a valid Java environment before you resolved all the required modules. Code executed in a module that is not resolved is in limbo and is bound to run into problems. There are also several security related issues.
- Having a procedural solution will prevent management systems from predicting what module will resolve to what module. Within the OSGi specifications, we spent countless hours to make the systems predictable for this reason. This is necessary because in the future more and more systems will be managed (I can’t wait for that day!). Inserting a user class in to the resolution process leaves the management system in the blind. Interestingly, Java made the same mistake with security permissions. Abstracting the permissions as a class is good OO practice, but it kills any possible optimization. A declarative approach could have significantly reduced the 20%-30% overhead of Java security while the flexibility that the model offers is rarely used.
Wasn’t there anything I liked? Well, I like the idea of the repositories. The OSGi framework maintains the repository internally with its installed bundles, it left external repositories outside the specification by standardizing the installation API. Repositories allow the framework to download or find modules just before activation. This is an interesting model, pursued by Maven and the OSGi Alliance with OBR.
However, I think the JSR 277 repository model is too simplistic. It codifies the current maven repository model, which is still immature and will likely change over time. For example, the current model takes only two extra constraints into account: OS, and platform. Unfortunately, life is seriously more complicated. OS’s have their own versioning scheme, processors have compatibility matrices (i.e. an x86 runs on a i586, win32 is compatible with WinXP but not vice versa), the OS is often a variation of OS and window system. Encoding these constraints in the file name is obviously bound to collide with reality one day. In contrast, the OSGi OBR uses a generic requirement-capability model derived from JSR 124 that is much better suited for finding the right module.
Well, the paper document I have reviewed is heavily covered with my marker and I could continue for more pages. However, it is too much for this blog. Well, ok, last complaint: the syntax for version ranges, it is too tempting to leave it alone (though it is not that important). The industry more or less has standardized on versions with a major, micro, minor number and a string qualifier. JSR 277 adds an update number that is between minor and qualifier (Maybe one day someone can explain to me why we need all those number while the only thing that is signaled is a change that is backward compatible or not, but developers seem to like to have all those numbers). However, I have no problem adding another number beneath minor. I do have a problem with a version range syntax that is obtuse and non-standard.
Intervals have a mathematical notation that is easy to understand. Parentheses are non-inclusive and brackets are inclusive. The interval 1 < x <= 5 is therefore represented as (1,5]. [2.3,5.1) indicates any version that is more or equal to 2.0 and less than 5.1. Simple, elegant, has been around since Pythagoras. Choosing this notation for the OSGi specifications was a no-brainer.
Now, let us take a look at what JSR 277 brewed. They use the same terminals as regular expressions, but they have a very different meaning. A partial version can be suffixed with a + (anything later) or a * (anything with the same prefix). So if you say 1+ you mean [1,∞) or 1. 1* is [1,2). The JSR uses square brackets to fix the first positions while floating the last. For example, 1.[2.3+] is [1.2.3, ∞) and 1.[2.3*] is [1.2.3,1.2.4). The JSR 277 syntax has no easy formulation for the not uncommon case [1,5). The only way to express this is with concatenation of version ranges: 1*;2*;3*;4*. Ok, enough about this strange syntax.
Conclusion. JSR 277 takes a simplistic view of the world, ignoring many real life problems: consistency, optionality, split packages, etc. The only way this EG can pass its final review is lack of attention for detail of the Executive Committee or alternatively, serious muscle power of SUN. JSR 277 is a missed opportunity for Java. I do not doubt that this specification will end up in Java 7, but it will further fragment the Java world for no technical reason. Not only is this specification impossible to implement on J2ME any time soon, it will also leave the many OSGi adopters out in the cold. Why?
Peter Kriens
Subscribe to:
Posts (Atom)