Monday, July 2, 2007

To Declare Or Not to Declare

A few weeks ago I described the “hibernate” problem, how to let multiple bundles independently use a Hibernate Session Factory without requiring a priori knowledge of each other. These postings are not just finding practical solutions to today’s problems with today’s specifications. These are my research subjects; I am trying to keep future generations of the OSGi specification clean and in the right spirit so I am playing with options. Next time I will try to be more pragmatic.

The solution I found to share the Session Factory was service based, it allowed bundles to contribute Hibernate domain classes, and it allowed bundles to use a collective of domain classes. However, in this solution there still lurked a class loading problem caused by a peculiar habit of Hibernate. In the solution this problem was solved with a simple Require-Bundle. However, though this solution works, it is a hack that can easily fall apart if the required classes are refactored into another bundle. So for today, this solution is good enough, for tomorrow we need to understand the root problems and come up with solutions to this root problem. So lets dive deeper into the problem and analyze what is going on.

When Hibernate creates a Session Factory, it actually creates a dynamic class proxy to its implementation class. This in itself is not a problem was it not for the fact that it uses the client’s class loader to create the proxy. The client therefore must have visibility to classes that it does not use. That is, Hibernate assumes all the classes itself sees are visible to the client. In a modular system, this is obviously not the case. A simple fix is to let the client bundle use Require-Bundle for the Hibernate bundle, however, this Require-Bundle has its own set of problems, see the OSGi R4.1 specification.

It is easy to take the high ground and tell Hibernate to get its act together. However, this is not a very productive stance for most of us. Despite the increasing popularity of OSGi Technology, the majorities of JARs out there are not prepared for proper modularity.

These type of class loading problem are endemic because so many developers have (ab)used class loaders for home grown extension mechanisms, due to a lack of a proper extension mechanisms in Java á la the OSGi service registry. All of these home grown mechanisms just fall apart when it becomes necessary to share the VM with multiple versions of the same package. They also can not provide class space consistency, which is important for applications that want to work reliably. Therefore, the best solution would be if the OSGi specifications addressed these issues directly.

The OSGi CPEG asked me to come up with an OSGi RFP about this problem so I Skyped my favorite OSGi Invited Researcher: Richard S. Hall. We did some brainstorming and in the past few weeks we did some prototyping to better understand the problems. Richard modified Apache Felix to support our experiments and I used these modifications with a simple web based notes taking application that uses Hibernate.

The first approach we took was the declarative “implicit-wire” directive. We came to this solution because the deep structure of the problem is that when you import a package, the consequence of this import is that you should import additional packages. That is, in the Hibernate example, when you import org.hibernate.cfg, you should also import net.sf.cglib.*, even if you have no direct dependency on these packages. Richard therefore created an x-implicitwire directive on an Export-Package clause. The value of this directive is a list of packages that should be added to the import of any bundle importing the package. For example:

Export-Package: org.hibernate.cfg;x-implicitwire:=”net.sf.cglib.proxy,net.
sf.cglib.core,net.sf.cglib.reflect, org.hibernate.proxy”


If Apache Felix has to wire to the org.hibernate.cfg package, it will now automatically add wires to the packages from the x-implicitwire directive as well, thereby ensuring that the importing bundle can see the proper packages when Hibernate attempts to create a proxy for the Session Factory.

The declarative approach is quite simple and it did solve the problem. The prototype code, a simple web based notes program, worked fine. The only change that was necessary was the bnd file that created the hibernate bundle. However, we were not convinced that this solution could solve world hunger; there are many cases where the declarative approach with x-implicitwire is not feasible because the set of required implicit wires is not known ahead of time. A crucial example is generic Aspect Oriented Programming, where the set of additional classes is completely open-ended. That is, an AspectJ program can weave any possible class in another class.
The only solution to this problem is allowing another bundle to add imports during runtime. After some discussions Richard found the courage, and time, to add an addRequire function to the Bundle Context object. Imports added this way are treated as DynamicImport-Package clauses. That is, they are consulted last and always until the import is found.

I created a simple extender bundle that inspects all bundles when they become resolved. I chose the import of the org.hibernate.cfg package as trigger. That is, when a bundle imported this package, it would automatically get the net.sf.cglib.* packages. As can be expected, also this approach worked fine and did not require anything special from the client bundle, no changes were necessary. However, unfortunately there is a race condition to which we do no have a good solution. If the extender bundle gets started after the client bundle, we have a problem. The resolved client bundle could get started and create a Session Factory before the extender has a change to add the imports. Alternatively, the additional imports could be added persistent, in that case the install event could trigger the adding of the imports. In that case, the window for the race condition is much smaller. This problem is prevented with the declarative approach of x-implicitwire.

So what is the best solution? I am really not sure. I like the simplicity of the declarative approach. It allows a clean and very simple solution for problems where the implicit imports are known ahead of time. A very important advantage of this approach is that it does not have race conditions. However, when these imports are dynamic, the problem is no longer usable. The procedural approach of dynamically adding an import in runtime has the required flexibility but has the problem of the race condition. It will require deployers to use the start level to ensure that the extender bundle is started before the client bundles are installed.

What do you think?

Peter Kriens

8 comments:

  1. Peter, would the bundle dependent upon an ACTIVE extender bundle have that dependency in its MANIFEST.MF? Because, if so, could you not have some mechanism in the core platform whereby the dependent bundle could modify its own start level to ensure that it started after the extender bundle?

    Just curious. BTW, thanks for the great insight into the travails with libraries like Hibernate!

    ReplyDelete
  2. All these problems are clearly solvable, but they also create complexity. The nice thing about the x-implicitwire is, that it does not have this dependency on start level.

    If I had to solve it today, Require-Bundle will probably solve most of your problems. Again, this is a research, it would be nice to find a solution that is simple, elegant, and actually works ...

    Peter Kriens

    ReplyDelete
  3. hi peter,

    add imports during runtime with version can help in most cases. This helps many homegrown solutions to move to OSGI gradually(in a phased manner).

    regards,
    Bharavi

    ReplyDelete
  4. The autowire declaration seems like a better solution for this problem.

    The timing window in the extender bundle approach is a consequence of a bigger problem, which is that the knowledge of autowire dependencies is no longer in the bundle that is exporting the packages, but has now moved into a completely different bundle.

    With the autowire declaration, the Hibernate bundle can assure that clients importing Hibernate APIs also import the proxy classes that they don't know about but that Hibernate requires them to be able to load.

    With the extender bundle, the Hibernate bundle does not assure that the imports are done properly, and it is left to an extender bundle that you hope is loaded to do that job. There is a timing window, and there is no documentation in the manifest about what is going to happen when a bundle imports Hibernate API packages. In fact, what happens is "magic" and it depends on which other bundles are loaded. This is a maintainability problem.

    So, while the extender bundle solves a problem, it creates more problems.

    I'm not an expert on AOP, but it seems to me that in practice the possible set of classes that can be woven in an application is not infinite as it is in theory. An application should know what its possible aspects are, and can export those and include autowire declarations for them.

    If not, then you can fall back to using Require-Bundle, which is no worse than what we have today.

    ReplyDelete
  5. What happened with this about 1 year old issue. Is it resolved by means of the rarely described

    DynamicImport-Package: *

    directive?

    Although I agree that the possible set of classes that can be woven in an concrete application is not infinite, it is infinite from a library perspective that is integrated in diffrent contexts and applications. May be it is of use if it would be possible to parameterize a library bundle at startup time rather than at compile time to import additional packages/classes from a well known domain. Without such an option it would be necessary to rebuild the library for each domain to import the specific addtional packages. However the DynamicImport directive seems to solve this issue.

    Robert Saenger

    ReplyDelete
  6. Dynamic-Import rarely solves issues, but it does create them. Dynamic Import is reverting back to all the sins of the linear classpath.

    The best solution is to use Require-Bundle on a bundle that contains both hibernate and its core libraries. This will take care of the wrong dependency. One of the few cases where Require-Bundle is useful ...

    ReplyDelete
  7. If I understand it right *clean* dynamic import is still an issue to be discussed.

    I wonder that Dynamic Import should be as week as linear classpath search. My thought was, that it is a bit more powerful because it takes the subordinate dependencies appropriately into account (of course the required anchor class may be accidentialy found as it would be the case for linear classpath search). So in cases where there is more than one version on the classpath the OSGi advantages are more or less lost.

    As I wrote in my last post, under some circumstances it may not be possible to specify a concrete dependency at assembly time neither at a package nor at a bundle level.

    I have the case where I have to deal with a legacy library that havily makes use of Class.forName() invocations. The concrete classes to be resolved vary from application to application that make use of this library and are mainly used for decoration purposes. E.g. application A may introduce ADecorator and app B BDecorator and so on (I'm taking about hundrets up to thousands of such classes per application and I'm convinced that the OSGi service pattern is not applicable to announce all these classes).

    However, it would be possible to declare the import requirements at deploy time. Thus rather finalizing the imported packages in the Manifest while building the bundle Jar it should be possible to dynamically configure (inject additional import packages) at deploy or at least at runtime. Until this is not possible I can either use Dynamic Import or create dedicated library JAR instances that solely consist of distinct Manifest characteristics in order to reflect the different import requirements.

    ReplyDelete
  8. The trick is not to use Class.forName. If you can't influence the class selection through some mechanism then you have to revert to Dynamic-ImportPackage (it was added for this reason) but you loose a lot of the advantages of OSGi.

    You might want to take a look at fragments. They can sometimes be useful for this kind of legacy code.

    ReplyDelete