Java Module Layers and OSGi Bundles

NOTE: This article discusses my initial attempt at representing OSGi bundles as JPMS modules.  The implementation of this exposed a flaw in the approach.  You can jump to the Lessons Learned section for the conclusion.  Since then I have done another attempt at properly representing OSGi bundles as real JPMS modules.  You can see that article here.

With the Java 9 release, the Java Platform Module System (JPMS) is coming.  The JPMS will finally modularize the class libraries provided by the JVM.  In addition, the JPMS can be used by developers to modularize applications.  This allows developers to split their applications into modules.  These modules can then specify what other modules they require and what packages they export for use by other modules.

The OSGi specification has been providing a module system for Java applications for a long time already which also allows developers to modularize their applications into modules (a.k.a. bundles).  With OSGi, developers have created many modular applications that are also extensible by installing more bundles provided by third parties.  I have been involved with two projects that do just that.

1. The Eclipse IDE uses OSGi as its module system.  Eclipse plug-ins can be installed to provide additional tooling support for the IDE.
2. WebSphere Liberty uses OSGi as its module system which allows the server to be configured with only the features which are required by the applications provisioned to the server.

When JPMS is released, developers can start to deliver their own Java modules for the JPMS.  What happens when developers want to use Java modules to compose applications which are running on a container that is built using the OSGi module system?  Will such a container be able provide its APIs in such away that JPMS modules can require and access them?  For this post I will focus on this scenario which is similar to WebSphere Liberty.  But before I go into the details of the problem lets first explore the JPMS and the concept of Layers.

JPMS Layers and Modules

A layer in JPMS is a static set of resolved modules.  Each layer has a single parent layer except the empty layer which has no parent.  Layers are hierarchical and can have no cycles.  A layer provides the JVM with a graph which determines how classes are located during class loading.  Once a layer is created, none the modules within the layer can change.  This allows the class loading graph to be locked in when the layer is created.  Modules in one layer can require any module provided in a parent layer.  This includes all parents in the hierarchy all the way down to the empty layer   In order to update a module, the complete layer in which a module is contained must be thrown away and recreated in order to provide a new resolution graph for the layer.  If any module within a layer in the hierarchy needs to be updated, then that layer as well as any children of that layer must be torn down and recreated.  If you need to load up another module provided by JVM in the boot layer, then the complete JVM has to be restarted so that the JVM can recreate the boot layer.

This provides a stable and predictable class loading behavior but it poses a problem for containers that are built using the OSGi module system.  The OSGi module system is much more dynamic.  Modules (bundles) in OSGi are not placed in hierarchical layers which are resolved in orderly stages like the JPMS layers.  In OSGi, bundles that have no dependency on each other can be resolved independently in time of each other.  Not only that, but new bundles can be installed and existing bundles can be updated or uninstalled.  All this without tearing down the Framework or effecting existing bundles within the framework that do not depend on the other bundles being updated, installed or uninstalled.  How would a container built on a dynamic module system be able to provide a JPMS layer which can be used as a parent of a another layer which contains JPMS modules?  Modules in JPMS can only load classes from packages exported from modules within their own layer or one of the layers in their parent hierarchy.  If a container is providing APIs which are exported by OSGi bundles then any API which can be used by applications composed of JPMS modules must be represented within a JPMS layer somehow.  The following diagram illustrates the possible layers with this scenario:

The boot layer contains the JPMS modules which were configured with the JVM when it was launched.  In this diagram, the framework implementation has not yet been migrated to a JPMS module itself.  It is loaded as part of an unnamed module using the old -cp or -jar option when launching the JVM.  The Framework then has two bundles installed: bundle.a and bundle.b.  The Framework resolves these two bundles and creates a class loader for each bundle.  Each class loader in Java 9 now has its own unnamed module associated with it.  Any classes defined by the bundle class loader will be associated with this unnamed module that is associated with the bundle's class loader.  In this scenario there are three distinct unnamed modules.  The goal of this experiment is to figure how how to represent bundle.a and bundle.b as JPMS modules in a layer and then create another JPMS layer as a child which has JPMS modules that require bundle.a and bundle.b as modules.

My Experiment

Over the past few days I have been investigating a way to use the JPMS Layer API in order to construct such an OSGi-JPMS layer.  My goal was to do this completely on top of a standard OSGi Framework implementation without any modifications to the Framework itself.  Now lets dig into the details of my experiment.  My first step was to figure out how to construct a layer based on a finite set of resolved bundles that are installed within the OSGi Framework.  For this experiment, I ignored the fact that the bundles within the bundle JPMS layer could be refreshed at anytime.  To do this, I decided to place the code which constructs this layer into an OSGi bundle itself.  I could have done it with a launcher outside of the framework, but it seemed more simple just to put it into a bundle which can be installed onto any framework no matter how it is launched.

Creating a Layer is a simple process for JPMS modules containing a module-info.class and you want the layer implementation to do all the work for resolving the modules and providing the modular class loader behavior.  The module-info.class file identifies the module, what it provides, and what it requires.  This file gives the layer implementation all it needs to resolve and provide a class loader implementation for the module.   But in my case I wanted to reuse existing bundle class loaders to back the JPMS modules that represented the OSGi bundles.  I did not have any interest in the JPMS layer actually resolving the bundles or providing any class loader behavior. This must continue to be handled by the OSGi Framework.  It turns out the Layer API does provide the ability to map modules to customized class loaders.  So my first step was to discover the set of bundle class loaders I wanted to include in the bundle layer.  To do this, I used the FrameworkWiring findProviders method to find bundle wirings for host bundles which have a class loader associated with them.  The wirings were used to create a ModuleFinder.  A ModuleFinder is used by JPMS in order to find ModuleReference/ModuleDescriptor objects for something called a Configuration.  My ModuleFinder implementation mocked up ModuleDescriptions based on the set of bundle's symbolic names, versions, and exported packages.  I did not bother with any requirements for the ModuleDescriptors for the bundles since I did not care about resolving the bundles in the context of the JPMS.

The next step is to create a Configuration.  Configurations, similar to Layers, have a single parent Configuration.  I based the bundle configuration off the boot configuration by using the resolveRequires method.  Once a configuration is created, it can be used to create a layer.  Here I used the defineModules method which takes a Function<String, ClassLoader> that is used to map a module name to a class loader.  This function allows us to map the bundle JPMS modules to the real bundle class loaders.  There are a couple of gotchas here.

1. It is dangerous to use the actual bundle class loader because errors will occur if any classes are already defined for the packages exported by the bundle when the Layer is created.  To work around this I simply created an empty class loader that uses the bundle's class loader as its parent.
2. All the classes defined by the bundle's class loader will not have the JPMS bundle module associated with them as returned by the Class getModule method.  Instead they will be associated with the unnamed module associated with the bundle's class loader.  This is an important detail when working out how to create the JPMS layer built on top of the bundle layer.

Now that the bundle layer has been created, it can be used to create additional child layers which can then require the bundles by referencing their symbolic names.  This can be done by using the built-in ModuleFinder of method which provides a finder that can discover JPMS modules that use the module-info.class to describe themselves.  This finder can then be used to create a resolved configuration based off the configuration of the bundle jpms Layer.  But there is a catch.  Even though the new configuration correctly resolves the JPMS modules which require the OSGi bundles, they still will not successfully be able to load classes from the packages exported by the OSGi bundles. This is because the JPMS layer only grants read access to the classes defined by bundle modules existing in the bundle JPMS layer.  This makes sense but the actual classes defined by the bundle's class loaders are associated with the unnamed module of the bundle's class loader.

Here is where this experiment turns into a hack, to be honest.  In order to get this to work I had to invoke the addReads method.  This method would allow the modules in the child JPMS layer to actually be able to read (load) the classes from the unnamed modules that are actually backing the bundle JPMS modules.  But the addReads method MUST be called from code loaded by the module that wants read access added to themselves.  In order to do this I ended up augmenting the JPMS module content on the fly by implementing my own ModuleReader.  This module reader injected a specific class into each module which I could then call reflectively in order to "convince" Java 9 into thinking the module itself is really calling addReads.  This is even more awkward due to the fact that the package this injected class is in has to be accessible by the calling code.  To make this "simple," I just exported that package from each JPMS module.  This did not take a lot of code, but it is rather tricky and far from obvious.  After that though, it was possible to get JPMS modules that require OSGi bundles using their bundle symbolic names.  You can find the complete experiment on GitHub at https://github.com/tjwatson/osgi-jpms-layer.

Lessons Learned

I learned a bit by doing this experiment, but in the end I don't think it is really a viable solution to the problem because of the hacks needed to grant read access to the unnamed modules.  I think it has convinced me that there may be a viable solution by creating a true JPMS Layer which gets real JPMS modules associated with the bundle class loaders upfront so that each bundle class loader is able to define classes that are associated with named class loaders.  I hope to get time in the next week or so to do that experiment.  The high level things I need to learn are:

1. Can I intercept the bundle class loaders before they define any classes such that I can map them to their JPMS module representations?  I am pretty sure I can use an OSGi WeavingHook to do this since weaving hooks get called before classes are defined.
2. Can I manage to create a linear layer hierarchy as bundles are resolved such that the latest layer can be used to build child JPMS layers from?  This depends on the order layers search for modules.  The findModule method seems to indicate the "local" layer is searched before asking its parent layer.  I find that hard to believe because it would seem to allow one layer to override things like the java.base module.  But this would be critical so that we can ignore stale modules from layers that have bundles that have been uninstalled, updated, or refreshed.

If I cannot manage to figure out how to have a linear parent chain that ignores stale modules then we still should be able to create layers that represent OSGi bundles with real modules, but the layer may not be able to accurately reflect the full dynamic nature of the OSGi Framework.