Thursday, September 29, 2016

InfoQ: A comparison of OSGi and the Java 9 Java Platform Module System

In Java 9, OSGi and the Future of Modularity, Neil Bartlett and Kai Hackbarth provide part 1 of an excellent comparison of OSGi modularity and the current state of Java 9's Java Platform Module System (JPMS) modularity.
One of the most common complaints about OSGi is that it can increase complexity for a developer. There is a grain of truth here, but people who make this complaint are mistaking the medicine for the disease.
Modularity is not a magic dust that can be sprinkled onto an application just before release. It is a discipline that must be followed throughout all phases of design and development. Developers who adopt OSGi early and apply modular thinking before writing a line of code realise enormous gains[...]
Project Jigsaw started with a goal of being simple, but the JPMS specification has increased enormously in complexity: the interplay of modules with class loaders; hierarchical layers and configurations; re-exporting requirements; weak modules; static requirements; qualified exports; dynamic exports; inherited readability across layers; multi-module JARs; automatic modules; unnamed modules… all these features have been added as the need for them became clear. A similar process happened in OSGi, just with a 16-year head start.
The article is well written and provides a clear understanding of the differences between OSGi and JPMS as it stands today. Looking forward to part 2.

Monday, September 19, 2016

Want to become an OSGi Developer Certified Professional?


The next OSGi Developer Certification - Professional exam is taking place on the afternoon of Monday Oct 24 in Ludwigsburg, Germany. That's the day before the OSGi Community Event 2016.

This is your opportunity to validate your knowledge and experience with OSGi technology and specifications.

You should have some practical experience of working with OSGi and a good understanding of the OSGi R5 specification topics, but don't worry we do provide an electronic copy of the R5 specification for you to use during the exam so you don't need to commit the spec to memory!

The exam is 3.5 hours and is made up of three sections:
  • Section 1 - written questions and answers, both multiple choice and open questions
  • Section 2 - a practical assignment with working deliverables developed and submitted for assessment
  • Section 3 - a practical troubleshooting assignment.
Please see https://www.osgi.org/osgi-certification/developer-certification/professional/ for further details on exam topics, what you need to bring with you and other useful information.

Its US $500 per candidate sitting the exam. A special student discounted price of US $200 is available with proof of student status.

To book a place please visit our eventbrite page.

Can't make Ludwigsburg?  If you can't make it to Ludwigsburg but are interested in taking the OSGi Developer Certification - Professional exam please drop us an email to let us know and suggest a location that would work for you.

Friday, September 16, 2016

See OSGi Alliance at Smart Summit London Next Week

Smart Summit London is taking place Sept 21-22 at the Olympia Conference Centre in London.


The conference is made up of three tracks and a join networking exhibition.  The tracks will have in-depth information focused on Smart Home, Smart Cities and Industrial IoT.

The OSGi Alliance will be represented by Christer Larsson (OSGi Alliance VP of EMEA) and Tim Ward (OSGi Alliance Co-chair of the IoT Expert Group ) who will be taking parr in the conferences.

The conference has two packed days with over 160 leading speakers and is expecting over 700 attendees.

Exhibit only passes are available for free, there is a charge for conference passes. The OSGi Alliance has a number of 20% discount codes available for the conference.  If you are interested in receiving one of these please contact us by email.

Also be sure to let us know by email if you are going to be attending the conference and would like to meet with Christer, Tim or any of the other OSGi Alliance members who will be there.

You can find out more about the event and how to register for the exhibits or the conference on the Smart Summit London website.

Wednesday, August 24, 2016

OSGi BoF - We Want Your Topic Suggestions

The OSGi Alliance will be hosting a Birds of a Feather (BoF) on the evening of Tuesday 25 Oct at the OSGi Community Event in Ludwigsburg, Germany.  The BoF will be open to all attendees of the OSGi Community Event and EclipseCon Europe (registration for the conference Oct 25 to Oct 27 is open so book your tickets today). Full details on the timing and room location will be confirmed n due course.

Birds of a feather
By Richard Taylor
(originally posted to Flickr as Birds of a feather)
[CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)],
via Wikimedia Commons
The BoF is a regular occurrence at our annual event.  This year we are asking you, the OSGi community, to provide us with your suggestions on topics that you think would be good to consider and discuss at the BoF.  You can send us your ideas, even if you are not able to join us for the event or the BoF. We would, of course, love to have you there on person too to participate, so please do attend if you can.

To make your suggestions please visit our online form. You can optionally provide us with your name and email address if you would like to receive an update on the final BoF topic list.

The program for the conference has been announced.  We will have 1 OSGi Keynote, 1 tutorial, 25 talks, the BoF, the OSGi IoT Demo, and lots of opportunity to meet and mingle with your peers and colleagues and participate in the included conference evening social events.  Attendees to OSGi Community Event also get full access to the EclipseCon schedule and talks, so there is plenty to keep you busy for all 3 days.

Paul Fraser is presenting on OSGi enRoute and he has put together a 40-second fun video to entice you to join his talk - Its Beautiful enRoute. The talk is taking place on Tuesday 25 October between 15.45 and 16.20 in Seminarräume 1-3.


Finally, but definitely importantly, many thanks to our OSGi Community Event sponsors, whose support ensures we can continue to provide an excellent conference value every year.

For 2016 we are pleased to have sponsorship so far from:


If you would like to explore how your company could join the sponsors please contact us by email.

And a final big thank you to our event partners EclipseCon Europe with whom we are co-located and who provide all of the logitstics and organisation for running OSGi Community Event.

Monday, August 8, 2016

MODCONF 2016 - CFP Open Now

A new conference, MODCONF, is being held in November this year.  As you can hopefully
guess by the name its all about Modularity and software. Clearly this is very applicable to OSGi so when the OSGi Alliance was invited to support the conference we were pleased to accept.

The first MODCONF will take place in Darmstadt, Germany on November 15, 2016.  The conference is being organised by Liferay but it will not be covering Liferay products. Co-located with the event is Liferay DevCon, which will cover their products.

The Call For Papers for the conference is open now and submissions from anyone in the OSGi Community are encouraged and very welcome.  You can find full details of the type of talks that they are looking for and how to submit a talk online here. Of course be sure to have a punch Title and Abstract to submit.

Also please ensure you list your submission as a Modularity Conference Talk as the talk type when submitting.

You will probably note that the CFP deadline is officially Aug 12.  I do however have it on good authority that they will be pleased to receive OSGi related submissions up to the end of August.  Thats still not too much time, so the sooner you can submit the better,

For those of you who don't want to provide a talk but are interested in attending you can find full details on how to register here.

We hope you can help ensure OSGi is well represented in the conference program and help make this new conference a success.  Please contact the Marketing team at the OSGi Alliance or the MODCONF organisers if you have any questions.

Friday, August 5, 2016

OSGi with Java Modules all the way down

In my previous blog post I discussed an experiment that creates a JPMS Bundle Layer which can represent resolved OSGi bundles as JPMS modules.  This would allow child JPMS layers to be created that have modules that require the OSGi bundles as they would any other module.

In that experiment I took a hybrid approach where the Framework implementation and the OSGi bundles themselves did not really live in the JPMS world.  Instead I dynamically created a layer on top that attempted to represent that OSGi world indirectly within the JPMS world.  Then real JPMS modules could be configured to work on top of this facade layer that represented the OSGi world.  This can be thought of as taking a top down approach to migrating to JPMS.  Unfortunately this approach has a major shortcoming because all classes that are loaded in the OSGi bundle layer will be associated with an unnamed module.

The fact that the bundle classes are associated with an unnamed module caused me to have to do a major hack to grant access to modules representing the OSGi bundles.  This hack involved injecting code into the jpms modules which could invoke the addReads method in order to grant the necessary access to the unnamed module of the bundle class loaders.  This does not seem like a real viable solution for running JPMS modules on top of and OSGi bundle layer.

I learned much about how the JMPS layer works during that experiment.  The hybrid has a major flaw because the delegation graph of class loaders involved are not associated with named modules all the way down.  A better way would be to do a bottom up approach where each layer involved has class loaders which are mapped to one or more named modules.  This way when the JPMS layer resolves the modules on top it will automatically grant read access as normal from a requiring module to all of its required modules it got resolved to.  The following diagram illustrates how the layers would look:

The boot layer contains the JPMS modules which were configured with the JVM when it was launched.  In this diagram, the framework launcher has also been migrated to Java 9 in order to have it create a Layer for the class loader used to load the framework implementation.  This layer configures a single module named system.bundle.  This allows all the classes for the Framework implementation to be associated with the system.bundle module.  Next is the bundle layer.  This layer is configured to map each bundle class loader to a named module representing the bundle.  Finally we have a module layer which uses all the built-in module class loaders of Java 9 for JPMS.

My Experiment

Over the past few days I have been reworking my github project (OSGi-JPMS layer) to investigate if this approach is possible.  Again, I am trying to do this without requiring any modifications to the OSGi framework implementation itself and I am using only OSGi specified APIs.  This approach uses a bottom up strategy for JPMS modules.  With that in mind the first thing to do is to modify the OSGi Framework launcher to create the system.bundle module.

The system.bundle Module

I did not want to modify the framework itself to make it a real JPMS module.  Instead I decided to modify the existing Equinox launcher to create a layer itself which maps the class loader it creates to load the OSGi Framework with a system.bundle module.  While the Equinox launcher is specific to launching the Equinox Framework a similar thing could be done to launch any standard OSGi Framework.

The system.bundle acts as the OSGi bundle that exports for all the non-java.* packages available in the boot layer.  This allows OSGi bundles to use Import-Package to depend on packages from the boot layer.  In order to grant the system.bundle class loader access to all packages available from the boot layer I have to generate a ModuleDescriptor programatically which requires all modules from the boot layer.  The layer must be created with the system.bundle module resolved which maps the module to the class loader used to load the framework implementation before any classes are defined in packages that we want to be exported by system.bundle module.  The ModuleDescriptor used for the system.bundle must specify that it exports the packages from the framework implementation, otherwise JPMS will still associate them with the unknown module. With this modified launcher, any classes defined in the packages we declared in the ModuleDescriptor will be associated with the system.bundle module.  You can find the changes I made to the equinox launder on github at https://github.com/tjwatson/rt.equinox.framework/tree/tjwatson/jpms.  You may notice I hard coded the list of packages to export from the system.bundle module.  This was a hack to get going quickly.  Ideally these packages would be discovered programmatically.

The Bundle Layer

One important detail to understand about JPMS layers is that the class loaders that are mapped to by the modules within a layer MUST NOT have defined any classes in packages for which a ModuleDescriptor declares as exports or conceals.  This implies that the bundle layer used to represent bundle JPMS modules must be created as early as possible and ideally before any classes are loaded using the bundle class loaders.  In order to achieve this I changed the bundle osgi.jpms.layer to a system.bundle fragment still named osgi.jpms.layer.  The OSGi R6 Framework specification added a new feature which allows system.bundle fragments to be activated when the Framework is initializing before the rest of the bundles get activated.  This allows for the code controlling the bundle layer to get in place in order to intercept any class defines from bundle class loaders.  That way we can map the bundle class loaders for resolved bundles to their respective JPMS modules before any classes are defined.  I used a WovenClassListener and WeavingHook to achieve this.  Here I am not interested in actually weaving any class bytes, but these OSGi hooks allow for us to hook directly into the bundle class loader just before it is about to define a class.

We can now insert the code in the correct place to create the bundle layer.  I used a similar approach as before to achieve this, but some more information is needed now that the bundle classes will belong to a named module.  Here are the steps:
  1. Discover all resolved host bundles and map their symbolic name to their wiring.  Note that we could get conflicts if multiple bundles are installed with the same symbolic name.  For this experiment I choose only one to map into the bundle layer.
  2. Create a module finder that is backed by the bundle wirings.  The finder is what creates the ModuleReference and ModuleDescriptor objects to represent the bundles.  The following information is used from the wiring:
    • The bundle symbolic name is the module name.
    • The bundle version is the module version.
    • The the package capabilities are the exports for the module.
    • Private packages must be discovered to specify the module's concealed packages.  Here the private packages are treated as exported by the module instead of concealed.  I will explain why later.
    • Dependencies on other bundles for class loading must become module requirements.
  3. Create a configuration using the bundle finder.  Default to using the system.bundle layer configuration as the parent configuration.
  4. Create a layer that maps each module name to the bundle wiring class loader.
Creating this layer exposes some issues with JPMS that make it difficult and sometimes impossible to properly represent OSGi bundles as modules.
  1. JPMS-ISSUE-001 - Reflection is used by almost any framework in Java and the OSGi Framework is no exception.  In JPMS the JVM will not allow reflection to be used on any class that is not known to JPMS as an exported package.  Once I successfully got every class from a bundle associated with a JPMS module I found that the framework could no longer call Class.newInstance() for bundle activator classes contained in concealed packages!!  In order to get that to work I had to treat every private package from a bundle as exported by the ModuleDescriptor for the bundle.  This will also be necessary for other dependency injection containers on OSGi, for example, Declarative Services.  I also imagine this has to cause issues for other DI containers such as Spring and CDI.
  2. JPMS-ISSUE-002 - Private packages must be discovered and specified to JPMS.  As pointed out already, I had to make the private packages exported by JPMS, but first I tried to make them concealed.  Either way, all packages that are associated with a module must be known to JPMS as either exported or concealed.  If this is not done then the classes from unknown packages will be associated with the unnamed module.  This places an extra burden on the OSGi module system because in OSGi there was no reason for the framework to discover the private packages ahead of time.
  3. JPMS-ISSUE-003 - JPMS must be aware of the OSGi bundle dependencies for class loader access.  If the module descriptors representing OSGi bundles do not declare any module requires then JPMS will not grant the read access required to use a class from another module.  The bundle class loaders will continue to be able to load the classes from other bundles according to import-package and require-bundle rules, but when the class is actually used the JVM will throw access exceptions.  This forces us to translate the OSGi dependencies into module requires.  If there are multiple bundles with the same symbolic name then there is no way to tell JPMS which version of the bundle a module depends on.
  4. JPMS-ISSUE-004 - JPMS layers do not allow cycles between modules.  OSGi bundles are allowed to have cycles.  Since we must make JPMS aware of the OSGi bundle dependencies this restricts us to only bundles that have no cycles.
  5. JPMS-ISSUE-005 - JPMS layers provide a static module resolution graph.  This will prevent OSGi from successfully resolving dynamic package imports if they require read access to a new module.
  6. JPMS-ISSUE-006 - JPMS layers allow for multiple versions of the same module but it does not appear that modules within that layer or contained child layers can influence which version of the module they get resolved to.
  7. JPMS-ISSUE-008 - JPMS layers do not allow for split packages.  If the OSGi bundles are resolved with split packages then the bundle layer cannot be created.
If you can look past these issues we are left with a layer that can represent a static set of resolved OSGi bundles as real JPMS modules and we can use that layer to create child JPMS layers for loading other JPMS modules.

OSGi Bundle Dynamics

The bundle layer we have now represents a static set of resolved OSGi bundles in a Framework.  But the bundles in an OSGi Framework are not static.  They can be uninstalled, updated, re-resolved, and new bundles can be installed.  How can this dynamic nature be represented in JPMS layers?  The approach I took was to create a linear graph of layers where the youngest child layer represents the current state of the bundles.  This would look something like this:
In this scenario we started out with bundle.a and bundle.b resolved in the bundle layer 1.  Then we created a module layer 1 to resolve jpms.a and jpms.b modules.  Then bundle.b was updated and bundle.c was installed and then bundle.b was refreshed in order to flush out its old content and class loader.  This leaves bundle layer 1 with a "dead" bundle.b module which also makes module layer 1 stale.  So we decide to discard module layer 1 and create module layer 2 for jpms.a and jpms.b modules.  To do that we need a new bundle layer that represents the current set of resolved bundles.

Here we cannot discard bundle layer 1 because it still has at least one valid module bundle.a.  We also cannot represent bundle.a module in a new layer because we may have already loaded classes from packages contained in bundle.a.  Instead of throwing away bundle layer 1 a new bundle layer 2 is created that uses bundle layer 1 as its parent.  Bundle layer 2 will contain all the new versions of modules that are not already represented in the parent layers.  This allows the new bundle.b to shadow the "dead" bundle.b module in bundle layer 1.  This appears to work.  The only JPMS module that cannot be shadowed by a child layer is the java.base module.  But we are left with a pretty big issue:
  • JPMS-ISSUE-007 - Discarded modules from a JPMS layer will be pinned in memory until the complete layer is discarded.  This ultimately leads to a huge class loader leak because we cannot properly free up our stale bundle class loaders.  It also causes issues for bundles that are uninstalled completely.  The "dead" modules for these bundles will continue to be available since nothing is shadowing them from child layers.  I suppose we could create a empty module that has the same name but exports nothing, but that will still allow modules on top to resolve when they shouldn't.
Currently the code for the experiment is located in github at https://github.com/tjwatson/osgi-jpms-layer/tree/tjwatson/moduleClassLoader I did this in the tjwatson/moduleClassLoader branch.


Conclusion


This approach allows for a pretty accurate representation of a static set of resolved OSGi bundles as JPMS modules.  But we are left with several issues that need to be addressed before this can be considered a truly viable solution.  Some may decide these are permanent restrictions of JPMS that we will have to live with going forward.  But I believe there are some tweaks to JPMS that could go a long ways to making this approach close to a complete solution.  Listed below are some changes that would help.  I listed them in the order of importance, but I think 1 and 2 are a close tie for most important.

  1. Allow for code that manages a JPMS layer to have more control for establishing read access for the modules contained in the managed layer.  The Module addReads method allows for read access to be added for a module dynamically at runtime.  But it has a restriction that it must be called by a class defined by the module that wants new read access.  It would be a great help if we could call addReads from the management code that created the layer.  Perhaps an addReads(Module wantsRead, Module toTarget) method on Layer that checks the caller module is the same module get created the Layer?  This could be used to solve a large set of issues outlined above:
    • JPMS-ISSUE-003 - We could avoid having to make JPMS aware of the OSGI dependencies if we would be allowed to establish the read access ourselves when the bundle layer is created.
    • JPMS-ISSUE-004 - If we avoid having to make JPMS aware of the OSGi dependencies then we no longer have worry about restricting cycles.
    • JPMS-ISSUE-005 - If we can dynamically add reads then we can enable dynamic package import to work by dynamically adding read access to the provider of the package at runtime.
    • JPMS-ISSUE-008 - If we avoid having to make JPMS aware of the OSGi dependencies then we no longer have to worry about restricting split packages.
  2. Allow for reflection on classes from concealed packages.  Many dependency injection containers depend on being able to act upon concealed classes in order to construct objects and inject the objects with dependencies.  Forcing implementation details to be exported so that these classes can be acted upon by DI containers is wrong.
    • JPMS-ISSUE-001 - We would no longer have to declare the bundle private packages as exported by the JPMS module.  Instead they can remain concealed as they should be.
  3. Allow for sub-graphs of modules to be discarded within a layer.
    • JPMS-ISSUE-007 - This would allow us to flush out the "dead" modules which should never be used anymore.
  4. Allow a layer to map a class loader to a default named module.  Any classes from unknown packages to the JPMS would be assigned this named module instead of the unnamed module.
    • JPMS-ISSUE-002 - This would allow us to avoid having to scan for private packages.  Instead we would map the bundle classloader to a module and that module could be used for the private packages.
  5. Allow the JPMS requires statement to specify a module version.
    • JPMS-ISSUE-006 - This would allow us to represent multiple versions of a bundle within the bundle layer and give JPMS modules the ability to specify which version they want.
My hope is that this experiment is useful in providing constructive feedback to the JPMS expert group.  I hope they consider enhancing JPMS to make JPMS layers more usable with existing module systems like OSGi.

Monday, August 1, 2016

Java Module Layers and OSGi Bundles

NOTE: This article discusses my initial attempt at representing OSGi bundles as JPMS modules.  The implementation of this exposed a flaw in the approach.  You can jump to the Lessons Learned section for the conclusion.  Since then I have done another attempt at properly representing OSGi bundles as real JPMS modules.  You can see that article here.

With the Java 9 release, the Java Platform Module System (JPMS) is coming.  The JPMS will finally modularize the class libraries provided by the JVM.  In addition, the JPMS can be used by developers to modularize applications.  This allows developers to split their applications into modules.  These modules can then specify what other modules they require and what packages they export for use by other modules.

The OSGi specification has been providing a module system for Java applications for a long time already which also allows developers to modularize their applications into modules (a.k.a. bundles).  With OSGi, developers have created many modular applications that are also extensible by installing more bundles provided by third parties.  I have been involved with two projects that do just that.
  1. The Eclipse IDE uses OSGi as its module system.  Eclipse plug-ins can be installed to provide additional tooling support for the IDE.
  2. WebSphere Liberty uses OSGi as its module system which allows the server to be configured with only the features which are required by the applications provisioned to the server.
When JPMS is released, developers can start to deliver their own Java modules for the JPMS.  What happens when developers want to use Java modules to compose applications which are running on a container that is built using the OSGi module system?  Will such a container be able provide its APIs in such away that JPMS modules can require and access them?  For this post I will focus on this scenario which is similar to WebSphere Liberty.  But before I go into the details of the problem lets first explore the JPMS and the concept of Layers.

JPMS Layers and Modules


A layer in JPMS is a static set of resolved modules.  Each layer has a single parent layer except the empty layer which has no parent.  Layers are hierarchical and can have no cycles.  A layer provides the JVM with a graph which determines how classes are located during class loading.  Once a layer is created, none the modules within the layer can change.  This allows the class loading graph to be locked in when the layer is created.  Modules in one layer can require any module provided in a parent layer.  This includes all parents in the hierarchy all the way down to the empty layer   In order to update a module, the complete layer in which a module is contained must be thrown away and recreated in order to provide a new resolution graph for the layer.  If any module within a layer in the hierarchy needs to be updated, then that layer as well as any children of that layer must be torn down and recreated.  If you need to load up another module provided by JVM in the boot layer, then the complete JVM has to be restarted so that the JVM can recreate the boot layer.

This provides a stable and predictable class loading behavior but it poses a problem for containers that are built using the OSGi module system.  The OSGi module system is much more dynamic.  Modules (bundles) in OSGi are not placed in hierarchical layers which are resolved in orderly stages like the JPMS layers.  In OSGi, bundles that have no dependency on each other can be resolved independently in time of each other.  Not only that, but new bundles can be installed and existing bundles can be updated or uninstalled.  All this without tearing down the Framework or effecting existing bundles within the framework that do not depend on the other bundles being updated, installed or uninstalled.  How would a container built on a dynamic module system be able to provide a JPMS layer which can be used as a parent of a another layer which contains JPMS modules?  Modules in JPMS can only load classes from packages exported from modules within their own layer or one of the layers in their parent hierarchy.  If a container is providing APIs which are exported by OSGi bundles then any API which can be used by applications composed of JPMS modules must be represented within a JPMS layer somehow.  The following diagram illustrates the possible layers with this scenario:


The boot layer contains the JPMS modules which were configured with the JVM when it was launched.  In this diagram, the framework implementation has not yet been migrated to a JPMS module itself.  It is loaded as part of an unnamed module using the old -cp or -jar option when launching the JVM.  The Framework then has two bundles installed: bundle.a and bundle.b.  The Framework resolves these two bundles and creates a class loader for each bundle.  Each class loader in Java 9 now has its own unnamed module associated with it.  Any classes defined by the bundle class loader will be associated with this unnamed module that is associated with the bundle's class loader.  In this scenario there are three distinct unnamed modules.  The goal of this experiment is to figure how how to represent bundle.a and bundle.b as JPMS modules in a layer and then create another JPMS layer as a child which has JPMS modules that require bundle.a and bundle.b as modules.

My Experiment


Over the past few days I have been investigating a way to use the JPMS Layer API in order to construct such an OSGi-JPMS layer.  My goal was to do this completely on top of a standard OSGi Framework implementation without any modifications to the Framework itself.  Now lets dig into the details of my experiment.  My first step was to figure out how to construct a layer based on a finite set of resolved bundles that are installed within the OSGi Framework.  For this experiment, I ignored the fact that the bundles within the bundle JPMS layer could be refreshed at anytime.  To do this, I decided to place the code which constructs this layer into an OSGi bundle itself.  I could have done it with a launcher outside of the framework, but it seemed more simple just to put it into a bundle which can be installed onto any framework no matter how it is launched.

Creating a Layer is a simple process for JPMS modules containing a module-info.class and you want the layer implementation to do all the work for resolving the modules and providing the modular class loader behavior.  The module-info.class file identifies the module, what it provides, and what it requires.  This file gives the layer implementation all it needs to resolve and provide a class loader implementation for the module.   But in my case I wanted to reuse existing bundle class loaders to back the JPMS modules that represented the OSGi bundles.  I did not have any interest in the JPMS layer actually resolving the bundles or providing any class loader behavior. This must continue to be handled by the OSGi Framework.  It turns out the Layer API does provide the ability to map modules to customized class loaders.  So my first step was to discover the set of bundle class loaders I wanted to include in the bundle layer.  To do this, I used the FrameworkWiring findProviders method to find bundle wirings for host bundles which have a class loader associated with them.  The wirings were used to create a ModuleFinder.  A ModuleFinder is used by JPMS in order to find ModuleReference/ModuleDescriptor objects for something called a Configuration.  My ModuleFinder implementation mocked up ModuleDescriptions based on the set of bundle's symbolic names, versions, and exported packages.  I did not bother with any requirements for the ModuleDescriptors for the bundles since I did not care about resolving the bundles in the context of the JPMS.

The next step is to create a Configuration.  Configurations, similar to Layers, have a single parent Configuration.  I based the bundle configuration off the boot configuration by using the resolveRequires method.  Once a configuration is created, it can be used to create a layer.  Here I used the defineModules method which takes a Function<String, ClassLoader> that is used to map a module name to a class loader.  This function allows us to map the bundle JPMS modules to the real bundle class loaders.  There are a couple of gotchas here.
  1. It is dangerous to use the actual bundle class loader because errors will occur if any classes are already defined for the packages exported by the bundle when the Layer is created.  To work around this I simply created an empty class loader that uses the bundle's class loader as its parent.
  2. All the classes defined by the bundle's class loader will not have the JPMS bundle module associated with them as returned by the Class getModule method.  Instead they will be associated with the unnamed module associated with the bundle's class loader.  This is an important detail when working out how to create the JPMS layer built on top of the bundle layer.
Now that the bundle layer has been created, it can be used to create additional child layers which can then require the bundles by referencing their symbolic names.  This can be done by using the built-in ModuleFinder of method which provides a finder that can discover JPMS modules that use the module-info.class to describe themselves.  This finder can then be used to create a resolved configuration based off the configuration of the bundle jpms Layer.  But there is a catch.  Even though the new configuration correctly resolves the JPMS modules which require the OSGi bundles, they still will not successfully be able to load classes from the packages exported by the OSGi bundles. This is because the JPMS layer only grants read access to the classes defined by bundle modules existing in the bundle JPMS layer.  This makes sense but the actual classes defined by the bundle's class loaders are associated with the unnamed module of the bundle's class loader.

Here is where this experiment turns into a hack, to be honest.  In order to get this to work I had to invoke the addReads method.  This method would allow the modules in the child JPMS layer to actually be able to read (load) the classes from the unnamed modules that are actually backing the bundle JPMS modules.  But the addReads method MUST be called from code loaded by the module that wants read access added to themselves.  In order to do this I ended up augmenting the JPMS module content on the fly by implementing my own ModuleReader.  This module reader injected a specific class into each module which I could then call reflectively in order to "convince" Java 9 into thinking the module itself is really calling addReads.  This is even more awkward due to the fact that the package this injected class is in has to be accessible by the calling code.  To make this "simple," I just exported that package from each JPMS module.  This did not take a lot of code, but it is rather tricky and far from obvious.  After that though, it was possible to get JPMS modules that require OSGi bundles using their bundle symbolic names.  You can find the complete experiment on GitHub at https://github.com/tjwatson/osgi-jpms-layer.

Lessons Learned


I learned a bit by doing this experiment, but in the end I don't think it is really a viable solution to the problem because of the hacks needed to grant read access to the unnamed modules.  I think it has convinced me that there may be a viable solution by creating a true JPMS Layer which gets real JPMS modules associated with the bundle class loaders upfront so that each bundle class loader is able to define classes that are associated with named class loaders.  I hope to get time in the next week or so to do that experiment.  The high level things I need to learn are:

  1. Can I intercept the bundle class loaders before they define any classes such that I can map them to their JPMS module representations?  I am pretty sure I can use an OSGi WeavingHook to do this since weaving hooks get called before classes are defined.
  2. Can I manage to create a linear layer hierarchy as bundles are resolved such that the latest layer can be used to build child JPMS layers from?  This depends on the order layers search for modules.  The findModule method seems to indicate the "local" layer is searched before asking its parent layer.  I find that hard to believe because it would seem to allow one layer to override things like the java.base module.  But this would be critical so that we can ignore stale modules from layers that have bundles that have been uninstalled, updated, or refreshed.
If I cannot manage to figure out how to have a linear parent chain that ignores stale modules then we still should be able to create layers that represent OSGi bundles with real modules, but the layer may not be able to accurately reflect the full dynamic nature of the OSGi Framework.

Blog Archive