OSGi Core R8 Specification Draft Available - Get Connected

The OSGi Core Expert Group has released a draft of the upcoming OSGi Core Release 8 specification. The draft includes two new specifications for the Core Framework as well as a number of smaller improvements.

• OSGi Connect - Provides a mechanism to connect bundles in the Framework with content managed outside of the Framework itself
• OSGi Condition Service - Provide a mechanism to signal that a condition is satisfied at runtime

For this post, I will focus on the OSGi Connect specification included in chapter 60 of the draft specification. Karl Pauls previously blogged about OSGi Connect last year and now I would like to describe the progress made since that post for the draft specification.

A Bit of History

Over the last 20 years, the OSGi Framework has provided a solid foundation to develop modular software.  This foundation is built upon the various layers provided by the Core OSGi Framework itself.

At the bottom of the foundation is the module layer.  This layer provides the rules for how modules (or bundles) can share or hide the packages they include in their own bundle JAR.  The Framework provides the class loader implementation that enforces the rules of the module layer.  This includes a rich capabilities and requirements model and a resolver that wires up the requirements to the available capabilities in the Framework.  Once a bundle is installed and resolved it is then able to participate in the layers that are built on top of the module layer.

The lifecycle layer provides control over the activation of the bundles resolved in the Framework.  Activation provides an entry point for each bundle to allow them to interact with the Framework and other bundles installed in the Framework.  A bundle may include a bundle activator which allows them to execute code when they are activated.  The activation concept also allows for more powerful runtimes to be built on top that can use introspection of the bundle capabilities and provide a mechanism for enabling the capabilities of the bundle at runtime.  This concept is called the extender pattern.  Two good examples of the extender pattern are the OSGi Declarative Services specification and the CDI Integration specification.  These two examples provide powerful dependency injection runtimes for developers.

The service layer provides a dynamic, concise and consistent programming model for developers, simplifying the development and deployment of services by de-coupling the service's specification from its implementations. This model allows developers to bind to services only using their interface specifications. The selection of a specific implementation, optimized for a specific need or from a specific vendor, can thus be deferred to runtime. The service layer model provides a common layer which allows for things like declarative service components and CDI components have dependencies on services available in the service registry.  This is powerful because it allows the components from declarative services and CDI to interact with each other through a shared layer without needing to know the details of each other’s runtimes.

The combination of the activation model with the service model in OSGi provides for an unparalleled platform for developing loosely coupled components that are highly configurable at runtime. In order to use this powerful tool, developers must be able to live within the confines of the OSGi Framework module layer. In some scenarios, it is challenging to live within the OSGi module layer as it is defined today. The following are a few examples:

• Modules built into a jlink image. The jlink tool was introduced in Java 9 along with the Java Platform Module System (JPMS) and provides a way to “right size” the JVM along with a set of application modules to provide an image that only includes what is required by an application’s modules. At runtime Java modules are typically loaded by class loaders provided by the JVM.  In the case of a jlink image all the modules are loaded by a single class loader.  This limits the ability to load bundle content out of the modules included in a jlink image.  Similar limitations exist with modules provided on the module path.
• Applications natively compiled using something like Graal Substrate.  With native-image compilation not only is it not possible to have a custom class loader, the classes at runtime are not really loaded at all.  Instead they simply exist with the execution of the native-image. This limits the ability to load bundles out of content compiled into the native-image.
• Bundles included on the class path or more generally loaded by some form of the URLClassLoader. The concept of the Java class path has been around since the early days of Java and many of today’s popular frameworks take advantage of the class path environment.  For example, Spring Boot loader mimics the class path when a Spring Boot application is packaged as a uber JAR (a JAR with many embedded JARs).  It has been challenging to integrate OSGi technologies into such environments.
• Other environments that compile Java into alternative deployment artifacts, such as Android, limit the ability of the framework to control the actual installation and deployment of new bundles.

The Idea to Connect

It would be helpful to allow content that lives outside of the module layer to participate in the lifecycle and service layers of OSGi. The new OSGi Connect specification introduces a mechanism that allows content managed outside of the Framework control to be represented (or connected) with a bundle installed in the Framework.  Because the content is managed outside of the Framework itself it may not follow all the rules of the OSGi module layer. For example, multiple bundles connected to outside content may be loaded by the same class loader. The class loader loading the content may not provide the same isolation as the Framework managed class loaders. With Connect now that content has the ability to participate in the activation model as well as the service layer of the framework.

The Connect specification introduces a concept of a module connector. The module connector is called by the framework when the framework needs to access content of the bundle. The content of the bundle includes the following:

• A list of entries contained in the bundle and access to read from them.  For example, a declarative service component XML file, the bundle manifest, or any other resources included in the bundle.
• An optional map of bundle manifest headers. Typically bundle manifest headers are specified in the bundle’s META-INF/MANIFEST.MF entry, but for connect content the headers may come from an alternate source.
• An optional class loader for the bundle. If a class loader is provided then the framework must use it for the bundle and must not create a framework managed class loader.

In order to install connect bundles, the Framework must be created with a module connector instance (defined by the interface org.osgi.framework.connect.ModuleConnector). The module connector hooks into the initialization of the framework as well as the activation, allowing it to interact with the Framework lifecycle and service layers. More importantly the module connector hooks into the installation of bundles in the Framework.  When a bundle is installed into the Framework a location string and an optional input stream to the bundle content is provided by the installer.  When no input stream content is provided the Framework must determine what the content of the bundle is.  Typically the Framework implementation will attempt to convert the location string into a URL and load the bundle content from there.

When a module connector is used the Framework must first ask the module connector if it can provide content for the bundle location.  If the module connector can provide content then it supplies a connect module to the framework (defined by the interface org.osgi.framework.connect.ConnectModule). A connect module is then associated with the connect bundle installed in the framework.  For each revision of the connect bundle the connect module provides connect content to the Framework (defined by the interface org.osgi.framework.connect.ConnectContent). A connect content provides access to read entries out of the content, provides an optional class loader for the content and may provide the bundle manifest headers for the content.

To create a new Framework that uses a module connector a new framework factory has been introduced with the interface org.osgi.framework.connect.ConnectFrameworkFactory.  A Framework implementation that supports the Connect specification provides a ConnectFrameworkFactory just like the Framework provides the org.osgi.framework.launch.FrameworkFactory.  A launcher that is looking for a factory to create a Framework instance can use the Java ServiceLoader to load a ConnectFrameworkFactory implementation.  This factory can then be used to create a new Framework instance that uses a module connector instance.

Seeing it in Action

While developing the Connect specification, Karl Pauls (Apache Felix project lead) and myself (Tom Watson - Eclipse Equinox project lead) have been busy implementing the Connect specification in our respective OSGi Framework implementations. I have also been working on a project called Atomos that I used as a proof of concept to test out the ability of the Connect specification to connect bundles to various content from different environments, such as: a jlink image, Graal substrate, Spring Boot uber JAR and an Android Dexified JAR (still a work in progress).

With the Connect specification entering into the draft phase, Karl and I thought it would be a good idea to have my Atomos project contributed to the Apache Felix project.  This has been done and is now available in the GitHub repository https://github.com/apache/felix-atomos

Among other things, the Atomos project provides a runtime with a module connector and a launcher that can be used to easily launch a framework and a set of bundles contained on the class path or module path.  For example, to launch a framework with the Apache Felix Gogo console you could have the following directory containing the necessary JARs:

bundles/

bundles/atomos.osgi.framework-0.0.1-SNAPSHOT.jar

bundles/jline-3.14.0.jar

bundles/org.apache.felix.atomos.runtime-0.0.1-SNAPSHOT.jar

bundles/org.apache.felix.gogo.command-1.1.0.jar

bundles/org.apache.felix.gogo.jline-1.1.0.jar

bundles/org.apache.felix.gogo.runtime-1.1.2.jar

bundles/org.eclipse.osgi-3.16.0.tjwatson_osgiConnect11.jar

The org.apache.felix.atomos.runtime JAR is the Atomos runtime snapshot built from the felix-atomos GitHub repo.  It contains the Atomos module connector implementation and the AtomosLauncher class which discovers the framework implementation and launches it with the Atomos module connector. The org.eclipse.osgi JAR is a snapshot of the Equinox framework that implements the connect specification. The atomos.osgi.framework JAR is a Java module that acts as a facade for the Framework implementation so that the Atomos runtime module can require it instead of requiring directly the org.eclipse.osgi module.  This allows Atomos to work with other Framework implementations, such as Felix.  The atomos.osgi.framework JAR is only required if you are launching from the module path.

To launch Atomos with the Gogo console using the class path, the following Java command can be used:

java -cp "bundles/*" org.apache.felix.atomos.launch.AtomosLauncher

To launch using the module path instead the following can be used:

java -p bundles -m org.apache.felix.atomos.runtime

Both will give you a gogo console with the bundles, Framework implementation, and Atomos all loaded from the same class loader.  The Atomos runtime does support Java 8 when using the class path, but class path mode can also be used with Java 11. If you are using Java 11 or higher you will notice that a number of other bundles are installed from the modules included in the JVM.  Atomos discovers all the modules that got loaded and will map each of them as an OSGi connect bundle.  This includes not only the modules from the specified module path, but also the modules from the boot layer of the JVM.  It includes things like the java.base module.  For example, if you run the Gogo command “lb” you will see things like the following:

g! lb | grep java.base

   38|Active     |    1|java.base (11.0.6)|11.0.6

Atomos will read the module descriptors and generate an equivalent OSGi bundle manifest for it so that it can be represented as a connected bundle in the Framework.

At this point, you have a fully functional OSGi Framework instance.  You can even install other bundles dynamically.  Any additional bundles that you install will not be connect bundles because they will not have been included on the original class path or module path at launch.  That means they will have the typical Framework managed class loader and have all the expected behaviors of living in the OSGi module layer.

The Atomos project also has a number of examples that can be looked at in https://github.com/apache/felix-atomos/tree/master/atomos.examples. This includes a jlink, Spring loader and a few Graal Substrate examples.  The Spring Loader and substrate examples all include a version of the Apache Felix WebConsole and the substrate examples also include the Felix SCR (Declarative Services implementation) and a set of test bundles that have declarative service components.  On my laptop the substrate examples can be launched in an impressive 40 milliseconds.

We are also working on a Maven plugin in Atomos to make the configuration of a Graal substrate compilation easier and automatic.  Also in the works is the ability to produce something that can easily be used on Android.

I am excited about the upcoming OSGi Core R8 specification release with the Connect specification.  I think this will help enable the use of OSGi technologies in environments that were not previously easy to do and in some cases not possible.  Now go download the draft specification and read more details about the new Connect specification.

OSGi R7 Highlights: Java 9 Support

With the Java 9 release, the Java Platform Module System (JPMS) is a major feature that impacts many areas of the Java Platform.  The JPMS is used to modularize the class libraries provided by the Java Platform.  In addition, it has been proposed that the JPMS can be used by developers to modularize their own applications, although the OSGi Alliance cautions against its use here.

Meanwhile the OSGi Core specification has provided a stable platform for developing modular Java applications for well over 17 years.  The OSGi Core R7 specification continues this tradition by providing updates to the OSGi Core specification which are fully compatible with previous versions of the OSGi Core specification.  OSGi Core R7 Framework implementations will be able to run on top of Java 9 without requiring any changes to existing OSGi application bundles (assuming no internal packages from the Java platform are being used).

The OSGi Core R7 specification has a couple of new features that build upon some features new to Java 9.  This includes support for multi-release JARs and runtime discovery of packages provided by the JPMS modules loaded by the platform.  First I will tell you about the multi-release JAR support.

Multi-Release JAR Support

Java 9 introduces a new type of JAR called a multi-release JAR.  A multi-release JAR file allows for a single JAR to support multiple major versions of the Java platform. For example, a multi-release JAR file can depend on both the Java 8 and Java 9 major platform releases, where some class depend on APIs in Java 8 and other class depend on APIs in Java 9.

The purpose of multi-release JARs is to support alternative implementation of select classes to deal with changes in the visible APIs of the Java platform.  That is, it is not meant as a means to supply new function or new API on different Java platform versions. As a best practice new APIs should be released in an updated version of an existing bundle or released as part of a new bundle.

The OSGi Core R7 specification adds support for multi-release JAR files. An OSGi bundle file can be a multi-release JAR by adding the following manifest header to the bundle manifest:

 Multi-Release: true

Any JAR included on the bundle class path can also be independently declared as a multi-release JAR by including the same manifest header:

  Multi-Release: true

in its own JAR manifest. For example, an OSGi bundle can include third-party multi-release JARs on its own bundle classpath. In this case the third-party multi-release JARs will be loaded as multi-release JARs by the OSGi Framework even if the bundle JAR itself is not a multi-release JAR.

When a JAR on the bundle class path is a multi-release JAR, then the Framework must search the JAR's versioned directories when attempting to locate a class or resource in the JAR. For more information on how classes and resources are loaded from multi-release JARs see the multi-release JAR documentation.

Different implementations of a Java class for different versions of the Java platform can affect the requirements of the bundle.  For example, what packages need to be imported from the running Java platform. The R7 Framework supports supplemental manifest files that can be used to specify alternative values for the Import-Package and Require-Capability manifest headers for different versions of the Java platform.

When a bundle file is a multi-release JAR, then the Framework must also look for a supplemental manifest file, OSGI-INF/MANIFEST.MF, in the versioned directories. For example:

  META-INF/versions/9/OSGI-INF/MANIFEST.MF

The Framework finds the highest versioned manifest available in the multi-release JAR which can be applied to the running Java platform and uses the Import-Package and Require-Capability manifest headers as replacements for the values specified in the bundle's manifest.

The purpose of the supplemental manifest is to specify requirements on capabilities provided by different versions of the Java platform which are needed by the implementation classes for that Java platform version. As a best practice, the supplemental manifests should not contain additional requirements on capabilities which are not supplied by the Java platform for the Java version associated with the supplemental manifest.

For more information about multi-release JAR support you can refer to the following sections of the OSGi Core R7 specification:

1. Bundle File Multi-release JAR
2. Bundle class path Multi-release Container

Runtime Discovery of Java Platform Packages

Now that the Java platform is modularized in Java 9, the runtime can be configured to load only the modules which are required by the application.  This allows for a smaller custom runtime that is tailored to the needs of a specific application.  The set of java.* packages provide by the running Java platform is no longer constant for a specific version of the Java platform.

Bundles typically depend on different packages by using requirements specified with the Import-Package header, but as of the OSGi Core R4 specification, bundles have been prohibited from using Import-Package to specify requirements on java.* packages.  Instead bundles have used the osgi.ee namespace (or the deprecated Bundle-RequiredExecutionEnvironment header) to specify a dependency on specific versions of the Java platform. It has been assumed that the set of java.* packages available for a specific version of the Java platform is constant and therefore there is no need to specify additional requirements on specific java.* packages.

With Java 9 this is no longer a valid assumption.  With OSGi Core R7, the osgi.ee namespace should only be used to specify the minimal level of the Java platform required by the Java class bytecode level included in the bundle.  Dependencies on specific java.* packages should to be specified using the Import-Package header.

The OSGi Core R7 specification now allows bundles to use the Import-Package header to import java.* packages.  The OSGi Framework implementation is required to discover the all available java.* packages in the running Java platform and automatically provide the java.* package as a separate system packages (see the system packages configuration property).  The system bundle is the only bundle that is allowed to provide java.* packages. It is an error for normal bundles to attempt to export java.* packages using the Export-Package header.

When importing java.* packages only the package name should be specified, no other matching attributes are needed.  For example:

  Import-Package: java.sql

Note that bundle class loaders in R7 continue to have full visibility to all available java.* packages available in the running Java platform.  The class and resource loads for the java.* packages will continue to be boot delegated by the framework implementation even when the bundle does not specify any Import-Package header. The java.* package requirements are only necessary to prevent a bundle from resolving if the necessary java.* packages are not available at runtime.  For example, if a bundle must only resolve if the java.sql module is loaded.

For more information about java* package support you can refer to the following sections of the OSGi Core R7 specification:

1. Execution Environment
2. Parent Delegation 

Bundles Supporting R6 and R7 Frameworks

The OSGi Core R6 specification prohibits bundles from importing java.* packages and will throw a BundleException if such a bundle is attempted to be installed. Bundles that must work on OSGi R6 and earlier Frameworks but also want to run on R7 Frameworks have two options:

1. Do not import the java.* packages.  The bundle will continue to work on an R7 framework because java.* packages are still boot delegated by the bundle class loaders. This implies the bundle may resolve when running on Java 9 even when all the java.* packages required by the bundle are not available.
2. Package the bundle as a multi-release JAR to allow alternative values for Import-Package to be specified.

A bundle that must still be able to be installed on an OSGi R6 or earlier Framework must not specify java.* packages in their main bundle manifest Import-Package header. A supplemental manifest must also be included to be able to specify java.* package imports.

This supplemental manifest must contain the original Import-Package header from the main bundle manifest as well as any java.* package required by the bundle. When running on an OSGi R6 Framework, this supplemental manifest will be ignored and therefore the bundle will install successfully as the bundle did not specify an import for the java.* packages in the main manifest.

If a bundle will always require OSGi R7 or greater Framework, there is no need to use multi-release JARs for this purpose even when the bundle must support Java 8 or earlier. This is because OSGi R7 frameworks must provide the java.* package capabilities on all Java platform versions the framework implementation supports. As long as the java.* package is available on the running Java platform, the bundle with an import for the java.* package must resolve.

I advise using option 1 here because it is by far the most simple approach.  If you really have a strong need to use this new R7 feature then I recommend you do not try to produce a single bundle version that supports both R6 and R7 Frameworks.  Instead simply release a new version of your bundle that only supports OSGi R7 or greater Frameworks.

In Summary

The OSGi Core R7 release continues the tradition by providing a stable modular platform with updates to the OSGi Core specification which are fully compatible with previous versions of the OSGi Core specification.  OSGi bundle developers can continue to develop advanced modular applications and run their applications on Java 9 without having to commit to migrating to the green initial release of JPMS.

The OSGi Core R7 specification does add new functionality in support of new features available in Java 9, such as the support for multi-release JARs and importing of java.* packages.  The OSGi specifications will continue to evolve in order to provide the necessary tools for developing advanced modular applications.

OSGi R7 Highlights: Proposed Final Draft Now Available

I am pleased to announce that the OSGi Alliance has published the Proposed Final Drafts of the OSGi Core R7 and Compendium R7 specifications. We expect that the final versions of these specifications will be published in April 2018 after OSGi Alliance member approval.

The R7 release builds upon the long history of the OSGi Alliance’s leadership in Java modularity and reflects a significant amount of effort from the technical members of the OSGi Alliance expert groups over the last 2 years. Thanks go to all of the members who have contributed to this release.

R7 represents many significant new features and capabilities and provides an open standards-based approach for a number of modern valuable and simple-to-use technologies important to Java developers.

This blog post is the start of a series of blog posts from the technical experts at the OSGi Alliance to share some of the key highlights of R7. The blog posts in this series will come out over the coming weeks and cover the following topics:

• Java 9 Support – Multi-release JAR support and runtime discovery of the packages provided by the JPMS modules loaded by the platform.
• Declarative Services – Constructor injection and component property types.
• JAX-RS – A whiteboard model for building JAX-RS microservices.
• Converter – A package for object type conversion.
• Cluster Information – Support for using OSGi frameworks in clustered environments.
• Transaction Control – An OSGi model for transaction life cycle management.
• Http Whiteboard – Updates to the Http Whiteboard model.
• Push Streams and Promises – The Promises packages is updated with new methods and an improved implementation and the new Push Streams package provides a stream programming model for asynchronously arriving events.
• Configurator and Configuration Admin – Configuration Admin is updated to support the new Configurator specification for delivering configuration data in bundles.
• LogService – A new logging API is added which supports logging levels and dynamic logging administration and a new Push Stream-based means of receiving log entries is also added.
• Bundle Annotations – Annotations that allow the developer to inform tooling on how to build bundles.
• CDI – Context and Dependency Injection support for OSGi developers.

Stay tuned and I hope you find the technical information in the blog post series useful to you as developers!

OSGi and JSR 376 Java Platform Module System (JPMS)

The OSGi Alliance is aware that there are a number of conversations taking place about the Public Review draft of JSR 376 Java Platform Module System (JPMS). This is planned to be a major feature of Java 9 and is under ballot to complete Public Review. The OSGi Alliance will analyze JPMS and may comment on our findings in the future.

In the meantime our Expert Groups are busy working on the OSGi Release 7 which is planned for October this year. You can find the RFPs and RFCs for the new Release 7 features at https://github.com/osgi/design and draft specifications at https://www.osgi.org/developer/specifications/drafts/.

BJ Hargrave
OSGi Alliance CTO

InfoQ: A comparison of OSGi and the Java 9 Java Platform Module System

In Java 9, OSGi and the Future of Modularity, Neil Bartlett and Kai Hackbarth provide part 1 of an excellent comparison of OSGi modularity and the current state of Java 9's Java Platform Module System (JPMS) modularity.

One of the most common complaints about OSGi is that it can increase complexity for a developer. There is a grain of truth here, but people who make this complaint are mistaking the medicine for the disease.
Modularity is not a magic dust that can be sprinkled onto an application just before release. It is a discipline that must be followed throughout all phases of design and development. Developers who adopt OSGi early and apply modular thinking before writing a line of code realise enormous gains[...]

Project Jigsaw started with a goal of being simple, but the JPMS specification has increased enormously in complexity: the interplay of modules with class loaders; hierarchical layers and configurations; re-exporting requirements; weak modules; static requirements; qualified exports; dynamic exports; inherited readability across layers; multi-module JARs; automatic modules; unnamed modules… all these features have been added as the need for them became clear. A similar process happened in OSGi, just with a 16-year head start.

The article is well written and provides a clear understanding of the differences between OSGi and JPMS as it stands today. Looking forward to part 2.

JSR 294 SuperPackages

This blog is updated after a pleasant conversation with Alex Buckley over email in which he explained some of the finer points. These sections are marked with []. JSR 294 has produced their public draft of superpackages! (Somehow the exclamation mark seems necessary with this name ... ) Anyway, superpackages are a new construct for Java 7 to improve the modularity of the Java language. Originally JSR 277 was going to take care of modularity but Gilad Bracha (formerly Sun) thought that deployers could not be trusted with language elements and spun off JSR 294 after he published a spin about superpackages in his blog. Gilad left, but the JSR 294 expert group churned on and they produced a public draft, this blog is a review of that draft.

Superpackages address the ever present need to modularize. Types encapsulate types, fields and methods, packages encapsulate types, and superpackages encapsulate, packages and nested superpackages. When Java was designed packages were intended to be a set of closely related types, a.k.a. a module. However, systems have become larger and larger and packages turned out to not have the right granularity for many. It could have worked if packages had been nested, but this would of course have limited the flexibility of the programmer and Sun used to favor configuration over convention.

For the OSGi specifications we came up with the JAR file as a module of packages. Packages can be exported and imported, allowing the developer to keep classes private or expose them to be used by others. Modularity in OSGi is therefore a deployment concept, the same packages can be members of different modules/bundles. OSGi service platforms enforce these rules with the aid of class loaders.

However, purists want to have the modularity in the language itself, therefore superpackages were born in Sunville. Superpackages group a set of named (super-)packages and export types from these packages.

First let me make clear that I have not seen anything in the public review draft that would make it hard for OSGi to support superpackages. The current incarnation of the OSGi framework will be more or less oblivious of superpackages. The accessibility rules are enforced by the VM and the system works with normal class/resource loading rules. Superpackages and its member types must come from the same class-loader/bundle, but that does not look like an issue. So from an OSGi point of view there is no reason for concern with JSR 294 as long as a superpackage and its member packages come from the same bundle.*

So the remainder is just a technical review of the technical merits of 294, the OSGi specifications are not affected by it.

The superpackages specification is surprisingly complex and there is no overview. The spec is a set of changes to the existing Java Language Specification. I'll try to show you my understanding.

The current model is depicted in the following picture.


When you begin a class you give it a package name with the package keyword:

package com.foo;
class A { ... }

That is, the class defines its membership to a package. This is different from a superpackage that honors its name looking at the complexity it introduces. The following picture shows a similar diagram when superpackages are introduced.


This clearly is supercomplex in comparison with the current model. The major drawback is the redundancy that is introduced by this model. When I went to school ages ago I was thought that redundancy is evil. Over time I learned that redundancy can improve a system but the cost always remains. In this case the redundancy actually seems to create a very brittle structure that makes deployment unnecessary rigid. Let's explore.

[] Before first. As you can see, the Package is dashed now. The reason is that Sun is a bit ambiguous about packages in this specification. In this specification a package is treated as a name with structure (com.foo.bar is a sub-superpackage of com.foo), this is not uncommon. However, in Java a package is (well should be) a first class citizen but it often is not. Packages do provide access rules and there is the java.lang.Package class (not in java.lang.reflect). However, this class is not reflective; it is impossible to find the classes belonging to a package. Would you accept that a class could not enumerate its fields and methods? So why should a package not be able to enumerate its members? In contrast, a superpackage is in java.lang.reflect can enumerate is contained superpackages and, strangely enough, its member types, not its packages. Interestingly, to be able to enumerate the types the VM must be able to enumerate the packages because the superpackage class file only enumerates the package names, not the individual members. I would advise the JSR 294 expert group to take the packages seriously and allow full reflection.

First, superpackage links are always bidirectional and can therefore easily be wrong. A class file points to the superpackage file and the superpackage file points to the package of the class files. That is, if you move a package to another superpackage the superpackage definition file and the sources of the types must be modified. Fortunately wildcards can be used in the superpackage to identify the exported classes, though this means that most (if not all) changes in a type require recompilation of the superpackage definition.

The same is true for superpackage parenthood (enclosement) and membership (nesting). This link is also bidirectional, the parent must list its children and the children must list their parent. A superpackage is restricted to one parent, it is a true hierarchy. By the way, the top of the hierarchy is the unnamed superpackage. Oh yes, also all superpackages with a simple name (no dot in them) are automatically visible (in scope) to any superpackage if I read 7.4.5 correctly (took me some time to figure that out).

[]The unnamed superpackage is really special. Any top level super package is automatically a member of this supersuperpackage. Any class can see any exported type from a top level superpackage. I missed this the first time, the specification could make this more clear. Because the rules for the unnamed superpackage are so different. For example, membership is automatic, all exported members are visible to anyone, and the unnamed package is not open for reflection, it is represented as null. I wonder if this unnamed superpackage should not just be named the global space. I.e. there is no supersuperpackage so do not imply it by calling the superpackage that shall not be named?

The data structure for superpackages is not elegant. Though we have good refactoring tools today in Eclipse, IDEA, and Netbeans, I do not think it is a good excuse to design data structures that are so error prone.

However, the restriction is something that worries me most because it seems to create a system that I have hard time to see how it should work. The restriction is intentional, from section 7.4.5:

If a superpackage did not have to declare which superpackage it is nested in, then the following problem could occur. Consider these superpackages, where Outer.Inner does not declare that it is nested in Outer.

superpackage Outer {
member superpackage Outer.Inner;
}
superpackage Outer.Inner {
member package foo;
export foo.C;
}

If a type outside the Outer.Inner superpackage tries to access foo.C, then the access
would succeed because foo.C is exported from Outer.Inner and neither C.class nor the superpackage file for Outer.Inner mentions the fact that Outer.Inner is a non-exported nested superpackage of Outer. The intent of the Outer superpackage - to restrict access to members of Outer.Inner - is subverted.

Clearly, they have chosen restriction over convenience. However, the consequences of this are quite far reaching. Let us take a look at the access rules. I always need pictures for these things so we need a legend:

The first access rule in 5.4.4. in the specification reads that type C is accessible to type D if any of the following conditions is true:

1. Type C is public and is not a member of a named superpackage
   
   
   
2. Type C is public and both type C and D are a member of the same superpackage S.
   
   
   
3. Type C is public and C is an exported member of named superpackage S and D is a member of the enclosing superpackage O of superpackage S
   
   
   
4. Type C and D reside in the same package p.
   
   

At first could not understand how one of the most common cases, a library provider, could work with superpackages. The rules state that a type can only see what is available to its superpackage. This seems to exclude visibility between peer superpackages. For example, if OSGi would put all its specification packages in the org.osgi super package, a member type of the com.acme package could not see the OSGi exported types. However, after a lot of puzzling I found that in 7.4.2 it states that "A superpackage name can be simple or qualified (§6.2). A superpackage with a simple name is trivially in scope in all superpackage declarations."

I guess this means means that a super package has all superpackages with simple names as superpackage members? If this interpretation is true, then any "top" level superpackage would be visible to anybody else. Therefore, the following example should work:

[] The previous is not correct, the magic is the unnamed superpackage. I missed the rule (even after looking after being told) that any top level superpackage makes its exports available to any type in the system, regardless if it is a simple or complex name. That is exported types of top level superpackages are global. The use of the unnamed superpackage confused me because the rule is so different from normal superpackages. Silly me.


It seems Sun is slowly moving to conventions over configuration! Influence of the Ruby guys they hired? A name with no dots means general membership is clearly convention. However, it raises a number of issues.

• If the package must have a simple name, how do we handle uniqueness? Package names normally are scope with the reverse domain name, like org.osgi... However, org.osgi is not a simple name? [] This is thus not an issue, a superpackage can have a dotted name, the trick is that it must be a top level package, i.e. not being enclosed.
  
• It seems that top level packages are special. Then why is a superpackage not just defined in a single file that allows nested superpackages without naming them? This model would significantly simplify the model where the VM must find resources from all over the system that have obligatory relations. A lot of potential errors could be removed this way.
• []Despite my minsunderstanding, the previous point is still relevant. It is not clear why superpackage members are spread out over the file system while they are closely dependent on each other with bidirectional links.
  

I guess I must be missing something ...

Versioning
One would expect that in AD 2007 any modularity system in Java should have an evolution model. Alas, I have not been able to find any hint, not even a manifest header, of versions and how a superpackage should evolve. Obviously, this needs to be addressed. Superpackages are related to large systems and large systems do not pop in existence nor do they suddenly disappear. They live a long time and need to change over time.

Defining the Content of a Superpackage
The spec says that a superpackage has only member types and nested superpackages. However, the superpackage file contains a list of packages and lists the nested superpackages. The exports, however, list the exported type names and the exported superpackages.

These data structures are specified in the superpackage declaration and this is a file that the average developer will love to hate. This file must list all the packages of a superpackage; wildcarding or using the hierarchy is not allowed. Each package must be entered in full detail. Same for member superpackages as well as exported superpackages. Exported types can, however, use a short cut by specifying the exported types with the on-demand wildcard. That is, you can export com.foo.* indicating that you accept all types in the com.foo package (or all nested types in the com.foo type!). This sounds cool until you look at normal practice. A very common case is that implementation classes in a package have a name that ends in Impl. However, the wildcard in on-demand specifications is all or nothing. This likely means that all exported types must be enumerated by hand. Painful!

Deployment Versus Language
The key driver to remove JSR 294 from JSR 277 was that deployment artifacts should have no influence on the language. I am not so sure of that. One of the key insights I have obtained in the last few years is that there are many ways to slice and dice a JAR. With the myriad of profiles and configurations in Java, it is likely that you must deploy the same code in different ways to different platforms. For example, one of the really useful tricks of the bnd tool is the possibility to copy code from other bundles. Though many people can get upset about this, it provides me with a way to minimize dependencies without introducing redundancy because there is still one source code base.

The current solution of superpackages is highly rigid and static with its double linked structure. A class can only be a member of one superpackage. I really prefer the more flexible solution of a description that defines how a set of classes are modularized. It is a pity that the JCP does not work from a requirements phase so that these differences could be discussed without a worked out proposal on the table.

Restrictions and Security
In section 7.4.5 an example with an Outer and Outer.Inner superpackage is given that elucidates why the nested package must name their enclosing package. However, without a security manager anybody can easily access any packages to their liking. Access restrictions are conveniences, not security.

It would have been a better solution to add a SuperpackagePermission that specifies which packages can be named members or not. This would be similar to the OSGi PackagePermission. This would be a safe way to control access, the current model pays a very high price (only a single parent, double pointers) but does not provide security, just a slight barrier.

OSGi and Superpackages
Assuming that superpackages can come from different class loaders (see footnote), it is likely that the OSGi specifications need an additional header that reflects the super package dependencies. This dependency is a nice intermediate between Import-Package and Require-Bundle, albeit at the cost of a lot of added complexity and maintenance.

Interestingly, in the work for the next release we are being pushed from multiple sides to provide an entity that stands for an application. In J2EE and MIDP the context of an application is clear because applications are strictly contained in a silo. In OSGi platforms the situation is more fluid because applications collaborate. Superpackages could be a handle in this direction but in the current incarnation it will likely not work.

[]It looks like that a top level superpackage and its children must come from a single class loader. This is bad, because it means that in JSR 277 and OSGi it is impossible to deploy a member superpackages in modules. A common use case is that an enterprise has an application consisting of multiple bundles. Superpackages could have been used to minimize the exposure of internal API to other residents. However, bundles in OSGi and modules in JSR 277 require their own class loader, implying that a top level superpackage must be deployed with all its enclosed superpackages and code in one module/bundle. I guess OSGi Fragments can be abused to still allow partial delivery and update of an application, but this is not very elegant.

Conclusion
I wish I could be positive about JSR 294 but it I can't. The lack of requirements document makes it hard to judge the JSR on its own merits so this conclusion is my personal opinion.

I think that the current solution is unnecessary complex, there is too much redundancy and there is too much to specify; information that is usually quite volatile during development. The current model unnecessarily allows too many potential errors. Also versioning must be addressed. And if I understand the model with simple names being available to all superpackages then a solution must be envisioned to allow superpackage names to be unique.

However, the key aspect I differ with is if we need a language construct for modularity. Maybe I am blinded by almost ten years of OSGi modularity but JAR based modularity seems to provide more than superpackages provide at great additional expense. So if superpackages must be added to the language, please simplify it and provide a more convenient method to specify its contents. Better, consider how much JAR based modularity could add to the language.

Peter Kriens

* = There is a slight concern with section 5.3.5 of the Classfile and VM changes of the 294classfilevm.html file. In the last paragraph it states that a superpackage and its member types must be loaded from the same class loader. I interpret this as its direct members, however, one could interpret this as an member type of the enclosed super packages. If this unlikely interpretation is true, all superpackages would have to come from the same class loader, which seems silly. However, 7.1.1 defines type membership transitively giving credence to the silly interpretation. Needs work.