New RFPs available for feedback

Over the past while we have started making RFPs available at OSGi for public comments before they are finalized and voted on. This has worked really well for the OSGi Cloud RFP and the OSGi/CDI integration RFP.

Four new RFPs have been made available as part of this process. The RFPs describe uses cases and requirements which will ultimately feed into RFC work that will be the basis of future OSGi specifications.

RFP 143 OSGi Connect

The people at PojoSR have done some great work in showing how you can use OSGi Services in cases where you may not want to take on OSGi modularity (yet). RFP 143 captures requirements for creating a specification around this. The RFP can be found here: RFP 143.

RFP 150 HTTP Service Updates

The OSGi HTTP Service is widely used and it provides a very nice programming model for sevlets in an OSGi environment. However the specification is in need for an update to modernize it in relation to the Servlet spec itself. Additionally, the whiteboard pattern is introduced for servlet. You can find the RFP here: RFP 150.

RFP 151 Declarative Service Updates

A variety of updates are proposed to the Declarative Services specification. They can be found in RFP 151.

RFP 152 EJB Integration

EJB integration with OSGi has been done at Apache Aries and Glassfish. Other appservers such as JBoss and others also provide some form of interaction between EJBs and OSGi. This RFP aims a writing a specification for this integration to make OSGi-enabled EJBs portable. You can find the RFP here: RFP 152.

If you have thoughts or opinions about the above topic, have a read of the RFPs and put your comments on the associated bugs.

Core Release 5 and Enterprise Release 5 specifications

The OSGi Alliance has just published the recently approved Core Release 5 and Enterprise Release 5 specifications and made them available for download.

Some highlights from the new specifications include:

OSGi Core Release 5

• New Resource API for modeling generic capabilities and requirements.
• New Version Range class.

OSGi Enterprise Release 5

• New Repository Service Specification provides declarative access to artifact repositories based on the generic capabilities and requirements model. Where traditional repositories have typically provided artifacts based on their name, version and group, the OSGi Repository can provide artifacts based on capabilities, such as packages exported, services provided, extender functionality provided or custom-defined capabilities.
• New Resolver Service Specification. Based on the generic capabilities and requirements model, a management agent can use the Resolver service to compute the set of necessary resources needed to satisfy the given set of requirements. The Resolver is designed to work with the Repository Service, if available.
• New Subsystems Service Specification provides the ability to group multiple bundles into a single manageable entity, allows for complete isolation as well as various sharing models of code, services, and resources through a management agent. The Subsystem Service Specification defines an archive format to package multiple bundles, the Enterprise Subsystem Archive (.esa).
• New Service Loader Mediator Specification addresses common problems of bundles that rely on the java.util.ServiceLoader API to load custom Service Provider Implementations. It describes how to use the service registry for lookup of Service Providers as well as a solution for existing code to continue functioning using Service Loader API in a OSGi environment.
• New Common Namespaces Specification for use with the generic OSGi capabilities and requirements model.
• The Extender Namespace allows a bundle that requires an extender, such as Declarative Services or Blueprint, to express this dependency.
• The Contract Namespace provides a shorthand for many Import-­‐Package statements for technologies which span multiple packages.
• The Service Namespace allows a bundle to express that it provides or consumes a certain service.
• Updated JMX Management Model Specification.
• Object names now contain framework name and UUID, which allow multiple frameworks to be represented side-­by-­side.
• Updated the JMX API to reflect the latest Core API, specifically the bundle wiring API.
• Many improvements as requested by users, often focused on limiting the amount of data communicated via JMX APIs.
• Updated Configuration Admin Specification.
• Added targeted PIDs, which can be useful when configuring multiple versions of the same bundle through Configuration Admin.
• Added persistent change count to make it easier to detect changes.
• Added Synchronous Configuration Listener.

The specification PDFs, companion code jars and javadoc are now all available for download from the OSGi website.

OSGi/CDI integration RFP available for public comments

One of the new topics being discussed in the OSGi Enterprise Expert Group is integration with CDI (JSR 299). In short - the idea is that CDI beans in OSGi can receive their injections from the OSGi Service Registry. Additionally, CDI beans themselves can be registered in the OSGi Service Registry so that other bundles running in the OSGi framework can use them as OSGi Services.

Obviously, other component models already exist as OSGi specifications: we have Declarative Services and Blueprint. The CDI/OSGi integration will bring the CDI programming model into OSGi, and especially developers from the JavaEE world could be interested in continuing to use CDI inside OSGi. In my opinion it's not one size fits all. Developers have their own preferences and having a choice of programming models is good.
Also, the nice thing about the OSGi Service Registry-based approach is that all these component models work together, so I can write a Blueprint component and consume that as an OSGi Service in my CDI bean. And any other combination is also possible.

The OSGi/CDI discussion takes place in RFP 146. The RFP is a requirements document, once we agree on the requirements the details of the technical design will be discussed in an RFC.
As we did in the past with the Cloud Computing RFP 133, we're inviting everyone to comment or refine the requirements.
You can find the OSGi/CDI RFP here: OSGi RFP 146.

Repository Complex Requirements
Besides the OSGi/CDI RFP, another RFP has also been published for comments. This is RFP 148 Complex Repository Requirements. This is a much more narrow-focused RFP that aims at extending the capabilities of the Repository Service which is part of the upcoming R5 Enterprise Release (already available as EA Draft). Currently the Repository query API only accepts requirements within a single namespace. RFP 148 aims at extending this to allow requirements that cross namespaces. An example of this would be: I need a bundle that provides the package javax.servlet and has the Apache License.
Why wasn't this part of the Repository spec in the first place? Very simple, we ran out of time. So rather than rushing this feature in or delaying the Repository spec, we decided to work on it after the initial Repository Release. That's why we're looking at this RFP now, you can find it here: OSGi RFP 148.

Compendium 4.3 and Residential 4.3 published!

After some delays, I am happy to make the Compendium Version 4.3 and Residential Version 4.3 specification documents available for download.

Compendium Version 4.3

The Compendium Version 4.3 document adds all the specifications that were introduced in the Enterprise Version 4.2 document including

• Remote Services Admin
• JTA
• JDBC
• JNDI
• JPA
• Web Applications
• SCA Configuration

The Compendium Version 4.3 document also introduced the new Coordinator specification.

The Coordinator service provides a mechanism for multiple parties to collaborate on a common task without a priori knowledge of who will collaborate in that task. A collaborator can participate by adding a Participant to the Coordination. The Coordination will notify the Participants when the coordination is ended or when it is failed.

And there were also enhancements to some of the existing specifications. Configuration Admin is updated to allow multiple bundles to access the same configuration; and extend the security model to allow configuration "regions".

Declarative Services is updated to allow service references to receive service updates; to allow greedy service bindings; and support the use of compile time annotations to simplify authoring component descriptions.

Event Admin is also updated to support out-of-order asynchronous delivery of events.

Residential Version 4.3

We are also pleased to make available the first edition of the Residential specification.

The services of this Residential Specification have been designed with the residential market in mind. Requirements and management protocols for this environment are defined in the specifications by consortias like the Home Gateway Initiative (HGI), the Broadband Forum (BBF) and the UPnP Forum. These specifications provide requirements for execution environments in a Consumer Premises Equipment (CPE) and other consumer devices, as well as protocols for the management of residential environments.

The DMT Admin service has been updated to version 2.0 with a set of major improvements including overlapping subtrees, mount points, and scaffold nodes. These changes provide the basis for use with the TR-069 protocol.

A new Residential Device Management specification defines a Residential Management Tree, the RMT. This tree provides a general DMT Admin object model that allows browsing and managing the OSGi platform remotely over different protocol adapters.

The TR-157 Amendment 3 Software Module Guidelines chapter provides guidelines for implementers of the TR-157a3 Internet Gateway Device Software Modules specification on an OSGi platform.

The DMT Admin service and the TR-069 protocol have different semantics and primitives. The new TR069 Connector Service specification provides an API based on the TR-069 Remote Procedure Calls concept that is implemented on top of DMT Admin. This connector supports data conversion and the object modeling constructs defined in the DMT Admin service.

The specification pdfs, companion code jars and javadoc are now all available for download from the OSGi website.

Follow-up on the 2nd Cloud Workshop

The second OSGi Cloud Workshop was held during EclipseCon/OSGi DevCon 2012 last March. It was a very interesting morning with some good presentations and some great discussion. You can still find the presentations linked from here: http://www.osgi.org/Design/Cloud.

We learned that people are already widely using OSGi in Cloud environments, and part of the morning was spent discussing what OSGi could do to make it even more suitable for use in the Cloud. As a result of that a number of topics were proposed for people active in the OSGi Alliance to look at. You can find a summary of these topics here: https://mail.osgi.org/pipermail/cloud-workshop/2012-March/000112.html.

Last week the OSGi Enterprise Expert Group and the Residential Expert Group met to discuss these topics and to find potential routes to address them. Below you can find the results of these discussions. In this list I'll start each topic with the requirement as posted earlier to the cloud-workshop mailing list. The follow-ups below describe the thinking that we came to during the recent EEG/REG meeting.


1. Topic: Make it possible to describe various Service unavailable States. A service may be unavailable in the cloud because of a variety of reasons:

• Maybe the amount of invocations available to you is exhausted for today.
• Maybe your credit card expired 
• Maybe the node running the service crashed. 
• etc...

It should be possible to model these various failure states and it should also be possible to register 'callback' mechanisms that can deal with these states in whatever way is appropriate (blacklist the service, wait a while, send an email to the credit card holder, etc).

1. Follow-up: A potential new RFP is under discussion around monitoring and management. This RFP is currently being discussed in the Residential Expert Group, but it should ultimately be useful to all contexts in which OSGi is run. The requirements in this RFP could address some of the service quality issues referred to in this topic.

Additionally, there was a discussion whether it would make sense to extend the OSGi ServiceException so that various types of service failures could be reported (i.e. payment needed, quota exceeded, etc).

2. Topic: WYTIWYR (What you test is what you run) It should be possible to quickly deploy and redeploy.

2. Follow-up: One of the requirements that this expresses is the need to remotely run a test suite in an existing (remote) framework. There are test OSGi test frameworks that support this kind of behavior today (Pax Exam, Arquillian and others), but possibly they need to be enhanced with a remote deployment/managament solution that is cloud-friendly, for example the REST-based OSGi Framework management as is being discussed in RFC 182.

2b. Topic: There was an additional use-case around reverting the data (and configuration) changes made during an upgrade. If we need to downgrade after an upgrade then we may need to convert the data/configuration back into its old state.

2b. Follow-up: It might be possible to achieve this by providing an OSGi API to snapshot the framework state. This API could allow the user to save the current state and to retrieve a past saved state. When reverting to a past deployment this operation could be combined with a pluggable compensation process that converts the data back, if applicable.
The idea of snapshotting the framework state will be explored in a new RFP that is to be created soon. The data compensation process itself is most likely out of scope for OSGi.

3. Topic: Come up with a common and agreed architecture for Discovery. This should include consideration of Remote Services, Remote Events and Distributed Configuration Admin.

3. Follow-up: This is the topic of the new RFC 183 Cloud Discovery.

4. Topic: Resource utilization. It should be possible to measure/report this for each cloud node. Number of threads available, amount of memory, power consumption etc… Possibly create OSGi Capability namespaces for this. 

4. Follow-up: This relates to the monitoring RFP mentioned above.

5. Topic: OBR scaling. Need to be able to use OBR in a highly available manner. Should support failover and should hook in with discovery. 

5. Follow-up: The Repository service as defined in OSGi Enterprise R5 spec chapter 132 (see http://www.osgi.org/News/20120326 for download instructions of the latest draft) provides a stateless API which can work with well-known HA solutions (replication, failover, etc). Additionally, the Repository supports the concept of referrals, allowing multiple, federated repositories to be combined into a single logical view.

The discovery piece is part of RFC 183.

6. Topic: We need subsystems across frameworks. Possibly refer to them as 'Ecosystems'. These describe a number of subsystems deployed across a number of frameworks. 

6. Follow-up: While the general usefulness of this isn't disputed, there is nobody at this point in time driving this. If people strongly feel it should be addressed they should come forward and help out with defining the solution to addressing the issue.

7. Topic: Asynchronous services and asynchronous remote services. 

7. Follow-up: This is the topic of RFP 132 which was recently restarted. RFP 132 is purely about asynchronous OSGi services. Once this is established, asynchronous remote services can be modeled as a layer on top.

8. Topic: Isolation and security for applications 

• For multi-tenancy 
• Protect access to file system 
• Lifecycle handling of applications 
• OBR - isolated OBR (multiple tenants should not see each other's OBR) 

This all needs to be configurable.

8. Follow-up: Clearly separate VMs provide the best isolation, while separate JavaVMs within a single OS-level VM also provide fairly strong isolation (however, be aware of possible side effects of native code and possible resource exhaustion). Nested OSGi Frameworks and Subsystem Regions also provide isolation to a certain degree (see Graham's post on Subsystems), but the level of protection that is required clearly depends on the required security for the given application. The deployer can choose from these options as a target for deploying bundles and/or subsystems.

9. Topic: It should be possible to look at the cloud system state: 

• where am I (type of cloud, geographical location)? 
• what nodes are there and what is their state? 
• what frameworks are available in this cloud system? 
• where's my OBR? 
• what state am I in? 
• what do I need here in order to operate? 
• etc… 

9. Follow-up: This is part of what is being discussed in RFC 183 Cloud Discovery.

10. Topic: There should be a management mechanism for use in the cloud 

• JMX? Possibly not 
• REST? Most likely 

Management of application state should also be possible in addition to bundle/framework state 

10. Follow-up: A cloud-friendly REST-based management API for the framework is currently being worked on in RFC 182. Once that is established it can also form the baseline for Subsystems management technology which can be used for application-level management.

11. Topic: Deployment - when deploying replicated nodes it should be possible to specify that the replica should not be deployed on certain nodes, to avoid that all the replicas are deployed on the same node.

11. Follow-up: This also relates to discovery as discussed in RFC 183. A management agent can use this information to enforce such a constraint.

12. Topic: Single Sign-on for OSGi.

12. Follow-up: One member company has done a project in relation to this on top of the User Admin Service. A new RFP will be created to discuss this requirement further.

So there you are - the ideas from the cloud workshop were greatly appreciated and provide very useful input into future work. If you're interested in following the progress, as usual we're planning to release regular early access drafts of the documents that are relatively mature. Or, if you're interested in taking part in the creation of these specs, join in! For more information see: http://www.osgi.org/About/Join or contact me (david at redhat.com) or anyone else active in OSGi for more information.

Standard Applications on the Horizon

Those following the OSGi draft specifications will have seen that the OSGi Alliance has been working in the area of "bundle collections" for quite some time. By bundle collections, I mean the ability to define and deploy a set of bundles as a single entity. Many OSGi runtimes already provide such capabilities, and some example include:

• Apache Aries Applications
• Apache Geronimo Applications
• Apache Karaf Features
• Eclipse Virgo Plans and PARs
• Eclipse Platform Features
• IBM WebSphere Application Server Applications, Composites and Features
• Oracle GlassFish Applications
• Paremus Service Fabric Systems

When you see a list that long, and I've no doubt there are many more, it's clear that this is an area crying out for standardization. Digging a little deeper on these bundle collections, we see a lot of commonality between them; they define collections of bundles (obviously), they give the collection an identity, they allow version ranges when identifying the content bundles, and they also often allow locking down of versions (e.g. for a specific tested deployment). There's also some variability; many, but not all, provide isolation to prevent undesirable interactions between collections deployed to the same runtime. This is similar to the isolation you get with Java EE applications on an applications server, but rather than replicating the Java EE model, they also allow bundles to be shared by applications to remove duplication and reduce disk and memory footprint (allows common libraries and frameworks to be packaged and deployed just once). Other types of collections do not impose isolation and are often used as a way to simplify the assembly of runtimes.

So, where are we in the standards? Well, there are two parts to this:

The first part is an enabler in the OSGi Core Framework. To enable isolation to be applied to collections of bundles, the OSGi Core defined a set of framework hooks. Hooks hide things from bundles such that they cannot resolve to, or find, them. Core 4.2 had already added the Service hooks and Core 4.3 complemented these with Resolver and Bundle hooks. Hooks are a low-level core features. It was never the intention that these would be used by most OSGi developers, and so something else was needed.

The second part is the specification that uses the core enabler, but makes it simple to develop, test and deploy these bundle collections. That's where the new Subsystems specification comes in. Subsystems defines a programming model that uses a format familiar to bundle developers for developing bundle collections. It simplifies the task further still by defining different types of bundle collection which have default isolation policies matching the most common isolation models. The subsystem types defined in the specification are:

• Application - allows the collection to use packages and services from shared bundles, but does not share its own with others. This type fits closely with what people would typically think of as an application running in an application server
• Composite - does not use or share any packages or services unless explicitly configured to do so. This enables the collection of bundles to keep some aspects of their internals private, only exposing what they want the rest of the runtime to see or provide. A common use case for composites is where you have teams providing parts for solutions and those parts are composed of multiple bundles and have public APIs and then implementation details they do not wish to be externalized.
• Feature - freely uses and shares any packages and services it provides or are visible to it. This simplifies the management of bundle collections but does not isolate them. A common use case for these is to make the definition of runtimes or solutions simpler.

The following is an example of an application subsystem containing three bundles:

Subsystem-ManifestVersion: 1

Subsystem-Name: Acme Foo Application

Subsystem-SymbolicName: com.acme.foo.application

Subsystem-Version: 1.2

Subsystem-Type: osgi.subsystem.application

Subsystem-Content: com.acme.foo.web;version="[1.0.0,2.0.0)",

com.acme.foo.biz;version="[1.0.0,2.0.0)",

com.acme.foo.persistence;version="[1.0.0,2.0.0)"

If you've developed bundles in the past, then many of the headers will look familiar. This particular application has a name (Subsystem-SymbolicName) and a version (Subsystem-Version), including human readable information (Subsystem-Name). It has a type (Subsystem-Type) that identifies it as an application which means its packages and services are not exposed outside the subsystem. Finally, it lists its content (Subsystem-Content). The type of the content defaults to bundle, but other types are permitted (such as fragments, or other subsystems, e.g. to nest a feature within an application), and a subsystem implementation can also choose to enable its own custom types (not required to be supported by different subsystem implementations).

One other important area of management that subsystems addresses is that of provisioning. You'll see from the example that it lists content but not the specific versions, nor any bundles it might require in support of that content. If you install that definition and provide no further details, it will resolve the subsystem to choose the exact bundle versions and any necessary dependency bundles. A second option is to provide a pre-defined deployment that specifies the exact content and supporting bundles. That way, you can test a specific deployment before putting it into production.

The Subsystem specification was made available in the OSGi Enterprise Release 5 Proposed Final Draft in March. As with all OSGi specifications, they're written for implementors, but as projects and vendors begin to provide implementations I expect to see user-level documentation emerge and then users can enjoy the flexibility of creating modular standard applications, composites and features for a number of target environments.

Graham Charters

Goodbye

This is my last blog on the OSGi Web site ... After 237 blog posts it is time to say goodbye. I'd like to thank the OSGi Alliance for giving me a sounding board for these years. I'd like to thank everybody in the Alliance in the past years for their support and giving me a chance to work with them. I am excited to start my new work but there is sadness that my time as OSGi Technical Director has ended. I've been incredibly privileged to work on this technology, it was a fantastic adventure.

I wish the OSGi Alliance the best. I think the specifications are mostly rock solid and I think the increasing adoption reflect this. The last OSGi DevCon/EclipseCon was a big success and it was great to see how the OSGi service model is making inroads. Though there is never an opportune time to leave, I do feel that the specifications are mature now and the most crucial parts are in place. So to that extent my work is done.

I will continue blogging on http://softwaresimplexity.blogspot.com, I do hope you will follow me there. I will also tweet as @pkriens. Also do not forget to connect on LinkedIn and/or provide a recommendation.

Thanks for having had the patience to read this blog,

  Peter Kriens