Compendium 4.3 and Residential 4.3 published!

After some delays, I am happy to make the Compendium Version 4.3 and Residential Version 4.3 specification documents available for download.

Compendium Version 4.3

The Compendium Version 4.3 document adds all the specifications that were introduced in the Enterprise Version 4.2 document including

• Remote Services Admin
• JTA
• JDBC
• JNDI
• JPA
• Web Applications
• SCA Configuration

The Compendium Version 4.3 document also introduced the new Coordinator specification.

The Coordinator service provides a mechanism for multiple parties to collaborate on a common task without a priori knowledge of who will collaborate in that task. A collaborator can participate by adding a Participant to the Coordination. The Coordination will notify the Participants when the coordination is ended or when it is failed.

And there were also enhancements to some of the existing specifications. Configuration Admin is updated to allow multiple bundles to access the same configuration; and extend the security model to allow configuration "regions".

Declarative Services is updated to allow service references to receive service updates; to allow greedy service bindings; and support the use of compile time annotations to simplify authoring component descriptions.

Event Admin is also updated to support out-of-order asynchronous delivery of events.

Residential Version 4.3

We are also pleased to make available the first edition of the Residential specification.

The services of this Residential Specification have been designed with the residential market in mind. Requirements and management protocols for this environment are defined in the specifications by consortias like the Home Gateway Initiative (HGI), the Broadband Forum (BBF) and the UPnP Forum. These specifications provide requirements for execution environments in a Consumer Premises Equipment (CPE) and other consumer devices, as well as protocols for the management of residential environments.

The DMT Admin service has been updated to version 2.0 with a set of major improvements including overlapping subtrees, mount points, and scaffold nodes. These changes provide the basis for use with the TR-069 protocol.

A new Residential Device Management specification defines a Residential Management Tree, the RMT. This tree provides a general DMT Admin object model that allows browsing and managing the OSGi platform remotely over different protocol adapters.

The TR-157 Amendment 3 Software Module Guidelines chapter provides guidelines for implementers of the TR-157a3 Internet Gateway Device Software Modules specification on an OSGi platform.

The DMT Admin service and the TR-069 protocol have different semantics and primitives. The new TR069 Connector Service specification provides an API based on the TR-069 Remote Procedure Calls concept that is implemented on top of DMT Admin. This connector supports data conversion and the object modeling constructs defined in the DMT Admin service.

The specification pdfs, companion code jars and javadoc are now all available for download from the OSGi website.

Follow-up on the 2nd Cloud Workshop

The second OSGi Cloud Workshop was held during EclipseCon/OSGi DevCon 2012 last March. It was a very interesting morning with some good presentations and some great discussion. You can still find the presentations linked from here: http://www.osgi.org/Design/Cloud.

We learned that people are already widely using OSGi in Cloud environments, and part of the morning was spent discussing what OSGi could do to make it even more suitable for use in the Cloud. As a result of that a number of topics were proposed for people active in the OSGi Alliance to look at. You can find a summary of these topics here: https://mail.osgi.org/pipermail/cloud-workshop/2012-March/000112.html.

Last week the OSGi Enterprise Expert Group and the Residential Expert Group met to discuss these topics and to find potential routes to address them. Below you can find the results of these discussions. In this list I'll start each topic with the requirement as posted earlier to the cloud-workshop mailing list. The follow-ups below describe the thinking that we came to during the recent EEG/REG meeting.


1. Topic: Make it possible to describe various Service unavailable States. A service may be unavailable in the cloud because of a variety of reasons:

• Maybe the amount of invocations available to you is exhausted for today.
• Maybe your credit card expired 
• Maybe the node running the service crashed. 
• etc...

It should be possible to model these various failure states and it should also be possible to register 'callback' mechanisms that can deal with these states in whatever way is appropriate (blacklist the service, wait a while, send an email to the credit card holder, etc).

1. Follow-up: A potential new RFP is under discussion around monitoring and management. This RFP is currently being discussed in the Residential Expert Group, but it should ultimately be useful to all contexts in which OSGi is run. The requirements in this RFP could address some of the service quality issues referred to in this topic.

Additionally, there was a discussion whether it would make sense to extend the OSGi ServiceException so that various types of service failures could be reported (i.e. payment needed, quota exceeded, etc).

2. Topic: WYTIWYR (What you test is what you run) It should be possible to quickly deploy and redeploy.

2. Follow-up: One of the requirements that this expresses is the need to remotely run a test suite in an existing (remote) framework. There are test OSGi test frameworks that support this kind of behavior today (Pax Exam, Arquillian and others), but possibly they need to be enhanced with a remote deployment/managament solution that is cloud-friendly, for example the REST-based OSGi Framework management as is being discussed in RFC 182.

2b. Topic: There was an additional use-case around reverting the data (and configuration) changes made during an upgrade. If we need to downgrade after an upgrade then we may need to convert the data/configuration back into its old state.

2b. Follow-up: It might be possible to achieve this by providing an OSGi API to snapshot the framework state. This API could allow the user to save the current state and to retrieve a past saved state. When reverting to a past deployment this operation could be combined with a pluggable compensation process that converts the data back, if applicable.
The idea of snapshotting the framework state will be explored in a new RFP that is to be created soon. The data compensation process itself is most likely out of scope for OSGi.

3. Topic: Come up with a common and agreed architecture for Discovery. This should include consideration of Remote Services, Remote Events and Distributed Configuration Admin.

3. Follow-up: This is the topic of the new RFC 183 Cloud Discovery.

4. Topic: Resource utilization. It should be possible to measure/report this for each cloud node. Number of threads available, amount of memory, power consumption etc… Possibly create OSGi Capability namespaces for this. 

4. Follow-up: This relates to the monitoring RFP mentioned above.

5. Topic: OBR scaling. Need to be able to use OBR in a highly available manner. Should support failover and should hook in with discovery. 

5. Follow-up: The Repository service as defined in OSGi Enterprise R5 spec chapter 132 (see http://www.osgi.org/News/20120326 for download instructions of the latest draft) provides a stateless API which can work with well-known HA solutions (replication, failover, etc). Additionally, the Repository supports the concept of referrals, allowing multiple, federated repositories to be combined into a single logical view.

The discovery piece is part of RFC 183.

6. Topic: We need subsystems across frameworks. Possibly refer to them as 'Ecosystems'. These describe a number of subsystems deployed across a number of frameworks. 

6. Follow-up: While the general usefulness of this isn't disputed, there is nobody at this point in time driving this. If people strongly feel it should be addressed they should come forward and help out with defining the solution to addressing the issue.

7. Topic: Asynchronous services and asynchronous remote services. 

7. Follow-up: This is the topic of RFP 132 which was recently restarted. RFP 132 is purely about asynchronous OSGi services. Once this is established, asynchronous remote services can be modeled as a layer on top.

8. Topic: Isolation and security for applications 

• For multi-tenancy 
• Protect access to file system 
• Lifecycle handling of applications 
• OBR - isolated OBR (multiple tenants should not see each other's OBR) 

This all needs to be configurable.

8. Follow-up: Clearly separate VMs provide the best isolation, while separate JavaVMs within a single OS-level VM also provide fairly strong isolation (however, be aware of possible side effects of native code and possible resource exhaustion). Nested OSGi Frameworks and Subsystem Regions also provide isolation to a certain degree (see Graham's post on Subsystems), but the level of protection that is required clearly depends on the required security for the given application. The deployer can choose from these options as a target for deploying bundles and/or subsystems.

9. Topic: It should be possible to look at the cloud system state: 

• where am I (type of cloud, geographical location)? 
• what nodes are there and what is their state? 
• what frameworks are available in this cloud system? 
• where's my OBR? 
• what state am I in? 
• what do I need here in order to operate? 
• etc… 

9. Follow-up: This is part of what is being discussed in RFC 183 Cloud Discovery.

10. Topic: There should be a management mechanism for use in the cloud 

• JMX? Possibly not 
• REST? Most likely 

Management of application state should also be possible in addition to bundle/framework state 

10. Follow-up: A cloud-friendly REST-based management API for the framework is currently being worked on in RFC 182. Once that is established it can also form the baseline for Subsystems management technology which can be used for application-level management.

11. Topic: Deployment - when deploying replicated nodes it should be possible to specify that the replica should not be deployed on certain nodes, to avoid that all the replicas are deployed on the same node.

11. Follow-up: This also relates to discovery as discussed in RFC 183. A management agent can use this information to enforce such a constraint.

12. Topic: Single Sign-on for OSGi.

12. Follow-up: One member company has done a project in relation to this on top of the User Admin Service. A new RFP will be created to discuss this requirement further.

So there you are - the ideas from the cloud workshop were greatly appreciated and provide very useful input into future work. If you're interested in following the progress, as usual we're planning to release regular early access drafts of the documents that are relatively mature. Or, if you're interested in taking part in the creation of these specs, join in! For more information see: http://www.osgi.org/About/Join or contact me (david at redhat.com) or anyone else active in OSGi for more information.

Standard Applications on the Horizon

Those following the OSGi draft specifications will have seen that the OSGi Alliance has been working in the area of "bundle collections" for quite some time. By bundle collections, I mean the ability to define and deploy a set of bundles as a single entity. Many OSGi runtimes already provide such capabilities, and some example include:

• Apache Aries Applications
• Apache Geronimo Applications
• Apache Karaf Features
• Eclipse Virgo Plans and PARs
• Eclipse Platform Features
• IBM WebSphere Application Server Applications, Composites and Features
• Oracle GlassFish Applications
• Paremus Service Fabric Systems

When you see a list that long, and I've no doubt there are many more, it's clear that this is an area crying out for standardization. Digging a little deeper on these bundle collections, we see a lot of commonality between them; they define collections of bundles (obviously), they give the collection an identity, they allow version ranges when identifying the content bundles, and they also often allow locking down of versions (e.g. for a specific tested deployment). There's also some variability; many, but not all, provide isolation to prevent undesirable interactions between collections deployed to the same runtime. This is similar to the isolation you get with Java EE applications on an applications server, but rather than replicating the Java EE model, they also allow bundles to be shared by applications to remove duplication and reduce disk and memory footprint (allows common libraries and frameworks to be packaged and deployed just once). Other types of collections do not impose isolation and are often used as a way to simplify the assembly of runtimes.

So, where are we in the standards? Well, there are two parts to this:

The first part is an enabler in the OSGi Core Framework. To enable isolation to be applied to collections of bundles, the OSGi Core defined a set of framework hooks. Hooks hide things from bundles such that they cannot resolve to, or find, them. Core 4.2 had already added the Service hooks and Core 4.3 complemented these with Resolver and Bundle hooks. Hooks are a low-level core features. It was never the intention that these would be used by most OSGi developers, and so something else was needed.

The second part is the specification that uses the core enabler, but makes it simple to develop, test and deploy these bundle collections. That's where the new Subsystems specification comes in. Subsystems defines a programming model that uses a format familiar to bundle developers for developing bundle collections. It simplifies the task further still by defining different types of bundle collection which have default isolation policies matching the most common isolation models. The subsystem types defined in the specification are:

• Application - allows the collection to use packages and services from shared bundles, but does not share its own with others. This type fits closely with what people would typically think of as an application running in an application server
• Composite - does not use or share any packages or services unless explicitly configured to do so. This enables the collection of bundles to keep some aspects of their internals private, only exposing what they want the rest of the runtime to see or provide. A common use case for composites is where you have teams providing parts for solutions and those parts are composed of multiple bundles and have public APIs and then implementation details they do not wish to be externalized.
• Feature - freely uses and shares any packages and services it provides or are visible to it. This simplifies the management of bundle collections but does not isolate them. A common use case for these is to make the definition of runtimes or solutions simpler.

The following is an example of an application subsystem containing three bundles:

Subsystem-ManifestVersion: 1

Subsystem-Name: Acme Foo Application

Subsystem-SymbolicName: com.acme.foo.application

Subsystem-Version: 1.2

Subsystem-Type: osgi.subsystem.application

Subsystem-Content: com.acme.foo.web;version="[1.0.0,2.0.0)",

com.acme.foo.biz;version="[1.0.0,2.0.0)",

com.acme.foo.persistence;version="[1.0.0,2.0.0)"

If you've developed bundles in the past, then many of the headers will look familiar. This particular application has a name (Subsystem-SymbolicName) and a version (Subsystem-Version), including human readable information (Subsystem-Name). It has a type (Subsystem-Type) that identifies it as an application which means its packages and services are not exposed outside the subsystem. Finally, it lists its content (Subsystem-Content). The type of the content defaults to bundle, but other types are permitted (such as fragments, or other subsystems, e.g. to nest a feature within an application), and a subsystem implementation can also choose to enable its own custom types (not required to be supported by different subsystem implementations).

One other important area of management that subsystems addresses is that of provisioning. You'll see from the example that it lists content but not the specific versions, nor any bundles it might require in support of that content. If you install that definition and provide no further details, it will resolve the subsystem to choose the exact bundle versions and any necessary dependency bundles. A second option is to provide a pre-defined deployment that specifies the exact content and supporting bundles. That way, you can test a specific deployment before putting it into production.

The Subsystem specification was made available in the OSGi Enterprise Release 5 Proposed Final Draft in March. As with all OSGi specifications, they're written for implementors, but as projects and vendors begin to provide implementations I expect to see user-level documentation emerge and then users can enjoy the flexibility of creating modular standard applications, composites and features for a number of target environments.

Graham Charters

Goodbye

This is my last blog on the OSGi Web site ... After 237 blog posts it is time to say goodbye. I'd like to thank the OSGi Alliance for giving me a sounding board for these years. I'd like to thank everybody in the Alliance in the past years for their support and giving me a chance to work with them. I am excited to start my new work but there is sadness that my time as OSGi Technical Director has ended. I've been incredibly privileged to work on this technology, it was a fantastic adventure.

I wish the OSGi Alliance the best. I think the specifications are mostly rock solid and I think the increasing adoption reflect this. The last OSGi DevCon/EclipseCon was a big success and it was great to see how the OSGi service model is making inroads. Though there is never an opportune time to leave, I do feel that the specifications are mature now and the most crucial parts are in place. So to that extent my work is done.

I will continue blogging on http://softwaresimplexity.blogspot.com, I do hope you will follow me there. I will also tweet as @pkriens. Also do not forget to connect on LinkedIn and/or provide a recommendation.

Thanks for having had the patience to read this blog,

  Peter Kriens

What happened to pre-release versions?

In RFC 175, we proposed to introduce two new things for Core Release 5. One was a change to the version syntax to support pre-release versions and the other was to introduce a VersionRange class to provide a common implementation of version range operations. When the Core Release 5 spec was completed, it included the VersionRange class but did not include pre-release version support. So, what happened?

Pre-release, aka. snapshot, versions seemed like a good idea when first proposed. From the RFC:

During development of a bundle (or package), there is a period of time where the bundle has not been declared final. That is, the bundle has a planned version number once final, but that version number is not practically consumed until the bundle has been declared final. However, during development of the bundle, it must have a version number. This version number must be larger than the version number of the previous final version of the bundle but less than the version number intended for the bundle once final.

There are several usage patterns for version numbers which have emerged to deal with this problem. For example, some use an odd/even version number for the minor version to differentiate between development versions and final (release) versions. Some also place the build timestamp in the qualifier to distinguish all built versions of a bundle, but there is no clear marking which is the final version so dependents cannot mandate a final version.

So we proposed a change to the version syntax to open up space between version numbers so that before the unqualified version (e.g. 1.2.3) there would be pre-release versions. So 1.2.3-snapshot would be a lower version number than 1.2.3. It would have a total ordering over all versions and be backwards compatible with existing version syntax.

1.2.3- < 1.2.3-x < 1.2.3 = 1.2.3. < 1.2.3.x

However, we also had to work properly with existing version range syntax. For example, is the version 1.2.3-snapshot included in the range [1.2.3,2.0.0)? We defined two rules for this.

1. If a version range having an endpoint specified without a qualifier (e.g. [1.2.3,2.0.0)) would include a version with a release qualifier of the empty string (e.g. 1.2.3), then the version range must also include that version when identified as pre-release (e.g. 1.2.3-x).
2. If a version range having an endpoint specified without a qualifier (e.g. [1.2.3,2.0.0)) would exclude a version with a release qualifier of the empty string (e.g. 2.0.0), then the version range must also exclude that version when identified as pre-release (e.g. 2.0.0-x).

All together we had a complete design and I was able to implement the changes to the Version and VersionRange classes and write compliance tests. We even implemented it in Equinox. So from a runtime point of view, things looked OK.

But the big concern come around interactions with existing tooling, management and provisioning systems. These systems would not understand a bundle having a pre-release version string. They would require a lot of changes to properly handle support the pre-release version syntax.

Furthermore, we also become concerned about the mental complexity of pre-release versions. In numerous discussions within CPEG and with peers, people would get confused over the ordering of versions and whether some version was included in some range. If we, the "experts", couldn't keep it straight in our minds, we might expect others to also have a hard time.

So in the end, given the mental complexity and the downstream impact to tools, repositories and management systems, CPEG decided that the benefit of the changes was not sufficient to justify the cost of the change. So we agreed, after some lengthy discussions to discard the pre-release version proposal.

BJ Hargrave

Surprising Services

Services are arguably what OSGi is about but at the same time they are also the least understood. When we set out to design OSGi our key requirement was to create an ad-hoc collaborative programming model. Instead of components that are centrally managed and/or are closely coupled to their siblings, we looked for a model of independent peers. To understand what I mean:

In a peer-to-peer model you need to be able to find out who your peers are, and how you can collaborate with them. In the agent or actor model the peer is identified and messages are exchanged based on peer identity. Peer dependencies suffers from the transitive dependency aggregation, quite soon almost all agents are transitively dependent on each other and there is no more component reuse. This is the big ball of mud problem. To address this problem we came up with the service model.  OSGi services combine package based contracts with an instance broker, to provide a managed conduit between peers. 

That said, then what is actual the value of the services for application developers?

Well, the surprising effect of services is that the contracts can often be significantly simplified over traditional APIs because in most Java APIs the collaborative parts are mixed with instance coupling techniques (hacks?)  (DocumentBuilderFactory, InitialContextFactoryBuilder anyone?) and administrative aspects.

It turns out that with services you rarely need to define contracts for these aspects since they are taken care of by the service model (factories, but much more), or in most cases can stay inside the component. Many administrative aspects can be handled by the implementation. Service contracts can be limited to strictly the collaborative parts. And size does matter, the smaller the contract, the easier it is to use.

For example, assume you need to use a persistent queue that is used by a set of workers. Active MQ, Amazon SQS, etc. have a significant number of API calls about maintaining the queues, setting the properties, and interacting with it. However, virtually all of those aspects can be defined in Configuration Admin, the only collaborative aspects are how does the worker get its task and how do you queue a task?

The by far simplest solution I know is to define a contract where only the worker registers a Worker service and a MessageQueue service:

  @Component(properties="queue=myqueue")
  public class MyWorker implements Worker<MyTask> {
      MessageQueue queue;


      public void work(MyTask task) { 
         ...
         queue.post( AnotherTask.class, another );
      }


      @Reference(target="(queue=myqueue)")
      void setQueue( MessageQueue queue ) {
         this.queue = queue;
      }
  }

This queuing contract is sufficient to allow a wide variety of different implementations. Implementing this with the Amazon Simple Queue Service is quite easy. A puny little bundle can look for the services, uses the queue service property to listen to queues and dispatch the messages. In this case, the web based AWS console can be used to manage the queues, no code required. A more comprehensive implementation can use Configuration Admin to manage queues, or it can create queues on demand. Implementing this on another persistent queue can be done quite different without requiring any change in the components that act as senders and workers.

If there is one rule about simplifying software that works consistently then it is hiding. Something that you could not see can't bug you. OSGi services are by far the most effective way to hide implementation details; minimizing what must be shared. Our industry is predicted to double the amount of code in the next 8 years, we better get our services on or this avalanche will bury us.

 Peter Kriens

Coordinator

Last year we introduced the Coordinator in the Compendium 4.3. Unfortunately, this release 4.3 was held up over some legal issues. However, it will soon be available, in the 4.3 Compendium as well as the Enterprise 5.0.

The Coordinator service is a bit my baby. When we started with OSGi almost 14 years ago one of the first things I proposed was to start with a transaction manager. I'd just read in 3days Transaction Processing from Gray & Reuters and was thrilled, that had been the best non-fiction book I ever read. Obviously the ACID properties were interesting, and very informative to see how they could be implemented, but the most exciting part was the coordination aspect of transactions. Transactions, as described in the seminal book, allowed different parties (the resource managers) to collaborate on a task without prior knowledge of each other. Resource managers when called could discover an ongoing transaction and join it. The transaction guaranteed them a callback before the task was finished. This of course is a dream in a component model like OSGi where you call many different parties of which most you have no knowledge of. Each called service could participate in the ongoing task and be informed when the task was about to be finished. When I proposed a transaction manager the guys around the table looked at me warily and further ignored me, transactions in an embedded environment?

We got very busy but I kept nagging and the rest kept looking if I was silly in the head. In 2004 I finally wrote RFC 98, a modest proposal for a transaction manager light. Of course I immediately ran into the situation that, even though few if any had used it, that there was an already existing Java Transaction API (JTA). Prosyst did some work on this since they saw some value but in the end it was moved to a full blown JTA service. This was not what I wanted because JTA is weird (from my perspective then) because it distinguishes too much between the Container people and the application people. OSGi is about peer-to-peer, containers are about control from above. Try to act like a resource manager with XA (which would give the coordination aspects), however, it looks like it was made difficult on purpose.

The it hit me, I always ran into the opposition because I used a name that too many people associated with heavy and complexity. Though a full blown distributed high performance robust transaction manager is to say the least a non-trivial beast I was mostly interested in the coordination aspects inside an OSGi framework, a significantly simpler animal. So choose to change the name! The Coordinator service was born!

The first RFC was a very simple thread based Coordinator service. When you're called in your service you can get the current Coordination (or you can start one). Each thread had its own current Coordination. Anybody could then join the Coordination by giving it a participant. The Coordination can either fail or succeed, after which all the participants are called back and informed of the outcome. Anybody that has access to the Coordination can fail it, the Coordination will also fail with time out if it is not terminated before a deadline.

So how would you use it? The following code shows how a Coordination is started:

  Coordinator  coordinator = ...

  Coordination c = coordinator.create("work",0);
  try {

    doWork(c);
  } catch( Throwable t ) {

    c.fail(t);
  } finally {

    c.end(); 
  }

This template is very robust. The Coordination is created with a name and a timeout. The work is then done in a try/catch/finally block. The catch block will fail the Coordination. Calling end on a failed Coordination will throw an exception so the exception does not get lost. A worker would do the following to participate in the Coordination:

 Coordinator coordinator = ... 

 void foo() {

   doPrepare();
   if ( !coordinator.addParticipant(this))

     doFinish();
 }

A worker can use the Coordination service to add itself as a participant. It is then guaranteed to get a call back when the current Coordination is terminated.

An example use case is batching a number of updates. Normally you can significantly optimize if you can delay communications by batching a number of updates. However, how you do you know when you had the last update so you can initiate the batch? If there is no Coordinator, the updates are done immediately, with a Coordinator they can be batched until the Coordination is terminated.


During the specification process a number of features were added: direct Coordinations (not thread based), a variable store per Coordination, and a reflective API.

I guess it will take some time before the Coordinator is really taken advantage of since the model is quite foreign to most developers. However, I am convinced that the service is really what OSGi is all about: coordinated collaborative components.

Peter Kriens