What happened to pre-release versions?

In RFC 175, we proposed to introduce two new things for Core Release 5. One was a change to the version syntax to support pre-release versions and the other was to introduce a VersionRange class to provide a common implementation of version range operations. When the Core Release 5 spec was completed, it included the VersionRange class but did not include pre-release version support. So, what happened?

Pre-release, aka. snapshot, versions seemed like a good idea when first proposed. From the RFC:

During development of a bundle (or package), there is a period of time where the bundle has not been declared final. That is, the bundle has a planned version number once final, but that version number is not practically consumed until the bundle has been declared final. However, during development of the bundle, it must have a version number. This version number must be larger than the version number of the previous final version of the bundle but less than the version number intended for the bundle once final.

There are several usage patterns for version numbers which have emerged to deal with this problem. For example, some use an odd/even version number for the minor version to differentiate between development versions and final (release) versions. Some also place the build timestamp in the qualifier to distinguish all built versions of a bundle, but there is no clear marking which is the final version so dependents cannot mandate a final version.

So we proposed a change to the version syntax to open up space between version numbers so that before the unqualified version (e.g. 1.2.3) there would be pre-release versions. So 1.2.3-snapshot would be a lower version number than 1.2.3. It would have a total ordering over all versions and be backwards compatible with existing version syntax.

1.2.3- < 1.2.3-x < 1.2.3 = 1.2.3. < 1.2.3.x

However, we also had to work properly with existing version range syntax. For example, is the version 1.2.3-snapshot included in the range [1.2.3,2.0.0)? We defined two rules for this.

1. If a version range having an endpoint specified without a qualifier (e.g. [1.2.3,2.0.0)) would include a version with a release qualifier of the empty string (e.g. 1.2.3), then the version range must also include that version when identified as pre-release (e.g. 1.2.3-x).
2. If a version range having an endpoint specified without a qualifier (e.g. [1.2.3,2.0.0)) would exclude a version with a release qualifier of the empty string (e.g. 2.0.0), then the version range must also exclude that version when identified as pre-release (e.g. 2.0.0-x).

All together we had a complete design and I was able to implement the changes to the Version and VersionRange classes and write compliance tests. We even implemented it in Equinox. So from a runtime point of view, things looked OK.

But the big concern come around interactions with existing tooling, management and provisioning systems. These systems would not understand a bundle having a pre-release version string. They would require a lot of changes to properly handle support the pre-release version syntax.

Furthermore, we also become concerned about the mental complexity of pre-release versions. In numerous discussions within CPEG and with peers, people would get confused over the ordering of versions and whether some version was included in some range. If we, the "experts", couldn't keep it straight in our minds, we might expect others to also have a hard time.

So in the end, given the mental complexity and the downstream impact to tools, repositories and management systems, CPEG decided that the benefit of the changes was not sufficient to justify the cost of the change. So we agreed, after some lengthy discussions to discard the pre-release version proposal.

BJ Hargrave

Surprising Services

Services are arguably what OSGi is about but at the same time they are also the least understood. When we set out to design OSGi our key requirement was to create an ad-hoc collaborative programming model. Instead of components that are centrally managed and/or are closely coupled to their siblings, we looked for a model of independent peers. To understand what I mean:

In a peer-to-peer model you need to be able to find out who your peers are, and how you can collaborate with them. In the agent or actor model the peer is identified and messages are exchanged based on peer identity. Peer dependencies suffers from the transitive dependency aggregation, quite soon almost all agents are transitively dependent on each other and there is no more component reuse. This is the big ball of mud problem. To address this problem we came up with the service model.  OSGi services combine package based contracts with an instance broker, to provide a managed conduit between peers. 

That said, then what is actual the value of the services for application developers?

Well, the surprising effect of services is that the contracts can often be significantly simplified over traditional APIs because in most Java APIs the collaborative parts are mixed with instance coupling techniques (hacks?)  (DocumentBuilderFactory, InitialContextFactoryBuilder anyone?) and administrative aspects.

It turns out that with services you rarely need to define contracts for these aspects since they are taken care of by the service model (factories, but much more), or in most cases can stay inside the component. Many administrative aspects can be handled by the implementation. Service contracts can be limited to strictly the collaborative parts. And size does matter, the smaller the contract, the easier it is to use.

For example, assume you need to use a persistent queue that is used by a set of workers. Active MQ, Amazon SQS, etc. have a significant number of API calls about maintaining the queues, setting the properties, and interacting with it. However, virtually all of those aspects can be defined in Configuration Admin, the only collaborative aspects are how does the worker get its task and how do you queue a task?

The by far simplest solution I know is to define a contract where only the worker registers a Worker service and a MessageQueue service:

  @Component(properties="queue=myqueue")
  public class MyWorker implements Worker<MyTask> {
      MessageQueue queue;


      public void work(MyTask task) { 
         ...
         queue.post( AnotherTask.class, another );
      }


      @Reference(target="(queue=myqueue)")
      void setQueue( MessageQueue queue ) {
         this.queue = queue;
      }
  }

This queuing contract is sufficient to allow a wide variety of different implementations. Implementing this with the Amazon Simple Queue Service is quite easy. A puny little bundle can look for the services, uses the queue service property to listen to queues and dispatch the messages. In this case, the web based AWS console can be used to manage the queues, no code required. A more comprehensive implementation can use Configuration Admin to manage queues, or it can create queues on demand. Implementing this on another persistent queue can be done quite different without requiring any change in the components that act as senders and workers.

If there is one rule about simplifying software that works consistently then it is hiding. Something that you could not see can't bug you. OSGi services are by far the most effective way to hide implementation details; minimizing what must be shared. Our industry is predicted to double the amount of code in the next 8 years, we better get our services on or this avalanche will bury us.

 Peter Kriens

Coordinator

Last year we introduced the Coordinator in the Compendium 4.3. Unfortunately, this release 4.3 was held up over some legal issues. However, it will soon be available, in the 4.3 Compendium as well as the Enterprise 5.0.

The Coordinator service is a bit my baby. When we started with OSGi almost 14 years ago one of the first things I proposed was to start with a transaction manager. I'd just read in 3days Transaction Processing from Gray & Reuters and was thrilled, that had been the best non-fiction book I ever read. Obviously the ACID properties were interesting, and very informative to see how they could be implemented, but the most exciting part was the coordination aspect of transactions. Transactions, as described in the seminal book, allowed different parties (the resource managers) to collaborate on a task without prior knowledge of each other. Resource managers when called could discover an ongoing transaction and join it. The transaction guaranteed them a callback before the task was finished. This of course is a dream in a component model like OSGi where you call many different parties of which most you have no knowledge of. Each called service could participate in the ongoing task and be informed when the task was about to be finished. When I proposed a transaction manager the guys around the table looked at me warily and further ignored me, transactions in an embedded environment?

We got very busy but I kept nagging and the rest kept looking if I was silly in the head. In 2004 I finally wrote RFC 98, a modest proposal for a transaction manager light. Of course I immediately ran into the situation that, even though few if any had used it, that there was an already existing Java Transaction API (JTA). Prosyst did some work on this since they saw some value but in the end it was moved to a full blown JTA service. This was not what I wanted because JTA is weird (from my perspective then) because it distinguishes too much between the Container people and the application people. OSGi is about peer-to-peer, containers are about control from above. Try to act like a resource manager with XA (which would give the coordination aspects), however, it looks like it was made difficult on purpose.

The it hit me, I always ran into the opposition because I used a name that too many people associated with heavy and complexity. Though a full blown distributed high performance robust transaction manager is to say the least a non-trivial beast I was mostly interested in the coordination aspects inside an OSGi framework, a significantly simpler animal. So choose to change the name! The Coordinator service was born!

The first RFC was a very simple thread based Coordinator service. When you're called in your service you can get the current Coordination (or you can start one). Each thread had its own current Coordination. Anybody could then join the Coordination by giving it a participant. The Coordination can either fail or succeed, after which all the participants are called back and informed of the outcome. Anybody that has access to the Coordination can fail it, the Coordination will also fail with time out if it is not terminated before a deadline.

So how would you use it? The following code shows how a Coordination is started:

  Coordinator  coordinator = ...

  Coordination c = coordinator.create("work",0);
  try {

    doWork(c);
  } catch( Throwable t ) {

    c.fail(t);
  } finally {

    c.end(); 
  }

This template is very robust. The Coordination is created with a name and a timeout. The work is then done in a try/catch/finally block. The catch block will fail the Coordination. Calling end on a failed Coordination will throw an exception so the exception does not get lost. A worker would do the following to participate in the Coordination:

 Coordinator coordinator = ... 

 void foo() {

   doPrepare();
   if ( !coordinator.addParticipant(this))

     doFinish();
 }

A worker can use the Coordination service to add itself as a participant. It is then guaranteed to get a call back when the current Coordination is terminated.

An example use case is batching a number of updates. Normally you can significantly optimize if you can delay communications by batching a number of updates. However, how you do you know when you had the last update so you can initiate the batch? If there is no Coordinator, the updates are done immediately, with a Coordinator they can be batched until the Coordination is terminated.


During the specification process a number of features were added: direct Coordinations (not thread based), a variable store per Coordination, and a reflective API.

I guess it will take some time before the Coordinator is really taken advantage of since the model is quite foreign to most developers. However, I am convinced that the service is really what OSGi is all about: coordinated collaborative components.

Peter Kriens

OSGi DevCon 2012 / EclipseCon

Did you already make your arrangements for OSGi DevCon? It is almost 4 weeks before the conference starts and you sure do not want to miss it. Not only do we have a very strong program this year, we also have an OSGi Cloud workshop on Thursday. And, last but I hope not least, it will be the last time to meet me in my role as OSGi Evangelist/Editor/Technical Director/Gopher!

I really hope I will meet most of you on the conference!

   Peter Kriens

Cloud Workshop During OSGi DevCon/EclipseCon

The OSGi Alliance is organizing a second Cloud workshop during OSGi Devcon! Thursday March 29. The workshop will run from 9am to 1pm. Attendance is free but requires registration.

RFP 133, our document describing the areas we could look into, has matured over the past year. It is clear from this work that OSGi and the cloud are a natural fit. The next stage is working on actual specifications that will make building reliable and robust cloud based systems easier. For this, we need to set priorities.

We will start with a number of presentations from Paremus, JClouds, eBay and RedHat/JBoss about the work that has been done in this area. After this we will move on to a discussion about priorities since we want to start the RFC work.

We have a limited number of places, if you want to attend register quick because there are a limited number of places and we already have quite a list of registrations. Details can be found here.

 

I will definitely attend the Cloud workshop since it is close to my heart. It will, however, be one of my last activities as the OSGi Technical Director before moving on ...


Peter Kriens

Objects Revisited

Alan Kay is the  inventor of Smalltalk, the first fully truly object oriented language. I learned Smalltalk in the early eighties and almost everyday that I use Java I am crunching my teeth that James Gosling did not steal more ideas from Smalltalk. About 20 years ago, during an OOPSLA, Alan Kay presented the idea that data should always carry its own methods to access that data. His example was a tape (!) that would contain the data as well as the code to interpret that data.

I think this idea was very much at the core of the Java Management standard first proposed around 1997. Each device would have a Java VM on board and the management system could send little management programs that would be executed on the device. However good it sounded at the time (and I tried to push this idea in Ericsson) the idea never became successful, it was just too complex to make it work reliably on a larger scale where machines have different versions and are implemented in more languages I could ever learn in this life. It was just too complicated, error prone, and risky. Exchanging, or relying on, arbitrary code between loosely coupled machines turned out to be a surprisingly bad idea. Objects, however useful they are in many places, seem to be getting more and more in the way when you build larger distributed systems.

The reason is that objects are so ill suited to go outside their process is that they force the objects to expose their innards, the very thing objects try so hard to hide. Even if we could encapsulate the data during the transition as Alan Kay suggested we would create a huge burden on the receiver to understand (and trust) the code that encapsulates the data. We also created a huge dependency problem that the code provided with the data can actually correctly run on the receiver.

There has always been an impedance mismatch between persistence and object orientation. JPA does a decent job but there is something fishy when you need such huge, complicated, and performance intensive middleware only to simplify the life of the developer. Recently I've been doing some more thinking about this subject and I think that though objects work beautifully in a single process they are ill suited for anything that involves crossing the process boundary, which obviously includes persistence.

Last week during an OSGi EG conference call the problem came up again during the discussion of a specification: do we support serialization for some of the domain objects or not? What is often not realized is that serialization is a public interface since it is shared with the world, it is not an internal implementation detail. This is the essence of modularity, there is an inside and there is an outside. What is on the inside only can be changed what escaped from the inside must be carefully (and thus more expensively) evolved since its dependencies are unknown. 

The problem is acute with interface based programming. Two systems running a service defined in interface S (maybe separated in time) that need to communicate their domain objects can only do so if the specification for S defines a serialization format.  Putting a serializedVersionUID in an interface is a total waste of bits (although they do occur!). The only solution that I see is that we need to make the marshalling a first class citizen in the contract since the data representation is part of the public API.

However, what format should be used? The standard Java serialization format is quite awkward to parse except for implementation classes.There is good old XML but JSON is increasing in popularity and there are enough other serialization standards out there to fill books. SQL is also a kind of serialization format. Picking one without making others unhappy will be hard.

I've come to the conclusion that the best format is actually ... Java.  I started to use what I call data classes. These are classes with only public fields of primitives (or their wrappers), strings, data classes, and collections or arrays of data classes. This subset is very easy to (un)marshal to almost any available marshalling technique using simple rules and reflection. These data classes can act as a very convenient schema for my public interface to other processes, including me in the future (a.k.a. persistence). Since they are part of the Java type system they are easy to use and the compiler can do a lot of sanity type checking. And they can easily be versioned in OSGi.

The data classes are a solution to a problem I see becoming prevalent. It is against pure object orientation but I honestly do not see another solution; The shared code model just does not work very well. Sad, but I think it is time to declare defeat, maybe Java 8 should not steal from Smalltalk but the struct from C?

Peter Kriens

Java Generics are a Lemon

After working with Java for almost 15 years and deep knowledge of Java generics on the class format level I learned something very basic the really, really hard way. I knew the collections in Java were not that good in comparison what you find in other environments (immutable anyone?) but now I learned that even adding all that extra cruft on my classes is useless when you have a major refactoring.

This week I learned that for the collections and maps the get, remove, containsKey, containsValue, and equals methods do not use the generic type parameter. This means you can call it with any type and you do not get an error if you call it with a type that is not compatible with the generic type of the collection.

I found this out when I changed many Map types to take another key type, expecting that Eclipse would nicely point me out what to change. Well it does not. The puts and parameter calls are nicely pointed out but a significant amount of code fails because it always fails because the object is now no longer found. Fortunately I am saved by having hundreds of solid test cases that tell me where to look.

I understand these methods were not generified because things became too hairy. Why that did not raise concerns about the power of the generics at the time beats me.

Well, guess I learned something.

Peter Kriens