Processes?

The last few weeks have been quite hectic, closing the Core specifications and working hard on the Compendium. The Compendium contains two major new specifications: Remote Services and Blueprint. The process of creating these specifications seems to have surprised many, and angered some. There is even a blog that makes us look like fools, just making up a process as we go along. Maybe that is understandable, we are trying to run the specification process on a trust and consensus basis and that goes better in an informal atmosphere. People that trust each other just work more effective and are more productive.

However, we do have a formal process documented in RFC 75 and we try to follow it meticulously. This process document has two purposes. First it defines for newcomers the model of how the work will be executed. Second, it is provides clear rules for the worst case when consensus cannot be reached. Unfortunately, it is of course one of the hundreds of things we have to do and most people assumed that the OSGi process was like most other standard organizations. They never have read the process document, really understood the early presentations about the process, nor read one of Eric Newcomer's blogs.

The (seemingly) unique aspect of the OSGi is that it recognizes the role of a Specification Document Editor (SDE). The SDE is paid by the OSGi; it is not an employee of a member company. So far, I had the honor to play this role for the releases 2, 3, and 4.

This makes the RFCs are what their abbreviation say they are: Request For Comments, they are not the final specification. RFCs are input of the specification writing process, a document to reach consensus about a technical design between disparate parties. They can be compared to a design document. And don't we all know very well how the final product changes during development? The vote for the RFC indicates a technical agreement among the EG members, not a vote for the final specification. And frankly, I am a bit disappointed that there is a confusion because the OSGi specification documents look in my eyes very different from the RFCs ...

So let me quote the applicable chapter that describes the current phase (TSC is the technical steering committee that consists of the EG chairs and the Technical Director (me)):

5.6.1 Input
The input to this process shall be the RFC(s) and RFP(s) and any supporting documentation from the EG. This shall be provided to the SDE by the TSC. It is possible that a single Specification may incorporate more than a single RFC and RFP. This integration shall be at the direction of the TSC.

5.6.2 Actions
The SDE shall create the appropriate documents based on the content of one or more RFCs. Under ideal circumstances the formulation of the Specification would be a mechanical process but it is expected that the SDE will uncover inconsistencies or other issues in the RFC(s) which require clarification by the appropriate EG. In this case the SDE shall liaise with the appropriate EGs directly to resolve the issue. The SDE shall have at least one and preferably several review cycles with the appropriate EGs to ensure accuracy prior to completion of the Specification.

5.6.3 Output
When a Specification has been completed it shall be electronically signed using the OSGi Alliance certificate and then voted upon by the EG as stated in section 5.5.7.2. If the document is rejected then it is returned to the SDE together with a written explanation as to the problems with it. The SDE, in conjunction with the EG, shall then modify the document as needed to address the EGs concerns before the document re-enters the formal process.

I think this phase of the process is for a large part the reason that our specs have so few errata. The process of taking documents from an EG and explaining the contents in a consistent tends to discover a lot of issues. Inconsistencies with other parts of the OSGi specs, hidden compromises that do not make sense when looking at things as a whole, hidden assumptions and knowledge, overlaps with other parts of the spec, etc. However good the RFC editors are, it is hard to create a good technical design and at the same time understand, and take into consideration the overall context in which it will be placed. RFC editing is a secondary responsibility, while the SDE is doing this as a primary responsibility. Then again, the SDE has no power whatsoever, any change as well as the final specification, must be approved by the EG.
I do not think any RFC has come through this Specification Writing phase unscathed. However, nobody has ever denied that the specification was better than the input RFC(s) due to this phase, well so far at least.

The second related frustration of last week was a bug report in Eclipse complaining about my reading schedule because there are API changes in an RFC that they based their product on. Since about two years, we addressed public concerns that we were too closed, by publishing interim drafts of the RFCs as well as specifications. Obviously, we made it crystal clear that there are no guarantees about the final specification. Worse, we had virtually no feedback for the RFCs so far and that we are now being banged on the head for fixing issues that come up late during specification writing. It ain't over 'til the fat lady sings ...

That said, I am not denying we have a problem. Though the core went out on the planned date, but the compendium is 4 weeks delayed and it is not completely clear Remote Services and Blueprint will be finished in that time frame. Part of the problem is that the OSGi Alliance has only one SDE, and that person (me) is only being paid part-time to work on it. The EGs have been very active lately and this has significantly increased the workload. We need to fix this somehow. However, I do think we should keep the SDE role for the sake of the final quality.

However, in my opinion, a specification is not cost free for a community, a bad specification can actually be quite expensive. I will not use any names here. I pride myself to work for an organization that actually wants to publish high quality specifications and is willing to pay the (sometimes) steep price. Tim Diekmann's (co-chair of the Enterprise EG) mail signature is:

"There is never enough time to do it right, but there is always enough time to do it over" -- Murphy's Law

Well, so far, we actually always tried to do it right because specs cannot be done over. Even if it sometimes really hurts.

Peter Kriens

P.S. Processes rate slightly above licenses on my list of favorite subjects ... Back to RFC 119 and RC 124.