Reified Types: Conversions Using Generic Types

One of the most interesting aspects of the Blueprint specification was the ReifiedType. I ran into this nifty class again today because we want create a Conversion service that is useful by the new upcoming OSGi shell service. We had decided to reuse the Blueprint conversion model, but now as a separate service, something which we probably should have done already for Blueprint.

The problem that Reified Type solved for us is how to convert an Object to a specific target type by taking any generic information into account. For example, if you convert an int[] to a Collection<Long> you want to be sure that the collection contains Long objects and not Integer objects. That is, a naive implementation would convert this to Collection<Integer>, which would miserably fail later on when someone uses the collection.

I can hear many readers thinking: "But generic information is erased, isn't it????" Yup, generics are erased, but not completely! When you have an object there is no way to know the generic parameters (the parameters between the angled brackets like T, K, V etc.). I am not sure why they decided to not provide this information with the object because it seems very doable but it clearly is not present. However, the VM maintains extensive generic information everywhere else but in the object. If you have an instance field then it can describe exactly what its generic type constraints are. For example, if you have a field numbers, declared like:

Map<String,Class<T>> numbers;

Then you can get the generic type constraints with the getGenericType method that returns a Type. Virtually all reflective methods have been duplicated to provide this Type object where they used to return a Class object.

OSGi APIs are written in a restrictive subset of Java so that they can be used in a wide variety of VMs. Though we found a solution allowing us to use generics in Java 1.4 (JSR 14), we must assume a target of Java 1.4 with the minimum execution environment for our daily work. This means there is no Type class we can rely on in our APIs.

Obviously, in enterprise scenarios the VM is Java 5 or later, and the Type class is readily available. Is there a way we can eat our cake (use the generics info in our Conversion service) and have it too (no dependency on Java 5)?

The first solution that comes to mind is to duplicate all the sub-types of the Type interface in our own name space. There are actually quite a few sub-types of the Type interface. These sub-types models (and more!) all the flexibility of the generics constraints system: type variables, arrays, wild-cards (super, extends), and parametrized types.

Hmm, not only is duplicating no fun, it also turned out that the hierarchy is not that easy to work with. For our Conversion service we only need to know the raw type of a type parameter. That is, if our target type is Collection<Long> than the we only need to know that the type parameter is Long.

After struggling with this problem we found an interesting solution: the ReifiedType class. A Reified Type collapses all the intermediate levels of wildcards, variables, and arrays and provides direct access to the most specific types that can be used. It turns out that by traversing the network of Type objects it is always possible to end in a raw class, even with variables, wildcards, and arrays.

A Reified Type is always associated with a single raw class. The raw class is the class without any generics information. It provides access to the type parameters by their position. For example, the Map<String, ? extends Number> has a raw class of Map and 2 type parameters: String and Number. For this example, the previous Refied Type has a size() of 2, and provides a ReifiedType<String> and a ReifiedType<Number> for the type parameters.

In the Blueprint specification we provide a concrete implementation for Java 1.4 VMs. In Java 1.4, there are no generics so this implementations hard codes the number of type parameters to 0 by definition. We will do the same in the Conversion service. However, users of this service will be able to pass subclasses that provide the generic information to get better conversions.

So it looks like this was a case where we could have our cake and eat it too! You can already look at Reified Type in the Blueprint specification and in the next release I hope we can have a new Conversion service based on this model.

Peter Kriens

Mea Culpa

Yesterday the OSGi Alliance introduced the new look and feel and I choose the rather despicable way of spamming many of this blog loyal readers with blogs that were never meant to be published, at least not yet. By republishing the blog to slide into its wonderful new skin I made the mistake to select all blogs and forgot to deselect some of the drafts that were in the pipeline (or forgotten).

Excuses, at least I hope you enjoy the new look & feel as well as the updated content.

Peter Kriens

To Coordinate in OSGi

Since day one of the OSGi Framework (over 12 years ago) I have been trying to get a light weight transaction model into the specifications. After several attempts that were skillfully aborted by others or way to heavy for what I had in mind I had actually given up. That is, until the last face-to-face meeting in Mountain View. David Bosschaert (Redhat and EEG co-chair) was looking for a better Configuration Admin solution than the Managed Service (Factory) services provided. One of the key requirements he had was that of a composite update. In Configuration Admin updates are done per PID. A PID is a persistence identity for a Dictionary that contains the configuration properties. David argued that in larger enterprise applications there is need to compose the configuration properties out of a number of smaller dictionaries that each represent a configuration aspect. For example, there could be a com.example.system.config PID for system configuration and a com.example.http PID for configuring an HTTP server.

An very good idea I think but he was confronted with a serious problem. Though Configuration Admin nowadays allows the use of multiple PIDs per service, it does not give any timing guarantee other then that each update will be done on another thread than the setter. The problem with the multiple configuration updates is thus that you could get parts of your configuration milliseconds apart. That is normally not good because changes in configuration are potentially causing expensive operations. After several attempts to find a good solution we realized that the transaction model could solve this problem rather nicely. If the Configuration Admin was transaction aware it could wait with updating the target services and sending out events until the commit part of the transaction.

Now there is something funny with transactions, they have a weird effect on system developers. The moment you start talking about transactions they seem them gain 10 pounds and age 10 years. Transactions are seen as very heavy weight because of the recovery requirements and setup. And acting as a resource manager is a non-trivial task with the XA API. Having lost the battle to use real transactions several times (Framework, Dmt Admin, Deployment Admin) I was not prepared to start such a battle again. That is, until it dawned on me that what I really was looking for was a coordination API, transaction are providing much more than I needed.

My side of the software fence has been relatively free of persistence problems, the whole ACID part of transactions was never my favorite part. I liked transactions because it allowed coordinated collaboration between different parties. In an OSGi system you never know how the processor flows through different services when you call another party. In this model there are many cases where you could things more efficiently, or atomically, if you only knew when the task that was being worked upon was being finished. The coordination part of the transactions was always what I liked so much because I knew there was going to be a callback at the end of the transaction.

So could the answer be a light-weight coordination API? If the Configuration Admin was updated to use this API, it could not send out notifications or update managed targets until all the changes were made, that is, at the end of the coordination. So how could this look like (notice: API is work in progress):

Coordinator coordinator = ... ;
ConfigurationAdmin admin = ... ;

void updateConfigurations(Map configs) {
Coordination c = coordinator.begin("update configurations");
try {
 for ( Map.Entry e : configs.entrySet() ) {
   Configuration c = admin.getConfiguration(e.getKey());
   c.update( e.getValue() );
 }
 if ( c.end() == OK )
   return;
 // log!
} finally {
   terminate(); // Ensure proper termination
}
}

So how does this look like for the participants? For example, how would Configuration Admin schedule its updates when it would use coordinations? Well, a participant must indicate it wants to participate in a coordination. The method to start a participation is on the Coordinator service. The code to participate is as follows for a schedule method of a Runnable that wants to delay scheduling the Runnable's until the coordination is ended:

 final List queue = new ArrayList();

void schedule( Runnable r ) {
if ( coordinator.participate( this ) ) {
 synchronized(queue) {
   queue.add(r);
 }
} else
  executor.execute(r);
}

The Coordinator will callback the participant on either the failed() method or the ended() method. The failed() method can be called concurrently with the initiating thread. The ended() method is always called on the initiating thread.

 // the coordination failed, clear the result
public void failed() {
synchronized(queue) {
  queue.clear();
}
}

// the coordination ended ok, clear the result
public void ended() {
for ( Runnable r : queue )
  executor.execute(r);
queue.clear();
}

The coordination API therefore allows two completely different implementations to synchronize their work on a common task. This API seems incredibly useful to optimize many of our existing admin APIs: from the framework itself to Remote Service Admin. I wish I had realized much earlier not to call it a transaction API ...

Peter Kriens

P.S. This Coordination API is work in progress, there is no promise this work will ever end up in an official OSGi spec.

How to use Config Admin?

Config Admin is one of the most powerful services the OSGi standardized but is often badly understood. The only reason why it seems hard is because it was designed for highly dynamic long living environments. Most applications start, read their configuration, do something, and then die because someone types control-C or the application exists the VM. In a dynamic environment your configuration can change at any time and you're expected to react to these changes. Users tend to like this model.

What we did is merge the initial "get my configuration" phase that so many people are so used to with the notification that there is a configuration change. With the intent to simplify the coding we only have one mechanism: update. The initial phase is a guarantee that the the Configuration Admin service will update you, even if there is no data provided yet. Many programmers hate this model because they do not feel in charge, they want to grab their configuration when they need and not wait until some louse Config Admin calls them. Though this is feasible, it just a lot more work. Relax, and leave the initiative to Config Admin and it all falls into its place.

The key concept of Config Admin is that the receiver of the configuration registers a Managed Service with a property called service.pid. The value of the property is some unique identification. The Config Admin will then call the service's update(Dictionary) method. If it has a configuration for that service.pid, the value will be a set of properties in the Dictionary. If there has been no configuration set, it will call will null. The key thing is to let the Config Admin control you instead of trying to be charge. Another type of Inversion of Control ...

Lets do a simple example of an echo service on an internet port:

public EchoServer implements ManagedService, BundleActivator {
EchoServerImpl server = EchoServerImpl(-1); // Dummy server

public void update(Dictionary props ) {
 int port = -1;

 if ( props != null) {
    Object o = props.get("port");
    if ( o != null )
      port = (Integer) o;
 }
 if ( server.getPort() != port ) {
   server.quit();
   server = new EchoServerImpl(port);
 }
}
}

To set Config Admin data, take a look at the Felix Webconsole, Felix FileInstall, or the Knopflerfish environment. They all support a range of (G)UIs to create configuration records.

Peter Kriens

Bundlefest in Girona!

Next week the OSGi organizes yet another bundlefest in Girona. This time we convene with the Residential Expert Group (REG). The week will start off with a REG meeting and then the remainder will be hacking with bundles for the REG release's compliance test suites and reference implementations. If you were a member, you could have participated in the fun! I am really looking forward to this week because it is a nice group of people to hang out with, the weather looks good in Spain next week, the hotel seems very nicely located, and the residential area is becoming hot again.

There is a surprising amount of activity in the residential area. All the signs are green for this area to finally happen. Many operators are now starting to develop residential gateways. Big projects are happening all over the world, which is now creating more and more interests of vendors. Where we had a hard time selling OSGi in this world a decade ago, today it is more or less a given. Ten years ago the technology was deemed unproven, today that is hard to argue. There are also no real alternatives to the comprehensive execution model that the OSGi provides.

To see how OSGi is solving real problems in gigantic applications running on enterprise servers but also able to hold its own in the embedded world is incredible. As I argued many times before, I do believe we've developed a software model (µservices!) that is surprisingly fundamental. It it is very exciting to be part of this development.

Peter Kriens

P.S. The Enterprise Specification is now available in book form, you can order it now!

Duct Tape

I discovered software when a high school friend got a programmable TI-57 calculator. This calculator made me fall in love with developing software; a love affair that has not ended yet. Falling in love made me want to read everything about her. In the pre-google era books were used for that purpose and I read all the books I could get my hands on. At that time software books were mostly about structured programming. The basic idea was that you take a function, decompose it in smaller function, and so on. The mantra for this decomposing was:

• low coupling
• high cohesion

The idea of coupling was not hard to understand but I had a bit of a problem with cohesion, it was more fuzzy. And I never liked the design magic that structured programming required. Therefore, when I discovered my mistress' objects around '83 I fell in love all over again. In the period between the calculator and the discovery of object oriented programming I had studied, started a company, and developed a number of editorial systems for newspapers. Until that time most with assembly and crafting a lot of code for networking, databases, operating system, and even a GUI (eat your heart out!). As any new generation we worked hard to ignore the lessons from our predecessors and objects seemed to be the perfect vehicle to put the old geezers in their place. Move out of the way Tom DeMarco, Edward Yourdan, and Michael Jackson, we're coming! So the cohesion and coupling mantra's went out of the window and we showed how data encapsulation with inheritance brought you all these goodies.

Most of you know the results. though objects work very well on the medium scale, once the systems grow and evolve they tend to create systems that resemble spaghetti and become hard to maintain. Something got lost along the way. Interestingly, spaghetti code was exactly the problem that drove structured programming.

Problems that we solve on one level appear almost identical on higher levels is a hallmark of fractal problems. The size of systems has increased exponentially since I started and spaghetti problems that we thought we had solved reappear in a different incarnation. So how do we address the spaghetti problem at our current scale? Could the old structured programming mantra help us when we apply it to objects? Lets take a look.

Modules were defined for functions, not for classes but when we took about a module today we talk about a set of classes with restricted accessibility. Now, classes have a tendency to act as duct tape: very useful and darned flexible. As the joke goes: if you can’t fix it with duct tape, you probably just haven’t used enough. However, classes can also be just as sticky and they can get surprisingly entangled when you're not paying attention. So even if OO modules are restricting accessibility, they've lots of classes sticking out eager to get entangled with other classes, preferably from other modules.

Clearly interfaces solve the sticky type problem by type-separating the client and the provider of an object. However, they do not help in getting in getting an instance without getting entangled with the implementation class. We have to solve that within the OO paradigm: Listeners, Factories, and Dependency Injection. However, these patterns never completely got rid of the stickiness of the implementation. Listeners require the client to know the library implementation, Factories require the factory implementation to know the implementations some way and with DI frameworks there is some all-encompassing God-XML that is sticky as hell. The big problem is that in ALL those cases the implementation class somehow leaks out of the module ready to stick. Often as a string instead of a class but the only advantage of that is cosmetically: your dependency graphs look better because the analyzers usually miss these couplings. However, the dependency still is as concrete as a class coupling and just as bad, just looks better.

So our current toolbox of patterns may hide the coupling to the implementation classes but they do not really get completely rid of it. They also have some surprisingly bad characteristics that we somehow got used to. These patterns are extremely biased to the client and forget that the module that provides the instance might have something interesting to say as well. They all force the provider to provide an object at the time they deem right without letting the provider decide if it is ready and if it really wants to provide an object. The provider has no way to signal when it is ready and maybe even offer multiple instances. Again, Factories and DI are surprisingly one sided.

Let us start with context. When you call a factory or use DI then the caller/receiver can provide parameters but the callee has nothing to say. An instance is created out of the blue from one of its classes. If there is any context, it must be stored in static fields, which is an evil software practice. A module that provides an implementation can not easily different instances based it inside knowledge. For example, maybe a provider tracks instantiated machines in the cloud and wants to offer an object per machine. Sorrt, can't do that. The provider module can only provide some manager where the manager then provides listeners, etc. etc. This lack of context makes APIs seriously more complex.

Similarly, a providing module has nothing to say about timing. The caller of the Factory or the DI engine make the decision when to instantiate the class from the module. The providing module has nothing to say about when it is actually ready to provide such an object. This makes startup ordering and dependencies really hard to manage.

We've all been working with these patterns for so long that it is very hard to see how limited they are. However, in a truly modular system the client module and providing module are peers that collaborate. Each module can play the client role and provider role in different times and neither the client role nor the provider role is passive. None of the existing patterns can handle this peer-to-peer module, all fall short in handling type coupling to the implementation class, the dynamic context, and the timing.

In a perfect world, each module should be able to offer an object to other modules. And each module should be able to react to the availability of those objects. This is the Broker pattern. If I need a Log object, I ask for a it (or get injected when it is available). With the Broker pattern, the providing module can decided when to offer, solving the timing problem. The providing module can instantiate as many objects as it likes and offers them, solving the context problem. And last but not least, it is the providing module that creates the object and thereby never exposing the implementation class.

That said, there is one thing that is still fuzzy in this model: who defines the collaboration contract? Is this the provider? Well, that makes it hard to use other providers for the same collaboration. After a lot of hard thinking it seems obvious that modules are not enough, we need to reify this contract in runtime. If we reify the contract then modules can provide this contract and use this contract without being bound to a specific implementation. Both providers and clients would be peers.

So if we talk about software modularity than in an Object Oriented world we need to augment modules with something that reifies the contract between the modules. So far we've struggled with half baked solutions like Factories and the current DI model to address the problems caused by the lack of a reified contract. These contracts are much more important for a design than module.

I strongly feel that today we're in an almost identical situation as in the eighties when objects addressed the shortcomings of structured programming. This was a hard sell because many people looked at the implementation details and not at the design primitives that OO provided: encapsulation of data, inheritance, and polymorphism. Making people see the primitive instead of the vtables and descriptors was hard work.

I believe that today we need a design primitive that allows us to reason about the collaboration of modules. A design primitive that allows us to design large scale systems and still understand the fundamental architecture of what've we created. A primitive that allows us to see a picture of a complex system and understand quickly how it can be extended. OSGi µServices are very close to that primitive I believe.

I must admit that several times in my life I was unfaithful to my mistress. When we first got married she was all Smalltalk and I loved her for it. But over time she developed a lot of C++ behavior and I not so happy about that. Fortunately, she decided to pick herself up and become so much more dynamic with Java. But if I am really honest I was eying other software the last few years, still missing some of her old dynamism. However, since I saw the potential of her µServices I was head over heals in love again.

Peter Kriens

The Catwalk

I guess there is something in the air at the moment that makes people worried OSGi is not successful quickly enough because there are not 7 million Java programmers using the OSGi API on a day to day basis. Kirk Knoernschild gave us the choice between Feast or Famine and SD Times told us OSGi is too complex for the enterprise developer. Well, feasts tend to end in hangovers and I do agree the OSGi API is not very useful for most Java web developers. Is OSGi actually a technology that is used by a (web) application programmers? Will web developers have to start using Service References and Bundle objects? I don't think so.

If you develop, for example, a social networking app then you should not be concerned about low level infra structure details and OSGi should completely stay out of your face, this is the concept of high cohesion. OSGi should never be visible on the (web) application level. However, if you write a middleware component and need to interact with the applications then you need domain objects that represent these applications. Bundles and Service References are then the perfect domain objects that allows you to interact with that app on a system level. For example, the Spring DM extender bundle leverages the OSGi API to allow a developer to write POJOs oin his bundle. Many middle-ware APIs can be simplified because the OSGi API provides detailed information about the callers, making the APIs significantly more cohesive.

OSGi itself does not simplify application development, it only enables third parties to provide frameworks that then can simplify the life of the application developers, or empower them. They function provided by OSGi is the collaborative model that makes different frameworks no islands on their own but actually allows them to work together. OSGi defines the collaboration mechanisms, not the nice to have convenience functions for web development. What OSGi allows is breaking a huge application problem in smaller collaborating parts. The cumulative complexity of a collection of small parts is less than those parts combined. However, to enable this collaborative model we must enforce a number of modularity rules. Rules that are limiting on the component level to create more flexibility on the deployment level.

Unfortunately, those rules break a lot of existing code. We often talk about modularity but in reality we tend to create highly coupled components. When these "components" are run on OSGi they crash against the module boundaries because OSGi enforces these boundaries. Many people forget that a class encoded in a string in an XML file is creating as much coupling as that class used in your code. The only advantage is that these strings do not show up in your automated dependency graph ... OSGi is just the unfortunate messenger of evil hidden coupling.

Application servers adopted OSGi because their problem domain is highly complicated and large. So large that strong modularity was their only option to keep things manageable and OSGi was perfect for this because it already contained some of their domain objects. Most Java application developers develop web apps. Web apps are a highly restricted domain that has been extensively simplified by tools and libraries. Improving on this has a very high threshold. This is the same problem as with the combustion engine and helicopter; there are better technologies in principle but the incumbents of a huge head start in optimization. Therefore we've adopted the WAR format. WAR files will make it easier to start running web applications on OSGi without having to introduce new tools for the developers: their WARs will run on OSGi unchanged. Over time they then decompose their WARs into smaller bundles.

There is one innovation in OSGi that I think is highly undervalued: the µServices. µServices are a paradigm shift as important as the move from structured programming to object oriented programming. µServices are the pinnacle of modularity. If they're so good, why does it take so much time before everybody uses them? Well, SD Times provided some insight, they said that a new technology X is irrelevant because developers have been building fantastic systems for a long time without X. It is hard to better illustrate the reasons why paradigm shifts are so hard and can take multiple decades.

As with OO, there is a chicken-egg problem. To take advantage of µServices you need components that provide and consume these µServices. Our problem so far has been that the major adopters (Eclipse/App Servers/Spring) picked OSGi for its class loaders on steroids and treated the µServices as an extra. But things are changing. Last EclipseCon it was clear that µServices are moving to the front. People that could not care less about services now publicly declared their support for them. Eclipse provides now good tooling for µServices, which will make services more attractive for many Eclipse developers. I am sure this will create the needed network effect.

Kirk notes how our industry is more fashion driven than the fashion industry and both authors complain that OSGi is not visible on the catwalk. And that is correct because OSGi is the catwalk, present in every fashion show picture and sustaining virtually any application that runs on a Java EE Application server based on OSGi, which are actually most of them.