Metatypes

Many, many, many years ago I worked a lot with LDAP and the ISO communication standards like X.500 and X.700. Looking at the Metatype specification I am afraid I had some bad influence on that spec. Looking back, I probably would be able to make it a bit simpler today, however, I do believe that it is one of the specifications in the that under-appreciated. Actually, a lot of people have no clue what it is.

So what problem does the Metatype spec solve? The problem it addresses is runtime configuration. The OSGi Configuration Management specification makes it possible to configure bundles with properties. The problem we faced that gave us Metatype is that we needed a schema to describe those properties to editors and validators. At the time I was involved in creating automatically created user interfaces based on type information. The Metatype specification provided such type information for the configuration properties. In the first incarnation of Metatype we did not describe a format (seems silly and I forgot why), only hard to use Java interfaces. Later versions added a Metatype Service and an XML based format for describing the properties.

The Metatype specication has been in existence for quite a long time there are as far as I know very few editors built for it. Actually, the only one I am familiar with is the Apache Felix Webconsole. I am very interested in Metatypes because the triplet metatype + configuration + components (DS) provides a surprisingly powerful programming model. DS can control the life cycle of components based on the configuration data that is available. Creating a factory configuration causes the activation of a component (if it is satisfied of course) and deleting that configuration deactivates the component. Each component is provided with the properties of that factory instance. For example:

@Component(configurationPolicy=required)
 public class FactoryDemo {
   @Activate
   void activate( Map props ) {
      ...
   }
 }

Such a component will follow the life cycle of a configuration or factory configuration of the (factory) PID com.example.FactoryDemo (the name of the component). This model is one of the cornerstones of good use of OSGi: configuration driven design. This model allows us to register services that are under control of the deployer. For example, assume you want to make an interface to Amazon S3. Where do you store your acces key and secret key? If you have any real world experience you will of course not hard code them. So in a good design you would write the following code would register a Store service:

@Component(configurationPolicy=required)
  public class BlobStoreComponent implements Store {
    String domain;
    AmazonS3Client client;

    @Activate
    void activate( Map props ) {
      String secretKey = (String) props.get(".secret.key");
      String accessKey = (String) props.get(".access.key");
      AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey); 
      client = new AmazonS3Client(credentials);
      domain = (String) props.get("domain");
    }
    ...
  }

With this component, the deployer can now decide how many, and for which domains a BlobStore service should be registered. This is quite a big step in information hiding with very little work (try doing this in other environments). However, setting these properties is always awkward, to say the least. Adding a Metatype file in the bundle would make the editing easier. Set lets add a Metatype XML file in OSGI-INF/metatypes/com.example.BlobStore.xml:

«metatype:MetaData xmlns:metatype='http://www.osgi.org/xmlns/metatype/v1.1.0'»
  «OCD 
    name='Blob store component config' 
    id='aQute.metatype.aws.BlobStoreComponent' 
    localization='aQute.metatype.aws.BlobStoreComponent$Config'»
    «AD name='Domain' id='domain' cardinality='0' required='true' type='String'/»
    «AD name='Secret key *' id='.secretKey' cardinality='0' required='true' type='String'/»
    «AD name='Access key *' id='.accessKey' cardinality='0' required='true' type='String'/»
    «AD name='Region' id='region' cardinality='0' required='true' type='String'»
      «Option label='US standard' value='US_Standard'/»
      «Option label='US west' value='US_West'/»
      «Option label='EU ireland' value='EU_Ireland'/»
      «Option label='AP singapore' value='AP_Singapore'/»
    «/AD»
  «/OCD»
  «Designate pid='aQute.metatype.aws.BlobStoreComponent'»
    «Object ocdref='aQute.metatype.aws.BlobStoreComponent'/»
  «/Designate»
«/metatype:MetaData»

This gives us a decent GUI on the webconsole:

Unfortunately, to get this GUI it looks like we're back in XML hell? Nope! bnd comes to the rescue! In the latest release 1.42 there is really nice support for create the Metatypes from a Java interface, using optional annotations to provide additional information. The previous GUI was created automatically from the following interface:

interface Config {
   String domain();
   String _secretKey();
   String _accessKey();
   Region region();
 };

The title is derived from the configuration interface name and the property key labels come from the method names, it uses the camel case and characters like $ and _ to turn them into something more pleasant to read for the labels. The _ in front of secretKey and accessKey indicate private properties, they will not be registered as a service property. Notice that he Region, which is an Amazon enum, is used to create a popup list with readable labels and the exact names of the enum. bnd will automatically create this for you. So what do you have to do to make this all work?

@Component(designate=BlobStoreComponent.Config.class,immediate=true)
 public class BlobStoreComponent implements Store {
  interface Config {
    String domain();
    String _secretKey();
    String _accessKey();
    Region region();
  };
  Config config;
  AmazonS3Client client;
 
  @Activate
  void activate(Map map) {
    config = Configurable.createConfigurable(Config.class, map);
    AWSCredentials credentials = new BasicAWSCredentials(
       config._accessKey(), config._secretKey());
    client = new AmazonS3Client(credentials);
    client.createBucket(config.domain(),config.region());
  }

  public InputStream get(String name) {
    S3Object obj = client.getObject(config.domain(), name);
    return obj.getObjectContent();
  }

  public void put(String name, InputStream input) {
    client.putObject(config.domain(), name, input, null);
  } 
 }

This is all you have code ... No property handling, not need to check for null properties (by default the fields are required), not mucking around deep in XML, all type safe ... The same package that contains the optional  annotations contains a tiny little class that creates a proxy that maps the methods to the property and performs type conversion if needed.

If you want to experience OSGi heaven (or at least the road towards OSGi heaven), download the alpha version of bndtools and play with it. This code and other samples are provided so you can directly use them with bndtools. You can find  this code at an aQute github repository.The Metatype and Components chapter of the bnd manual define the details.

For me, the metatype support is fixing a crucial extra step to take advantage of OSGi. I am looking forward to make many examples in the future that take advantage of the inredible power of metatype + configuration + components. For ant lovers, the latest version is available from the download page. For maven, this will require some collaboration with the maven bundle guys but I will work on this. Enjoy, and hope to see you next week at OSGi Devcon/EclipseCon!

Peter Kriens

PS. The hackathon (or hackaton as I erroneously called it) is on Thursday morning but I do not have the room yet. If you're coming send me a mail and I'll keep you posted.

Modularity is Hard, lets do a Jigsaw?

Reply to an interesting blog at DZone from Martijn Verburg about modularity, OSGi, and Jigsaw. The blog was about the eternal discussion of using the package as the atom of dependency or the JAR. So one more time.

After putting up with the argument of package versus JAR as dependency point I've noticed a pattern in people. Initially the proponents of the Jigsaw/maven model of JAR dependencies can only indignantly come up with one argument: it is simpler. We actually added Require-Bundle under protest for Eclipse, which actually models most of the Jigsaw/maven view of the world. However, once people start to use OSGi they discover that the model is not simple, it is simplistic. Yes, if you make a "hello world" Require Bundle might be simpler, but when you have to maintain a 400.000 line code base over many years you quickly discover why it is not simple but simplistic. I've not met an experienced OSGi user that argues for Require-Bundle, and I am pretty sure that Eclipse would not have pressed for Require-Bundle if they had known what they know now.

Having been on the OSGi side for more than ten years I came to compare it to plumbing. On the OSGi side the plumbing is done well and we got rid of the smell but when I visit the other side I cannot be but amazed at how messy and smelly life is there, like plumbing in the medieaval times. Though there is a lot of amazing functionality out there, invariably when you analyze these wonderful applications and libraries you find hundreds to thousands of type references that are impossible to satisfy in run time. Ticking bombs waiting to hit you when they cause the most harm. Versioning is absent, it is a mess, or it is not very useful at all. Beautiful castles but built on quick sand.

The tragedy of OSGi is that the model puts a lot of these hidden problems up front in your face of letting them explode at the customer's (always wondered if this is the reason our industries invests so much in logging). Most open source libraries never worry about their dependencies and drag in transitively many megabytes of unneeded code. Hey, the only time you notice this is when maven starts downloading the Internet and that is only once due to caching. In OSGi you have to handle the dependencies up front, without OSGi you can put your head in the sand and "prove" that it works by running it for some time without a Class Not Found Exception. If people build bridges like the average WAR I probably would buy a boat.

So yes, OSGi has a threshold that is not easy to cross, but there is nothing tooling cannot do. Afterall, we have a compiler that handles the byte codes for us, in the same vein tools can remove most of the additional metadata for OSGi. Fortunately, the tooling is improving, take a look at bndtools, which is supposed to integrate with Apache Sigil soon. This is an Eclipse plugin that works very well inside Eclipse but also provides a small ant plugin that can build workspaces in a headless build, providing identical results inside and outside Eclipse. I also know IBM is investing heavily in OSGi tooling for Websphere. There is of course Eclipse PDE and a derivative of PDE with SpringSource but these tooling are not very oriented to package dependencies, hope they will change too.

So I challenge anyone to argue why the Jigsaw/maven's model is technically superior to OSGi's package model. So the question is: do we want to move the Java industry forward by picking the technically superior model or do we keep  building on quicksand?

Peter Kriens

OSGi's Role

With the increased adoption of OSGi the issue of when to use the OSGi API pops up more and more often. One misconception, popularized with Dependency Injection, is that NO code should touch the OSGi API.

The reason code should not touch a specific API is cohesion. If you write a program to control the brakes of a car you'd likely not want to include code to optimize fuel consumption. Cohesion is about keeping focus in your module; the more concerns get mixed, the less cohesive the module gets. Low cohesion modules are bad because they are more complicated to maintain, which translates to extra cost, and they are less usable in another model because now the module requires both the brake semantics as well as the fuel semantics to be applicable for any new model. The less cohesive, the smaller the chance all the domains match the new usage area for the module.

So it is very true that you do not want to mix OSGi API with your shopping cart code for an ecommerce site, these are different domains. The quest for high cohesion forbids mixing these domains.

However (you knew this was coming), there are many cases where the application infrastructure is the domain. Looking at open source you find that (often too) large swaths of code are not related to some application domain but they are about the infrastructure of the application, or said differently, the plumbing. Application plumbing can be very beneficial; it is all about reducing application domain code. Dynamic class loading, annotation processing, reflection, and many other techniques are all used to reduce the amount of application code with plumbing code. Clever plumbing code is reusable so over several systems the total complexity will be much less.

Application plumbing is exactly the domain of OSGi.

So once you start using class loaders or play with Class.forName realize that you are acting as an application plumber. Whatever you're doing, once your code uses a class loader your no longer in Kansas. If you have to use Class.forName and class loaders you're like a plumber in the medieval ages. You might be a craftsman but the environment gives you very little support so you're largely on your own, devising your own tricks. There might be some common patterns you picked up from your local guild but there are no standards in the Java VM.

Once you realize you're into application plumbing you should realize that you're right at the heart of OSGi's application domain. In all those cases the OSGi API is perfectly suitable to use and gives you much cleaner and reusable solutions than continuing to abuse the Java APIs in this areas that were never intended to be used for application plumbing.

So to conclude that one should never use OSGi APIs is misunderstanding the cohesion issue; once you do application plumbing it is much more advantageous to use the OSGi API than any alternative route. The OSGi gives you a comprehensive framework for application plumbing that is unmatched in the software world.

Peter Kriens

Exception Hygiene

Arguably the worst decision in Java was to have checked exceptions. I've never been thrilled by Java as a language (after Smalltalk all other languages seem, well, just not something to get excited about) but checked exceptions seem to get to me every time. I just had an another bad experience with this exceptionally bad idea.

I recently had to work on some open source code, the code was about a specific servlet. After working around some of the limitations I decide to just fork it. The servlet was not working very reliable and I could not really pinpoint where the problem was. So I decided to just remove all low level exception handling and handle all exceptions in the service method of the servlet. The authors had done the traditional way. This is making APIs without general throw clauses so you have to catch all the bad stuff and then turn it into some specific exception. It was amazing how readable the code became, and, even better how easy it was now to diagnose because anything that went wrong always ended up in the catch block of the service method. Easy to set break points, and easy to handle. The other improvement was that I now could get rid of all of the messy logging code that also cluttered the code.  Virtually all code was logging exceptions and this logging could now be done in one place. Removing checked exceptions this way is imho the cheapest and most effective way to improve your code.

So how do I handle exceptions? Well, for me a function all has two exits: a normal return and an exception. I do not believe exception types matter, only badly designed APIs where exceptions are abused for normal behavior require their types. By having this simplified model, any API can throw any exception and I usually declare "throws Exception" on my API methods so I do not have to catch any lower level exceptions. This model makes it easy for an exception to bubble up to the top level method that knows what to do with an exception. Obviously intermediate functions must use the finally block to ensure that any any resources are properly closed.

For the OSGi APIs we've not followed this model because lots of people still need to get used to ignoring checked exceptions. However, if I see how far we've come since the early days then I think we can one day expect this style to be common.

Peter Kriens