Monday, August 24, 2015

Building the Internet of Things with OSGi



by Tim Verbelen, iMinds, Ghent University


Tim Verbelen One year ago, I was speaking on my first EclipseCon Europe about the results of my PhD research (Mobilizing the Cloud with AIOLOS). Since then, I have been working for iMinds as a research engineer. iMinds is a digital research centre in Belgium, which joins the 5 Flemish universities in conducting strategic and applied research in areas such as ICT, Media and Health.
Therefore, iMinds is uniquely positioned to bring together multi-disciplinary teams to work on various emerging topics. I myself am working within the iMinds IoT lab, which works on a wide area of topics related to IoT, ranging from antenna design and wireless MAC protocols, to software security and distributed computing, the last one being my main expertise.
In our research, we try to not only come up with theoretical solutions, but also strive to create tangible results in the form of demonstrations and proof of concepts. As a researcher, you get more freedom in choosing which technology to use in building your solutions, in contrary to in industry, where you are often tied to a lot of legacy software. For IoT, the choice of using OSGi was made early on, which proved to be a good fit for a lot of IoT requirements.
One challenge in IoT environments is the hardware heterogeneity you have to cope with. You need to deploy (parts of) software on a wide variety of devices ranging from embedded devices up to high-end servers in the Cloud. As OSGi was initially designed for service gateways, it is well suited to run on even lower-end devices. The recent work in the Enterprise Expert group also equipped it with a lot of features to operate in a server environment. This makes OSGi a perfect fit as a base for an IoT platform, as the modularity allows you to pick and place the software modules you need on any of your devices.
A second challenge where OSGi really shines is the dynamics you have to cope with when developing IoT applications. As the physical world is constantly changing, you will need to adapt to new devices that are coming online and other devices that disappear at runtime. This is incredibly hard to manage in software as the complexity increases. The OSGi bundle and service model already handles these dynamics and offers the developer a nice and easy way to cope with this using for example Declarative Services.
Third, OSGi offers a nice solution for software distribution with the Remote Services specification. This enables you to delay the decision on which part has to run on which device until deployment time or even at runtime, instead of having this fixed already at development time. This gives you a lot more flexibility in deploying complex applications on a distributed infrastructure.
In order to even better match IoT industry requirements, the OSGi Alliance has recently started anIoT Expert Group, which will build on the already available specification work and where additional IoT-specific RFPs can be submitted to become part of the OSGi specification.
In my talk, I will present and demo some IoT use cases we have developed, and illustrate how we really benefit from using OSGi. If you are interested in OSGi and/or IoT, you are invited to attend my session,OSGi for IoT: the good, the bad and the ugly at EclipseCon 2015 in Ludwigsburg.

Reposted with permission from Tim Verbelen, original post at Eclipsecon Europe 2015

Wednesday, August 19, 2015

OSGi Residential Release 6 Specification

The OSGi Residential Expert Group is excited about the new specifications introduced in the OSGi Residential Release 6 Specification. You can find it at: http://www.osgi.org/Specifications/HomePage. It contains a number of new specifications mostly dealing with device interoperability and configuration, monitoring and management services. As with the previous Residential 4.3 Specification, the OSGi Alliance and the Home Gateway Initiative (HGI) synchronized their work in yearly workshops.
We are proud to present the following new service specifications that are now part of the OSGi ecosystem:

Device Abstraction Layer - The Device Abstraction Layer specification provides a unified interface for application developers to interact with sensors, devices, etc. connected to a gateway. Application developers don't have to deal with protocol specific details. This greatly simplifies the development of applications. This abstraction layer is also the basis to support the integration of semantic technologies in future specifications.

Device Abstraction Layer Functions - The Device Abstraction Layer Functions specification defines a minimal set of basic device operations and the related properties. They can be extended or replaced to cover domain specific scenarios. The set is not closed and can be incorporated with vendor specific functions. There is support for control, monitoring and metering information.

EnOcean Device Service Specification - This specification defines how OSGi bundles can both discover and control EnOcean devices, and act as EnOcean devices and interoperate with EnOcean clients. In particular, a Java mapping is provided for the standard representation of EnOcean devices called EnOcean Equipment Profile (EEP).

Network Interface Information Service - This service specification defines services that provide a standard way for bundles to receive notifications about changes in the IP network interfaces and IP addresses.

Resource Monitoring - The Resource Monitoring specification defines an API for applications to monitor resources consumed by any set of bundles. This includes hardware resources but also covers other resource types as well. Data derived from monitoring enables applications to take decisions on management actions to apply. Resource management actions are mentioned as examples in this specification, including actions on the lifecycle of components, bundles, the framework and the JVM, Java threads, and the raising of exceptions.

Serial Device Service Specification - The Serial Device Service specification defines an API to communicate with controllers, devices and other equipment that is connected via a serial port.

USB Information Category – This specification adds a new Device Access Category for USB devices to the Device Access specification in order to handle, for example, the integration of communication protocols, for example, ZigBee or Z-Wave via USB dongles.


Please share the news, review the specification and give us your feedback.

Andreas Kraft & Kai Hackbarth (co-chairs Residential Expert Group)

Tuesday, August 18, 2015

More jars on Maven Central and JCenter

In my last blog post, I announced the availability of the Release 6 specifications including their companion code jars which were released to Maven Central and JCenter. Those companion code jars were collections of companion code associated with the specification documents. For example, the osgi.enterprise jar contained all the packages for the all the APIs specified in the OSGi Enterprise specification document. Those jars are meant for compile time use only and are not meant for runtime use by being installed as a bundle in an OSGi framework. In fact, those jars now contain unresolvable requirements to prevent their use at runtime.

But what if you want to use the APIs at runtime? To support using the APIs at runtime, OSGi has now made the companion code for individual specifications available as individual companion code bundles. These bundles are now also available from Maven Central and JCenter. So if, for example, you need the Configuration Admin Service API at runtime (and you are not using a Configuration Admin Service implementation bundle which already exports the API), you can get the org.osgi:org.osgi.service.cm:1.5.0 bundle and install it. Each of the companion code bundles is versioned at the version of the specification they are for. Since the current release of the Configuration Admin Service Specification is version 1.5, that is also the version of its companion code bundle.

There are now individual companion code bundles for all of the specifications in the OSGi Compendium, OSGi Enterprise, and OSGi Residential Release 6 specifications:
org.osgi.application-1.0.0
org.osgi.jmx-1.1.0
org.osgi.namespace.contract-1.0.0
org.osgi.namespace.extender-1.0.1
org.osgi.namespace.implementation-1.0.0
org.osgi.namespace.service-1.0.0
org.osgi.service.application-1.1.0
org.osgi.service.async-1.0.0
org.osgi.service.blueprint-1.0.2
org.osgi.service.cm-1.5.0
org.osgi.service.component-1.3.0
org.osgi.service.component.annotations-1.3.0
org.osgi.service.coordinator-1.0.2
org.osgi.service.dal-1.0.0
org.osgi.service.dal.functions-1.0.0
org.osgi.service.deploymentadmin-1.1.0
org.osgi.service.device-1.1.0
org.osgi.service.dmt-2.0.1
org.osgi.service.enocean-1.0.0
org.osgi.service.event-1.3.1
org.osgi.service.http-1.2.1
org.osgi.service.http.whiteboard-1.0.0
org.osgi.service.io-1.0.0
org.osgi.service.jdbc-1.0.0
org.osgi.service.jndi-1.0.0
org.osgi.service.jpa-1.0.0
org.osgi.service.log-1.3.0
org.osgi.service.metatype-1.3.0
org.osgi.service.metatype.annotations-1.3.0
org.osgi.service.monitor-1.0.0
org.osgi.service.networkadapter-1.0.0
org.osgi.service.prefs-1.1.1
org.osgi.service.provisioning-1.2.0
org.osgi.service.remoteserviceadmin-1.1.0
org.osgi.service.repository-1.1.0
org.osgi.service.resolver-1.0.1
org.osgi.service.resourcemonitoring-1.0.0
org.osgi.service.rest-1.0.0
org.osgi.service.serial-1.0.0
org.osgi.service.serviceloader-1.0.0
org.osgi.service.subsystem-1.1.0
org.osgi.service.tr069todmt-1.0.1
org.osgi.service.upnp-1.2.0
org.osgi.service.usbinfo-1.0.0
org.osgi.service.useradmin-1.1.0
org.osgi.service.wireadmin-1.0.1
org.osgi.util.function-1.0.0
org.osgi.util.measurement-1.0.1
org.osgi.util.position-1.0.1
org.osgi.util.promise-1.0.0
org.osgi.util.tracker-1.5.1
org.osgi.util.xml-1.0.1

Note: the Core API is only available in the osgi.core jar for compile time use since the Core API is exported at runtime by the framework implementation and thus should not be exported by bundles.

Sunday, August 16, 2015

Taking Exception

Did you ever look at how we as developers are handling our exceptions? Open source lets us see that we've developed an intriguing number of ways of handling exceptions. Let's take a look at the myriad ways developers handle their exceptions.

First the closeted C developer that was forced to use Java.

  int main(String[] args, int argc) {
    FileInputStream file_input_stream;
    int first_char;
    try {
      if (argc != 0)
        throw new IllegalArgumentException("Not enough arguments");
    } catch (IllegalArgumentException illegal_argument) {
      System.err.format("Exception: " + illegal_argument.getMessage());
      return -1;
    }
    try {
      file_input_stream = new FileInputStream(args[0]);
    } catch (FileNotFoundException e) {
      return ENOENT;
    }
    try {
      first_char = file_input_stream.read();
    } catch (IOException e) {
      try {
        file_input_stream.close();
      } catch (IOException ee) {
        return EIO;
      }
      return EIO;
    }
    if (first_char > 0) {
      System.out.format("first character is %c\n", first_char);
      try {
        file_input_stream.close();
      } catch (IOException e) {
        return EIO;
      }
    } else {
      try {
        file_input_stream.close();
      } catch (IOException ee) {
        return EIO;
      }
      return EEOF;
    }
    return EOK;
  }

Then the testosterone driven developer that basically reasons that if the caller does not want its checked exceptions they better swallow Runtime Exceptions!
  
public void main(String args[]) {
    try (FileInputStream input = new FileInputStream(args[0]);) {
      int c = input.read();
      if (c > 0)
        System.out.format("first character is %c%n", c);
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }

There are of course the persnickety developers that feel that since exceptions are good, more exception classes must be better. They wrap the exception in their own, better, exception, thereby creating a profound stack trace. Their variation looks like:


  public void main(String args[]) throws MeTooException {
    try (FileInputStream input = new FileInputStream(args[0]);) {
      int c = input.read();
      if (c > 0)
        System.out.format("first character is %c%n", c);
    } catch (Exception e) {
      throw new MeTooException(e);
    }
  }

And then we have the financial developer that figured out his productivity is measured by the lines of code he produces, regardless how mindless they are. They are especially vicious combined with the persnickety approach that wraps each exception in their own variation.

  public void main(String args[]) throws MeTooException {
    try (FileInputStream input = new FileInputStream(args[0]);) {
      int c = input.read();
      if (c > 0)
        System.out.format("first character is %c%n", c);
    } catch (FileNotFoundException e) {
      log("File not found Exception");
    } catch (EOFException e) {
      log("File EOF Exception");
    } catch (ClosedChannelException e) {
      log("Closed Channel Exception");
    } catch (ConnectIOException e) {
      log("Connect IO Exception");
    } catch (FileSystemException e) {
      log("File System Exception");
    } catch (FileLockInterruptionException e) {
      log("File Lock Interrupt Exception");
    } catch (InterruptedIOException e) {
      log("Interrupted IO Exception");
    } catch (MalformedURLException e) {
      log("Malformed URL Exception");
    } catch (IIOException e) {
      log("IIO Exception");
    } catch (RemoteException e) {
      log("Remote Exception");
    } catch (ProtocolException e) {
      log("Protocol Exception");
    } catch (SocketException e) {
      log("Socket Exception");
    } catch (SSLException e) {
      log("SSL Exception");
    } catch (SyncFailedException e) {
      log("Sync Failed Exception");
    } catch (UnknownHostException e) {
      log("Unknown Host Exception");
    } catch (JarException e) {
      log("Jar Exception");
    } catch (ZipException e) {
      log("Zip Exception");
    } catch (IOException e) {
      log("IO Exception");
    }catch (SecurityException e) {
      log("Security Exception");
    }
  }
Then we have the 'what checked exceptions?' developer that worked out how they can bypass the type system to throw a non-runtime exception without the caller knowing it:

  public static void main(String args[]) {
    try (FileInputStream input = new FileInputStream(args[0]);) {
      int c = input.read();
      if (c > 0)
        System.out.format("first character is %c%n", c);
    } catch (Exception e) {
      Throw.asUncheckedException(e);
    }
  }
  public static class Throw {
    public static void asUncheckedException(Throwable throwable) {
      Throw. asUncheckedException0(throwable);
    }
    @SuppressWarnings("unchecked")
    private static  void asUncheckedException0(Throwable throwable) throws E {
      throw (E) throwable;
    }
  }

Fortunately we can all hate the ostrich developers that swallow exceptions. Any experienced Java developer knows what it means to trace a problem for hours only to find that some idiot had not reported an error. A better argument for licensing software professionals is hard to find.

  public static void main(String args[]) {
    try (FileInputStream input = new FileInputStream(args[0]);) {
      int c = input.read();
      if (c > 0)
        System.out.format("first character is %c%n", c);
    } catch (Exception e) {}
  }

And then we have the pragmatic developer that realizes that there is no difference between checked and runtime exceptions. Hated by its consumers that are still believing in the myth of checked exceptions:



  public static void main(String args[]) throws Exception {
    try (FileInputStream input = new FileInputStream(args[0]);) {
      int c = input.read();
      if (c > 0)
        System.out.format("first character is %c%n", c);
    }
  }

So in which camp am I? Well, you probably have guessed that I am in the pragmatic camp.  My reasoning is that checked exceptions do not exist, get over it. Bad Idea.

Let me explain why.

I am a firm believer in contract based design and OSGi is imho the best example of this model. In such a world a function call succeeds when the contract is obeyed by the consumer and the provider. However, in the real world there are cases where the contract cannot be fulfilled. Exceptions are for signalling this failure to the consumer. Maybe the input arguments are wrong, one of the downstream calls fails, or a disk goes haywire. The number of things that can go wrong are infinite so it is infeasible to figure out what to do about this failure except to ensure that the state of the current object remains correct.

In all most all cases if anything could be done, then the provider already should have done it. It is crucial to realize that exceptions are therefore by definition not part of the contract. For example, bnd does not see a change in the throws clause as a binary change.

When an exception happens, the consumer could try an alternative strategy but it must never try to understand the reason of the failure for this creates very brittle code. This is especially true in a component world like OSGi where the actual implementations on a call stack can vary. The function succeeds when no exception is thrown, and the function fails when contract could not be obeyed.

Any information in the exception is for the human user to figure out the root problem so that the software contracts can be adjusted to cover the exceptional case or some repair initiated. When an exception happens it is the root cause that the user needs to know. Wrapping exceptions obscures this root cause as we all realize when we see that the root happened 17 exceptions deep and our environment decided to only show 16.

Handling the checked exceptions creates a tight coupling between the consumer and provider for no reason since the consumer should treat all exceptions equal: the contract could not be obeyed, the cause for the consumer is irrelevant. The type, message, and other information of the exception is only intended for the end users to diagnose the problem.

Once you accept this way of thinking about exceptions you realize that checked exceptions were a really bad idea since they give the impression that the consumer should do something specific with them while the best thing is in all most all cases to forward the original exception to the function that is responsible for error handling on that thread.

 I started throwing Exception on all my methods a long time ago. I am often resented for this because users of my code are often still under the illusion that checked exception have utility. So they feel forced to obscure there code in the myriad of ways described in this blog. Well, get over it, the emperor has no clothes.

Since the runtime does not distinguish between checked and unchecked exceptions Oracle could probably provide a compiler annotation that would disable the checking for checked exception. Would be a relieve to also get rid of this nonsensical throws Exception line in my code. To conclude, checked exceptions were a failed experiment. Maybe we should start accepting this.

Peter Kriens
@pkriens

Wednesday, August 5, 2015

Release 6 of OSGi Compendium, OSGi Enterprise and OSGi Residential specifications available

I am pleased to announce that the OSGi Expert Groups and the OSGi membership has approved Release 6 of the OSGi Compendium, OSGi Enterprise and OSGi Residential specifications. They are now available for download from OSGi at http://www.osgi.org/Specifications/HomePage. You can download the specification PDFs, the API JAR files, and also the javadoc.

For you maven users, I also released the API JAR files to Maven Central and Jcenter. The maven coordinates are org.osgi:osgi.cmpn:6.0.0, org.osgi:osgi.enterprise:6.0.0, and org.osgi:osgi.residential:6.0.0. (I just uploaded these, so it make take a bit of time for them to be visible.)

Now time for some vacation... :-)

Avoiding Class Loader Muck with the Weaver. Wonkish

We're quite busy to finalize OSGi enRoute (the 1.0 date is September 1) and one of the significant parts of OSGi enRoute is bnd(tools). A shameful secret of bnd is that it knows a lot about OSGi but its plugin system is actually not based on OSGi. Fifteen years ago when bnd started OSGi was deemed too heavy to run as a command line tool. Ah, the crimes we commit in the name of optimizations.

The bnd plugin system regularly makes me want to pull out my (by now grey) hair;  it confirms my believe that class loaders originated from Dante's ninth circle. Things that work correct without effort in an OSGi environment are yucky in a raw class loader environment. They work, but the solutions are surprisingly fragile without writing another OSGi.

That said, bndlib is used in several OSGi frameworks; it is a real bundle. As can be expected, OSGi is pretty obnoxious when it comes to class loader muck. (Google for OSGi and class loader problems.) The worst part is of course that OSGi is actually right in being obnoxious.

Over time bnd got expanded with several repository types, a resolver, and other plugins. In bndtools (the Eclipse version) we actually install these plugins as bundles but due to reason I will not explain here, they were not always used. So we needed to clean up this mess.

Initially I created a mucky class loader solution that would do the evil thing of searching a class name in all bundles, a.k.a. buddy loading. Evil because you are in lala land, the whole OSGi engine to make sure proper classes are aligned is utterly ignored. You also create wiring dependencies that are not visible to the OSGi framework and this can cause other nastiness. Yuck.

After making my hands dirty and getting it to work I discussed the solution with BJ Hargrave. He, however, came up with a very interesting solution: the Weaving Hook.

The Weaving Hook is normally used to do byte code weaving. I am not that crazy about weaving because it is hard to make it work in an open and shared environment like OSGi. However, in this case it was actually quite easy and useful. Let me elucidate you.

When you change the byte codes of the classes, you need to add imports for any new dependencies. However, nothing in the Weaving specification forces you to weave, you can also use it to just add imports any bundle you want.

So this is what we ended up doing. In bndtools (or any OSGi environment) we register a Weaving Hook service. The hook tracks bundles that can provide plugins (They opt-in with an attribute on an exported package) and then adds these packages to the bndlib bundle as new imports.

This solution worked the first time and provided us with all the OSGi goodies. For the OSGi Framework it looks like the bndlib bundle now imports the plugin bundles. No more class loader yuck!

Peter Kriens