OSGi DevCon/EclipseCon 2011, Did you Register Yet?

This is the last week you can register with the early registration discount, so you better get your socks on! And you should register because we have been able to build (again) a very interesting program. This years we even had BJ Hargrave make a video to convince us to pick his presentation.

A first this year is an introduction to OSGi by yours truly. Though virtually all developers on EclipseCon are using OSGi, there are still many developers that have no good working knowledge of what OSGi really is. So if you want to become enlightened in 20 minutes, come to the Introduction to OSGi.

If those 20 minutes of introduction are not sufficient (can't imagine though), then you might want to try out one of the tutorials. We have the following tutorials:

* Developing Apache Aries Enterprise OSGi Applications, by Graham Charters (IBM), Jeremy Hughes (IBM).
* Developing OSGi-enabled Java EE Applications with GlassFish
* Apache ACE in the Cloud, by Marcel Offermans (Luminis), Karl Pauls (Luminis)

Even though the inside joke in the OSGi expert groups is that we always start with a telegraph pole but end up with a tooth pick, we've actually been quite busy inside the OSGi. We added lots of new things to the new specification. BJ Hargrave (IBM ) will elucidate you know about all the new gadgets in the OSGi 4.3 Technical Update. There is also an Update on the OSGi Enterprise Expert Group by David Bosschaert (Redhat) and Tim Diekmann (TIBCO), who are the co-chairs of the Enterprise Expert Group.

OSGi would not be OSGi if we did not have our eyes on the cloud. Richard Nicholson (Paremus), who happens to be our new OSGi Alliance President, will inform you about progress in the OSGi Cloud workgroup as well as efforts in his company in his OSGi and Cloud presentation.

If you want to be really entertained then the 10 signs you're doing OSGi wrong by
Jeff McAffer (EclipseSource ) and Paul VanderLei (Band XI International ) is always highly recommended. Two years ago I almost had tears in my eyes and I actually also learned a few things.

One of the (in my eyes then) most undervalued parts of the OSGi service specifications is Declarative Services (DS). DS is very lightweight, takes all the pain out of using services, and is surprisingly powerful. And it is also simple to use (especially with the annotations). If I had to do the OSGi framework all over again, I sure would fight for including DS in the framework. Felix Meschberger (Adobe) is the Apache Felix implementer of DS and will tell give you the Declarative Services - Dependency Injection OSGi Style presentation.

In the big enterprise world there is Virgo from the Eclipse Foundation, Aries from Apache, but there is a lesser known third player in the distributions that is slowly building up momentum: Apache Karaf. Guillaume Nodet (Progress) will tell you more about about Karaf in his Building and deploying OSGi applications using Apache Karaf presentation.

There are many more good talks that are all too good to miss, take a look and select!

Oh, yes. Do not forget that we have the OSGi BoF/Reception Tuesday, 19:30. The OSGi Alliance will spring for drinks. You can attend this event even if you've not registered for OSGi DevCon/EclipseCon so no excuse to not be there when you're in the bay area.

Hope to see you in Santa Clara next month, do not forget to register!

Peter Kriens

bndtools hackaton

I've talked with a few people that are using bndtools and there is some interest to spent some time hacking on bndtools, Neil Bartlett's Eclipse plugin for OSGi development. The tool is getting traction and people want to extend the tool in different directions. So I will be organizing a hackaton for bndtools on OSGi DevCon/EclipseCon on Wednesday night (March 23, Santa Clara). I'll make sure some of the right people are there and that we will have a place to hack. If you want to participate, let me know at Peter.Kriens@aQute.biz.

Peter Kriens

Anecdotes of Frustration

I am doing a little survey of what moments of frustration we're having as developers of software that relies heavily on external software (both open source, proprietary, and in-house but from other departments). I am looking for anecdotes how you, or others, wasted time and money in your normal developer work flow because something went wrong with an external software product. I am not looking for theories, I'd like to hear real stories, a.k.a. anecdotes.

To ease the processing, I'd like to see the following format:

1. The situation/environment
2. What were you trying to accomplish
3. How did you do it
4. What went wrong
5. How to prevent it next time
   
6. What was the economic cost of the mishap

For example:

1. Developing a web site for a small workshop
2. We were trying to provide textual search to the past orders
   
3. We integrated open open source X into our website and then suddenly we got Null Pointer Exceptions in our web layer, which was not at all connected.
4. It turned out that some of the packages were repeated in product X and because X was first on the class path the web layer started to use X's packages and they were an older version.
5. Reverse the order on the class path
6. We lost about a man day figuring it out

Another:

1. Creating an order taking application for the iPad
2. We were field testing the application with potential customers a to prepare for the launch at a large conference that was held within two weeks.
   
3. One of the field testers contacted us and said that he had noticed we were using a GPL package to communicate with our back-end. He was a hobby developer and asked us for our sources so he could review the source code. However, our code contained trade secrets and we were not willing to provide the source base to him. He then threatened to contact the EFF.
4. We worked three day straight to replace the GPL code in our product, which we actually had not been aware that it was in our product.
5. We lost about 20 man days and got a few extra gray hairs
   

From the examples it should be clear I am not looking for OSGi centered problems; I am mostly interested in problems related to the handling of external products. Keywords:

1. Legal, problems with licenses, patents.
   
2. Versioning, surprising mismatches
   
3. Deploying
4. Escrow
5. Java
6. Security
7. ...
   

You can send these anecdotes to anecdotes@aQute.biz. I would appreciate it if you could ask others as well. You can of course send multiple anecdotes but please use different emails to simplify the processing. I will not use the anecdotes in any way that can be connected to the companies or individuals, sending them anonymous or without actual company or individual names is ok.

If I receive a substantial number of these anecdotes then I will process them and write a number of blogs about common problems that everybody seem to have. This of course will be a fine input to the OSGi specification process.

Looking forward to your anecdotes, keep them coming!

Peter Kriens

Error Messages

The increased adoption of OSGi is fantastic but it also means more and more people start using it without properly understanding the technology. These people tend to get upset when things do not immediately work the way they expect and blame it on the technology. The clarity of the error messages are therefore an important aspect of the usability. If they are clear and simple to understand people will be easier to get along with the technology. However, dependency graphs with constraints are notoriously difficult to explain in textual messages.

In this vein Richard S. Hall (Apache Felix) contacted me to discuss a very interesting presentation he is working on. Going through the presentation I joked that I thought the error messages listed in the presentation were completely unreadable. Richard is the conscience of the OSGi Alliance and very much driven to keep things simple and to provide as much help as possible to end users. So that joke hurt. That started a long thread about error messages. We mostly focused on this message:

Constraint violation for package 'bar' when resolving
module 11.0 between existing import 8.0.bar BLAMED ON
[[11.0] package;
(&(package=bar)(version>=1.0.0)(!(version>=2.0.0)))]
and uses constraint 10.0.bar BLAMED ON [[11.0] package;
(&(package=foo)(version>=1.0.0)(!(version>=2.0.0))),
[9.0] package;
(&(package=boz)(version>=1.0.0)(!(version>=2.0.0)))]

I've noticed it before but I have a huge deficiency; a blurp of unformatted text like this message is for me completely unreadable. When I run into such a amorphous blurb, I need to paste it in an editor and put some visual structure in it to grasp the contents. So lets format it:

Constraint violation for package 'bar' when resolving module 11.0
between existing import 8.0.bar
  BLAMED ON [[11.0] package;
    (&(package=bar)(version>=1.0.0)(!(version>=2.0.0)))]
    and uses constraint 10.0.bar
  BLAMED ON
    [[11.0] package;
    (&(package=foo)(version>=1.0.0)(!(version>=2.0.0))),
    [9.0] package;
    (&(package=boz)(version>=1.0.0)(!(version>=2.0.0)))]

Hmm. This seems about a uses constraint violation that is caused by a module that can see the package bar through two different paths in the network of dependencies. That is, the class space for package bar in module 11 contains two definitions of package bar. One comes from a module 11 importing package bar directly from module 8 and the other is a chain of modules. Module 11 seems to import package foo from module 9, where module 9 also exports boz that has a uses constraint on package bar. Module 9 seems to import package bar from module 10. Phew! After some talking with Richard, I came to the conclusion that we have this problem:



This diagram uses the syntax used in the OSGi specifications. A white box is an import, a black box is an export, a yellow box is a bundle, and the white cylinder in the package boxes is the uses constraint.

In this case, module 11 can receive bar objects that are loaded by module 8 as well as module 10, not good. Clearly, any sane person will concur that uses constraints are good because they detect run-time errors early and allow a resolver to prevent these run-time errors from happening. However, how should we report such a complex problem in a log message?

After some sparring Richard came back a few days later with an alternative proposal:

Module com.acme.eleven[11.0] can see package 'bar' from
com.acme.eight[8.0] and com.acme.ten[10.0]. The first
is due to com.acme.eleven[11.0]'s requirement
(&(package=bar)(version>=1.0.0)(!(version>=2.0.0)))
that is satisfied by com.acme.eight[8.0]. The second
is due to the dependency chain from
com.acme.eleven[11.0]'s requirement
(&(package=foo)(version>=1.0.0)(!(version>=2.0.0)))
which is satisfied by com.acme.nine[9.0], that has a
requirement
(&(package=boz)(version>=1.0.0)(!(version>=2.0.0)))
which is satisfied by com.acme.ten[10.0] that uses
package 'bar'.

Ouch, another blurb. The problem is that Richard is someone that is extremely good in parsing these text blurbs (he still prefers vi over Eclipse), but I suck at it. I need a visual representation ! However, this message did improve the original if properly formatted:

Module com.acme.eleven[11.0] can see package 'bar'
 from com.acme.eight[8.0]
 and  com.acme.ten[10.0].

The first is due to com.acme.eleven[11.0]'s requirement
    (&(package=bar)(version>=1.0.0)(!(version>=2.0.0)))
that is satisfied by com.acme.eight[8.0].

The second is due to the dependency chain from
  com.acme.eleven[11.0]'s requirement
    (&(package=exporter2.woz)(version>=1.0.0)(!(version>=2.0.0)))
which is satisfied by com.acme.nine[9.0],
that has a requirement
    (&(package=foo)(version>=1.0.0)(!(version>=2.0.0)))
which is satisfied by com.acme.ten[10.0]
that uses package 'bar'.

This clearly is in the right direction but the lack of formatting and the, in my opinion, overly verbose text is still not to my liking. Personally, I like conciseness, it makes it easier to see the problem; verbose template text has a tendency to shut my brain down. However, Richard argues that a lot of people reading the text have no idea what OSGi is and need the verbosity. Not sure, maybe there needs to be an explanation of the problem without the actual arguments or a web page that details what is going on. If you have no clue, I think it is too ambitious to expect an understanding from a mere error message. In the old, long gone days of computing, all error messages had numbers that one could look up, maybe we should start doing that again.

So Richard has taken two shots at it. Now let me provide a variation on the error message that I think would be best. As a strong believer in leveraging the visual brain I think formatting and white space are not a luxury but a necessity. So my message is highly visualized:

[E128] Uses constraint violation in com.acme.importer[11.0] on package 'bar'

A bundle can see classes from the same package via multiple class loaders.
To prevent this case (which results in Class Cast Exceptions), an
exporter package  can declare what packages it uses with the uses:
directive. The framework transitively detects cases where a bundle
can be confronted with objects from the same package but coming
from different class loaders. This error indicates that no resolution
could be found and this constraint violation is one of the problems.

In this case com.acme.eleven (module 11.0) sees package 'bar' through
the following 2 dependency chains:

1)  com.acme.eleven[11.0]
    import: (&(package=bar)(version>=1.0.0)(!(version>=2.0.0)))
     |
    export: {package=bar, version=1.2}
  com.acme.eight[8.0]

2) com.acme.eleven[11.0]
    import: (&(package=foo)(version>=1.0.0)(!(version>=2.0.0)))
     |
    export: {package=foo, version=1.5, uses:=boz}
 com.acme.nine[9.0]
    import: (&(package=boz)(version>=1.0.0)(!(version>=2.0.0)))
     |
    export: {package=boz, version=1.5, uses:="bar"}
    export: {package=bar, version=1.2}
 com.acme.ten[10.0]

Richard has not seen this example but he will undoubtedly react. And we would you to react as well because we just do not know what is best. Richard is willing to change the error messages of the Apache Felix framework if people come up with a better solution than we have now.

Peter Kriens

P.S. Richard became a father yesterday of his second child: Brady William Hall. Congratulations!

Accidental Complexity and Modularity

This week I watched a discussion unfold that seems quite typical for the pain many people feel when they port code to OSGi. The problem started with the infamous class not found exceptions that happen when you try to dynamically load classes from the big void. After a couple of iterations where context class loaders, buddy loading, and other hacks were discussed things seem to get worse and worse until someone asked: "Why do you actually load all those classes dynamically?" After a silence, the somewhat reluctant answer was that the code looked a bit over the top in this area ...

If I would order a new construction on my house I would be pretty surprised (upset would probably a better word) when the constructor would spent a substantial portion of his paid time experimenting with different building techniques to make his life easier. Fortunately, this is not very likely because I can understand what construction workers are doing (most of the time) and the skills they possess are not the skills required to optimize their tools and techniques. For example, painters are unlikely to insert thousands of logging statements in their paint (20110101 10:22.03223 INFO: Latex: Paint drying from brush@3128AC at position 3121912@121212C ?) or develop their own brushes. We are different: our customers usually have no clue how we do it and our tools are built with the same skills as the applications we make. We are the only large industry with an ailing tool market.

I do know from experience how easy it is to go overboard by attempting to simplify, believe me. The most gratifying code is that annotations library that reduces your domain code from a thousand repetitive lines to that 1 line of code (with 99 annotations). Unfortunately, simplifications are in the eye of the beholder. Newcomers must still learn the annotations to understand what the code is doing. If (too) many developers make such "simplifications" we moved the complexity from repetitive simple code to a very hard to understand (and maintain) system. Still, it is fun to do, much more fun than learn what is already out there.

The last few years I've started to see more and more how "simplifications" locally can make things more complex globally. Tricks and hacks that seem to work wonderfully well for a problem tend create brittle and hard to maintain systems. Not because they are inherently bad but because the large number of one time rules they introduce makes the overall systems hard to understand. This is increasingly becoming a problem because today we build our systems out of many open source components. Unfortunately, each open source component has a tendency to bring its own bag of tricks and hacks. The big problem we face today is how to combine these components into a coherent application and not having to learn and fight the combined accidental complexity. That is, complexity is no longer a qualative problem but it has become a quantitative problem.

Systems build from collaborating components is an area that is eminently addressed by OSGi. But one should be warned that there is no free lunch. To achieve OSGi's extensive benefits requires that modules hide their tricks and hacks inside the module boundaries and use the OSGi primitives to collaborate with their peers. Any hack that assumes global visibility (as does dynamic loading) is bound to destroy these hard won benefits, if not today then surely tomorrow.

So the bad news is that OSGi is disruptive, you cannot take your existing code base, sprinkle some OSGi pixy-dust on it and then suddenly all your problems are magically gone. (If you have such a pixy-dust, lets talk, I have some Facebook shares to sell.) This is not related to how OSGi was designed, this is related to the fact that OSGi is based on the principles of strong modularity. There is no silver bullet to modularize your code base overnight because modularity is a design principle, not a technique you can haphazardly apply. If you read David Parnas' seminal paper about modularity you see exactly what I mean; he decomposes a system in different ways but using the principles of modularity he got an overall simpler and much more resilient solution.

When the first combustion engine cars were developed they looked like horse carriages as today most enterprise applications look like Java EE applications. In the last few years the OSGi has spent a lot of effort to make it easier for Java EE applications to use to the OSGi highway; moving your WAR to OSGi has become trivial with the WAB specification. However, we should not forget that to go at full speed leaving your design unchanged is at your own peril (and cost).

Peter Kriens

JCP Is Dead?

After 4 years it seems Oracle broke Apache's last straw, they left the JCP. Together with the independents. Sun originally promised to make Java a free platform, and I believed them. Guess I am naive but Sun's track record was pretty good at the time. Oracle originally objected to the fields of use restrictions before they bought Java but changed their mind now they're the owner and I guess IBM was bought off. It is surprising how Oracle seems to be aiming straight at their foot, do they regard the open source community as a minor aspect of Java? Is this (long gone after Android & iPhone) mobile market worth the cost of alienating their most fervent supporters?

It is a mess because I do not want to choose. I like the big corporations because the give credence to the Java platform and provide numerous resources to work on the platform. I do not think Java could have been created in an open source environment of volunteers. However, I also like Apache and other open source foundations because they provide diversity and innovation beyond corporate interest. And then again I do not like big corporations because politics often trump best technical solutions and I do not like open source communities because they too often mix idealism and work. Working together they keep each other in a good balance.

What is needed, and what was clearly promised by Sun all those years ago, is an independent organization where industry and open source community can meet and standardize without any one party being special. The OSGi is such an organization. An organization that has a lot of experience with Java specifications as well. Maybe the OSGi Alliance needs to make some organizational changes but it is and independent and experienced place to standardize where there currently is no other place like it.

It is sad to see how a closed eco-system like Apple's iPhone and iPad is thriving and how we are going to waste a lot of energy in the next years because we direct that energy on destroying and not creation. The tragedy of the commons revisited.

Peter Kriens

P.S. Neil Bartlett and I will hold another master class int the US Bay Area on January 18-21. The first one was a very good experience (also according to the participants!), though intense.

Modularization

Reading blogs about Java modularization gives me the sincere impression that (too) few people understand what modularization really is all about and how OSGi implements it.

Modularization is a fractal, it is a process that has happened several times over the last 50 years. The reason for these disruptive process was that it allowed us to scale software development to the next level. Assembly was a clear form of modularization where the bits were hidden from us humans. Functions with local variables were the next step. The industry then went on with modules (functions + data) and objects that are basically modules that could be instantiated. Java added packages, which grouped classes, again a form of modularization.

What these efforts did was encapsulate, they created a boundary between inside and outside and controlled the internal name space by importing and exporting. You could import from the outside world and you could control what was exported. What you did on the inside was your problem, it was not visible to the outside world. For example, it is impossible to detect that the name of a local variable used in a function. This kind of encapsulation is important, it means what happens in the inside is invisible to other parts of the system. Invisibility is good, you can't be wrong about something you do not know!

Except for OSGi, other modularity efforts seem to forget about this crucial encapsulation required for the next modularization step. The efforts I've seen so far focus on accessibility. The difference between accessibility and visibility is that accessibility requires a single name space. Where in OSGi each bundle (the module) has its own name space as we've done in the past when we took a modularization step, in other forms this aspect is ignored. All other proposals I've seen so far assume that there is a single name space for classes.

A single name space works well for small to medium systems. However, today we are building systems that consists of a large number of subsystems. Forcing a single name space on these large systems is in the long run asking for problems. The clearest example is if you have two subsystems that depend on different versions of a third subsystem than you're in trouble. Unless you build on OSGi.

Peter Kriens