Accidental Complexity and Modularity

This week I watched a discussion unfold that seems quite typical for the pain many people feel when they port code to OSGi. The problem started with the infamous class not found exceptions that happen when you try to dynamically load classes from the big void. After a couple of iterations where context class loaders, buddy loading, and other hacks were discussed things seem to get worse and worse until someone asked: "Why do you actually load all those classes dynamically?" After a silence, the somewhat reluctant answer was that the code looked a bit over the top in this area ...

If I would order a new construction on my house I would be pretty surprised (upset would probably a better word) when the constructor would spent a substantial portion of his paid time experimenting with different building techniques to make his life easier. Fortunately, this is not very likely because I can understand what construction workers are doing (most of the time) and the skills they possess are not the skills required to optimize their tools and techniques. For example, painters are unlikely to insert thousands of logging statements in their paint (20110101 10:22.03223 INFO: Latex: Paint drying from brush@3128AC at position 3121912@121212C ?) or develop their own brushes. We are different: our customers usually have no clue how we do it and our tools are built with the same skills as the applications we make. We are the only large industry with an ailing tool market.

I do know from experience how easy it is to go overboard by attempting to simplify, believe me. The most gratifying code is that annotations library that reduces your domain code from a thousand repetitive lines to that 1 line of code (with 99 annotations). Unfortunately, simplifications are in the eye of the beholder. Newcomers must still learn the annotations to understand what the code is doing. If (too) many developers make such "simplifications" we moved the complexity from repetitive simple code to a very hard to understand (and maintain) system. Still, it is fun to do, much more fun than learn what is already out there.

The last few years I've started to see more and more how "simplifications" locally can make things more complex globally. Tricks and hacks that seem to work wonderfully well for a problem tend create brittle and hard to maintain systems. Not because they are inherently bad but because the large number of one time rules they introduce makes the overall systems hard to understand. This is increasingly becoming a problem because today we build our systems out of many open source components. Unfortunately, each open source component has a tendency to bring its own bag of tricks and hacks. The big problem we face today is how to combine these components into a coherent application and not having to learn and fight the combined accidental complexity. That is, complexity is no longer a qualative problem but it has become a quantitative problem.

Systems build from collaborating components is an area that is eminently addressed by OSGi. But one should be warned that there is no free lunch. To achieve OSGi's extensive benefits requires that modules hide their tricks and hacks inside the module boundaries and use the OSGi primitives to collaborate with their peers. Any hack that assumes global visibility (as does dynamic loading) is bound to destroy these hard won benefits, if not today then surely tomorrow.

So the bad news is that OSGi is disruptive, you cannot take your existing code base, sprinkle some OSGi pixy-dust on it and then suddenly all your problems are magically gone. (If you have such a pixy-dust, lets talk, I have some Facebook shares to sell.) This is not related to how OSGi was designed, this is related to the fact that OSGi is based on the principles of strong modularity. There is no silver bullet to modularize your code base overnight because modularity is a design principle, not a technique you can haphazardly apply. If you read David Parnas' seminal paper about modularity you see exactly what I mean; he decomposes a system in different ways but using the principles of modularity he got an overall simpler and much more resilient solution.

When the first combustion engine cars were developed they looked like horse carriages as today most enterprise applications look like Java EE applications. In the last few years the OSGi has spent a lot of effort to make it easier for Java EE applications to use to the OSGi highway; moving your WAR to OSGi has become trivial with the WAB specification. However, we should not forget that to go at full speed leaving your design unchanged is at your own peril (and cost).

Peter Kriens