The KISS principle (acronym for "Keep It Simple, Stupid") states that design simplicity should be a key goal and unnecessary complexity avoided. It serves as a useful principle in a wide array of disciplines, such as software development, animation, photography, engineering, and strategic planning.
Today Google Alerts dumped me into two discussions where OSGi seems to be on the wrong side of the divide regarding the KISS principle. The first is an extensive debate in the dev@tomcat.apache.org mailing list, the second is the JSR 277 mailing list.
The Tomcat developers have been asked to osgify their popular web server. This resulted in a discussion that was very interesting to read. Several people seem to have a disdain for the OSGi zealots (which I guess is me) but there seems to be a general uneasiness with this request. However, my key trigger for this blog was that in this discussion I noticed that several people regard OSGi as overly complex and heavy. The JSR 277 discussion raised the same problem from another direction. From the JSR 291 interoperation discussion: "I wanted something in Java that was simpler and cleaner that OSGi-users could leverage if they wanted.". Is OSGi already ossified?
The key problem to decide this question is, as so often the case, requirements. Is Java complex? Hard to answer, if you want to write a "Hello World" it is humongous overkill. If you are writing a large scale web application it is nice to have a solid foundation. The key phrase in the definition of KISS is unnecessary complexity. Complexity is in the eye of the beholder.
First, the OSGi API is very small by any measure. There is only one mandatory package that only has 26 class files, including the security and exception classes. And you can already leverage OSGi without writing Java code. Can you call it intrusive when you can turn a JAR into a bundle by adding one or two manifest lines? All the complexity people complain about is optional. That said, I do agree that even optional things create some kind of conceptual complexity. So maybe there is a lot of cruft in the options?
Let us take a feature like import packages which is often considered hard. Most dependency mechanisms like Maven, Ivy, and also the original JSR 277 are based on the Require-Bundle concept. I.e. you declare a dependency on the wrapper.
Wrapper based dependency models are simple to understand for us humans. It would work actually very well in practice if JARs would be highly cohesive and not change their constituent packages over time. Which raises the question: Do you want the woman, or the dress?
We developers have discovered the wonders of refactoring and things do change. OSGi is a module system, it is about being able to change software while not breaking the deployments. Though import packages might be harder to understand initially than Require-Bundle, Import-Package does create more resilient systems because it minimizes coupling. That is why people that have used both systems usually decide to standardize on Import-Package (and bnd makes this quite easy).
Similar to the
uses: directive. In the Tomcat discussion it was stated that the uses directives made the manifest unreadable. It was much nicer to create a manifest by hand, it looked better. I do agree from an aesthetic point of view but who reads a manifest? The manifest is intended to be read by the OSGi Framework. The uses directive is an incredible important concept to maintain classpace consistency when the same class can be loaded multiple times through different class loaders. If you have lived in a world where this model is supported, you understand that uses have a very high value because they minimize runtime problems. I do not say that OSGi has no unnecessary complexity at all. Obviously, the specification has been evolving over almost ten years. We have to tried to keep it is simple as possible, but we are as fallible as all of us. However, requirements have always come first in OSGi and fulfilling requirements just forbid you to make things too simple because they force you to understand the problem.
Obviously the key question is, are many of these requirements highly specialized cases not useful for the majority. Or are they just solving problems that awaits anybody that ventures into the wonderful land of modularity?
Peter Kriens
First: I'm a big fan of OSGi technology. I've recently brought it into my organization and now use it just about everywhere. Nevertheless, I still feel that configuration is the Achilles's heel that complicates OSGi. It's what made my team fight it at first and is still the issue that causes the most headaches.
ReplyDeleteFirst, let's be honest: only trivial bundles require "one or two manifest lines." Real-world scenarios demand a tool like bnd to dig through our code and figure out the complexities of required dependencies, exported packages, etc., and most of my manifests wind up over fifty lines long.
Bnd is an excellent tool, but it still requires integration into the build process. In our team environment, at least, that's much, much easier said than done, and is complicated a hundredfold by the vast and disparate wasteland of Java build tools that were never designed to handle corner cases. (For starters, try integrating projects using bnd with Eclipse RCP applications.) It's hardly the fault of OSGi or bnd, but it's still something that haunts me daily.
One of the primary reasons for this difficulty is the reliance on the Java manifest. Simply put: the file is used for too many things. Code signing, OSGI configuration and a plethora of extensions all crowd together to form what we would call very low cohesion if we were describing a class. I've wished from the beginning that OSGi used a separate configuration file.
The ramifications for this are surprisingly widespread. For instance, I once tried to create an annotation library that could be used to automatically generate OSGi configuration. Unfortunately, even though I could create a correct manifest (thanks in no small part to your bnd library), every single build system required a separate and distinct step to get that manifest incorporated into the final archive. Throw in even one more unexpected step - code signing, for instance - and the whole thing fell apart. I have since abandoned the idea.
The same goes for extension information. Since I find Declarative Services to be one of the most useful facets of OSGi, I wrote (and regularly use) an annotation library that can automatically generate Declarative Services configuration files. With the Java 6 annotation processing API, they can even be generated automatically without any intervention from the developer. But the spec still requires a Service-Component line in the manifest, and the only way to do that turns out to be manually. Refactor one class and - kaboom! - the whole thing breaks. I've been thinking of creating my own version for a while now that is identical to the Declarative Services specification but, rather than relying on the Service-Component attribute of the manifest, simply looks up every XML file in, say, a META-INF/SCR directory.
It seems to me that OSGi was built by brilliant individuals that understood the best practices of software engineering quite well. Nevertheless, the configuration process neglects so many of those same practices. There is low cohesion, with too many unrelated things thrown into the manifest. There are tight coupling and duplication, with class names repeated in un-refactorable (any chance that is a word?) text files. Etc.
Forgive me if I've gone on a bit of a tangent or hijacked your blog - and feel free to edit or remove this post if you so desire. My message to you is this: OSGi is really, really close. Simplify configuration with the same best practices that you have already obviously mastered and R5 will be fantastic.
I do agree that configuration is weak in OSGi, and as a result others have provided so many different ways to configure that the picture in this area has become confusing. However, I think your use case is very illustrative. There are just too many options and it will be impossible to align all those requirements.
ReplyDeleteThe problems you signal is a build problem. Though we can do some things (we are talking now about allowing a Service-Component header with wildcards, and maybe we should have a usable default), build systems are clearly out of scope for OSGi right now.
Interestingly, I like having things in the manifest instead of the plethora of XML files one find in other systems. The manifest describes the contents, the fact that a bundle can contain so many different content type does not make this uncohesive imho because it is all metadata.
You can instruct Eclipse to refactor class names in text files as well. Anyway, any configuration with a text file has a similar problem.
I have personally started working on a build system because I recognize many of your issues. However, after already spending quite a bit of time I started to realize that even when I would build the best possible build system in the world, it would be unlikely I could ever get paid for it because it is hard to compete with free products like maven and ant.
Anyway, if you have complexity, it is bound to show up somewhere. The point of my blog was to talk about unnecessary complexity. Do you think all your problems would disappear if we had had a separate configuration file? In that case, I'd love to see an example of this.
Peter Kriens
Thanks for such a great blog.
ReplyDeleteAt the moment, I'm using an in-house Maven plugin to create my manifest. It simply adds Require-Bundle entries for nearest (traverses dependency tree) bundle dependencies. The nearest non-bundle dependencies are embedded and added to the Bundle-ClassPath.
----------
e.g.
Bundle-ClassPath: .,
dependency/xml-apis/xml-apis/1.0.b2/xml-apis-1.0.b2.jar,
dependency/xom/xom/1.1/xom-1.1.jar
----------
The embedded jars are then programmatically added as resources to the MavenProject instance. Export-Package is configured with regex include/exclude filters.
Anyway, I'd really like to incorporate Bnd in my plugin so I can eliminate the use of Require-Bundle. During that transition, I'd also like to get rid of as much manual configuration as possible. I plan to use annotations and apt wherever possible to dynamically configure Bnd. I know you've expressed some reservations about annotation-based manifest configuration in the past [1].
It would really help me if you could give some guidance:
Is something like @NonPublicAPI acceptable for filtering out packages/classes? The annotation would be @Documented so non-OSGi consumers could understand the restriction.
Which parts of configuration could/should be annotation driven?
Can you direct me to any existing projects which might help me?
Best Regards,
Luke
[1] - http://www.aqute.biz/Blog/2007-05-21
Peter,
ReplyDeleteYou're absolutely right: so many of the problems stem from build systems. I'll also agree with you that there isn't enough money in build systems to buy half a sandwich. That said, if you have some ideas for an improved Java build system, I would eagerly contribute in any way possible on a volunteer basis. It may not pay well, yet I am still a young man and plan to be using Java for quite some time; a better build system is number one on my wish list.
Getting back on topic, there have been two times that stand out when I've wished that OSGi used a different configuration file than the manifest. The first was when I started using OSGi and was trying the most basic tutorials out there. Nothing worked. The problem was that I was using Maven and placing the manifest in the src/main/resources/META-INF folder, which is supposed to get copied verbatim to the final artifact. However, Maven will automatically overwrite that manifest with it's own! Default NetBeans project? You have to know to switch to file view and edit the pre-existing manifest in the root folder. Eclipse? The manifest needs to be in a META-INF folder under the root. Of all necessary complexity, a novice OSGi developer shouldn't have to fight that battle.
The second time was when I tried to create an annotation-driven OSGi configuration tool, as mentioned previously. Again, the build system, coupled with limitations of annotation processors, gets in the way. Generating configuration into non-manifest files, however, is a snap.
Ultimately, each build system and IDE treats the manifest as a special file. And yes - that means that it really is the fault of the build system. Nevertheless, this is the world we live in. Moving OSGi configuration out of the manifest would be a big step towards decoupling it from the surrounding environment. It's the same basic principle that OSGi promotes so well: decouple separate parts in order to better manage the unexpected.
To answer your final question directly: no, I don't think every problem would disappear just by moving configuration to a different file, I only think it would be one helpful step. For what it's worth, I think OSGi has done a great job managing complexity, and I certainly prefer working with the current format over the mounds of XML configuration found elsewhere.
Luke,
It seems like we are coming from a similar place with annotations. I started a project a while ago on Java.net and published my code there (https://naked-osgi.dev.java.net/). The goal was to configure exports, activators, bundle name, version and Declarative Services using annotations and bnd-style parsing for imports. As you'll note from the project page, however, I've since abandoned the concept due to limitations of the Java annotation processing API and the aforementioned troubles with Java build systems. Currently, I'm trying to work with a more limited scope, generating Declarative Services configurations, and have had much greater success. If you are interested in building off what I have done or just picking my brain, feel free to contact me via the project owner email link on the project page. (Fair warning: the code base is not in the best of shape since I have changed directions so many times to try and get around limitations. Code cleanup is my primary goal before a 1.0 release, followed closely by documentation.)
- Evan