Monday, December 12, 2011

OSGi DevCon March 2012, Reston Virginia

After a slow start we suddenly got a lot and very good submissions for the OSGi DevCon 2012 in Reston Virgina. Though this made the life of the program committee harder (thanks to BJ Hargrave (IBM), Neil Bartlett (Paremus), David Bosschaert (JBoss), and Christer Larsson (Makewave)!) it resulted in a fabulous program. There is really no excuse for not visiting this premier OSGi conference in the US considering we're also throwing in a free registration for Agile Connected and EclipseCon which are co-located!

You can find all the submissions here: http://tinyurl.com/osgidevcon2012selections

There really is a lot of interesting stuff:
  • Apache Felix Web Console - Web Based OSGi Framework Administration — The Apache Felix Web Console has been created out of a need to remotely administer an OSGi Framework. This administration includes maintenance of bundles, editing Configuration, and introspecting ...
  • Best Practices for (Enterprise) OSGi applications — Since the first release of the OSGi Enterprise specification in March 2010 the use of OSGi in the enterprise has increased dramatically. Moving traditional Java EE applications to an OSGi stack is...
  • Dynamic RIAs with Equinox and Vaadin — When you want to use OSGi on the server-side, there are only a few options when it comes to dynamic and modular UIs. Besides RAP (Rich Ajax Platform), Vaadin is another UI toolkit that integrates...
  • Experiences from Building the Fastest OSGi Container on the Planet — What is the fastest OSGi container? Felix or...
  • Liberate your components with OSGi services — Converting any large application to be OSGi based is a difficult and complex process. Many projects find the fences that OSGi put in place puts insurmountable barriers in the way of success. Many...
  • Mastering OSGi with Ease — This tutorial is a compressed version of the renowned Masterclass on OSGi. Its goal is to take you into areas of OSGi that are extremely useful, but seldom discovered through independent...
  • Micro Services in JavaScript — Although modularity concepts from OSGi might not map cleanly to programming languages other than Java, the ideas around OSGi Micro Services might be universally applicable. In this talk we will...
  • Moving the Guidewire platform to OSGi — Guidewire Software builds advanced applications for the insurance industry. With over a hundred customers in a dozen countries, including global giants like AXA, Geico, and Tokyo Marine, our...
  • OSGi in the cloud - quo vadis? — Modularity as in OSGi solves a major issue with architecting elastic applications for the cloud. Cloud resources are inherently dynamic in their nature and undergo frequent changes either due to...
  • Smart Home Mashups: A New Application Opportunity — The number of smart devices in the home is exploding. These devices no longer just include TVs, Bluray players and game consoles. They now include door locks, thermostats, refrigerators, washer...em—
  • What's new in the OSGi Enterprise Release 5.0 — The Enterprise OSGi Release 5.0 will be generally available at EclipseCon/OSGiCon 2012. A number of significant enhancements have been made since the previous release, ...
The details of the schedule can be found from our the OSGi DevCon home page. We're planning an OSGi BOF where we will be presenting our new (then still draft) specifications and providing some really interesting news.

REGISTRATION FOR THE CONFERENCE IS NOW OPEN.
Registration for OSGi DevCon 2012 and EclipseCon 2012 is open and you should book by Dec 31, 2011 to secure the best price.  Employees of OSGi Members are also entitled to an extra discount by using the coupon code: OSGI

If you come for the OSGi DevCon conference please do not forget to check that OSGi is of your interest during the registration since we found out this year that this is actually important for future OSGi DevCons. 

Finally, don’t forget to make your hotel reservation too.  The conference hotel room booking system for the Hyatt Regency Reston is also available; it is best to book early to secure your place.

Looking forward to seeing you in Reston in March next year!

OSGi DevCon Committee:

Peter Kriens
Mike Francis
(corrected 13/12/2011: left David out of the program committee list)

Thursday, November 10, 2011

OSGi DevCon Subsmissions, You've Got Until Nov 18!



The submission system for OSGi DevCon was supposed to close tomorrow but as seems customary nowadays the submission time has been extended to serve all those people out there that did not have enough time ... So the deadline is now November 18!

We already have an interesting list of submissions but it would be nice to have a larger set. There are so many issues that seem to generate some controversy that I would expect more pundits. Some topics that seem to scream for the use of the wonderful soap box a presentation at OSGi DevCon is:
  • Jigsaw versus OSGi
  • Package imports versus Module dependencies
  • Manifest first like PDE versus manifest as output like bndtools
  • Subsystem versus bundles
  • Inversion of Control versus Delegation of Control
  • Semantic Versioning
  • Relase models
  • Extending type safety into the Module system
  • Statics versus instances
  • Services versus Central control
I would love to see an OSGi DevCon where we get really good discussions trying to get to the heart of these issues, it really can help to further the software industry.  Where are the academics to help us find the answers to these issues? But you are of course also more than welcome to submit your position when you're not an academic!

Of course we also want presentations about experiences (good or bad) or that show interesting products/frameworks based on OSGi.. There is always room and more than enough interest in these presentations.

So given this extra week, you no longer have an excuse not to submit a presentation. Looking at the issues that are currently playing, combined with the enthusiasm of the EclipseCon organization I expect this to become a very good OSGi DevCon/EclipseCon in 2012. You're surely do not want to miss this premier event to interact with the key players in the OSGi eco system.

You can submit here until Nov 18 2012: http://www.eclipsecon.org/2012/submissions-are-open

Hope to see many of you in Reston in March 2012.

Peter Kriens

Monday, October 24, 2011

What to Depend On

Why do we need modularity in the first place? Lets get rid of this extraneous layer and make life simpler as Erlang's Joe Armstrong argues. Then again, maybe not.

The reason we need modules is that Object Oriented (OO) technology is starting to fail us. OO gave us a tremendous boost in productivity for 25 years but it is running out of steam for today's systems and especially for statically typed languages like Java. The reason is simply too much coupling. Using a single class from a library drags in its whole transitive dependency, to run even a simple program we have to download the internet. The independent evolution of these dependencies tend to create nightmares like JAR hell. After assemblers, subroutines/functions, modules, classes, and packages it is time for the next paradigm in software! A paradigm that must explicitly address the problem of too much coupling in Java systems.

Jigsaw and JBoss A7 claim to provide the solution to the dependency pains of Java by providing modularity since modularity provides encapsulation; we all know that encapsulation can reduce coupling. But do they? Sure, they make it much simpler to assemble a large class path because it becomes largely self assembling. However, they do not fundamentally change the way the class path works today. That is, they actually only provide a more convenient way towards JAR hell (Personally, I prefer a slightly different destination).

Worse, the hallmark of modularity, encapsulation, is severely compromised by the Jigsaw/JBoss model and Java's best practices.Virtually all Java code has become contaminated with class loading hacks to fight coupling. Java interfaces provided us with a tool to make clean decoupled class diagrams but at the same time created the problem of instance coupling: how to get the instance that implements that interface. In Java, instance coupling is generally achieved with dynamic class loading hacks. Unfortunately, most of those patterns need to violate the module's boundaries to get access to the client and implementation classes.

Once you see this problem (and it is not obvious), the solution is rather clear: we need a concept to handle the interaction between modules, a contract. Just like a class can implement one or more interfaces (the type contract), a module should be able to provide one or more services (the module contract). A module should depend on the services it needs, not on the JAR that holds those implementations. Exactly like a type should depend on an interface and not on an implementation class.

How to specify the service contract? An interface? The granularity of an interface is too small in almost all of the most trivial cases. In general a contract describes a collaboration between multiple actors, defined in interfaces, abstract or base classes, permissions, utility functions, etc.  Like Jigsaw, could we (ab)use the virtual module as the "interface" of a module? Nope. That model has no representation in the type system and is therefore fundamentally unsafe. It also is wrong to restrict a module to providing a single service; remember why interfaces were added to Java? Only because we had single inheritance and it was felt that the 1:1 relation was too constraining.

The logical choice in Java is obviously the package. A Java package is already the Java module: it provides encapsulation, a namespace, is part of the Java language, is part of the type system, it has a name, can easily be versioned, and it has the proper granularity for a contract. A module should provide a number of packages and depend on a number of packages that specify contracts between the modules. Just like methods provide the contract between classes. Since the dependencies are expressed in packages, modules can be substituted.

Anybody that doubts package imports/exports are a bad idea should consider that OSGi also supports the module-to-module dependency of Jigsaw/Maven/JBoss, it can even separate types. Since each OSGi user has the choice, why do you think that over time they use packages? Maybe because it works?

The need for package imports/exports is crystal clear when you really understand the violation of encapsulation caused by the interaction of the Jigsaw/JBoss programming model and popular Java patterns. Five years from now I might be able to say: "I told you so," but I still have hope that enough people will put in an effort to understand the looming problem to prevent another 5 years lost to the gains of true Java modularity.

Peter Kriens



Saturday, October 15, 2011

Another bndtools Hackathon

The house feels rather silent after 4 days of intense hacking on bndtools with PK Sörelde, Neil Bartlett, Joakim, Stuart McCulloch, David Savage (remote from the UK), and Ferry Huberts here in my house. We closed a lot of bugs and we (well almost) reached our goal to run bndtools on the OSGi build faster than anything else. Then again, maybe I should say we reached our goal since Neil just informed me with Skype from an English bus that he was now really close.

Neil Bartlett started already quite some time ago to create the ultimate OSGi IDE based on Eclipse. Since he was using bnd to do the low level building I convinced him to also use the more advanced build features in bnd, which resulted in a fruitful collaboration. This brought ComActivity into the game. A couple of years ago they hired me as a consultant to help them with their OSGi work. At that time we used bnd as an Eclipse plugin to do the build. Last year they switched to bndtools since it has a much more extensive GUI. PK Sörelde from ComActivity even contributed the version release management tool. With over an increasing number of listeners on the bndtools list we were getting in an area that bndtools needed to become rock solid stable so it could be used by more people. Two weeks ago PK proposed to do an intense round of finalizing open issues and I gladly offered my house.

We also had Stuart McCulloch from Sonatype here to help us out with maven. Since maven is quite popular as an OSGi build tool with the maven bundle plugin, which is based on bnd, collaborating is quite natural. Initially we wanted to do the complete build in bndtools but after brainstorming we found out that we could use the m2-eclipse plugin quite elegantly for building, leaving the bundle development (coding, manifest, debugging, etc) to bndtools. Using m2-eclipse will give us full fidelity with the offline and IDE build.

The coming week we will test the heck out of bnd and bndtools and hope to release a rock solid fast new version next week.

So lots of bugs closed, lots of fun (the weather was fantastic!), interesting discussions, good food, lots of plans, and a very nice result.

Peter Kriens

Monday, October 10, 2011

American Pie

"Steve Jobs died," someone yelled in the San Francisco Hilton lobby. First a wave of sadness descended upon me and then I got angry at myself for giving in to celebritis. Yes, he is a big part part of the tools I use daily to get my work done and he is part of the many Apple gadgets around the house as well as much of their content, as so many people today. And it is likely he will sorely be missed in the coming years for not providing his constant stream of innovations. Every death is sad because something is lost forever but the death of a well known figure should not feel like someone close to you died. So why was it so hard to shake of this sadness?

There is of course the identification: Steve Jobs was only 3 years older, he worked in the same industry, and in moments of grandeur I have envisioned myself as a Steve Jobs if only I had been born a bit earlier and in the US instead of Rotterdam. When someone that plays such a role for you his death confronts you with your own mortality. So was this sadness only caused by a rather solipsistic fear of death?

I guess there was his constant presence through my career. When I soldered my first computer, he made the Apple II. When I designed our newspaper page make-up terminal in the eighties we got inspired by Smalltalk and Steve Jobs was inspired by the Alto, both closely related results from Xerox' PARC Place. Also Postscript, another result of PARC Place, was a shared inspiration though admittedly Apple made it viable with their LaserWriter. Around 1985 I developed an object oriented language called CO2. I had been inspired by Brad Cox, the inventor of Objective-C. So when Steve Jobs came up with the NeXT, largely based on Objective-C, I had to get one. A mind blowing experience only numbed by the excruciatingly slow optical drive.

So maybe the reason for the sadness was not egocentric but more the loss of the rare someone that shared the same enthusiasm for the combination of computers and esthetics. Someone that struggled with the same problems (though he admittedly much more successful than I ever did) in the eighties. Maybe the sadness is caused by losing the subject of a my rather rare envy. Though I share his focus on tomorrow but I do lack his ruthlessness and risk taking nature that is required for great products. His gusto to break backward compatibility only so that tomorrow's product could become better than anything else. I unfortunately tend to cave in because the others really seem desperate to want to have that feature ...

Of course I can rationalize that the almost absolute powers granted to a Steve Jobs are an anomaly and that in general we have to balance the interest of different parties. The route I've taken in my life is to find a better way to develop software, software based on collaboration and open environments like the OSGi Alliance. In such an environment compromises are inevitable but maybe the sadness is caused because Steve Jobs' death shines such a harsh light on all the compromises I agreed to only for backward compatibility and special interests.

And maybe one should not analyze those feelings to deeply and just accept the sadness, it will pass. And maybe after all it is just celebritis; for someone with very few heroes I guess am just not used to it.

Peter Kriens


Thursday, September 8, 2011

Events

These months are quite hectic. I could just finish the draft for the Residential Specification hours before my flight left for the "Masterclass on OSGi" in Stockholm. Neil Bartlett and I had the masterclass in Stockholm because we had been asked to organize it there. We were told there were lots of eager OSGi afficionados that would love to come. In the end we had a single person from Sweden (and he came from the other coast) so from that perspective Stockholm was not good. However, the location, Djüronaset, was absolutely fantastic.
This must be one of the best locations in the world to learn OSGi! Beautiful spot, sauna, swimming pool, coffee and tea all day around, and a good conference room. Even the food was excellent. Combine this with a very good class and you have the recipe for  a thoroughly enjoyable week. This was a very international master class as we had people from Holland, England, Belgium, Sweden, Germany, US, Bulgaria, and Serbia. And one, ehh, guess Australia. Gerd Wütherich had brought Sheepie, a toy sheep secretly put in his backpack by his daughter. Sheepie had many adventures during the week, all photographed by Nils Hartmann for the book about Sheepie. I think we could actually provide Sheepie with the Masterclass certificate because she became an OSGi master along the way.

All participants were experienced so we could focus on the really interesting parts. With an experienced group like this it is possible to really discuss problems people have in day to day practice. Those 4 days were gone before we knew it and saying goodbye was actually hard. We are looking for a similar location on the East coast of the US so if you have suggestions, let us know. Believe, this is fun.

The rest of the week was largely catching up after having been glued to my Framemaker for almost three months. And of course doing the chores for the different events that are coming up. First the OSGi DevCon in Reston, VA next march. The Call for Papers will go out next week and we have started planning the details. Not much time for this as next week we have a meeting in Palo Alto, CA for a combined CPEG and EEG meeting. I had promised many people to provide feedback to their work so that is taking my time.

And then, last but absolutely not least, the OSGi Community Event! This year in Darmstadt, hosted by Deutsche Telekom. The Community Event Puzzle was solved but I will not give out the answer, really interested to see how many people found the word. You can tell me next week! Really looking forward to this event, hoping to meet the OSGi crowd and more. I am moderating a panel about "The Future of OSGi", a subject you can understand that is close to my heart. I've lined up some great people so this will be very interesting. I'll be flying to Darmstadt right after the weekend that I returned from California. I sure hope to see you there.

Then after returning from Darmstadt another week playing catchup so that the week after I can go to JavaOne where BJ Hargrave and I have 3 events going on. An OSGi BOF, a presentation about "Why OSGi?" (Isn't it obvious) and "Migrating to OSGi Services First." And then it is already October ...

Busy busy busy ...

Peter Kriens

1. Updated West coast to East coast


Wednesday, August 24, 2011

A Puzzle for the Community Event

Next month we're having the community event in Darmstadt, Germany, Sept. 20 and 21 (with Alex Blewitt as keynote btw). Now before you click away because you were just hoping for some juicy Jigsaw information, I'd like to challenge you! I was inspired by Masquerade, an amazing puzzle book, after seeing a documentary about it on BBC 4. I therefore decided to create my own puzzle. Not in the league of Masquerade but I think it is sufficient complex to keep some geeks like us busy. The first prize is a free registration for the Community Event. It also happens to be the only prize (and I'll buy you a drink at the Community Event if you found all the answers).

If you're really clever enough to solve it but just too slow to win then do not hesitate to send me a mail because I am interested in anybody that is clever enough to solve this puzzle.

Kit Williams, the author of Masquerade went into recluse after his puzzle went viral in the late seventies. He is living in a hideaway in Gloucestershire in England still making amazing art but in general shying away from public. I am pretty sure this puzzle is too geeky to get that kind of following.

Anyway, go solve the puzzle if you can!

Peter Kriens



Thursday, August 18, 2011

Join us for the Community Event, 20&21 September 2011

It is time to register for the OSGi Community Event in Darmstadt September 20 & 21! We have a really good agenda this year and lots of OSGi people will be present. Looking at the lunches map, we should have quite a lot of attendants as we're in the heart of the OSGi Continent.

I'll be chairing a panel on "What Are the Major Tasks to Tackle Within the Next Two Years?" and I will be looking forward to your input.

If you register in the next two days you will get a substantial early bird discount.

I hope I will see you in Darmstadt next month!

  Peter Kriens

Tuesday, August 16, 2011

The Package as Contract

Contract based design is, from my perspective, one of the best insights our industry had for the past 50 years. With contract based design you separate the providers of the contract and the consumers of the contract. In Java, there is a sometimes tendency to see these contracts as just an interface. The provider is the implementer and the consumer is the client of the interface. I think this is misguided because an interface is way too limited for real life software contracts.

A software contract is a collaboration scenario; the contract designer is the choreographer allowing the provider and consumer safely do their dance. In this choreography, you sometimes need an interface to describe the role of the consumer of the contract; we often call those interfaces listeners. Other times, the onus is on the provider and then we frequently revert to names like managers or servers. Sometimes we need real classes to provide event objects, exceptions, permissions, or small helper classes to make the overall contract easier to use. Sometimes, though rarely, there is even a need for private classes to implement parts of the contract in the contract. It should be clear that a contract is much more than just an interface.

Contracts change over time and this raises the issue of versioning. Most of the time when we talk about versioning we only talk about the provider and the consumer versions. However, the whole idea of contract based the design is that the consumer does not depend on the provider, it only depends on the contract. So in reality, the most important version is not the provider's version, it is the contract version! In most build systems this is very awkward as it forces a contract to be specified by an artifact, the only thing that is versioned. Since the best contracts are well defined, highly cohesive contracts this creates a proliferation of artifacts, containing only a few packages. This is cumbersome so people tend to aggregate them and version them together. Often even bunching them up with a provider! However, a side effect of aggregating contracts is that every contract evolves at the speed of the fastest changing contract, causing many unnecessary version update ripples through your system. If you include a provider, the speed even increases more.

By far the best solution I know how to handle contracts is to take the package in Java very serious. A package is really a module, it has encapsulation (classes, interfaces, and resources)  and it can control accessibility. It provides all we need to define a contract. The only missing thing in plain Java is that packages are unfortunately not versioned.

OSGi was made for design by contract from the start and that is why we've taken packages so serious. In OSGi packages are versioned. This is a lot easier that versioning artifacts, although many believe the opposite. The consequences of a change in a contract package is a lot easier to understand than a change in an aggregation of thousands of classes. In the OSGi we have a lot of experience with package versioning and most packages are still at surprisingly low numbers. We actually maintain the version of a package as resource in the package or an annotation on the package.

Designing by contract also explains why OSGi bundles import and export packages and not require aggregations of thousands of classes. The intention is even that shared packages should only be contract packages. In an ideal OSGi system there is no need of sharing implementation classes.

With this experience behind our belts it is therefore hard to see how Jigsaw will provide a model that seem to ignore these lessons of contract based design in a modular world.

Peter Kriens

Sunday, July 17, 2011

OSGi and Lunches

About two years ago I found out that a French OSGi expert (Jerome Moliere) actually lived around the corner. It was quite a surprise because I live in the nick of the woods in a small village in the south of France. However, it is a pretty good combination, the south of France and somebody you can discuss OSGi with. There are a large number of pretty good lunch places within 15 km radios and there about 30.000 wineries in a circle of about 100km. So since that day we regularly meet for lunch and enjoy the good life.

So why should you care? Well, I ran into a link to a Google Shared Spaces. Google shared spaces is one of those powerful but simple ideas. A shared space is a link where people can share information that is then visually aggregated. You can share a map, voting, a drawing, text, shopping list, etc. Anybody that has the link can modify the shared space. I had to think of one of those shared spaces when I saw the Map cluster. Would it not be fun to find out who in your neighborhood has some interest in OSGi? Having lunch or a drink in the local bar might be really helpful. Just think of how much more fun it is to complain about class loaders when you're not alone? Or just think how nice it is to show something cool that you've made to someone that actually has a clue?

For this reason I've created a shared map where you can just add your name and location to the map. I know we developers are not known for our outstanding social skills but it is worth a try! So do not be shy or lazy. And if you get a good chapter together you can always invite me for lunch!

You can find the map here: http://goo.gl/61MKk

Peter Kriens

(Follow on Twitter)

Monday, June 27, 2011

Negative Qualifiers

Some face time with your colleagues can sometimes be really productive. Last week we had an EG meeting at IBM in my home town Montpellier. As I've known BJ Hargrave (IBM) and Richard Hall (Oracle/Apache Felix) for so many years I'd invited them to stay in our guest house. After arriving around noon we took the afternoon off to solve world hunger and climate change over a glass of rosé. As too often happens in these important discussions, we were seriously distracted with versioning strategies.

The OSGi version is defined as linearly increasing where the floor is the shortest form of the version. That is, 1.2.3 is less than any other version that starts with 1.2.3. A side effect of this model is that it is hard to model pre-release versions. A common approach is to use a snapshot version for artifacts that are on their way to becoming a release version. For example, 1.2.3-SNAPSHOT indicates that this artifact is becoming version 1.2.3 but is not there yet.

You can do this in OSGi with the qualifier but it is awkward because you want the final release version to be sort highest. Like 1.2.3.A1, 1.2.3.A2, 1.2.3.FINAL. The fact that you cannot just release 1.2.3 but have to use 1.2.3.FINAL generally offends our highly developed aesthetic sense.

The maven solution is to use a magic keyword: -SNAPSHOT. However, we felt a bit uncomfortable with a magic keyword. The qualifier in OSGi is left open for enterprises to implement a release strategy and it would be nice if they could do the same with a qualifier that came before the actual release. So why not have negative qualifiers?

The syntax for the OSGi qualifiers specifies a period as separate between the version and the qualifier, for example 1.2.3.qualifier.  If we allow a minus sign instead of the period we can interpret the qualifier as coming before the version instead of after. That is, 1.2.3-qualifier is actually less than 1.2.3. This contains the special case of -SNAPSHOT but enables the use of qualifiers to track build, patch, or other variations.

So what about ranges? Bundles that depend on artifacts normally specify a range like [1.2.3,1.2.4). I think this should include a version like 1.2.3-SNAPSHOT but there was some discussion about this, some wanted snapshot only when the range indicated snapshots, e.g.[1.2.3-,1.2.4-). For me this is awful because it requires rebuilding and retesting after you released. In think [1.2.3,1.2.4) must accept any negative qualifiers on 1.2.3. The discussion is ongoing.

Negative qualifiers would really help bndtools. We could always build with a 1.2.3-${tstamp} to capture the build time. If one day the artifact is released, we take the JAR and convert the minus in the version to a period. For example, the artifact has version 1.2.3-201106261020 if it comes from the development tree/repository. Once it is released, we change the manifest to version 1.2.3.201106261020 inside the release repository. No need to recompile or retest.

There are some loose ends with version ranges but I am sure we can work this out. However, the fact that got excited about this idea (world hunger has to wait another meeting) does of course not mean that the OSGi will start using this model tomorrow. BJ is working on an RFP to there is hope for the future. If you have any ideas, let us know.

Peter Kriens

Friday, June 24, 2011

Ease of Use and Spring

Things should be as simple as possible, but not simpler.

As true as this is, the boundary between simple and simplistic is unfortunately highly subjective. If something works for you, don't change it. Unfortunately, many developers are in a situation where the environment and problems are highly complex and OSGi can simplify such situations.

The primary value that OSGi brings to the table is when you have to maintain a large code base over long periods of time, you have many external dependencies, you need to provide middleware, if you work with multiple developers, or if your development is spread over multiple locations. For those situations OSGi provides a type safe module extension to Java that can significantly reduce your development complexity.

So what is the usability issue? Unfortunately, both the modularity and type safety are at odds with prevailing class loading patterns in Java. OSGi is often the messenger telling you that your baby is neither modular nor really type safe. For example, Dependency Injection in Spring requires access to implementation classes from the configuration, exactly the classes you should really keep private. Worse, dynamic loading is only type safe if there is a single namespace in the universe. A single namespace causes DLL hell aka JAR hell that OSGi prevents by managing multiple class name spaces in a type safe way.

These are not OSGi problems, if Jigsaw one day will be modular and type safe it will have exactly the same issues!

To get the many benefits of modularity and type safety you must therefore change the way you think about Java ... If something is inside your module everything works as before but if you want to work with other modules you need to consider type safety and module boundaries. Object Oriented Design forced you to consider inheritance, encapsulation, and object identity, things that did not exist in Structured Design. Java modularity requires just such a paradigm shift. What we need is an inter module communication mechanism that truly hides implementation classes and enforces the type safety across modules. OSGi services are such a mechanism.

As Spring heavily relies on dynamic class loading it ran head on into the module boundaries and multiple name spaces. The resulting mess is what causes the negativity. Though Spring DM included OSGi services the problem was that they were more supported than embraced, resulting in many Spring DM/Blueprint modules to use the existing dynamic class loading model. However, this results in the import/export of large numbers of implementation classes. Exporting/Importing implementation classes increases the public space, the opposite of modularity. The history would be very different if the Spring libraries had embraced services, setting an example for true modularity.

There are only two solutions to this problem: Change OSGi or change the world. As I do believe there is no value in a little bit of modularity I can only see false advertising value in punching convenient holes in the modularity story of OSGi.  I do believe it is worth keep strong modularity and its enforcement. Very large applications like Eclipse, Websphere, JDeveloper, and others demonstrate that at certain scales OSGi becomes a necessity. And as more and more open source projects start to embrace OSGi it will become easier and easier to build OSGi based systems.

That said, as Rod justly points out, there are lots of people that are happy with Tomcat and depend on hard core dynamic class loaders. The OSGi service based programming model could provide benefits to these applications even if it did not enforce the boundaries. We're therefore thinking about a version of OSGi that inherits the class loading architecture of whatever environment it runs in; a JAR on the class path is then treated as a bundle. For example, in a WAR all entries in the WEB-INF/lib directory would be available as bundles. Karl Pauls tested this and it is a testament to modularity that many existing bundles actually worked out of the box. Though this solution obviously does not help when you are in JAR hell, it does provide many of the benefits of OSGi in your own familiar environment. And one day, when JAR hell finally arrives, the OSGi Service Platform is there to serve you.

Peter Kriens

Wednesday, June 15, 2011

Masterclass on OSGi

Neil Bartlett and I are organizing a Masterclass on OSGi in the Stockholm archipelago from August 30 to September 2. In this masterclass we provide a thorough schooling in OSGi (using bndtools) but we also have a chance to explore the areas the attendees are interested in. The last Masterclass was a great success and we intend to repeat this success in Stockholm. One interesting aspect was that experienced people that had been using OSGi for several years were surprised how much easier OSGi had become.

Though we've a places left there is a bit of urgency because the hotel indicated that the rooms are becoming scarce. As the masterclass usually runs into the wee hours of the night it would be a pity if you had to stay in another hotel.

You can find all the details here: Masterclass on OSGi.

Peter Kriens

Thursday, May 26, 2011

The Unbearable Lightness of Jigsaw

Yesterday Mark Reinhold posted a link to the new requirements on Jigsaw. He states a more ambitious goal to make Jigsaw not just for VM developers only as they said before. The good news is that OSGi is recognized. The even better news is that now there are requirements it is obviously clear that OSGi meets virtually all of them, and then some. The bad news is that despite OSGi meeting the requirements it does not look likely that Jigsaw will go away anytime soon.

As this round will likely get me another 15 minutes of fame, let me use this fame to try to explain why I consider the Jigsaw dependency model harmful to the Java eco-system.

Jigsaw's dependency model must be based on Maven concepts according to requirement #1. This model is the same as Require-Bundle in OSGi. It is based on a module requiring other modules, where modules are deployment artifacts like JARs. To deploy, you transitively traverse all their dependencies and put them in the module system to run. That is, a module for Jigsaw is a deployment module, not a language module.

It is a popular solution because it is so easy to understand. However, Einstein already warned us: "A solution should be as simple as possible, but not simpler." The problem of managing today's code base for large applications is complex and all of us in the Java eco-system will be harmed with a simplistic solution.

The key problem is that if you depend on one part of a JAR you inherit all its transitive dependencies. For example, take the JAR Spring Aspects JAR. Even if my code only uses the org.springframework.beans.factory.aspectj package in this JAR, the Spring Aspects JAR will unnecessarily provide me with on dao, orm, jpa, as well as the transaction package. If I deploy my application  I am forced to drag in the dependencies of these packages, which drag in other JARs, which each have additional dependencies, etc. etc. During deployment tens (if not hundreds) of JARs are dragged in even though those packages can never be reached from my code. Not only is this wasteful in memory, time, and complexity, it also inevitably causes problems in compatibility. For example, if the jpa dependency and the orm dependency transitively depend on different versions of the log4j JAR I cannot even run my code for no real reason whatsoever.  In practice this problem is often ignored, for example maven will take the first log4j version it encounters, regardless if this is a suitable version. The value of type safety is questionable if that kind of fudging is acceptable.

Maven users are already all too familiar with this problem: many are complaining that maven downloads the Internet. This is not bashing maven, maven only does what it is told to do in the poms. In Tim O'Brien's blog he explains that Maven does not download the Internet, People download the Internet. Though he rightly blames the poms, he ignores the fact that the dependency model makes it hard to work differently. It is like you want a candy bar but you can only get the whole supermarket. He advises each JAR to be self contained and completely modularized to not drag in extraneous dependencies. Agree. Unfortunately this means you get lots and lots of JARs. And with lots of JARs on your module path without side by side versioning the compatibility issues like the above log4j example explode. And Jigsaw is not likely to have simultaneous multiple versions in run-time when critically reading the requirement.

In Unix dependency managers the transitive aspect is not such a big deal because dependencies are relatively coarse grained and there is a reason why you usually have to build stuff to make it compatible with your platform. Java is (thank God) different. Java dependencies are not only much more fine grained, they are binary portable to any platform, most important of all, often they have no dependency on an implementation but use a dependency on a contract instead. Unix was brilliant 40 years ago and is still impressive today but let's not undervalue what Java brought us. We must understand the differences before we revert to a 40 year old technology from an operating system to solve the problems of a modern and unique programming environment like Java.

The key advantage that Java brings to the dependency landscape is that each class file encodes all its dependencies. Give me your class file and I will tell you what packages you depend on. For example, when I compile my JAX-RS class against the jersey JAR(s) you will find that the class file only depends on the javax.ws.rs package even though the Jersey file contains lots of implementation packages. However, in the Jigsaw world I must select a module JAR that provides that package. Best case is to pick the JSR 311 API JAR for compilation to ensure I do not accidentally pick up implementation classes. However, I do not want to make the life of the deployer hard by unnecessarily forcing a specific implementation in run-time. So what module should I pick to depend on?

Despite common believe, we do not need fidelity across all phases between the build time and run-time; we need to compile against API and run against implementations that are compatible with that API. A good Java module system must allow me to depend on an API module at compile time and select an appropriate implementation at deployment-time. Forcing the compile time dependency to be the same as the run-time dependency creates brittle systems, believe me. The problem can be solved by separating the concept of a deployment module (the JAR) and the language module.

We could define the concept of a language module in Java and allow the JAR to contain a number of these modules. Surprisingly (for some), Java already provides such a language module, they are called packages. Packages provide a unique name space, privacy, and grouping, all hallmarks of modularity. The JLS introduction even names them as similar to Modula's modules. Actually, module dependencies on packages are already encoded in the class files. The only thing we need to add to the existing system is versioning information on the package and record this version in the class file. During deploy-time a resolver can take a set of JAR's (which can be complete) and calculate what modules should be added depending on the actual environment. E.g. if it runs on a Mac it might deploy another set of additional JAR's then if it runs on the PC or Linux. 

To manage real world dependencies we need this indirection to keep this manageable. From 2010 to 2017 the amount of software will double. If we want to have a fighting chance to keep up we must decouple compile time from deployment time dependencies and have a module system that can handle it.

It seems so obvious to build on existing concept in the Java language that it begs an explanation why the transitive dependency model on JARs is so popular. The only reason I can think of is that is easier for the module system providers to program. In the Jigsaw model traversing the dependencies is as easy as taking the module name + version, combining it with a repository URL, doing some string juggling and using the result as a URL to your module. Voila! Works with any web server or file system!

Package dependencies require an indirection that uses a mapping from package name to deployment artifact. The Jigsaw/Maven model is like having no phone number portability because the operators found a lookup too difficult. Or worse, it is like having a world without Google ...
 
So for the convenience of a handful of module system providers (in the case of Jigsaw just one because it is not a specification) we force the rest of us to suffer? Though people have learned to use dynamic class loading and not telling the truth in their poms to work around dependency problems inherent in the dependency model we should not encode this in my favorite language. Should we not try to address this problem when a new module system is started? Or better, use an already mature solution that does not suffer the impedance mismatch between compile-time and run-time?

At JavaOne 2007 Gilad Bracha, then Sun's man in charge of JSR 277, told us he had started JSR 294 because he did not trust the deployers to handle the language issues. At that particular time this remark puzzled me.

Peter Kriens

P.S. Opinions stated in this blog are mine alone and do not have to agree with the OSGi Alliance's position

Tuesday, May 24, 2011

What If OSGi Ran Your Favorite Language?

My bookshelf is full of books about programming languages, from APL to Z. They are a little bit dusty because the last decade OSGi kept me confined to Java. However, I think it is time we should broaden our scope. There is just more life out there then just Java (really!).

If you look at the OSGi API then you see that the class loader modularity is strongly tied to Java. I actually do not know any other language that provides a way to intercept class loading in the flexible way that Java allows this to be done. It is also clear that most other languages out there are not type safe, something the OSGi has been fanatic to maintain and support.

However, the most important part of the OSGi programming model has always been the µservices, an Inter Module Communication (IMC) mechanism, that has virtually no overhead but is a surprisingly powerful design primitive; µservices encode so many important design principles.

OSGi µServices are also the cornerstone of Distributed OSGi. As so much code is not written in Java, how could we provide the service model to other languages? Obviously JVM languages can usually use the Java API but that is likely not optimal for a language, you like to take advantage of the unique possibilities each language provides.

Now, I could sit down and dust off my Smalltalk and write a service registry with the OSGi semantics for Smalltalk. With a bit of effort I could probably do it for C(++), maybe even Basic? However, it is clear that most language experts do not live in my house. So the question to you is, how would you make the µservice model look in another language while remaining compatible to the OSGi semantics?

Write a blog or google document how your µservice API would look in your favorite language and place a comment on this blog with a link. I am really curious how those APIs would look like. Give me your Python, Ruby, Haskell, Scheme, Clojure, Smalltalk, Javascript, ActionScript, Scala, Cobol, Perl, PHP, .NET, Beta, ML, Modula, Pascal, Fortran, Forth, Erlang, Postscript, Bash, and any of those other languages you like!

Peter Kriens
P.S. This is something I am highly interested in which does not necessarily imply that the OSGi Alliance is interested in pursuing this.

Friday, May 20, 2011

What You Should Know about Class Loaders

The more (open) source code I see the more I realize that so many developers do not understand the implications of class loaders and just try different things until it seems to work. Not that the Class Loader API is that hard but the implications of Class loaders are often not understood. In a modular environment, class loader code wreaks havoc.

Unfortunately, Class Loaders have become popular as the Java extension mechanism. Developers love the idea that you can take a String and turn it into a Class, there are so many hacks you can do with this! However, what is often not realized that in a modular world this just cannot work. The implicit assumption is that each Class object can be identified with a unique name. It should not take much more than 2 seconds to realize that in a modular world we also require the name of the module because the Class name is no longer unique.  That is, class unique.X can actually occur in multiple modules. So given the string "unique.X" there is no way to identify from which module to load this class. In a modular world there are multiple class name spaces!

What is the advantage of having multiple class name spaces? Well, it solves a huge problem in larger applications that are built from many parts that come from different sources. Think open source. Take an example of logging. Logging is so popular that everybody does it, unfortunately with different libraries. In practice, it becomes hard to ensure that all your dependencies use the same logging library. In a modular world you can connect each library to its correct logging module. In a non modular world, the first one wins because libraries are placed on a linear search path; you often have no clue who won until you get an error. Obviously this problem is not limited to logging. It is actually kind of flabbergasting that a weak mechanism like the class path is acceptable for professional applications.

So why are developers using strings so often to get a class? Well, the core reason is that Java does not have an extension mechanism. No, the Service Loader is not an extension mechanism, it is encoding of a bad convention leaking its implementation out of all its methods. Extensions through class loading are highly popular, very often in Factory patterns. Sadly, extensions are exactly the place where you want to cross module boundaries. What is the use of an extension that happens to be in your own module? So these strings often contain the name of an implementation class that implements a collaboration interface. The three most important rules of modularity are: Hide, Hide, and Hide. Do you really want to expose the implementation class from a module? Isn't that the antithesis of modularity?

In spite of these problems, the pattern to use class loaders for extension mechanisms has become highly popular. Today we even have generations that actually do not realize how bad it is to use global variables (class names) to access context through static variables. My programming teacher in 1979 would have set me back a class when I would have proposed such a solution. However, as you know, you go to war with the army you have, not the army you might want or wish to have at a later time. So the class loading hack has become highly popular in lieu of an alternative. So popular that it is used in places where it is not even needed. Worse, it will bite you whenever you move your code base to a modular world.

So here are number of rules/hints you can use to prepare your software for a modular world (and simplify your life at the same time).
  1. Do not use class loaders - If you feel the urge to use a class loader think again. And again. Class loaders are highly complex beasts and your view of the world might not match the world your code is actually used in.
  2. Do not use class loaders yet - Only for the experts.
  3. Extensions - If you have to connect to other modules use a Inter Module Communication (IMC) mechanism. A good IMC provides access to instances and not classes. If you do not have one, just make one yourself it is that not much code. Even better, use the OSGi service registry. Best is a real OSGi framework but if you're not yet ready for that step use something like PojoSR from Karl Pauls.
  4. Class.forName is Evil - Class.forName will pin classes in memory forever (almost, but long enough to cause problems). If you're forced to do dynamic class loading use ClassLoader.loadClass instead. All variations of the Class.forName suffer from the same problem. See BJ Hargrave's blog about this.
  5. Understand the Context - I've just debugged GWT code that was unnecessarily mucking around with class loaders when it needs to deserialize the request. The first token was a GWT class but the remainder of the request was the interface and parameters related to the delegate (the object implementing the RPC method.) However, for some reason they used the Thread Context Class Loader (you're in deep s**t if that one becomes necessary), and Class.forName. Completely unnecessary because they had access to the delegate object and thus its class loader. The class loader of this object must have access the RPC interface as well as all its parameters. There is a similar situation for Hibernate. You can name the classes in an XML file or you can give Hibernate the class objects for the entity classes. If you give it the class objects Hibernate works fine in a modular world. Give it XML and it chokes because in a modular world there is no 1:1 relation ship to class name and Class object. Reasoning about the context in which a class is loaded can prevent problems.
  6. Centralize - If you really need class loaders centralize and isolate your code in a different package. Make sure your class loading strategies are not visible in your application code. Module systems will provide an API that allows you to correctly load classes from the appropriate module. So when you move to a module system you only have to port one piece of code and not change your whole code base.
  7. Never assume that a class name identifies one class - The relation between class name and Class object is 1:n, not 1:1. You might desperately want it to be true but in a modular world this does not make it true.
  8. DynamicImport-Package and buddy class loading are not an alternative - If you browse the web you find lots of well meaning people pointing to  these "solutions". Unfortunately, these "solutions" move you right back all the problems of the class path.
The trick to modularity is to make modules that assume as little as possible of the outside world and communicate through well defined interfaces. If you do not know something, you cannot be wrong about it. The pesky factories in Java that use dynamic class loading to get an implementation class just do not fit well in this world. If you're not ready for OSGi yet but want to prepare your code for the future then following the previous rules will make your code more portable and easier to maintain.

Peter Kriens

Tuesday, May 17, 2011

Start with Why (OSGi) ...

Someone pointed me at a TED talk from Simon Sinek that explains why it is so easy to forget to communicate the why when you try to evangelize a technology. Why have I been working hard these last 13 years, traveling a million miles, to develop and promote a technology like OSGi? Tirelessly explaining how OSGi works for sometimes skeptical mainstream developers?

The why lies buried in my youthful encounter with objects in the early eighties. Discovering objects was like an epiphany, suddenly a lot of the techniques I had found to work well were put in a framework and got names. The excitement of reading the 1981 Byte issue on Smalltalk was immense. Objects were exactly what I wanted for my software development! In that excited mode I convinced my manager that we should switch to objects as soon as possible because it would save us so much in the future. I painted a rosy picture in the sky about how software could be developed in the future of reusable classes. Developing an application would largely consist of picking the right classes, and voilá! I got the budget easily.

Objects did provide a significant benefit and 10-15 years after the Byte issue on Smalltalk they had therefore become mainstream. Though it is still easy to point out how out industry is struggling with complexity I do believe objects provided a significant value in allowing us to handle much more complex problems. However, the vision of assembling an application out of components never happened. What happened instead is that most large object oriented programs turn into a big ball of mud that is so entangled that it becomes impossible to find reusable components in the haystack.

Thirty years after the Byte issue on Smalltalk the world looks very different. Open source software has fundamentally changed the nature of software. Open source seems to have achieved the quest of reusable components because today these open source projects are reused on a grand scale.

However, when I struggle with integrating different software packages and their myriad of dependencies I am not so sure anymore. Badly documented, missing crucial metadata, extremely sensitive to their environment, complex configuration, incompatible versions, avalanches of logging, tons of XML, and mysterious crashes, then I know we're not there yet, it is still way too hard. There should be a better way! How can we better leverage open source projects so that these projects are easier to use together?

The model we envisioned in the eighties was plug an play, the way we can go to the department store and get drills, nuts, fasteners, and bolts from different vendors that work together because they use standards so they work together. This is the dream that kept me going for the past decade. OSGi is the only software standard that provides a substrate on which reusable components can flourish. Though OSGi has not reached the grail, it is arguably closer than anything else.

I am convinced that the next big step in software is the OSGi model: bundles collaborating through substitutable services. However, just like the mainstream procedural developers had to adjust their ways to take advantage of objects, today's mainstream developers have to adjust their software to take advantage of OSGi (and not run head first against the enforced module boundaries). Some developers reject OSGi because they deem it too complex but what is not too complex if you do not understand the why? Procedural developers also felt that objects were way too complex and wasteful. An investment in a technology is always required to get its benefits.

After thirty years the vision of reusable components has come tantalizing close with OSGi. However, to go that last mile we need your help to make OSGi the rock-solid foundation for the mainstream software developers.

Peter Kriens

P.S. As always, these are my personal opinions and are not necessarily aligned with the OSGi Alliance

Blog Archive