Attributes, attributes, and attributes

A lot of thanks for the people that sent me some persistence files.  It was a bit humbling for me that I only got less than 10 replies, but those good people are highly appreciated! If you feel guilty now, then don't hesitate to still sent me some files to my personal email!

Conclusion? Well, there is quite a variety out there and there is still a lot of diversity, lots of experimentation, and lots of struggling going on. Let me reflect on the most striking issue I noticed: Meta Data.

Persistence means your data is going to make a trip in a time capsule. Java was designed as an in-memory in-process programming language and this has as consequence that you need to provide some extra information how that trip should be made. What classes are persisted, where they are persisted, with what data types, under what name, etc. The most common approach is to use the javax.persistence annotations to markup a persistence object.

For smaller projects this seems to work well but I noticed that the larger projects went outside Java and escaped to text files. I have seen hibernate XML mapping files, proprietary XML files, and one project uses a Domain Specific Language (DSL) they developed with the Eclipse Enterprise Modeling Framework (EMF).

The advantages of annotations should be clear:

1. Minimizes redundancy since they interact with the Java type information
2. Can be safely refactored in all IDEs
3. Can use small names because of Java's package scoping
4. Type safe

However, one of the projects that used XML did not restrict itself to the persistence problem alone. The attributes of the persistent entities are not only used in Java to persist, they are also reflected as forms in the user interfaces and must be validated before they are stored.This meta-data is used in the GUI (potentially in another language), the core code, and the database (SQL Schema).  For each attribute, there are quite a large number of attributes on those domain attributes that must be defined by the developer. Using a proprietary file format or a DSL obviously provides for ample space to capture this information. Generators can then be used create the annotated data objects, SQL Data Definition Language files (DDL), HTML forms, Javascript validation, etc. This, of course, has the big disadvantage of requiring an extra step in the build. 

I must admit that  I find this model quite attractive considering that most enterprise applications must manage a very large number of attributes in a large number of entities and having a single definition of the domain model sounds awfully attractive; it can prevent a lot of redundancy. The surprise is of course that there is no dominant syntax for this information. The existing data description languages (ASN.1 or XML Schema anyone?) look utterly unusable for this purpose; these languages allow such complex data to be specified that generating Java, Javascript, HTML, etc. will be awfully complicated.

So what do you think? Just annotations on the Java Entity objects (potentially with extra validation annotations that can be used by the HTML5 GUI) or have a special domain object  specification file?

Peter Kriens @pkriens

Problems with Persistence (begging)

I am still struggling with the OSGi persistence story. I am therefore doing some research on OSGi and persistence and I still find the whole Java persistence story quite confusing and complex. Part of my problem is that I see lots of frameworks but it is quite hard to see code that really uses those frameworks. Virtually all tutorials and examples look highly contrived and seem to ignore issues like caching, security and look rather lax with transactions.  

I wonder if people reading this blog could share with me a typical production source or class file showing:

• How entities are defined
• The persistent use of the entity, i.e. the part where the SQL will be generated. I.e. places where the PersistenceManager, EntityManager, SQL generation is used
• How results are cached

A single source or class file per issue is best. Adding a small description how you use persistence (Aries, JPA, JDO, JDBC, etc), the primary issues you face, and describe your environment is highly appreciated. 

I know from my own experience that there is often a feeling that your own code is not up for showing to others but please send me the raw unadulterated code; I need to see how it is today, not how you think it should be. Obviously I am not interested in what the code does or where it is used so feel free to remove comments (if any!) and change names (or just send class files). I am just looking for a couple of hundred of real world samples to extract the patterns that are actually popular in our industry. Obviously I will not share that code with anyone and treat it fully confidential. 

So in this case, do not ask what the OSGi can do for you, but for once, ask what you can do for the OSGi! ;-)

Please? Come on, it only takes 3 minutes. Send your 4 files to: Peter.Kriens@aQute.biz Thanks!

If I get some interesting result I promise share the conclusions with you in this blog. Deal?

Peter Kriens @pkriens

OSGi Services versus Extenders

In the OSGi community there are two patterns that are widely used, but looking at a recent Stackoverflow question, not so widely understood. So an attempt to elucidate.

The best metaphor I can find for bundles, services, and extenders is our day to day society. In a society with interact with others to achieve our goals. In this interaction we often play a role in a scenario. For example, in a commercial transaction we have the seller and a buyer role and the contract that governs the transaction is the scenario. In the OSGi µservice model, a µservice is one of the actors in the scenario described by the service package. The µservice model is eminently suitable for this role playing since, unlike other service models, is based from the ground up to be dynamic, just like the real world. For example, the existence of a Bluetooth service is not a means to interact with any Bluetooth devices, nope, each Bluetooth service signals that there is a Bluetooth device available in the area and provides the means to talk to that device. It is hard to overestimate the value of this tool. Though there is a place for registering your service in the Bundle's activator, OSGi becomes really powerful when you make this registration dynamic, fully taking dependencies into account, which is trivial to do with Declarative Services. Services are therefore a powerful a domain design tool that solves untold ordering, boilerplate, and concurrency problems.

In our society metaphor, bundles are the people. They control the behavior and choose what scenarios to play in, what services to consume and what services to provide.  Though µservices are a great tool for many scenarios we have to participate in, there are also many not so interesting scenarios that are repetitive, boring, and not in our prime interest. An example. Our family has two cars. In one car I can always open the car door since it detects that I have a key, the other car not. Since I always forget that I actually have to get the key, insert it in the lock, and open the car, I dislike this stupid car. By detecting my presence through the key, the smart car saved me from looking foolish.

We can mimic this behavior in OSGi because we can find out what bundles are installed and get notifications when new ones are installed. Just as our smart car can detect the key in my pocket, so can a bundle detect specific cargo in other bundles and react accordingly. Just yesterday I discussed database migration in OSGi, a perfect example. A Bundle that interacts with a database is written to expect a specific schema in the database it uses. When a new version comes along, the database needs to be updated to match this new schema. Instead of the domain bundle updating the schema itself, a chore, the bundle could contain a description of the new schema. A database schema extender bundle (for example based on Liquibase) could detect the presence of this schema description and, if necessary, adjust the database. Extender bundles can take information from an active bundle and ensure that certain chores are done on the bundle's behalf. An extender like Declarative Services is controlled from an XML file in your bundle to provide all the chores of setting up your components, handling dependencies, etc. In general, any information you can provide declarative (i.e. in resources in your bundle) should use extenders so that the bundle can focus on its domain work. 

Extenders react to the life cycle of the bundles that provide the parameters. This makes it a very powerful to deliver information as well. Paremus' packager packs native executables inside bundles and manage's the life cycle of this native application according to the extendee bundle's life cycle. That is, starting the bundle will start the application, stopping it will stop it. Providing a single model for deploying code, static content, native executables is surprisingly powerful. I have a bundle that can map the static directory in a bundle to a web server, ensuring proper cache control, range requests, and version control. I even use this model to deploy configuration data. You should see these extenders as humble servants that are ready for you and provide lots of value as long as you carry their key.

So when should you use the service pattern and when the extender pattern? 

The service model is usually about the core domain stuff your bundle is supposed to do, it is generally right where the primary focus of your bundle is, it is where your code should be. The extender model is about removing chores and boilerplate from the domain bundles to specialized bundles. And the trick of this all is to understand how the dynamics of the services and bundles unexpectedly make things easier.

Peter Kriens @pkriens

P.S. The book Antifragile from Taleb describes the effect that unreliable dynamic components can make extremely reliable systems.

The Transaction Composability Problem

Entering the enterprise world from an embedded background feels a bit like Alice must have felt when she entered Wonderland. Sometimes you feel very big, other times you feel awfully small. Worse, you often do not know your relative size in that new and wondrous world. One of these areas for me is persistence and its associated transaction model.

The reason I have to understand this area better is that for the enRoute project we will have to provide a persistence model that makes sense in a world build out of reusable components. If these components are closely tied to a specific database, JPA, and transaction manager implementations than reuse will be minimal, forfeiting the purpose. There are many issues to solve but analyzing this landscape one thing seems to pop up: the transaction composability problem. A problem quite severe in a reusable component model like OSGi.

Transactions provide the Atomic, Consistent, Isolated, and Durable (ACID) properties to a group of operations. This grouping is in general tied to a thread. A method starts a transaction and subsequent calls on that thread are part of the grouping until the transaction is either committed or rolled back. The easy solution is to start a transaction at the edge of the system (RMI, servlet, queue manager, etc.) and rollback/commit when the call into the application code returns. However,  transactions are related to locks in the databases (and other resource managers) it is therefore crucial to minimize the number of grouped operations to increase throughput and minimize deadlocks. Generating HTML inside a transaction can seriously reduce throughput. Therefore, application developers need to handle transactions in their code.
One of the issues these developers must handle is how to treat an existing transaction. Should they join it or suspend it? Is being called outside a transaction allowed? Since methods can be called in many different orders and from many different places it is very hard to make assumptions about the current transaction state in a method. For example method foo() and bar() can each begin a transaction but then foo() can not call bar() or vice versa.
The corresponding complexity and associated boilerplate code resulted in declarative transactions. Annotations provide the suspend and joining strategy and something outside the application takes care of the details. EJB and Spring containers provide this functionality by proxying the instances or weaving the classes.
Back to OSGi. Since transactions will cross cut through all components we need to define a single transaction model that can be shared by all, just like the service model.
Obviously, we will need to register a Transaction Manager service so the different components can at least manage transactions together. However, do we need to prescribe a declarative model since this seems to be the best practice? Do we then pick EJB or Spring annotations? Support both? Or make new? Or are we moot on this and allow others to provide this support with extenders? This would be similar to the service model, Blueprint, iPOJO, Dependency Manager, and DS are all in business to make life easier for code using services, a similar model could be followed for transactions?
I am very interested in hearing feedback on this.

OSGi BOF at OSGi Community Event

The OSGi Alliance will be hosting an OSGi BOF at the OSGi Community Event next week.  The BOF is scheduled for 19.00hrs on Weds 30 Oct in the Silchersaal room at the conference which is located at Forum am Schlosspark in Ludwigsburg.

Peter Kriens will provide a short introduction to "OSGi en Route" a new initiative from the Alliance which is intended to make it easier for developers to get started with using OSGi.  Peter will be keen to answer any questions and get your feedback and suggestions on this initiative.

As with any BOF the goal is for everyone to participate, so if you have any burning questions, problems, or 'itches you need to scratch' then please join us for the BOF to share them with us.  These can be anything you want to discuss, from a specific problem you are facing with OSGi development, through to more general community or OSGi futures questions.

We had a great BOF at the Community Event last year with fantastic participation and we are looking forward to repeating it again this year.

If you unfortunately aren't able to join us but have something you would like raised feel free to submit a comment to this post or email me and we will try and cover this during the session and will feed back to you.

bndtools 2.2.2!

There is a new release out for bndtools! What is it? Well, bndtools is an Eclipse plugin that provides some very nice tools to develop OSGi bundles, it is also used by the OSGi Alliance itself to build the RI's and run the test suites. Since it is based on bndlib, it provides lots of functions that work well together with the maven bundle plugin, ant, sbt, gradle, and many other build environments. The bndtools plugin provides a friendly environment to develop bundles that automatically can then also be built in a continuous integration setup, using one of the aforementioned plugins.

The 2.10 release was a major release in which we added baselining. This release has many smaller features but definitely did improve a lot of things.

The easiest way to get started (or update) with this Eclipse plugin is through the Eclipse Marketplace (In the rather unexpected Help menu). You can also install it manually from its update site: http://bndtools-updates.s3.amazonaws.com

Enjoy, and let the team know how they are doing ...

   Peter Kriens @pkriens

Are you coming to Ludwigsburg next week?

The OSGi Community Event 2013 is taking place next week in Ludwigsburg, Germany between Tues 29 and Thurs 31 Oct.

We have a schedule that is overflowing with OSGi goodness.  The conference kicks off with a "Mastering OSGi with Ease" tutorial on Tuesday morning. From Tues afternoon to close on Thursday there are 26 OSGi talks ranging from embedded, to enterprise, to cloud and covering topics from development hints, tips and best practices, to case studies on how OSGi is being in the world. In addition Thursday morning sees an OSGi Keynote from Ian Robinson of IBM called Travelling Light for the Long Haul.

The full schedule is available online.

There will also be an OSGi BOF on one of the evenings offering the opportunity to find out more about some of the current OSGi Alliance initiatives and to discuss OSGi opportunities and challenges that you see.  Further details on the BOF will be announced soon.

The Community Event is co-located with EclipseCon Europe and there is still time to register to secure your place.  All attendees also get full access to the EclipseCon Europe conference. Details on registration, location and hotels are available on the Community Event homepage.

There are of course plenty of social activities and opportunities to network with your friends and peers, and of course drink and eat plenty of German beer and food!  Tuesday evening sees a Stammtisch and Cirque d'Eclipse; and a reception will be held on Wednesday evening.

It you are joining us for the Community Event you may also be interested in the popular full day Code Camp that the OSGi Users' Forum Germany are running the day before the Community Event (Monday 28 Oct) - Building OSGi based HTML5 Web Application with Peter Kriens and Neil Bartlett. Full details of the agenda and how to register for this are available here.

If you have any questions please email the OSGi Community Event Program Committee.

We hope you can join us and look forward to seeing you next week.