Problems with Persistence (begging)

I am still struggling with the OSGi persistence story. I am therefore doing some research on OSGi and persistence and I still find the whole Java persistence story quite confusing and complex. Part of my problem is that I see lots of frameworks but it is quite hard to see code that really uses those frameworks. Virtually all tutorials and examples look highly contrived and seem to ignore issues like caching, security and look rather lax with transactions.  

I wonder if people reading this blog could share with me a typical production source or class file showing:

• How entities are defined
• The persistent use of the entity, i.e. the part where the SQL will be generated. I.e. places where the PersistenceManager, EntityManager, SQL generation is used
• How results are cached

A single source or class file per issue is best. Adding a small description how you use persistence (Aries, JPA, JDO, JDBC, etc), the primary issues you face, and describe your environment is highly appreciated. 

I know from my own experience that there is often a feeling that your own code is not up for showing to others but please send me the raw unadulterated code; I need to see how it is today, not how you think it should be. Obviously I am not interested in what the code does or where it is used so feel free to remove comments (if any!) and change names (or just send class files). I am just looking for a couple of hundred of real world samples to extract the patterns that are actually popular in our industry. Obviously I will not share that code with anyone and treat it fully confidential. 

So in this case, do not ask what the OSGi can do for you, but for once, ask what you can do for the OSGi! ;-)

Please? Come on, it only takes 3 minutes. Send your 4 files to: Peter.Kriens@aQute.biz Thanks!

If I get some interesting result I promise share the conclusions with you in this blog. Deal?

Peter Kriens @pkriens

OSGi Services versus Extenders

In the OSGi community there are two patterns that are widely used, but looking at a recent Stackoverflow question, not so widely understood. So an attempt to elucidate.

The best metaphor I can find for bundles, services, and extenders is our day to day society. In a society with interact with others to achieve our goals. In this interaction we often play a role in a scenario. For example, in a commercial transaction we have the seller and a buyer role and the contract that governs the transaction is the scenario. In the OSGi µservice model, a µservice is one of the actors in the scenario described by the service package. The µservice model is eminently suitable for this role playing since, unlike other service models, is based from the ground up to be dynamic, just like the real world. For example, the existence of a Bluetooth service is not a means to interact with any Bluetooth devices, nope, each Bluetooth service signals that there is a Bluetooth device available in the area and provides the means to talk to that device. It is hard to overestimate the value of this tool. Though there is a place for registering your service in the Bundle's activator, OSGi becomes really powerful when you make this registration dynamic, fully taking dependencies into account, which is trivial to do with Declarative Services. Services are therefore a powerful a domain design tool that solves untold ordering, boilerplate, and concurrency problems.

In our society metaphor, bundles are the people. They control the behavior and choose what scenarios to play in, what services to consume and what services to provide.  Though µservices are a great tool for many scenarios we have to participate in, there are also many not so interesting scenarios that are repetitive, boring, and not in our prime interest. An example. Our family has two cars. In one car I can always open the car door since it detects that I have a key, the other car not. Since I always forget that I actually have to get the key, insert it in the lock, and open the car, I dislike this stupid car. By detecting my presence through the key, the smart car saved me from looking foolish.

We can mimic this behavior in OSGi because we can find out what bundles are installed and get notifications when new ones are installed. Just as our smart car can detect the key in my pocket, so can a bundle detect specific cargo in other bundles and react accordingly. Just yesterday I discussed database migration in OSGi, a perfect example. A Bundle that interacts with a database is written to expect a specific schema in the database it uses. When a new version comes along, the database needs to be updated to match this new schema. Instead of the domain bundle updating the schema itself, a chore, the bundle could contain a description of the new schema. A database schema extender bundle (for example based on Liquibase) could detect the presence of this schema description and, if necessary, adjust the database. Extender bundles can take information from an active bundle and ensure that certain chores are done on the bundle's behalf. An extender like Declarative Services is controlled from an XML file in your bundle to provide all the chores of setting up your components, handling dependencies, etc. In general, any information you can provide declarative (i.e. in resources in your bundle) should use extenders so that the bundle can focus on its domain work. 

Extenders react to the life cycle of the bundles that provide the parameters. This makes it a very powerful to deliver information as well. Paremus' packager packs native executables inside bundles and manage's the life cycle of this native application according to the extendee bundle's life cycle. That is, starting the bundle will start the application, stopping it will stop it. Providing a single model for deploying code, static content, native executables is surprisingly powerful. I have a bundle that can map the static directory in a bundle to a web server, ensuring proper cache control, range requests, and version control. I even use this model to deploy configuration data. You should see these extenders as humble servants that are ready for you and provide lots of value as long as you carry their key.

So when should you use the service pattern and when the extender pattern? 

The service model is usually about the core domain stuff your bundle is supposed to do, it is generally right where the primary focus of your bundle is, it is where your code should be. The extender model is about removing chores and boilerplate from the domain bundles to specialized bundles. And the trick of this all is to understand how the dynamics of the services and bundles unexpectedly make things easier.

Peter Kriens @pkriens

P.S. The book Antifragile from Taleb describes the effect that unreliable dynamic components can make extremely reliable systems.

The Transaction Composability Problem

Entering the enterprise world from an embedded background feels a bit like Alice must have felt when she entered Wonderland. Sometimes you feel very big, other times you feel awfully small. Worse, you often do not know your relative size in that new and wondrous world. One of these areas for me is persistence and its associated transaction model.

The reason I have to understand this area better is that for the enRoute project we will have to provide a persistence model that makes sense in a world build out of reusable components. If these components are closely tied to a specific database, JPA, and transaction manager implementations than reuse will be minimal, forfeiting the purpose. There are many issues to solve but analyzing this landscape one thing seems to pop up: the transaction composability problem. A problem quite severe in a reusable component model like OSGi.

Transactions provide the Atomic, Consistent, Isolated, and Durable (ACID) properties to a group of operations. This grouping is in general tied to a thread. A method starts a transaction and subsequent calls on that thread are part of the grouping until the transaction is either committed or rolled back. The easy solution is to start a transaction at the edge of the system (RMI, servlet, queue manager, etc.) and rollback/commit when the call into the application code returns. However,  transactions are related to locks in the databases (and other resource managers) it is therefore crucial to minimize the number of grouped operations to increase throughput and minimize deadlocks. Generating HTML inside a transaction can seriously reduce throughput. Therefore, application developers need to handle transactions in their code.
One of the issues these developers must handle is how to treat an existing transaction. Should they join it or suspend it? Is being called outside a transaction allowed? Since methods can be called in many different orders and from many different places it is very hard to make assumptions about the current transaction state in a method. For example method foo() and bar() can each begin a transaction but then foo() can not call bar() or vice versa.
The corresponding complexity and associated boilerplate code resulted in declarative transactions. Annotations provide the suspend and joining strategy and something outside the application takes care of the details. EJB and Spring containers provide this functionality by proxying the instances or weaving the classes.
Back to OSGi. Since transactions will cross cut through all components we need to define a single transaction model that can be shared by all, just like the service model.
Obviously, we will need to register a Transaction Manager service so the different components can at least manage transactions together. However, do we need to prescribe a declarative model since this seems to be the best practice? Do we then pick EJB or Spring annotations? Support both? Or make new? Or are we moot on this and allow others to provide this support with extenders? This would be similar to the service model, Blueprint, iPOJO, Dependency Manager, and DS are all in business to make life easier for code using services, a similar model could be followed for transactions?
I am very interested in hearing feedback on this.