Despite common belief, splitting up an application into a number of bundles is not automatically introducing modularity. The almost magic benefits of modularity are caused by the particular decomposition. Many decompositions can actually increase complexity, only the right decompositions reduce complexity. This was perfectly demonstrated by the father of modularity David Parnas. In his seminal paper On the Criteria To Be Used in Decomposing Systems into Modules (PDF), written in 1972. Parnas takes a simple problem, a text indexer, and shows two different decompositions. He then added the requirement that the source text files could be very large, no longer fitting in memory. The first decomposition required changing each module, the second decomposition only had one affected module.
Parnas' paper clearly shows why creating the correct decomposition is so hard, it is about predicting the future. With this disclaimer, there are some general rules that apply. These rules are in their essence the same as we use daily in writing our object oriented code. The most important being information hiding. Information hiding works because if you do not know something, you cannot make false assumptions about it. Assumptions that could break your code when violated in runtime.
With modularity, we have the fantastic tool that the rules inside the module are different from the rules outside the module. As a module author we can control what we expose from the inside and what is hidden from the outside. We can make sure that outsiders cannot make assumptions about our internal workings because they do not have access to them. By minimizing our dependencies on other modules we can reduce our own assumptions of the outside world. The less we assume, the more resilient our module will be to changes in the outside world.
In OSGi we even quite far with this model. In the initial model, you could basically only export API, not implementation classes. Each bundle could only talk to other bundles through services. A bundle is not even allowed to make assumptions about its and other modules' life-cycle. Though for many this sounds extreme, it was put in place to minimize assumptions that are so easily broken.
But there is more to modularity than just robustness and resilience. The proper decomposition can also simplify by requiring much less code. In 1984 I was the designer for an object oriented window system used in a page makeup terminal, this was my largest object oriented design so far. This design process was accompanied with an uneasy feeling that I was not designing, I was only postponing. Every abstraction seems to just postpone the really hard problems. However, one day I realized that I was done without ever having felt that I solved those really hard problems, they seem to have disappeared. In small this effect is visible with recursive algorithms. With the right decomposition, you need very little code, with the wrong decomposition you wrestle with the special cases of start and ending. The right decomposition somehow removes the need for much code.
I do not have a good understanding where the complexity goes when you have a proper modular structure but I have seen it too many times to know it happens. The problem is that once you found a proper decomposition, it feels so natural that you can't understand why you did not see this immediately.
Modularity provides benefits when it is used in the proper way, splitting an application into parts is not guaranteed to give you these benefits. If the cohesion is low and the coupling high between modules it is likely to give you more pain than gain. But there are more ways you can kill the benefits. If you break through abstractions, you likely will be broken when the implementation behind those abstractions change. Unfortunately, there are patterns in Java, mainly class loading related, that are fundamentally not modular and can easily kill the benefits of modularity.
However, if you refactor that application and you hit that right composition of modules, I can ensure you that you will immediately know why it is worth it.
Peter Kriens
OSGi Blog
Friday, August 28, 2009
Wednesday, August 19, 2009
A Simple Module System
To progress the work in JSR 294, BJ Hargrave, Richard Hall and undersigned have submitted a proposal for a simple module system. The purpose of this module system is to simple enough that can be supported both by Jigsaw and OSGi. In reality, this will likely be sufficient for most people. More advanced features, for example some of the special requirements that Jigsaw has with regard to the JDK modularization, can then be supported in a module system specific way.
This proposal is based on the module accessibility keyword, as is the current proposal, and there are no additional visibility rules. Module boundaries are defined by the artifact in which the modules are delivered. Modules can require other modules that must then become completely visible to the requiree.
The approach of finding a common denominator looks very promising. I am looking forward to continue working with the JSR 294 EG to iron out all the details.
Peter Kriens
This proposal is based on the module accessibility keyword, as is the current proposal, and there are no additional visibility rules. Module boundaries are defined by the artifact in which the modules are delivered. Modules can require other modules that must then become completely visible to the requiree.
The approach of finding a common denominator looks very promising. I am looking forward to continue working with the JSR 294 EG to iron out all the details.
Peter Kriens
Monday, July 6, 2009
No Pain No Gain
Being in the spotlight for OSGi has many good sides but it always pains when you get heavily criticized for a presentation about OSGi & Java Modularity that you gave. However, you usually learn more from criticism than from praise. Lets see if we can digest the criticism in William's blog and learn something from it.
From the first part I can see the criticism that OSGi is old technology and we keep (re)inventing component models. I guess there is a fine line between old and mature but it seems a tad unfair to hold age, without any further qualifications, against us. On the contrary, for stability, robustness, and usability a proven technology seems hard to beat. Then again, a promise is always more luring than hard reality I guess.
With the criticism that OSGi has many component models I cannot but disagree. OSGi has a single component model, since day one, and nothing has changed since then. True, on top of the OSGi component models we have different programming models because the needs in that area are quite diverse. However, all these programming models work seamlessly together through the single and only OSGi bundle and service model.
So what's left. In the end, William Louth makes a valid and interesting point: I said in my presentation that going modular is not without pain, class loaders are the major culprits in Java. However, I do not think this is a problem with OSGi, it is a fundamental problem when you modularize applications. Modules cannot see every class in the application, this global view is the antithesis of modularity. However, almost all class loader tricks I see that cause problems require this global view of the class path. Without addressing this problem, I do not think we can reap the fruits of going modular. How can Jigsaw will be able to solve these problems without destroying the benefits of modularity? Moving us from class path hell into module path hell.
The problem we're facing is that class loaders have become a very successful way to extend applications, something they were never designed for. In OSGi, we started with an extension model that makes the coupling between the modules very explicit: services. The class loading architecture was designed to support these services. The OSGi service model allows modules (bundles) to be very self contained and, ideally, only to export packages that are used to define the service.
So the big problem the OSGi faces is how to handle legacy code that heavily depends on class loaders for what OSGi does with services.
I spent considerable time thinking about this. Together with others, we created an RFP investigating all problems we could lay our hands on. My personal conclusion is that an application that requires a global unrestricted view of the class path cannot be modular. Yes, there are hacks that allow you to find the class your class loader need, but these hacks revert to class searching which implies that valuable properties like version constraints and class space constraints are no longer taken into account. My fear is that providing easy "solutions" to these practices will destroy the modular value that OSGi provides. I have found that these visibility problems are very fundamental. Time will tell if Sun is able to solve this problem with their "... much improved replacement."
Peter Kriens
From the first part I can see the criticism that OSGi is old technology and we keep (re)inventing component models. I guess there is a fine line between old and mature but it seems a tad unfair to hold age, without any further qualifications, against us. On the contrary, for stability, robustness, and usability a proven technology seems hard to beat. Then again, a promise is always more luring than hard reality I guess.
With the criticism that OSGi has many component models I cannot but disagree. OSGi has a single component model, since day one, and nothing has changed since then. True, on top of the OSGi component models we have different programming models because the needs in that area are quite diverse. However, all these programming models work seamlessly together through the single and only OSGi bundle and service model.
So what's left. In the end, William Louth makes a valid and interesting point: I said in my presentation that going modular is not without pain, class loaders are the major culprits in Java. However, I do not think this is a problem with OSGi, it is a fundamental problem when you modularize applications. Modules cannot see every class in the application, this global view is the antithesis of modularity. However, almost all class loader tricks I see that cause problems require this global view of the class path. Without addressing this problem, I do not think we can reap the fruits of going modular. How can Jigsaw will be able to solve these problems without destroying the benefits of modularity? Moving us from class path hell into module path hell.
The problem we're facing is that class loaders have become a very successful way to extend applications, something they were never designed for. In OSGi, we started with an extension model that makes the coupling between the modules very explicit: services. The class loading architecture was designed to support these services. The OSGi service model allows modules (bundles) to be very self contained and, ideally, only to export packages that are used to define the service.
So the big problem the OSGi faces is how to handle legacy code that heavily depends on class loaders for what OSGi does with services.
I spent considerable time thinking about this. Together with others, we created an RFP investigating all problems we could lay our hands on. My personal conclusion is that an application that requires a global unrestricted view of the class path cannot be modular. Yes, there are hacks that allow you to find the class your class loader need, but these hacks revert to class searching which implies that valuable properties like version constraints and class space constraints are no longer taken into account. My fear is that providing easy "solutions" to these practices will destroy the modular value that OSGi provides. I have found that these visibility problems are very fundamental. Time will tell if Sun is able to solve this problem with their "... much improved replacement."
Peter Kriens
Tuesday, June 23, 2009
Hi, We're OSGi. We mean no harm
Dear James,
I read your interesting interview with eWeek. There were many parts where we agreed, but I was also slightly puzzled with your observations about OSGi. "OSGi is this thing that kind of came from a different universe that's being used for modularity." I do agree that many people see the quality of the OSGi specs as out this world, but that seems a bit exagerated. I do agree we did not start out in the enterprise application space, we started in embedded world where space and performance constraints are pervasive, but after all I thought we both lived in the Java universe. However, this seems at odds with your remarks like "So we needed something that was a lot lighter weight." and "... OSGi's just too much fat."
Hmm, the OSGi core API is 27 classes. That is a all. Security, Module layer, Life cycle layer, and Service Layer. Exceptions, permissions, and interfaces. And one of them is even deprecated! Just the module layer in Jigsaw seems to have more classes, and they just got started ... Or did you mean the implementations? With Concierge at around 80k for an R3 implementation and Felix at 350k it seems a stretch to call us fat? OSGi is even deployed in smart cards.
It's true, our documentation is a bit fat. The core is described in 300 pages. Though we have lots of pictures! And we have virtually no errata, despite the fact that OSGi has been used in tens of thousands of applications over the last decade.
I'd like to tell you a little anecdote. In 1997 I tried to convince Ralph Johnson about Java. My key argument was that Java was so nicely small and therefore easy to understand. Only 11 packages! Ralph, a famous Smalltalker, looked at me wearily and said: "Just wait." Oh boy, was he right. That lesson still drives me every day to keep OSGi lean and mean, annoying many people along the way, but I guess that is the price one needs to pay.
If you think project Jigsaw will be leaner than OSGi, well, modularity is a problem where size does matter. You cannot demonstrate modularity with a Hello World because modularity solves the problem of large evolving code bases. [deleted]
So, James, I think that are lots of details where we did not get it perfect, but OSGi's weight is not one of them. The misconceptions about OSGi at Sun stand to cost our industry a lot of money and pain in the coming years. Project Jigsaw's simplicity is a fallacy, hidden by the fact that they do not address the hard issues that OSGi has now been working on for over a decade. All major application servers are today based on OSGi, not because it was fun or a hype, but because they had no choice. These are applications in themselves that have a scale where modularity is not an option but a necessity. The success of open source will move many Enterprise applications in this same realm.
If you believe that a simplistic solution can address the needed scale, well than indeed we do live in another universe. However, please remember that we're here as friends, to help, and mean no harm.
Your's sincerely,
Peter Kriens
I read your interesting interview with eWeek. There were many parts where we agreed, but I was also slightly puzzled with your observations about OSGi. "OSGi is this thing that kind of came from a different universe that's being used for modularity." I do agree that many people see the quality of the OSGi specs as out this world, but that seems a bit exagerated. I do agree we did not start out in the enterprise application space, we started in embedded world where space and performance constraints are pervasive, but after all I thought we both lived in the Java universe. However, this seems at odds with your remarks like "So we needed something that was a lot lighter weight." and "... OSGi's just too much fat."
Hmm, the OSGi core API is 27 classes. That is a all. Security, Module layer, Life cycle layer, and Service Layer. Exceptions, permissions, and interfaces. And one of them is even deprecated! Just the module layer in Jigsaw seems to have more classes, and they just got started ... Or did you mean the implementations? With Concierge at around 80k for an R3 implementation and Felix at 350k it seems a stretch to call us fat? OSGi is even deployed in smart cards.
It's true, our documentation is a bit fat. The core is described in 300 pages. Though we have lots of pictures! And we have virtually no errata, despite the fact that OSGi has been used in tens of thousands of applications over the last decade.
I'd like to tell you a little anecdote. In 1997 I tried to convince Ralph Johnson about Java. My key argument was that Java was so nicely small and therefore easy to understand. Only 11 packages! Ralph, a famous Smalltalker, looked at me wearily and said: "Just wait." Oh boy, was he right. That lesson still drives me every day to keep OSGi lean and mean, annoying many people along the way, but I guess that is the price one needs to pay.
If you think project Jigsaw will be leaner than OSGi, well, modularity is a problem where size does matter. You cannot demonstrate modularity with a Hello World because modularity solves the problem of large evolving code bases. [deleted]
So, James, I think that are lots of details where we did not get it perfect, but OSGi's weight is not one of them. The misconceptions about OSGi at Sun stand to cost our industry a lot of money and pain in the coming years. Project Jigsaw's simplicity is a fallacy, hidden by the fact that they do not address the hard issues that OSGi has now been working on for over a decade. All major application servers are today based on OSGi, not because it was fun or a hype, but because they had no choice. These are applications in themselves that have a scale where modularity is not an option but a necessity. The success of open source will move many Enterprise applications in this same realm.
If you believe that a simplistic solution can address the needed scale, well than indeed we do live in another universe. However, please remember that we're here as friends, to help, and mean no harm.
Your's sincerely,
Peter Kriens
Friday, June 12, 2009
Classpath hell just froze over?
Other blogs seems to drive this blog nowadays. Just read Classpath hell just froze over. This raises the questions where OSGi stands in the relation with JSR 294. I am not speaking from an official OSGi point of view, but I can of course give my personal opinion. I therefore posted a comment on the original blog but it turned out to be its own blog ...
JSR 294 contains 2 parts:
The module-info.java file? I have my severe doubts and have expressed this in the expert group. I do not think we fully understand the interactions with IDEs like Eclipse, IntelliJ, Netbeans, JDeveloper, etc. Currently, the command line with javac is completely driving the design while imho the IDE will be the common case for anyone needing modularity. Nobody needs modularity for a Hello World program, and letting this use case drive the design seems plain wrong.
One of the problems with this approach that I'm seeing is that a lot of dependencies are already specified in the sources (import package anyone?) and that a smart IDE can help find the proper modules to import those packages from. However, selecting package for import needs a wide range of modules because when you're developing, you want completion support in the IDE. However, when you have compiled your code, the compiler knows exactly which module was selecrted out of this wide scope it was compiled against.
And not to forget the build tools, they will start having to interpret the module-info file and link to the appropriate module system to find their class path. Today, a build tool tells the compiler its class path, in the future it would first have to compile or interpret the Java file. This alone will probably kill 90% of the ant scripts because the class path is used in other places then compiling. Also maven will have to start to interact with this.
This is not all. The current approach for module-info.java is creating a meta-module system. Both Jigsaw, OSGi, and any other module systems can put their metadata in the file. This is the famous design by committee problem: let's each have it our own way and make the other optional. Good for vendors, bad for users. This is something that I am always fighting inside the OSGi. Having a meta-module system will cause severe fragmentation on the module layer. Some people hope for runtime interaction between module systems. Well, this will be very hard, if not impossible. Java module systems are just to complex to be able to map one to another without causing some severe pain.
Then about Jigsaw. It is very focused on breaking up the JDK into modules. I like the native installers but I dislike the fact that they put this native packaging stuff in my face. Java should abstract the platform so I can write code for any platform and distribute my code in a singular form. It's Java's original promise to deploy and manage this code on all the variety of VMs/OSs/packagers out there. There are hundreds of VM's and even more VM-package manager combinations. There is no way anybody can support all of these combinations. Write once, deploy everywhere, and then run anywhere? This all goes against the original promise and architecture of Java.
Jigsaw is too simple for the kind of applications in the enterprise space. Most of the class path problems are still in the module-path: split packages, no real hiding of classes (they will be protected by the module keyword only, not like OSGi invisible to other bundles), no multiple versions of the same JAR to solve hard dependency problems in large applications. Looking at Mark Reinhold's slides I think he agrees, Enterprise applications should be build on OSGi. However, small applications do not have to bother with modularity, so why allow Jigsaw to be used for applications at all? Unfortunately, if Jigsaw becomes part of the JDK delivery, people will start using it, causing immediate irreversible fragmentation.
Ok, to summarize. The module keyword is heavily supported by me and other OSGi people I know. It would allow us to put the bundles in the accessibility check of the Java VM. We're not there yet, but it looks good.
From a module system point of view I think we're moving into a direction of a meta-module system, where one of the two users of this meta-module system has an awful lot of homework left to do. After ten years of working on OSGi I would not yet dare to say that I completely understand Java module systems, I therefore shudder of the thought of a meta-module systems ...
Hope this helps to enlighten some issues.
Peter Kriens
JSR 294 contains 2 parts:
- the module keyword
- module-info.java, with dependencies
The module-info.java file? I have my severe doubts and have expressed this in the expert group. I do not think we fully understand the interactions with IDEs like Eclipse, IntelliJ, Netbeans, JDeveloper, etc. Currently, the command line with javac is completely driving the design while imho the IDE will be the common case for anyone needing modularity. Nobody needs modularity for a Hello World program, and letting this use case drive the design seems plain wrong.
One of the problems with this approach that I'm seeing is that a lot of dependencies are already specified in the sources (import package anyone?) and that a smart IDE can help find the proper modules to import those packages from. However, selecting package for import needs a wide range of modules because when you're developing, you want completion support in the IDE. However, when you have compiled your code, the compiler knows exactly which module was selecrted out of this wide scope it was compiled against.
And not to forget the build tools, they will start having to interpret the module-info file and link to the appropriate module system to find their class path. Today, a build tool tells the compiler its class path, in the future it would first have to compile or interpret the Java file. This alone will probably kill 90% of the ant scripts because the class path is used in other places then compiling. Also maven will have to start to interact with this.
This is not all. The current approach for module-info.java is creating a meta-module system. Both Jigsaw, OSGi, and any other module systems can put their metadata in the file. This is the famous design by committee problem: let's each have it our own way and make the other optional. Good for vendors, bad for users. This is something that I am always fighting inside the OSGi. Having a meta-module system will cause severe fragmentation on the module layer. Some people hope for runtime interaction between module systems. Well, this will be very hard, if not impossible. Java module systems are just to complex to be able to map one to another without causing some severe pain.
Then about Jigsaw. It is very focused on breaking up the JDK into modules. I like the native installers but I dislike the fact that they put this native packaging stuff in my face. Java should abstract the platform so I can write code for any platform and distribute my code in a singular form. It's Java's original promise to deploy and manage this code on all the variety of VMs/OSs/packagers out there. There are hundreds of VM's and even more VM-package manager combinations. There is no way anybody can support all of these combinations. Write once, deploy everywhere, and then run anywhere? This all goes against the original promise and architecture of Java.
Jigsaw is too simple for the kind of applications in the enterprise space. Most of the class path problems are still in the module-path: split packages, no real hiding of classes (they will be protected by the module keyword only, not like OSGi invisible to other bundles), no multiple versions of the same JAR to solve hard dependency problems in large applications. Looking at Mark Reinhold's slides I think he agrees, Enterprise applications should be build on OSGi. However, small applications do not have to bother with modularity, so why allow Jigsaw to be used for applications at all? Unfortunately, if Jigsaw becomes part of the JDK delivery, people will start using it, causing immediate irreversible fragmentation.
Ok, to summarize. The module keyword is heavily supported by me and other OSGi people I know. It would allow us to put the bundles in the accessibility check of the Java VM. We're not there yet, but it looks good.
From a module system point of view I think we're moving into a direction of a meta-module system, where one of the two users of this meta-module system has an awful lot of homework left to do. After ten years of working on OSGi I would not yet dare to say that I completely understand Java module systems, I therefore shudder of the thought of a meta-module systems ...
Hope this helps to enlighten some issues.
Peter Kriens
Wednesday, June 10, 2009
OSGi Case Studies == Pain?
During JavaOne Atlassian presented a case study of OSGi and offered Atlassian Plugins. The presentation was very dualistic. One one side it was very positive towards OSGi but it also was (overly) critical, mostly because they ran in many problems with legacy code. This caused reactions, like the blog OSGi Case Studies == Pain.
As the OSGi evangelist I would probably say it a bit sweeter, but in principle the blog is right: don't use OSGi unless the benefit offsets the cost. Which should of course be true for everything you do. If you have a legacy application of a couple of thousands lines of code, don't bother with OSGi, it will probably be just in your way at its current incarnation.
The reason for the pain is not some unnecessary complexity in OSGi, on the contrary, with less than 30 well documented classes OSGi is actually quite simple in contrast with almost any other API I know. However, enforced modularity is painful because it confronts you with all the entanglements in your code, and even worse, the hacks and shortcuts in the libraries you use. Strong modularity puts your code in a straightjacket, and often that is no fun when you have legacy code that enjoys the anarchy of the Java class path.
Is that pain worth it? Well, both industry and science seems to have clear consensus that modularity provides lots of benefits for large applications. All the major Java app servers are not based on OSGi because they think it is fancy (notice that most do not provide the API to application developers), the sheer size of these systems just require strong modularity to survive their evolution.
Eli Whitney was one of the key drivers of the industrial revolution because he had the idea of interchangeable parts for guns. However, it took the effort of many others and long years before this simple idea could finally be put into practice. During that transition, the skilled gun making craftsmen of those days poked fun of the whole idea and told each other jokes about how interchangeable parts weren't. I think OSGi is that simple idea of interchangeable parts for software. It is a very, very hard problem, but I am convinced it is worth pursuing. Once you have experienced how you can mix and match bundles to construct a large part of an application it is imposible to ever go back.
OSGi is by far the furthest along this idea but I am the first to admit we're not there. It will require more time and effort of many, including you, to modularize our codebases and develop the tools to simplify that process. So yes, it can be a pain to move legacy code towards strong modularity but I am sure that when your applications are big enough the gain will more than compensate the pain. As many can already testify, including Atlassian.
Peter Kriens
As the OSGi evangelist I would probably say it a bit sweeter, but in principle the blog is right: don't use OSGi unless the benefit offsets the cost. Which should of course be true for everything you do. If you have a legacy application of a couple of thousands lines of code, don't bother with OSGi, it will probably be just in your way at its current incarnation.
The reason for the pain is not some unnecessary complexity in OSGi, on the contrary, with less than 30 well documented classes OSGi is actually quite simple in contrast with almost any other API I know. However, enforced modularity is painful because it confronts you with all the entanglements in your code, and even worse, the hacks and shortcuts in the libraries you use. Strong modularity puts your code in a straightjacket, and often that is no fun when you have legacy code that enjoys the anarchy of the Java class path.
Is that pain worth it? Well, both industry and science seems to have clear consensus that modularity provides lots of benefits for large applications. All the major Java app servers are not based on OSGi because they think it is fancy (notice that most do not provide the API to application developers), the sheer size of these systems just require strong modularity to survive their evolution.
Eli Whitney was one of the key drivers of the industrial revolution because he had the idea of interchangeable parts for guns. However, it took the effort of many others and long years before this simple idea could finally be put into practice. During that transition, the skilled gun making craftsmen of those days poked fun of the whole idea and told each other jokes about how interchangeable parts weren't. I think OSGi is that simple idea of interchangeable parts for software. It is a very, very hard problem, but I am convinced it is worth pursuing. Once you have experienced how you can mix and match bundles to construct a large part of an application it is imposible to ever go back.
OSGi is by far the furthest along this idea but I am the first to admit we're not there. It will require more time and effort of many, including you, to modularize our codebases and develop the tools to simplify that process. So yes, it can be a pain to move legacy code towards strong modularity but I am sure that when your applications are big enough the gain will more than compensate the pain. As many can already testify, including Atlassian.
Peter Kriens
Wednesday, June 3, 2009
Subscribe to:
Posts (Atom)