OSGi: Go Forth And Extend

This entry is my take on a trait of the current OSGi architecture unparalleled to my knowledge in any other modular environment. Before I dive in I must introduce in brief said architecture. Detailed descriptions can be found in the public OSGi specification.

OSGi Layers

Modular

This is what OSGi is best known for. The modular layer introduces the notion of bundle. with strict visibility rules based on the bundle metadata. One interesting thing to notice is that the modular layer is static. If you code an application based on this layer you will not have any hot code update. The application would begin with a main() method contained in a bundle from where the first classes and threads will be created. These will only be classes that other bundles expose to the "main bundle". These classes can than turn around and create other classes based on what their respective bundles can see, and so the process continues. Here we can have multiple versions existing side by side and resource sharing is done via a graph of ClassLoaders rather than the traditional ClassLoader tree. The second interesting thing to notice is that this layer deals only with class sharing. E.g. we have no concept of sharing instantiated (or "live") objects between the bundles. In short the modular layer today is what project Jigsaw is going to be when Java 7 comes out. But I digress...

Lifecycle

Lifecycle binds the modular and service layers in a common state machine. This is where dynamics are introduced to OSGi. Some states deal with the modular layer and describe when a bundle can participate in class (and resource) sharing. These are "installed", "resolved", and "uninstalled". The rest deal with the service layer and describe when a bundle can participate in instance sharing (and code execution). These state are "starting", "active", and "stopping".

These two sets form two state machines with the service layer machine being embedded in the modular layer machine's "resolved" state. So we can break the lifecycle into two sublayers: modular lifecycle and service lifecycle.

Service

This is where the interesting stuff happens. Each bundle has an activation hook. When the bundle is activate the lifecycle layer calls the hook to make the Resolved->Starting->Active transition. When the bundle is stopped the lifecycle layer call the hook make the reverse Active->Stopping->Resolved transition. When activated the bundle hook object can mushroom into a runtime object structure. Some of the objects comprising this structure can be shared with other bundles. I.e. while the modular layer was about class sharing the service layer is about instance sharing. So the two layers complement each other. A bundle must first share the classes that comprise the API of another bundle before it can share runtime instances of these classes. Some of the activation hooks can also start threads to drive application control flow. To do the actual instance sharing each activation hook receives as a parameter a private accessor to the inter-bundle environment (the service registry).

Extender

The extender architectural pattern defines how a central bundle (the extender) can manage a certain facet of the logic of other bundles. This is kind of like dependency injection but instead of objects entire chunks of logic are injected! For example a Service Layer Runtime can drive the service interactions of a bundle based on metadata found in that bundle. Similarly an extender can manage the URL space of another bundle by discovering and registering it's servlets and resources with the HTTP server. Yet another extender can set up the database and the JPA entity manager. The applications of the pattern are endless. It's key elements are:

1. Track the lifecycle of other bundles.
2. Search newly resolved bundles for relevant metadata and interpret it.
3. If needed load resources from the bundle.
4. If needed load classes from the bundle and instantiate them.
5. If needed export the instantiated objects on behalf of the bundle (as services).
6. If needed import live objects (e.g. services) on behalf of the bundle.
7. If needed start threads to drive the internals of the bundle.

Looking at this list we might be reminded of the way the operations performed by the OSGi core (i.e. the "framework") . And this is exactly the case...

The bootstrap extender

The OSGi service layer follows the extender pattern quite closely:

1. Track the lifecycle of other bundles.
   We can imagine the modular layer notifies the service layer the bundle has entered the Resolved state.
2. Search newly resolved bundles for relevant metadata and interpret it.
   The service layer looks for a Bundle-Activator header in the bundle manifest and interprets it as a fully qualified class name.
3. If needed load classes from the bundle.
   The service layer loads the BundleActivator class through the bundle's own ClassLoader.
4. If needed instantiate the loaded classes.
   The BundleActivator is instantiated using it's default constructor.
5. If needed start threads to drive the internals of the bundle.
   To do this the service layer calls the activator, which can if it so chooses start threads to drive application control flow.

So there we are - the service layer of the OSGi framework is nothing more than the first ever extender. The service registry is a runtime structure maintained by this extender and is completely separate from the bundle graph maintained at the modular layer. Part of the methods of BundleContext and Bundle form a separate sub-API that has nothing to do with bundles:

• From BundleContext we have getServiceReference(),getServiceReferences(), registerService(),addServiceListener(),addRemoveListener()
• From Bundle we have getRegisteredServices(),getServicesInUse()

In fact these methods can be moved into a ServiceContext, leaving Bundle/BundleContext to deal only with resource searching and class loading.

Now any bundle can do it

Before OSGi 4.0 it was exclusively the prerogative of the OSGi core (bundle 0) to play god with other bundles. This is because the OSGi API did not provide a way for one bundle to cause another bundle to load a class. I.e. it was not possible for one bundle to drive the module-level behavior of another bundle. OSGi 4.0 gave the fire of the gods to mortal bundles via a single method called Bundle.loadClass(). This essentially enabled the extender pattern. Now a bundle can instantiate classes belonging to another bundle. Later OSGi 4.1 increased this power via the Bundle.getBundleContext() method. Now an extender can also control the service-layer interactions of another bundle.

Let us now examine what doors into the OSGi core an extender can use to implement each major step of it's operation:

1. Track the lifecycle of other bundles.
   Use BundleContext.addBundleListener(). The extender should add the SynchronousBundleListener flavor. This guarantees that all methods on Bundle and BundleContext that depend on the state of the extended bundle will work as expected. I.e. the extender will always observe the target bundle in it's current state.
2. Search newly resolved bundles for relevant metadata.
   Use Bundle.getHeaders, Bundle.getEntry , Bundle.getEntryPaths , Bundle.findEntries to search through the content of the bundle. Using these methods it is even possible to obtain every single class file in the bundle and use that information (i.e. the class names) to load the classes later on. The important trait of all these methods is that none of them pass through the ClassLoader of the extended bundle. So the search will not cause premature resource consumption.
3. Load resources from the bundle.
   Use Bundle.getResource, Bundle.getResources. These use the ClassLoader and cause class-space resolution for the extended bundle. On the other hand they can be used to search through jars embedded in the extended bundle.
4. If needed load classes from the bundle.
   Use Bundle.loadClass on the extended bundle. This uses that bundle's ClassLoader to get a Class object. I.e. the extender looks "through the eyes" of another bundle and that bundle's imports and exports alone determine if the desired class can be loaded. At the same time the loaded class does not have to be exported from the target bundle. That's right - the extender can touch into the internal classes of another bundle as long as it knows their fully qualified names. These names are obtained in the "metadata hunt" that took place in the previous phase. Notice that we can manipulate these classes because the reflection API is common to all bundles and is provided by the system ClassLoader.
5. Import/Export services on behalf of the bundle.
   Use Bundle.getBundleContext() on the manipulated bundle. To export services we must load their implementation classes as described above, create instances via reflection and than register using the context of the manipulated bundle.

Using steps 1 through 4 we can implement the OSGi service registry as a bundle. Using step 5 we can bridge our home-grown registry with the one provided by the OSGi core.

Eclipse - extenders only

In fact there is a project out there that uses these exact means to implement it's own complete replacement of the OSGi service registry. This ofcource is the Eclipse project and it's extension registry. As far as I know Eclipse does not take step 5. It is still possible to use both OSGi services and Eclipse's own extension-point/extension system in one application because bundles are still activated via the standard OSGi hook, which gives them the opportunity to step into the OSGi service layer. Despite this until recently it was relatively hard to mix extensions and services. Today the Peaberry project finally enables the seamless mixing of extensions and classic OSGi services.

Actually the entire extension-point/extension model is built on the extender pattern. Each extension point is in fact an extender - it pulls resources and data from another bundle and mixes them into a runtime structure that it hosts. Eclipse goes one step further: if each extension point must have code that hunts for metadata inside other bundles why not extract this functionality into a common extender. That extender will assemble the metadata from all bundles and provide a metadata repository accessible to everyone. This is kind of a meta-extender. In Eclipse's this metadata repository is called the the extension registry.

OSGi - extenders vs. services

Eclipse is a vivid demonstration of the maturity that OSGi has reached. With the addition of a couple of methods the OSGi API can be split almost cleanly into modular-API and service-API. Subsequently two types of bundles emerge: extenders and applications. The extenders operate almost purely on top of the modular half of the API: they pull out classes and resources from the application bundles and mix them into runtime structures. These structures do not have to reflect the underlying bundles in any way. E.g. looking at the way an extender has arranged servlets into an alias space we can no longer see the boundaries between the bundles from, which these servlets were loaded. The applications are now full of metadata that describes to various extenders, which pieces of the app need to be pulled out and mixed into the runtime structure maintained by the respective extender. A bundle that acts as an extender with respect to a certain type of application bundles can be an application with respect to another extender. Replace "extender" with "extension-point" and "application" with "extension" and we get the Eclipse runtime model. Unlike Eclipse, OSGi provides a simpler, lower level API that places no restrictions on the format of the metadata. The downside is that extenders have to do the bundle tracking and metadata parsing by themselves. For an in-depth comparison between the Eclipse and OSGi runtime models read this excellent article by Neil Bartlett

Than there is the bootstrap extender: the service layer. Unlike most extenders this one hosts a truly generic runtime structure: the service registry. In a sense the service layer pulls out objects from other bundles and mixes them into this runtime structure. The service registry does preserve the division between bundles - we can say that the service and modular layers of OSGi are "aligned" with respect to bundle boundaries. I suppose this alignment has caused people to believe the two layers are inseparable when in fact they represent two distinct but complementary halves of the OSGi runtime. Because the service registry is available on any OSGi framework bundles that can't subscribe to a specialized extender can always opt for the generic playground hosted by the OSGi service layer. I.e. (most) applications choose to use the service half of the OSGi API, but (as Eclipse demonstrates) this is no longer mandatory.

All of this brings up the question if the OSGi API is not due for a makeover. Maybe we need to explicitly split the modular and service realms? I have some wildly speculative ideas on the subject, which warrant their own blog entry. Until then - Go forth and extend!