Loading Maven dependencies for OSGi Bundles – Apache Solr / Lucene in Apache Karaf

Hi everyone,
Today its a great learning for me and my team trying to build Apache Solr within Apache Karaf. I am gonna talk about missing dependencies issue related with Karaf using Felix bundling tool in maven. This is quite common and I took example of Solr for here to explain you all.

What is Apache Karaf?
Apache Karaf is a small OSGi based runtime which provides a lightweight container onto which various components and applications can be deployed.

What is Apache Solr?
Solr is the popular, blazing fast open source enterprise search platform from the Apache LuceneTM project. Its major features include powerful full-text search, hit highlighting, faceted search, near real-time indexing, dynamic clustering, database integration, rich document (e.g., Word, PDF) handling, and geospatial search.

Creating Client Project using SolrJ
Solrj is a java client to access solr. It offers a java interface to add, update, and query the solr index. For adding this feature to my karaf container, we created a pom.xml with minimal dependency of

Problem
<!-- SOLR CORE AND ITS DEPENDENCIES -->
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-core</artifactId>
<version>4.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-solrj</artifactId>
<version>4.3.1</version>
</dependency>

Then when deploying in Karaf it showed some error when you read headers
karaf@root> headers <START_ID>

Reason
It will show you some jars and dependencies highlighted in red which means that we failed to deploy them in maven pom.xml. For this, we tried including all the dependencies that were missing and its like a fission effect where the list of missing dependencies started growing more and more. For this, there is a simple approach that will eliminate the missing dependency issue. Here is the sample of felix definition in our pom.xml

Solution
<build>
<plugins>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<version>2.3.7</version>
<extensions>true</extensions>
<executions>
<execution>
<id>bundle</id>
<phase>package</phase>
<goals>
<goal>bundle</goal>
</goals>
</execution>
</executions>
<configuration>
<instructions>
<Bundle-Name>BUNDLE NAME</Bundle-Name>
<Bundle-Description>This bundle will be used as a common registry
service across the platform</Bundle-Description>
<Bundle-SymbolicName>${project.artifactId}</Bundle-SymbolicName>
<Import-Package>${bundle.import.package}</Import-Package>
<Export-Package>${bundle.export.package}</Export-Package>
true</Embed-Transitive>
*;scope=compile|runtime;</Embed-Dependency>
<DynamicImport-Package>${bundle.dynamicImport.package}</DynamicImport-Package>
</instructions>
</configuration>
</plugin>
</plugins>
</build>

Having the property defined in real world as


<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<bundle.import.package>com.**.**,*</bundle.import.package>
<bundle.private.package>com.**.**</bundle.private.package>
<bundle.export.package></bundle.export.package>
<bundle.dynamicImport.package>
org.apache.log,
all.missing.dependencies
</bundle.dynamicImport.package>
</properties>

Sample
Now we need to replace all.missing.dependencies with all the jars that are listed as red in karaf. In my case for Apache Solr, I added the below given list

<bundle.dynamicImport.package$gt;
com.sun.jdmk.comm,
com.sun.management,
com.sun.msv.grammar,
com.sun.msv.grammar.trex,
com.sun.msv.reader,
com.sun.msv.reader.trex.ng,
com.sun.msv.reader.util,
com.sun.msv.verifier,
com.sun.msv.verifier.regexp,
com.sun.msv.util,
com.vividsolutions.jts.algorithm,
com.vividsolutions.jts.geom,
com.vividsolutions.jts.io,
com.vividsolutions.jts.operation.union,
com.vividsolutions.jts.operation.valid,
com.vividsolutions.jts.simplify,
com.vividsolutions.jts.util,
javax.jms,
javax.portlet,
org.apache.log,
org.jboss.netty.bootstrap,
org.jboss.netty.buffer,
org.jboss.netty.channel,
org.jboss.netty.channel.group,
org.jboss.netty.channel.socket.nio,
org.relaxng.datatype,
org.relaxing.datatype,
sun.misc
</bundle.dynamicImport.package$gt;

Reasoning
The magic here that we added all the missing / unavailable dependencies as dynamic import to the Felix builder, that will create a manifest in the jar keeping them in dynamic import section. This will tell Apache Karaf container that the jars / packages are not needed to be part of bundle. This way, you shall eliminate all the missing dependency issue with the classes that need not be bundled along with the core jars.

Here is my Full POM file
<?xml version="1.0"?>
<project
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.kousik.package</groupId>
<artifactId>yourartifact</artifactId>
<version>0.5</version>
</parent>
<groupId>com.package</groupId>
<artifactId>artifact-impl</artifactId>
<version>0.5</version>
<name>artifact-impl</name>
<url>http://maven.apache.org</url&gt;
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<bundle.import.package>com.package.api,*</bundle.import.package>
<bundle.private.package>com.package.impl.ClassImpl</bundle.private.package>
<bundle.export.package></bundle.export.package>
<bundle.dynamicImport.package>
com.sun.jdmk.comm,
com.sun.management,
com.sun.msv.grammar,
com.sun.msv.grammar.trex,
com.sun.msv.reader,
com.sun.msv.reader.trex.ng,
com.sun.msv.reader.util,
com.sun.msv.verifier,
com.sun.msv.verifier.regexp,
com.sun.msv.util,
com.vividsolutions.jts.algorithm,
com.vividsolutions.jts.geom,
com.vividsolutions.jts.io,
com.vividsolutions.jts.operation.union,
com.vividsolutions.jts.operation.valid,
com.vividsolutions.jts.simplify,
com.vividsolutions.jts.util,
javax.jms,
javax.portlet,
org.apache.log,
org.jboss.netty.bootstrap,
org.jboss.netty.buffer,
org.jboss.netty.channel,
org.jboss.netty.channel.group,
org.jboss.netty.channel.socket.nio,
org.relaxng.datatype,
org.relaxing.datatype,
sun.misc
</bundle.dynamicImport.package>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.package.api</groupId>
<artifactId>search-api</artifactId>
<version>0.5</version>
</dependency>
<!-- SOLR CORE AND ITS DEPENDENCIES -->
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-core</artifactId>
<version>4.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-solrj</artifactId>
<version>4.3.1</version>
</dependency>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.5.6</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-sandbox</artifactId>
<version>4.0.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<version>2.3.7</version>
<extensions>true</extensions>
<executions>
<execution>
<id>bundle</id>
<phase>package</phase>
<goals>
<goal>bundle</goal>
</goals>
</execution>
</executions>
<configuration>
<instructions>
<Bundle-Name>Project Impl</Bundle-Name>
<Bundle-Description>This bundle will be used as a common registry
service across the platform</Bundle-Description>
<Bundle-SymbolicName>${project.artifactId}</Bundle-SymbolicName>
<Import-Package>${bundle.import.package}</Import-Package>
<Export-Package>${bundle.export.package}</Export-Package>
true</Embed-Transitive>
*;scope=compile|runtime;</Embed-Dependency>
<DynamicImport-Package>${bundle.dynamicImport.package}</DynamicImport-Package>
</instructions>
</configuration>
</plugin>
</plugins>
</build>
<repositories>
</repositories>
</project>

Also you shall download from here pom.xml
Now that we know the technique, we started usign the same approach for other services too.

Thanks,
Kousik Rajendran.

Advertisements

About kousikraj
Director & Chief Enterprise Architect at Sphata Systems. Expert in building Cloud based Solutions, Product Engineering, enterprise platforms, RIA, Mobile and Healthcare Applications.

3 Responses to Loading Maven dependencies for OSGi Bundles – Apache Solr / Lucene in Apache Karaf

  1. irene says:

    I did the same, but some things are differents for me.
    1) I have more missing dependencies:
    jline,
    org.eclipse.jetty.server,
    org.eclipse.jetty.server.bio,
    org.eclipse.jetty.server.handler,
    org.eclipse.jetty.server.nio,
    org.eclipse.jetty.server.session,
    org.eclipse.jetty.server.ssl,
    org.eclipse.jetty.servlet,
    org.eclipse.jetty.util.component,
    org.eclipse.jetty.util.log,
    org.eclipse.jetty.util.ssl,
    org.eclipse.jetty.util.thread
    2) This error appears: Caused by: java.lang.NoClassDefFoundError: org/apache/solr/common/params/SolrParams

    Any Idea?

    • kousikraj says:

      Can you explain more on your environment, I see the libraries that are needed to run on Jetty Server. In my project I dont use any JETTY components or libraries. You would have used them as part of your project specific needs. Check for all the imports in your Project and if that demands any Jetty server kindly add them to your in pom.xml.
      Mostly you should add like this

      <bundle.dynamicImport.package>
      …[all old entries]…,
      org.eclipse.jetty.server,
      org.eclipse.jetty.server.bio,
      org.eclipse.jetty.server.handler,
      org.eclipse.jetty.server.nio,
      org.eclipse.jetty.server.session,
      org.eclipse.jetty.server.ssl,
      org.eclipse.jetty.servlet,
      org.eclipse.jetty.util.component,
      org.eclipse.jetty.util.log,
      org.eclipse.jetty.util.ssl,
      org.eclipse.jetty.util.thread
      </bundle.dynamicImport.package>

      Check this out and post me your result.

      Thanks,
      Kousik Rajendran.

      • irene says:

        I’m trying to use solrj from an own bundle. I don’t need JETTY neather, but when I deploy the bundle these imports are identified as missed.
        Ok, I included them as DynamicImport-Package and everything is ok, until my activator is executed, and a java.lang.NoClassDefFoundError referent to solrj appears.
        I can’t understand why, because my MANIFEST.MF seems to include them as Embedded-Artifacts and as Bundle-ClassPath.

        This is my POM.xml

        4.0.0

        xxx.jitcore.pom
        osgi-pom-super
        0.0.1-SNAPSHOT

        xxx.domain
        domain-search-solr
        0.0.1-SNAPSHOT
        bundle

        xxx.service
        service-search-api
        0.0.2-SNAPSHOT
        provided

        org.apache.solr
        solr-solrj
        4.3.1

        javax.servlet
        servlet-api
        2.5

        org.apache.solr
        solr-core
        4.3.1

        org.slf4j
        slf4j-simple
        1.5.6

        org.apache.lucene
        lucene-sandbox
        4.0.0

        org.osgi
        org.osgi.core
        4.2.0
        provided

        commons-logging
        commons-logging
        1.1.1
        provided

        org.apache.felix
        maven-bundle-plugin
        true

        bundle
        package

        bundle

        true
        META-INF

        ${project.name}
        xxx.search.solr.activator.SolrActivator
        true
        *;scope=compile|runtime
        true
        false
        lib
        *

        com.sun.jdmk.comm,
        com.sun.management,
        com.sun.msv.grammar,
        com.sun.msv.grammar.trex,
        com.sun.msv.reader,
        com.sun.msv.reader.trex.ng,
        com.sun.msv.reader.util,
        com.sun.msv.verifier,
        com.sun.msv.verifier.regexp,
        com.sun.msv.util,
        com.vividsolutions.jts.algorithm,
        com.vividsolutions.jts.geom,
        com.vividsolutions.jts.io,
        com.vividsolutions.jts.operation.union,
        com.vividsolutions.jts.operation.valid,
        com.vividsolutions.jts.simplify,
        com.vividsolutions.jts.util,
        javax.jms,
        javax.portlet,
        org.apache.log,
        org.jboss.netty.bootstrap,
        org.jboss.netty.buffer,
        org.jboss.netty.channel,
        org.jboss.netty.channel.group,
        org.jboss.netty.channel.socket.nio,
        org.relaxng.datatype,
        org.relaxing.datatype,
        sun.misc,
        jline,
        org.eclipse.jetty.server,
        org.eclipse.jetty.server.bio,
        org.eclipse.jetty.server.handler,
        org.eclipse.jetty.server.nio,
        org.eclipse.jetty.server.session,
        org.eclipse.jetty.server.ssl,
        org.eclipse.jetty.servlet,
        org.eclipse.jetty.util.component,
        org.eclipse.jetty.util.log,
        org.eclipse.jetty.util.ssl,
        org.eclipse.jetty.util.thread

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: