Hi everyone,
Today its a great learning for me and my team trying to build Apache Solr within Apache Karaf. I am gonna talk about missing dependencies issue related with Karaf using Felix bundling tool in maven. This is quite common and I took example of Solr for here to explain you all.
What is Apache Karaf?
Apache Karaf is a small OSGi based runtime which provides a lightweight container onto which various components and applications can be deployed.
What is Apache Solr?
Solr is the popular, blazing fast open source enterprise search platform from the Apache LuceneTM project. Its major features include powerful full-text search, hit highlighting, faceted search, near real-time indexing, dynamic clustering, database integration, rich document (e.g., Word, PDF) handling, and geospatial search.
Creating Client Project using SolrJ
Solrj is a java client to access solr. It offers a java interface to add, update, and query the solr index. For adding this feature to my karaf container, we created a pom.xml with minimal dependency of
Problem
<!-- SOLR CORE AND ITS DEPENDENCIES -->
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-core</artifactId>
<version>4.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-solrj</artifactId>
<version>4.3.1</version>
</dependency>
Then when deploying in Karaf it showed some error when you read headers
karaf@root> headers <START_ID>
Reason
It will show you some jars and dependencies highlighted in red which means that we failed to deploy them in maven pom.xml. For this, we tried including all the dependencies that were missing and its like a fission effect where the list of missing dependencies started growing more and more. For this, there is a simple approach that will eliminate the missing dependency issue. Here is the sample of felix definition in our pom.xml
Solution
<build>
<plugins>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<version>2.3.7</version>
<extensions>true</extensions>
<executions>
<execution>
<id>bundle</id>
<phase>package</phase>
<goals>
<goal>bundle</goal>
</goals>
</execution>
</executions>
<configuration>
<instructions>
<Bundle-Name>BUNDLE NAME</Bundle-Name>
<Bundle-Description>This bundle will be used as a common registry
service across the platform</Bundle-Description>
<Bundle-SymbolicName>${project.artifactId}</Bundle-SymbolicName>
<Import-Package>${bundle.import.package}</Import-Package>
<Export-Package>${bundle.export.package}</Export-Package>
true</Embed-Transitive>
*;scope=compile|runtime;</Embed-Dependency>
<DynamicImport-Package>${bundle.dynamicImport.package}</DynamicImport-Package>
</instructions>
</configuration>
</plugin>
</plugins>
</build>
Having the property defined in real world as
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<bundle.import.package>com.**.**,*</bundle.import.package>
<bundle.private.package>com.**.**</bundle.private.package>
<bundle.export.package></bundle.export.package>
<bundle.dynamicImport.package>
org.apache.log,
all.missing.dependencies
</bundle.dynamicImport.package>
</properties>
Sample
Now we need to replace all.missing.dependencies with all the jars that are listed as red in karaf. In my case for Apache Solr, I added the below given list
<bundle.dynamicImport.package$gt;
com.sun.jdmk.comm,
com.sun.management,
com.sun.msv.grammar,
com.sun.msv.grammar.trex,
com.sun.msv.reader,
com.sun.msv.reader.trex.ng,
com.sun.msv.reader.util,
com.sun.msv.verifier,
com.sun.msv.verifier.regexp,
com.sun.msv.util,
com.vividsolutions.jts.algorithm,
com.vividsolutions.jts.geom,
com.vividsolutions.jts.io,
com.vividsolutions.jts.operation.union,
com.vividsolutions.jts.operation.valid,
com.vividsolutions.jts.simplify,
com.vividsolutions.jts.util,
javax.jms,
javax.portlet,
org.apache.log,
org.jboss.netty.bootstrap,
org.jboss.netty.buffer,
org.jboss.netty.channel,
org.jboss.netty.channel.group,
org.jboss.netty.channel.socket.nio,
org.relaxng.datatype,
org.relaxing.datatype,
sun.misc
</bundle.dynamicImport.package$gt;
Reasoning
The magic here that we added all the missing / unavailable dependencies as dynamic import to the Felix builder, that will create a manifest in the jar keeping them in dynamic import section. This will tell Apache Karaf container that the jars / packages are not needed to be part of bundle. This way, you shall eliminate all the missing dependency issue with the classes that need not be bundled along with the core jars.
Here is my Full POM file
<?xml version="1.0"?>
<project
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.kousik.package</groupId>
<artifactId>yourartifact</artifactId>
<version>0.5</version>
</parent>
<groupId>com.package</groupId>
<artifactId>artifact-impl</artifactId>
<version>0.5</version>
<name>artifact-impl</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<bundle.import.package>com.package.api,*</bundle.import.package>
<bundle.private.package>com.package.impl.ClassImpl</bundle.private.package>
<bundle.export.package></bundle.export.package>
<bundle.dynamicImport.package>
com.sun.jdmk.comm,
com.sun.management,
com.sun.msv.grammar,
com.sun.msv.grammar.trex,
com.sun.msv.reader,
com.sun.msv.reader.trex.ng,
com.sun.msv.reader.util,
com.sun.msv.verifier,
com.sun.msv.verifier.regexp,
com.sun.msv.util,
com.vividsolutions.jts.algorithm,
com.vividsolutions.jts.geom,
com.vividsolutions.jts.io,
com.vividsolutions.jts.operation.union,
com.vividsolutions.jts.operation.valid,
com.vividsolutions.jts.simplify,
com.vividsolutions.jts.util,
javax.jms,
javax.portlet,
org.apache.log,
org.jboss.netty.bootstrap,
org.jboss.netty.buffer,
org.jboss.netty.channel,
org.jboss.netty.channel.group,
org.jboss.netty.channel.socket.nio,
org.relaxng.datatype,
org.relaxing.datatype,
sun.misc
</bundle.dynamicImport.package>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.package.api</groupId>
<artifactId>search-api</artifactId>
<version>0.5</version>
</dependency>
<!-- SOLR CORE AND ITS DEPENDENCIES -->
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-core</artifactId>
<version>4.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-solrj</artifactId>
<version>4.3.1</version>
</dependency>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.5.6</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-sandbox</artifactId>
<version>4.0.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<version>2.3.7</version>
<extensions>true</extensions>
<executions>
<execution>
<id>bundle</id>
<phase>package</phase>
<goals>
<goal>bundle</goal>
</goals>
</execution>
</executions>
<configuration>
<instructions>
<Bundle-Name>Project Impl</Bundle-Name>
<Bundle-Description>This bundle will be used as a common registry
service across the platform</Bundle-Description>
<Bundle-SymbolicName>${project.artifactId}</Bundle-SymbolicName>
<Import-Package>${bundle.import.package}</Import-Package>
<Export-Package>${bundle.export.package}</Export-Package>
true</Embed-Transitive>
*;scope=compile|runtime;</Embed-Dependency>
<DynamicImport-Package>${bundle.dynamicImport.package}</DynamicImport-Package>
</instructions>
</configuration>
</plugin>
</plugins>
</build>
<repositories>
</repositories>
</project>
Also you shall download from here pom.xml
Now that we know the technique, we started usign the same approach for other services too.
Thanks,
Kousik Rajendran.
I did the same, but some things are differents for me.
1) I have more missing dependencies:
jline,
org.eclipse.jetty.server,
org.eclipse.jetty.server.bio,
org.eclipse.jetty.server.handler,
org.eclipse.jetty.server.nio,
org.eclipse.jetty.server.session,
org.eclipse.jetty.server.ssl,
org.eclipse.jetty.servlet,
org.eclipse.jetty.util.component,
org.eclipse.jetty.util.log,
org.eclipse.jetty.util.ssl,
org.eclipse.jetty.util.thread
2) This error appears: Caused by: java.lang.NoClassDefFoundError: org/apache/solr/common/params/SolrParams
Any Idea?
Can you explain more on your environment, I see the libraries that are needed to run on Jetty Server. In my project I dont use any JETTY components or libraries. You would have used them as part of your project specific needs. Check for all the imports in your Project and if that demands any Jetty server kindly add them to your in pom.xml.
Mostly you should add like this
<bundle.dynamicImport.package>
…[all old entries]…,
org.eclipse.jetty.server,
org.eclipse.jetty.server.bio,
org.eclipse.jetty.server.handler,
org.eclipse.jetty.server.nio,
org.eclipse.jetty.server.session,
org.eclipse.jetty.server.ssl,
org.eclipse.jetty.servlet,
org.eclipse.jetty.util.component,
org.eclipse.jetty.util.log,
org.eclipse.jetty.util.ssl,
org.eclipse.jetty.util.thread
</bundle.dynamicImport.package>
Check this out and post me your result.
Thanks,
Kousik Rajendran.
I’m trying to use solrj from an own bundle. I don’t need JETTY neather, but when I deploy the bundle these imports are identified as missed.
Ok, I included them as DynamicImport-Package and everything is ok, until my activator is executed, and a java.lang.NoClassDefFoundError referent to solrj appears.
I can’t understand why, because my MANIFEST.MF seems to include them as Embedded-Artifacts and as Bundle-ClassPath.
This is my POM.xml
4.0.0
xxx.jitcore.pom
osgi-pom-super
0.0.1-SNAPSHOT
xxx.domain
domain-search-solr
0.0.1-SNAPSHOT
bundle
xxx.service
service-search-api
0.0.2-SNAPSHOT
provided
org.apache.solr
solr-solrj
4.3.1
javax.servlet
servlet-api
2.5
org.apache.solr
solr-core
4.3.1
org.slf4j
slf4j-simple
1.5.6
org.apache.lucene
lucene-sandbox
4.0.0
org.osgi
org.osgi.core
4.2.0
provided
commons-logging
commons-logging
1.1.1
provided
org.apache.felix
maven-bundle-plugin
true
bundle
package
bundle
true
META-INF
${project.name}
xxx.search.solr.activator.SolrActivator
true
*;scope=compile|runtime
true
false
lib
*
com.sun.jdmk.comm,
com.sun.management,
com.sun.msv.grammar,
com.sun.msv.grammar.trex,
com.sun.msv.reader,
com.sun.msv.reader.trex.ng,
com.sun.msv.reader.util,
com.sun.msv.verifier,
com.sun.msv.verifier.regexp,
com.sun.msv.util,
com.vividsolutions.jts.algorithm,
com.vividsolutions.jts.geom,
com.vividsolutions.jts.io,
com.vividsolutions.jts.operation.union,
com.vividsolutions.jts.operation.valid,
com.vividsolutions.jts.simplify,
com.vividsolutions.jts.util,
javax.jms,
javax.portlet,
org.apache.log,
org.jboss.netty.bootstrap,
org.jboss.netty.buffer,
org.jboss.netty.channel,
org.jboss.netty.channel.group,
org.jboss.netty.channel.socket.nio,
org.relaxng.datatype,
org.relaxing.datatype,
sun.misc,
jline,
org.eclipse.jetty.server,
org.eclipse.jetty.server.bio,
org.eclipse.jetty.server.handler,
org.eclipse.jetty.server.nio,
org.eclipse.jetty.server.session,
org.eclipse.jetty.server.ssl,
org.eclipse.jetty.servlet,
org.eclipse.jetty.util.component,
org.eclipse.jetty.util.log,
org.eclipse.jetty.util.ssl,
org.eclipse.jetty.util.thread