OSGi and Transitive Dependencies - apache-felix

I’m using Felix Framework for my OSGi project, but I’ve came across a severe problem concerning third party dependencies.
I’m using eclipse and maven-bundle-plugin to generate my bundles from the sources and the MANIFEST.MF from the POM.XML file. So far so good. however when I have some third party dependency in my bundle, I find myself looking for an infinite list of JARs, which usually are not bundles, and putting them in my /bundle Felix directory until no more dependencies are missing.
I call this process “Downloading the Internet for my OSGi application to work”.
What am I doing wrong? Sure I must be doing something very wrong, because I can’t imagine anyone having a bundle A that depends on B, which then depends on C and D, and then those two will depend on several others and so on… to go look for ALL those dependencies manually using google or maven central! That's insane!
What is the correct way to automate this? I would love to have one of the two solutions:
1) Be able to create a massive JAR file with all of its dependencies embedded, but exporting only the packages I want, and, of corse, not importing any package.
2) (My preferred solution) Having a way to get all my dependencies into individual JAR files that I can simply paste into the /bundle directory.
3) (Even more preferred) Having a way to use third party JARs without downloading 8GB of dependencies to my project.
I have found tools that do me this, but just for direct (1st degree) dependencies, leaving transitive dependencies for me to solve manually.
This problem is critical. The lack of such a tool hampers the usage of OSGi. I’ve searched and searched and searched, I’ve came across all the 101 solutions such as PAX, bndtools, and friends, but it seems that they do not solve this issue…
Please help me. Please provide a living example if you can, people like me around the world will benefit from the solution to this problem.
Thanks!
-
-
Edit: I am attaching a example project where I'm trying to use JScience but the resulting JAR bundle keeps asking me for new Imports, i.e., it is not self-contained.
Link to the example: https://www.dropbox.com/s/svo3nu3vawvv2xn/RequireJscienceExample.zip?dl=0
I usually try to convert 3rd party JARs into bundles using Eclipse, but they always have to Import packages that I do not have, so it is an endless situation as you have stated.
I could not find any documentation concerning the tag Conditional_Package for maven-bundle-plugin. However related searches show me the inline option that I have tried before without success.
I created a basic project where I have one class using JScience library, and in its POM.XML I have the following:
<plugins>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<version>2.3.7</version>
<extensions>true</extensions>
<configuration>
<instructions>
<Bundle-SymbolicName>${project.artifactId};singleton:=true
</Bundle-SymbolicName>
<Bundle-Version>${project.version}</Bundle-Version>
<Export-Package>shared.properties.api, shared.properties.base
</Export-Package>
<Embed-Dependency>!org.osgi.*;scope=compile|runtime;inline=true</Embed-Dependency>
<Embed-Transitive>true</Embed-Transitive>
</instructions>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
</plugin>
</plugins>
I am saying maven to inline all packages which are not from the osgi framework itself. And Looking at the resulting JAR it looks pretty good, I now have embedded only packages instead of entire JARs (however it looks to me that I didn’t need all those inlined packages since I’m using just two of them). Moreover, if I open the MANIFEST.MF file I can see this problematic line:
Manifest-Version: 1.0
Bnd-LastModified: 1414164534170
Build-Jdk: 1.6.0_65
Built-By: Pedro
Bundle-ManifestVersion: 2
Bundle-Name: RequireJscienceExample
Bundle-SymbolicName: RequireJscienceExample;singleton:=true
Bundle-Version: 0.0.1.SNAPSHOT
Created-By: Apache Maven Bundle Plugin
Embed-Dependency: !org.osgi.*;scope=compile|runtime;inline=true
Embed-Transitive: true
Import-Package: org.joda.convert,org.xml.sax <------ Problem...
Tool: Bnd-1.50.0
saying that I’m missing org.joda.convert and org.xml.sax.
What amazes me is that we are talking about a library (JScience) that states being OSGi compatible: http://jscience.org/
What am I missing? I really cannot afford not using JScience. And I have rejected several 3rd party libraries before, that would spare me development time, because of these OSGi 3rd party integration difficulties.

Why not just let Maven to resolve transitive dependency and download them for you.
Once you add them in pom.xml, IDE like Eclipse (m2e plugin actually) can already resolve, download and show resulted jars. (You can use mvn dependency:tree from command line as well)
Then review, and exclude unwanted, e.g. optional or with packages already exported by other bundle.
And yes, use provided scope e.g. for org.osgi.*
<Embed-Dependency>*;scope=compile|runtime;inline=true</Embed-Dependency>
<Embed-Transitive>true</Embed-Transitive>

Related

GWT and SpringBoot - Is there a smart way to deal with conflicting dependencies?

In an ideal world, you have your GWT app compiled to javascript, you serve it as static resource and you behind the scenes you have your back end code running on a JVM, and life goes well.
But that ideal world is called production during runtime.
However, during development time, when you would like to make use of the gwt code server...
You're GWT compile time dependencies are needed during runtime (sources + classes), for debugging and recompilation purposes of the GWT module.
At the same time, you may wish to have back-end supported by something like spring-boot 1.3.5.RELEASE.
In this case, spring boot, suffering multiple frequent releases, is at this point in time wanting to add as managed dependency, for example:
<hibernate-validator.version>5.2.4.Final</hibernate-validator.version>
Which, of course, is a very good thing. It is one of the many good features of spring, in fact.
Gwt, on the other hand, see link bellow,
http://www.gwtproject.org/doc/latest/DevGuideValidation.html
is still requiring you to use:
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-validator</artifactId>
<version>4.1.0.Final</version>
<scope>runtime</scope>
</dependency>
Now, if you were to say: I do not care about productive development, just compile me something that works properly in production.
Then indeed, you can trivially solve the above problem by structuring your project in such manner:
ROOT
--- Backend-api
--- Backend-Impl
--- Gwt-Front-end
---- War or stand alone Spring jar module to glue it all together
Where you make the Gwt-Front end depend on the back-end APIs, for some RPC services and DTOs. The pom.xml dependencies you can for the most part manage only in your front-end module, independently of whichver dependencies you have in the back-end.
Ultimately, you make a war, or a spring boot runable jar, that carries your gwt code as static resources and carries your back-end code with all of its dependencies, namely Hirbernate validator latest version.
However, when you are trying to get a pom that works for development purposes, as far as i can see, you are stuck having to globally to manage the dependencies that are common between the back-end and the front-end layer in the ROOT pom.xml, and downgrading your dependencies to the version required by gwt.
That is, in the ideal world scenario.
You have your ROOT pom.xml simply declaring your modules and the order by which they get build. And you get your back-end-impl to have the power to state it wants to inherit dependencies from the spring-boot-starter pom.xml, etc...
In contrast to the ideal scenario, on the pom.xml configuration that actually helps you during the development time...
Well, you can have to revisit the Root pom.xml.
And you have to add in managed dependencies on this root pom.xml, so that for all those common conflicting dependencies between your GWT front-end and spring boot back-end, you always have to hammer the version of the GWT and put a downgrade in place.
This will ensure that when you do gwt:run, dev mode recompile, etc... you do not end up having the gwt compiler trying to javascript compile your Hibernate version 5, but instead the version that is indeed supported by GWT 4.1 final. Of course you might end up having a surprise in back-end code, one of these days by puting in place such a hack...
Does anybody have an idea of how to properly organize a multi module project where back-end and gwt based front-end are allowed to have conflicting depency requirements?
Ultimately, if the answer is no, I believe I will prefer to have network wastage and added communication delay, by having a pure stand-alone spring boot backend that integrates with a pure gwt stand alone jetty, where the back-end on this jetty does nothing more than dumbly kick the requests to the actual spring-boot backend. It is kind of pathetic to have gwt jetty back-end that gets called to do 1+1 by the UI and which forwards the computation to a second back-end running sprin boot that actual knows how to do the 1+1... but if this is the most productive way of working, ensuring that development is productive and the production will run without surprises, so be it.
Thanks for any feedback, and Ideally, I would like to see a multi-pom project that you can import as existing maven project into eclipse, and demonstrates how this can be achieved.
Figure out where the transitive dependency is that you want to exclude when compiling GWT code.
<dependency>
<groupId>spring-dependency-with-hibernate</groupId>
<artifactId>some-spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-validator</artifactId>
</exclusion>
</exclusions>
</dependency>
This way only the GWT provided hibernate version will be available at runtime.
Yes, what I wanted to do can in fact be done, though not trivially... not at all.
To setup a maven multi module pom that can be used productively in the sense that you can get code swapping both for back-end and front-end code without needing to choose either the gwt validator dependencies, or the springboot validator dependencies, essentially I organize my project as follows.
You have:
LEVEL 1: myproject-root
----- LEVEL 2: myproject-backend
---------- LEVEL 3: myproject-backend-api
---------- LEVEL 3: myproject-backend-impl
------ LEVEL 2: myproject-frontend
------ LEVEL 2: myproject-runner
-----------LEVEL 3: myproject-runner-jar
-----------LEVEL 3: myproject-runner-war
In myproject-root, you declare the submodules, naturally.
You do dependency management on some inocuous plugins such as surefire plugin. You control java source and target compile language. You do very little more than that. If you have some plugins that you can configure cross cutingly, you might as well put those in the plugin management.
In the myproject-backend, you essentially make it inherit into the dependency management the springboot depencies, by adding to your dependency management:
<dependencyManagement>
<dependencies>
<!-- Inherit dependencies from spring boot dependency management pom -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>${version.spring.boot}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
You declare the two submodules, and you're out.
You do not declare any cross cuting dependencies here. You only use the dependency management. The front-end component will be hiting the api submodule, and you do not want to be creating a dependency magnet in the backend pom, so here you are conservative as much as possible.
In the myproject-backend-api, you start by letting this submodule be a jar sub-module. Down the line, you will want to branch the api component further, by subdividing api-s by entrypoint nature.
To begin with the -api component will have apis needed for the communication between server and frontend, kind of like the typically seen "shared" in the documentation.
Here you are as careful as possible to have as little as possible in what regards to elements in the dependencies element. The front-end will be using these api's and you do not want your api package depending on any spring boot apis.
In the backend-impl, you write your actual business logic and you are free to depend on the full spring boot stack. The rule is, make sure no one ever points to this dependency magnet, especially, never let the gwt component even smell the existence of springboot libraries. Do not refer it ever.
In the myproject-frontend, you do not branch it into different sub-modules, as you might have done with the api component.
Here you make a monolitic component for gwt code so that you can more easily have the gwt maven plugin working for you doing your hot code recompilation.
In this component you depend on the myproject-api, as well as on any relevant gwt library, such as gwt-user, hibernate validator with the old version needed by gwt, etc...
The rule for the myproject-frontend is to declare every single dependency in the component as optional. You do not want to take any transitive depency on the component.
In production all you need is the myproject-frontend.jar/META-INF/resources/MyProjectModule.html...
And all such static resources that springboot will have to detect and serve.
While you are in development mode, however, what you want to do in this front end component is to:
(a) tune your gwt-plugin to not run the jetty server
<noServer>true</noServer>
The point being, if run this jetty server the only thing it would have in its classpath would be the front end code and the api, and definitely not the impl code.
And second, you do not want to use the gwt jetty, you either want to use the springboot tomcat or the springboot jetty. Much better to use the embedded containers of springboot with the appropriate websocket libraries and such.
(b) With springboot you want to server your static resources from the META-INF/resources/ or the public folder. I prefer the resources as it would be where you would normally put JSF components as well.
In any case, because the place static resources come from is special, you want to tune your gwt plugin to compile your java sources into the write folder.
<webappDirectory>${project.build.directory}/classes/META-INF/resources</webappDirectory>
So something like above.
Finally, and this was my main mistake, you do not run your code from where you write your front-end code.
That is wrong, because that forces your front-end code to need springboot depencies, which will cause you extreme pain due to conflicting libraries.
So for dodging this problem, you create the myproject-runner component, and if you want to be very detailed, you make that one a pom parent component so that you can fork your packaging into either jar or war.
I personally like the executable jar, much more to the point, than the war option. I will never want to deploy a sprinboot application in something like a heavy weight weblogic... but whatever rocks your boat.
In any case, the spring boot runner is your ultimate dependency magenet.
This component will depend on about every module you have and their transitive dependencies. But because you separate your runner component from the front-end code, and you make all the dependencies in the front-end code optinal...
Well, you essentially only bundle springboot, and the your gwt static resources.
While you are developing, what you do is .... once you know it ... simple.
(A) you start your spring boot jar application by triggering the main file. If you really want to deploy a war file, that-s your choice you can also build a war file and deploy it to a tomcat or whatever...
(b) you go to your frontend component, rirght click it, chooose run as Maven Build, and gwt:run.
The gwt run will start up the gwt code server, that will be completely blind to your backend code, the only thing it will see in front of its eyes is the code your have in the frontend component as well as all the gwt dependencies you've added in as optinal.
You can finally hot swap code in the back-end if you-re using springboot dev.
You can finally hot swap code in that monolitic frontend component, once you-ve started teh gwt plugin.
Conclusion, it-s possible, but its a hell of a mess.
Glad this problem is out of the way.
I believe I had similar problems and discovered the following while using Eclipse IDE, GWT Google Plugin, SpringBoot and Maven.
The recipe is not simple but it worked for my team.
GWT-dev and Jetty server doesn't work with SpringBoot.
You have to fix the classpath due to Jetty and gwt-dev hardcoded dependencies, only the dependencies gwt-dev and Jetty were built against will work.
I had to split the GWT UI project into two Maven artifacts.
One artifact was a WebApp capable of running only within Eclipse, with all SpringBoot libraries excluded from the classpath.
The other artifact was a deployable SpringBoot JAR with "war" maven packaging type and a WAR overlay of the Eclipse dedicated artifact, including only GWT specicic content.
I hope this helps
Just create dedicated maven profile for running GWT codeserver and add 'gwt-dev' dependency ONLY there
<profiles>
<profile>
<id>gwt-dev</id>
<dependencies>
<dependency>
<groupId>com.google.gwt</groupId>
<artifactId>gwt-dev</artifactId>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>gwt-maven-plugin</artifactId>
<version>${gwt.version}</version>
<executions>
<execution>
<id>run-codeserver</id>
<goals>
<goal>run-codeserver</goal>
</goals>
</execution>
</executions>
<configuration>
<sourceLevel>${java.version}</sourceLevel>
<module>kn.iopm.documentserver.IDocServer</module>
<extraJvmArgs>-Xmx1G -Xms512M -Xss1G -Dlog.root=${project.build.directory}/log
</extraJvmArgs>
<runTarget>index.html</runTarget>
<persistentunitcachedir>${project.build.directory}</persistentunitcachedir>
<webappDirectory>${project.build.outputDirectory}/public</webappDirectory>
<deploy>${project.build.directory}/gwt-deploy</deploy>
<codeServerWorkDir>${project.build.directory}/gwt</codeServerWorkDir>
<launcherDir>${project.build.outputDirectory}/static</launcherDir>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>

Maven/Eclipse - Quick iteration acceptance Testing a project packaged as a WAR file

Eclipse makes working with multi module maven projects easy because you don't have to re-build and re-install a module before dependent modules will see changes to it. So you just change the code and eclipse updates the dependencies magically in the background.
I want to achieve this behaviour for acceptance testing as well.
I have
storage-service
storage-service-war
storage-service-acceptance-tests
If I use embedded jetty or tomcat to test inside the storage-service-war project then obviously code changes are immediately viewable in the tests, but I cannot see any way to achieve the same quick iteration of testing when testing from storage-service-acceptance-tests.
Every way I look at it it seems that I have to build storage-service-war and then use the artefact generated from that, but it seems like overkill when you only want to change one line.
Does anyone have a good method for doing this?
Cheers
Piers
So answering my own question :D The solution I came up with will not work on CI it will likely only work when doing a local build as it makes use of the relative paths of the projects. At the bottom I outline a more robust but more complex approach that should satisfy eclipse and CI.
I was looking at setting attachClasses to true for the war plugin configuration of the war project.
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.4</version>
<configuration>
<attachClasses>true</attachClasses>
</configuration>
</plugin>
</plugins>
You can then reference the jar in the dependent project as follows
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>storage-service-war</artifactId>
<version>${project.version}</version>
<classifier>classes</classifier>
</dependency>
Then I was thinking I could run my tests from within the acceptance test module using embedded jetty or tomcat and pointing them to the web.xml defined in the war project using a relative path.
This works fine with maven via the commandline but fails in eclipse :(
The problem with this is that the jar produced by attach classes is not picked up by the eclipse m2e integration see -https://bugs.eclipse.org/bugs/show_bug.cgi?id=365419 unfortunately it wont be fixed.
So My solution for the moment is to manually add the storage-service-war project to my acceptance test project build path in eclipse. Its not great but it works
The above solution is a bit hacky but the alternative outlined is a bit more involved.
By splitting the project into the following I think it would be possible to have correct eclipse integration and projects that work on CI
storage-service
storage-service-core
storage-service-war
storage-service-acceptance-tests
storage-service-config
The core project contains the logic and source of the webapp and is of type jar, the config contains the web.xml and any other config files and is also of type jar. Then the acceptance-tests and war project are both of type war and serve merely to package the core project into a war and extract the config to the webapp/WEB-INF dir so that they may share a common setup.

GWT maven compiler output directory

I need to set output directory location for GWT compiler using maven plugin.
I've investigated the GWT compiler and gwt-maven-plugin docs and did not find any options for this.
For example:
I have two modules named editor and console and gwt-maven-plugin compiles them to %ROOT%/editor and %ROOT%/console dirs.
But I need to have these modules compiled into %ROOT%/ria/editor and %ROOT%/ria/console dirs to reduce effort to maintain the URL based security and other stuff.
Is it possible to set ria directory in plugin configuration to avoid unneeded new ANT task creation for moving compiled modules stuff to needed place?
I had to un-check the checkbox highlighted in the below screenshot
<plugin>
<artifactId>gwt-maven-plugin</artifactId>
<configuration>
<webappDirectory>${project.build.directory}/${project.build.finalName}/ria</webappDirectory>
</configuration>
</plugin>
See http://mojo.codehaus.org/gwt-maven-plugin/compile-mojo.html#webappDirectory
It won't play well with the DevMode though, unless you run in -noserver mode (which I'd recommend anyway for any middle-sized/big project)

How do I manage a 3rd party jar dependency with Eclipse, Maven and Jenkins?

We're trying to manage a project made of multiple Eclipse plug-ins.
One plug-in has a dependency to a 3rd party plug-in. It imports a class from a library named bpmn2. This library is a jar file and NOT included in the Maven remote repository.
Locally, in Eclipse, we have the library checked out into the workspace and referenced in the classpath of the Eclipse project. The plug-in manifest doesn't explicitly state the dependency to bpmn2. Which works locally...
We try to use Jenkins for continuous integration. The Multi-Plug-in-Project is managed using Maven and multiple POM files, using the Maven tycho plug-in.
The problem is that Maven doesn't care for the locally present library bpmn2 (of course). So we thought that using Maven install:install to install bpmn2 to the local Maven repository
./mvn install:install-file -Dfile=/home/someUser/bpmn2/org.eclipse.bpmn2_0.7.0.201111021300.jar -DgroupId=org.eclipse.bpmn2 -DartifactId=bpmn2 -Dversion=0.7.0 -Dpackaging=jar -DlocalRepositoryPath=/var/lib/jenkins/localRep/
and adapting the corresponding POM with a dependency entry for the library
<dependencies>
<dependency>
<groupId>org.eclipse.bpmn2</groupId>
<artifactId>bpmn2</artifactId>
<version>0.7.0</version>
<type>jar</type>
</dependency>
</dependencies>
would work. But it didn't.
The output of Maven is:
[ERROR] Cannot resolve project dependencies:
[ERROR] Software being installed: modeltype.bpmn2 1.0.0.qualifier
[ERROR] Missing requirement: modeltype.bpmn2 1.0.0.qualifier requires 'bundle org.eclipse.bpmn2 0.7.0' but it could not be found
[ERROR]
The question is:
How could we better integrate the 3rd party library?
The local classpath reference is not the optimal solution for the greater picture, I think. Should everyone who develops for the project install the bpmn2 library and only use the dependency in the manifest?
And what are we doing wrong with Maven? The local repository is
/var/lib/jenkins/localRep
and after installing the library to the repository, it seemed that the created dir structure was okay.
/org/eclipse/bpmn2/bpmn2/0.7.0/bpmn2-0.7.0.jar
Can somebody help?
The simplest solution would be to set up a repository server for your company and proxy all your calls through that server.
I use Nexus from Sonatype.
It acts as a proxy when you need to download artifacts from remote locations, like maven central, but it also has an ability to setup repositories to store non-publcicly distributed artifacts.
The whole setup process is very well documented here -> http://www.sonatype.com/books/nexus-book/reference/.
Note, that open-source edition of the product is very good and is enough for your purposes.
Just to be fair, there is a competing product called Artifactory. You may read about it here -> http://www.jfrog.com/products.php
I had a similar scenario where I need to include a third party library (not loaded in maven repository) in Maven Web project.
Following approach helped us make the code portable.
Solution:
Created the "lib" directory under WEB_INF as below:
<<Project_Base_Dir>>/src/main/webapp/WEB-INF/lib
Copied the third party JARs to LIB directory.
Updated POM.xml to use the SYSTEMPATH as below:
<dependency>
<groupId>GROUP_ID</groupId>
<artifactId>ARTIFACT_ID</artifactId>
<version>VERSION</version>
<systemPath>${basedir}/src/main/webapp/WEB-INF/lib/<<JAR_FILENAME>></systemPath>
<scope>system</scope>
</dependency>
If you can't use Nexus, you can use scope system :
Dependency Scopes
system: This dependency is required in some phase of your
project's lifecycle, but is
system-specific. Use of this scope
is discouraged: This is considered an
"advanced" kind of feature and should
only be used when you truly understand
all the ramifications of its use,
which can be extremely hard if not
actually impossible to quantify.
This scope by definition renders your
build non-portable. It may be
necessary in certain edge cases. The
system scope includes the
<systemPath> element which points to
the physical location of this
dependency on the local machine. It is
thus used to refer to some artifact
expected to be present on the given
local machine an not in a repository;
and whose path may vary
machine-to-machine. The systemPath
element can refer to environment
variables in its path: ${JAVA_HOME}
for instance.
You can use it with
Be careful, using this is probably making your build not portable.

How to add a jar to the server lib for *embedded* JBoss?

I need to add a crypto provider to the embedded JBoss used by Seam for integration tests.
For the regular JBoss it's a simple matter, just drop the files into the /lib folder of the server instance. With the embedded JBoss, however, things seem to be different. I've tried putting the jars in /embedded-jboss/bootstrap/lib and /embedded-jboss but no change, the classes are not seen.
I've read http://community.jboss.org/wiki/EmbeddedAndJavaSE and also looked in the source of org.jboss.embedded.Bootstrap but I haven't found a way yet. The build is done with Maven if it matters.
Thanks a lot for any suggestions.
I found an answer in the meantime: they can be added to the classpath of the plugin that runs the tests, like below. Works for me.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<!-- snip -->
<additionalClasspathElements>
<additionalClasspathElement>add/here/your/jar</additionalClasspathElement>
</additionalClasspathElements>