I am using Jenkins as my CI server for a project that makes use of a native library. The project is in scala and I am using sbt to compile and run the unit tests. One of the libraries that I am using is a java (jni) wrapper around a c-library.
I have added the location of the library to the LD_LIBRARY_PATH and location of the jar to CLASSPATH in my .bashrc so that I can run the project and unit tests from the command line.
How do I do this for Jenkins?
I have recently had a problem when copying artifacts from remote nodes, which was fixed by adding the following to the advanced setting "JVM options" of the relevant node:
-Djava.library.path=/lib/x86_64-linux-gnu/
This is quite simple and is obvious to anyone reviewing the settings, for example when replicating the node configuration to use a similar machine. I do not recommend touching system- or user-wide scripts, in general.
Related
I'm trying to run a scala/akka based program on a linux cluster machine. I was following the tutorial link (this was the only example I could find):
http://doc.akka.io/docs/akka/2.0/intro/getting-started-first-scala.html
It says to use the command to obtain the akka library:
git clone git://github.com/akka/akka.git
But it doesn't have any jars files inside it as per the tutorial.
I'm not able to get a basic akka-scala-sbt combo working on linux. So any help on this is much appreciated. I'm not aware (not able to find any clear source) of the commands needed to compile/execute with and without using SBT.
Java version: "1.8.0_31"
Scala version: "2.11.5"
Akka: I'm not sure, I did git clone, believe its the latest
SBT: 0.13.9
Java,Scala are already installed on the cluster, I had to just use module load.
You can start with this simple example: https://github.com/izmailoff/SmallAkkaProjectExample. It will help you to understand how to compile and run a project.
Usually the steps are:
Start SBT
Compile code
Run it from SBT
or:
Start SBT
Compile executable JAR
Run the JAR from command line
If you want a more advanced example take a look at: https://github.com/izmailoff/remote-akka-server-template
In either of these projects you don't need to download any libraries/JARs. SBT will download everything you need automatically.
In short, you need to understand how to build projects with SBT and how to run them - not related to Akka. Separately from that you need to know how Akka runs, i.e. ActorSystem, kernel, etc.
Not long before I had digged into Ant and use it doing simple practice(compile,unit test,package).Now my usual work is under Eclipse for java development,the usual work can all be simply done through Eclipse' function:in-time compiling、click button to deploy、various wizard..
So I am wondering under Eclipse do we have a need for Ant.
And if it is, at which aspect should we use Ant along with Eclipse.
Building applications is not about just building them in your IDE. You need to setup an automated build and deploy method that can be run for, say, continuous integration and deployment. The built package has to be deloyed to various environments and finally production. You cannot build them out of your own box. You cannot build it out of your IDE. You need separate build system. That is where build tools like ANT come in.
Also note that the capabilities of IDEs for doing build, deploy is limited and that is not their strength. It is, however, possible to make use of ANT and its build scripts and run them out of eclipse. This way, you have a single set of scripts for both your local ( developer ) build and the main build process. With ANT being JAVA based, you can also suit your build tool to your needs by writing custom tasks etc.
Also, when you make use of something as standard as ANT for your build system, you can move to a different IDE like Intellij IDEA and still make use of the build system that you know.
Do you need Ant if you can build through the IDE? Of course not! You can see that yourself.
Should you still have a IDE independent build with standard tools? Of course!
Note I didn't say Ant. There are a number of build tools such as Maven or Gradle that are made to build Java programs. However, there are reasons why you want an IDE independent build system:
You have a Continuous Integration Server: You do have one. Right? If you don't get one. A Continuous Integration Server automatically builds your project (and runs every test) each time a change is committed to the version control system. It catches mistakes. It also provides an OFFICIAL BUILD. No more It worked on my machine! excuses. If the Continuous Integration Server can't build it and successfully run the tests, the software isn't good.
You have others who need to work with your project on a different computer in a different environment: The problem with IDEs is that its specific to your system. Someone else checks out your project, and they may be using a different IDE. They maybe have a slightly different setup. An independent build scripting language like Ant guarantees that they can rebuild your project as intended.
If you share a project with someone else, you need an independent standard method way of being able to build that project and to know exactly what is required for that build to work. IDEs can hide that information.
I am a benchmark tester using Ant everyday. Our environment has 7 test machines and 1 console machine. For each test run, the ant script updates the working copy, restarts the database, build the product and the Grinder with proper properties and deploy them to the machines, and starts them and the monitoring softwares. With the 1000-line-buildfile, I can just type in an "ant A B C D E"-like line in the terminal on the console machine. I don't think it's a good idea to do this in a IDE like eclipse or NetBeans or anything else.
Also, in my company's product, ant is widely used. We use ant to build sample data for the benchmark test; we use ant to update Tomcat which is going to be bundled with our product; we also use ant to build and run selenium tests.
What's more, when you want to release the source code, including the IDE project files is not a good idea because there are a lot of IDEs. And what if the user just want to get a fresh build from your latest code (if you don't provide nightly build)? Providing a set of ant (or other build tool) buildfiles is a good idea. With this, you can manage your classpath for the build.
The shortest answer here wold be flexibility. The IDEs may came with pre-defined functionalities and may even allow you to choose some options using wizards. It is like generating html with Dreamweaver or such tools. It will never be as flexible as the pure thing. For example NetBeans uses Ant underneath.
Quoted from: Netbeans: Build Tools
The default NetBeans IDE project system is built directly on top of
the Ant build system. All of the project commands, such as "Build
Project" and "Run File in Debugger", call targets in the project's Ant
script. You can therefore build and run your project outside the IDE
exactly as it is built and run inside the IDE.
Even after you use tools for that purpose you are most likely still to brush up. So, what I am saying is, what the IDEs do for you is only a subset of what you can do with the pure ant or maven( if you use it). For simple tasks such as compile and run, IDEs do pretty much a good job! But if you have a use case as complex as deploying on CI IDEs won't help you. You need the whole flexibility of your build tools.
I'm trying to convert a Netbeans 6.9.1 project into a scripted build (without netbeans). Of course, it fails (or I wouldn't be asking for help).
In the failure it says that the org.apache.commons.httpclient package does not exist. (Of course, it worked when we ran the build in Netbeans).
Now I know exactly where the commons-httpclient.jar file is located in my project structure, but I can't seem to tell it to the compiler via the ant build files and the netbeans property files.
Perhaps related to this is when I ran "ant -v" to build my software, it said,
Property lib.mystuff.classpath has not been set. This variable is important, I guess, because
the file nbproject/project.properties uses lib.mystuff.classpath in its definition of javac.classpath, which of course tells the Java compiler where to find the JARs.
So...when moving a Netbeans project to a netbeans-independent scripted build, how can the build script set these properties? Also, how can I ensure that the jar file gets included in the ant build?
I appreciate any help I can get, as I am a Java newbie.
UPDATE AFTER ACCEPTING ANSWER FROM vkraemer:
There are a few best practices for build scripts for production software:
Put everything needed for a build under a single directory tree. (Netbeans = fail)
Put everything in source code control. (I did that)
The first line of the build script should clear all environment variables.
The next section of the build script should explicitly set all environment variables to values which are known to work.
The next part of the build should be able to execute using command-line programs such as javac, ant, cc, etc, and must not depend on firing up an IDE such as Eclipse or Netbeans.
It is a shame that Netbeans makes this hard.
I did a quick look in a Java Application project and found the following...
javac.classpath = ${libs.MyStuff.classpath}
libs.MyStuff.classpath is defined in %HOME%/.netbeans/6.9.1/build.properties.
You may be able to get by doing the following...
ant -Dlibs.MyStuff.classpath=c:\a\b\c.jar
You would need to do more if you have multiple jar files in the MyStuff library that you created in NetBeans.
Item 2 of the Joel Test is "Can you make a build in one step", but what is the best (or commonly accepted) way to achieve this? What are the pros and cons of using the IDE's command-line interface to do a headless build, as opposed to maintaining a build script that is completely independent of the IDE (e.g. using Ant or Maven in the case of a Java project).
I ask this question because I am experimenting with Maven, and was a bit surprised to find that even when using the m2eclipse plugin it's not really feasible to turn off Eclipse's Java Builder and delegate the whole build process to Maven while working on the project. It seems that if I want to migrate the project to Maven, I'll end up having to maintain two equivalent but different build processes, for example the command-line build will use the DataNucleus Maven plugin for enhancing JDO classes, while the IDE build will use the DataNuclues plugin for Eclipse; the command-line build will use the Tomcat Maven plugin, while the IDE will use Eclipse's web tools platform. This redundancy seems unfortunate.
I gather that NetBeans uses Ant for its build, which sounds like it would solve this problem. But unfortunately I'm using neither NetBeans nor Ant!
EDITED TO ADD: I found that I can set up a Maven builder in Eclipse with specific goals that call some of the Maven plugins that I'm using (e.g. in my case the goals "datanucleus:enhance process-resources" take care of JDO enhancement, resource copying, and native2ascii conversion). This leaves java compilation up to Eclipse, but still achieves some degree of integration with Maven.
The eclipse builder gives you intermediate and repeated building, but the "Joel Test" is really about being able to go from source to ready-to-deploy something in a single step. If you're using Maven, there's several ways of going about it - including just invoking Maven as a command line script from within Eclipse.
mvn package
Is what I most frequently used for a quick run through the whole build setup. You can extend Maven through the POM and inject some additional mechanisms if you want to. If you get a little more complex, it's often most effective to start using multi-module POM and maven setups to get functional tests integrated and running with the rest of the code.
Basically, let eclipse "do it's thing" with it's builders and take advantage of what it provides, but when you're ready to use the end product have it invoked from the command line through a continuous integration server (Hudson is a nice easy one to get and set up - free too: http://hudson-ci.org/). Presumably if you're looking at using Maven, you'll also have an instance of Archiva or Nexus set up as your DSL for the resulting libraries. You can have hudson invoke "mvn deploy" for regular checkin builds, or "mvn release:prepare && mvn release:perform" for when you're ready to cut the release (separate builds in Hudson work best for this)
You can do a headless build in an IDE. In eclipse you can execute an arbitrary shell command as an "External Tool". This is also true in IDEA and netbeans with a little effort. At the very least, this is a convenient test of the headless build that you should run whenever you make changes to the build configuration.
Also, I'd like to add that the build should be one-step as a minimum requirement. It should also be easy to set up and easy to debug. If it takes longer than an couple of hours to set up a new developers environment, then the one-step process is likely to be less than optimal. This is with the caveat that if you add or replace team members quarterly or annually this is less of an issue.
I'm finding it difficult to phrase this question well, as there are quite a few generic terms (run, configuration, launch, etc.). Here goes:
You can save run configurations in a .launch file. (in the Run Configuration Dialog, under the Common tab, Save as a shared file.
We check these in to SVN. The developers can pass them around, and it helps getting new devs running a working application quicker.
I'd like to check these out as part of our build and use them to programatically run the application, the tests, etc, without spinning up the whole IDE.
What would be the best way to run a .launch file outside of the UI?
Edit: I am trying to unify the tests run on the build server and the IDE. I do not
particularly want to give up integrated debugging, which would be the case with an ant script to run the tests .
This is probably more a problem for integration testing with multiple bundles, or unit testing a whole bundle, where you'd like to mock up extensions.
there is an eclipse plugin built over JUnit, called TPTP. It provides an automation client which can be used to launch the test from eclipse with no gui. maybe it helps
Ant4Eclipse may provide a good starting point on how to do this.
Unfortunately, this is limited to Java Applications and JUnit configurations; I am more interested in PDE applications and Plugin JUnit tests.
I have recently had alot of success building an Eclipse RCP app inside a Hudson CI server using Eclipse Buckminster. It took a bit of doing, but once I setup both features, made my RCP product be based on features, and added the Buckminster query files and the like, it worked. There is a Hudson/Jenkins Buckminster plugin that allowed me to have hudson build the application.
After saving the launch configurations for each test fragment, I created hudson commands to invoke them (yes one line per test fragment unfortunately), but after that I got the automated CI build that I wanted.
You could also use the shell command Eclipse uses. To get it:
Run your program in Eclipse
Go to the "Debug" view
Right-click on the process (probably the second item in the tree) and select "Properties"
Copy shell command and delete the agentlib flag to run in bash
I think you don't need to use the .launch configurations to run the tests. If you build an application using the Eclipse Build System, then you can use the AntRunner application from Eclipse to run your units tests. This doesn't start the whole IDE.
This article describes how to run the tests during your build process. With this process, you use a special "Test" Eclipse and load the plugins you want to test.
Perhaps running the configurations the way you would run your own custom run configurations would help here. It is described in this article.