run Scala code in package (LinkedIn Norbert) - scala

I am trying to run a Scala object called NorbertClusterClientMain which is in package com.linkedin.norbert.cluster. The source code for it is in folder examples/src/main/scala of rhavyn's open-source branch of LinkedIn Norbert, and I am working on a Linux command line.
Although I've been told that running Scala code in a package is like running Java in a package, I am in examples/src/main/scala but cannot use this command:
$ scala com.linkedin.norbert.cluster.NorbertClusterClientMain
I am getting "No such file or class on classpath", even though the file exists.
I was successfully able to compile Norbert with
$ mvn clean -DskipTests install
How can I run the NorbertClusterClientMain? Please let me know. I appreciate your help.

It is the same. So, in this case, it is looking for this file:
./com/linkedin/norbert/cluster/NorbertClusterClientMain.class
This is how Java works, and since "running" a Scala program is just running java passing the Scala library in the classpath, it has to be the same.
How did you compile it, by the way? Nevermind, saw your comment. At the directory you ran mvn, you should probably be able to run it like this:
scala -cp target com.linkedin.norbert.cluster.NorbertClusterClientMain
Failing that, find the class file, and pass the directory where com/ is to the classpath.

Your mvn script produced JAR and class files within target directory:
./target/com/.../<someClassName1>.class
./target/com/.../<someClassName2>.class
... etc
./target/<someJarName1>.jar
./target/<someJarName2>.jar
... etc
Great! Now do the same thing that you must do for java; include in your classpath:
the target base directory (this "picks up" all class files in directory hierarchy beneath target)
each jar file (this "picks up" all class files in directory hierarchy within each JAR)
scala -cp target:target/<someJarName1>.jar:target/<someJarName2>.jar:... etc ./com.linkedin.norbert.cluster.NorbertClusterClientMain
Here -cp (or equivalently, CLASSPATH environment variable) is the java classpath and so has the same syntax and rules as java.
BTW: "sbt" is a standard, powerful, usable way to build scala projects. It uses Ivy to "pull" dependencies from code repositories (i.e. mvn++). The best way to get started with it is to download sbt example projects, search for "sbt tutorial" blogs and read the sbt docs. :)

I don't anything about the code base, but that class is in the examples subproject.
This shows that it loads normally. (I haven't configured anything, because I don't know anything about the code base.)
apm#mara:~/clones/norbert$ cd examples
/home/apm/clones/norbert/examples
apm#mara:~/clones/norbert/examples$ ls
pom.xml src target
apm#mara:~/clones/norbert/examples$ mvn exec:java -Dexec.mainClass=com.linkedin.norbert.cluster.NorbertClusterClientMain
[INFO] Scanning for projects...
<snip...>
[INFO] --- exec-maven-plugin:1.2.1:java (default-cli) # norbert-examples ---
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at com.linkedin.norbert.cluster.NorbertClusterClientMain$.main(NorbertClusterClientMain.scala:22)
at com.linkedin.norbert.cluster.NorbertClusterClientMain.main(NorbertClusterClientMain.scala)
... 6 more
Here's how to add args:
https://stackoverflow.com/a/9846103/1296806
I don't use maven much anymore.
Edit: I don't use Scala 2.7.7 much anymore either.

try $ scala ./com.linkedin.norbert.cluster.NorbertClusterClientMain
or -cp .
you "current directory" might not be in classpath

Norbert uses a Scala 2.7 so directly using Scala from CLI may not work for you. Therefore find all the dependency jar using Maven and use it.
This is how I did it.
First, check out the code:
$ git clone https://github.com/rhavyn/norbert
$ cd norbert/
Build and install the dependencies in local repository first:
$ mvn clean install
Setup a classpath variable which we will use later for examples/ folder:
$ cd examples/
$ export CP=$(mvn dependency:build-classpath | grep -A1 'Dependencies classpath:' | tail -1)
Run server from examples/ folder:
$ java -cp $CP:target/classes com.linkedin.norbert.network.javaapi.NorbertJavaNetworkServerMain arg0 arg1
log4j:ERROR Could not find value for key log4j.appender.R
log4j:ERROR Could not instantiate appender named "R".
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 2
at com.linkedin.norbert.network.javaapi.NorbertJavaNetworkServerMain.main(NorbertJavaNetworkServerMain.java:33)
Run client from examples/ folder:
$ java -cp $CP:target/classes com.linkedin.norbert.cluster.NorbertClusterClientMain localhost 1011
log4j:ERROR Could not find value for key log4j.appender.R
log4j:ERROR Could not instantiate appender named "R".
> h2013-12-20 13:59:44,323 - WARN [pool-1-thread-2-SendThread(0.0.3.243:2181):ClientCnxn$SendThread#1120] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:364)
at sun.nio.ch.Net.connect(Net.java:356)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:623)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1009)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1036)
2013-12-20 13:59:46,192 - WARN [pool-1-thread-2-SendThread(0.0.3.243:2181):ClientCnxn$SendThread#1120] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:364)
at sun.nio.ch.Net.connect(Net.java:356)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:623)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1009)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1036)
Thats it. Now all you have to do is setup the required services like Zookeper etc.

Related

sbt failed to start with NoClassDefFoundError

On a remote Windows 10 system sbt failes to start with a NoClassDefFoundError:
C:\WORKBENCH\BPF\my-project>sbt -Djavax.net.ssl.trustStoreType=WINDOWS-ROOT -Dsbt.boot.credentials="C:\WORKBENCH\BPF\my-project\credentials.txt" -Dsbt.boot.directory=C:\WORKBENCH\BPF\.sbt\boot -Dsbt.coursier.home=C:\WORKBENCH\BPF\coursier -v
# Executing command line:
"C:\Program Files (x86)\Java\jre1.8.0_321\bin\java.exe"
-Djavax.net.ssl.trustStore="C:\Program Files (x86)\Java\jre1.8.0_321\lib\security\cacerts"
-Djavax.net.ssl.trustStorePassword=changeit
-Xms1024m
-Xmx1024m
-Xss4M
-XX:ReservedCodeCacheSize=128m
-cp
"C:\Program Files (x86)\sbt\bin\sbt-launch.jar"
xsbt.boot.Boot
-Djavax.net.ssl.trustStoreType=WINDOWS-ROOT
-Dsbt.boot.credentials=C:\WORKBENCH\BPF\my-project\credentials.txt
-Dsbt.boot.directory=C:\WORKBENCH\BPF\.sbt\boot
-Dsbt.coursier.home=C:\WORKBENCH\BPF\coursier
java.lang.ClassCastException: java.lang.NoClassDefFoundError cannot be cast to xsbti.FullReload
at sbt.internal.XMainConfiguration.run(XMainConfiguration.java:59)
at sbt.xMain.run(Main.scala:46)
at xsbt.boot.Launch$.$anonfun$run$1(Launch.scala:149)
at xsbt.boot.Launch$.withContextLoader(Launch.scala:176)
at xsbt.boot.Launch$.run(Launch.scala:149)
at xsbt.boot.Launch$.$anonfun$apply$1(Launch.scala:44)
at xsbt.boot.Launch$.launch(Launch.scala:159)
at xsbt.boot.Launch$.apply(Launch.scala:44)
at xsbt.boot.Launch$.apply(Launch.scala:21)
at xsbt.boot.Boot$.runImpl(Boot.scala:78)
at xsbt.boot.Boot$.run(Boot.scala:73)
at xsbt.boot.Boot$.main(Boot.scala:21)
at xsbt.boot.Boot.main(Boot.scala)
[error] [launcher] error during sbt launcher: java.lang.ClassCastException: java.lang.NoClassDefFoundError cannot be cast to xsbti.FullReload
Tried with different SBT versions (1.6.2, 1.5.8, 1.4.9) by changing project/build.properties): The Stack Trace differs depending on the version, but it is always a NoClassDefFoundError.
The SBT version specified in the project is successfully downloaded by the launcher.
I suspect local file permission problems as cause (e.g. due to security policies) therefore i moved the boot and the coursier cache directory. However, this did not bring the desired success.
Does anyone have an idea what the Problem could be?
(I did not do much findings about it.) In my case i could fix it by changing to appropriate java version. Initially i ran with java 18v mistakenly and then changing to java 11v it worked.
I don't have a real diagnosis or causal explanation, but I did solve a similar problem (identical error message, Linux) by clearing out the sbt cache (the .sbt folder within your user directory).

Jrebel not initializing for embedded tomcat

I have been unable to get Jrebel to run. I am running a Hippo Java project on mac with the Eclipse plugin. I run the project using mvn from a command line (not through the eclipse IDE). I have followed these instructions for Hippo and JRebel
I have located the JRebel Eclipse plugin files here:
/Applications/Eclipse.app/Contents/Eclipse/plugins
From here I copied the contents of
org.zeroturnaround.eclipse.embedder_6.5.0.RELEASE/jrebel/
which contains 2 files: jrebel.jar and jrebel.plugininfo
I put these copies into a created directory ~/Tools/jrebel and pointed the environment variable REBEL_HOME to it.
export REBEL_HOME=~/Tools/jrebel
When I echo $REBEL_HOME I get the correct path. When I cd to that path and ls I get the 2 files.
In my hippo project, from the command line I run
mvn clean verify -Djrebel
I get build success. Then I run
mvn -Pcargo.run -Djrebel
This gives the error:
[INFO] [talledLocalContainer] Error opening zip file or JAR manifest missing : ${env.REBEL_HOME}/jrebel.jar
[INFO] [talledLocalContainer] Error occurred during initialization of VM
[INFO] [talledLocalContainer] agent library failed to init: instrument
I have read here to specify the path of jrebel.jar to -javaagent , but I don't know what this means or how to do it?
Hippo has built in JRebel configurations in the pom.xml. These should be activated with the -Djrebel flag. Why isn't this working?
The error shows that in that process REBEL_HOME isn't defined. I don't see anything wrong with what you did here so there may be some peculiarity to your local setup or to MacOS (I'm not a mac user). In any case your environment variable is not being passed to the process.
Everything you said works for me on Linux (Mint).
You can try setting the property in the cargo profile.
Or you could set the variable in your global environment.

Cannot find parser that supports .groovy

When I'm trying to start changelog.groovy via liquibase command line it tells me that
Unexpected error running Liquibase: Cannot find parser that supports changelog.groovy
I'm doing the next: java -jar liquibase.jar update
My liquibase.properties are:
driver=org.postgresql.Driver
classpath=C:\Users\Andrii\org.postgresql.Driver.jar;C:\Users\Andrii\liquibase-3.5.1-bin\lib\liquibase-groovy-dsl-1.2.2-SNAPSHOT.jar
changeLogFile=D:\changelog.groovy
url=jdbc:postgresql://localhost:5432/test
username=postgres
password=rup
It finds those jars since if I change something in that path it will tell that jars cannot be found.
I downloaded the groovy-liquibase-dsl project, build it and added a jar into classpath. What am I doing wrong?
To make it work, you need to additionally include groovy and groovy-sql jars in Liquibase's classpath.
So say you store all the jars in C:\Users\Andrii\LiquibaseDependencies, update your the classpath property of your file as such:
classpath=C:\Users\Andrii\LiquibaseDependencies\org.postgresql.Driver.jar;
C:\Users\Andrii\LiquibaseDependencies\liquibase-groovy-dsl-1.2.1.jar;
C:\Users\Andrii\LiquibaseDependencies\groovy-2.4.6.jar;
C:\Users\Andrii\LiquibaseDependencies\groovy-sql-2.4.6.jar

Running Scala^Z3 with Scala 2.10

I installed Scala^Z3 on my Mac OSX (Mountain Lion, JDK 7, Scala 2.10, Z3 4.3) successfully (following this: http://lara.epfl.ch/w/ScalaZ3). Everything went fine except that I cannot run any example from this website (http://lara.epfl.ch/w/jniz3-scala-examples) without getting this nasty error:
java.lang.NoClassDefFoundError: scala/reflect/ClassManifest
at .<init>(<console>:8)
at .<clinit>(<console>)
at .<init>(<console>:7)
...
Caused by: java.lang.ClassNotFoundException: scala.reflect.ClassManifest
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 29 more
I think this happens because of the incompatibility between the Scala 2.9.x and 2.10.x in handling reflections. As I was able to run the same set of examples under Scala 2.9.x. My question is, is there anyway to go around this and run Scala^Z3 under Scala 2.10?
From looking at the project propertie and build file (https://github.com/psuter/ScalaZ3/blob/master/project/build.properties and https://github.com/psuter/ScalaZ3/blob/master/project/build/scalaZ3.scala) I infer that scalaZ3 is currently provided for scala 2.9.2 only. There is no cross version support at the moment.
You might try to get the code and compile it yourself after having changed the version to scala 2.10.0 in the "build.properties" file.
See this page for instructions on how to compile it: https://github.com/psuter/ScalaZ3.
If you're lucky, the code will compile as is under scala 2.10. If you're not, there might be some small fixes to do. Cross your fingers.
If you are not in a hurry, you could also bug the Scala^Z3 authors and ask them for scala 2.10 version of the library.
I'm copying the instructions from my response to your issue on GitHub, as it may help someone in the future.
The current status is that the old sbt project does not seem mix well with Scala 2.10. Here are the instructions for a "manual" compilation of the project, for Linux. This works for me with Z3 4.3 (grabbed from the Z3 git repo) and Scala 2.10. After installing Z3 per the original instructions:
First compile the Java files:
$ mkdir bin
$ find src/main -name '*.java' -exec javac -d bin {} +
Then compile the C files. For this, you need to generate the JNI headers first, then compile the shared library. The options in the commands below are for Linux. To find our where the JNI headers are, I run (new java.io.File(System.getProperty("java.home")).getParent in a Scala console (and add /include to the result).
$ javah -classpath bin -d src/c z3.Z3Wrapper
$ gcc -o lib-bin/libscalaz3.so -shared -Wl,-soname,libscalaz3.so \
-I/usr/lib/jvm/java-6-sun-1.6.0.26/include \
-I/usr/lib/jvm/java-6-sun-1.6.0.26/include/linux \
-Iz3/4.3/include -Lz3/4.3/lib \
-g -lc -Wl,--no-as-needed -Wl,--copy-dt-needed -lz3 -fPIC -O2 -fopenmp \
src/c/*.[ch]
Now compile the Scala files:
$ find src/main -name '*.scala' -exec scalac -classpath bin -d bin {} +
You'll get "feature warnings", which is typical when moving to 2.10, and another warning about a non-exhaustive pattern match.
Now let's make a jar file out of everything...
$ cd bin
$ jar cvf scalaz3.jar z3
$ cd ..
$ jar uf bin/scalaz3.jar lib-bin/libscalaz3.so
...and now you should have bin/scalaz3.jar containing everything you need. Let's try it out:
$ export LD_LIBRARY_PATH=z3/4.3/lib
$ scala -cp bin/scalaz3.jar
scala> z3.scala.version
Hope this helps!
This does not directly answer the question, but might help others trying to build scalaz3 with scala 2.10.
I built ScalaZ3 with Scala 2.10.1 and Z3 4.3.0 on Windows 7. I tested it with the integer constraints example at http://lara.epfl.ch/w/jniz3-scala-examples and it is working fine.
Building Z3
The Z3 4.3.0 download at codeplex does not include libZ3.lib file. So I had to download the source and build it on my machine. The build process is quite simple.
Building ScalaZ3
Currently, build.properties has sbt version 0.7.4 and scala version 2.9.2. This builds fine. ( I had to make some minor modifications to the build.scala file. Modify z3LibPath(z3VN).absolutePath to z3LibPath(z3VN).absolutePath + "\\libz3.lib" in the gcc task. )
Now if I change scala version to 2.10.1 in build.properties, I get "Error compiling sbt component 'compiler-interface'" error on launching sbt. I have no clue why this happens.
I then changed sbt version to 0.12.2 and scala version to 2.10.1, and started with the fresh source. I have also added build.sbt in project root folder containing scalaVersion := "2.10.1". This is required since in sbt 0.12.2, the file build.properties is supposed to be used for specifying the sbt version only. More info about sbt version differences at (https://github.com/harrah/xsbt/wiki/Migrating-from-SBT-0.7.x-to-0.10.x).
I get error Z3Wrapper.java:27: cannot find symbol LibraryChecksum. This happens because the file LibraryChecksum.java which is supposted to be generated by the build (project\build\build.scala) is not generated. Looks like the package task does not execute the tasks in (project\build\build.scala). The tasks compute-checksum, javah, and gcc are not executed. This may be happening because sbt 0.12.2 expects the build.scala file to be directly under the project folder.
I then copied the LibraryChecksum.java generated from the previous build, the build then goes through. The generated jar file does not contain scalaz3.dll.
I then executed javah and gcc tasks manually. The command for these tasks can be copied from the log of successful build with scala 2.9.2(I made appropriate modifications to the commands for scala 2.10.1). Here also, I had to make some changes. I had to explicitly add full path of scala-library.jar classpath of the javah task.
I then added the lib-bin\scalaz3.dll to the jar file using jar uf target\scala-2.10\scalaz3.jar lib-bin/scalaz3.dll.

Scala SBT / Maven2 Error on OSX: "Error Opening Zip File" -> MissingRequirementError

I have a project that builds well on Unix boxes (http://www.github.com/jhclark/ducttape).
However, using SBT 0.11.2 (and a few other versions of SBT), it will not build on my Mac (OSX 10.5). I get the following cryptic error message:
$ ~/bin/sbt compile (master*? 20:11)
[info] Loading project definition from /Users/jon/Documents/workspace- scala/ducttape/project
[info] Set current project to ducttape (in build file:/Users/jon/Documents/workspace-scala/ducttape/)
[info] Compiling 104 Scala sources to /Users/jon/Documents/workspace-scala/ducttape/target/scala-2.9.2/classes...
[error] error while loading <root>, error in opening zip file
[error] {file:/Users/jon/Documents/workspace-scala/ducttape/}default-024416/compile:compile: scala.tools.nsc.MissingRequirementError: object scala not found.
[error] Total time: 2 s, completed May 27, 2012 8:12:09 PM
This happens even after I clean things out thoroughly with:
sbt clean clean-files
rm -rf ~/.ivy2 ~/.m2 ~/.sbt
I suspect that the real error is happening in Maven2, which SBT uses for dependency management (see also Maven : error in opening zip file when running maven).
However, I'm stumped after several days. Any ideas?
I had similar problems when attempting to use an older version of the sbt-extras launcher with sbt-0.11.3. In my case, it attempted to download a file that didn't exist, and attempted to unzip the 404 error page. The most recent sbt-extras launcher has been fixed for sbt-0.11.3.
Yours sounds different, but they may be due to the the shutdown of scala-tools.org. If you can, I recommend upgrading to sbt-0.11.3.
If you need to continue to use 0.11.2, you should use the 0.11.3-2 launcher, put sbt.version=0.11.2 in project/build.properties, and disable the scalaTools repo in your build.sbt. Mark Harrah posted info on the SBT mailing list.
If this doesn't help, for some reason you're downloading corrupted .jars from somewhere.
Run the command find ~/.ivy2 ~/.m2 ~/.sbt -name "*.jar" -exec unzip -qqt {} \; to find which jar(s) are corrupted. The contents of the corrupt jar may give you a clue as to what's going wrong.
For completion, I had the same problem and it was a corrupt zip/jar file.
However, Dave's command line checks for the usual classpath directories, where your jar may be.
The one causing me troubles was in fact in my projects ./lib folder. I struggled with this for hours, hopefully this may help someone else.
PS: Thanks a lot Dave!
I just had the same problem (on OSX, and with Scala 2.10.2), but the problem turned out not to be in any of my dependency manager repos, or with SBT or Scala (as for some people).
I had put a JAR into /Library/Java/Home/lib/ext/ for other reasons, but didn't make it group- and world-readable. Only root (it's owner) could read it. I tweaked the permissions, and voila. Scala and SBT suddenly work again.
cd /Library/Java/Home/lib/ext
sudo chmod g+r bcprov-jdk15on-1.47.jar
sudo chmod g+a bcprov-jdk15on-1.47.jar