I'm building a debian package from scala project using the sbt-native-packager plugin. The package produced locally (macos) from command line has different size and of course md5sum than the one I produce in jenkins (debian) using the same sbt command:
sbt debian:packageBin
Also the jenkins produced debian package throws an error when using the included jar, like some classes were not included:
ERROR org.apache.spark.deploy.yarn.ApplicationMaster - User class threw exception: java.lang.NoClassDefFoundError: shadeshapless/Generic
java.lang.NoClassDefFoundError: shadeshapless/Generic
I thought has to do something with the class names on debian environment so added the compiler option:
scalacOptions ++= Seq("-Xmax-classfile-name","255")
I expect the same debian package to be produced on the local and jenkins environments.
The resulting package will always depend on the machine as the version of the native build tool, in your case deb-pkg, affects how the package is produced. Sometimes a plugin generates code which might have timestamps that change, leading to different outputs.
Having said that. I personally hit this classfile issue with shading shapeless, sbt assembly and docker images. This however has nothing to do with the packaging.
Cheers,
Muki
Related
I am looking at using d-bus from Scala. I have added this to my build.sbt:
libraryDependencies ++= Seq(
...
"com.github.hypfvieh" % "dbus-java" % "3.0.2",
...
}
Building is find, but at runtime it can't find the UnixSocket class:
java.lang.RuntimeException: Could not load library from any given source: [SYSTEM_PATH, CUSTOM_PATH, CLASS_PATH]
at com.github.hypfvieh.system.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:89)
at com.github.hypfvieh.system.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:69)
at cx.ath.matthew.unix.UnixSocket.(UnixSocket.java:54)
I've tried various artefacts on Maven that might have this class, but without success.
Is there another dependency I should be using, or do I need to add the appropriate library by hand?
None of the (old) dbus-java classes/libraries are on Maven, those bindings never used Maven as the buildsystem. The original bindings can still be seen on the dbus git repo.
If you're on Debian, you should be able to get the libraries by running apt-get install libunixsocket-java. If you are not on Debian or a Debian derivative, you would have to compile from source.
Note that however the error that is happening is actually due to you not loading the JNI code library, that code is actually in the dbus-java repo.
I have been using sbt on windows and a custom build.sbt script in conjunction with an import Chisel._ in the top-level file in order to generate Verilog from my Chisel source successfully.
I'm trying to get an IDE working on Windows to expedite Chisel development. I've gone with the Eclipse based SCALA IDE http://scala-ide.org/download/sdk.html/
I want to compile the Chisel library so that the import Chisel._ can be resolved locally, without having to go off and download the source from the repository each timeand recompile the source. When I download the Chisel-master repo from Git and include the src\main folder in my SCALA project in the SCALA IDE, I get lots of syntax errors in the Chisel SCALA files that prevent me from building the project.
Has anyone done anything like this before on Windows or have any knowledge of working with the SCALA IDE as it may just be a case of undefined symbols in the project configuration?
Not sure exactly what you did with build.sbt respect to recompile (I think it download it only the first time, then it caches it for the future). But I'm using ScalaIDE for Chisel on linux, using the default build.sbt files, maybe you can try to get it working out of the box first to help narrow down the issue.
Here are the steps I took in order to get ScalaIDE work with Chisel:
the latest Scala IDE uses 2.11.8, the current Chisel repository defaults to 2.11.7. So I had to change all the build.sbt reference to scalaVersion from 2.11.7 to 2.11.8
I used sbteclipse
https://github.com/typesafehub/sbteclipse
To create importable the workspace to setup the compilation dependencies.
Except for chiselFrontEnd. For some reason, this package is not added to the dependency. I have to Add chiselFrontEnd as a javabuildpath dependency manually (Properties/JavaBuildPath, under Projects) for my own projects.
To resolve undefined symbols, you can also add a JAR onto the project build path using Project Properties > Java Build Path > Libraries > Add External JARs...
If you are getting your JARs through Maven / SBT, they should be in:
C:\Users\<name>\.ivy2\local\edu.berkeley.cs\chisel3_2.11\jars
If you are using publish-local with chisel3, your JARs should be in
C:\Users\<name>\.ivy2\cache\edu.berkeley.cs\chisel3_2.11\jars
Note that chisel3 is compiled into one JAR, including coreMacros and chiselFrontend sub-projects
Of course, this is a more quick-and-dirty solution compared to something that can parse SBT files.
We are creating a customize version of Spark since we are changing some lines of code from ALS.scala. We build the customize spark version using
mvn command:
./make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn
However, upon using the customized version of Spark, we run into this error:
Do you guys have some idea on what causes the error and how we might solve the issue?
I am actually using a jar file in the local machine by building them using sbt: sbt compile then sbt clean package and putting the jar file here: /Users/user/local/kernel/kernel-0.1.5-SNAPSHOT/lib.
However in the hadoop environment, the installation is different. Thus, I use maven to build spark and that's where the error comes in. I am thinking that this error might be dependent on using maven to build spark as there are some reports like this:
https://issues.apache.org/jira/browse/SPARK-2075
or probably on building spark assembly files
For our Scala development we currently use ivy + ant, but we are also trying to use sbt for our development workflow. This would be for the continuous incremental compilation when not using an IDE.
sbt uses ivy, so in theory this should work. But when using an ivy external file the tests won't compile.
To reproduce this you can even use the generated ivy.xml file from any sbt project.
Here are the steps to reproduce the error on a sbt project with tests,
from the sbt console run deliverLocal (deliver-local in previous versions of sbt)
copy the generated ivy file into your project home and rename it to 'ivy.xml'. From my understanding using this file should be equivalent to declaring the dependencies in build.sbt.
edit the build.sbt, add externalIvyFile() on one line and then comment all dependencies declarations
in the console, run reload, then test
compile will run just fine, but test will fail at compile time. None of the dependencies will be honoured, not even the production code of the current project.
What am I missing?
In my case it worked with the following build.sbt:
externalIvyFile()
classpathConfiguration in Compile := Compile
classpathConfiguration in Test := Test
classpathConfiguration in Runtime := Runtime
You just need the extra three lines in the end. Here is a link for more info: http://www.scala-sbt.org/release/docs/Detailed-Topics/Library-Management.html#ivy-file-dependency-configuration
Look for the Full Ivy Example. I hope it helps!
EDIT: Just to be complete - here is what pointed me to the above link: https://github.com/sbt/sbt/issues/849.
I'm currently using sbt to build and run my scala programs. I'm trying to use sbt.Process to execute system commands. I must be missing something because when I try to import sbt.Process in one of my files in src/ I get this error.
not found: value sbt
[error] import sbt.Process._
So it looks like I can't access the sbt package inside my src/ files. What do I need to do to access it? Thanks.
SBT's environment (v 0.7.x) is only available in your build file or a Plugin.
The easiest way to use sbt.Process library (until 0.9.x which will have Process as an independent library) is to copy (BSD License) Process.scala and ProcessImpl.scala into your project
There are different classpaths for running sbt and compiling your source files.
One classpath is for compilation of files in directory project/build (that one contains sbt jars and usually scala library 2.7.7) and the other one is for building source files of your project (that one contains your dependencies from lib and lib_managed and usually scala library 2.8.*). If you'd like to use sbt.Process in your source files you can do two things:
add sbt jar to lib or lib_managed for it to be available on your project's classpath
use snapshot version of scala 2.9, it would have sbt Process built-in as sys.process package
Wait for Scala 2.9, and then just use it out of scala.sys.process.
sbt package has became an integral part of the Scala standard library since version 2.9
...this API has been included in the Scala standard library for version 2.9.
quoted from sbt wiki
Here's the link (scroll down)
well, in order to use it, all you have to do (assuming you are using sbt for build), is to add in build.sbt file the following line of code: sbtPlugin := true it will add the needed dependencies to your project.
of course, this solution is only to get your imports with sbt package to work. you should refactor your code to use the new package scala.sys.process like Daniel C. Sobral suggested.