Last week we had the user directory permissions changed on our CI servers and we no longer have write access to the user home directory. Hence sbt fails to boot because it cannot write to ~/.ivy etc with the following stacktrace.
java.io.IOException: No such file or directory
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1006)
at xsbt.boot.Locks$.apply0(Locks.scala:34)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at xsbt.boot.Launch.locked(Launch.scala:238)
at xsbt.boot.Launch.app(Launch.scala:147)
at xsbt.boot.Launch.app(Launch.scala:145)
at xsbt.boot.Launch$.run(Launch.scala:102)
at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:35)
at xsbt.boot.Launch$.launch(Launch.scala:117)
at xsbt.boot.Launch$.apply(Launch.scala:18)
at xsbt.boot.Boot$.runImpl(Boot.scala:41)
at xsbt.boot.Boot$.main(Boot.scala:17)
at xsbt.boot.Boot.main(Boot.scala)
Error during sbt execution: java.io.IOException: No such file or directory
I know that there is a smattering of sbt variables we can set, such as the boot directory. I've not yet found a definitive list of all the variables that we can configure that otherwise default to a subdirectory of user home. I also have not found a way to set the default root directory, which could in turn impact all of these variables.
Can anyone point out how to configure sbt such that it does not use the user's home directory for any files?
The ivy path options mentioned in the other answers here were part of the solution. As pointed out in this answer you need to set ivy for both sbt itself and the project. Furthermore as I discovered on this sbt github issue comment, sbt needs yet another directory for its own stuff.
In total, I used the following three jvm properties in order to use a relative path to the project for everything that sbt otherwise uses the user's home directory for:
-Dsbt.global.base=./.sbt/
-Dsbt.ivy.home=./.ivy2/
-Divy.home=./.ivy2/
At least as of sbt 0.13.9, this is what is needed.
If you would like to run with your .ivy in a different location, and you're using sbt extras boot script, then you should just be able to do:
sbt -ivy /path/to/ivy package
and it'll download all the ivy stuffs to the location specified and run whatever command you need (in this example, package)
You can set -ivy when using sbt-extras (preferable), or you can use a JVM property:
-Dsbt.ivy.home=/path/to/ivy
I've also used this:
-Divy.default.ivy.user.dir=/path/to/ivy
Related
In this question, there is discussion of how to include jar file into an sbt project. I need both a .jar file and some .so library files.
Attempt 1:
I can move the jar file into my sbt lib/ directory, which is great, except that this application has a small jar which is just a wrapper around C++ software. The stuff I want to do is in the .so library files, and if I move the jar file to ./lib by itself, I get linking errors:
sbt:SimpleProject> run linearSalience
[info] Running linearSalience linearSalience
[error] (run-main-0) java.lang.UnsatisfiedLinkError: no java_salience in java.library.path
[error] java.lang.UnsatisfiedLinkError: no java_salience in java.library.path
[error] at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
Attempt 2:
I have tried putting simlinks to the desired libraries into the lib/ folder, but that didn't work. I don't want to copy the entire library into the lib/ folder, even if that would work, since it is almost 2 GB; and it would be silly to copy it for each project.
Attempt 3:
I tried setting java.library.path through the javaOptions of sbt, by adding the following line to build.sbt
javaOptions in run += "-Djava.library.path=/opt/path/to/lib/:/opt/path/to/sdk/java/src/"
The first path contains the .so files, the second the .jar file. In this case, the compiler couldn't even find the packages (which was not a problem while the .jar file was in the lib/ folder of sbt):
[info] Compiling 1 Scala source to /opt/optests/sbttest/target/scala-2.10/classes ...
[error] /opt/optests/sbttest/src/main/scala/SimpleApp.scala:6:12: object lexalytics is not a member of package com
[error] import com.lexalytics.salience.{Salience, Section, Sentence, Word}
[error] ^
[error] /opt/optests/sbttest/src/main/scala/SimpleApp.scala:23:23: not found: type Salience
[error] val session = new Salience("/opt/path/to/license.v5", "/opt/path/to/data")
[error] ^
...etc
Attempt 4:
I try to set the LD_LIBRARY_PATH environment variable (as suggested here)
[user#server ~]$ echo $LD_LIBRARY_PATH
/opt/path/to/lib/:/opt/path/to/sdk/java/src/
The result is the same error as in 3
It seems like all the questions on this topic are resolved by either putting single jar files into lib/ or using managed dependencies (as here). But I have a local-only repository with no online support, that is more than a single .jar file.
Ultimately, I need to get the library directory into java.library.path, but how do I do that?
Note: This is not a duplicate of any question that deals with only .jar files and has no mention of .so files.
When you use a JNI wrapper over a native library on JVM (it doesn't matter on the language, it can be Java or Scala), the library jar file usually contains only the JNI glue code, defining how to map some java api calls to the native library calls. This jar is just a regular library, so it can be dropped to the /lib sbt folder as usual.
The native library itself should be present at runtime within the java.library.path, so you were quite close. As you suggested, you can add native SDK to the javaOptions in run (along with the jar in /lib folder) and it should work.
I'm not an "real" developper, but I have the right to at least write some code, and add some Jars to the Eclipse build path, without spending hours trying to figure out if the Jars are actually in the Build Path.
My problem (error here below) was resolved in question [NoClassDefFoundError, cannot run MapReduceColorCount (Avro 1.7.7) by adding the correct Jars.
[cloudera#localhost ~]$ hadoop jar avroColorCount.jar exos.MapReduceColorCount2 inavro01 outavro01
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/mapreduce/AvroKeyInputFormat
at exos.MapReduceColorCount2.run(MapReduceColorCount2.java:71)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at exos.MapReduceColorCount2.main(MapReduceColorCount2.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
The following are the different ways I've tried to add Jars to the build path:
1. Maven: adding Dependencies through the POM file, they appear afterwards under "Maven Dependencies".
2. "Configure Build Path": the Jars are actually located in my local file system, thus I add the (library) folders, and the folders appear under the "Referenced Libraries".
3. Create a "lib" folder in the project folder, copy/paste the Jars (located in my local file system), do a project Refresh (the lib folder appears in the Package Explorer), select all Jars and right-click "Add to Build Path"
I confirm that my code show no warnings/errors while performing either method. I usually to an "Export ..." of the Jar file in order to execute it.
Example: I've tried adding to Build Path external Jars from Cloudera's CDH5 (Hadoop 2.3.0-cdh5.1.2 and Avro 1.7.5-cdh5.1.2) which are localted locally in /opt/lib
The only method that really worked was method 3. Why it doesn't work with methods 1. or 2. ?
Thank you in advance for your support
I could not reproduce success with method 3., I received a "cannot cast to namespace.customClass" error instead of the "NoClassDefFoundError" error.
I've found an answer for the latter error with a workaround based on two variables export:
export LIBJARS=avrojar1,avrojar2,jar3
export HADOOP_CLASSPATH=avrojar1:avrojar2:jar3
and then running the hadoop jar command with -libjars ${LIBJARS}.
This was tested with method 1. and method 3. respectively.
I conclusion, my case was specific to avro-related Jars only
Thanks
When I start the sbt console I get this:
alex#alex-K43U:~$ sbt console [warn] Alternative project directory
.sbt (/home/alex/.sbt) has been deprecated since sbt 0.12.0. [warn]
Please use the standard location: /home/alex/project [info] Loading
project definition from /home/alex/.sbt [info] Set current project to
default-22b2b7 (in build file:/home/alex/)
I just started using scala and sbt, so I'm not really sure what the warning means. It means that I have to move all the content of /home/alex/.sbt to /home/alex/project?
(I have this folder too: /home/alex/sbt which has a bin folder and a jansi-license.txt file. I think that's how I installed sbt).
You can run sbt from any dedicated folder besides your home to get rid of this warning.
This warn appears because you run sbt from the directory that contains the .sbt folder, that in your case is /home/alex/.sbt
Create a folder (named project or something else) inside /home/alex/ and run sbt from there. Don't move the .sbt data inside the new folder because the warn will appear again.
I've got an sbt (Scala) project that currently pulls artifacts from the web. We'd like to move towards a corporate-standardized Nexus repository that would cache artifacts. From the Nexus documentation, I understand how to do that for Maven projects. But sbt obviously uses a different approach. (I understand Ivy is involved somehow, but I've never used it and don't understand how it works.)
How do I tell sbt and/or the underlying Ivy to use the corporate Nexus repository system for all dependencies? I'd like the answer to use some sort of project-level configuration file, so that new clones of our source repository will automatically use the proxy. (I.e., mucking about with per-user config files in a dot-directory is not viable.)
Thanks!
Step 1: Follow the instructions at Detailed Topics: Proxy Repositories, which I have summarised and added to below:
(If you are using Artifactory, you can skip this step.) Create an entirely separate Maven proxy repository (or group) on your corporate Maven repository, to proxy ivy-style repositories such as these two important ones:
http://repo.typesafe.com/typesafe/ivy-releases/
http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/
This is needed because some repository managers cannot handle Ivy-style and Maven-style repositories being mixed together.
Create a file repositories, listing both your main corporate repository and any extra one that you created in step 1, in the format shown below:
[repositories]
my-maven-proxy-releases: http://repo.example.com/maven-releases/
my-ivy-proxy-releases: http://repo.example.com/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
Either save that file in the .sbt directory inside your home directory, or specify it on the sbt command line:
sbt -Dsbt.repository.config=<path-to-your-repo-file>
Good news for those using older versions of sbt: Even though, in the sbt 0.12.0 launcher jar at least, the boot properties files for older sbt versions don't contain the required line (the one that mentions repository.config), it will still work for those versions of sbt if you edit those files to add the required line, and repackage them into the sbt 0.12.0 launcher jar! This is because the feature is implemented in the launcher, not in sbt itself. And the sbt 0.12.0 launcher is claimed to be able to launch all versions of sbt, right back to 0.7!
Step 2: To make sure external repositories are not being used, remove the default repositories from your resolvers. This can be done in one of two ways:
Add the command line option -Dsbt.override.build.repos=true mentioned on the Detailed Topics page above. This will cause the repositories you specified in the file to override any repositories specified in any of your sbt files. This might only work in sbt 0.12 and above, though - I haven't tried it yet.
Use fullResolvers := Seq( resolver(s) for your corporate maven repositories ) in your build files, instead of resolvers ++= or resolvers := or whatever you used to use.
OK, with some help from Mark Harrah on the sbt mailing list, I have an answer that works.
My build class now looks like the following (plus some other repos):
import sbt._
//By extending DefaultWebProject, we get Jetty support
class OurApplication(info: ProjectInfo) extends DefaultWebProject(info) {
// This skips adding the default repositories and only uses the ones you added
// explicitly. --Mark Harrah
override def repositories = Set("OurNexus" at "http://our.nexus.server:9001/nexus/content/groups/public/")
override def ivyRepositories = Seq(Resolver.defaultLocal(None)) ++ repositories
/* Squeryl */
val squeryl = "org.squeryl" % "squeryl_2.8.0.RC3" % "0.9.4beta5"
/* DATE4J */
val date4j = "hirondelle.date4j" % "date4j" % "1.0" from "http://www.date4j.net/date4j.jar"
// etc
}
Now, if I delete the Squeryl tree from my machine's .ivy2/cache directory, sbt tries to grab it from the Nexus tree with the appropriate URL. Problem solved!
All you need is to define a property file sbt.boot.properties which will allow you to:
redefine the ivy cache location (I need that because it would be otherwise part of our roaming Windows profile, which is severely limited in disk space in our shop. See Issue 74)
define any other Maven repo you want
C:\HOMEWARE\apps\sbt-0.74\sbt.boot.properties
[scala]
version: 2.7.7
# classifiers: sources, javadoc
[app]
org: org.scala-tools.sbt
name: sbt
version: read(sbt.version)
class: sbt.xMain
components: xsbti
cross-versioned: true
classifiers: sources, javadoc
[repositories]
local
my-nexus: http://my.nexus/nexus/content/repositories/scala-tools/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
maven-local
# sbt-db: http://databinder.net/repo/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
# maven-central
# scala-tools-releases
# scala-tools-snapshots
[boot]
directory: project/boot
properties: project/build.properties
prompt-create: Project does not exist, create new project?
prompt-fill: true
quick-option: true
[log]
level: debug
[app-properties]
project.name: quick=set(test), new=prompt(Name)[p], fill=prompt(Name)
project.organization: new=prompt(Organization)[org.vonc]
project.version: quick=set(1.0), new=prompt(Version)[1.0], fill=prompt(Version)[1.0]
build.scala.versions: quick=set(2.8.0.RC2), new=prompt(Scala version)[2.8.0.RC2], fill=prompt(Scala version)[2.8.0.RC2]
sbt.version: quick=set(0.7.4), new=prompt(sbt version)[0.7.4], fill=prompt(sbt version)[0.7.4]
project.scratch: quick=set(true)
project.initialize: quick=set(true), new=set(true)
[ivy]
cache-directory: C:\HOMEWARE\projects\.ivy2\cache
Note: this sbt.boot.properties file is inspired from:
the one mentioned in the "Generalized Launcher" page of the sbt project.
the one found within sbt-0.74 itself!
I have commented any external Maven repository definition, and added a reference to my own Nexus Maven repo.
The launcher may be configured in one of the following ways in increasing order of precedence:
Replace the /sbt/sbt.boot.properties file in the jar.
Put a configuration file named sbt.boot.properties on the classpath. Put it in the classpath root without the /sbt prefix.
Specify the location of an alternate configuration on the command line. This can be done by:
either specifying the location as the system property sbt.boot.properties
or as the first argument to the launcher prefixed by '#'.
The system property has lower precedence.
Resolution of a relative path is:
first attempted against the current working directory,
then against the user's home directory,
and then against the directory containing the launcher jar.
An error is generated if none of these attempts succeed.
Define a sbt.bat wrapper (in order to be sure to specify your sbt.boot.properties) like:
C:\HOMEWARE>more C:\HOMEWARE\bin\sbt.BAT
#echo off
set t=%~dp0
set adp0=%t:C:\="%"
set SBT_DIR=%adp0%..\apps\sbt-0.74
dir C:\%SBT_DIR%\sbt-launch-0.7.4.jar
# if needed, add your proxy settings
set PROXY_OPTIONS=-Dhttp.proxyHost=my.proxy -Dhttp.proxyPort=80xx -Dhttp.proxyUser=auser -Dhttp.proxyPassword=yyyy
set JAVA_OPTIONS=-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m -Xmx512M -cp C:\HOMEWARE\apps\sbt-0.74\sbt-launch-0.7.4
set SBT_BOOT_PROPERTIES=-Dsbt.boot.properties="sbt.boot.properties"
cmd /C C:\HOMEWARE\apps\jdk4eclipse\bin\java.exe %PROXY_OPTIONS% %JAVA_OPTIONS% %SBT_BOOT_PROPERTIES% -jar C:\HOMEWARE\apps\sbt-0.74\sbt-launch-0.7.4.jar %*
And your sbt will download artifacts only from:
your Nexus
your local Maven repo.
Just tested at home with an old Nexus opensource 1.6 I had running, java 1.6, sbt07.4
C:\Prog\Java\jdk1.6.0_18\jre\bin\java -Xmx512M -Dsbt.boot.properties=sbt.boot.properties - jar "c:\Prog\Scala\sbt\sbt-launch-0.7.4.jar"
That gives:
[success] Build completed successfully.
C:\Prog\Scala\tests\pp>sbt
Getting Scala 2.8.0 ...
downloading http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-compiler/2.8.0/scala-compiler-2.
8.0.jar ...
[SUCCESSFUL ] org.scala-lang#scala-compiler;2.8.0!scala-compiler.jar (311ms)
downloading http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-library/2.8.0/scala-library-2.8.
0.jar ...
[SUCCESSFUL ] org.scala-lang#scala-library;2.8.0!scala-library.jar (185ms)
:: retrieving :: org.scala-tools.sbt#boot-scala
confs: [default]
2 artifacts copied, 0 already retrieved (14484kB/167ms)
[info] Building project test 0.1 against Scala 2.8.0
[info] using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7
If I try a funny value in the sbt.boot.properties file:
C:\Prog\Scala\tests\pp>sbt
Getting Scala 2.9.7 ...
:: problems summary ::
:::: WARNINGS
module not found: org.scala-lang#scala-compiler;2.9.7
==== nexus: tried
http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-compiler/2.9.7/scala-compiler-2.9.7.pom
-- artifact org.scala-lang#scala-compiler;2.9.7!scala-compiler.jar:
http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-compiler/2.9.7/scala-compiler-2.9.7.jar
So it does limit itself to the two repo I defined:
[repositories]
nexus: http://localhost:8081/nexus/content/repositories/scala
nexus2: http://localhost:8081/nexus/content/repositories/scala, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
(I commented everything else: local, maven-local, ...)
If I comment all repositories and put a funny value (2.7.9) for the scala version in the sbt.boot.properties, I do get (like the OP did)
C:\Prog\Scala\tests\pp>sbt
Error during sbt execution: No repositories defined.
If I put 2.7.7 (while still having all repo commented), yes, it won't generate an error:
C:\Prog\Scala\tests\pp>sbt
[info] Building project test 0.1 against Scala 2.8.0
[info] using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7
But that's only because it already had downloaded scala2.8.0 during my previous tries.
If I remove that library from my project/boot directory, then it will throw an Exception:
[info] using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7
> C:\Prog\Scala\tests\pp>sbt
Error during sbt execution: No repositories defined.
at xsbt.boot.Pre$.error(Pre.scala:18)
at xsbt.boot.Update.addResolvers(Update.scala:197)
...
at xsbt.boot.Boot$.main(Boot.scala:15)
at xsbt.boot.Boot.main(Boot.scala)
Error loading project: Error during sbt execution: No repositories defined.
edit the config file in sbt_home/conf "sbtconfig.txt"
add two line
-Dsbt.override.build.repos=true
-Dsbt.repository.config="C:/Program Files (x86)/sbt/conf/repo.properties"
the repo.properties content is
[repositories]
local
public: http://222.vvfox.com/public <-fix this ,write your local nexus group url
Well this has bugged me for a while so I found a guy that has written an SBT plugin for maven out on github called maven-sbt so all you have to do is include it in your plugins project and make your project mixin with maven.MavenDependencies and all your operations like update and publish-local work with your local maven. The nice thing about that is if you are like me, your org is all maven. So, all you libs are in you local maven repo but if for some reason you build with sbt first, then you start getting a bunch or jars in ivy too. What a waste of space, and time since you will still need to get them for your maven builds.
That said, I wish this were built into sbt so I would not need to add it to every project. Maybe as a processor at least. He mentioned in one thing I read that he would like to add it to 0.9 but I have not been able to find it.
I got this error because I had a blank file in ~/.sbt/repositories. Both adding repositories to the file and removing the file solved the problem.
I am using SBT as my build tool for building a Scala project.
My problem is, I can't configure SBT to download dependencies to my user home directory. Therefore I am looking for a per-user or even better a system-wide setting to tell SBT to put the Ivy cache directory somewhere else.
With maven there is the per-user settings.xml that can be used to configure the local repository.
I have read question How to override the location of Ivy’s Cache? and it's answers, but it seems it only describes how to configure the setting on a per project basis.
If there is no alternative I would go for a per-project setting, but I didn't get the answer from the mentioned question to work. Some more details would be most welcome, for example where to put the ivysettings.xml. I put it into the project's root directory and it didn't work.
The sbt.ivy.home property is only half of the solution. It controls where the sbt launcher downloads sbt itself (and related dependencies like the scala compiler and library, etc.) As noted by Joachim Hofer, it has no effect on where the dependencies declared by your project get downloaded.
To change that location, you must set the ivy.home property. So, to augment Joachim's first solution, you would set both system properties:
java -Dsbt.ivy.home=/tmp/.ivy2/ -Divy.home=/tmp/.ivy2/ -jar `dirname $0`/sbt-launch.jar "$#"
With these properties, the launcher will download both your project's and sbt's dependencies to the /tmp/.ivy2/ directory. Of course, you can put them in separate directories as well.
You can simply add an environment variable to your sbt launch shell script:
java -Dsbt.ivy.home=/tmp/.ivy2/ ...
See Library Management in the official documentation.
You should use sbt-extras if you don't do already.
Then, it's simply a flag you pass it:
sbt -ivy /path/to/.ivy2
Location of ivy files
I normally put the ivy.xml and ivysettings.xml files alongside by build file as follows:
build.xml
ivy.xml
ivysettings.xml
The ivy tasks resolve and retrieve should find both files.
For example:
<target name="init" description="--> retrieve dependencies with ivy">
<ivy:retrieve pattern="lib/[conf]/[artifact].[ext]"/>
</target>
Odd, that it's not working for you.
User specific settings
You can emulate the maven settings file in a couple of ways
1) include directive within the project ivysettings.xml
<ivysettings>
<include file="${user.home}/.ivy2/my-ivysettings.xml"/>
</ivysettings>
2) Set location from the build file
<target name="init" description="--> retrieve dependencies with ivy">
<ivy:settings file="${user.home}/.ivy2/my-ivysettings.xml" />
<ivy:retrieve pattern="lib/[conf]/[artifact].[ext]"/>
</target>
3) I've never tried this but I think you can override the default location using an ANT property
ant -Divy.settings.file=$HOME/.ivy2/my-ivysettings.xml
You can retrieve your home directory using Path.userHome.absolutePath, like shown below:
resolvers += Resolver.file("Local", file( Path.userHome.absolutePath + "/.ivy2/local"))(Resolver.ivyStylePatterns)
I suppose that you can also retrieve environment variables using System.getenv and concatenate in the same way, like shown below:
resolvers += Resolver.file("Local", file( System.getenv("IVY_HOME") + "/whatever/it/is"))(Resolver.ivyStylePatterns)
For editing the cache location during the SBT boot itself, see Sbt Launcher Configuration in the official documentation.
Basically, to get it to work system-wide, you'd have to:
Put a configuration file named sbt.boot.properties somewhere where it's accessible system-wide (the default one is listed at the link above).
Call the launcher with the additional system property sbt.boot.properties set to point to your configuration file.
Set the cache-directory entry (in the [ivy] section) to the location of your ivy cache.
This configuration doesn't seem to carry over to normal SBT usage, though, unfortunately.
sbt -ivy /tmp/.ivy2 compile
Reference: man sbt
Options:
-ivy path: path to local Ivy repository (default: ~/.ivy2)