How to specify JVM maximum heap size "-Xmx" for running an application with "run" action in SBT? - scala

My application does large data arrays processing and needs more memory than JVM gives by default. I know in Java it's specified by "-Xmx" option. How do I set SBT up to use particular "-Xmx" value to run an application with "run" action?

For forked processes you should look at Build.scala
To modify the java options for forked processes you need to specify them in the Build.scala (or whatever you've named your build) like this:
val buildSettings = Defaults.defaultSettings ++ Seq(
//…
javaOptions += "-Xmx1G",
//…
)
This will give you the proper options without modifying JAVA_OPTS globally, and it will put custom JAVA_OPTS in an sbt generated start-script
For non forked processes it's most convenient to set the config via sbtopts or sbtconfig depending on your sbt version.
Since sbt 0.13.6 .sbtconfig is deprecated. Modify /usr/local/etc/sbtopts along these lines:
-J-Xms512M
-J-Xmx3536M
-J-Xss1M
-J-XX:+CMSClassUnloadingEnabled
-J-XX:+UseConcMarkSweepGC
-J-XX:MaxPermSize=724M
-J-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
You can also create an .sbtopts file in the root of your SBT project using the same syntax as in the /usr/local/etc/sbtopts file. This makes the project self-contained.
Before sbt 0.13.6 you could set the options in .sbtconfig for non forked processes:
Check where sbt is:
$ which sbt
/usr/local/bin/sbt
Look at the contents:
$ cat /usr/local/bin/sbt
#!/bin/sh
test -f ~/.sbtconfig && . ~/.sbtconfig
exec java ${SBT_OPTS} -jar /usr/local/Cellar/sbt/0.12.1/libexec/sbt-launch.jar "$#"
Set the correct jvm options to prevent OOM (both regular and PermGen):
$ cat ~/.sbtconfig
SBT_OPTS="-Xms512M -Xmx3536M -Xss1M
-XX:+CMSClassUnloadingEnabled
-XX:+UseConcMarkSweepGC -XX:MaxPermSize=724M"
If you want to set SBT_OPTS only for the current run of sbt you can use env SBT_OPTS=".." sbt as suggested by Googol Shan. Or you can use the option added in Sbt 12: sbt -mem 2048. This gets unwieldy for longer lists of options, but it might help if you have different projects with different needs.
Note that CMSClassUnloadingEnabled in concert with UseConcMarkSweepGC helps keep the PermGen space clean, but depending on what frameworks you use you might have an actual leak on PermGen, which eventually forces a restart.

In sbt version 12 onwards there is an option for this:
$sbt -mem 2048

If you run sbt on linux shell, you can use:
env JAVA_OPTS="-Xmx512m" sbt run
This is my usually used command to run my sbt project.

.sbtconfig is deprecated starting with SBT 0.13.6. Instead, I configured these options in /usr/local/etc/sbtopts in the following way:
-J-Xms512M
-J-Xmx3536M
-J-Xss1M
-J-XX:+CMSClassUnloadingEnabled
-J-XX:+UseConcMarkSweepGC
-J-XX:MaxPermSize=724M
-J-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005

Try this:
class ForkRun(info: ProjectInfo) extends DefaultProject(info) {
override def fork = Some(new ForkScalaRun {
override def runJVMOptions = super.runJVMOptions ++ Seq("-Xmx512m")
override def scalaJars = Seq(buildLibraryJar.asFile, buildCompilerJar.asFile)
})
}

There's one way I know of. Set the environment variable JAVA_OPTS.
JAVA_OPTS='-Xmx512m'
I have not found a way to do this as a command parameter.

Use JAVA_OPTS for setting with environment variable.
Use -J-X options to sbt for individual options, e.g. -J-Xmx2048 -J-XX:MaxPermSize=512
Newer versions of sbt have a "-mem" option.

The javaOptions += "-XX:MaxPermSize=1024" in our build.sbt as referenced by #iwein above worked for us when we were seeing a java.lang.OutOfMemoryError thrown while running Specs2 tests through sbt.

The environment variable is _JAVA_OPTIONS, which needs to be set.
Once you set _JAVA_OPTIONS, and when you sbt, sbt will show the message using JAVA_OPTIONS and the values.
Alternatively you could set javaOption in the sbt or .scala file
e.g
javaOptions += "-Xmx1G"
From sbt shell you could run show javaOptions to see the values that are set.

sbt lets you list the JVM options you need to run your project on a file named
.jvmopts
in the root of your project.
then add the java options that you want
cat .jvmopts
-Xms512M
-Xmx4096M
-Xss2M
-XX:MaxMetaspaceSize=1024M
it is tested and works in windows 10
https://www.lagomframework.com/documentation/1.4.x/scala/JVMMemoryOnDev.html

javaOptions in Test += "-Xmx1G"
This sets the JVM options for tests. Works also with jvm forking (fork in Test := true).

Related

How to set SCALACTIC_FILL_FILE_PATHNAMES=yes (or other environment variable) for scalac from sbt?

I would like to use full paths in ScalaTest reporter. The value LineInFile.filePathname tells me I should:
Please set the environment variable SCALACTIC_FILL_FILE_PATHNAMES to yes at compile time to enable this feature.
Can I do this from sbt, or do I have to launch SBT with a different environment? Scalac seems to be running in the same JVM as sbt, and there is a question Scala: Unable to set environment variable telling me I cannot modify my own environment.
I have tried following, none of them seems to set the variable for the compiler:
scalacOptions ++= Seq(
"-deprecation", "-unchecked",
"-DSCALACTIC_FILL_FILE_PATHNAMES=yes"
),
Compile / envVars := Map("SCALACTIC_FILL_FILE_PATHNAMES" -> "yes"),
Test / envVars := Map("SCALACTIC_FILL_FILE_PATHNAMES" -> "yes"),
As far as I know, sbt doesn't run scalac in a separate process (which would allow it to set environment variables), so you'd have to run sbt in an environment with that set.
Note that -D sets Java properties. While it's fairly common for software to allow environment variables and properties to be treated similarly (e.g. Lightbend Config), that's not standard.
The mechanism for setting the environment will depend on how you are launching sbt. If using a Bourne-style shell like bash:
export SCALACTIC_FILL_FILE_PATHNAMES=yes
sbt
or
SCALACTIC_FILL_FILE_PATHNAMES=yes sbt
IDEs and CI/CD tooling may launch sbt directly, in which case check the documentation for those tools.

Run project with java options via sbt

I am running my fat jar with command java -Djava.security.krb5.conf=/krb5.conf -jar my.jar.
How to run my app with this option via sbt?
$ sbt -Djava.security.krb5.conf="module\\src\\main\\resources\\krb5.conf" run doesn't work. Error:
ctl-scala>sbt -Djava.security.krb5.conf="ctl-core\src\main\resources\krb5.conf" ctl-ui-backend/run
Warning: invalid system property 'java.security.krb5.conf'
[info] Loading global plugins from C:\Users\User\.sbt\0.13\plugins
[info] Loading project definition from C:\Users\User\IdeaProjects\ctl-scala\project
[info] Set current project to ctl (in build file:/C:/Users/User/IdeaProjects/ctl-scala/)
[error] No valid parser available.
[error] ctl-core\\src\\main\\resources\\krb5.conf
[error] ^
Can you try sbt -J-Djava.security.krb5.conf="module/src/main/resources/krb5.conf" run
The -J causes the sbt launcher to pass those as options to the JVM.
As sbt manual it will pass JAVA_OPTS environment variable to the java and if you can not set the variable .jvmopts will hold the java commandline arguments. so if you are in bash :
export JAVA_OPTS="-Djava.security.krb5.conf=/krb5.conf"
before sbt command will pass the argument to java runtime.
You can force sbt to fork a new JVM when running the application, and set the desired java options with the following settings in the build.sbt file:
fork := true,
javaOptions ++= Seq(
"-Djava.security.krb5.conf=/krb5.conf"
)
Simply run the run task and it should start the application in its own JVM with the required java options.
Another option, is to use .sbtopts file. It should be in the root folder, next to sbt.build. Its content should be the java options, prefixed with -J, as written in previously answers, to tell sbt to pass those options to the JVM.
For example its content can be:
-J-Djava.security.krb5.conf=/krb5.conf

Change sbt boot and ivy dirs via environment variables

Is it possible to set .sbt and .ivy2 directories via environment variables?
I can override these parameters when i run sbt like that:
sbt -Dsbt.boot.directory=/tmp/.sbt/boot -Dsbt.ivy.home=/tmp/.ivy2 version
I thought this would work:
export sbt.boot.directory=/tmp/.sbt/boot
export sbt.ivy.home=/tmp/.ivy/home
sbt version
but it doesn't work...
Second question: how can i bypass sbt config launcher? (http://www.scala-sbt.org/0.13/docs/Launcher-Configuration.html) when running sbt?
You can set JAVA_OPTS or SBT_OPTS environment variables with the runtime parameters for the SBT. The difference is that JAVA_OPTS effects also other Java based applications while SBT_OPTS only effects the SBT. For example:
export SBT_OPTS="-Dsbt.ivy.home=/tmp/.ivy/home -Dsbt.boot.directory=/tmp/.sbt/boot"
When you run the SBT from the command-line, it will use these parameters. However, if you launch it from IntelliJ IDEA, it will not use them. IDEA launches the SBT JAR directly and SBT's VM parameters have to be configured in the IDEA sbt settings per project.
For the second question, it is unclear what you mean by bypass. It is possible to run the SBT jar directly instead of the sbt launcher script. For example:
java -server -Xmx1536M -Dsbt.ivy.home=/tmp/.ivy/home -jar /<install-path>/sbt/bin/sbt-launch.jar

Why does sbt compile fail with StackOverflowError?

I am working on a Scala project that has been in existence for a few years but is new to me. My task is to upgrade it from Scala 2.9.3 to 2.11.7, along with its dependencies. I have gotten past the errors and warnings, but I cannot get the project to compile successfully in SBT. I always get a StackOverflowError in pretty much the same place. The stacktrace looks like this, but details vary with the Xss setting (currently 4M, but have tried as high as 24M):
java.lang.StackOverflowError
at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:698)
at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5395)
at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5422)
at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5369)
at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5373)
at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5471)
at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5479)
at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:644)
at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:698)
at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5395)
at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5422)
SBT_OPTS looks like this:
-Xmx2G -Xss4M -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
I can 'make' the project successfully in Intellij, and others can pull my changes from GitHub and compile the project in sbt, so the issue seems to be local to my machine (a recent quad-core Macbook Pro with 16GB RAM). Other Scala/sbt projects compile successfully for me on this machine.
Here are other relevant details:
Scala version: 2.11.7
Java version: java version "1.8.0_66" (build 1.8.0_66-b17)
sbt version: 0.13.7 (have also tried 0.13.9)
I have completely rebuilt the ivy2 cache and cleared the lib_managed directory. The version of the scala-compiler.jar is the same as is used on at least one machine that can 'sbt compile' the code successfully. I did a clean reinstall of sbt (via brew remove sbt, manual removal of ~/.sbt directory, then brew install sbt).
I have not tried to isolate the line of source code being compiled when the error occurs. I have assumed it would be more productive to look for a configuration issue or dependency conflict somewhere.
Any suggestions for further troubleshooting will be appreciated.
[Added...] It may be helpful to add that, as an experiment, I downloaded the Scala language source code from https://github.com/scala/scala and got the following very similar error trying to sbt compile it:
java.lang.StackOverflowError
at scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.outerValue(ExplicitOuter.scala:229)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:441)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:352)
at scala.reflect.internal.Trees$class.itransform(Trees.scala:1345)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)
at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:44)
at scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.scala$reflect$internal$Trees$UnderConstructionTransformer$$super$transform(ExplicitOuter.scala:219)
at scala.reflect.internal.Trees$UnderConstructionTransformer$class.transform(Trees.scala:1693)
at scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.transform(ExplicitOuter.scala:291)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:459)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:352)
at scala.reflect.internal.Trees$class.itransform(Trees.scala:1347)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)
Here is something interesting. From this post I found out about launching sbt with a -d flag for debugging info. Got the following output:
Kevins-MacBook-Pro:scala kdoherty$ sbt -d
[process_args] java_version = '1.8.0_66'
# Executing command line:
java
-Xmx2G
-Xss4M
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-Xmx384m
-Xss512k
-XX:+UseCompressedOops
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
-jar
/usr/local/Cellar/sbt/0.13.9/libexec/sbt-launch.jar
So somewhere my SBT_OPTS settings are being overridden (by defaults, I guess). Now I need to find where those defaults are coming from.
I figured it out. Once I knew that the -d flag would tell me what settings SBT was actually using, I saw that the values in my SBT_OPTS environment variable were being clobbered by other, lower settings. Where were those coming from? From my JAVA_OPTS env variable! I should have noticed them sooner, but now I know I can keep those Java options as they are and override them by adding the SBT-specific settings to my /usr/local/etc/sbtopts file, using the somewhat awkward format of
-J-Xmx2G
-J-Xss2M
Using the values shown I was able to run sbt compile successfully on my project.
I hope someone finds this useful.
Add to the bottom of /usr/local/etc/sbtopts
-J-Xmx4G
-J-Xss4M
All set.
I just added -Xss in my Intellij sbt properties and the issue is resolved.
Intellij SBT properties
Relevant parts from the output of sbt -h:
# jvm options and output control
JAVA_OPTS environment variable, if unset uses ""
.jvmopts if this file exists in the current directory, its contents
are appended to JAVA_OPTS
SBT_OPTS environment variable, if unset uses ""
.sbtopts if this file exists in the current directory, its contents
are prepended to the runner args
Thus, in my case I solved the problem by creating a file .sbtopts with content
-J-Xmx4G
-J-Xss4M
in the project directory.
Note: Running sbt -d shows what settings have been used, for instance:
$ sbt -d
[addSbt] arg = '-debug'
[process_args] java_version = '8'
# Executing command line:
java
-Xms1024m
-XX:ReservedCodeCacheSize=128m
-XX:MaxMetaspaceSize=256m
-Xmx2G
-Xss2M
-jar
/path/to/sbt-launch.jar
-debug
I was unable to get this to work via the provided answers. build.sbt settings and the sbtopts file both failed to solve this error for me. What I had to do was run sbt with the -mem flag, for example:
sbt -mem 2048 compile
and now my projects built. If using IntelliJ you can also go to
Preferences > Build, Execution, Deployment > Build Tools > sbt
and set Maximum heap size, MB to whatever target you need.
I discovered the SBT_OPT setting in the bin/sbt file of my sbt install was affecting the memory values set in my projects build.sbt
updating the existing -Xss value in this file from 1M to 8M raised the memory size of the Scalac stack to a point I stopped getting StackOverflow exceptions in the sbt-invoked compiler. This seemed odd because the sbt documented approach to setting stack size in the compiler is to do this is with the -J-Xss setting.
Sbt doesn't seem to actually enable you to set the compiler's stack memory. While a build.sbt accepts the following configuration as a valid setting, it doesn't seem to apply the value in the compiler:
scalacOptions in ThisBuild ++= Seq(-J-Xss8M)
I suspect that is a bug or non-implemented functionality
I too run into this problem recently and I discovered a working solution for it with sbt 1.3.13.
create a .sbtopts file under project root
add -J-Xss100M (or any thread stack size you think suitable) to the .sbtopts file
There are multiple Correct Answers already. But what worked for me is below,
// Created and Added a File: .jvmopts in the Project Root Folder with below Parameters.
-Xms3022m
-Xmx4048m
-Xss124m
-XX:MaxPermSize=4048m
-XX:MaxMetaspaceSize=512m
-XX:+CMSClassUnloadingEnabled
-XX:ReservedCodeCacheSize=128m

How do I set a system property for my project in sbt?

I'm sure I'm missing something really simple... I want to set the system property java.awt.headless to true for my sbt project. Reading the page on properties I think that I need to use system or systemOptional. In my project file I've tried things like:
lazy val javaAwtHeadless = system[Boolean]("java.awt.headless")
Setting it as a user property (e.g. lazy val javaAwtHeadless = property[Boolean]) and setting the accompanying value in build.properties made the property visible in the sbt console but not within sbt's Scala console (via System.getProperty("java.awt.headless")).
set java.awt.headless true from the sbt console works, including being set in the Scala console, but it doesn't persist to the next time I launch sbt.
A straightforward method would be to edit the batch file or shell script that you use to run sbt and add -Dprop=val
If I needed this option for all sbt tasks, I'd set it as follows in build.sbt
javaOptions += "-Djava.awt.headless=true"
If it was just for one task, eg: run, you can scope that:
javaOptions in Runtime += "-Djava.awt.headless=true"
If you're trying to set SBT properties, like plugin settings, then the following worked for me with 0.13+. The following however did work, when trying to pass in Liquibase settings, like password, from our CI frameworks.
In your build.sbt
Ugly, but supplies defaults, and optionally grabs from System.properties. This way you've got your default and override cases covered.
def sysPropOrDefault(propName:String,default:String):String = Option(System.getProperty(propName)).getOrElse(default)
liquibaseUsername := sysPropOrDefault("liquibase.username","change_me")
liquibasePassword := sysPropOrDefault("liquibase.password","chuck(\)orris")
From the commandline
Now just override via -Dprop=value like you would with Maven or other JVM programs. Note props appear before SBT task.
sbt -Dliquibase.password="shh" -Dliquibase.username="bob" liquibase:liquibase-update