SBT killed when running out of ram - scala

I am trying to run my scala application in a docker container with 800MB of RAM, (docker run -m=800m mindlevel) but it gets killed before it is done compiling.
[info] Compiling 10 Scala sources to /root/mindlevel-backend/target/scala-2.12/classes...
[info] 'compiler-interface' not yet compiled for Scala 2.12.1. Compiling...
[info] Compilation completed in 8.631 s
/usr/local/sbt/bin/sbt-launch-lib.bash: line 58: 57 Killed "$#"
I have tried to restrict the SBT heap with:
CMD ["sbt", "-mem", "700", "run"]
But it still gets killed at the same place.
Without any restrictions on the docker container it runs fine.
When running it in an EC2 micro instance (1GB of RAM) it also crashes, with or without restrictions on SBT and Docker.
Is there some restriction that I am missing?

To get it to run with less than 1GB heap I precompiled the project with sbt assembly which makes a jar packaged with your project and all of your dependencies.
Install it by adding:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")
to project/assembly.sbt
Then generate the jar with sbt assembly which will place the jar in ./target/scala-<VERSION>/<PROJECT>-assembly-1.0.jar
Then simply run your project with:
scala target/scala-<VERSION>/<PROJECT>-assembly-1.0.jar
Since no compilation step is needed on the low memory instance now a lot of projects will be able to run even though it is less memory than recommended.

Related

Sbt not loading the project from its current directory

When I run SBT in my Scala project that contains a build.sbt file, SBT seems to ignore it and loads the project definition from a different folder and set the current project to that different directory hsperfdata which is weird because this is used as part of Java's performance counter which has nothing to do with what I'm trying to do.
Also SBT was installed via SDKMAN and when I try to specify a lower SDK version 1.2.8 instead of 1.4.6 it will always use 1.4.6. version 1.2.8 is also specified in my build.properties file as well.
My stack trace:
xxxxx#MC-N369748 topic-api % sbt
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[info] welcome to sbt 1.4.6 (AdoptOpenJDK Java 1.8.0_275)
[info] loading project definition from /private/var/folders/hs/0vkhntfd557c8z8g8b_p7ly00000gp/T/hsperfdata_adu-tp01/project
[info] set current project to hsperfdata_adu-tp01 (in build file:/private/var/folders/hs/0vkhntfd557c8z8g8b_p7ly00000gp/T/hsperfdata_adu-tp01/)
[info] sbt server started at local:///Users/xxxxx/.sbt/1.0/server/143baa47709c908603e6/sock
[info] started sbt server
sbt:hsperfdata_xxxxx>
When we had this issue, it was caused by OS X restricting the permissions of the java executables. Since the process is not able to read/write to the current directory, sbt diverts to the tmp folder.
In System Preferences => Security & Privacy make sure the Full Disk Access property is turned on for the appropriate executables (probably java and javac in this case) or move your source code our of what OS X considers the protected part of the file system (we moved our builds from the user home directory to /code and sbt worked fine).

Tasks stuck with sbt 1.0

Following the "https://www.coursera.org/learn/scala-spark-big-data/supplement/R7FJ9/eclipse-tutorial" course, I'm trying to create a eclipse project from sbt, by adding sbteclipse plugin.
sbt new scala/hello-world.g8
cd hello-world
echo 'addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "5.1.0")' > project/plugins.sbt
sbt> eclipse
It should last a minute and return several successful, but instead it stay still, it gets stuck. The message I've received is
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
I've installed sbt 1.0.2, it fails, then I'd install sbt 0.13.16 and still fails. Checking the sbt version it says it's 1.0.2 (although I'd uninstalled sbt 1.0.2 to install the older version).
checking https://github.com/typesafehub/sbteclipse/issues/346 it seen like this is a brand new issue, it's supose the gets fix by typing sbt.version=0.13.16 in project/build.properties, but even after that the problem persist. I don't know if problem persist because is a fix for Mac and I'm working on windows 10, or if the solution saids is for sbt.version=0.13.15 and I'm using sbt.version=0.13.16.
Checking the \scala hello world\hello-world\eclipse file it says:
[error] [C:\Users\israel.rodriguez\Documents\eclipse\scala hello world\hello-world\project\plugins.sbt]:1: unclosed character literal
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore?
Terminate batch job (Y/N)?
checking the \scala hello world\hello-world\project\plugins.sbt I found :
'addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "5.1.0")'
when removing the ' ' at the beginning and the end. the \scala hello world\hello-world\eclipse file saids :
[info] Loading project definition from
C:\Users\israel.rodriguez\Documents\eclipse\scala hello
world\hello-world\project
[info] Set current project to hello-world (in build file:/C:/Users/israel.rodriguez/Documents/eclipse/scala%20hello%20world/hello-world/)
and it remain there for eternity. with the same message on windows shell :
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
MaxPermSize=256m; support was removed in 8.0
My question is, what's wrong over here? ,and how can I solve or work around this. Sorry for the rookie question, is my firt time with sbt, scala or spark.
I was in exact same scenario with the same problem as mentioned above. Here is how I overcame it.
Firstly there is no problem in steps that you follow everything is correct except:
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "5.1.0")
Use latest version for the sbt plugin form, which is:
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "5.2.4")
The latest version handles the hanging console problem as described here, also there is no need to use '' at the beginning and the end.
Step: 1. Download and install sbt from here (windows sbt-1.0.4.msi)
Step 2. Go to cmd prompt and type sbt about, it will display something like this
C:\Users\rajnish.kumar>sbt about
"C:\Users\rajnish.kumar\.sbt\preloaded\org.scala-sbt\sbt\"1.0.4"\jars\sbt.jar"
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[warn] Executing in batch mode.
[warn] For better performance, hit [ENTER] to switch to interactive mode, or
[warn] consider launching sbt without any commands, or explicitly passing 'shell'
[info] Loading project definition from C:\Users\rajnish.kumar\project
[info] Set current project to rajnish-kumar (in build file:/C:/Users/rajnish.kumar/)
[info] This is sbt 0.13.15
[info] The current project is {file:/C:/Users/rajnish.kumar/}rajnish-kumar 0.1-SNAPSHOT
[info] The current project is built against Scala 2.10.6
[info] Available Plugins: sbt.plugins.IvyPlugin, sbt.plugins.JvmPlugin, sbt.plugins.CorePlugin, sbt.plugins.JUnitXmlReportPlugin, sbt.plugins.Giter8TemplatePlugin
[info] sbt, sbt plugins, and build definitions are using Scala 2.10.6
I'm not sure why it says "This is sbt 0.13.15"; if anybody knows please answer it here.
Step 3. Go to your workspace form cmd prompt and type and hit sbt new sbt/scala-seed.g8, type project name as (e.g. hello) when asked.
Step 4. Now move to your project directory i.e cd hello
Step 5. Now type echo addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "5.2.4") > project/plugins.sbt
Step 6. Make sure you are inside the hello project and then instead of calling sbt > eclipse (if you will do like this way then it will get stuck in the same infinite loop as you're getting) only call sbt and it will start the server on default port.
Step 7. Now type eclipse, now it will start downloading all dependency, eventually it will show Successfully created Eclipse project files for project(s):
Step 8. Import project (hello) to Eclipse as normal and it will work. Please not do not call sbt > eclipse in single go call them one by one first call sbt and when server is started then only call Eclipse.
I had the same issue when I start reading about scala.
Concerning the problem of [\plugins.sbt]:1: unclosed character literal] use :
echo addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "5.1.0") > project/plugins.sbt
Remove the simple quote from the begining and the end of the echo first parameter
I had the very same issue as you describe. I use Windows 10 and wonder if it just occurs on that system.
It works if I use sbteclipse 5.2.4 and some other small modifications:
cd hello-world
echo 'addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "5.2.4")' > project/plugins.sbt
Next, I need to go to project/plugins.sbt and remove the ' at the beginning and end of the line.
Now, do NOT enter sbt > eclipse! This will just pipe the sbt console into a the newly created file and you won't execute eclipse. Instead, typ the following into the command line:
sbt eclipse

In an sbt 0.13.7 project, compile the compiler-interfaces without compiling the project code

In a freshly checked out sbt ( 0.3.7 ) project and empty ivy cache, is it possible to trigger compilation of the compiler-interface(s) needed without compiling the project itself? I have poked around but haven't found a way.
Currently if a compiler-interface is required it will be created during compilation of the project. I would like to have this compiled directly in a separate command if possible. This would allow CircleCi to cache it saving 1-3 minutes with every build because it could be cached in the dependencies section of the circle.yml.
In sbt 0.13.12 compile:compileIncremental seems to do the trick. I ran inspect compile and inspected its dependencies to find the command.

Deploying Play Framework 2.3.x (Java style) on Ubuntu 14.04 production server

I have a Play application on my laptop, which I am trying to get running on my production server. Locally the application compiles and works without errors.
On my production server, I do the following:
root#example:~/mysite# ./activator
[info] Loading project definition from home/mysite/project
[info] Set current project to mysite (in build file:/home/mysite/)
[mysite] $ ~run
--- (Running the application, auto-reloading is enabled) ---
[info] play - Listening for HTTP on /0:0:0:0:0:0:0:0:9000
(Server started, use Ctrl+D to stop and go back to the console...)
[info] Compiling 4 Scala sources and 2 Java sources to /home/mysite/target/scala-2.11/classes...
[info] 'compiler-interface' not yet compiled for Scala 2.11.1. Compiling...
Killed
root#example:~/mysite#
As you can see, `compiler-interface' is not yet compiled. I am wondering why this is. Is this because I don't have the Scala compiler installed?
Probably the Killed part is because you don't have enough memory to compile your application.
Either way I think it would be better to deploy your app already compiled than compile it on place. You can do that using ./activator dist.
This will generate a zip file in YOUR_PROJECT_DIR/target/universal named YOUR-PROJECT-NAME-VERSION.zip with everything already compiled and with every needed dependency packaged already. You can then upload this file to somewhere in your production server, extract it and then start the server using
$ EXTRACTED_FOLDER/bin/YOUR-APP-NAME
If you still want to build it in your production server, try to reduce the memory used by activator using
$ ./activator -mem 200 # MAX 200mb
// (...)
[PROJECT-NAME] ~start
Note that ~run will start your server in dev mode (ie sensible information like stacktraces etc can be shown on error) and recompile it on changes, so you may want to use start or ~start instead.

Play2 run via Sbt in dev mode on Osx Causes Out of Memory

I can't seem to find similar issues.
On OSX, play2 bin seems to run okay but using sbt causes oom errors and scala files never complete compiling
--- (Running the application from SBT, auto-reloading is enabled) ---
[info] play - Listening for HTTP on port 9000...
(Server started, use Ctrl+D to stop and go back to the console...)
[info] Compiling 15 Scala sources and 2 Java sources to
/Users/tripled153/Development/src/Foundation/playApp/target/scala-2.9.1/classes...
Java HotSpot(TM) 64-Bit Server VM warning: Exception
java.lang.OutOfMemoryError occurred dispatching signal Unknown Signal
to handler- the VM may need to be forcibly terminated
I'm running with extended memory settings but still no luck. Running in staged (prod) mode appears okay though.
rundev.sh
export SBT_OPTS="-Xms512M -Xmx3536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:MaxPermSize=512M"
sudo sbt clean run