I am trying to dockerize the Rest API built with Scala Play so I can use AWS EC2 or Kubernetes to deploy it.
Following this:
:
https://guilhebl.github.io/scala/backend/docker/play/2017/08/23/scala-play-docker-sbt-native-packager-example/ and
it failed using this sequence of command:
sbt playGenerateSecret
sbt dist
sbt docker:publishLocal
docker run -p 9000:9000 -e APPLICATION_SECRET="token from above" play-scala-rest-api-example:1.0-SNAPSHOT
with this official example app:
https://github.com/playframework/play-samples/tree/2.8.x/play-scala-rest-api-example
Other teams need a Dockerfile to deploy the app to AWS EC2 with Jenkins. Right now it seems like a Dockerfile is not generated until sbt docker:publishLocal
It would be better if we can find some official tutorials showing how to do this.
Answers from mfirry:
sbt docker:stage
from https://www.scala-sbt.org/sbt-native-packager/formats/docker.html#tasks
Related
I was following the tutorial for build an scalable websocket app in Heroku, following the tutorial Writing Scalable WebSocket Applications on Heroku and trying to set up Play Framework to work on Eclipse IDE, downloaded play-samples-play-java-hello-world-tutorial from Play Framework site, added sbt-plugin annotation in plugins.sbt file, git bashed from project folder and tried the command sbt, it gives me, following error and I can't proceed with compile command:
server failed to start on local:sbt-server-cc02d04ca19b451f676c. java.io.IOException: Could
not create lock for \\.\pipe\sbt-server-cc02d04ca19b451f676c_lock, error 5
I'm trying to use the sbt-native-packager to produce a Docker image of my Scala play app, I followed the steps described at http://www.scala-sbt.org/sbt-native-packager/formats/docker.html
This is my configuration:
on my plugins.sbt I added the dependency for sbt native packager:
// SBT Native
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.2.1")
on my build.sbt I added the plugins for Universal and Docker:
.enablePlugins(PlayScala, JavaAppPackaging)
I also added some extra properties:
javaOptions in Universal ++= Seq(
// JVM memory tuning
"-J-Xmx1024m",
"-J-Xms512m",
// Since play uses separate pidfile we have to provide it with a proper path
// name of the pid file must be play.pid
s"-Dpidfile.path=/var/run/${packageName.value}/play.pid",
// Use separate configuration file for production environment
s"-Dconfig.file=/usr/share/${packageName.value}/conf/production.conf",
// Use separate logger configuration file for production environment
s"-Dlogger.file=/usr/share/${packageName.value}/conf/logback.xml"
)
// exposing the play ports
dockerExposedPorts in Docker := Seq(9000, 9443)
Then I generate the docker image using the plugin and SBT CLI:
docker:publishLocal
the dockerfile gets generated at ./target/docker/Dockerfile
when I inspect the file I see:
FROM openjdk:latest
WORKDIR /opt/docker
ADD opt /opt
RUN ["chown", "-R", "daemon:daemon", "."]
USER daemon
ENTRYPOINT ["bin/root"]
CMD []
which doesn't seem to contain all the necessary steps to run the app, when I use docker build . I get :
java.nio.file.NoSuchFileException: /var/run/root/play.pid
It seems like the Dockerfile is missing some steps where it should mkdir /var/run/{APP_NAME}/
(* creating folder inside docker container instance)
and chown that folder in order for play to create the PID file.
how to fix the above error ?
What's the error message when starting the docker image and how do you start it?
Then there are a couple of notable things.
play ships with native-packager
You shouldn't have to add any plugin, but only configure docker relevant stuff. You already linked the correct documentation for the package format ( docker ).
Archetypes Vs Formats
Your configuration won't work without the play plugin. Take a look at http://www.scala-sbt.org/sbt-native-packager/archetypes/java_app/index.html which explains how to configure a simple application build.
I'd also recommend to read the format and Archetypes section here:
http://www.scala-sbt.org/sbt-native-packager/introduction.html#archetype-plugins
Native docker build
Native-packager currently generates two docker files, which is confusing and not relevant. Sorry for that confusion. We plan to remove the redundant docker file.
Simply go one level deeper and run the docker build command.
Hope that helps,
Muki
I have a question about seemingly unnecessary recompilations by SBT. I have the following scenario: I’m running SBT inside a docker container, by attaching the volume with my application source code to the container, and launching the container with sbt as entry point. If I continuously run SBT inside that container, it doesn’t recompile whole app, which is good.
However, if I start SBT natively on OS X, it does the full recompilation. If after that I start it again inside docker, it again does the full recompilation. It takes very long, and is really annoying. What could be the reason for such behavior?
Here's how I launch the SBT in the container:
docker run --name=bla -it --net=host -v /Users/me/.ivy2:/tmp/.ivy2 \
-v /Users/me/.aws/config:/root/.aws/config \
-v /Users/me/.sbt:/root/.sbt \
-v /Users/me/projects/myapp:/src 01ac0b888527 \
/bin/sh -c 'sbt -Dsbt.ivy.home=/tmp/.ivy2 -Divy.home=/tmp/.ivy2 -jvm-debug 5005 -mem 3072'
My Java, Scala and SBT versions are the same on the host and in the container. Concretely: Scala 2.11.8, Java 1.8.0_77, SBT 0.13.11
Okay, after a day of debugging I've found the way around this problem.
SBT invalidates the compiled classes mainly based on the following rules:
Canonical path
Last modification date
That is, paths and modification dates have to be exactly same for
Source code files
Ivy dependencies jars
JRE's jars
First 2 points are quite easy to achieve, because it's just mapping of docker volumes. The crucial thing is to map to exactly same path as on the host machine. For example, if you work on OS X as I do, path your project sources probably looks like this: /Users/<username>/projects/bla, so in your docker run command you have to do something like:
docker run ... -v /Users/<username>/projects/bla:/Users/<username>/projects/bla ...
You don't care about timestamps for sources and ivy jars, because they will be exactly same (it's the same files).
Where you have to care about timestamps is the JRE stuff. I build the docker image with JRE baked in (using sbt-docker plugin), so I ended up reading modification date of local JRE libs and setting same dates inside the image:
new mutable.Dockerfile {
...
val hostJreTimestamp = new Date(new File(javaHome + "/jre/lib/rt.jar").lastModified()).toString
val hostJceTimestamp = new Date(new File(javaHome + "/jre/lib/jce.jar").lastModified()).toString
runRaw(s"""touch -d "$hostJreTimestamp" $javaHome/jre/lib/rt.jar""")
runRaw(s"""touch -d "$hostJceTimestamp" $javaHome/jre/lib/jce.jar""")
...
}
And, of course, JRE should also be installed to exactly same path as on the host, which might be problematic if you used to install Java from RPM, for example. I ended up downloading server JRE (which is distributed as .tar.gz) and extracting it to the right path manually.
So, long story short, it worked in the end. No recompilation, no long waiting time. I was able to find the relevant information from 2 main sources: SBT source code, particularly this function: https://github.com/sbt/sbt/blob/0.13/compile/inc/src/main/scala/sbt/inc/IncrementalCommon.scala#L271, and enabling SBT debug output in build.sbt:
logLevel := Level.Debug
incOptions ~= { _.copy(apiDebug = true, relationsDebug = true) }
(prepare for a lot of output)
This is just a guess. As rumoku suggest it may be an issue with repositories but I think it's related with SBT itself. From SBT's point of view you are running two different machines and it must compile all when it thinks the files have changed.
I don't know how SBT or the compiler identifies the versions of Scala and Java but it may be the case that even though you have the exact same versions of Java and Scala in both environments SBT thinks they are different. Which makes sense as the are different OS.
We are creating a customize version of Spark since we are changing some lines of code from ALS.scala. We build the customize spark version using
mvn command:
./make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn
However, upon using the customized version of Spark, we run into this error:
Do you guys have some idea on what causes the error and how we might solve the issue?
I am actually using a jar file in the local machine by building them using sbt: sbt compile then sbt clean package and putting the jar file here: /Users/user/local/kernel/kernel-0.1.5-SNAPSHOT/lib.
However in the hadoop environment, the installation is different. Thus, I use maven to build spark and that's where the error comes in. I am thinking that this error might be dependent on using maven to build spark as there are some reports like this:
https://issues.apache.org/jira/browse/SPARK-2075
or probably on building spark assembly files
Where can I override the repo URL that SBT uses to fetch its launcher when the SBT instance is provided by Travis-CI?
http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.8/sbt-launch.jar is currently giving 404s and, as a result, the builds are failing on Travis with:
$ sbt clean dependencyUpdates coverage test coverageReport
Detected sbt version 0.13.8
Downloading sbt launcher for 0.13.8:
From http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.8/sbt-launch.jar
To /home/travis/.sbt/launchers/0.13.8/sbt-launch.jar
Download failed. Obtain the jar manually and place it at /home/travis/.sbt/launchers/0.13.8/sbt-launch.jar
I ran into the same problem today, and logged an issue for travis-ci: https://github.com/travis-ci/travis-ci/issues/4527
As a workaround, you can download the sbt-launcher.jar by adding a before_script section to your .travis.yml
before_script:
- mkdir -p $HOME/.sbt/launchers/0.13.8/
- curl -L -o $HOME/.sbt/launchers/0.13.8/sbt-launch.jar http://dl.bintray.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.8/sbt-launch.jar
We just deployed the fix to production. SBT builds should be working now.
Sorry for the inconvenience.
https://github.com/travis-ci/travis-ci/issues/4527#issuecomment-124123880