Unnecessary recompilations by SBT - scala

Question

I have a question about seemingly unnecessary recompilations by SBT. I have the following scenario: I’m running SBT inside a docker container, by attaching the volume with my application source code to the container, and launching the container with sbt as entry point. If I continuously run SBT inside that container, it doesn’t recompile whole app, which is good.
However, if I start SBT natively on OS X, it does the full recompilation. If after that I start it again inside docker, it again does the full recompilation. It takes very long, and is really annoying. What could be the reason for such behavior?
Here's how I launch the SBT in the container:
docker run --name=bla -it --net=host -v /Users/me/.ivy2:/tmp/.ivy2 \
-v /Users/me/.aws/config:/root/.aws/config \
-v /Users/me/.sbt:/root/.sbt \
-v /Users/me/projects/myapp:/src 01ac0b888527 \
/bin/sh -c 'sbt -Dsbt.ivy.home=/tmp/.ivy2 -Divy.home=/tmp/.ivy2 -jvm-debug 5005 -mem 3072'
My Java, Scala and SBT versions are the same on the host and in the container. Concretely: Scala 2.11.8, Java 1.8.0_77, SBT 0.13.11

Answer 1

Okay, after a day of debugging I've found the way around this problem.
SBT invalidates the compiled classes mainly based on the following rules:
Canonical path
Last modification date
That is, paths and modification dates have to be exactly same for
Source code files
Ivy dependencies jars
JRE's jars
First 2 points are quite easy to achieve, because it's just mapping of docker volumes. The crucial thing is to map to exactly same path as on the host machine. For example, if you work on OS X as I do, path your project sources probably looks like this: /Users/<username>/projects/bla, so in your docker run command you have to do something like:
docker run ... -v /Users/<username>/projects/bla:/Users/<username>/projects/bla ...
You don't care about timestamps for sources and ivy jars, because they will be exactly same (it's the same files).
Where you have to care about timestamps is the JRE stuff. I build the docker image with JRE baked in (using sbt-docker plugin), so I ended up reading modification date of local JRE libs and setting same dates inside the image:
new mutable.Dockerfile {
...
val hostJreTimestamp = new Date(new File(javaHome + "/jre/lib/rt.jar").lastModified()).toString
val hostJceTimestamp = new Date(new File(javaHome + "/jre/lib/jce.jar").lastModified()).toString
runRaw(s"""touch -d "$hostJreTimestamp" $javaHome/jre/lib/rt.jar""")
runRaw(s"""touch -d "$hostJceTimestamp" $javaHome/jre/lib/jce.jar""")
...
}
And, of course, JRE should also be installed to exactly same path as on the host, which might be problematic if you used to install Java from RPM, for example. I ended up downloading server JRE (which is distributed as .tar.gz) and extracting it to the right path manually.
So, long story short, it worked in the end. No recompilation, no long waiting time. I was able to find the relevant information from 2 main sources: SBT source code, particularly this function: https://github.com/sbt/sbt/blob/0.13/compile/inc/src/main/scala/sbt/inc/IncrementalCommon.scala#L271, and enabling SBT debug output in build.sbt:
logLevel := Level.Debug
incOptions ~= { _.copy(apiDebug = true, relationsDebug = true) }
(prepare for a lot of output)

Answer 2

This is just a guess. As rumoku suggest it may be an issue with repositories but I think it's related with SBT itself. From SBT's point of view you are running two different machines and it must compile all when it thinks the files have changed.
I don't know how SBT or the compiler identifies the versions of Scala and Java but it may be the case that even though you have the exact same versions of Java and Scala in both environments SBT thinks they are different. Which makes sense as the are different OS.