I'm noticing very slow build times for my Play 2.3 project on our CircleCI server relative to my local build machine. One reason for this is that on every build, SBT prints out the following:
[info] 'compiler-interface' not yet compiled for Scala 2.11.4. Compiling...
then proceeds to spend about a minute compiling the compiler. On my laptop, this happened exactly one time, then never again. My assumption is that on my laptop, the compiler gets cached somewhere so that it doesn't have to be rebuilt again the next time, whereas on the CI server, that cache directory gets blown out.
CircleCI allows you to specify cache directories that won't get destroyed between builds (or more accurately, get destroyed, then rebuilt) so a simple solution for me would be to tell CircleCI to save that directory.
Problem is, I don't know where it is. I'm already saving ~/.sbt and ~/.ivy2 and it hasn't helped. Is the compiler built to a different location that I can cache? Any other ideas on how to make this problem go away short of downgrading my Scala compiler?
I looked on my setup, and it seems that the compiler-interface should indeed be in ~/.sbt or ~/.ivy2.
Your problem is probably coming from how you save these directories. Are you sure you save them for the correct user? You have to save them for the use that launch the sbt command.
#joe: you can do sbt compile:test instead of sbt compile to also get the test dependencies in the cache (as well as ensure the compile interface gets included in the cache).
Related
Package A depends on package B (0.1.0). I can modify package B, re-publish it (0.1.1), and then switch back over to package A, I run SBT clean, restart SBT, then run compile, and it grabs the latest version and understands that the classes have changed.
Ensime, however, does nothing. It seems to have some memory of the old class. Removing the .ensime_cache does not help. Recreating the .ensime file seems to show it resolving the latest version of the package that I want, but does nothing. Closing and re-opening the files does not help. Running M-x ensime does not do anything. Inspecting the class shows the old version, which is apparently living somewhere, but I don't know where.
Please help me understand what ensime is doing.
OK - The problem was my understanding of how emacs via ensime is communicating with SBT. I was running my own SBT in a shell, which ensime, naturally, knew nothing about.
If, however, I run ensime-sbt, then the clean/re-run, then ensime picks up the changes and all editing can progress peacefully. So it appears that it runs its own internal sbt, even when you're not running ensime-sbt explicitly. Or if it's not doing that, the effect is at least the same...
I've tried to run application inside firejail without network. When I run sbt in host system, all goes ok, sbt finds all libraries. Then I try to run sbt inside restricted environment with read-write access to ~/.sbt, ~/.ivy2, ~/.m2 and project folders and read-only access to .sbt binary. Sbt loses all dependent libraries and try go get them from network, which is disabled.
Where do sbt store library cache and why do it refuse to take old compiled and ready to use state?
update: sbt fails on the time it tries to build global plugins. It does not get to the stage where it performs project build. I've tried to add offline := true both to the project and global sbt configuration, but sbt tries to update plugins nevertheless.
update2: I've managed to run sbt successfully without rebuilding all global plugins. That requires access to /etc. But I would prefer not to give access to /etc. What information do SBT need from /etc? Hostname or something else? Why on earth it make cache depend on such things?
update3:
I've discovered the exact cause for SBT failing to recognize already downloaded libraries. /etc/passwd is strictly needed for SBT to work correctly. Could anyone point in what way SBT uses login info exactly?
This is getting tedious. I created a test project, simple java hello, to test a problem I have in another project. Having created the project, I could run it and build it to a jar.
Then I added some ivy stuff and tried a build which did not work. The ivy stuff consisted of ivy.xml ivysettings.xml and 2 tasks added to the otherwise default netbeans build.xml.
So then I removed the ivy files and the ivy sections from build.xml. I then did a diff on that and the previous unedited build.xml file and they are identical.
So now why after a clean build do I get this error?
/home/tester/WORKSPACE/NetBeans/Foo/nbproject/build-impl.xml:63: Source resource does not exist: /home/tester/WORKSPACE/NetBeans/Foo/lib/nblibraries.properties
Correct that file does not exist and it does not exist in any other project I have access to whether they use ivy or not.
I have reset everything I did yet Netbeans cannot continue. I have tried to work out how the hell nb actually builds a project but got lost. Just don't know what to say.
I guess nb and myself differ on what clean actually means and also on whether a task should be able to produce exact same results everytime given the same input, something that nb does not seem to be able to do. That or it cannot recover from the real world or their documentation just hides the useful information like how to reset a project.
I'm assuming there is some nb state that needs to be cleaned up manually but have no idea where to start. Any ideas please?
#Tim Biegeleisen, your comment got me thinking.
I shut down/restarted netbeans then tried the build again and it worked. So it seems that ivy is still running inside netbeans and has some form of state for my project that it will not release. I'll have to dig around to see if there is a way to reset ivy state without restarting nb but this whole process is turning in to a very tedious onerous task. I seem to be spending more time on the dev tools than I am on solving my coding tasks.
So the solution/workaround to my problem seems to be to restart netbeans to clear ivy state.
Currently all our project builds with Maven on Windows. We were not successful with making incrementally compiled code to work in run time (50% of the cases it was failing with some kind of error), so to benefit from warm compiler and (maybe?) properly working incremental compilation we think about moving to SBT. However currently I have only one sprint to work on it, and I'm afraid to put all the eggs in a single basket and try to migrate whole project in a sprint. I need to find a way to make this change gradual, so I could advance one module at a time. So here are the main questions:
How can I include SBT modules in Maven build (or maybe vise versa, having my "parent" in SBT, yet part of the modules still building with Maven)?
How can we still benefit from IDE support (currently IntelliJ 13), like updated indices on changes in pom / Build.scala, task & goals invocation and so on?
Any advises on subject are highly appreciated.
Eventually we did the switch and don't regret it. Writing SBT tasks is easier, because it's plain Scala. Incremental compilation works now (used to fail in Maven with java.lang.InternalError: Enclosing method not found when deployed to JBoss) and build time is significantly faster. Unfortunately we did not find a way to make a gradual switch, so we had to take the risk. Incremental compiled jars still didn't work, yet Typesafe are about to fix this issue in 2.11.6
When I edit a scala file in my Play 2 app, sometimes only a few files are recompiled, but often the entire codebase must be recompiled:
[info] Compiling 1 Scala source to /home/michael/code/superglot/target/scala-2.10/classes...
[success] Compiled in 1s
versus
[info] Compiling 2 Scala sources to /home/michael/code/superglot/target/scala-2.10/classes...
[info] Compiling 52 Scala sources and 1 Java source to /home/michael/code/superglot/target/scala-2.10/classes...
[success] Compiled in 13s
However I see no discernible pattern for when a full recompile is necessary. If I add somewhitespace to a model or controller class it may only compile that file, but doing the same to a comparable file will trigger a recompile.
I would love to have as many of my reloads be closer to 1s, because currently I'm waiting for a full recompile more often than not. I'd gladly refactor my code to make the areas I'm working on faster to reload, but I don't know what I could even do to achieve that. Are frequent recompiles normal for a typical Play 2 app, or is there something anomalous about mine?
In general, if you change the "source API" of a file, dependencies of that file are recompiled. The source API consists of the signatures of non-private methods and types. So, if you have a file that everything depends on, changes to signatures in that file may cause a lot of recompilation. Also, when an ancestor's API changes, all descendents must be recompiled.
You can get some additional information from last compile, such as what triggered other files to be recompiled. (In a multi-module build, last <project-name>/compile) You can
If adding insignificant whitespace causes other files to be recompiled, it is always a bug, often in scalac itself. An example of such a bug is SI-7361 (not that it is useful to anyone besides compiler developers!) that was worked around in sbt here. For these to be fixed, we need a reproducible test case. (Given the effort often involved in that, you might wait for 0.12.4 or 0.13.0 to see if those fix your problem.)
0.13.0 has some improvements that will hopefully reduce what gets invalidated when APIs change.
I think you are going for the slightly wrong approach here. As there is no way to successfully determine which changes will cause the full re-compilation, that path leads nowhere.
For learning purposes perhaps it is helpful to be aware of the compiler's internals to a better extent, but from a productivity perspective there is a much better choice available, and that is jRebel, who have been giving out free Scala licenses for years.
JRebel
From here, you can get a free Scala license within minutes.
Then go ahead and add this to the sbt config. It has to go straight into the sbt file.
Linux/Mac
java -noverify -javaagent:/opt/zt/jrebel/jrebel.jar \
-Xmx1024M -Xss2M -XX:MaxPermSize=512m -XX:+CMSClassUnloadingEnabled -jar \
`dirname $0`/sbt-launch-0.12.jar "$#"
Windows
set SCRIPT_DIR=%~dp0
java -noverify -javaagent:c:/opt/zt/jrebel/jrebel.jar \
-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m -Xmx1024M -Xss2M \
-jar "%SCRIPT_DIR%\sbt-launch-0.12.jar" %*
At this stage, if you've done everything right, when you re-start the SBT console you will see that the jRebel Agent has been initiated and you will benefit from the real-time redeployment of your changes. When you Save in your IDE, JRebel will reload only the changes you've made.