I've tried to run application inside firejail without network. When I run sbt in host system, all goes ok, sbt finds all libraries. Then I try to run sbt inside restricted environment with read-write access to ~/.sbt, ~/.ivy2, ~/.m2 and project folders and read-only access to .sbt binary. Sbt loses all dependent libraries and try go get them from network, which is disabled.
Where do sbt store library cache and why do it refuse to take old compiled and ready to use state?
update: sbt fails on the time it tries to build global plugins. It does not get to the stage where it performs project build. I've tried to add offline := true both to the project and global sbt configuration, but sbt tries to update plugins nevertheless.
update2: I've managed to run sbt successfully without rebuilding all global plugins. That requires access to /etc. But I would prefer not to give access to /etc. What information do SBT need from /etc? Hostname or something else? Why on earth it make cache depend on such things?
update3:
I've discovered the exact cause for SBT failing to recognize already downloaded libraries. /etc/passwd is strictly needed for SBT to work correctly. Could anyone point in what way SBT uses login info exactly?
Related
I have recently installed SBT on Docker Ubuntu machine to get started with Scala. When I started docker initially, it started grabbing all the Java, sbt JAR's from the remote locations (https://repo.scala-sbt.org/scalasbt/debian/sbt-0.13.17.deb).
But, whenever I run sbt command, it again starts downloading the sbt JAR. Is there a way of maintaining a global cache whereby artifacts are only downloaded once and not every time I remote to docker container?
My solution to this was a multi stage build.
Have a “base” docker image.
Copy in only build.sbt, projects.sbt and the file which sets out the sbt version from your project.
That defines the required dependencies. The last line in that base image is “sbt update” - I.e fetch them. That “base image” has the dependencies in it… and is reuseable. Just remember to run it when you change library versions etc to rebuild it.
In the ”build” image… copy over the project and proceed as normal… make sure sbt is resolving from maven-local, and it should use the “cache”… which is already in place from the paragraph above.
I’d be interested to hear other answers, but that’s my solution… YMMV :-).
That works for me on a cloud / Kube pipeline.
I'm noticing very slow build times for my Play 2.3 project on our CircleCI server relative to my local build machine. One reason for this is that on every build, SBT prints out the following:
[info] 'compiler-interface' not yet compiled for Scala 2.11.4. Compiling...
then proceeds to spend about a minute compiling the compiler. On my laptop, this happened exactly one time, then never again. My assumption is that on my laptop, the compiler gets cached somewhere so that it doesn't have to be rebuilt again the next time, whereas on the CI server, that cache directory gets blown out.
CircleCI allows you to specify cache directories that won't get destroyed between builds (or more accurately, get destroyed, then rebuilt) so a simple solution for me would be to tell CircleCI to save that directory.
Problem is, I don't know where it is. I'm already saving ~/.sbt and ~/.ivy2 and it hasn't helped. Is the compiler built to a different location that I can cache? Any other ideas on how to make this problem go away short of downgrading my Scala compiler?
I looked on my setup, and it seems that the compiler-interface should indeed be in ~/.sbt or ~/.ivy2.
Your problem is probably coming from how you save these directories. Are you sure you save them for the correct user? You have to save them for the use that launch the sbt command.
#joe: you can do sbt compile:test instead of sbt compile to also get the test dependencies in the cache (as well as ensure the compile interface gets included in the cache).
I followed the getting start video on Jetbrains website to setup IntelliJ IDEA 13.1 Community Edition to work with Scala. Scala plugin v0.36.431 had been installed. While I created a new Scala SBT project with wizard, there was no src/ directory structure generated in the project. Only two sbt files were generated:
scala-course/
├── build.sbt
└── project
└── plugins.sbt
From the video and other document I know that there should be a src/ directory structure, including src/main/scala, src/test/scala, etc. sbt uses the same directory structure as Maven for source files by default.
I can create those folders manually and mark it as source root. However it is trivial. So my question is: Why IntelliJ IDEA new project wizard doesn't generate the directory structure as said in document? Was I doing something wrong? I checked the preferences and couldn't find anything that seems related.
Normally it should create these folders automatically. It may take a while though - it takes couple of seconds in my case.
When creating project make sure you have selected Scala -> SBT, then proceed with the wizard.
Once the Finish is clicked, the project will be loaded. This part takes couple of seconds, and I can see no src/main/scala nor src/test/scala generated until it's done. Observe the bottom of the screen to see when it's done.
Once the process is finished, you'll see the folders.
If that's not the case, check the settings. You should have the Create directories for empty content roots automatically checked. You may want to check Use auto-import to automatically propagate changes in the build.sbt.
After changing the settings (if the change is required) you may need to refresh the project, as seen in picture below.
This can also happen if you do not have a JDK selected. For some reason you no longer get the option to select an SDK so you must make sure you have configured this before hand. To fix this do the following:
From the welcome screen, go to
Configure -> Project defaults -> Project structure and add the jdk.
Source:
What's the reason for "Error:Cannot determine Java VM executable in selected JDK"?
Thanks to lpiepiora, with his hint I find out the reason.
Because my sbt is newly installed, there is nothing in ~/.ivy2/cache/ and ~/.sbt/boot/. sbt needed to download required dependencies from repositories on network. It happened that my proxy to internet had something wrong, download stuck.
And also need to notice that, if quit IntelliJ IDEA when sbt is running in background, the next time you'll get error of waiting for some lock file. Have to remove the lock file on filesystem and restart IntelliJ IDEA again.
After fixed the network problem, everything work as promised. It requires several minutes, depends on network speed, to download required jar files. After finished, the src/ directory structure is created.
I followed the instructions in this thread but I had a java crash in the final phase in sbt with the configuration bellow and I think this info maybe useful:
The problem happened with IDEA 2016.2, sbt 0.13.8 (I tried later to import using 0.13.12 but the crash was the same), scala 2.11.8 and ubuntu 16.04.
The only way I could make it work was to use java 8 instead of 9.
error: error while loading package, Missing dependency 'object java.lang.Object in compiler mirror', required by /home/jbamaral/.sbt/boot at xsbt.boot.Boot.main(Boot.scala)
...
stack log here
...
[error] scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found.
As you must known, the first time you run SBT, it starts to download his dependencies:
>sbt sbt-version
Getting org.scala-sbt sbt 0.12.4
downloading repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/main/0.12.4/jars/main.jar ...
The problem is that I'm living in Africa and the ISP is not very good: for some unknown reason, they are some domains from which I can not download any file. For example, if I enter to the site of WAMP and try to download the installer, the download begins but it never ends. Normally I fix this issue by using a proxy (such as HideMyAss or ZendProxy), which servers as an intermediate between the site and the ISP, allowing me of download the file.
typesafe.com seems to be one of the not-work-for-download-in-africa domains, as consequence, SBT can not download his dependencies.
The question is, there is a way of download manually the dependencies? (I have downloaded them using a proxy, but I do not known in which folder put them in order to avoid SBT to try to download them)?
Create a directory in the root of your sbt project called lib and put the jars there.
I have near zero knowledge of Maven/Ivy but know that these technologies are behind SBT's ability to download and manage libraries automatically.
When I was trying to generate an IntelliJ Idea project from SBT it (Idea or the generator plug-in) has failed to find cached libraries stored in the common Ivy 2 cache. I've solved the problem by setting up the SBT project to use the old scheme storing downloaded libraries under the project directory. Another project of mine uses old SBT version which does it this way by default. In both cases I use clean command to remove library binaries before backing up and update after to download all the libraries again. Which means I download the same library files many times.
The question is how to set up a local repository to cache the libraries so that SBT update will download the libraries from the local cache if available? Would be nice to use the existing ~/.ivy2/cache directory for the cache if possible.
I tend to believe this is probably fairly simple when you have at least basic Ivy 2 knowledge.
Ivan, this may help.
sbt
> deliver-local
[info] delivering ivy file to /.../target/scala-2.9.1/ivy-1.0.xml
Then, you'll need to find IntelliJ equivalent steps, but in Eclipse you configure the build path with an additional library, "IvyDE Managed Dependencies" offered by the IvyDE plugin
1) browse to the target/scala-version/ivy-1.0.xml file
2) select desired configurations (compile, runtime, test, provided, etc.)
Now you have an ivy representation of your sbt configuration.
This works wonderfully with one annoying exception: on "sbt clean", ivy config file gets blown away and you have to repeat the above. Eclipse project clean preserves ivy config, btw.
Keep posted if this approach can be applied to IntelliJ