How to prevent SBT from recompiling modified .class files? - scala

In our project, we have a enhance post-process to the .class files generated by compile. This enhance step actually modifies the generated .class files then overrides it.
enhance <<= enhance triggeredBy (compile in Compile)
The problem is that sbt has a mechanism called incremental recompilation. It monitors the generated .class file. Every time the enhancer overrides the generated .class file, sbt recognises these modifications and recompiles related sources in next compile command.
To us, a recompilation is a very time-consuming work. We want to stop sbt from recompiling modified .class file. That may mean making sbt only monitor source changes, not output changes.
I did some searching on this. But I found little things about this. Now I know a trait called Analysis is likely responsible for the mapping from sources to output .class files. So I ask help from you guys.
Ps: we may solve this problem by putting the output of enhance to another folder, but it is not preferred.

sbt strongly discourages mutations to files. You should generate different files instead. By doing so, you will solve your problem, since the incremental compiler of sbt will still be looking at the unaltered .class files. You will have some rewiring to do:
Send the outputs of the compile task somewhere else:
classDirectory in Compile := crossTarget.value / "classes-orig"
Processing these .class files with your tool, and send them to crossTarget.value / "classes" (the original classDirectory:
enhance <<= enhance triggeredBy (compile in Compile)
enhance := {
val fromDir := (classesDirectory in Compile).value
val toDir := crossTarget.value / "classes"
...
}
Rewire productDirectories to use crossTarget.value / "classes" anyway (otherwise it'll go looking in your modified classDirectory:
productDirectories in Compile := Seq(crossTarget.value / "classes")
Make sure that products depends on your enhance task:
products in Compile <<= (products in Compile) dependsOn enhance
You may need some more rewiring if you have resources (see copyResources). But basically you should be able to get there.

I said about that sbt monitors the output .class file. When a .class file is modified, it recompiles the .class file's source.
After some research, we found that sbt notices file's modification by its last modified time. That is to say, we can fool sbt by rolling back the last modified time after the modification, so that sbt won't notice any changes.
So, our solution is simple but effective:
find all .class files
note down their last modified time
do the enhance
put back the former last modified time
This is a small trick. We still expect more robust solutions.

Description:
A little like Chenyu, I had to write a plugin for SBT 1.x, that would enhance compiled classes and later I wanted to make sure that those enhanced classes were used for building the jar.
I did not want to hack this solution, so Chenyu's answer was not acceptable to me and sjrd's answer was very helpful but adjusted to SBT 0.13.
So here is my working solution, with little comments:
Code:
object autoImport {
val enhancedDest = settingKey[File]("Output dir for enhanced sources")
}
def enhanceTask: Def.Initialize[Task[Unit]] = Def.task {
val inputDir = (classDirectory in Compile).value
val outputDir = enhancedDest.value
enhance(inputDir, outputDir)
...
}
override def projectSettings: Seq[Def.Setting[_]] = Seq(
enhancedDest := crossTarget.value / "classes-enhanced",
products in Compile := Seq(enhancedDest.value), // mark enhanced folder to use for packaging
// https://www.scala-sbt.org/1.0/docs/Howto-Dynamic-Task.html#build.sbt+v2
compile in Compile := Def.taskDyn {
val c = (compile in Compile).value // compile 1st.
Def.task {
(copyResources in Compile).value // copy resources before enhance
enhanceTask.value // enhance
c
}
}.value
)

Related

Get full classpath without compilation in SBT

I need a piece of code using SBT, possibly internals, which acquires the full classpath of an SBT project without invoking a compilation. Normally I would use "Runtime / fullClasspath", but that needs the project to be compiled first. Is there any way to get the fullClasspath without triggering a compile? I thought the build.sbt alone determined the classpath, and compilation (in theory) isn't necessary.
I asked on the SBT gitter channel, and there it was also mentioned to use dependencyClasspath. dependencyClasspath doesn't require compilation of the root project, but it does require compilation of all the dependents. So that doesn't solve it yet for me. I'm looking for the complete classpath required to running the root project, without compiling its constituents.
I'm interested in any way to work around this, so if there are any farfetched workarounds, those are welcome too.
An example of what I have now:
Global / printMainClasspath := {
val paths = (rootProject / fullClasspath).value
val joinedPaths = paths
.map(_.data)
.mkString(pathSeparator)
println(joinedPaths)
}
This works partially, it creates a task "printMainClasspath" which prints the full classpath. But if I call it in the sbt shell, it compiles the code first. I'd like the classpath to be printed without invoking a full compile of the project. Ideally, only all build.sbt's of all constituent projects are compiled. Is there a way?
val dryClasspath = taskKey[Seq[File]]("dryClasspath")
dryClasspath := {
val data = settingsData.value
val thisProj = thisProjectRef.value
val allProjects = thisProj +: buildDependencies.value.classpathTransitiveRefs(thisProj)
val classDirs = allProjects.flatMap(p => (p / Runtime / classDirectory).get(data))
val externalJars = (Runtime / externalDependencyClasspath).value.map(_.data)
classDirs ++ externalJars
}

How IntelliJ (or sbt-structure) identifies test sources?

Background
I've created template-scala-project which, among other amenities, defines configurations FunctionalTest and AcceptanceTest (in addition to IntegrationTest, which is provided by SBT).
This way, the relevant bits of the directory structure are:
src/main/scala - compile sources - Compile
src/test/scala - test sources - Test
src/it/scala - it sources - IntegrationTest
src/ft/scala - ft sources - FunctionalTest
src/at/scala - at sources - AcceptanceTest
Behavior in SBT
I can run only functional tests, for example, like this:
ft:test
Everything works as I've planned. I can even share test sources to it sources, or ft sources, or at sources... which is a common practical requirement.
Behaviour in IntelliJ
IntelliJ recognizes that src/test/scala and src/it/scala are test sources. IntelliJ does not have any distinction between them, I mean: no distinction between test sources and integration test sources, ... but it's OK. All I need is that src/it/scala is recognized as test sources. And it works as that.
However, IntelliJ does not recognize src/ft/scala as test sources; IntelliJ does not recognize src/at/scala as test sources.
I have inspected the XML file produced by sbt-structure but I was unable to understand the pattern or logic behind it. However (and apparently!) src/ft/scala and src/at/scala should appear under <configuration id="test"> in order to be eligible for being considered as test sources.
Question
In order to test my hypothesis above, I would like to force src/ft/scala to appear under <configuration id="test">, employing "something" in the build.sbt file. How could I accomplish that?
If you want a directory in IntelliJ to be considered a Test Source directory, you can configure it as such by simply right clicking it and selecting Mark Directory As > Test Sources Root
That changes your project structure configuration to let IntelliJ know that tests reside there. The same is true of your resources if those need to be marked appropriately.
With THAT said, I suppose I'm not 100% sure if IntelliJ's use of sbt will recognize that and run them appropriately, but I would expect it to.
After some experimentation, I apparently found something which works well enough. I cannot claim that I've found a solution, but at least it seems to work as I would expect: IntelliJ recognizes FunctionalTest (test) sources and AcceptanceTest (test) sources, the same way it recognizes Test sources and IntegrationTest sources.
Answer:
Create a configuration which extends Runtime and Test. Doing this way, src/ft/scala appears under <configuration id="test">, as the XML file is produced by sbt-structure. See the example below for configuration FunctionalTest:
lazy val FunctionalTest = Configuration.of("FunctionalTest", "ft") extend (Runtime, Test)
As a bonus, I show below some other settings which I find useful to be associated to configuration FunctionalTest:
lazy val forkSettings: Seq[Setting[_]] =
Seq(
fork in ThisScope := false,
parallelExecution in ThisScope := false,
)
lazy val ftSettings: Seq[Setting[_]] =
inConfig(FunctionalTest)(Defaults.testSettings ++ forkSettings ++
Seq(
unmanagedSourceDirectories in FunctionalTest ++= (sourceDirectories in Test).value,
unmanagedResourceDirectories in FunctionalTest ++= (resourceDirectories in Test).value,
))
I also did something similar for configuration AcceptanceTest.
You can find a project which exercises such kind of configurations at:
http://github.com/frgomes/template-scala-project

SBT not generating Avro classes

I'm creating a new project in sbt, and I'm having a hard time getting avro files to generate. I'm using avrohugger in my sbt plugins:
$ cat ~/.sbt/0.13/plugins/plugins.sbt
addSbtPlugin("org.ensime" % "sbt-ensime" % "2.0.1")
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "5.2.3")
addSbtPlugin("com.julianpeeters" % "sbt-avrohugger" % "1.1.0")
And, as recommended on the avrohugger github page, I'm defining the settings for the avro tasks in build.sbt:
sourceGenerators in Compile += (avroScalaGenerate in Compile).taskValue
(avroSpecificSourceDirectory in Compile) := new java.io.File("src/main/resources/avro")
(avroScalaSpecificCustomNamespace in Compile) := Map("" -> "avro")
(avroSpecificScalaSource in Compile) := new java.io.File("target/generated-sources/avro")
I have tried also using avroScalaGenerate and avroScalaGenerateScavro to no avail. I've tried running the specific task step (sbt avroScalaGenerateSpecific), which succeeds but has no visible output, even when run with show. clean and compile produces the expected output in target for the regular classes but not for anything from avro. So all together, I'm left with no error messages but no results.
After some digging, this was a few different issues.
First was that (avroSpecificSourceDirectory in Compile) := new java.io.File("src/main/resources/avro") is not the full namespace, and that actually made a difference. I was using my.cool.namespace because my folder was incorrect and also had the .'s in it. Remaking the folder structure to match the expected namespace, and changing this to slashes, helped it find the files it needed.
I was able to remove the namespace mapping, since my avro definitions didn't need it, and I didn't need the output folder because the default is fine.
Finally, I fixed which generator was being used in the compile step, and used the specific record generator throughout the build file.
So now, the relevant and completed build.sbt part looks like this:
//add avro generation to the compile step
sourceGenerators in Compile += (avroScalaGenerateSpecific in Compile).taskValue
//tell the avro compiler where to look for avro sources.
(avroSpecificSourceDirectory in Compile) := new java.io.File("src/main/resources/my/cool/namespace/avro")
This now generates avro when needed without any errors.

sbt : Is there a better way of structuring large build.sbt files

Problem description: large sbt file
Our current build.sbt spanns 250+ lines.
We have two problems:
readability
current approach grouping of data and comments:
// Plugins ///////////////////////////////////////////////////
enablePlugins(DockerPlugin)
// basic configuration : projects ///////////////////////////
name := """projectName"""
lazy val projectName =
(project in file(".")).....
logic reuse
We have some configuration logic we would like to share between different projects.
Question
Is there a way to include other *.sbt files?
Or do you have a suggestion how to solve this problem without resorting to write a sbt plugin?
Is there a way to include other *.sbt files?
Yes, you can simply put parts of your build.sbt file into other *.sbt files within your project root. SBT picks up all *.sbt files and merges them together, as there was only a single large file.
One thing you can do is to factor out parts of your build info into scala files in the project directory.
E.g. in our build we have a file Dependencies.scala in the project directory, which contains all dependencies of the various projects in one place:
object Dependencies {
val akka_actor = "com.typesafe.akka" %% "akka-actor" % "2.3.13"
// ...
}
This can then be imported from the build.sbt:
import Dependencies._
lazy val foo = Project(...) dependsOn (akka_actor, ...)
You can also put tasks and commands into objects in the project directory.
Update: One thing I often do when looking for inspiration about how to organize a build is to look at the build of complex, high-profile scala projects such as akka. As you can see, they have moved a lot of logic into scala files in the project directory. The build itself is defined in AkkaBuild.scala.

Importing in Scala - How to add a jar to the classpath permanently

So I'm having trouble importing a package in scala. I downloaded the package, breeze, from github, because I wanted to sample from probability distributions.
I'm used to Python, where I can just download a package, include it in the path, and then import it in the code. So I'm very new to the idea of using a separate "build tool" to use 3rd party packages.
So I downloaded the "breeze" source code from github, installed sbt, and then within the source code for breeze, I ran sbt, and then I used the "assembly" command to get a .jar for breeze.
If I want to use the scala interpreter, I can import the package just fine with
scala -cp breeze-master/target/scala-2.11/breeze-parent-assembly-0.8.jar
The problem is that I want to use this package in a separate piece of code that I'm writing in a file called Chromosome.scala. And when I try to import the package (as seen below), I get an error:
error: not found: object breeze
Here's my code:
// Chromosome.scala
import breeze.stats.distributions._
class Chromosome(s:Int, bitstring:Array[Boolean]) {
val size:Int = s;
val dna:Array[Boolean] = bitstring;
var fitness:Int = 0;
def mutate(prob:Float):Unit = {
// This method will randomly "mutate" the dna sequence by flipping a bit.
// Any individual bit may be flipped with probability 'pm', usually small.
val pm:Float = prob;
// The variable bern is an instance of a Bernoulli random variable,
// whose probability parameter is equal to 'pm'.
var bern = new Bernoulli(pm);
//Loop through the 'dna' array and flip each bit with probability pm.
for (i <- 0 to (size - 1)) {
var flip = bern.draw();
if (flip) {
dna(i) = !(dna(i));
}
}
}
“A script?” What is this and what is its connection to your SBT project? Scala scripts include their own launch command for the Scala interpreter / compiler ( / REPL…). If you want to access things beyond the standard library, you'll have to inclulde them there. Alternately, you can use the SBT Start Script plug-in to produce a launcher script that will include the project dependencies. It will only work locally, though you can write some text processing and other shell scripting to produce a portable launch bundle.
It looks like there's some understandable confusion about what sbt is supposed to do for you.
First off, you generally don't need to download a package from github and build it from source. In the rare cases that you do (such as when you require features that have not made it into a release of the library), sbt can handle the grunt work.
Instead, you tell sbt a little about the project you're building (including what its dependencies are), and sbt will download them, compile your code, and set up the runtime classpath for the scala interpreter (amongst myriad other build-related tasks).
Just follow the directions on the breeze wiki. Specifically, create a build.sbt file in your project's root folder and copy this into it:
libraryDependencies ++= Seq(
// other dependencies here
"org.scalanlp" % "breeze_2.10" % "0.7",
// native libraries are not included by default. add this if you want them (as of 0.7)
// native libraries greatly improve performance, but increase jar sizes.
"org.scalanlp" % "breeze-natives_2.10" % "0.7",
)
resolvers ++= Seq(
// other resolvers here
// if you want to use snapshot builds (currently 0.8-SNAPSHOT), use this.
"Sonatype Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots/",
"Sonatype Releases" at "https://oss.sonatype.org/content/repositories/releases/"
)
// Scala 2.9.2 is still supported for 0.2.1, but is dropped afterwards.
// Don't use an earlier version of 2.10, you will probably get weird compiler crashes.
scalaVersion := "2.10.3"
Put your source into the appropriate folder (by default, src/main/scala) and run sbt console. This command will download the dependencies, compile your code, and launch the Scala interpreter. At this point you should be able to interact with your class.