I'm creating a new project in sbt, and I'm having a hard time getting avro files to generate. I'm using avrohugger in my sbt plugins:
$ cat ~/.sbt/0.13/plugins/plugins.sbt
addSbtPlugin("org.ensime" % "sbt-ensime" % "2.0.1")
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "5.2.3")
addSbtPlugin("com.julianpeeters" % "sbt-avrohugger" % "1.1.0")
And, as recommended on the avrohugger github page, I'm defining the settings for the avro tasks in build.sbt:
sourceGenerators in Compile += (avroScalaGenerate in Compile).taskValue
(avroSpecificSourceDirectory in Compile) := new java.io.File("src/main/resources/avro")
(avroScalaSpecificCustomNamespace in Compile) := Map("" -> "avro")
(avroSpecificScalaSource in Compile) := new java.io.File("target/generated-sources/avro")
I have tried also using avroScalaGenerate and avroScalaGenerateScavro to no avail. I've tried running the specific task step (sbt avroScalaGenerateSpecific), which succeeds but has no visible output, even when run with show. clean and compile produces the expected output in target for the regular classes but not for anything from avro. So all together, I'm left with no error messages but no results.
After some digging, this was a few different issues.
First was that (avroSpecificSourceDirectory in Compile) := new java.io.File("src/main/resources/avro") is not the full namespace, and that actually made a difference. I was using my.cool.namespace because my folder was incorrect and also had the .'s in it. Remaking the folder structure to match the expected namespace, and changing this to slashes, helped it find the files it needed.
I was able to remove the namespace mapping, since my avro definitions didn't need it, and I didn't need the output folder because the default is fine.
Finally, I fixed which generator was being used in the compile step, and used the specific record generator throughout the build file.
So now, the relevant and completed build.sbt part looks like this:
//add avro generation to the compile step
sourceGenerators in Compile += (avroScalaGenerateSpecific in Compile).taskValue
//tell the avro compiler where to look for avro sources.
(avroSpecificSourceDirectory in Compile) := new java.io.File("src/main/resources/my/cool/namespace/avro")
This now generates avro when needed without any errors.
Related
I have created basic Scala Play application with https://www.playframework.com/getting-started play-scala-seed. This project compiles and runs with sbt run. But I have another Scala project that compiles and runs and which I have submitted to my local Ivy repository with command sbt publishLocal. This other project was saved at C:\Users\tomr\.ivy2\local\com.agiintelligence\scala-isabelle_2.13\master-SNAPSHOT as a result of this command.
Then I imported (exactly so - imported, no just opened) my Play project in IntelliJ and I used Project - Open Module Settings - Project Settings - Libraries to add com.agiintelligence jar from my ivy2 location. After such operations IntelliJ editor recognizes com.agiintelligence classes. That is fine.
But when I am trying to run my Play application with sbt run, I experience the error message not found: object com that is exactly when compiling import com.agiintelligence line in my Scala controller file of Play application.
Of course - such error has been reported and resolved with, e.g. object play not found in scala application
But that solution suggests to append build.sbt file. My build.sbt file is pretty bare:
name := """agiintelligence"""
organization := "com.agiintelligence"
version := "1.0-SNAPSHOT"
lazy val root = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.13.5"
libraryDependencies += guice
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "5.0.0" % Test
// Adds additional packages into Twirl
//TwirlKeys.templateImports += "com.skaraintelligence.controllers._"
// Adds additional packages into conf/routes
// play.sbt.routes.RoutesKeys.routesImport += "com.skaraintelligence.binders._"
My Play application contains (as can bee seen from the IntelliJ project pane) some tens of 'external libraries' (it shows my com.agiintelligence jar as well), but why should I add my own ivy2 library in build.sbt file if no other libraries are listed here? What is different with my library? It is on my computer, in the repository as expected already?
Of course, I can try to add it build.sbt and issue sbt update and see what happens, but I can not understand this logic? Can someone explain it and provide some clue to intelligible solution of my error message?
My Play application contains (as can bee seen from the IntelliJ project pane) some tens of 'external libraries'
Those are probably just transitive dependencies of your Play dependency, that is why sbt downloaded all of them and put them in your classpath so you could use them without you needing to tell it about them; because the pom of Play already did.
It is not that the build tool or the IDE magically added all those dependencies for you because they read your mind and magically understood you wanted them. And that for some reason the magic stopped working for your own library.
Why it is not sufficient to list it Project-Setting--External Libraries in IntelliJ only?
That is sufficient for the IDE to work, but not for the build tool. The build tool is independent of the IDE; it doesn't know about it. sbt just knows about the dependencies you configured in your definition file.
Even more, you should always configure your dependencies on your build tool and then import that in the IDE; rather than the opposite. IDEs are graphical tools, so their state can not be committed, can not be shared, can not keep track of changes, can not be used in CI / CD environments; additionally, different teammates may want to use different IDEs.
I resolved the error message by adding line in build.sbt file
libraryDependencies += "de.unruh" %% "scala-isabelle" % "master-SNAPSHOT"
and by subsequent run of sbt update.
Error is solved, but the main question still stand - why I had to do this? Why there are tens of dependencies that are not listed in build.sbt and why should I list my dependency in build.sbt and why it is not sufficient to list it Project-Setting--External Libraries in IntelliJ only?
OK, comment by #Luis_Miguel_Mejía_Suárez gave the explanation, that comment is the actual and expected answer to my question.
I am currently using ScalaTest 3.0.1 for my Scala 2.11.8 project, with sbt 0.13.18 as the build tool. The IDE is Intellij.
The project is a Spark project, and I'm using a temp directory at root/temp/ for checkpoints and warehousing. When I run the tests multiple times the checkpoints keep getting added to, eventually reaching a very large size. I want to keep the checkpoints around after the test runs so they can be used for verification, but I would like to delete them before the next run.
How can you accomplish this using sbt and ScalaTest in Intellij?
I've been looking into setting up an sbt clean with my temp directory in build.sbt but I can't seem to get it to delete the directory, and reading the documentary hints it will only delete the files sbt has created, so I abandoned that idea.
I'm currently looking into setting up my ScalaTest set up to have a BeforeAll event which will delete the directory, but I'm not sure this is the correct approach and I've been having issues getting it to work.
Based on https://stackoverflow.com/a/48659771/5205022 clean can include temp directory
cleanFiles += baseDirectory.value / "temp"
Another option is to create a custom task that cleans temp, for example using better-files
lazy val deleteTestTemp = taskKey[Unit]("Delete test temp directory")
deleteTestTemp := {
import better.files._
val temp = (baseDirectory.value / "temp").toScala
if (temp.exists) temp.delete()
}
where project/plugins.sbt contains
libraryDependencies += "com.github.pathikrit" %% "better-files" % "3.9.1"
we could clean and then test like so
deleteTestTemp;test
In our project, we have a enhance post-process to the .class files generated by compile. This enhance step actually modifies the generated .class files then overrides it.
enhance <<= enhance triggeredBy (compile in Compile)
The problem is that sbt has a mechanism called incremental recompilation. It monitors the generated .class file. Every time the enhancer overrides the generated .class file, sbt recognises these modifications and recompiles related sources in next compile command.
To us, a recompilation is a very time-consuming work. We want to stop sbt from recompiling modified .class file. That may mean making sbt only monitor source changes, not output changes.
I did some searching on this. But I found little things about this. Now I know a trait called Analysis is likely responsible for the mapping from sources to output .class files. So I ask help from you guys.
Ps: we may solve this problem by putting the output of enhance to another folder, but it is not preferred.
sbt strongly discourages mutations to files. You should generate different files instead. By doing so, you will solve your problem, since the incremental compiler of sbt will still be looking at the unaltered .class files. You will have some rewiring to do:
Send the outputs of the compile task somewhere else:
classDirectory in Compile := crossTarget.value / "classes-orig"
Processing these .class files with your tool, and send them to crossTarget.value / "classes" (the original classDirectory:
enhance <<= enhance triggeredBy (compile in Compile)
enhance := {
val fromDir := (classesDirectory in Compile).value
val toDir := crossTarget.value / "classes"
...
}
Rewire productDirectories to use crossTarget.value / "classes" anyway (otherwise it'll go looking in your modified classDirectory:
productDirectories in Compile := Seq(crossTarget.value / "classes")
Make sure that products depends on your enhance task:
products in Compile <<= (products in Compile) dependsOn enhance
You may need some more rewiring if you have resources (see copyResources). But basically you should be able to get there.
I said about that sbt monitors the output .class file. When a .class file is modified, it recompiles the .class file's source.
After some research, we found that sbt notices file's modification by its last modified time. That is to say, we can fool sbt by rolling back the last modified time after the modification, so that sbt won't notice any changes.
So, our solution is simple but effective:
find all .class files
note down their last modified time
do the enhance
put back the former last modified time
This is a small trick. We still expect more robust solutions.
Description:
A little like Chenyu, I had to write a plugin for SBT 1.x, that would enhance compiled classes and later I wanted to make sure that those enhanced classes were used for building the jar.
I did not want to hack this solution, so Chenyu's answer was not acceptable to me and sjrd's answer was very helpful but adjusted to SBT 0.13.
So here is my working solution, with little comments:
Code:
object autoImport {
val enhancedDest = settingKey[File]("Output dir for enhanced sources")
}
def enhanceTask: Def.Initialize[Task[Unit]] = Def.task {
val inputDir = (classDirectory in Compile).value
val outputDir = enhancedDest.value
enhance(inputDir, outputDir)
...
}
override def projectSettings: Seq[Def.Setting[_]] = Seq(
enhancedDest := crossTarget.value / "classes-enhanced",
products in Compile := Seq(enhancedDest.value), // mark enhanced folder to use for packaging
// https://www.scala-sbt.org/1.0/docs/Howto-Dynamic-Task.html#build.sbt+v2
compile in Compile := Def.taskDyn {
val c = (compile in Compile).value // compile 1st.
Def.task {
(copyResources in Compile).value // copy resources before enhance
enhanceTask.value // enhance
c
}
}.value
)
So I'm having trouble importing a package in scala. I downloaded the package, breeze, from github, because I wanted to sample from probability distributions.
I'm used to Python, where I can just download a package, include it in the path, and then import it in the code. So I'm very new to the idea of using a separate "build tool" to use 3rd party packages.
So I downloaded the "breeze" source code from github, installed sbt, and then within the source code for breeze, I ran sbt, and then I used the "assembly" command to get a .jar for breeze.
If I want to use the scala interpreter, I can import the package just fine with
scala -cp breeze-master/target/scala-2.11/breeze-parent-assembly-0.8.jar
The problem is that I want to use this package in a separate piece of code that I'm writing in a file called Chromosome.scala. And when I try to import the package (as seen below), I get an error:
error: not found: object breeze
Here's my code:
// Chromosome.scala
import breeze.stats.distributions._
class Chromosome(s:Int, bitstring:Array[Boolean]) {
val size:Int = s;
val dna:Array[Boolean] = bitstring;
var fitness:Int = 0;
def mutate(prob:Float):Unit = {
// This method will randomly "mutate" the dna sequence by flipping a bit.
// Any individual bit may be flipped with probability 'pm', usually small.
val pm:Float = prob;
// The variable bern is an instance of a Bernoulli random variable,
// whose probability parameter is equal to 'pm'.
var bern = new Bernoulli(pm);
//Loop through the 'dna' array and flip each bit with probability pm.
for (i <- 0 to (size - 1)) {
var flip = bern.draw();
if (flip) {
dna(i) = !(dna(i));
}
}
}
“A script?” What is this and what is its connection to your SBT project? Scala scripts include their own launch command for the Scala interpreter / compiler ( / REPL…). If you want to access things beyond the standard library, you'll have to inclulde them there. Alternately, you can use the SBT Start Script plug-in to produce a launcher script that will include the project dependencies. It will only work locally, though you can write some text processing and other shell scripting to produce a portable launch bundle.
It looks like there's some understandable confusion about what sbt is supposed to do for you.
First off, you generally don't need to download a package from github and build it from source. In the rare cases that you do (such as when you require features that have not made it into a release of the library), sbt can handle the grunt work.
Instead, you tell sbt a little about the project you're building (including what its dependencies are), and sbt will download them, compile your code, and set up the runtime classpath for the scala interpreter (amongst myriad other build-related tasks).
Just follow the directions on the breeze wiki. Specifically, create a build.sbt file in your project's root folder and copy this into it:
libraryDependencies ++= Seq(
// other dependencies here
"org.scalanlp" % "breeze_2.10" % "0.7",
// native libraries are not included by default. add this if you want them (as of 0.7)
// native libraries greatly improve performance, but increase jar sizes.
"org.scalanlp" % "breeze-natives_2.10" % "0.7",
)
resolvers ++= Seq(
// other resolvers here
// if you want to use snapshot builds (currently 0.8-SNAPSHOT), use this.
"Sonatype Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots/",
"Sonatype Releases" at "https://oss.sonatype.org/content/repositories/releases/"
)
// Scala 2.9.2 is still supported for 0.2.1, but is dropped afterwards.
// Don't use an earlier version of 2.10, you will probably get weird compiler crashes.
scalaVersion := "2.10.3"
Put your source into the appropriate folder (by default, src/main/scala) and run sbt console. This command will download the dependencies, compile your code, and launch the Scala interpreter. At this point you should be able to interact with your class.
I'm doing some refactoring that made compiler temporally give errors in several files. I'd like to work with them one by one (starting with common dependencies) and need some tool to check if modification is correct.
sbt compile is inconvenient because it gives too many errors and spends much time for compiling things that have no good.
I'm searching for a way to compile single file with sbt or a method for extracting sbt side libraries definition to pass them to a normal scalac compiler
There was a similar topic: How to compile just some files with sbt? that turned out to be source code error discussion rather that sbt functionality disclosure.
You could add the following line to build.sbt:
sources in Compile <<= (sources in Compile).map(_ filter(_.name == "Particular.scala"))
Then fix Particular.scala, then edit build.sbt and put the name of the next source file. If you keep the sbt console open, reload will re-read the .sbt file after you modify it.
I just wanted to mention here that I came across sbt-compile-quick-plugin (https://github.com/etsy/sbt-compile-quick-plugin). It does what it says on the tin, just add addSbtPlugin("com.etsy" % "sbt-compile-quick-plugin" % "1.3.0") to your project/plugins.sbt, then you can just start up sbt and run compileQuick /path/to/your/file