I want to add some files to test a library that I am writing.
The tests are available in a compressed file in a URI and I just want to download that file and uncompress its contents to a folder before testing.
I was reading the documentation on SBT and there is a Generate sources/resources task.
Also, it seems easy to uncompress a zip file in Scala (see this question).
So I think, I could glue those 2 pieces together, but I wonder if there is some simpler solution.
How about this (syntax for Sbt 0.13.2), in your build.sbt:
resourceGenerators in Test += Def.task {
val location = url("http://path/to/your/zip-file.zip")
IO.unzipURL(location, resourceManaged.value / "my-custom-files").toSeq
}.taskValue
Related
I am using scalapb in a project that needs to have access to the FileDescriptorSet. Is there a way to have scalapb generate the .desc file in addition to all other class files? Or is there some programatic way of obtaining a FileDescriptorSet from what is already generated?
Yes, to both questions.
If you are using sbt-protoc, you can have the following definition in your SBT file:
PB.protocOptions in Compile := Seq(
"--descriptor_set_out=" +
(baseDirectory in Compile).value.getParentFile / "src" / "main" / "resources" /"out.desc"
)
One caveat is that you would have to create src/main/resources yourself, otherwise you would get an error. It would probably be better to generate into resourceManaged (that would also require creating a directory ahead of time, since protoc doesn't do that)
You can also build a FileDescriptorSet at run time. For each proto file, ScalaPB generates a Scala object with scalaDescriptor (and also javaDescriptor if that's more convenient). The descriptors contains a list of their dependencies which are also FileDesciptors - you can traverse that tree structure and build a list of FileDescriptors which is essentially a FileDescriptorSet.
I am working on an SBT project that generates an RPM via the sat-native-packager. One of the items that I want to pull into the RPM is a ZIP file that was published from a separate project using the sat-pack plugin. This ZIP file contains a number of JAR files, along with multiple scripts for invoking them.
I have the following in my RPM project's build.sbt:
libraryDependencies += ("com.mycompany" %% "aputils" % "1.0.0-SNAPSHOT").artifacts(Artifact("aputils", "zip", "zip"))
// Task to download and unpack the utils bundle
lazy val unpackUtilsTask = taskKey[Unit]("Download the utils bundle to the target directory")
unpackUtilsTask := {
val log = streams.value.log
val report: UpdateReport = (update in Rpm).value
val filter = artifactFilter(extension = "zip")
val matches: Seq[File] = report.matching(filter)
matches.foreach{ f =>
log.info(s"Filter match: ${f}")
IO.copyFile(f, target.value)
}
}
When I run this task, it does not match any entries in the UpdateReport. Nothing is printed, and no files are copied to target/. If I modify the task to instead print all of the files in the UpdateReport:
report.allFiles.foreach(f => log.info(s"All files: $f))
I see a number of JAR files, but not my ZIP file. The JAR files turn out to be all of the JAR files that are contained in the ZIP file. I am not sure why the ZIP is being unpacked and its contents are being listed as dependencies like this. If I mark the dependency as notTransitive, then the contained JARs are not listed in the report, but the ZIP still isn't included either.
This project is using SBT 0.13.15. I would prefer not to update it to 1.x at this time, but will do so if I must.
I will need to unzip the ZIP file under target/ eventually so I can define one or more packageMapping entries to pull the files into the RPM, but that seems easy enough to do via sbt.IO, if I can first just get a reference to the original ZIP file that is pulled down from our Artifactory server.
This didn't get any responses after a couple days, but I'll post the answer that I was able to come up with after more trial and error.
I was on the right track by examining the UpdateReport, but I wasn't looking at the right data within it. I needed to drill down to find a ModuleReport, which would show me where the .zip file was being downloaded to on the build machine. Once I have that path, it is trivial to unpack it to target/ using IO.unzip(). Here is how my task ended up looking:
libraryDependencies += ("com.mycompany" %% "aputils" % "1.0.0-SNAPSHOT").artifacts(Artifact("aputils", "zip", "zip"))
// Task to unzip the utils ZIP file to the target directory so we can define a package mapping
lazy val unpackUtilsTask = taskKey[Unit]("Download the utils bundle to the target directory")
unpackUtilsTask := {
val log = streams.value.log
val cReport: ConfigurationReport = (update in Compile).value.configuration("compile").get
cReport.modules.foreach{ mReport =>
if (mReport.module.name.startsWith("aputils")) {
mReport.artifacts.foreach{ case (art, f) =>
log.info(s"Unpacking aputils bundle: ${f.getAbsolutePath}")
IO.unzip(f, target.value)
}
}
}
}
packageBin in Rpm := ((packageBin in Rpm).dependsOn(unpackUtilsTask)).value
The last line attaches the task to the task that builds the RPM, so it will be unzipped before the RPM is built, and we can define packageMappings to put the contents of the .zip file into the generated RPM.
I'm working on a Scala.js cross project where the jvm folder represents my server application and jsrepresents my scala.js code.
Whenever i compile my scala.js code via sbt crossJS/fastOptJS the compiled JS ends up in ./js/target/scala-2.11/web-fastopt.js.
I need to have this compiled JS file accessible in the resources of the server project in the jvm folder, so i can server it through my web application. I think i have to do something with artifactPath but i can't seem to get any results from my experiments thus far.
You can simply set the artifactPath of the fastOptJS task (or the fullOptJS task) to the (managed) resources directory of your JVM project:
// In the JS project's settings
artifactPath in fastOptJS in Compile :=
(resourceManaged in jvm in Compile).value /
((moduleName in fastOptJS).value + "-fastopt.js"))
This will put it in the directory, if the you run the fastOptJS task. However, it will not be included in sbt's resources task and it will not automatically be triggered, if you launch your server. Therefore:
// In the JVM project's settings
resources in Compile += (fastOptJS in js).value.data
A couple of notes:
The first step is only necessary, if your web-server does only serve specific directories. Otherwise the second one is enough, as this adds the file to the resources already; where it lies is secondary.
Setting the crossTarget, as in #ochrons' answer will also output all the .class and .sjsir files in the resource directory.
Have a look at Vincent Munier's sbt-play-scalajs for out-of-the-box sbt-web / Scala.js integration (it follows a slightly different approach: It copies the file from the js project, rather than directly placing it in the JVM project. Useful if you have multiple JVM projects).
You can configure the Scala.js SBT plugin to output the JavaScript file in folder of your choosing. For example like this:
// configure a specific directory for scalajs output
val scalajsOutputDir = Def.settingKey[File]("directory for javascript files output by scalajs")
// make all JS builds use the output dir defined later
lazy val js2jvmSettings = Seq(fastOptJS, fullOptJS, packageJSDependencies) map { packageJSKey =>
crossTarget in(js, Compile, packageJSKey) := scalajsOutputDir.value
}
// instantiate the JVM project for SBT with some additional settings
lazy val jvm: Project = sharedProject.jvm.settings(js2jvmSettings: _*).settings(
// scala.js output is directed under "web/js" dir in the jvm project
scalajsOutputDir := (classDirectory in Compile).value / "web" / "js",
This will also store -jsdeps.js and .js.map files in the same folder, in case you want to use those in your web app.
For a more complete example, check out this tutorial which addresses many other issues of creating a more complex Scala.js application.
I want to download a zip file (my database) from a URL and extract it in specific folder (e.g. resource). I want to do it in my project build sbt file.
What would be the appropriate way to do that?
I know that sbt.IO has unzip and download. I couldn't find a good example that uses download (those I found were not working).
Is there any sbt plugin to do this for me?
It's not clear when you want to download and extract, so I'm going to do it with a TaskKey. This will create a task you can run from the sbt console called downloadFromZip, which will just download the sbt zip and extract it to a temp folder:
lazy val downloadFromZip = taskKey[Unit]("Download the sbt zip and extract it to ./temp")
downloadFromZip := {
IO.unzipURL(new URL("https://dl.bintray.com/sbt/native-packages/sbt/0.13.7/sbt-0.13.7.zip"), new File("temp"))
}
This task can be modified to only run once if the path already exists:
downloadFromZip := {
if(java.nio.file.Files.notExists(new File("temp").toPath())) {
println("Path does not exist, downloading...")
IO.unzipURL(new URL("https://dl.bintray.com/sbt/native-packages/sbt/0.13.7/sbt-0.13.7.zip"), new File("temp"))
} else {
println("Path exists, no need to download.")
}
}
And to have it run on compilation, add this line to build.sbt (or project settings in Build.scala).
compile in Compile <<= (compile in Compile).dependsOn(downloadFromZip)
I've been writing an SBT plugin that generates resources into resource_managed. I'm now looking to include these generated resources in the generated jar as the SBT docs detail:
Generating resources:
By default, generated resources are not included in the packaged source artifact. To do so, add them as you would other mappings. See Adding files to a package
I've read the docs but honestly how to do this I can't figure out. Can anyone explain it or point me to another project that does this so I can see how they do it?
First just to clarify, they are included in jars containing compiled classes. They are not included in jars containing sources.
By default, generated resources are not included in the packaged
source artifact.
For packageBin the generated files should already be included - just make sure you return all generated files from the generator method. Assuming you want to package them in the sources artifact, this is what you have to do.
Let's assume you have a generator that generates a property file.
lazy val generatePropertiesTask = Def.task {
val file = (Compile / resourceManaged).value / "stack-overflow" / "res.properties"
val contents = s"name=${name.value}\nversion=${version.value}"
IO.write(file, contents)
Seq(file)
}
resourceGenerators in Compile += generatePropertiesTask.taskValue
To include that in the generated sources you have to tell sbt to where the res.properties must be copied in the generated sources artefact. The task, which generates the packaged sources is called packageSrc, therefore you have to set mappings scoped to that task.
mappings in (Compile, packageSrc) += {
((resourceManaged in Compile).value / "stack-overflow" / "res.properties") -> "path/in/jar/res.properties"
}
Because your generator can generate many tasks, and mapping each by hand would be a tedious task, sbt gives you an utility to map multiple paths at once.
mappings in (Compile, packageSrc) ++= {
val allGeneratedFiles = ((resourceManaged in Compile).value ** "*") filter { _.isFile }
allGeneratedFiles.get pair relativeTo((resourceManaged in Compile).value)
}
The first line finds all generated files using path finders and second line maps them to their path in the target jar.