How do I copy a zip dependency to the target directory in an SBT build? - scala

I am working on an SBT project that generates an RPM via the sat-native-packager. One of the items that I want to pull into the RPM is a ZIP file that was published from a separate project using the sat-pack plugin. This ZIP file contains a number of JAR files, along with multiple scripts for invoking them.
I have the following in my RPM project's build.sbt:
libraryDependencies += ("com.mycompany" %% "aputils" % "1.0.0-SNAPSHOT").artifacts(Artifact("aputils", "zip", "zip"))
// Task to download and unpack the utils bundle
lazy val unpackUtilsTask = taskKey[Unit]("Download the utils bundle to the target directory")
unpackUtilsTask := {
val log = streams.value.log
val report: UpdateReport = (update in Rpm).value
val filter = artifactFilter(extension = "zip")
val matches: Seq[File] = report.matching(filter)
matches.foreach{ f =>
log.info(s"Filter match: ${f}")
IO.copyFile(f, target.value)
}
}
When I run this task, it does not match any entries in the UpdateReport. Nothing is printed, and no files are copied to target/. If I modify the task to instead print all of the files in the UpdateReport:
report.allFiles.foreach(f => log.info(s"All files: $f))
I see a number of JAR files, but not my ZIP file. The JAR files turn out to be all of the JAR files that are contained in the ZIP file. I am not sure why the ZIP is being unpacked and its contents are being listed as dependencies like this. If I mark the dependency as notTransitive, then the contained JARs are not listed in the report, but the ZIP still isn't included either.
This project is using SBT 0.13.15. I would prefer not to update it to 1.x at this time, but will do so if I must.
I will need to unzip the ZIP file under target/ eventually so I can define one or more packageMapping entries to pull the files into the RPM, but that seems easy enough to do via sbt.IO, if I can first just get a reference to the original ZIP file that is pulled down from our Artifactory server.

This didn't get any responses after a couple days, but I'll post the answer that I was able to come up with after more trial and error.
I was on the right track by examining the UpdateReport, but I wasn't looking at the right data within it. I needed to drill down to find a ModuleReport, which would show me where the .zip file was being downloaded to on the build machine. Once I have that path, it is trivial to unpack it to target/ using IO.unzip(). Here is how my task ended up looking:
libraryDependencies += ("com.mycompany" %% "aputils" % "1.0.0-SNAPSHOT").artifacts(Artifact("aputils", "zip", "zip"))
// Task to unzip the utils ZIP file to the target directory so we can define a package mapping
lazy val unpackUtilsTask = taskKey[Unit]("Download the utils bundle to the target directory")
unpackUtilsTask := {
val log = streams.value.log
val cReport: ConfigurationReport = (update in Compile).value.configuration("compile").get
cReport.modules.foreach{ mReport =>
if (mReport.module.name.startsWith("aputils")) {
mReport.artifacts.foreach{ case (art, f) =>
log.info(s"Unpacking aputils bundle: ${f.getAbsolutePath}")
IO.unzip(f, target.value)
}
}
}
}
packageBin in Rpm := ((packageBin in Rpm).dependsOn(unpackUtilsTask)).value
The last line attaches the task to the task that builds the RPM, so it will be unzipped before the RPM is built, and we can define packageMappings to put the contents of the .zip file into the generated RPM.

Related

Prevent looping recompilation in SBT

I am using SBT to build my scala project. AFter the compilation of a submodule which sues fastOptJS, I need to push the compiled files to another module within the same project, I designed a custom command fastOptCopy to do so.
lazy val copyjs = TaskKey[Unit]("copyjs", "Copy javascript files to public directory")
copyjs := {
val outDir = baseDirectory.value / "public/js"
val inDir = baseDirectory.value / "js/target/scala-2.11"
val files = Seq("js-fastopt.js", "js-fastopt.js.map", "js-jsdeps.js") map { p => (inDir / p, outDir / p) }
IO.copy(files, true)
}
addCommandAlias("fastOptCopy", ";fastOptJS;copyjs")
However, when I enter into the sbt console and type
~fastOptCopy
it keeps compiling, copying, compiling, copying, ... in an infinite loop. I guess that because I am copying the files, it thinks that the sources have changed and retriggers compilation.
How can I prevent this?
You can exclude specified files from watchSources in sbt configuration
http://www.scala-sbt.org/0.13/docs/Triggered-Execution.html
watchSources defines the files for a single project that are monitored
for changes. By default, a project watches resources and Scala and
Java sources.
Here is a similar question:
How to not watch a file for changes in Play Framework
watchSources := watchSources.value.filter { _.getName != "BuildInfo.scala" }

download a zip from url and extract it in resource using SBT

I want to download a zip file (my database) from a URL and extract it in specific folder (e.g. resource). I want to do it in my project build sbt file.
What would be the appropriate way to do that?
I know that sbt.IO has unzip and download. I couldn't find a good example that uses download (those I found were not working).
Is there any sbt plugin to do this for me?
It's not clear when you want to download and extract, so I'm going to do it with a TaskKey. This will create a task you can run from the sbt console called downloadFromZip, which will just download the sbt zip and extract it to a temp folder:
lazy val downloadFromZip = taskKey[Unit]("Download the sbt zip and extract it to ./temp")
downloadFromZip := {
IO.unzipURL(new URL("https://dl.bintray.com/sbt/native-packages/sbt/0.13.7/sbt-0.13.7.zip"), new File("temp"))
}
This task can be modified to only run once if the path already exists:
downloadFromZip := {
if(java.nio.file.Files.notExists(new File("temp").toPath())) {
println("Path does not exist, downloading...")
IO.unzipURL(new URL("https://dl.bintray.com/sbt/native-packages/sbt/0.13.7/sbt-0.13.7.zip"), new File("temp"))
} else {
println("Path exists, no need to download.")
}
}
And to have it run on compilation, add this line to build.sbt (or project settings in Build.scala).
compile in Compile <<= (compile in Compile).dependsOn(downloadFromZip)

Include generated resources in a jar (SBT)

I've been writing an SBT plugin that generates resources into resource_managed. I'm now looking to include these generated resources in the generated jar as the SBT docs detail:
Generating resources:
By default, generated resources are not included in the packaged source artifact. To do so, add them as you would other mappings. See Adding files to a package
I've read the docs but honestly how to do this I can't figure out. Can anyone explain it or point me to another project that does this so I can see how they do it?
First just to clarify, they are included in jars containing compiled classes. They are not included in jars containing sources.
By default, generated resources are not included in the packaged
source artifact.
For packageBin the generated files should already be included - just make sure you return all generated files from the generator method. Assuming you want to package them in the sources artifact, this is what you have to do.
Let's assume you have a generator that generates a property file.
lazy val generatePropertiesTask = Def.task {
val file = (Compile / resourceManaged).value / "stack-overflow" / "res.properties"
val contents = s"name=${name.value}\nversion=${version.value}"
IO.write(file, contents)
Seq(file)
}
resourceGenerators in Compile += generatePropertiesTask.taskValue
To include that in the generated sources you have to tell sbt to where the res.properties must be copied in the generated sources artefact. The task, which generates the packaged sources is called packageSrc, therefore you have to set mappings scoped to that task.
mappings in (Compile, packageSrc) += {
((resourceManaged in Compile).value / "stack-overflow" / "res.properties") -> "path/in/jar/res.properties"
}
Because your generator can generate many tasks, and mapping each by hand would be a tedious task, sbt gives you an utility to map multiple paths at once.
mappings in (Compile, packageSrc) ++= {
val allGeneratedFiles = ((resourceManaged in Compile).value ** "*") filter { _.isFile }
allGeneratedFiles.get pair relativeTo((resourceManaged in Compile).value)
}
The first line finds all generated files using path finders and second line maps them to their path in the target jar.

How can I download and add static files to a project using SBT

I want to add some files to test a library that I am writing.
The tests are available in a compressed file in a URI and I just want to download that file and uncompress its contents to a folder before testing.
I was reading the documentation on SBT and there is a Generate sources/resources task.
Also, it seems easy to uncompress a zip file in Scala (see this question).
So I think, I could glue those 2 pieces together, but I wonder if there is some simpler solution.
How about this (syntax for Sbt 0.13.2), in your build.sbt:
resourceGenerators in Test += Def.task {
val location = url("http://path/to/your/zip-file.zip")
IO.unzipURL(location, resourceManaged.value / "my-custom-files").toSeq
}.taskValue

Sbt Package Command Do Not Copy Resources

I am using sbt for a simple, small GUI projects that load icons from src/main/scala/resources. At first, everything works fine and I can compile. package, and run. The generated jar and class files all have the resource folder in it. Then I do the clean command. I re-run the compile and package, and suddently the application crashes. I check the generated jars and classes, and found out that the resources folder are not copied this time.
Running the application now gives me the NullPointerException pointing to the line where I load the resource (icon).
I didn't change the sbt build files or anything in the project. Just run clean and re-run compile and package. I don't know where to start looking for the problem. Where should I start looking? What am I doing wrong?
EDIT (the minimal example)
The project is a standard Scala template from typesafe's g8 (https://github.com/typesafehub/scala-sbt.g8). Here's my Build.Scala:
import sbt._
import sbt.Keys._
object ObdscanScalaBuild extends Build {
val scalaVer = "2.9.2"
lazy val obdscanScala = Project(
id = "obdscan-scala",
base = file("."),
settings = Project.defaultSettings ++ Seq(
name := "project name",
organization := "thesis.bert",
version := "0.1-SNAPSHOT",
scalaVersion := scalaVer,
// add other settings here
// resolvers
// dependencies
libraryDependencies ++= Seq (
"org.scala-lang" % "scala-swing" % scalaVer,
"org.rxtx" % "rxtx" % "2.1.7"
)
)
)
}
It builds the code fine previously. Here's the project code directory structure:
It works fine and output this directory inside the jar at first:
And suddently, when I do a clean and compile command via the sbt console, it didn't copy the resource directory in the jar or in the class directory (inside target) anymore. I can't do anything to get the resource directory copied to target now, except by restoring previous version and compile it one more time. I restore the previous version via Windows' history backup.
Is it clear enough? Anything I need to add?
EDIT:
After moving the files to src/main/resources, the compiled files now contains the resources. But now, I can't run it in eclipse. Here's my code:
object ControlPanelContent {
val IconPath = "/icons/"
val DefaultIcon = getClass.getResource(getIconPath("icon"))
def getImage(name: String) = {
getClass.getResource(getIconPath(name))
}
def getIconPath(name: String) = {
IconPath + name + ".png"
}
}
case class ControlPanelContent(title: String, iconName: String) extends FlowPanel {
name = title
val icon: ImageIcon = createIcon(iconName, 64)
val pageTitle = new Label(title)
protected def createIcon(name: String, size: Int): ImageIcon = {
val path: Option[URL] = Option(ControlPanelContent.getImage(name))
val img: java.awt.Image = path match {
case Some(exists) => new ImageIcon(exists).getImage
case _ => new ImageIcon(ControlPanelContent.DefaultIcon).getImage
}
val resizedImg = img.getScaledInstance(size, size, Image.SCALE_SMOOTH)
new ImageIcon(resizedImg)
}
}
The TLDR version is this, I guess:
getClass.getResource("/icons/icon.png")
which works if I call from sbt console command. Here's the result when I call the code from sbt console:
scala> getClass.getResource("/icons/icon.png")
res0: java.net.URL = file:/project/path/target/scala-2.9.2/classes/icons/icon.png
which when runned gives the following exception:
Caused by: java.lang.NullPointerException
at javax.swing.ImageIcon.<init>(Unknown Source)
at thesis.bert.gui.ControlPanelContent.createIcon(ControlPanel.scala:54)
at thesis.bert.gui.ControlPanelContent.<init>(ControlPanel.scala:33)
at thesis.bert.gui.controls.DTC$.<init>(Diagnostics.scala:283)
at thesis.bert.gui.controls.DTC$.<clinit>(Diagnostics.scala)
... 60 more
EDIT 2: It works now. I just deleted the project from eclipse, re-run sbt eclipse and it magically works. Not sure why (maybe caching?).
The SBT convention for resources is to put them in src/main/resources/, not src/main/scala/resources/. Try moving your resources folder up one level. Its content should then be included, meaning that you will get icons and indicator folders inside the generated jar file (directly at the root level, not inside a resources folder).
If you put the resources in scala, I think it copies only the files that are compiled (i.e. .class files resulting from scala compilation).
If it doesn't solve your problem, can you post the lines of code you use to load the resource?