I want to download a zip file (my database) from a URL and extract it in specific folder (e.g. resource). I want to do it in my project build sbt file.
What would be the appropriate way to do that?
I know that sbt.IO has unzip and download. I couldn't find a good example that uses download (those I found were not working).
Is there any sbt plugin to do this for me?
It's not clear when you want to download and extract, so I'm going to do it with a TaskKey. This will create a task you can run from the sbt console called downloadFromZip, which will just download the sbt zip and extract it to a temp folder:
lazy val downloadFromZip = taskKey[Unit]("Download the sbt zip and extract it to ./temp")
downloadFromZip := {
IO.unzipURL(new URL("https://dl.bintray.com/sbt/native-packages/sbt/0.13.7/sbt-0.13.7.zip"), new File("temp"))
}
This task can be modified to only run once if the path already exists:
downloadFromZip := {
if(java.nio.file.Files.notExists(new File("temp").toPath())) {
println("Path does not exist, downloading...")
IO.unzipURL(new URL("https://dl.bintray.com/sbt/native-packages/sbt/0.13.7/sbt-0.13.7.zip"), new File("temp"))
} else {
println("Path exists, no need to download.")
}
}
And to have it run on compilation, add this line to build.sbt (or project settings in Build.scala).
compile in Compile <<= (compile in Compile).dependsOn(downloadFromZip)
Related
I am working on an SBT project that generates an RPM via the sat-native-packager. One of the items that I want to pull into the RPM is a ZIP file that was published from a separate project using the sat-pack plugin. This ZIP file contains a number of JAR files, along with multiple scripts for invoking them.
I have the following in my RPM project's build.sbt:
libraryDependencies += ("com.mycompany" %% "aputils" % "1.0.0-SNAPSHOT").artifacts(Artifact("aputils", "zip", "zip"))
// Task to download and unpack the utils bundle
lazy val unpackUtilsTask = taskKey[Unit]("Download the utils bundle to the target directory")
unpackUtilsTask := {
val log = streams.value.log
val report: UpdateReport = (update in Rpm).value
val filter = artifactFilter(extension = "zip")
val matches: Seq[File] = report.matching(filter)
matches.foreach{ f =>
log.info(s"Filter match: ${f}")
IO.copyFile(f, target.value)
}
}
When I run this task, it does not match any entries in the UpdateReport. Nothing is printed, and no files are copied to target/. If I modify the task to instead print all of the files in the UpdateReport:
report.allFiles.foreach(f => log.info(s"All files: $f))
I see a number of JAR files, but not my ZIP file. The JAR files turn out to be all of the JAR files that are contained in the ZIP file. I am not sure why the ZIP is being unpacked and its contents are being listed as dependencies like this. If I mark the dependency as notTransitive, then the contained JARs are not listed in the report, but the ZIP still isn't included either.
This project is using SBT 0.13.15. I would prefer not to update it to 1.x at this time, but will do so if I must.
I will need to unzip the ZIP file under target/ eventually so I can define one or more packageMapping entries to pull the files into the RPM, but that seems easy enough to do via sbt.IO, if I can first just get a reference to the original ZIP file that is pulled down from our Artifactory server.
This didn't get any responses after a couple days, but I'll post the answer that I was able to come up with after more trial and error.
I was on the right track by examining the UpdateReport, but I wasn't looking at the right data within it. I needed to drill down to find a ModuleReport, which would show me where the .zip file was being downloaded to on the build machine. Once I have that path, it is trivial to unpack it to target/ using IO.unzip(). Here is how my task ended up looking:
libraryDependencies += ("com.mycompany" %% "aputils" % "1.0.0-SNAPSHOT").artifacts(Artifact("aputils", "zip", "zip"))
// Task to unzip the utils ZIP file to the target directory so we can define a package mapping
lazy val unpackUtilsTask = taskKey[Unit]("Download the utils bundle to the target directory")
unpackUtilsTask := {
val log = streams.value.log
val cReport: ConfigurationReport = (update in Compile).value.configuration("compile").get
cReport.modules.foreach{ mReport =>
if (mReport.module.name.startsWith("aputils")) {
mReport.artifacts.foreach{ case (art, f) =>
log.info(s"Unpacking aputils bundle: ${f.getAbsolutePath}")
IO.unzip(f, target.value)
}
}
}
}
packageBin in Rpm := ((packageBin in Rpm).dependsOn(unpackUtilsTask)).value
The last line attaches the task to the task that builds the RPM, so it will be unzipped before the RPM is built, and we can define packageMappings to put the contents of the .zip file into the generated RPM.
I've been writing an SBT plugin that generates resources into resource_managed. I'm now looking to include these generated resources in the generated jar as the SBT docs detail:
Generating resources:
By default, generated resources are not included in the packaged source artifact. To do so, add them as you would other mappings. See Adding files to a package
I've read the docs but honestly how to do this I can't figure out. Can anyone explain it or point me to another project that does this so I can see how they do it?
First just to clarify, they are included in jars containing compiled classes. They are not included in jars containing sources.
By default, generated resources are not included in the packaged
source artifact.
For packageBin the generated files should already be included - just make sure you return all generated files from the generator method. Assuming you want to package them in the sources artifact, this is what you have to do.
Let's assume you have a generator that generates a property file.
lazy val generatePropertiesTask = Def.task {
val file = (Compile / resourceManaged).value / "stack-overflow" / "res.properties"
val contents = s"name=${name.value}\nversion=${version.value}"
IO.write(file, contents)
Seq(file)
}
resourceGenerators in Compile += generatePropertiesTask.taskValue
To include that in the generated sources you have to tell sbt to where the res.properties must be copied in the generated sources artefact. The task, which generates the packaged sources is called packageSrc, therefore you have to set mappings scoped to that task.
mappings in (Compile, packageSrc) += {
((resourceManaged in Compile).value / "stack-overflow" / "res.properties") -> "path/in/jar/res.properties"
}
Because your generator can generate many tasks, and mapping each by hand would be a tedious task, sbt gives you an utility to map multiple paths at once.
mappings in (Compile, packageSrc) ++= {
val allGeneratedFiles = ((resourceManaged in Compile).value ** "*") filter { _.isFile }
allGeneratedFiles.get pair relativeTo((resourceManaged in Compile).value)
}
The first line finds all generated files using path finders and second line maps them to their path in the target jar.
I want to add some files to test a library that I am writing.
The tests are available in a compressed file in a URI and I just want to download that file and uncompress its contents to a folder before testing.
I was reading the documentation on SBT and there is a Generate sources/resources task.
Also, it seems easy to uncompress a zip file in Scala (see this question).
So I think, I could glue those 2 pieces together, but I wonder if there is some simpler solution.
How about this (syntax for Sbt 0.13.2), in your build.sbt:
resourceGenerators in Test += Def.task {
val location = url("http://path/to/your/zip-file.zip")
IO.unzipURL(location, resourceManaged.value / "my-custom-files").toSeq
}.taskValue
Using Intellij IDEA with Scala plugin.
When doing a Build -> Rebuild Project I get the following make warnings:
Output path ProjectRootFolder\project\target\idea-test-classes intersects with a source root. Only files that were created by build will be cleaned.
Output path ProjectRootFolder\project\target\idea-classes intersects with a source root. Only files that were created by build will be cleaned.
The project was generated with SBT gen-idea plugin.
The two output paths mentioned in the warnings are set as output path and test output path for the build module of the project under Project Structure -> Modules -> ProjectName-build -> Paths -> Use module compile output path.
Looking at the Sources tab for both the ProjectName module and ProjectName-build modules I saw that there is no place where ProjectRootFolder\project\target was marked as Source.
It seems the warnings were caused by the fact that the project and . folders were marked as Sources in the ProjectName-build module.
Since the SBT build module is not needed when using IDEA to build the project, one solution would be to generate the IDEA project without that module:
sbt gen-idea no-sbt-build-module
More details here: https://github.com/mpeltonen/sbt-idea/issues/180
UPDATE
Removing the build module is actually problematic since the build.scala file will show a lot of warnings because the required libraries would be missing.
A solution would be to unmark . and project from being Sources of the build module, which is also troublesome since it would need to be done after each gen-idea.
A better solution would be to use sbt to build the project instead of make. To achieve that remove the make before launch step in the IDEA run configuration and add a SBT products step instead.
I get the same warning and it didn't cause any problems so far.
Judging by this code it seems that they simply delete only files generated by IDE, otherwise they would want to delete everything in target directory. They are playing safe by checking if there could be any source files there:
// check that output and source roots are not overlapping
final List<File> filesToDelete = new ArrayList<File>();
for (Map.Entry<File, Collection<BuildTarget<?>>> entry : rootsToDelete.entrySet()) {
context.checkCanceled();
boolean okToDelete = true;
final File outputRoot = entry.getKey();
if (JpsPathUtil.isUnder(allSourceRoots, outputRoot)) {
okToDelete = false;
}
else {
final Set<File> _outRoot = Collections.singleton(outputRoot);
for (File srcRoot : allSourceRoots) {
if (JpsPathUtil.isUnder(_outRoot, srcRoot)) {
okToDelete = false;
break;
}
}
}
if (okToDelete) {
// do not delete output root itself to avoid lots of unnecessary "roots_changed" events in IDEA
final File[] children = outputRoot.listFiles();
if (children != null) {
filesToDelete.addAll(Arrays.asList(children));
}
else if (outputRoot.isFile()) {
filesToDelete.add(outputRoot);
}
}
else {
context.processMessage(new CompilerMessage(
"", BuildMessage.Kind.WARNING, "Output path " + outputRoot.getPath() + " intersects with a source root. Only files that were created by build will be cleaned.")
);
// clean only those files we are aware of
for (BuildTarget<?> target : entry.getValue()) {
clearOutputFiles(context, target);
}
}
}
I did a simple thing: I put a jar file I've made myself into /lib directory of a Play application. However, it doesn't work:
def index = Action {
val a = new com.mypackage.Class123 // error! not found Class123
Ok(views.html.index("hello"))
}
It already said sbt, compile, gen-idea but everything is still the same.
By the way, the file /lib/mypackage.jar is 38Mb for some reason.
Try reload in sbt. This is needed to rescan the build file; it might also be needed to rescan the lib directory (I'm not sure).