How do I create a tarball and a zip for a single module using distinct configurations? - scala

I have a multi-project build with a particularly messy module which contain several mainClasses. I would like to create several distribution packages for this messy module, each distribution package employing distinct file sets and employing different formats. Ideas?

This is the answer from the sbt-nativer-packager issue tracker where the same question was posted.
I'm adding this from the gitter chat as well:
I'm just arriving in this chat room and my knowledge of sbt-native-packager is virtually zero... but anyway... looks to me that JavaAppPackaging and other archetypes should actually be configurations extended from Universal. In this scenario, I would just create my own configuration extended from JavaAppPackaging and tweak the necessary bits according to my needs. And, finally, if the plugin just picks mappings in ThisScope... it would pick my own scope, and not JavaAppPackaging... and not Universal.
So, let's go through this one by one.
The sbt-native-packager plugin always pick mappings in Universal. This is not ideal. It should conceptually pick mappings in ThisScope
SBT native packager provides two categories of AutoPlugins: FormatPlugins and ArchetypePlugins. FormatPlugins provide a new package format, e.g. UniversalPlugin (zip, tarball) or DebianPlugins (.deb). These plugins form a a hierarchy as they are build on top of each other:
SbtNativePackager
+
|
|
+-------+ Universal +--------+
| |
| + |
| | |
+ + +
Docker +-+ Linux +-+ Windows
| |
| |
+ +
Debian RPM
mappings, which define a file -> targetpath relation, are inherited with this pattern
mappings in ParentFormatPluginScope := (mappings in FormatPluginScope).value
So for docker it looks like this
mappings in Docker := (mappings in Universal).value
The linux format plugins use specialized mappings to preserve file permissions, but are basically the same.
Since sbt-native-packager plugin always pick mappings in Universal, I have to redefine mappings in Universal in each of my configurations
Yes. If you want to define your own scope and inherit the mappings and change them you have to do this, like all other packaging plugins, too. I recommend putting this code into custom AutoPlugins in your project folder.
For example (not tested, imports may be missing )
import sbt._
object BuilderRSPlugin extends AutoPlugin {
def requires = JavaAppPackaging
object autoImport {
val BuilderRS = config("builderrs") extend Universal
}
import autoImport._
override lazy val projectSettings = Seq(
mappings in BuilderRS := (mappings in Universal).value
)
}
looks to me that JavaAppPackaging and other archetypes should actually be configurations extended from Universal
JavaAppPackaging is an archetype, which means this plugin doesn't bring any new packaging formats, thus no new scopes. It configures all the packaging formats it can and enables them.
You package stuff by specifying the scope:
universal:packageBin
debian:packageBin
windows:packageBin
So if you need to customize your output format you are doing this in the respecting scope.
mappings in Docker := (mappings in Docker).value.filter( /* what ever you want to filter */)

See: https://github.com/sbt/sbt-native-packager/issues/746
IMPORTANT: This is an "answer in progress". IT DOES NOT WORK YET!
This is an example of how one could achieve this.
The basic idea is that we add configurations for different packages to be generated. Each configuration tells which files will be present in the package. This does not work as expected. See my comments after the code.
lazy val BuilderRS = sbt.config("BuilderRS").extend(Compile,Universal)
lazy val BuilderRV = sbt.config("BuilderRV").extend(Compile,Universal)
addCommandAlias("buildRS", "MessyModule/BuilderRS:packageZipTarball")
addCommandAlias("buildRV", "MessyModule/BuilderRV:packageBin") // ideally should be named packageZip
lazy val Star5FunctionalTestSupport =
project
.in(file("MessyModule"))
.enablePlugins(JavaAppPackaging)
.settings((buildSettings): _*)
.configs(Universal,BuilderRS,BuilderRV)
.settings(inConfig(BuilderRS)(
Defaults.configSettings ++ JavaAppPackaging.projectSettings ++
Seq(
executableScriptName := "rs",
mappings in Universal :=
(mappings in Universal).value
.filter {
case (file, name) => ! file.getAbsolutePath.endsWith("/bin/rv")
},
topLevelDirectory in Universal :=
Some(
"ftreports-" +
new java.text.SimpleDateFormat("yyyyMMdd_HHmmss")
.format(new java.util.Date())),
mainClass in ThisScope := Option(mainClassRS))): _*)
//TODO: SEE COMMENTS BELOW ===============================================
// .settings(inConfig(BuilderRV)(
// Defaults.configSettings ++ JavaAppPackaging.projectSettings ++
// Seq(
// packageBin <<= packageBin in Universal,
// executableScriptName := "rv",
// mappings in ThisScope :=
// (mappings in Universal).value
// .filter {
// case (file, name) => ! file.getAbsolutePath.endsWith("/bin/rs")
// },
// topLevelDirectory in Universal :=
// Some(
// "ftviewer-" +
// new java.text.SimpleDateFormat("yyyyMMdd_HHmmss")
// .format(new java.util.Date())),
// mainClass in ThisScope := Option(mainClassRV))): _*)
Now observe configuration BuilderRV which in comments.
It is basically the same thing as configuration BuilderRS, except that we are now deploying a different shell script in the bin folder. There some other small differences, but not relevant to this argumentation. There are two problems:
The sbt-native-packager plugin always pick mappings in Universal. This is not ideal. It should conceptually pick mappings in ThisScope.
Since sbt-native-packager plugin always pick mappings in Universal, I have to redefine mappings in Universal in each of my configurations. And this is a problem because mappings in Universal is defined as a function of itself in all configurations: the result is that we ended up chaining logic to mapppings in Universal each time we redefined it in each configuration. This causes trouble in this example in particular because the configuration BuilderRV (the second one) will perform not only its filter, but also the filter defined in BuilderRS (the first one), which is not what I want.

Related

Multiple SBT Configurations should be exclusive, but they all activate at the same time - why?

I have defined a minimal build.sbt with two custom profiles ‘dev’ and ‘staging’ (what SBT seems to call Configurations). However, when I run SBT with the Configuration that was defined first in the file (dev), both Configuration blocks are executed - and if both modify the same setting, the last one wins (staging).
This seems to break any notion of conditional activation, so what am I doing wrong with SBT?
For reference, I want to emulate the conditionally activated Profiles concept of Maven e.g. mvn test -P staging.
SBT version: 1.2.1
build.sbt:
name := "example-project"
scalaVersion := "2.12.6"
...
fork := true
// Environment-independent JVM property (always works)
javaOptions += "-Da=b"
// Environment-specific JVM property (doesn’t work)
lazy val Dev = config("dev") extend Test
lazy val Staging = config("staging") extend Test
val root = (project in file("."))
.configs(Dev, Staging)
.settings(inConfig(Dev)(Seq(javaOptions in Test += "-Dfoo=bar")))
.settings(inConfig(Staging)(Seq(javaOptions in Test += "-Dfoo=qux")))
Command:
# Bad
sbt test
=> foo=qux
a=b
# Bad
sbt clean dev:test
=> foo=qux
a=b
# Good
sbt clean staging:test
=> foo=qux
a=b
Notice that despite of the inConfig usage you're still setting javaOptions in Test, i.e. in the Test config. If you remove in Test, it works as expected:
...
.settings(inConfig(Dev)(javaOptions += "-Dfoo=bar"))
.settings(inConfig(Staging)(javaOptions += "-Dfoo=qux"))
(also Seq(...) wrapping is unnecessary)
Now in sbt:
> show Test/javaOptions
[info] *
> show Dev/javaOptions
[info] * -Dfoo=bar
> show Staging/javaOptions
[info] * -Dfoo=qux
You can achieve the same result by scoping each setting explicitly (without inConfig wrapping):
.settings(
Dev/javaOptions += "-Dfoo=bar",
Staging/javaOptions += "-Dfoo=qux",
...
)
(here Conf/javaOptions is the same as javaOptions in Conf)

run sbt command for subprojects / set every command from Plugin?

I found the an SBT-recipe for parameters and Build Environment.
I would now like to be able to change buildEnv while running SBT. Basically
I can't manage to find a programmatic solution for:
> set every buildEnv := BuildEnvPlugin.autoImport.BuildEnv.Development or running BuiltinCommands.set from a wrapping command.
My basic solution doesn't scale to sub-/aggregated projects
val devCmd = Command.command("dev"){ state =>
Project extract state appendWithSession (Seq(buildEnv := BuildEnv.Development), state)
}
How can I change all aggregated settings as well?
I just didn't find this simple solution initially:
override def projectSettings: Seq[Setting[_]] = commands += devCmd
lazy val devCmd = BasicCommands
.newAlias("dev", "set every buildEnv := BuildEnvPlugin.autoImport.BuildEnv.Development")

Copy specific files in SBT

How would you create an sbt task to copy specific files (or file patterns) to a specific directory?
(For example, to copy client/target/foobar.js to dist/js and server/target/web/public/*.* to dist/assets).
I do not want to write a plugin to achieve this, but using an existing plugin is OK.
If you really want it to be a separate task, you can start with the copyResourcesTask in Defaults.scala:
def copyResourcesTask =
(classDirectory, resources, resourceDirectories, streams) map { (target, resrcs, dirs, s) =>
val cacheFile = s.cacheDirectory / "copy-resources"
val mappings = (resrcs --- dirs) pair (rebase(dirs, target) | flat(target))
s.log.debug("Copy resource mappings: " + mappings.mkString("\n\t", "\n\t", ""))
Sync(cacheFile)(mappings)
mappings
}
and modify it (just change mappings to the ones you want, and the tasks you depend on). Alternately, modify mappings in an existing task. See Mapping Files in documentation.

ADD/COPY files with sbt-native-packager's docker support

I'm using sbt-native-packager 1.0.0-M5 to create my docker image. I need to add a file that's not a source file or in the resource folder. My docker commands are as follows:
dockerCommands := Seq(
Cmd("FROM", "myrepo/myImage:1.0.0"),
Cmd("COPY", "test.txt keys/"), // <-- The failing part
Cmd("WORKDIR", "/opt/docker"),
Cmd("RUN", "[\"chown\", \"-R\", \"daemon\", \".\"]"),
Cmd("USER", "daemon"),
ExecCmd("CMD", "echo", "Hello, World from Docker")
)
It fails with: msg="test.txt: no such file or directory"
So after digging around a bit it seems I need to have test.txt in target/docker/stage. Then it works. But how do I get it there automatically? The file is actually in the root folder of the project.
I managed to get it to work by adding the file to mappings in Universal. So for you, you would need something like this:
mappings in Universal += file("test.txt") -> "keys/test.txt"
You won't need the COPY command if you do this, by the way.
Now, I'm not sure if this is going to add this mapping to other sbt-native-packager plugins. I hope a commenter can tell me whether or not this is true, but my intuition is that it will do so, which might be a dealbreaker for you. But any workaround is better than none, right? If you use Build.scala you could maybe use a VM argument to tell sbt whether or not to add this mapping...
You may place all additional files (which must be included in container image) into folder src/universal. Content of that folder will be automatically copied in /opt/app folder within your container image. You don't need any additional configuration. See "Getting started with Universal Packaging" for additional info.
The files located in /src/universal will be available in the runtime directory for the Scala app in the Docker container. This means that if your app has /src/universal/example.txt, then it can be accessed with scala.io.Source.fromFile("./example.txt")
I was able to get this working using dockerPackageMappings:
dockerPackageMappings in Docker += (baseDirectory.value / "docker" / "ssh_config") -> "ssh_config"
dockerCommands := (dockerCommands.value match {
case Seq(from#Cmd("FROM", _), rest#_*) =>
Seq(
from,
Cmd("Add", "ssh_config", "/sbin/.ssh/config")
) ++ rest
})
For sbt-docker plugin, not sbt-native-packager
I was able to add files this way:
For example, to add a file located in src/main/resources/docker/some-file.ext
dockerfile in docker := {
val targetPath = "/usr/app"
// map of (relativeName -> File) of all files in resources/docker dir, for convenience
val dockerFiles = {
val resources = (unmanagedResources in Runtime).value
val dockerFilesDir = resources.find(_.getPath.endsWith("/docker")).get
resources.filter(_.getPath.contains("/docker/")).map(r => dockerFilesDir.toURI.relativize(r.toURI).getPath -> r).toMap
}
new Dockerfile {
from(s"$namespace/$baseImageName:$baseImageTag")
...
add(dockerFiles("some-file.ext"), s"$targetPath/some-file.ext")
...
}
}
You can add an entire directory to a Docker image's file system by first making it available using dockerPackageMappings, and then COPYing it as an additional Docker command.
import NativePackagerHelper._
dockerPackageMappings in Docker ++= directory(baseDirectory.value / ".." / "frontend" )
dockerCommands ++= Seq(
Cmd("COPY", "frontend /opt/frontend"),
)

Combine several sbt tasks into one

I'm a little confused on the Scala/SBT documentation for creating Scala tasks. Currently I can run the following from the command line:
sbt ";set target := file(\"$PWD/package/deb-upstart\"); set serverLoading in Debian := com.typesafe.sbt.packager.archetypes.ServerLoader.Upstart; debian:packageBin; set target := file(\"$PWD/package/deb-systemv\"); set serverLoading in Debian := com.typesafe.sbt.packager.archtypes.ServerLoader.SystemV; debian:packageBin; set target := file(\"$PWD/package/rpm-systemd\"); rpm:packageBin"
This resets my target each time to a different directory (deb-upstart, deb-systemv and rpm-systemd) and runs an sbt-native-package task for each of those settings. (Yes, I realizing I'm compiling it three different times; but sbt-native-packager doesn't seems to have a setting for the artifact directory)
This works fine from a bash prompt, but I've been trying to put the same target into jenkins (replacing $PWD with $WORKSPACE) and I can't seem to get the quote escaping correct. I thought it might be easier just to have a task in either by build.sbt or project/Build.scala that runs all three of those tasks, changing out the target variable each time (and replacing $PWD or $TARGET with the full path of the base directory).
I've attempted the following:
lazy val packageAll = taskKey[Unit]("Creates deb-upstart, deb-systenv and rpm-systemd packages")
packageAll := {
target := baseDirectory.value / "package" / "deb-upstart"
serverLoading in Debian := com.typesafe.sbt.packager.archetypes.ServerLoader.Upstart
(packageBin in Debian).value
target := baseDirectory.value / "package" / "deb-systemv"
serverLoading in Debian := com.typesafe.sbt.packager.archetypes.ServerLoader.SystemV
(packageBin in Debian).value
target := baseDirectory.value / "package" / "rpm-systemd"
(packageBin in Rpm).value
}
But the trouble is the .value causes the tasks to get evaluated before my task is even run, so they don't get the new target setting (as stated in this other question: How can I call another task from my SBT task?)
So, I figured this out for you :)
As you already mentioned, combining a multiple tasks in a single one where some of the tasks depend on the same setting, doesn't work out as expected.
Instead we do the following
Create a task for each of our custom steps, e.g. packaging debian for upstart
Define an alias that executes these commands in order
Define tasks
lazy val packageDebianUpstart = taskKey[File]("creates deb-upstart package")
lazy val packageDebianSystemV = taskKey[File]("creates deb-systenv package")
lazy val packageRpmSystemD = taskKey[File]("creates rpm-systenv package")
Example task implementation
The implementation is pretty simple and the same for each of
the tasks.
// don't forget this
import com.typesafe.sbt.packager.archetypes._
packageDebianUpstart := {
// define where you want to put your stuff
val output = baseDirectory.value / "package" / "deb-upstart"
// run task
val debianFile = (packageBin in Debian).value
// place it where you want
IO.move(debianFile, output)
output
}
Define alias
And now compose these tasks into a single alias with
addCommandAlias("packageAll", "; clean " +
"; set serverLoading in Debian := com.typesafe.sbt.packager.archetypes.ServerLoader.SystemV" +
"; packageDebianSystemV " +
"; clean " +
"; set serverLoading in Debian := com.typesafe.sbt.packager.archetypes.ServerLoader.Upstart" +
"; packageDebianUpstart " +
"; packageRpmSystemD")
You can look at the complete code here.
Update
Setting the SystemLoader inside the alias seems to be the right
way to solve this. A clean is unfortunately necessary between
each build for the same output format.