SBT: Exclude class from Jar - scala

I am converting a legacy jar project to SBT and for strange reasons that are not easily solved, this project comes with "javax/servlet/Servlet.class" inside it. So I need to somehow exclude this class from the jar file generated by package-bin. How do I accomplish this ?. Preferably I would like to exclude using a wildcard (i.e. javax.*).
The SBT assembly plugin does look like it has features that will do this, but I am worried that relying on sbt assembly means that my jar project will not work in a muliti module project (i.e. if I include it as a dependency in a war file then the war projects needs to be told to run assembly on the dependent jar project rather than package-bin - but I may be mistaken here).

Each task declares the other tasks and settings that it uses. You can use inspect to determine these inputs as described on Inspecting Settings and in a recent tutorial-style blog post by John Cheng.
In this case, the relevant task used by packageBin is mappings. The mappings task collects the files to be included in the jar and maps them to the path in the jar. Some background is explained on Mapping Files, but the result is that mappings produces a value of type Seq[(File, String)]. Here, the File is the input file providing the content and the String is the path in the jar.
So, to modify the mappings for the packageBin task, filter out the paths from the default mappings that you don't want to include:
mappings in (Compile,packageBin) ~= { (ms: Seq[(File, String)]) =>
ms filter { case (file, toPath) =>
toPath != "javax/servlet/Servlet.class"
}
}
mappings in (Compile,packageBin) selects the mappings for the main package task (as opposed to test sources or the packageSrc task).
x ~= f means "set x to the result of applying function f to the previous value of x". (See More About Settings for details.)
The filter drops all pairs where the path corresponds to the Servlet class.

I came up with this solution, it defines a new compile task which depends on the previous compile task (thus effectively allowing me to hook in right after the source is compiled and before it's packaged)
def mySettings = {
// add functionality to the standard compile task
inConfig(Compile)(Seq(compile in Compile <<= (target,streams,compile in Compile) map{
(targetDirectory, taskStream, analysis) =>
import taskStream.log
// this runs after compile but before package-bin
recursiveListFiles(targetDirectory, ".*javax.*".r) foreach {
file =>
log.warn("deleting matched resource: " + file.getAbsolutePath())
IO.delete(file)
}
analysis
})) ++
Seq(name := "MyProject", version := "1.0", exportJars := true)
}
def recursiveListFiles(f: File, r: Regex): Array[File] = {
val these = f.listFiles
val good = these.filter(f => r.findFirstIn(f.getName).isDefined)
good ++ these.filter(_.isDirectory).flatMap(recursiveListFiles(_, r))
}
Its a little bit more complicated than what I had hoped but it allows me to do all sorts of modifications prior to packaging (in this case searching the target folder deleting all class files that matches a regular expression). Also it accomplished my second goal of sticking with the default SBT lifecycle.

Related

How to change the naming structure of the dependent libraries in the zip produced by sbt-native-packager?

I am using the Universal plugin of the sbt-native-packager to create a zip package. I am using the below setting for creating a default structure:
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.1.4")
enablePlugins(JavaAppPackaging)
Currently all my project dependencies in the zip fall under the lib folder e.g.
lib/
ch.qos.logback.logback-classic-1.1.3.jar
dom4j.dom4j-1.6.1.jar
How do I change the name of all the libraries to contain only the artifactId and version of the jar and not the complete name. For example, for the above, I want something like this:
lib/
logback-classic-1.1.3.jar
dom4j-1.6.1.jar
The logic for this is hard coded in the JavaAppPackaging archetype. However you can remap your library dependencies.
mappings in Universal := (mappings in Universal).value.map {
case (file, dest) if dest.startsWith("lib/") =>
file -> changeDestination(dest)
case mapping => mapping
}
def changeDestination(dest: String): String = ???
Next you need to change the scriptClasspathOrdering which is responsible for the app_classpath defined in the BashStartScriptPlugin.
scriptClasspathOrdering := scriptClasspathOrdering.value.map {
case (file, dest) if dest.startsWith("lib/") =>
file -> changeDestination(dest)
case mapping => mapping
}
The destination folder should be lib/ as the bash script assumes this.
Note that I have not tested this as this is a very uncommon use case. However the main idea should be clear :)
cheers,
Muki

Include generated resources in a jar (SBT)

I've been writing an SBT plugin that generates resources into resource_managed. I'm now looking to include these generated resources in the generated jar as the SBT docs detail:
Generating resources:
By default, generated resources are not included in the packaged source artifact. To do so, add them as you would other mappings. See Adding files to a package
I've read the docs but honestly how to do this I can't figure out. Can anyone explain it or point me to another project that does this so I can see how they do it?
First just to clarify, they are included in jars containing compiled classes. They are not included in jars containing sources.
By default, generated resources are not included in the packaged
source artifact.
For packageBin the generated files should already be included - just make sure you return all generated files from the generator method. Assuming you want to package them in the sources artifact, this is what you have to do.
Let's assume you have a generator that generates a property file.
lazy val generatePropertiesTask = Def.task {
val file = (Compile / resourceManaged).value / "stack-overflow" / "res.properties"
val contents = s"name=${name.value}\nversion=${version.value}"
IO.write(file, contents)
Seq(file)
}
resourceGenerators in Compile += generatePropertiesTask.taskValue
To include that in the generated sources you have to tell sbt to where the res.properties must be copied in the generated sources artefact. The task, which generates the packaged sources is called packageSrc, therefore you have to set mappings scoped to that task.
mappings in (Compile, packageSrc) += {
((resourceManaged in Compile).value / "stack-overflow" / "res.properties") -> "path/in/jar/res.properties"
}
Because your generator can generate many tasks, and mapping each by hand would be a tedious task, sbt gives you an utility to map multiple paths at once.
mappings in (Compile, packageSrc) ++= {
val allGeneratedFiles = ((resourceManaged in Compile).value ** "*") filter { _.isFile }
allGeneratedFiles.get pair relativeTo((resourceManaged in Compile).value)
}
The first line finds all generated files using path finders and second line maps them to their path in the target jar.

How to create a custom package task to jar a subset of classes in SBT

I am trying to define a separate package task without modifying the original task in compile configuration. This new task will package only a subset of classes conforming an API which we need to be able to share with other teams so they can write plugins for our application. So the end result will be two jars, one with the full application and a second one with a subset of the classes.
I approached this problem by creating a different configuration which I called pluginApi and would redefine the packageBin task within this new configuration so it does not change the original definition of packageBin. This idea was taken from here:
How to create custom "package" task to jar up only specific package in SBT?
In my build.stb I have:
lazy val PluginApi = config("pluginApi") extend(Compile) describedAs("Custom plugin api configuration")
lazy val root = project in file(".") overrideConfigs (PluginApi)
This effectively creates my new configuration and I can call
sbt pluginApi:packageBin
Which generates the complete jar in the same way as compile:packageBin would do. I then try to modify the mappings in the new packageBin task with:
mappings in (PluginApi, packageBin) ~= { (ms: Seq[(File, String)]) =>
ms filter { case (file, toPath) =>
toPath.startsWith("some/path/defining/api")
}
}
but this has no effect. I think the reason is because the call to pluginApi:packageBin is delegated to compile:packageBin rather than it being a cloned task.
I can redefine a new packageBin within the new scope like:
packageBin in PluginApi := {
}
However I would have to rewrite all packageBin functionality instead of reusing existing code. Also, in case that rewriting is unavoidable I am not sure how that implementation would be.
Could somebody provide an example about how to achieve this?
You could have it done as follows
lazy val PluginApi = config("pluginApi").extend(Compile)
inConfig(PluginApi)(Defaults.compileSettings) // you have to have standard
mappings in (PluginApi, packageBin) := {
val original = (mappings in (PluginApi, packageBin)).value
original.filter { case (file, toPath) => toPath.startsWith("some/path/defining/api") }
}
unmanagedSourceDirectories in PluginApi := (unmanagedSourceDirectories in Compile).value
Note that, if you keep your sources in src/main/scala you'll have to override unmanagedSourceDirectories in the newly created configuration.
Normally the unmanagedSourceDirectories contains the configuration name. E.g. src/pluginApi/scala or src/pluginApi/java.
I have had similar problems (with more than one jar per project). Our project uses ant - here you can do it, you just will repeat yourself a lot.
However, I have come to the conclusion that this scenario (2 JARs for one project) actually can be simplified by splitting the project - i.e. making 2 modules out of it.
This way, I don't have to "fight" tools which assume project==artifact (like sbt, maybe maven?, IDEA's default setting,...).
As a bonus point the compiler helps me to verify that my dependencies are correct, i.e. that I did not accidentally make my API package depend on the implementation package - when compiling everything together and only splitting classes apart in the JAR step, you do run the risk of getting an invalid dependency in your setup which you would only see when testing, because during compile time everything is compiled together.

How to avoid re-compiling generated source code

Generating boilerplate source code with sbt works fine:
sourceGenerators in Compile <+= sourceManaged in Compile map { srcDir =>
DslBoilerplate.generate(srcDir, Seq(
"path/to/a/definition/file"
))
}
When I run sbt compile this also compiles the generated source code files, thus producing some class files. I just don't want the generated source code to be re-compiled every time I re-compile the project during development.
So, from the class files I made a jar file and used this instead of the generated source/class files (I deleted those). This worked fine, now having access to the generated code through the jar file. Is there a way though to let sbt do the 4 steps (if needed?) in the initial project build?:
generate source code files
compile those files
create a jar from the produced class files
delete source and class files
(In this question they use the sbt.IO.jar method to create a jar but there they already have existing files...)
Or is there another better approach than making a jar to avoid re-compiling generated source code?
Update 1 (see update 2 below)
Thanks, Seth, for your answer! It worked great to avoid generating the source files with each project compilation since the cache now remembers that they have been created. I'll certainly use this feature, thanks!
But this was actually not what I had in mind with my original question. Sorry for not being clear enough. It might be clearer if we think of this as 2 transformations happening:
Input file ---1---> Source file (*.scala) ---2---> Target file (*.class)
where the transformations are
generation of source code (from some information in an input file) and
compilation of the generated source code
This all works fine when I compile the project with sbt compile.
But then if I "rebuild the project" (in IntelliJ), the generated source code (from the sbt compilation) will compile again, and that's what I want to avoid - but at the same time have access to that code. Is there any other way to avoid compilation than placing this code in a jar and then delete the source and target files?
So I tried to continue along that line of thought wrestling with sbt to make it create a source and target jar - still can't make the target jar. This is what I came up with so far (with help from this):
sourceGenerators in Compile += Def.task[Seq[File]] {
val srcDir = (sourceManaged in Compile).value
val targetDir = (classDirectory in Compile).value
// Picking up inputs for source generation
val inputDirs = Seq("examples/src/main/scala/molecule/examples/seattle")
// generate source files
val srcFiles = DslBoilerplate.generate(srcDir, inputDirs)
// prepare data to create jars
val srcFilesData = files2TupleRec("", srcDir)
val targetFilesData = files2TupleRec("", targetDir)
// debug
println("### srcDir: " + srcDir)
println("### srcFilesData: \n" + srcFilesData.mkString("\n"))
println("### targetDir: " + targetDir)
println("### targetFilesData: \n" + targetFilesData.mkString("\n"))
// Create jar from generated source files - works fine
val srcJar = new File("lib/srcFiles.jar/")
println("### sourceJar: " + srcJar)
sbt.IO.jar(srcFilesData, srcJar, new java.util.jar.Manifest)
// Create jar from target files compiled from generated source files
// Oops - those haven't been created yet, so this jar becomes empty... :-(
// Could we use dependsOn to have the source files compiled first?
val targetJar = new File("lib/targetFiles.jar/")
println("### targetJar: " + targetJar)
sbt.IO.jar(targetFilesData, targetJar, new java.util.jar.Manifest)
val cache = FileFunction.cached(
streams.value.cacheDirectory / "filesCache",
inStyle = FilesInfo.hash,
outStyle = FilesInfo.hash
) {
in: Set[File] => srcFiles.toSet
}
cache(srcFiles.toSet).toSeq
}.taskValue
def files2TupleRec(pathPrefix: String, dir: File): Seq[Tuple2[File, String]] = {
sbt.IO.listFiles(dir) flatMap {
f => {
if (f.isFile && f.name.endsWith(".scala")) Seq((f, s"${pathPrefix}${f.getName}"))
else files2TupleRec(s"${pathPrefix}${f.getName}/", f)
}
}
}
Maybe I still don't need to create jars? Maybe they shouldn't be created in the source generation task? I need help...
Update 2
Silly me!!! No wonder I can't make a jar with class files if I filter them with f.name.endsWith(".scala"), dohh
Since my initial question was not that clear, and Seth's answer is addressing an obvious interpretation, I'll accept his answer (after investigating more, I see that I should probably ask another question).
You want to use FileFunction.cached so that the source files aren't regenerated every time.
Here's an example from my own build:
Compile / sourceGenerators += Def.task[Seq[File]] {
val src = (Compile / sourceManaged).value
val base = baseDirectory.value
val s = streams.value
val cache =
FileFunction.cached(s.cacheDirectory / "lexers", inStyle = FilesInfo.hash, outStyle = FilesInfo.hash) {
in: Set[File] =>
Set(flex(s.log.info(_), base, src, "ImportLexer"),
flex(s.log.info(_), base, src, "TokenLexer"))
}
cache(Set(base / "project" / "flex" / "warning.txt",
base / "project" / "flex" / "ImportLexer.flex",
base / "project" / "flex" / "TokenLexer.flex")).toSeq
}.taskValue
Here the .txt and .flex files are input files to the generator. The actual work of generating the source files is farmed out to my flex method, which returns a java.io.File:
def flex(log: String => Unit, base: File, dir: File, kind: String): File =
...
You should be able to adapt this technique to your build.
FileFunction.cached is described in the API doc and in the sbt FAQ under "How can a task avoid redoing work if the input files are unchanged?" (http://www.scala-sbt.org/0.13/docs/Faq.html). (It would be nice if the material on caching was referenced from http://www.scala-sbt.org/0.13/docs/Howto-Generating-Files.html as well; currently it isn't.)

Add specific directory and its content to Universal target

I am switching from maven to sbt for a Scala project I am working on. I used to work with the maven assembly plugin where you can map any directory in the workspace to a target directory in the assembly. I didn't find any equivalent in sbt-native-package, it worth provide this feature for the Universe kind.
I understood that everything that is present in the universal subdirectory is copied to the package as such, and it works like a charm, but I lack something like the following snippet.
mappings in Universal += {
directory("my/local/dir") -> "static/dirInPackage"
}
I would like to know if there is already a way to do that, in such case, I would be happy to know how to do it, and I propose my help to commit documentation for that part if you want.
If there is no way to do this kind of customization, I will be happy to propose a patch for that after having discussed specifications.
By the way, great job, your packager is working very well, thanks !
After having discussed with the sbt-native-manager team and a first "rejected" pull request, here is the way to do this directory mapping in the build.sbt file (see pull request https://github.com/sbt/sbt-native-packager/pull/160 which provides mode detailed documentation) :
mappings in Universal <++= (packageBin in Compile, target ) map { (_, target) =>
val dir = target / "scala-2.10" / "api"
(dir.***) pair relativeTo(dir.getParentFile)
}
To reduce verbosity of the above snippet, there is an issue (https://github.com/sbt/sbt-native-packager/issues/161) to propose a more human readable way to express this directory mapping:
mappings in Universal ++= allFilesRelativeTo(file(target / "scala-2.10" / "api"))
From https://github.com/sbt/sbt-native-packager
If you'd like to add additional files to the installation dir, simply add them to the universal mappings:
import com.typesafe.sbt.SbtNativePackager.Universal
mappings in Universal += {
file("my/local/conffile") -> "conf/my.conf"
}
You could use a simple map on top of the directory method result.
==> directory method documentation: MappingsHelper.directory
For example:
// Packaging the content of /src/main/resources under conf add the following:
mappings in Universal ++= (directory("src/main/resources").map(t => (t._1, t._2.replace("resources", "conf"))))
This one seems to be the simplest example that worked for me
Takes all files in res/scripts/ and puts it in the bin/ directory when unzipped.
// In build.sbt
mappings in Universal <++= (packageBin in Compile) map { jar =>
val scriptsDir = new java.io.File("res/scripts/")
scriptsDir.listFiles.toSeq.map { f =>
f -> ("bin/" + f.getName)
}
}
If you choose a file that's not created, it will be created for you, for example assets/ will make a new assets folder with the files. If you want files inside of this one using this approach you'll have to make a new Seq at least that's what I did. Here's my example
assets/
├── scripts
│   └── install_dependencies.sh
└── urbangrizzly.database
and the appropriate build.sbt section:
mappings in Universal <++= (packageBin in Compile) map { jar =>
val assetsDir = new java.io.File("assets/")
val scriptsDir = new java.io.File("assets/scripts")
assetsDir.listFiles.toSeq.map { files =>
files -> ("assets/" + files.getName)
} ++ scriptsDir.listFiles.toSeq.map { files =>
files -> ("assets/scripts/" + files.getName)
}
}
If you need more, just keep using the ++ operator to concatenate the lists