I have an sbt project with a few subprojects, each of which publishes some artifacts and has a fairly extensive test suite.
When I run the build on my CI server, I want to publish the artifacts to a staging location and run the tests after the publishing task. Since others may want the artifacts, I'd like to tell sbt that I want it to build all the artifacts for all subprojects, then run all the tests, since by default it seems to run them interleaved in an unspecified order.
I have a ScopeFilter giving me access to all my subprojects, so I can make my ciBuild task depend on something like the following
(test in Test).all(subprojectScopeFilter).dependsOn(myArtifactsTask.all(subprojectScopeFilter))`
However, that doesn't seem to have any real effect on the order, and I definitely see some subprojects running tests before others have run their myArtifactsTask. I'm guessing that I don't fully understand how all works and it might be saying that each independent subproject's test task depends on that same subproject's myArtifactsTask? If that's the case, how can I specify what I want? Is it documented somewhere that I've missed? The manual describes the basics of all but not how it interacts with other constructs.
SBT will resolve automatically the order between task and projects and build them in that order.
What you could do is - let's assume you have three projects. Root and two sub-projects. I assume that the key myArtifactTask is defined in the root.
project/Build.scala
object MyBuild extends Build {
val myArtifactTask = TaskKey[Unit]("my-artifact-task", "My Artifact Task")
}
The myArtifactTask is implemented in both sub-projects.
subproject-a/build.sbt
myArtifactTask := {
println("myArtifactTask:project-a")
}
subproject-a/build.sbt
myArtifactTask := {
println("myArtifactTask:project-b")
}
What you want to do is to define your root's build.sbt in a way that it calls myArtifactTask in both projects. Then you could define new task testedArtifact which would depend on myArtifactTask.
build.sbt
lazy val testedArtifact = taskKey[Unit]("Runs myArtifactTask followed by tests")
lazy val inAnyProjectButRoot: ScopeFilter = ScopeFilter (
inAnyProject -- inProjects(ThisProject)
)
myArtifactTask := {
myArtifactTask.all(inAnyProjectButRoot).value
}
testedArtifact := {
(test in Test).all(anyProjectButRoot).value
}
testedArtifact <<= testedArtifact.dependsOn(myArtifactTask)
Now calling testedArtifactin the root project will first call all myArtifactTasks in sub-projects followed by tests.
Related
I'm attempting to compile and run scala code that generates a file in SBT (in this case its a swagger file).
the following sbt executes fine. The jar is built and executed and the swagger.zip file the execution creates is in the target directory. I can't, however, seem to get the zip file to get published to my artifactory repo like my standard jar files would.
Any idea on what I'm missing?
publishArtifact in (Compile, packageBin) := false
publishArtifact in run := true
val myZipTask = taskKey[File]("swagger-zip")
myZipTask := {
file("swagger.zip")
}
addArtifact(Artifact("swagger", "zip", "zip"), myZipTask )
The run command doesn't trigger any publishing tasks, as that's not what it's for. I think you have tried to turn on publishing when run is called with "publishArtifact in run := true", but that code is not sufficient to achieve that; you'd need to attach many other Tasks to that command. That's not what you want anyway.
Your case fits entirely within the expected usage of the addArtifact helper (see the docs, and the code)
As you need to run code to generate artefacts, you should provide that code as a Task argument to the addArtifact helper, i.e.
val myZipTask = taskKey[File]("return the swagger-zip file")
val runZipCodeTask = taskKey[Unit]("run the swagger-zip code")
// See http://www.scala-sbt.org/0.13.2/docs/faq.html, “How can I create a custom run task, in addition to run?”
// and https://stackoverflow.com/questions/23409993/defining-sbt-task-that-invokes-method-from-project-code
fullRunTask(runZipCodeTask, Compile, "ZipGeneratorMainClass")
myZipTask := {
runZipCodeTask.value
file("target/swagger/swagger.zip")
}
addArtifact(Artifact("swagger", "zip", "zip"), myZipTask)
Then run "sbt publish"
Working demo
See https://gist.github.com/RichardBradley/5384a5e0da2427df237f42fe512b30b8 for a working demo.
Note about scopes - consider generating this file in a meta-project, not the main project
See https://stackoverflow.com/a/23416018/8261
(Long question ahead. Simplified tl;dr at the bottom).
I have two ScalaJS projects built with SBT - "myapp" and "mylib", in the following directory structure
root/build.sbt
root/myapp/build.sbt
root/myapp/jvm/
root/myapp/js/
root/myapp/shared/
root/mylib/build.sbt
root/mylib/jvm
root/mylib/js
root/mylib/shared
lib exports an artifact named "com.example:mylib:0.1", which as used as a libraryDependency for myapp.
myapp and mylib are in separate repositories, contain their own build files, and should be able to be build completely separately (i.e. they must contain their own individual build config).
In production, they will be built separately with mylib being first published as a maven artifact before building myapp separately.
In development however, I want to be able to merge these into a parent SBT project so that both can be developed in parallel without needing to use publishLocal after each change.
In a traditional (not scalajs) project this would be quite easy
$ROOT/build.sbt:
lazy val mylib = project
lazy val myapp = project.dependsOn(mylib)
However in ScalaJS, we actually have two projects inside each module - appJVM, appJS, libJVM and libJS. As such, the above configuration only finds the aggregate root project and does not correctly apply the dependsOn configuration to the actual JVM and JS projects.
(i.e. myapp and mylib build.sbt each contains two projects, and an aggregate root project)
Ideally I'd like to be able to do something like the following
lazy val mylibJVM = project
lazy val myappJVM = project.dependsOn(mylibJVM)
lazy val mylibJS = project
lazy val myappJS = project.dependsOn(myappJS)
Unfortunately this just creates new projects within the root instead of importing the subprojects themselves.
I've also tried various combinations of paths (such as)
lazy val mylibJVM = project.in(file("mylib/jvm"))
But this doesn't see configuration in build.sbt file in mylib
Ultimately I keep running up against the same problem - when importing an existing multi-project SBT project into a parent sbt file, it imports the root project, but does not seem to provide a way to import a subproject from an existing multimodule SBT file in a way that lets me add dependsOn configuration to it.
tl;dr
If I have
root/mylib/build.sbt with multiple projects defined and
root/myapp/build.sbt with multiple projects defined
Is it possible to import individual subprojects into root/build.sbt instead of the root project from the submodule?
i.e. Can I have two layers of multiproject builds.
After spending a lot of time digging through SBT source code, I managed to figure out a solution. This isn't clean, but it works. (For bonus points, it imports correctly into IntelliJ).
// Add this function to your root build.sbt file.
// It can be used to define a dependency between any
// `ProjectRef` without needing a full project definition.
def addDep(from:String, to:String) = {
buildDependencies in Global <<= (
buildDependencies in Global,
thisProjectRef in from,
thisProjectRef in to) {
(deps, fromref, toref) =>
deps.addClasspath(fromref, ResolvedClasspathDependency(toref, None))
}
}
// `project` will import the `build.sbt` file
// in the subdirectory of the same name as the `lazy val`
// (performed by an SBT macro). i.e. `./mylib/build.sbt`
//
// This won't reference the actual subprojects directly,
// will but import them into the namespace such that they
// can be referenced as "ProjectRefs", which are implicitly
// converted to from strings.
//
// We then aggregate the JVM and JS ScalaJS projects
// into the new root project we've defined. (Which unfortunately
// won't inherit anything from the child build.sbt)
lazy val mylib = project.aggregate("mylibJVM","mylibJS")
lazy val myapp = project.aggregate("myappJVM","myappJS")
// Define a root project to aggregate everything
lazy val root = project.in(file(".")).aggregate(mylib,myapp)
// We now call our custom function to define a ClassPath dependency
// between `myapp` -> `mylib` for both JVM and JS subprojects.
// In particular, this will correctly find exported artifacts
// so that `myapp` can refer to `mylib` in libraryDependencies
// without needing to use `publishLocal`.
addDep("myappJVM", "mylibJVM")
addDep("myappJS","mylibJS")
I am trying to define a separate package task without modifying the original task in compile configuration. This new task will package only a subset of classes conforming an API which we need to be able to share with other teams so they can write plugins for our application. So the end result will be two jars, one with the full application and a second one with a subset of the classes.
I approached this problem by creating a different configuration which I called pluginApi and would redefine the packageBin task within this new configuration so it does not change the original definition of packageBin. This idea was taken from here:
How to create custom "package" task to jar up only specific package in SBT?
In my build.stb I have:
lazy val PluginApi = config("pluginApi") extend(Compile) describedAs("Custom plugin api configuration")
lazy val root = project in file(".") overrideConfigs (PluginApi)
This effectively creates my new configuration and I can call
sbt pluginApi:packageBin
Which generates the complete jar in the same way as compile:packageBin would do. I then try to modify the mappings in the new packageBin task with:
mappings in (PluginApi, packageBin) ~= { (ms: Seq[(File, String)]) =>
ms filter { case (file, toPath) =>
toPath.startsWith("some/path/defining/api")
}
}
but this has no effect. I think the reason is because the call to pluginApi:packageBin is delegated to compile:packageBin rather than it being a cloned task.
I can redefine a new packageBin within the new scope like:
packageBin in PluginApi := {
}
However I would have to rewrite all packageBin functionality instead of reusing existing code. Also, in case that rewriting is unavoidable I am not sure how that implementation would be.
Could somebody provide an example about how to achieve this?
You could have it done as follows
lazy val PluginApi = config("pluginApi").extend(Compile)
inConfig(PluginApi)(Defaults.compileSettings) // you have to have standard
mappings in (PluginApi, packageBin) := {
val original = (mappings in (PluginApi, packageBin)).value
original.filter { case (file, toPath) => toPath.startsWith("some/path/defining/api") }
}
unmanagedSourceDirectories in PluginApi := (unmanagedSourceDirectories in Compile).value
Note that, if you keep your sources in src/main/scala you'll have to override unmanagedSourceDirectories in the newly created configuration.
Normally the unmanagedSourceDirectories contains the configuration name. E.g. src/pluginApi/scala or src/pluginApi/java.
I have had similar problems (with more than one jar per project). Our project uses ant - here you can do it, you just will repeat yourself a lot.
However, I have come to the conclusion that this scenario (2 JARs for one project) actually can be simplified by splitting the project - i.e. making 2 modules out of it.
This way, I don't have to "fight" tools which assume project==artifact (like sbt, maybe maven?, IDEA's default setting,...).
As a bonus point the compiler helps me to verify that my dependencies are correct, i.e. that I did not accidentally make my API package depend on the implementation package - when compiling everything together and only splitting classes apart in the JAR step, you do run the risk of getting an invalid dependency in your setup which you would only see when testing, because during compile time everything is compiled together.
I'm using SBT to build a scala project. I want to define a very simple task, that when I input generate in sbt:
sbt> generate
It will invoke my my.App.main(..) method to generate something.
There is a App.scala file in myproject/src/main/scala/my, and the simplified code is like this:
object App {
def main(args: Array[String]) {
val source = readContentOfFile("mysource.txt")
val result = convert(source)
writeToFile(result, "mytarget.txt");
}
// ignore some methods here
}
I tried to add following code into myproject/build.sbt:
lazy val generate = taskKey[Unit]("Generate my file")
generate := {
my.App.main(Array())
}
But which doesn't compile since it can't find my.App.
Then I tried to add it to myproject/project/build.scala:
import sbt._
import my._
object HelloBuild extends Build {
lazy val generate = taskKey[Unit]("Generate my file")
generate := {
App.main(Array())
}
}
But it still can't be compiled, that it can't find package my.
How to define such a task in SBT?
In .sbt format, do:
lazy val generate = taskKey[Unit]("Generate my file")
fullRunTask(generate, Compile, "my.App")
This is documented at http://www.scala-sbt.org/0.13.2/docs/faq.html, “How can I create a custom run task, in addition to run?”
Another approach would be:
lazy val generate = taskKey[Unit]("Generate my file")
generate := (runMain in Compile).toTask(" my.App").value
which works fine in simple cases but isn't as customizable.
Update: Jacek's advice to use resourceGenerators or sourceGenerators instead is good, if it fits your use case — can't tell from your description whether it does.
The other answers fit the question very well, but I think the OP might benefit from mine, too :)
The OP asked about "I want to define a very simple task, that when I input generate in sbt will invoke my my.App.main(..) method to generate something." that might ultimately complicate the build.
Sbt already offers a way to generate files at build time - sourceGenerators and resourceGenerators - and I can't seem to notice a need to define a separate task for this from having read the question.
In Generating files (see the future version of the document in the commit) you can read:
sbt provides standard hooks for adding source or resource generation
tasks.
With the knowledge one could think of the following solution:
sourceGenerators in Compile += Def.task {
my.App.main(Array()) // it's not going to work without one change, though
Seq[File]() // a workaround before the above change is in effect
}.taskValue
To make that work you should return a Seq[File] that contains files generated (and not the empty Seq[File]()).
The main change for the code to work is to move the my.App class to project folder. It then becomes a part of the build definition. It also reflects what the class does as it's really a part of the build not the artifact that's the product of it. When the same code is a part of the build and the artifact itself you don't keep the different concerns separate. If the my.App class participates in a build, it should belong to it - hence the move to the project folder.
The project's layout would then be as follows:
$ tree
.
├── build.sbt
└── project
├── App.scala
└── build.properties
Separation of concerns (aka #joescii in da haus)
There's a point in #joescii's answer (which I extend in the answer) - "to make it a separate project that other projects can use. To do this, you will need to put your App object into a separate project and include it as a dependency in project/project", i.e.
Let's assume you've got a separate project build-utils with App.scala under src/main/scala. It's a regular sbt configuration with just the Scala code.
jacek:~/sandbox/so/generate-project-code
$ tree build-utils/
build-utils/
└── src
└── main
└── scala
└── App.scala
You could test it out as a regular Scala application without messing up with sbt. No additional setup's required (and frees your mind from sbt that might be beneficial at times - less setup is always of help).
In another project - project-code - that uses App.scala that is supposed to be a base for the build, build.sbt is as follows:
project-code/build.sbt
lazy val generate = taskKey[Unit]("Generate my file")
generate := {
my.App.main(Array())
}
Now the most important part - the wiring between projects so the App code is visible for the build of project-code:
project-code/project/build.sbt
lazy val buildUtils = RootProject(
uri("file:/Users/jacek/sandbox/so/generate-project-code/build-utils")
)
lazy val plugins = project in file(".") dependsOn buildUtils
With the build definition(s), executing generate gives you the following:
jacek:~/sandbox/so/generate-project-code/project-code
$ sbt
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Loading project definition from /Users/jacek/sandbox/so/generate-project-code/project-code/project
[info] Updating {file:/Users/jacek/sandbox/so/generate-project-code/build-utils/}build-utils...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/Users/jacek/sandbox/so/generate-project-code/project-code/project/}plugins...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 1 Scala source to /Users/jacek/sandbox/so/generate-project-code/build-utils/target/scala-2.10/classes...
[info] Set current project to project-code (in build file:/Users/jacek/sandbox/so/generate-project-code/project-code/)
> generate
Hello from App.main
[success] Total time: 0 s, completed May 2, 2014 2:54:29 PM
I've changed the code of App to be:
> eval "cat ../build-utils/src/main/scala/App.scala"!
package my
object App {
def main(args: Array[String]) {
println("Hello from App.main")
}
}
The project structure is as follows:
jacek:~/sandbox/so/generate-project-code/project-code
$ tree
.
├── build.sbt
└── project
├── build.properties
└── build.sbt
Other changes aka goodies
I'd also propose some other changes to the code of the source generator:
Move the code out of main method to a separate method that returns the files generated and have main call it. It'll make reusing the code in sourceGenerators easier (without unnecessary Array() to call it as well as explicitly returning the files).
Use filter or map functions for convert (to add a more functional flavour).
The solution that #SethTisue proposes will work. Another approach is to make it a separate project that other projects can use. To do this, you will need to put your App object into a separate project and include it as a dependency in project/project, OR package it as an sbt plugin ideally with this task definition included.
For an example of how to create a lib that is packaged as a plugin, take a look at snmp4s. The gen directory contains the code that does some code generation (analogous to your App code) and the sbt directory contains the sbt plugin wrapper for gen.
I want the test-compile action to put the contents of src/test/resources into target/scala_2.8.1/test-classes.
The reason for this is that I'm running SBT with IntelliJ IDEA, which puts the test-classes directory on the classpath when it runs tests.
My logback-test.xml file (configuration for logback logging framework) lives in src/test/resources, and it's not being found during testing.
This doesn't directly answer your question, but instead shows an the alternative that I use.
I add two new tasks: prepare and test-prepare.
lazy val prepare = task{
None
} dependsOn (compile, copyResources) describedAs ("Compiles main, copies main resources. Use for the before-action in the idea-sbt-plugin")
lazy val testPrepare = task{
None
} dependsOn (testCompile, copyResources, copyTestResources) describedAs ("Compiles main and test, copies all resources. Use for the before-action in the idea-sbt-plugin")
I then configure the 'before run' SBT action to these. This requires the idea-sbt-plugin.
I use the sbt-idea SBT Processor to configure the IntelliJ module setup. This includes target/scala_2.8.1/[test-]resources as a dependency.
While there might be a better way of doing it, this should work as well as long as nothing else overrides testCompile. Just add the following into your project definition.
override lazy val testCompile = testCompileAction dependsOn task {
import sbt.Process._
"cp -R src/test/resources/* target/scala_2.8.1/test-classes" ! log
None
}
override lazy val testCompile = testCompileAction dependsOn copyTestResources
override def testResourcesOutputPath = testCompilePath