Should plugins cache Initialize[Task[T]] objects? - scala

By trial and probably error, as a plugin author I've fallen into using the following style, which seems to work:
object AmazingPlugin extends AutoPlugin {
object autoImport {
val amaze = TaskKey[Amazement]("Do something totally effing amazing")
}
import autoImport._
lazy val ethDefaults : Seq[sbt.Def.Setting[_]] = Seq(
amaze in Compile := { amazeTask( Compile ).value },
amaze in Test := { amazeTask( Test ).value }
)
def amazeTask( config : Configuration ) : Initialize[Task[Amazement]] = Def.task {
???
}
}
Unfortunately, I do not actually understand how all of these constructs work, why it is an Initialize[Task[T]] I generate rather than a Task[T], for example. I assume that this idiom "does the right thing", which I take to mean that the amazeTask function generates some wrapper or seed or generator of an immutable Task for each Configuration and binds it just once to the appropriate task key. But it is entirely opaque to me how this might work. For example, when I look up Initialize, I see a value method that requires an argument, which I do not supply in the idiom above. I assume in designing the SBT configuration DSL tricks with implicits and/or macros were used, and shrug it off.
However, recently I've wanted to factor out some logic from my tasks into what are logically private functions, but which also require access to the value of tasks and settings. If just used private functions, gathering all the arguments would become repetitive boilerplate of the form
val setting1 = (setting1 in config).value
val setting2 = (setting2 in config).value
...
val settingN = (settingN in config).value
val derivedValue = somePrivateFunction( setting1, setting2 ... settingN )
everywhere I need to use get the value derived from settings. So it's better to factor all this into a derivedValue task, and I can replace all of the above with
derivedValue.value
Kewl.
But I do want derivedValue to be private, so I don't bind it to a task key. I just do this:
private def findDerivedValueTask( config : Configuration ) : Initialize[Task[DerivedValue]] = Def.task {
val setting1 = (setting1 in config).value
val setting2 = (setting2 in config).value
...
val settingN = (settingN in config).value
somePrivateFunction( setting1, setting2 ... settingN )
}
Now it works fine to implement my real, public task as...
def amazeTask( config : Configuration ) : Initialize[Task[Amazement]] = Def.task {
val derivedValue = findDerivedValueTask( config ).value
// do stuff with derivedValue that computes and Amazement object
???
}
Great! It really does work just fine.
But I have a hard time presuming that there is some magic by which this idiom does the right thing, that is generate an immutable Task object just once per config and reuse it. So, I think to myself, I should memoize the findDerivedValueTask function, so that after it generates a task, the task gets stored in a Map, whose keys are Configurations but whose values are logically Tasks.
But now my nonunderstanding of what happens behind the scenes bites. Should I store Initialize[Task[DerivedValue]] or just Task[DerivedValue], or what? Do I have to bother, does sbt have some clever magic that is already taking care of this for me? I really just don't know.
If you have read this far, I am very grateful. If you can clear this up, or point me to documentation that explains how this stuff works, I'll be even more grateful. Thank you!

Related

sbt val evaluated more than once

I'm creating a val into my build.sbt, made of a random string, to be used in the Setup and Cleanup methods for scalatest, like this:
val foo = Random.alphanumeric.take(3).mkString
...
Test / testOptions += Tests.Setup(() => {
// do stuff with it
})
...
Test / testOptions += Tests.Cleanup(() => {
// do stuff with the same string
}
but it seems that the two functions are actually re-evaluating the val, resulting in two different strings. It seems that the forking of the JVM (fork := true) does not play a role into it, so I'm kinda out of ideas. Is that intended and/or is there a way to fix it/finding another approach to the problem (native to Scala/sbt)?
Apparently the solution was easier than thought:
lazy val foo = SettingKey[String]("foo", "Random string")
foo := Random.alphanumeric.take(3).mkString
and then call foo.value in the sbt code after

Sane way to add members to a memder via anonymous subclass

I'm learning Scala along with a DSL called Chisel.
In Chisel there is an omnipresent pattern like:
class TestModule extends Module {
// overriding an io field of Module
val io = IO(new Bundle {
// add a new field, that was not defined in Bundle
val a = ...
})
// do something with io.a
}
As I figured out, this code can't be compiled ever since Scala 2.12.0 . Because of a change in type inference, the io field does not get the 'extended' type, and io.a is not accessible effectively from anywhere outside the definition of the anonymous subclass.
I can understand (more or less) the motivation of this change.
But this very implication of it looks quite strange to me.
I've managed to write some ways to overcome this problem, but none of them satisfies me completely.
So:
What is the shortest way to extend an overridden-field-object with new fields?
From a DSL-user point of view
From a DSL(library)-writer point of view
And being more general: what is the best way to add fields 'in-place'? And if there is no good way, why is adding fields in-place is so much left out?
My solutions for DSL-users:
add an explicit type definition, as supposed by the authors of the change
class TestModule extends Module {
val io: {val a: Type; ...} = IO(new Bundle {
val a = ...
...
})
}
This kinda doubles the typing. With more than about three additional fields this looks really scary. Even being broken into multiple lines. And in Chisel there usually are quite a lot of fields here.
assign the whole thing to another (non-overriding) field first
class TestModule extends Module {
val myIo = IO(new Bundle {
val a = ...
...
})
val io = myIo
}
Adds one unnecessary line, forces to invent another name... And looks like magic. Really.
make the class named instead of anonymous
class TestModule extends Module {
class MyIo extends Bundle {
val a = ...
...
}
val io = IO(new MyIo)
}
Still a line and a name. And looks a bit too involved for people who came to use a DSL without much will to dive deep into Scala (I'm not one of them, but I know a lot of such people).
As for DSL-writers, I can suggest only to write a macro. And, as far as I understand the problem, the macro should not only be plugged in place of the IO() function, but it should replace the whole val io = IO(...) assignment.

Override the tags for a test in ScalaTest

I'd like to apply the DRY principle to my ScalaTest test definitions. Specifically, I'd like to define an abstract test class that defines a bunch of tests. All of the tests call some function with parameters indicating the conditions to be tested. The definition of that function is left to the extending class. So far, this is doable.
Next, I'd like to tag any test that has ever failed and been fixed as a "regression" test, so I can run just those tests if I am so inclined.
But the tests are initially tagged in the abstract class. I need to override the tags, or add a tag, in the implementing class.
Is there a clean way of doing this? The documentation implies that there is, but so far I can't find an example of how to do it.
I never did find documentation on how to do it, but there was enough information in the ScalaDocs that I was able to figure it out. For the benefit of those who might want to do something like this, here is what you need to know:
First, you will want to define your own trait that you can mix-in to get this additional behavior. It will override the definition of tag(), like so:
trait _____ extends SuiteMixin with Informing { this: Suite with Informing =>
// with Informing, etc. is so that you can call info()
// to add comments to tests - not strictly needed for this
abstract override def tags : Map[String, Set[String]] = {
// implementation
}
}
The implementation must call super.tags, and then add whatever needs to be added to the resulting data structure before returning it. The keys of the result will be test names, the values will be sets of tag strings. NOTE: tests that have no tags will not be present, so you will not be able to depend on iterating over that object to find the test that you want to operate on. You will end up having to call this.testNames and iterating over that.
Here is an example of code that I wrote that shows how to pull this off.
abstract override def tags : Map[String, Set[String]] = {
val original = super.tags
val matching = <list of what to automatically add tags to>
if ( matching.isEmpty ) original
else {
val tests = this.testNames.toList
def extend( result: Map[String, Set[String]], test_list: List[String] ) : Map[String, Set[String]] =
if ( test_list.isEmpty ) result
else {
val matches = ( for ( p <- matching if ( <applicable> ) ) yield true ) contains true
if ( ! matches ) extend( result, test_list.tail )
else extend(
result.updated(
test_list.head,
result.getOrElse( test_list.head, Set[String]() )
+ "<tag-to-be-added>" ),
test.tail
)
}
extend( original, tests )
}
}
Hope this helps someone besides me.
Comments on how to do this in a more elegant or scala-esque manner are welcome and appreciated.

How do I test code that requires an Environment Variable?

I have some code that requires an Environment Variable to run correctly. But when I run my unit tests, it bombs out once it reaches that point unless I specifically export the variable in the terminal. I am using Scala and sbt. My code does something like this:
class something() {
val envVar = sys.env("ENVIRONMENT_VARIABLE")
println(envVar)
}
How can I mock this in my unit tests so that whenever sys.env("ENVIRONMENT_VARIABLE") is called, it returns a string or something like that?
If you can't wrap existing code, you can change UnmodifiableMap System.getenv() for tests.
def setEnv(key: String, value: String) = {
val field = System.getenv().getClass.getDeclaredField("m")
field.setAccessible(true)
val map = field.get(System.getenv()).asInstanceOf[java.util.Map[java.lang.String, java.lang.String]]
map.put(key, value)
}
setEnv("ENVIRONMENT_VARIABLE", "TEST_VALUE1")
If you need to test console output, you may use separate PrintStream.
You can also implement your own PrintStream.
val baos = new java.io.ByteArrayOutputStream
val ps = new java.io.PrintStream(baos)
Console.withOut(ps)(
// your test code
println(sys.env("ENVIRONMENT_VARIABLE"))
)
// Get output and verify
val output: String = baos.toString(StandardCharsets.UTF_8.toString)
println("Test Output: [%s]".format(output))
assert(output.contains("TEST_VALUE1"))
Ideally, environment access should be rewritten to retrieve the data in a safe manner. Either with a default value ...
scala> scala.util.Properties.envOrElse("SESSION", "unknown")
res70: String = Lubuntu
scala> scala.util.Properties.envOrElse("SECTION", "unknown")
res71: String = unknown
... or as an option ...
scala> scala.util.Properties.envOrNone("SESSION")
res72: Option[String] = Some(Lubuntu)
scala> scala.util.Properties.envOrNone("SECTION")
res73: Option[String] = None
... or both [see envOrSome()].
I don't know of any way to make it look like any/all random env vars are set without actually setting them before running your tests.
You shouldn't test it in unit-test.
Just extract it out
class F(val param: String) {
...
}
In your prod code you do
new Foo(sys.env("ENVIRONMENT_VARIABLE"))
I would encapsulate the configuration in a contraption which does not expose the implementation, maybe a class ConfigValue
I would put the implementation in a class ConfigValueInEnvVar extends ConfigValue
This allows me to test the code that relies on the ConfigValue without having to set or clear environment variables.
It also allows me to test the base implementation of storing a value in an environment variable as a separate feature.
It also allows me to store the configuration in a database, a file or anything else, without changing my business logic.
I select implementation in the application layer.
I put the environment variable logic in a supporting domain.
I put the business logic and the traits/interfaces in the core domain.

reproduce task dependencies in actual practise

the sbt task documentation shows an example of usage dependencies. It is very simple, artificial but it works! So I reproduced it in my project/scala.build without problem.
Note that I choose global scope to make tasks available for any project and any configuration
import sbt._
import Keys._
object TestBuild extends Build {
lazy val sampleTask = taskKey[Int]("A sample task")
lazy val intTask = taskKey[Int]("An int task")
override lazy val settings = super.settings ++ Seq(
intTask := 1 + 2 ,
sampleTask := intTask.value + 1
)
}
Now I'm trying to do something useful and enrich existing sbt key definitions with task that collects compiled class names
import sbt._
import Keys._
import sbt.inc.Analysis
import xsbti.api.ClassLike
import xsbt.api.Discovery.{isConcrete, isPublic}
object TestBuild extends Build {
lazy val debugAPIs = taskKey[List[String]]("list of all top-level definitions")
override lazy val settings = super.settings ++ Seq(
debugAPIs := getAllTop( compile.value )
)
private def getAllTop(analysis : Analysis) : List[String] =
Tests.allDefs(analysis).toList collect {
case c : ClassLike if isConcrete(c) && isPublic(c) => c.name
}
}
Now I get error from sbt:
Reference to undefined setting:
{.}/*:compile from {.}/*:debugAPIs (/home/sbt/project/build.scala:11)
So I have two questions:
How should I define debugAPIs properly so that it task would be available for all projects and all configurations?
How can I reproduce this error in synthetic configuration?
I'm more interested in the second question actually. I look for deep understanding of how sbt works because I'd like to write a plugin for it.
The problem is that you try to access a key value without a proper Scope.
The documentation gives us some hint here.
By default, all the keys associated with compiling, packaging, and
running are scoped to a configuration and therefore may work
differently in each configuration. The most obvious examples are the
task keys compile, package, and run; but all the keys which affect
those keys (such as source-directories or scalac-options or
full-classpath) are also scoped to the configuration.
Let's first focus on a very simple example, which maybe doesn't make much sense, but illustrates the problem. Lets assume that you want to redefine the compile task to itself.
override lazy val settings = super.settings ++ Seq (
compile := { compile.value }
)
Running this in SBT will give you an error, which is more or less like this
[error] {.}/*:compile from {.}/*:compile (/tmp/q-23723818/project/Build.scala:12)
[error] Did you mean compile:compile ?
We didn't specify the scope so SBT picked some defaults. The project was set to ThisBuild (meaning no specific project) and configuration set to Global. The setting was undefined in that context. However it's important to understand that a key is not a setting. The key can exist without scope, but the value of a key is attached to a scope. Note also that, if SBT won't find the value in the requested scope it can delegate to other scopes, but this is another topic.
How can we check this? Turns out that quite simple. Let's ignore the error, and let the SBT start.
If you type inspect compile you'll see that the inspect will look in compile:compile, where the value is defined. We can force it to look in a specific scope, e.g. inspect {.}/*:compile, will look in scope that gave us the error.
> inspect {.}/*:compile
[info] No entry for key.
Indeed it's undefined.
How to solve the issue? You have to give SBT the scope you're looking for. Naively you could try to add a configuration scope.
// this will NOT work
override lazy val settings = super.settings ++ Seq (
compile in Compile := { (compile in Compile).value }
)
Well but there is no global compile, there is only compile per project. You could overcome the issue by not overriding global settings, but the settings for a specific project, and specifying Compile configuration there.
lazy val root = project.in(file(".")).settings(Seq(
compile in Compile := {(compile in Compile).value}
): _*)
This would work,but what if you want to get the compile value regardless of where it is? This is where ScopeFilter comes in handy. Back to your original example. I assume you want to get compile's Analysis object from all the projects.
import sbt._
import Keys._
import sbt.inc.Analysis
import xsbti.api.ClassLike
import xsbt.api.Discovery.{isConcrete, isPublic}
object TestBuild extends Build {
val debugAPIs = taskKey[Seq[String]]("list of all top-level definitions")
val compileInAnyProject = ScopeFilter(inAnyProject, inConfigurations(Compile))
override lazy val settings = super.settings ++ Seq(
debugAPIs := {
getAllTop(compile.all(compileInAnyProject).value)
}
)
private def getAllTop(analyses : Seq[Analysis]) : Seq[String] =
analyses.flatMap { analysis =>
Tests.allDefs(analysis) collect { case c : ClassLike if isConcrete(c) && isPublic(c) => c.name }
}
}
What we created is a ScopeFilter filtering for any project, and in that projects for Compile configuration. Then we looked for all compile values.
You can configure the ScopeFilter to match your needs, and only filter for specific projects/configurations or even tasks. But the key to understand the problem is to remember that in SBT settings are always scoped.
Edit
You have asked how it comes that the compile is not defined globally but is available to every project. This is because there is Defaults.defaultSettings which define it. And each project include it. If you removed super.settings from your Build definition you'd see that among others compile is undefined.
And as if you should do it this way. Well overriding settings in your plugin is in general discouraged in Plugin Best Practices. However I recommend that you read it, together with Plugins chapter. It should give you an idea of how to proceed.
You can also get multiple values from multiple scopes by defining new task returning them. For example to get analyses with a project, you could use following piece of code.
object TestBuild extends Build {
val debugAPIs = taskKey[Seq[(String, String)]]("list of all top-level definitions")
val compileInAnyProject = ScopeFilter(inAnyProject, inConfigurations(Compile))
override lazy val settings = super.settings ++ Seq(
debugAPIs := {
getAllTop(analysisWithProject.all(compileInAnyProject).value)
}
)
lazy val analysisWithProject = Def.task { (thisProject.value, compile.value) }
private def getAllTop(analyses : Seq[(ResolvedProject, Analysis)]) : Seq[(String, String)] =
analyses.flatMap { case (project, analysis) =>
Tests.allDefs(analysis) collect { case c : ClassLike if isConcrete(c) && isPublic(c) => (project.id, c.name) }
}
}