How to create a task in SBT that calls a method? - scala

I am new developing tasks in SBT and I'm trying to figure out how to create a task that integrate my existing code.
In my code I have a singleton object that has run method that requires several parameters
object MyObject {
def run( param1: String, param2: Int, param3: String, ...) = {}
}
My question is: How can I define a Task in SBT that calls my run method specifying all its parameters in my build.sbt file?
I can imagine something like this in build.sbt
lazy val myTask: TaskKey[Seq[File]] = taskKey[Seq[File]]("My task")
lazy val myTaskRun = (sourceManaged, dependencyClasspath in Compile, runner in Compile, streams) map { (dir, cp, r, s) =>
val param1 = ...
val param2 = ...
val param3 = ...
val paramN = ....
MyObject.run( param1, param2, param3, ...)
Seq(file("path"))
}

1. You can use fullRunInputTask for that.
In your build.sbt
lazy val example = InputKey[Unit]("example", "Run something.")
fullRunInputTask( example, Compile, "somefun.CallMe")
Under src/main/scala/somefun/CallMe.scala
package somefun
object CallMe {
def main(args: Array[String]) : Unit = {
println("Params are: " + args.mkString(", "))
}
}
To call your task use example, e.g. "example 1 2 3"
2. You can create your own InputTask
see InputTask in SBT Doc
Creating a custom InputTask allows a flexible input parser (with suggestions on tab), allows linking with other tasks. It simple integrates better into SBT.

Related

How to invoke Spark functions (with arguments) from applications.properties(config file)?

So, I have a typesafe config file named application.properties which contains certain values like:
dev.execution.mode = local
dev.input.base.dir = /Users/debaprc/Documents/QualityCheck/Data
dev.schema.lis = asin StringType,subs_activity_date DateType,marketplace_id DecimalType
I have used these values as Strings in my Spark code like:
def main(args: Array[String]): Unit = {
val props = ConfigFactory.load()
val envProps = props.getConfig("dev")
val spark = SparkSession.builder.appName("DataQualityCheckSession")
.config("spark.master", envProps.getString("execution.mode"))
.getOrCreate()
Now I have certain functions defined in my spark code (func1, func2, etc...). I want to specify which functions are to be called, along with the respective arguments, in my application.properties file. Something like this:
dev.functions.lis = func1,func2,func2,func3
dev.func1.arg1.lis = arg1,arg2
dev.func2.arg1.lis = arg3,arg4,arg5
dev.func2.arg2.lis = arg6,arg7,arg8
dev.func3.arg1.lis = arg9,arg10,arg11,arg12
Now, once I specify these, what do I do in Spark, to call the functions with the provided arguments? Or do I need to specify the functions and arguments in a different way?
I agree with #cchantep the approach seems wrong. But if you still want to do something like that, I would decouple the function names in the properties file from the actual functions/methods in your code.
I have tried this and worked fine:
def function1(args: String): Unit = {
println(s"func1 args: $args")
}
def function2(args: String): Unit = {
println(s"func2 args: $args")
}
val functionMapper: Map[String, String => Unit] = Map(
"func1" -> function1,
"func2" -> function2
)
val args = "arg1,arg2"
functionMapper("func1")(args)
functionMapper("func2")(args)
Output:
func1 args: arg1,arg2
func2 args: arg1,arg2
Edited: Simpler approach with output example.

How to get return code in sbt of another internal command (not external command)?

My build.sbt file is like the following:
import sys.process._
def makeProject = Command.args("make", "<args>") {
(state, args) => {
val exec = Exec("run", None);
val makeCommand = ("make " + args.mkString(" "));
val newState = MainLoop.processCommand(exec, state);
// run is an internal command of sbt which can be executed from sbt terminal
// what I need is the return code of the `run` INTERNAL command
// if(exec's execution failed) do not execute the following makeCommand
makeCommand!;
newState;
}
}
I need to get the return code of Exec("run", None). If it ran successfully, the makeCommand after it should be executed - otherwise, do not execute the makeCommand.
You can define a task which depends on an internal task, like this:
myTask := {
val x = run.value
val args = Seq("my", "args")
val makeCommand = ("make " + args.mkString(" "))
makeCommand!
}
Now how to make the args more easily settable? I would suggest the most sbt idiomatic way would be to define them as settings in the build.sbt, and depend on them in your task.

Passing arguments to multiple tasks in SBT

In sbt 0.13.9, I want to be able to run a task which takes in arguments from the command line and then passes those arguments on to two other tasks.
My initial attempt was something along the lines of:
lazy val logTask = InputKey[Unit](...)
lazy val runTask = InputKey[Unit](...)
lazy val TestCase = config("testCase") extend Test
runTask in TestCase := Def.inputTaskDyn {
val args: Seq[String] = spaceDelimited("<arg>").parsed
runReg(args)
}.evaluated
logTask in TestCase := Def.inputTaskDyn {
val args: Seq[String] = spaceDelimited("<arg>").parsed
log(args)
}.evaluated
def runReg(args: Seq[String]) = Def.taskDyn {
val argString = args.mkString(" ")
(logTask in TestCase).toTask(argString).value
(testOnly in TestCase).toTask(s" $argString")
}
def log(args: Seq[String]) {
(runMain in TestCase).toTask(s" LoggingClass $args.mkString(" ")")
}
But then it complains of an Illegal Dynamic Reference argString in (logTask in TestCase).toTask(argsString).value
I've also tried something like:
runTask in TestCase := {
val args: Seq[String] = spaceDelimited("<arg>").parsed
log(args).value
runReg(args).value
}
which also has an Illegal Dynamic Reference for args.
Is there any way of passing in parsed arguments into two tasks and run one after the other?
Thanks for any help.
Instead of assigning args.mkString(" ") to a variable, just pass it without assigning to any variable like below:
(logTask in TestCase).toTask(args.mkString(" ")).value
Update 1:
This kind of issues can also be sorted out with lazy initialization in sbt. So, try something like below:
lazy val argString = args.mkString(" ")
(logTask in TestCase).toTask(argString).value

Scala script in 2.11

I have found an example code for a Scala runtime scripting in answer to Generating a class from string and instantiating it in Scala 2.10, however the code seems to be obsolete for 2.11 - I cannot find any function corresponding to build.setTypeSignature. Even if it worked, the code seems hard to read and follow to me.
How can Scala scripts be compiled and executed in Scala 2.11?
Let us assume I want following:
define several variables (names and values)
compile script
(optional improvement) change variable values
execute script
For simplicity consider following example:
I want to define following variables (programmatically, from the code, not from the script text):
val a = 1
val s = "String"
I want a following script to be compiled and on execution a String value "a is 1, s is String" returned from it:
s"a is $a, s is $s"
How should my functions look like?
def setupVariables() = ???
def compile() = ???
def changeVariables() = ???
def execute() : String = ???
Scala 2.11 adds a JSR-223 scripting engine. It should give you the functionality you are looking for. Just as a reminder, as with all of these sorts of dynamic things, including the example listed in the description above, you will lose type safety. You can see below that the return type is always Object.
Scala REPL Example:
scala> import javax.script.ScriptEngineManager
import javax.script.ScriptEngineManager
scala> val e = new ScriptEngineManager().getEngineByName("scala")
e: javax.script.ScriptEngine = scala.tools.nsc.interpreter.IMain#566776ad
scala> e.put("a", 1)
a: Object = 1
scala> e.put("s", "String")
s: Object = String
scala> e.eval("""s"a is $a, s is $s"""")
res6: Object = a is 1, s is String`
An addition example as an application running under scala 2.11.6:
import javax.script.ScriptEngineManager
object EvalTest{
def main(args: Array[String]){
val e = new ScriptEngineManager().getEngineByName("scala")
e.put("a", 1)
e.put("s", "String")
println(e.eval("""s"a is $a, s is $s""""))
}
}
For this application to work make sure to include the library dependency.
libraryDependencies += "org.scala-lang" % "scala-compiler" % scalaVersion.value

Scala run time code compilation

I am trying to port my java code to pure scala, so would appreciate any help in this regard.
The code below works by first translating my business logic into java code. Here I'm using freemarker template to do template based code generation. After file creation I then use the java comiler to compile the code and create a jar file, which is persisted in a temporary directory
I am currently using the javax.tools.* package that provides runtime compilation. What is the Scala equiavalent to that approach? I'd like to generate pure Scala code using freemarker templates and then run Scala compilation to create a jar file.
Below is sample Java code I am using to make this happen.
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
Iterable<? extends JavaFileObject> compilationUnits = Arrays.asList(javaFileObjects);
StringBuilder builder = new StringBuilder();
builder.append(service.getConfig().getProp("coreLib"));
builder.append(";" +result.getCodeContext().getOmClasspath());
builder.append(";" +jarBuilder.toString());
builder.append(";" +service.getConfig().getProp("tempCodeGen"));
String[] compileOptions = new String[]{"-d", result.getCodeContext().getOmClasspath(),"-cp",builder.toString()} ;
Iterable<String> compilationOptionss = Arrays.asList(compileOptions);
DiagnosticCollector<JavaFileObject> diagnostics = new DiagnosticCollector<JavaFileObject>();
CompilationTask compilerTask = compiler.getTask(null, stdFileManager, diagnostics, compilationOptionss, null, compilationUnits) ;
boolean status = compilerTask.call();
Here's some methods from my own code to compile a project and package it into a jar. Its very far from polished, or properly commented, but hopefully it will indicate where you need to start. I don't think you need to use String Builder as this is not performance critical:
def buildAll(name: String, projDir: String, mainClass: String = ""): Unit =
{
import scala.tools.nsc.{Settings,Global}
val relSrc : List[String] = List()
val maniVersion = "None"
def targDir: String = projDir + "/targ"
def srcDir: String = projDir + "/src"
def srcDirs: List[String] = srcDir :: relSrc
import java.io._
val sings = new scala.tools.nsc.Settings
new File(targDir).mkdir
sings.outputDirs.setSingleOutput(targDir.toString)
val comp = new Global(sings)
val crun: comp.Run = new comp.Run
def getList(fName: String): List[String] =
{
println("starting getList " + fName)
val file = new File(fName)
if (file.isDirectory) file.listFiles.toList.flatMap(i => getList(fName + "/" + i.getName))
else List(fName)
}
crun.compile(srcDirs.flatMap(i => getList(i)))
import sys.process._
("cp -r /sdat/projects/ScalaLibs/scala " + targDir + "/scala").!
import java.util.jar._
val manif = new Manifest
val mf = manif.getMainAttributes
mf.put(Attributes.Name.MANIFEST_VERSION, maniVersion)
if (mainClass != "") mf.put(Attributes.Name.MAIN_CLASS, mainClass)
val jarName = name + ".jar"
val jarOut: JarOutputStream = new JarOutputStream(new FileOutputStream(projDir + "/" + jarName), manif)
AddAllToJar(targDir, jarOut)
jarOut.close
}
def addToJar(jarOut: JarOutputStream, file: File, reldir: String): Unit =
{
val fName = reldir + file.getName
val fNameMod = if (file.isDirectory) fName + "/" else fName
val entry = new JarEntry(fNameMod)
entry.setTime(file.lastModified)
jarOut.putNextEntry(entry)
if (file.isDirectory)
{
jarOut.closeEntry
file.listFiles.foreach(i => addToJar(jarOut, i, fName + "/"))
}
else
{
var buf = new Array[Byte](1024)
val in = new FileInputStream(file)
Stream.continually(in.read(buf)).takeWhile(_ != -1).foreach(jarOut.write(buf, 0, _))
in.close
jarOut.closeEntry()
}
}
def AddAllToJar(targDir: String, jarOut: JarOutputStream): Unit =
new java.io.File(targDir).listFiles.foreach(i => addToJar(jarOut, i, ""))
You need to add the Scala Compiler to the build path. The Scala compiler takes a list of sourcefiles and produces the compiled class files in the directory set in the output directory. Getting to grips with the full capabilities of the compiler is a major task though.
And when using scala you don't need to use any free-marker similar tools. Since scala version 2.10 there is string interpolation feature.
val traitName = "MyTrait"
val packageName = "my.pack"
val typeParams = List("A", "B", "C")
s"""
|package ${packageName}
|
|trait ${traitName}[${typeParams.mkString(",")}] {
| ${typeParams.map(t => s"val ${t.toLowerCase()}: ${t}")}
|}
|
""".stripMargin
will yield:
package my.pack
trait MyTrait[A,B,C] {
List(val a: A, val b: B, val c: C)
}
No dependencies needed :)
If you would like to compile generated code in runtime, the simplest solution is twitter eval utility
See the suggestion here - Generating a class from string and instantiating it in Scala 2.10
Or the Eval class from twitter's util library -
https://github.com/twitter/util/blob/master/util-eval/src/main/scala/com/twitter/util/Eval.scala