I'm trying to implement a class evaluating the result of given scala script and input value and print the result of evaluation with scala.tools.nsc.interpreter.IMain. Here's my test code.
package some.test.package
import javax.script.{Compilable, CompiledScript}
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter.{IMain, JPrintWriter}
class Evaluator {
// Create IMain instance when created
val settings = new Settings()
settings.usejavacp.value = true
val writer = new JPrintWriter(Console.out, true)
val engine = new IMain(settings, writer)
val compiler = engine.asInstanceOf[Compilable]
var inputVal : Int = _
var compiledObj : CompiledScript = _
// compile given script
def compileScript(givenCode : String) {
compiledObj = compiler.compile(givenCode)
}
// evaluate
def evalCompiled(): Unit = {
compiledObj.eval()
}
// set input value
def setInput(givenInput:Int) {
inputVal = givenInput
}
// bind input variable
def bindInput() {
engine.bind("inputVal", "Int", inputVal)
}
}
object IMainTest {
def main(args:Array[String]): Unit = {
// create an instance
val evaluator = new Evaluator()
// first set input value to 3 and do evaluation
evaluator.setInput(3)
evaluator.bindInput()
evaluator.compileScript("def f(x:Int):Int=x+1; println(f(inputVal))")
evaluator.evalCompiled()
// let's change input value and re-evaluate
evaluator.setInput(5)
evaluator.evalCompiled()
}
}
My expectation was the program prints out '4' and '6' but the result was '4' and '4' printed out.
So, my question is...
Does IMain.bind just copies value not reference to some place that the engine can refer to?
Should I compile every time when I want to re-assign value of variable?
If the answer for 1 and 2 is 'YES', then is there any other way that I can achieve my purpose without re-bind and re-compile? I just expected re-assigning input value does not need additional bind operation or compile operation.
I think it's really hard to find IMain related examples or documents so any reference links might be also very helpful for me. Thank you.
Related
I have a pipeline with a set of PTransforms and my method is getting very long.
I'd like to write my DoFns and my composite transforms in a separate package and use them back in my main method. With python it's pretty straightforward, how can I achieve that with Scio? I don't see any example of doing that. :(
withFixedWindows(
FIXED_WINDOW_DURATION,
options = WindowOptions(
trigger = groupedWithinTrigger,
timestampCombiner = TimestampCombiner.END_OF_WINDOW,
accumulationMode = AccumulationMode.ACCUMULATING_FIRED_PANES,
allowedLateness = Duration.ZERO
)
)
.sumByKey
// How to write this in an another file and use it here?
.transform("Format Output") {
_
.withWindow[IntervalWindow]
.withTimestamp
}
If I understand your question correctly, you want to bundle your map, groupBy, ... transformations in a separate package, and use them in your main pipeline.
One way would be to use applyTransform, but then you would end up using PTransforms, which are not scala-friendly.
You can simply write a function that receives an SCollection and returns the transformed one, like:
def myTransform(input: SCollection[InputType]): Scollection[OutputType] = ???
But if you intend to write your own Source/Sink, take a look at the ScioIO class
You can use map function to map your elements example.
Instead of passing a lambda, you can pass a method reference from another class
Example .map(MyClass.MyFunction)
I think one way to solve this could be to define an object in another package and then create a method in that object that would have the logic required for your transformation. For example:
def main(cmdlineArgs: Array[String]): Unit = {
val (sc, args) = ContextAndArgs(cmdlineArgs)
val defaulTopic = "tweets"
val input = args.getOrElse("inputTopic", defaulTopic)
val output = args("outputTopic")
val inputStream: SCollection[Tweet] = sc.withName("read from pub sub").pubsubTopic(input)
.withName("map to tweet class").map(x => {parse(x).extract[Tweet]})
inputStream
.flatMap(sentiment.predict) // object sentiment with method predict
}
object sentiment {
def predict(tweet: Tweet): Option[List[TweetSentiment]] = {
val data = tweet.text
val emptyCase = Some("")
Some(data) match {
case `emptyCase` => None
case Some(v) => Some(entitySentimentFile(data)) // I used another method, //not defined
}
}
Please also this link for an example given in the Scio examples
I'm iterating over the command line input arguments in my Scala program and want to set several variables in a loop. My problem is that I'm using var's.
I don't want to use big libraries for this task, I want to keep it simple. The arguments contain simple key value pairs (like "-v myVar"), that's why I try to use the sliding function. Probably not the best choice for setting val's.
object Main {
def main(args: Array[String]): Unit = {
// args: Array("-c", "myFile", "-j", "update", "-d", "debugPath")
var config: String = null
var jobType: String = null
var debug: String = null
args.sliding(2,2).toList.collect {
case Array("-c", argProperty: String) => config = argProperty
case Array("-j", argJobType: String) => jobType = argJobType
case Array("-d", argDebug: String) => debug = argDebug
}
println("Config: " + config)
println("Job type: " + jobType)
println("Debug: " + debug)
}
}
The code compiles and delivers the correct output, but the 3 var's are not so nice. However, I could not find a solution for declaring 3 val's in a loop which are used outside the loop.
This code turns your arg list in to a Map from argument name to value:
val argMap = args.grouped(2).collect{ case Array(k, v) => k -> v }.toMap
val config = argMap.getOrElse("-c", defaultConfig)
This avoids using null and allows you to easily tell whether a parameter was supplied or not.
In scala, is there a way to access a class member,but the member name is a var?
Below is a code snippet, there is a class member called "one_star". I have a var whose value is "one_star", and I want use this var as the member name of the "case class".
case class Features(
// star about
var one_star: String = "0",
var two_star: String = "0",
var three_star: String = "0",
var four_star: String = "0",
var five_star: String = "0"
// other about
)
object Features {
def apply(): Features = {
val features = new Features()
var testVar = "one_star"
features.${testVar} = "1"
features
}
}
If you want to change field name dynamically, i.e. provide class variable name as value, find field that match given variable name and finally change the value for that field, there are several ways: the simple one is to use pattern match to check the field value and change instance value by yourself and return instance. However, it can be quite messy as you need to handle for every fields defined in your class and in case you have many fields, it can be quite cumbersome. Therefore, you will need some generic way to solve this problem.
Another approach is to use scala reflection. Scala reflection is designed for this, i.e. modifying your codes at runtime like your case and in more generic way. Following is a code snippet that change value of your instance for given field name.
case class Features(
// star about
var one_star: String = "0",
var two_star: String = "0",
var three_star: String = "0",
var four_star: String = "0"
// other about
) {
def copyInstance(variableName: String, value: String): Features = {
// 1. Initialize Features instance
val instance = Features()
// 2. Import scala reflection api.
import scala.reflect.runtime.{universe => ru}
// 3. Get the mirror of instance of Features class.
// Mirror will reflect to instance of Features case class, and we will use this instance mirror to modify its fields value.
val instanceMirror = ru.runtimeMirror(instance.getClass.getClassLoader).reflect(instance)
// 4. Now, Get the field whose value need to be changed - i.e. name you provide - variableName.
val field = ru.typeOf[Features].decl(ru.TermName(variableName)).asTerm
// 5. Get the mirror for that field so that we can read and write to this field.
val fieldMirror = instanceMirror.reflectField(field)
// 6. Finally, set the value to this field.
fieldMirror.set(value)
// 7. Return the changed instance.
instance
}
}
val features = Features()
val changedFeatures = features.copyInstance("one_star", "changed")
println(features)
println(changedFeatures)
//Result
Features(0,0,0,0)
Features(changed,0,0,0)
Also, note that you may need to handle the Exception for cases where invalid variable name and value is provided. In addition, if your case class contains >22 field parameters, certain features of case class are not available.
Scala is static type language and doesn't allow these language constructions. But you can use reflection (hard way) or pattern matching with code like this one (simple way):
class Features (
var one_star: String = "0",
var two_star: String = "0",
var three_star: String = "0",
var four_star: String = "0",
var five_star: String = "0") {
def setField(field: String, value: String): Unit = {
field match {
case "one_star" => one_star = value
case "two_star" => two_star = value
case "three_star" => three_star = value
case "four_star" => four_star = value
case "five_star" => five_star = value
}
}
}
This is possible using scala-reflect, although under most circumstances I would not recommend it.
import scala.reflect.runtime.universe._
val field = typeOf[Features].decl(TermName(testVar)).asTerm.accessed.asTerm
val mirror = runtimeMirror(classOf[Features].getClassLoader)
mirror.reflect(features).reflectField(field).set("1")
Are you sure you don't want to use or extend Map[String, String] for your class? So many properties is not typical.
I'm trying to write a class where when you call a function defined in the class, it will store it in an array of functions instead of executing it right away, then user calls exec() to execute it:
class TestA(val a: Int, newAction: Option[ArrayBuffer[(Int) => Int]]) {
val action: ArrayBuffer[(Int) => Int] = if (newAction.isEmpty) ArrayBuffer.empty[(Int) => Int] else newAction.get
def add(b: Int): TestA = {action += (a => a + b); new TestA(a, Some(action))}
def exec(): Int = {
var result = 0
action.foreach(r => result += r.apply(a))
result
}
def this(a:Int) = this(a, None)
}
Then this is my test code:
"delayed action" should "delay action till ready" in {
val test = new TestA(3)
val result = test.add(5).add(5)
println(result.exec())
}
This gives me a result of 16 because 3 was passed in twice and got added twice. I guess the easy way for me to solve this problem is to not pass in value for the second round, like change val a: Int to val a: Option[Int]. It helps but it doesn't solve my real problem: letting the second function know the result of the first execution.
Does anyone have a better solution to this?? Or if this is a pattern, can anyone share a tutorial of it?
Just save the result of the action in the 'result' variable (instatiate it with 'a') and use the previous result as input for the current iteration
def exec(): Int = {
var result = a
action.foreach(r => result = r.apply(result))
result
}
or use the more functional oriented solution that does the same
def exec(): Int = {
action.foldLeft(a)((r, f) => f.apply(r))
}
I have following code:
import java.io._
import com.twitter.chill.{Input, Output, ScalaKryoInstantiator}
import scala.reflect.ClassTag
object serializer {
val instantiator = new ScalaKryoInstantiator
instantiator.setRegistrationRequired(false)
val kryo = instantiator.newKryo()
def load[T](file:_=>_,name:String,cls:Class[T]):T = {
if (java.nio.file.Files.notExists(new File(name).toPath())) {
val temp = file
val baos = new FileOutputStream(name)
val output = new Output(baos, 4096)
kryo.writeObject(output, temp)
temp.asInstanceOf[T]
}
else {
println("loading from " + name)
val baos = new FileInputStream(name)
val input = new Input(baos)
kryo.readObject(input,cls)
}
}
}
I want to use it in this way:
val mylist = serializer.load((1 to 100000).toList,"allAdj.bin",classOf[List[Int]])
I don't want to run (1 to 100000).toList every time so I want to pass it to the serializer and then decide to compute it for the first time and serialize it for future or load it from file.
The problem is that the code block is running first in my code, how can I pass the code block without executing it?
P.S. Is there any scala tool that do the exact thing for me?
To have parameters not be evaluated before being passed, use pass-by-name, like this:
def method(param: =>ParamType)
Whatever you pass won't be evaluated at the time you pass, but will be evaluated each time you use param, which might not be what you want either. To have it be evaluated only the first time you use, do this:
def method(param: =>ParamType) = {
lazy val p: ParamType = param
Then use only p on the body. The first time p is used, param will be evaluated and the value will be stored. All other uses of p will use the stored value.
Note that this happens every time you invoke method. That is, if you call method twice, it won't use the "stored" value of p -- it will evaluate it again on first use. If you want to "pre-compute" something, then perhaps you'd be better off with a class instead?