What are the ways to convert a String into runnable code? - scala

I could not find how to convert a String into runnable code, for instance:
val i = "new String('Yo')"
// conversion
println(i)
should print
Yo
after the conversion.
I found the following example in another post:
import scala.tools.nsc.interpreter.ILoop
import java.io.StringReader
import java.io.StringWriter
import java.io.PrintWriter
import java.io.BufferedReader
import scala.tools.nsc.Settings
object FuncRunner extends App {
val line = "sin(2 * Pi * 400 * t)"
val lines = """import scala.math._
|var t = 1""".stripMargin
val in = new StringReader(lines + "\n" + line + "\nval f = (t: Int) => " + line)
val out = new StringWriter
val settings = new Settings
val looper = new ILoop(new BufferedReader(in), new PrintWriter(out))
val res = looper process settings
Console println s"[$res] $out"
}
link: How to convert a string from a text input into a function in a Scala
But it seems like scala.tools is not available anymore, and I'm a newbie in Scala so i could not figure out how to replace it.
And may be there are just other ways to do it now.
Thanks !

You can simple execute your code contained inside String using Quasiquotes(Experimental Module).
import scala.reflect.runtime.universe._
import scala.reflect.runtime.currentMirror
import scala.tools.reflect.ToolBox
// TO compile and run code we will use a ToolBox api.
val toolbox = currentMirror.mkToolBox()
// write your code starting with q and put it inside double quotes.
// NOTE : you will have to use triple quotes if you have any double quotes usage in your code.
val code1 = q"""new String("hello")"""
//compile and run your code.
val result1 = toolbox.compile(code1)()
// another example
val code2 = q"""
case class A(name:String,age:Int){
def f = (name,age)
}
val a = new A("Your Name",22)
a.f
"""
val result2 = toolbox.compile(code2)()
Output in REPL :
// Exiting paste mode, now interpreting.
import scala.reflect.runtime.universe._
import scala.reflect.runtime.currentMirror
import scala.tools.reflect.ToolBox
toolbox: scala.tools.reflect.ToolBox[reflect.runtime.universe.type] = scala.tools.reflect.ToolBoxFactory$ToolBoxImpl#69b34f89
code1: reflect.runtime.universe.Tree = new String("hello")
result1: Any = hello
code2: reflect.runtime.universe.Tree =
{
case class A extends scala.Product with scala.Serializable {
<caseaccessor> <paramaccessor> val name: String = _;
<caseaccessor> <paramaccessor> val age: Int = _;
def <init>(name: String, age: Int) = {
super.<init>();
()
};
def f = scala.Tuple2(name, age)
};
val a = new A("Your Name", 22);
a.f
}
result2: Any = (Your Name,22)
scala>
To learn more about Quasiquotes :
http://docs.scala-lang.org/overviews/quasiquotes/setup.html

I found a simple solution using the ToolBox tool :
val cm = universe.runtimeMirror(getClass.getClassLoader)
val tb = cm.mkToolBox()
val str = tb.eval(tb.parse("new String(\"Yo\")"))
println(str)
This is printing:
Yo

The scala compiler (and the "interpreter loop") are available here. For example:
https://github.com/scala/scala/blob/v2.11.7/src/repl/scala/tools/nsc/interpreter/ILoop.scala
A compiled jar that has that class will be in your scala distribution under lib\scala-compiler.jar.

One way could be to take/parse that string code and then write it yourself in some file as Scala code. This way it would be executed by Scala compiler when you will call it.
An example: just like Scala is doing with Java. It takes this code and then convert it into Java by making use of main method.
object Xyz extends App {
println ("Hello World!")
}

Related

How to run python code with args with python -c command from scala

I have python code, for example:
import sys
print('1')
print("2")
print(sys.argv[1])
Note that it can contain single, double quotes and accept arguments.
In my service I receive code by network and should execute it. I dont want to temporary save code as script on disk, so I want execute it with python -c command.
How I execute it in bash(https://stackoverflow.com/a/29565580/1996639):
$ python3 -c $'import sys\nprint(\'1\')\nprint("2")\nprint(sys.argv[1])' 3
1
2
3
My questions: how I can execute sample code from scala?
import sys.process._
def runCommand(cmd: String): (Int, String, String) = {
val stdoutStream = new ByteArrayOutputStream
val stderrStream = new ByteArrayOutputStream
val stdoutWriter = new PrintWriter(stdoutStream)
val stderrWriter = new PrintWriter(stderrStream)
val exitValue = cmd.!(ProcessLogger(stdoutWriter.println, stderrWriter.println))
stdoutWriter.close()
stderrWriter.close()
(exitValue, stdoutStream.toString, stderrStream.toString)
}
val code =
"""
|import sys
|
|print('1')
|print("2")
|print(sys.argv[1])
""".stripMargin
val arg = 3
val cmd = ???
val (exitCode, std, err) = runCommand(cmd)
You can take advantage of the fact that Scala's ProcessBuilder methods also accept Seq[String].
import sys.process._
object SO {
def runCommand(cmd: String*): (Int, String, String) = {
val stdoutStream = new java.io.ByteArrayOutputStream
val stderrStream = new java.io.ByteArrayOutputStream
val stdoutWriter = new java.io.PrintWriter(stdoutStream)
val stderrWriter = new java.io.PrintWriter(stderrStream)
val exitValue =
cmd.!(ProcessLogger(stdoutWriter.println, stderrWriter.println))
stdoutWriter.close()
stderrWriter.close()
(exitValue, stdoutStream.toString, stderrStream.toString)
}
def main(args: Array[String]): Unit = {
val code =
"""
|import sys
|
|print('1')
|print("2")
|print(sys.argv[1])
""".stripMargin
val arg = 3
println(runCommand("python3","-c", code, arg.toString))
}
}

Dynamic code evaluation in scala

What is the best way to inject a snippet of code to scala? something like eval in javascript and GroovyScriptEngine. I want to keep my rules/computations/formulas outside the actual data processing class. I have close to 100+ formulas to be executed. The data flow is same for all only the formulas change. What is the best way to do it in scala? and the number of formulas will grow over time.
You could use either scala-lang API for that or twitter-eval. Here is the snippet of a simple use case of scala-lang
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter.IMain
object ScalaReflectEvaluator {
def evaluate() = {
val clazz = prepareClass
val settings = new Settings
settings.usejavacp.value = true
settings.deprecation.value = true
val eval = new IMain(settings)
val evaluated = eval.interpret(clazz)
val res = eval.valueOfTerm("res0").get.asInstanceOf[Int]
println(res) //yields 9
}
private def prepareClass: String = {
s"""
|val x = 4
|val y = 5
|x + y
|""".stripMargin
}
}
or with twitter:
import com.twitter.util.Eval
object TwitterUtilEvaluator {
def evaluate() = {
val clazz = prepareClass
val eval = new Eval
eval.apply[Int](clazz)
}
private def prepareClass: String = {
s"""
|val x = 4
|val y = 5
|x + y
|""".stripMargin
}
}
I am not able to compile it at the moment to check whether I have missed something but you should get the idea.
I've found that scala.tools.reflect.ToolBox is the fastest eval in scala (measured interpreter, twitter's eval and custom tool). It's API:
import scala.reflect.runtime.universe
import scala.tools.reflect.ToolBox
val tb = universe.runtimeMirror(getClass.getClassLoader).mkToolBox()
tb.eval(tb.parse("""println("hello!")"""))

Need workaround for scala breeze matrix slicing and vector indexing

Because of the odd behaviour in method foo I cannot write methods like bar,
which I need:
import breeze.linalg.DenseMatrix
import breeze.linalg.DenseVector
class Test {
val dim = 3
val X:DenseMatrix[Double] = DenseMatrix.rand(dim,dim)
val u:DenseVector[Double] = DenseVector.fill(dim){1.0}
def foo:Unit = {
val i=0;
val row_i:DenseVector[Double] = X(i,::).t // OK
val s = u(i)+u(i) // OK
val j:Integer = 0
val row_j:DenseVector[Double] = X(j,::).t // does not compile (A)
val a = u(j)+u(j) // does not compile (B)
}
def bar(i:Integer):Double = u(i)+u(i) // does not compile (C)
}
Is there a workaround?
Thanks in advance for all replies.
Compilation errors:
(A) could not find implicit value for parameter canSlice:
breeze.linalg.support.CanSlice2[breeze.linalg.DenseMatrix[Double],Integer,collection.immutable.::.type,Result]
not enough arguments for method apply: (implicit canSlice:
breeze.linalg.support.CanSlice2[breeze.linalg.DenseMatrix[Double],Integer,collection.immutable.::.type,Result])
Result in trait TensorLike. Unspecified value parameter canSlice.
(B), (C)
could not find implicit value for parameter canSlice:
breeze.linalg.support.CanSlice[breeze.linalg.DenseVector[Double],Integer,Result]
not enough arguments for method apply: (implicit canSlice: breeze.linalg.support.CanSlice[breeze.linalg.DenseVector[Double],Integer,Result])Result
in trait TensorLike. Unspecified value parameter canSlice.
First off: convert your Integer to Int . That would take care of at least the first compilation error. Following update to your code does compile
import breeze.linalg.DenseMatrix
import breeze.linalg.DenseVector
import breeze.linalg.DenseMatrix
import breeze.linalg.DenseVector
class Test {
val dim = 3
val X:DenseMatrix[Double] = DenseMatrix.rand(dim,dim)
val u:DenseVector[Double] = DenseVector.fill(dim){1.0}
def foo:Unit = {
val i=0;
val row_i:DenseVector[Double] = X(i,::).t // OK
val s = u(i)+u(i) // OK
val j:Int = 0
val row_j:DenseVector[Double] = X(j,::).t // does not compile (A)
val a = u(j)+u(j) // does not compile (B)
}
def bar(i:Int):Double = u(i)+u(i) // does not compile (C)
}
From the repl:
// Exiting paste mode, now interpreting.
import breeze.linalg.DenseMatrix
import breeze.linalg.DenseVector
defined class Test
So your other errors also disappear for me (scala 2.11.8). Let me know if you have further issues.

Serializing to disk and deserializing Scala objects using Pickling

Given a stream of homogeneous typed object, how would I go about serializing them to binary, writing them to disk, reading them from disk and then deserializing them using Scala Pickling?
For example:
object PicklingIteratorExample extends App {
import scala.pickling.Defaults._
import scala.pickling.binary._
import scala.pickling.static._
case class Person(name: String, age: Int)
val personsIt = Iterator.from(0).take(10).map(i => Person(i.toString, i))
val pklsIt = personsIt.map(_.pickle)
??? // Write to disk
val readIt: Iterator[Person] = ??? // Read from disk and unpickle
}
I find a way to so for standard files:
object PickleIOExample extends App {
import scala.pickling.Defaults._
import scala.pickling.binary._
import scala.pickling.static._
val tempPath = File.createTempFile("pickling", ".gz").getAbsolutePath
val outputStream = new FileOutputStream(tempPath)
val inputStream = new FileInputStream(tempPath)
val persons = for{
i <- 1 to 100
} yield Person(i.toString, i)
val output = new StreamOutput(outputStream)
persons.foreach(_.pickleTo(output))
outputStream.close()
val personsIt = new Iterator[Person]{
val streamPickle = BinaryPickle(inputStream)
override def hasNext: Boolean = inputStream.available > 0
override def next(): Person = streamPickle.unpickle[Person]
}
println(personsIt.mkString(", "))
inputStream.close()
}
But I am still unable to find a solution that will work with gzipped files. Since I do not know how to detect the EOF? The following throws an EOFexception since GZIPInputStream available method does not indicate the EOF:
object PickleIOExample extends App {
import scala.pickling.Defaults._
import scala.pickling.binary._
import scala.pickling.static._
val tempPath = File.createTempFile("pickling", ".gz").getAbsolutePath
val outputStream = new GZIPOutputStream(new FileOutputStream(tempPath))
val inputStream = new GZIPInputStream(new FileInputStream(tempPath))
val persons = for{
i <- 1 to 100
} yield Person(i.toString, i)
val output = new StreamOutput(outputStream)
persons.foreach(_.pickleTo(output))
outputStream.close()
val personsIt = new Iterator[Person]{
val streamPickle = BinaryPickle(inputStream)
override def hasNext: Boolean = inputStream.available > 0
override def next(): Person = streamPickle.unpickle[Person]
}
println(personsIt.mkString(", "))
inputStream.close()
}

Creating serializable objects from Scala source code at runtime

To embed Scala as a "scripting language", I need to be able to compile text fragments to simple objects, such as Function0[Unit] that can be serialised to and deserialised from disk and which can be loaded into the current runtime and executed.
How would I go about this?
Say for example, my text fragment is (purely hypothetical):
Document.current.elements.headOption.foreach(_.open())
This might be wrapped into the following complete text:
package myapp.userscripts
import myapp.DSL._
object UserFunction1234 extends Function0[Unit] {
def apply(): Unit = {
Document.current.elements.headOption.foreach(_.open())
}
}
What comes next? Should I use IMain to compile this code? I don't want to use the normal interpreter mode, because the compilation should be "context-free" and not accumulate requests.
What I need to get hold off from the compilation is I guess the binary class file? In that case, serialisation is straight forward (byte array). How would I then load that class into the runtime and invoke the apply method?
What happens if the code compiles to multiple auxiliary classes? The example above contains a closure _.open(). How do I make sure I "package" all those auxiliary things into one object to serialize and class-load?
Note: Given that Scala 2.11 is imminent and the compiler API probably changed, I am happy to receive hints as how to approach this problem on Scala 2.11
Here is one idea: use a regular Scala compiler instance. Unfortunately it seems to require the use of hard disk files both for input and output. So we use temporary files for that. The output will be zipped up in a JAR which will be stored as a byte array (that would go into the hypothetical serialization process). We need a special class loader to retrieve the class again from the extracted JAR.
The following assumes Scala 2.10.3 with the scala-compiler library on the class path:
import scala.tools.nsc
import java.io._
import scala.annotation.tailrec
Wrapping user provided code in a function class with a synthetic name that will be incremented for each new fragment:
val packageName = "myapp"
var userCount = 0
def mkFunName(): String = {
val c = userCount
userCount += 1
s"Fun$c"
}
def wrapSource(source: String): (String, String) = {
val fun = mkFunName()
val code = s"""package $packageName
|
|class $fun extends Function0[Unit] {
| def apply(): Unit = {
| $source
| }
|}
|""".stripMargin
(fun, code)
}
A function to compile a source fragment and return the byte array of the resulting jar:
/** Compiles a source code consisting of a body which is wrapped in a `Function0`
* apply method, and returns the function's class name (without package) and the
* raw jar file produced in the compilation.
*/
def compile(source: String): (String, Array[Byte]) = {
val set = new nsc.Settings
val d = File.createTempFile("temp", ".out")
d.delete(); d.mkdir()
set.d.value = d.getPath
set.usejavacp.value = true
val compiler = new nsc.Global(set)
val f = File.createTempFile("temp", ".scala")
val out = new BufferedOutputStream(new FileOutputStream(f))
val (fun, code) = wrapSource(source)
out.write(code.getBytes("UTF-8"))
out.flush(); out.close()
val run = new compiler.Run()
run.compile(List(f.getPath))
f.delete()
val bytes = packJar(d)
deleteDir(d)
(fun, bytes)
}
def deleteDir(base: File): Unit = {
base.listFiles().foreach { f =>
if (f.isFile) f.delete()
else deleteDir(f)
}
base.delete()
}
Note: Doesn't handle compiler errors yet!
The packJar method uses the compiler output directory and produces an in-memory jar file from it:
// cf. http://stackoverflow.com/questions/1281229
def packJar(base: File): Array[Byte] = {
import java.util.jar._
val mf = new Manifest
mf.getMainAttributes.put(Attributes.Name.MANIFEST_VERSION, "1.0")
val bs = new java.io.ByteArrayOutputStream
val out = new JarOutputStream(bs, mf)
def add(prefix: String, f: File): Unit = {
val name0 = prefix + f.getName
val name = if (f.isDirectory) name0 + "/" else name0
val entry = new JarEntry(name)
entry.setTime(f.lastModified())
out.putNextEntry(entry)
if (f.isFile) {
val in = new BufferedInputStream(new FileInputStream(f))
try {
val buf = new Array[Byte](1024)
#tailrec def loop(): Unit = {
val count = in.read(buf)
if (count >= 0) {
out.write(buf, 0, count)
loop()
}
}
loop()
} finally {
in.close()
}
}
out.closeEntry()
if (f.isDirectory) f.listFiles.foreach(add(name, _))
}
base.listFiles().foreach(add("", _))
out.close()
bs.toByteArray
}
A utility function that takes the byte array found in deserialization and creates a map from class names to class byte code:
def unpackJar(bytes: Array[Byte]): Map[String, Array[Byte]] = {
import java.util.jar._
import scala.annotation.tailrec
val in = new JarInputStream(new ByteArrayInputStream(bytes))
val b = Map.newBuilder[String, Array[Byte]]
#tailrec def loop(): Unit = {
val entry = in.getNextJarEntry
if (entry != null) {
if (!entry.isDirectory) {
val name = entry.getName
// cf. http://stackoverflow.com/questions/8909743
val bs = new ByteArrayOutputStream
var i = 0
while (i >= 0) {
i = in.read()
if (i >= 0) bs.write(i)
}
val bytes = bs.toByteArray
b += mkClassName(name) -> bytes
}
loop()
}
}
loop()
in.close()
b.result()
}
def mkClassName(path: String): String = {
require(path.endsWith(".class"))
path.substring(0, path.length - 6).replace("/", ".")
}
A suitable class loader:
class MemoryClassLoader(map: Map[String, Array[Byte]]) extends ClassLoader {
override protected def findClass(name: String): Class[_] =
map.get(name).map { bytes =>
println(s"defineClass($name, ...)")
defineClass(name, bytes, 0, bytes.length)
} .getOrElse(super.findClass(name)) // throws exception
}
And a test case which contains additional classes (closures):
val exampleSource =
"""val xs = List("hello", "world")
|println(xs.map(_.capitalize).mkString(" "))
|""".stripMargin
def test(fun: String, cl: ClassLoader): Unit = {
val clName = s"$packageName.$fun"
println(s"Resolving class '$clName'...")
val clazz = Class.forName(clName, true, cl)
println("Instantiating...")
val x = clazz.newInstance().asInstanceOf[() => Unit]
println("Invoking 'apply':")
x()
}
locally {
println("Compiling...")
val (fun, bytes) = compile(exampleSource)
val map = unpackJar(bytes)
println("Classes found:")
map.keys.foreach(k => println(s" '$k'"))
val cl = new MemoryClassLoader(map)
test(fun, cl) // should call `defineClass`
test(fun, cl) // should find cached class
}