ProcessBuilder - Start another process / JVM in Scala - HowTo? - scala

I already handled to start another VM in Java.
See ProcessBuilder - Start another process / JVM - HowTo?
For some reason, I can't manage to do the same in Scala.
Here's my code
object NewProcTest {
def main(args :Array[String]) {
println("Main")
// val clazz = classOf[O3]
val clazz = O4.getClass
Proc.spawn(clazz, true)
println("fin")
}
}
object Proc{
def spawn(clazz :Class[_], redirectStream :Boolean) {
val separator = System.getProperty("file.separator")
val classpath = System.getProperty("java.class.path")
val path = System.getProperty("java.home") +
separator + "bin" + separator + "java"
val processBuilder =
new ProcessBuilder(path, "-cp",
classpath,
clazz.getCanonicalName())
processBuilder.redirectErrorStream(redirectStream)
val process = processBuilder.start()
process.waitFor()
System.out.println("Fin")
}
}
I've tried to define the main in an object and in class. Both within the same .scala file or within a separate one.
What am I doing wrong?

The issue seems to be that the class name for an object has a '$' suffix.
If you strip off that suffix, the Java invocation line triggered from ProcessBuilder works.
I've hacked something below to show a couple of test cases. I'm not yet sure yet why this is the case but at least it provides a workaround.
import java.io.{InputStreamReader, BufferedReader}
import System.{getProperty => Prop}
object O3 {def main(args: Array[String]) {println("hello from O3")}}
package package1 {
object O4 {def main(args: Array[String]) {println("hello from O4")}}
}
object NewProcTest {
val className1 = O3.getClass().getCanonicalName().dropRight(1)
val className2 = package1.O4.getClass().getCanonicalName().dropRight(1)
val sep = Prop("file.separator")
val classpath = Prop("java.class.path")
val path = Prop("java.home")+sep+"bin"+sep+"java"
println("className1 = " + className1)
println("className2 = " + className2)
def spawn(className: String,
redirectStream: Boolean) {
val processBuilder = new ProcessBuilder(path, "-cp", classpath, className)
val pbcmd = processBuilder.command().toString()
println("processBuilder = " + pbcmd)
processBuilder.redirectErrorStream(redirectStream)
val process = processBuilder.start()
val reader = new BufferedReader(new InputStreamReader(process.getInputStream()))
println(reader.readLine())
reader.close()
process.waitFor()
}
def main(args :Array[String]) {
println("start")
spawn(className1, false)
spawn(className2, false)
println("end")
}
}

Related

How to list out all the files in the public directory in a Play Framework 2.X Scala application?

Here is my controller
class Proguard extends Controller {
val proguardFolder = "/public/proguards/"
val proguardFolderFix = "/public/proguards"
val proguardSuffix = "proguard-"
val proguardExtension = ".pro"
val title = "# Created by https://www.proguard.io/api/%s\n\n%s"
def proguard(libraryName: String) = Action {
val libraries = libraryName.split(',')
val availableLibs = listInDir(proguardFolderFix)
val result = availableLibs.filter(libraries.contains).map(readFile).mkString
Ok(title.format(libraryName, result))
}
def list() = Action {
Ok(Json.toJson(listInDir(proguardFolder)))
}
private def listInDir(filePath: String): List[String] = {
getListOfFiles(Play.getFile(filePath)).map(_.getName.replace(proguardExtension, "").replace(proguardSuffix, ""))
}
def getListOfFiles(dir: File): List[File] = {
dir.listFiles.toList
}
def readFile(string: String): String = {
val source = scala.io.Source.fromFile(Play.getFile(s"$proguardFolder$proguardSuffix$string$proguardExtension"))
val lines = try source.mkString finally source.close()
lines
}
}
It worked totally okay in debug mode, but in production at Heroku dir.listFiles. is giving me NPE
I've tried different ways, but looks like only solution is move my files to s3 or database.

What are the ways to convert a String into runnable code?

I could not find how to convert a String into runnable code, for instance:
val i = "new String('Yo')"
// conversion
println(i)
should print
Yo
after the conversion.
I found the following example in another post:
import scala.tools.nsc.interpreter.ILoop
import java.io.StringReader
import java.io.StringWriter
import java.io.PrintWriter
import java.io.BufferedReader
import scala.tools.nsc.Settings
object FuncRunner extends App {
val line = "sin(2 * Pi * 400 * t)"
val lines = """import scala.math._
|var t = 1""".stripMargin
val in = new StringReader(lines + "\n" + line + "\nval f = (t: Int) => " + line)
val out = new StringWriter
val settings = new Settings
val looper = new ILoop(new BufferedReader(in), new PrintWriter(out))
val res = looper process settings
Console println s"[$res] $out"
}
link: How to convert a string from a text input into a function in a Scala
But it seems like scala.tools is not available anymore, and I'm a newbie in Scala so i could not figure out how to replace it.
And may be there are just other ways to do it now.
Thanks !
You can simple execute your code contained inside String using Quasiquotes(Experimental Module).
import scala.reflect.runtime.universe._
import scala.reflect.runtime.currentMirror
import scala.tools.reflect.ToolBox
// TO compile and run code we will use a ToolBox api.
val toolbox = currentMirror.mkToolBox()
// write your code starting with q and put it inside double quotes.
// NOTE : you will have to use triple quotes if you have any double quotes usage in your code.
val code1 = q"""new String("hello")"""
//compile and run your code.
val result1 = toolbox.compile(code1)()
// another example
val code2 = q"""
case class A(name:String,age:Int){
def f = (name,age)
}
val a = new A("Your Name",22)
a.f
"""
val result2 = toolbox.compile(code2)()
Output in REPL :
// Exiting paste mode, now interpreting.
import scala.reflect.runtime.universe._
import scala.reflect.runtime.currentMirror
import scala.tools.reflect.ToolBox
toolbox: scala.tools.reflect.ToolBox[reflect.runtime.universe.type] = scala.tools.reflect.ToolBoxFactory$ToolBoxImpl#69b34f89
code1: reflect.runtime.universe.Tree = new String("hello")
result1: Any = hello
code2: reflect.runtime.universe.Tree =
{
case class A extends scala.Product with scala.Serializable {
<caseaccessor> <paramaccessor> val name: String = _;
<caseaccessor> <paramaccessor> val age: Int = _;
def <init>(name: String, age: Int) = {
super.<init>();
()
};
def f = scala.Tuple2(name, age)
};
val a = new A("Your Name", 22);
a.f
}
result2: Any = (Your Name,22)
scala>
To learn more about Quasiquotes :
http://docs.scala-lang.org/overviews/quasiquotes/setup.html
I found a simple solution using the ToolBox tool :
val cm = universe.runtimeMirror(getClass.getClassLoader)
val tb = cm.mkToolBox()
val str = tb.eval(tb.parse("new String(\"Yo\")"))
println(str)
This is printing:
Yo
The scala compiler (and the "interpreter loop") are available here. For example:
https://github.com/scala/scala/blob/v2.11.7/src/repl/scala/tools/nsc/interpreter/ILoop.scala
A compiled jar that has that class will be in your scala distribution under lib\scala-compiler.jar.
One way could be to take/parse that string code and then write it yourself in some file as Scala code. This way it would be executed by Scala compiler when you will call it.
An example: just like Scala is doing with Java. It takes this code and then convert it into Java by making use of main method.
object Xyz extends App {
println ("Hello World!")
}

Serializing to disk and deserializing Scala objects using Pickling

Given a stream of homogeneous typed object, how would I go about serializing them to binary, writing them to disk, reading them from disk and then deserializing them using Scala Pickling?
For example:
object PicklingIteratorExample extends App {
import scala.pickling.Defaults._
import scala.pickling.binary._
import scala.pickling.static._
case class Person(name: String, age: Int)
val personsIt = Iterator.from(0).take(10).map(i => Person(i.toString, i))
val pklsIt = personsIt.map(_.pickle)
??? // Write to disk
val readIt: Iterator[Person] = ??? // Read from disk and unpickle
}
I find a way to so for standard files:
object PickleIOExample extends App {
import scala.pickling.Defaults._
import scala.pickling.binary._
import scala.pickling.static._
val tempPath = File.createTempFile("pickling", ".gz").getAbsolutePath
val outputStream = new FileOutputStream(tempPath)
val inputStream = new FileInputStream(tempPath)
val persons = for{
i <- 1 to 100
} yield Person(i.toString, i)
val output = new StreamOutput(outputStream)
persons.foreach(_.pickleTo(output))
outputStream.close()
val personsIt = new Iterator[Person]{
val streamPickle = BinaryPickle(inputStream)
override def hasNext: Boolean = inputStream.available > 0
override def next(): Person = streamPickle.unpickle[Person]
}
println(personsIt.mkString(", "))
inputStream.close()
}
But I am still unable to find a solution that will work with gzipped files. Since I do not know how to detect the EOF? The following throws an EOFexception since GZIPInputStream available method does not indicate the EOF:
object PickleIOExample extends App {
import scala.pickling.Defaults._
import scala.pickling.binary._
import scala.pickling.static._
val tempPath = File.createTempFile("pickling", ".gz").getAbsolutePath
val outputStream = new GZIPOutputStream(new FileOutputStream(tempPath))
val inputStream = new GZIPInputStream(new FileInputStream(tempPath))
val persons = for{
i <- 1 to 100
} yield Person(i.toString, i)
val output = new StreamOutput(outputStream)
persons.foreach(_.pickleTo(output))
outputStream.close()
val personsIt = new Iterator[Person]{
val streamPickle = BinaryPickle(inputStream)
override def hasNext: Boolean = inputStream.available > 0
override def next(): Person = streamPickle.unpickle[Person]
}
println(personsIt.mkString(", "))
inputStream.close()
}

Creating serializable objects from Scala source code at runtime

To embed Scala as a "scripting language", I need to be able to compile text fragments to simple objects, such as Function0[Unit] that can be serialised to and deserialised from disk and which can be loaded into the current runtime and executed.
How would I go about this?
Say for example, my text fragment is (purely hypothetical):
Document.current.elements.headOption.foreach(_.open())
This might be wrapped into the following complete text:
package myapp.userscripts
import myapp.DSL._
object UserFunction1234 extends Function0[Unit] {
def apply(): Unit = {
Document.current.elements.headOption.foreach(_.open())
}
}
What comes next? Should I use IMain to compile this code? I don't want to use the normal interpreter mode, because the compilation should be "context-free" and not accumulate requests.
What I need to get hold off from the compilation is I guess the binary class file? In that case, serialisation is straight forward (byte array). How would I then load that class into the runtime and invoke the apply method?
What happens if the code compiles to multiple auxiliary classes? The example above contains a closure _.open(). How do I make sure I "package" all those auxiliary things into one object to serialize and class-load?
Note: Given that Scala 2.11 is imminent and the compiler API probably changed, I am happy to receive hints as how to approach this problem on Scala 2.11
Here is one idea: use a regular Scala compiler instance. Unfortunately it seems to require the use of hard disk files both for input and output. So we use temporary files for that. The output will be zipped up in a JAR which will be stored as a byte array (that would go into the hypothetical serialization process). We need a special class loader to retrieve the class again from the extracted JAR.
The following assumes Scala 2.10.3 with the scala-compiler library on the class path:
import scala.tools.nsc
import java.io._
import scala.annotation.tailrec
Wrapping user provided code in a function class with a synthetic name that will be incremented for each new fragment:
val packageName = "myapp"
var userCount = 0
def mkFunName(): String = {
val c = userCount
userCount += 1
s"Fun$c"
}
def wrapSource(source: String): (String, String) = {
val fun = mkFunName()
val code = s"""package $packageName
|
|class $fun extends Function0[Unit] {
| def apply(): Unit = {
| $source
| }
|}
|""".stripMargin
(fun, code)
}
A function to compile a source fragment and return the byte array of the resulting jar:
/** Compiles a source code consisting of a body which is wrapped in a `Function0`
* apply method, and returns the function's class name (without package) and the
* raw jar file produced in the compilation.
*/
def compile(source: String): (String, Array[Byte]) = {
val set = new nsc.Settings
val d = File.createTempFile("temp", ".out")
d.delete(); d.mkdir()
set.d.value = d.getPath
set.usejavacp.value = true
val compiler = new nsc.Global(set)
val f = File.createTempFile("temp", ".scala")
val out = new BufferedOutputStream(new FileOutputStream(f))
val (fun, code) = wrapSource(source)
out.write(code.getBytes("UTF-8"))
out.flush(); out.close()
val run = new compiler.Run()
run.compile(List(f.getPath))
f.delete()
val bytes = packJar(d)
deleteDir(d)
(fun, bytes)
}
def deleteDir(base: File): Unit = {
base.listFiles().foreach { f =>
if (f.isFile) f.delete()
else deleteDir(f)
}
base.delete()
}
Note: Doesn't handle compiler errors yet!
The packJar method uses the compiler output directory and produces an in-memory jar file from it:
// cf. http://stackoverflow.com/questions/1281229
def packJar(base: File): Array[Byte] = {
import java.util.jar._
val mf = new Manifest
mf.getMainAttributes.put(Attributes.Name.MANIFEST_VERSION, "1.0")
val bs = new java.io.ByteArrayOutputStream
val out = new JarOutputStream(bs, mf)
def add(prefix: String, f: File): Unit = {
val name0 = prefix + f.getName
val name = if (f.isDirectory) name0 + "/" else name0
val entry = new JarEntry(name)
entry.setTime(f.lastModified())
out.putNextEntry(entry)
if (f.isFile) {
val in = new BufferedInputStream(new FileInputStream(f))
try {
val buf = new Array[Byte](1024)
#tailrec def loop(): Unit = {
val count = in.read(buf)
if (count >= 0) {
out.write(buf, 0, count)
loop()
}
}
loop()
} finally {
in.close()
}
}
out.closeEntry()
if (f.isDirectory) f.listFiles.foreach(add(name, _))
}
base.listFiles().foreach(add("", _))
out.close()
bs.toByteArray
}
A utility function that takes the byte array found in deserialization and creates a map from class names to class byte code:
def unpackJar(bytes: Array[Byte]): Map[String, Array[Byte]] = {
import java.util.jar._
import scala.annotation.tailrec
val in = new JarInputStream(new ByteArrayInputStream(bytes))
val b = Map.newBuilder[String, Array[Byte]]
#tailrec def loop(): Unit = {
val entry = in.getNextJarEntry
if (entry != null) {
if (!entry.isDirectory) {
val name = entry.getName
// cf. http://stackoverflow.com/questions/8909743
val bs = new ByteArrayOutputStream
var i = 0
while (i >= 0) {
i = in.read()
if (i >= 0) bs.write(i)
}
val bytes = bs.toByteArray
b += mkClassName(name) -> bytes
}
loop()
}
}
loop()
in.close()
b.result()
}
def mkClassName(path: String): String = {
require(path.endsWith(".class"))
path.substring(0, path.length - 6).replace("/", ".")
}
A suitable class loader:
class MemoryClassLoader(map: Map[String, Array[Byte]]) extends ClassLoader {
override protected def findClass(name: String): Class[_] =
map.get(name).map { bytes =>
println(s"defineClass($name, ...)")
defineClass(name, bytes, 0, bytes.length)
} .getOrElse(super.findClass(name)) // throws exception
}
And a test case which contains additional classes (closures):
val exampleSource =
"""val xs = List("hello", "world")
|println(xs.map(_.capitalize).mkString(" "))
|""".stripMargin
def test(fun: String, cl: ClassLoader): Unit = {
val clName = s"$packageName.$fun"
println(s"Resolving class '$clName'...")
val clazz = Class.forName(clName, true, cl)
println("Instantiating...")
val x = clazz.newInstance().asInstanceOf[() => Unit]
println("Invoking 'apply':")
x()
}
locally {
println("Compiling...")
val (fun, bytes) = compile(exampleSource)
val map = unpackJar(bytes)
println("Classes found:")
map.keys.foreach(k => println(s" '$k'"))
val cl = new MemoryClassLoader(map)
test(fun, cl) // should call `defineClass`
test(fun, cl) // should find cached class
}

Scala run time code compilation

I am trying to port my java code to pure scala, so would appreciate any help in this regard.
The code below works by first translating my business logic into java code. Here I'm using freemarker template to do template based code generation. After file creation I then use the java comiler to compile the code and create a jar file, which is persisted in a temporary directory
I am currently using the javax.tools.* package that provides runtime compilation. What is the Scala equiavalent to that approach? I'd like to generate pure Scala code using freemarker templates and then run Scala compilation to create a jar file.
Below is sample Java code I am using to make this happen.
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
Iterable<? extends JavaFileObject> compilationUnits = Arrays.asList(javaFileObjects);
StringBuilder builder = new StringBuilder();
builder.append(service.getConfig().getProp("coreLib"));
builder.append(";" +result.getCodeContext().getOmClasspath());
builder.append(";" +jarBuilder.toString());
builder.append(";" +service.getConfig().getProp("tempCodeGen"));
String[] compileOptions = new String[]{"-d", result.getCodeContext().getOmClasspath(),"-cp",builder.toString()} ;
Iterable<String> compilationOptionss = Arrays.asList(compileOptions);
DiagnosticCollector<JavaFileObject> diagnostics = new DiagnosticCollector<JavaFileObject>();
CompilationTask compilerTask = compiler.getTask(null, stdFileManager, diagnostics, compilationOptionss, null, compilationUnits) ;
boolean status = compilerTask.call();
Here's some methods from my own code to compile a project and package it into a jar. Its very far from polished, or properly commented, but hopefully it will indicate where you need to start. I don't think you need to use String Builder as this is not performance critical:
def buildAll(name: String, projDir: String, mainClass: String = ""): Unit =
{
import scala.tools.nsc.{Settings,Global}
val relSrc : List[String] = List()
val maniVersion = "None"
def targDir: String = projDir + "/targ"
def srcDir: String = projDir + "/src"
def srcDirs: List[String] = srcDir :: relSrc
import java.io._
val sings = new scala.tools.nsc.Settings
new File(targDir).mkdir
sings.outputDirs.setSingleOutput(targDir.toString)
val comp = new Global(sings)
val crun: comp.Run = new comp.Run
def getList(fName: String): List[String] =
{
println("starting getList " + fName)
val file = new File(fName)
if (file.isDirectory) file.listFiles.toList.flatMap(i => getList(fName + "/" + i.getName))
else List(fName)
}
crun.compile(srcDirs.flatMap(i => getList(i)))
import sys.process._
("cp -r /sdat/projects/ScalaLibs/scala " + targDir + "/scala").!
import java.util.jar._
val manif = new Manifest
val mf = manif.getMainAttributes
mf.put(Attributes.Name.MANIFEST_VERSION, maniVersion)
if (mainClass != "") mf.put(Attributes.Name.MAIN_CLASS, mainClass)
val jarName = name + ".jar"
val jarOut: JarOutputStream = new JarOutputStream(new FileOutputStream(projDir + "/" + jarName), manif)
AddAllToJar(targDir, jarOut)
jarOut.close
}
def addToJar(jarOut: JarOutputStream, file: File, reldir: String): Unit =
{
val fName = reldir + file.getName
val fNameMod = if (file.isDirectory) fName + "/" else fName
val entry = new JarEntry(fNameMod)
entry.setTime(file.lastModified)
jarOut.putNextEntry(entry)
if (file.isDirectory)
{
jarOut.closeEntry
file.listFiles.foreach(i => addToJar(jarOut, i, fName + "/"))
}
else
{
var buf = new Array[Byte](1024)
val in = new FileInputStream(file)
Stream.continually(in.read(buf)).takeWhile(_ != -1).foreach(jarOut.write(buf, 0, _))
in.close
jarOut.closeEntry()
}
}
def AddAllToJar(targDir: String, jarOut: JarOutputStream): Unit =
new java.io.File(targDir).listFiles.foreach(i => addToJar(jarOut, i, ""))
You need to add the Scala Compiler to the build path. The Scala compiler takes a list of sourcefiles and produces the compiled class files in the directory set in the output directory. Getting to grips with the full capabilities of the compiler is a major task though.
And when using scala you don't need to use any free-marker similar tools. Since scala version 2.10 there is string interpolation feature.
val traitName = "MyTrait"
val packageName = "my.pack"
val typeParams = List("A", "B", "C")
s"""
|package ${packageName}
|
|trait ${traitName}[${typeParams.mkString(",")}] {
| ${typeParams.map(t => s"val ${t.toLowerCase()}: ${t}")}
|}
|
""".stripMargin
will yield:
package my.pack
trait MyTrait[A,B,C] {
List(val a: A, val b: B, val c: C)
}
No dependencies needed :)
If you would like to compile generated code in runtime, the simplest solution is twitter eval utility
See the suggestion here - Generating a class from string and instantiating it in Scala 2.10
Or the Eval class from twitter's util library -
https://github.com/twitter/util/blob/master/util-eval/src/main/scala/com/twitter/util/Eval.scala