I can use Python to control gnuplot to print out plots in an interactive way as follows:
p = Popen(["/usr/local/bin/gnuplot"], shell=False, stdin=PIPE, stdout=PIPE)
p.stdin.write(r'set terminal gif;')
...
out, err = p.communicate()
How can I do the same thing with Scala? I have some skeleton code, but I'm not sure exactly how to fill in the missing gaps.
val gnuplot = "/usr/local/bin/gnuplot"
val pb = Process(gnuplot)
val pio = new ProcessIO(_ => (),
stdout => ...,
_ => ())
pb.run(pio)
This code works fine, get the inputStream, and write gnuplot commands with the stream:
def plot(): Unit = {
val inputStream = new SyncVar[OutputStream];
val gnuplot = "/usr/local/bin/gnuplot"
val pb = Process(gnuplot)
val pio = new ProcessIO(stdin => inputStream.put(stdin),
stdout => Source.fromInputStream(stdout).getLines.foreach(println),
stderr => Source.fromInputStream(stderr).getLines.foreach(println));
pb.run(pio)
val a = Array("set terminal gif", "set output \"hello.gif\"", "plot [-3.14:3.14] sin(x)").foreach { s =>
inputStream.get.write((s + "\n").getBytes)
}
inputStream.get.close()
}
It was non-trivial to rejigger your example:
def plot2(): Unit = {
val done = new CountDownLatch(1)
val cmds = List(
"set terminal gif",
"""set output "hello2.gif"""",
"plot [-3.14:3.14] sin(x)",
"exit"
)
val gnuplot = "/usr/bin/gnuplot"
val pb = Process(gnuplot)
val pio = BasicIO standard { out =>
for (c <- cmds) {
Console println s">$c"
out.write(s"$c\n".getBytes)
}
try out.close()
finally done.countDown()
}
val p = pb run pio
if (done.await(10, Seconds)) Console println s"Exited ${p.exitValue}"
else {
Console println "Stuck..."
p.destroy()
}
}
I don't know why this API feels so complicated.
It seems they'll want to add p.waitFor with timeout from Java 8.
Related
How can I get some kind of writeable stream connected to stdin (and also readable streams connected to stdout and stderr) when launching a process via scala.sys.process library? Here's the code that doesn't work (doesn't even print debug messages)
val p = Process("wc -l")
val io = BasicIO.standard(true)
val lines = Seq("a", "b", "c") mkString "\n"
val buf = lines.getBytes(StandardCharsets.UTF_8)
io withInput { w =>
println("Writing")
w.write(buf)
}
io withOutput { i =>
val s = new BufferedReader(new InputStreamReader(i)).readLine()
println(s"Output is $s")
}
You have a couple of problems.
First in your snippet you never connect your process with the io and never run it.
That can be done like this: p run io.
Second, the withInput & withOutput methods return a NEW ProcessIO they DON'T mutate the actual, and since you don't assign the return of those calls to a variable, you are doing nothing.
The following snippet fixes both problems, hope it works for you.
import scala.io.Source
import scala.sys.process._
import java.nio.charset.StandardCharsets
val p = Process("wc -l")
val io =
BasicIO.standard(true)
.withInput { w =>
val lines = Seq("a", "b", "c").mkString("", "\n", "\n")
val buf = lines.getBytes(StandardCharsets.UTF_8)
println("Writing")
w.write(buf)
w.close()
}
.withOutput { i =>
val s = Source.fromInputStream(i)
println(s"Output is ${s.getLines.mkString(",")}")
i.close()
}
p run io
Don't doubt to ask for clarification.
PS: it prints "Output is 3" - (Thanks to Dima for pointing the mistake).
I am trying to build a class that starts a system process which waits for stdin. The class should have another method which takes a string, inputs that into the system process, and return the process' output.
The reason is that starting the process involves loading a lot of data and hence takes a while.
I am trying to dummy-test this with bc, so that bc is started and waits for input. I would envision an interface like this:
case class BcWrapper(executable: File) {
var bc: Option[???] = None
def startBc(): Unit = bc = Some(???)
def calc(input: String): String = bc.get.???
def stopBc(): Unit = bc.get.???
}
I would like to be able to use it like this:
val wrapper = BcWrapper(new File("/usr/bin/bc"))
wrapper.startBc()
val result1 = wrapper.calc("1 + 1") // should be "2"
val result2 = wrapper.calc(???)
[...]
wrapper.stopBc()
This topic has been touched in multiple questions, but never fully answered for a use case like this one. This question or this one seems to come close. However, I am not sure how to implement the ProcessLogger, nor whether to use one in the first place.
Unfortunately, the Scala documentation is not very elaborate either.
Note that I do not want to read from stdin, but want to call a function.
The background is that I want to read a large file, read it line by line, preprocess the lines, pass them to the external process, and post-process the output.
You can get something similar, but simpler, like so.
import sys.process._
import util.Try
class StdInReader(val reader :String) {
def send(input :String) :Try[String] =
Try(s"/bin/echo $input".#|(reader).!!.trim)
}
usage:
val bc = new StdInReader("/usr/bin/bc")
bc.send("2 * 8") //res0: scala.util.Try[String] = Success(16)
bc.send("12 + 8") //res1: scala.util.Try[String] = Success(20)
bc.send("22 - 8") //res2: scala.util.Try[String] = Success(14)
Programs that send a non-zero exit-code (bc doesn't) will result with a Failure().
If you need more fine-grained control you might start with something like this and expand on it.
import sys.process._
class ProcHandler(val cmnd :String) {
private val resbuf = collection.mutable.Buffer.empty[String]
def run(data :Seq[String]) :Unit = {
cmnd.run(new ProcessIO(
in => {
val writer = new java.io.PrintWriter(in)
data.foreach(writer.println)
writer.close()
},
out => {
val src = io.Source.fromInputStream(out)
src.getLines().foreach(resbuf += _)
src.close()
},
_.close() //maybe create separate buffer for stderr?
)).exitValue()
}
def results() :Seq[String] = {
val rs = collection.mutable.Buffer.empty[String]
resbuf.copyToBuffer(rs)
resbuf.clear()
rs
}
}
usage:
val bc = new ProcHandler("/usr/bin/bc")
bc.run(List("4+5","6-2","2*5"))
bc.run(List("99/3","11*77"))
bc.results() //res0: Seq[String] = ArrayBuffer(9, 4, 10, 33, 847)
OK, I did some more research and found this. It appears to get at what you want but there are limitations. In particular, the process stays open for input until you want to get output. At that point IO streams are closed to insure all buffers are flushed.
import sys.process._
import util.Try
class ProcHandler(val cmnd :String) {
private val procInput = new java.io.PipedOutputStream()
private val procOutput = new java.io.PipedInputStream()
private val proc = cmnd.run( new ProcessIO(
{ in => // attach to the process's internal input stream
val istream = new java.io.PipedInputStream(procInput)
val buf = Array.fill(100)(0.toByte)
Iterator.iterate(istream.read(buf)){ br =>
in.write(buf, 0, br)
istream.read(buf)
}.takeWhile(_>=0).toList
in.close()
},
{ out => // attach to the process's internal output stream
val ostream = new java.io.PipedOutputStream(procOutput)
val buf = Array.fill(100)(0.toByte)
Iterator.iterate(out.read(buf)){ br =>
ostream.write(buf, 0, br)
out.read(buf)
}.takeWhile(_>=0).toList
out.close()
},
_ => () // ignore stderr
))
private val procO = new java.io.BufferedReader(new java.io.InputStreamReader(procOutput))
private val procI = new java.io.PrintWriter(procInput, true)
def feed(str :String) :Unit = procI.println(str)
def feed(ss :Seq[String]) :Unit = ss.foreach(procI.println)
def read() :List[String] = {
procI.close() //close input before reading output
val lines = Stream.iterate(Try(procO.readLine)){_ =>
Try(procO.readLine)
}.takeWhile(_.isSuccess).map(_.get).toList
procO.close()
lines
}
}
usage:
val bc = new ProcHandler("/usr/bin/bc")
bc.feed(List("9*3","4+11")) //res0: Unit = ()
bc.feed("4*13") //res1: Unit = ()
bc.read() //res2: List[String] = List(27, 15, 52)
bc.read() //res3: List[String] = List()
OK, this is my final word on the subject. I think this ticks every item on your wish list: start the process only once, it stays alive until actively closed, allows alternating the writing and reading.
import sys.process._
class ProcHandler(val cmnd :Seq[String]) {
private var os: java.io.OutputStream = null
private var is: java.io.InputStream = null
private val pio = new ProcessIO(os = _, is = _, _.close())
private val proc = cmnd.run(pio)
def feed(ss :String*) :Unit = {
ss.foreach(_.foreach(os.write(_)))
os.flush()
}
def ready :Boolean = is.available() > 0
def read() :String = {
Seq.fill[Char](is.available())(is.read().toChar).mkString
}
def close() :Unit = {
proc.exitValue()
os.close()
is.close()
}
}
There are still issues and much room for improvement. IO is handled at a basic level (streams) and I'm not sure what I'm doing here is completely safe and correct. The input, feed(), is required to supply the necessary NewLine terminations, and the output, read(), is just a raw String and not separated into a nice collection of string results.
Note that this will bleed system resources if the client code fails to close() all processes.
Note also that reading doesn't wait for content (i.e. no blocking). After writing the response might not be immediately available.
usage:
val bc = new ProcHandler(Seq("/usr/bin/bc","-q"))
bc.feed("44-21\n", "21*4\n")
bc.feed("67+11\n")
if (bc.ready) bc.read() else "not ready" // "23\n84\n78\n"
bc.feed("67-11\n")
if (bc.ready) bc.read() else "not ready" // "56\n"
bc.feed("67*11\n", "1+2\n")
if (bc.ready) bc.read() else "not ready" // "737\n3\n"
if (bc.ready) bc.read() else "not ready" // "not ready"
bc.close()
I start and have running a Scala process.
val dir = "/path/to/working/dir/"
val stockfish = Process(Seq("wine", dir + "stockfish_8_x32.exe"))
val logger = ProcessLogger(printf("Stdout: %s%n", _))
val stockfishProcess = stockfish.run(logger, connectInput = true)
The process reads from and writes to standard IO (console). How can I send a string command to the process if it's been already started?
Scala process API has ProcessBuilder which has in turn bunch of useful methods. But ProcessBuilder is used before a process starts to compose complex shell commands. Also Scala has ProcessIO to handle input or output. I don't need it too. I just need to send message to my process.
In Java I would do something like this.
String dir = "/path/to/working/dir/";
ProcessBuilder builder = new ProcessBuilder("wine", dir + "stockfish_8_x32.exe");
Process process = builder.start();
OutputStream stdin = process.getOutputStream();
InputStream stdout = process.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(stdout));
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(stdin));
new Thread(() -> {
try {
String line;
while ((line = reader.readLine()) != null) {
System.out.println("Stdout: " + line);
}
} catch (IOException e) {
e.printStackTrace();
}
}).start();
Thread.sleep(5000); // it's just for example
writer.write("quit"); // send to the process command to stop working
writer.newLine();
writer.flush();
It works quite well. I start my process, get InputStream and OutputStream from it, and use the streams to interact with the process.
It appears Scala Process trait provides no ways to write to it. ProcessBuilder is useless after process run. And ProcessIO is just for IO catching and handling.
Are there any ways to write to Scala running process?
UPDATE:
I don't see how I may use ProcessIO to pass a string to running process.
I did the following.
import scala.io.Source
import scala.sys.process._
object Sample extends App {
def out = (output: java.io.OutputStream) => {
output.flush()
output.close()
}
def in = (input: java.io.InputStream) => {
println("Stdout: " + Source.fromInputStream(input).mkString)
input.close()
}
def go = {
val dir = "/path/to/working/dir/"
val stockfishSeq = Seq("wine", dir + "/stockfish_8_x32.exe")
val pio = new ProcessIO(out, in, err => {})
val stockfish = Process(stockfishSeq)
stockfish.run(pio)
Thread.sleep(5000)
System.out.write("quit\n".getBytes)
pio.writeInput(System.out) // "writeInput" is function "out" which I have passed to conforming ProcessIO instance. I can invoke it from here. It takes OutputStream but where can I obtain it? Here I just pass System.out for example.
}
go
}
Of course it does not work and I failed to understand how to implement functionality as in my Java snippet above. It would be great to have advice or snippet of Scala code clearing my issue.
I think the documentation around Scala processes (specifically the usage and semantics of ProcessIO) could use some improvement. The first time I tried using this API, I also found it very confusing, and it took some trial and error to get my subprocess i/o working correctly.
I think seeing a simple example is probably all you really need. I'll do something really simple: invoking bc as a subprocess to do some trivial computations, and then printing the answers to my stdout. My goal is to do something like this (but from Scala rather than from my shell):
$ printf "1+2\n3+4\n" | bc
3
7
Here's how I'd do it in Scala:
import scala.io.Source
import scala.sys.process._
object SimpleProcessExample extends App {
def out = (output: java.io.OutputStream) => {
output.flush()
output.close()
}
def in = (input: java.io.InputStream) => {
println("Stdout: " + Source.fromInputStream(input).mkString)
input.close()
}
// limit scope of any temporary variables
locally {
val calcCommand = "bc"
// strings are implicitly converted to ProcessBuilder
// via scala.sys.process.ProcessImplicits.stringToProcess(_)
val calcProc = calcCommand.run(new ProcessIO(
// Handle subprocess's stdin
// (which we write via an OutputStream)
in => {
val writer = new java.io.PrintWriter(in)
writer.println("1 + 2")
writer.println("3 + 4")
writer.close()
},
// Handle subprocess's stdout
// (which we read via an InputStream)
out => {
val src = scala.io.Source.fromInputStream(out)
for (line <- src.getLines()) {
println("Answer: " + line)
}
src.close()
},
// We don't want to use stderr, so just close it.
_.close()
))
// Using ProcessBuilder.run() will automatically launch
// a new thread for the input/output routines passed to ProcessIO.
// We just need to wait for it to finish.
val code = calcProc.exitValue()
println(s"Subprocess exited with code $code.")
}
}
Notice that you don't actually call any of the methods of the ProcessIO object directly because they're automatically called by the ProcessBuilder.
Here's the result:
$ scala SimpleProcessExample
Answer: 3
Answer: 7
Subprocess exited with code 0.
If you wanted interaction between the input and output handlers to the subprocess, you can use standard thread communication tools (e.g., have both close over an instance of BlockingQueue).
Here is an example of obtaining input and output streams from a process, which you can write to and read from after the process starts:
object demo {
import scala.sys.process._
def getIO = {
// create piped streams that can attach to process streams:
val procInput = new java.io.PipedOutputStream()
val procOutput = new java.io.PipedInputStream()
val io = new ProcessIO(
// attach to the process's internal input stream
{ in =>
val istream = new java.io.PipedInputStream(procInput)
val buf = Array.fill(100)(0.toByte)
var br = 0
while (br >= 0) {
br = istream.read(buf)
if (br > 0) { in.write(buf, 0, br) }
}
in.close()
},
// attach to the process's internal output stream
{ out =>
val ostream = new java.io.PipedOutputStream(procOutput)
val buf = Array.fill(100)(0.toByte)
var br = 0
while (br >= 0) {
br = out.read(buf)
if (br > 0) { ostream.write(buf, 0, br) }
}
out.close()
},
// ignore stderr
{ err => () }
)
// run the command with the IO object:
val cmd = List("awk", "{ print $1 + $2 }")
val proc = cmd.run(io)
// wrap the raw streams in formatted IO objects:
val procO = new java.io.BufferedReader(new java.io.InputStreamReader(procOutput))
val procI = new java.io.PrintWriter(procInput, true)
(procI, procO)
}
}
Here's a short example of using the input and output objects. Note that it's hard to guarantee that the process will receive it's input until you close the input streams/objects, since everything is piped, buffered, etc.
scala> :load /home/eje/scala/input2proc.scala
Loading /home/eje/scala/input2proc.scala...
defined module demo
scala> val (procI, procO) = demo.getIO
procI: java.io.PrintWriter = java.io.PrintWriter#7e809b79
procO: java.io.BufferedReader = java.io.BufferedReader#5cc126dc
scala> procI.println("1 2")
scala> procI.println("3 4")
scala> procI.println("5 6")
scala> procI.close()
scala> procO.readLine
res4: String = 3
scala> procO.readLine
res5: String = 7
scala> procO.readLine
res6: String = 11
scala>
In general, if you are managing both input and output simultaneously in the same thread, there is the potential for deadlock, since either read or write can block waiting for the other. It is safest to run input logic and output logic in their own threads. With these threading concerns in mind, it is also possible to just put the input and output logic directly into the definitions { in => ... } and { out => ... }, as these are both run in separate threads automatically
I haven't actually tried this, but the documentation says that you can use a instance of ProcessIO to handle the Process's input and output in a manner similar to what you would do in Java.
var outPutStream: Option[OutputStream] = None
val io = new ProcessIO(
{ outputStream =>
outPutStream = Some(outputStream)
},
Source.fromInputStream(_).getLines().foreach(println),
Source.fromInputStream(_).getLines().foreach(println)
)
command run io
val out = outPutStream.get
out.write("test" getBytes())
You can get an InputStream in the same way.
I have two external processes to be run sequentially:
val antProc = Process(Seq(antBatch","everythingNoJunit"), new File(scriptDir))
val bossProc = Process(Seq(bossBatch,"-DcreateConnectionPools=true"))
val f: Future[Process] = Future {
println("Run ant...")
antProc.run
}
f onSuccess {
case proc => {
println("Run boss...")
bossProc.run
}
}
The result is:
Run ant...
Process finished with exit code 0
How do I run antProc until completion, then bossProc?
The following method seems to achieve the purpose. However, it's not a Future approach.
antProc.!<
bossProc.!<
You should be able to do something like this:
val antProc = Process(Seq(antBatch,"everythingNoJunit"), new File(scriptDir))
val bossProc = Process(Seq(bossBatch,"-DcreateConnectionPools=true"))
val antFut: Future[Process] = Future {
antProc.run
}
val bossFut: Future[Process] = Future {
bossProc.run
}
val aggFut = for{
antRes <- antFut
bossRes <- bossFut
} yield (antRes, bossRes)
aggFut onComplete{
case tr => println(tr)
}
The result of the aggFut will be a tuple consisting of the ant result and the boss result.
Also, be sure your vm that is running this is not exiting before the async callbacks can occur. If your execution context contains daemon threads then it might exit before completion.
Now if you want bossProc to run after antProc, the code would look like this
val antProc = Process(Seq(antBatch,"everythingNoJunit"), new File(scriptDir))
val bossProc = Process(Seq(bossBatch,"-DcreateConnectionPools=true"))
val antFut: Future[Process] = Future {
antProc.run
}
val aggFut = for{
antRes <- antFut
bossRes <- Future {bossProc.run}
} yield (antRes, bossRes)
aggFut onComplete{
case tr => println(tr)
}
I'm working with the Scala scala.sys.process library.
I know that I can capture the exit code with ! and the output with !! but what if I want to capture both?
I've seen this answer https://stackoverflow.com/a/6013932/416338 which looks promising, but I'm wondering if there is a one liner and I'm missing something.
I have the following utility method for running commands:
import sys.process._
def runCommand(cmd: Seq[String]): (Int, String, String) = {
val stdoutStream = new ByteArrayOutputStream
val stderrStream = new ByteArrayOutputStream
val stdoutWriter = new PrintWriter(stdoutStream)
val stderrWriter = new PrintWriter(stderrStream)
val exitValue = cmd.!(ProcessLogger(stdoutWriter.println, stderrWriter.println))
stdoutWriter.close()
stderrWriter.close()
(exitValue, stdoutStream.toString, stderrStream.toString)
}
As you can see, it captures stdout, stderr and result code.
You can use ProcessIO. I needed something like that in a Specs2 Test, where I had to check the exit value as well as the output of a process depending on the input on stdin (in and out are of type String):
"the operation" should {
f"return '$out' on input '$in'" in {
var res = ""
val io = new ProcessIO(
stdin => { stdin.write(in.getBytes)
stdin.close() },
stdout => { res = convertStreamToString(stdout)
stdout.close() },
stderr => { stderr.close() })
val proc = f"$operation $file".run(io)
proc.exitValue() must be_==(0)
res must be_==(out)
}
}
I figured that might help you. In the example I am ignoring what ever comes from stderr.
You can specify an output stream that catches the text:
import sys.process._
val os = new java.io.ByteArrayOutputStream
val code = ("volname" #> os).!
os.close()
val opt = if (code == 0) Some(os.toString("UTF-8")) else None
The one-line-ish use of BasicIO or ProcessLogger is appealing.
scala> val sb = new StringBuffer
sb: StringBuffer =
scala> ("/bin/ls /tmp" run BasicIO(false, sb, None)).exitValue
res0: Int = 0
scala> sb
res1: StringBuffer = ...
or
scala> import collection.mutable.ListBuffer
import collection.mutable.ListBuffer
scala> val b = ListBuffer[String]()
b: scala.collection.mutable.ListBuffer[String] = ListBuffer()
scala> ("/bin/ls /tmp" run ProcessLogger(b append _)).exitValue
res4: Int = 0
scala> b mkString "\n"
res5: String = ...
Depending on what you mean by capture, perhaps you're interested in output unless the exit code is nonzero. In that case, handle the exception.
scala> val re = "Nonzero exit value: (\\d+)".r.unanchored
re: scala.util.matching.UnanchoredRegex = Nonzero exit value: (\d+)
scala> Try ("./bomb.sh" !!) match {
| case Failure(f) => f.getMessage match {
| case re(x) => println(s"Bad exit $x")
| }
| case Success(s) => println(s)
| }
warning: there were 1 feature warning(s); re-run with -feature for details
Bad exit 3
The response provided by 'Alex Cruise' in your link is fairly concise, barring poorer performance.
You could extend sys.process.ProcessLogger to manage the
var out = List[String]()
var err = List[String]()
internally, with getters for the out.reverse and err.reverse results.
Here's a really simple Scala wrapper that allows you to retrieve stdout, stderr and exit code.
import scala.sys.process._
case class ProcessInfo(stdout: String, stderr: String, exitCode: Int)
object CommandRunner {
def runCommandAndGetOutput(command: String): ProcessInfo = {
val stdout = new StringBuilder
val stderr = new StringBuilder
val status = command ! ProcessLogger(stdout append _, stderr append _)
ProcessInfo(stdout.toString(), stderr.toString(), status)
}
}
I combined these and came up with this. The expected RC is there because I have a program I need to run in one project that returns 1 when it works. This does depend on the text of the Exception, but it will still do something reasonable it that doesn't match.
private val ProcessErrorP: Regex = "(.*): error=(\\d+),(.*)".r.unanchored
case class ProcessInfo(stdout: String, stderr: String, exitCode: Int, private val expectedRd: Int) {
def succeeded: Boolean = exitCode == expectedRd
def failed: Boolean = !succeeded
def asOpt: Option[String] = if (succeeded) None else Some(stderr)
}
/**
* Run a simple command
* #param command -- what to run
* #return -- what happened
*/
def run(command: String, expectedRc: Int = 0): ProcessInfo = {
try {
val stdout = new StringBuilder
val stderr = new StringBuilder
val status = command ! ProcessLogger(stdout append _, stderr append _)
ProcessInfo(stdout.toString(), stderr.toString(), status, expectedRc)
} catch {
case io: IOException =>
val dm = io.getMessage
dm match {
case ProcessErrorP(message, code, reason) =>
ProcessInfo("", s"$message, $reason", code.toInt, expectedRc)
case m: String =>
ProcessInfo("", m, 999, expectedRc)
}
}
}