I find it extreemly cool to use standard syntax like
import scala.sys.process._
val countLogger = ProcessLogger(line => {println ("out line: " + line)},
line => {println ("err line: " + line)})
val exitCode = ("cat prog.c" #&& "gcc prog.c -o prog -lm"
#&& "echo running, this may hang" #&& "prog.exe") ! countLogger
println("exitCode = " + exitCode)
It however happens that last process hangs. Is it possible to kill it on timeout?
You can wrap your process in a Future(blocking(_)) and if it doesn't return after the time-out, you call process.destroy().
That's what I have done for my small Processor library, e.g. see here. Instead of using ! to eagerly wait for the exit-code, you use the run method. Here is an adaption from the README:
import scala.concurrent._
import ExecutionContext.Implicits.global
import scala.sys.process._
val p = "sleep 100".run() // start asynchronously
val f = Future(blocking(p.exitValue())) // wrap in Future
val res = try {
Await.result(f, duration.Duration(2, "sec"))
} catch {
case _: TimeoutException =>
println("TIMEOUT!")
p.destroy()
p.exitValue()
}
Related
Just try some simple Task examples. The following code works fine
import monix.eval.Task
import monix.execution.CancelableFuture
import monix.execution.Scheduler.Implicits.global
import scala.util.Success
val task = Task { 1 + 1 }
val cancellable = task.runAsync {
case Right(result) => println(s"result is $result")
case Left(err) => System.out.println(s"ERROR: ${err.getMessage}")
}
but using runToFuture works only in sandbox, not when i run it in intelliJ (of course in intelliJ i run it inside object)
val task = Task { 1 + 1 }
val future: CancelableFuture[Int] = task.runToFuture
future.onComplete {
case Success(res) => println(s"result is: $res")
}
in intelliJ no printing 2, just
"C:\Program Files\Java\jdk1.8.0_192\bin\java.exe"
Process finished with exit code 0
What can be cause, i didn't expect stuck so early. Thanks in advance
When run as a stand-alone program, the program exits before the task completes, so you don't get any output. You need to wait for the task to complete.
Await.result(future, Duration.Inf)
How can I get some kind of writeable stream connected to stdin (and also readable streams connected to stdout and stderr) when launching a process via scala.sys.process library? Here's the code that doesn't work (doesn't even print debug messages)
val p = Process("wc -l")
val io = BasicIO.standard(true)
val lines = Seq("a", "b", "c") mkString "\n"
val buf = lines.getBytes(StandardCharsets.UTF_8)
io withInput { w =>
println("Writing")
w.write(buf)
}
io withOutput { i =>
val s = new BufferedReader(new InputStreamReader(i)).readLine()
println(s"Output is $s")
}
You have a couple of problems.
First in your snippet you never connect your process with the io and never run it.
That can be done like this: p run io.
Second, the withInput & withOutput methods return a NEW ProcessIO they DON'T mutate the actual, and since you don't assign the return of those calls to a variable, you are doing nothing.
The following snippet fixes both problems, hope it works for you.
import scala.io.Source
import scala.sys.process._
import java.nio.charset.StandardCharsets
val p = Process("wc -l")
val io =
BasicIO.standard(true)
.withInput { w =>
val lines = Seq("a", "b", "c").mkString("", "\n", "\n")
val buf = lines.getBytes(StandardCharsets.UTF_8)
println("Writing")
w.write(buf)
w.close()
}
.withOutput { i =>
val s = Source.fromInputStream(i)
println(s"Output is ${s.getLines.mkString(",")}")
i.close()
}
p run io
Don't doubt to ask for clarification.
PS: it prints "Output is 3" - (Thanks to Dima for pointing the mistake).
def fixture =
new {
val xyz = new XYZ(spark)
}
val fList: scala.collection.mutable.MutableList[scala.concurrent.Future[Dataset[Row]]] = scala.collection.mutable.MutableList[scala.concurrent.Future[Dataset[Row]]]() //mutable List of future means List[Future]
test("test case") {
val tasks = for (i <- 1 to 10) {
fList ++ scala.collection.mutable.MutableList[scala.concurrent.Future[Dataset[Row]]](Future {
println("Executing task " + i )
val ds = read(fixture.etlSparkLayer,i)
ds
})
}
Thread.sleep(1000*4200)
val futureOfList = Future.sequence(fList)//list of Future job in Future sequence
println(Await.ready(futureOfList, Duration.Inf))
val await_result: Seq[Dataset[Row]] = Await.result(futureOfList, Duration.Inf)
println("Squares: " + await_result)
futureOfList.onComplete {
case Success(x) => println("Success!!! " + x)
case Failure(ex) => println("Failed !!! " + ex)
}
}
I am executing one test case with sequence of Future List and List have collection of Future.I trying to execute same fuction multiple time parallely by help of using Future in scala.In my system only 4 job start in one time after completion of 4 jobs next 4 job will starting like that complete all the jobs. So how to start more than 4 job at a time and how main Thread will wait to complete all the Future thread ? I tried Await.result and Await.ready but not able to control main thread , for main thread control i m use Thread.sleep concept.this program is for read from RDBMS table and write in Elasticsearch. So how to control main thread main issue?
Assuming that you use the scala.concurrent.ExecutionContext.Implicits.global ExecutionContext you can tune the number of threads as described here:
https://github.com/scala/scala/blob/2.12.x/src/library/scala/concurrent/impl/ExecutionContextImpl.scala#L100
Specifically the following System Properties: scala.concurrent.context.minThreads, scala.concurrent.context.numThreads. scala.concurrent.context.maxThreads, and scala.concurrent.context.maxExtraThreads
Otherwise, you can rewrite your code to something like this:
import scala.collection.immutable
import scala.concurrent.duration._
import scala.concurrent._
import java.util.concurrent.Executors
test("test case") {
implicit val ec = ExecutionContext.fromExecutorService(ExecutorService.newFixedThreadPool(NUMBEROFTHREADSYOUWANT))
val aFuture = Future.traverse(1 to 10) {
i => Future {
println("Executing task " + i)
read(fixture.etlSparkLayer,i) // If this is a blocking operation you may want to consider wrapping it in a `blocking {}`-block.
}
}
aFuture.onComplete(_ => ec.shutdownNow()) // Only for this test, and to make sure the pool gets cleaned up
val await_result: immutable.Seq[Dataset[Row]] = Await.result(aFuture, 60.minutes) // Or other timeout
println("Squares: " + await_result)
}
I start and have running a Scala process.
val dir = "/path/to/working/dir/"
val stockfish = Process(Seq("wine", dir + "stockfish_8_x32.exe"))
val logger = ProcessLogger(printf("Stdout: %s%n", _))
val stockfishProcess = stockfish.run(logger, connectInput = true)
The process reads from and writes to standard IO (console). How can I send a string command to the process if it's been already started?
Scala process API has ProcessBuilder which has in turn bunch of useful methods. But ProcessBuilder is used before a process starts to compose complex shell commands. Also Scala has ProcessIO to handle input or output. I don't need it too. I just need to send message to my process.
In Java I would do something like this.
String dir = "/path/to/working/dir/";
ProcessBuilder builder = new ProcessBuilder("wine", dir + "stockfish_8_x32.exe");
Process process = builder.start();
OutputStream stdin = process.getOutputStream();
InputStream stdout = process.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(stdout));
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(stdin));
new Thread(() -> {
try {
String line;
while ((line = reader.readLine()) != null) {
System.out.println("Stdout: " + line);
}
} catch (IOException e) {
e.printStackTrace();
}
}).start();
Thread.sleep(5000); // it's just for example
writer.write("quit"); // send to the process command to stop working
writer.newLine();
writer.flush();
It works quite well. I start my process, get InputStream and OutputStream from it, and use the streams to interact with the process.
It appears Scala Process trait provides no ways to write to it. ProcessBuilder is useless after process run. And ProcessIO is just for IO catching and handling.
Are there any ways to write to Scala running process?
UPDATE:
I don't see how I may use ProcessIO to pass a string to running process.
I did the following.
import scala.io.Source
import scala.sys.process._
object Sample extends App {
def out = (output: java.io.OutputStream) => {
output.flush()
output.close()
}
def in = (input: java.io.InputStream) => {
println("Stdout: " + Source.fromInputStream(input).mkString)
input.close()
}
def go = {
val dir = "/path/to/working/dir/"
val stockfishSeq = Seq("wine", dir + "/stockfish_8_x32.exe")
val pio = new ProcessIO(out, in, err => {})
val stockfish = Process(stockfishSeq)
stockfish.run(pio)
Thread.sleep(5000)
System.out.write("quit\n".getBytes)
pio.writeInput(System.out) // "writeInput" is function "out" which I have passed to conforming ProcessIO instance. I can invoke it from here. It takes OutputStream but where can I obtain it? Here I just pass System.out for example.
}
go
}
Of course it does not work and I failed to understand how to implement functionality as in my Java snippet above. It would be great to have advice or snippet of Scala code clearing my issue.
I think the documentation around Scala processes (specifically the usage and semantics of ProcessIO) could use some improvement. The first time I tried using this API, I also found it very confusing, and it took some trial and error to get my subprocess i/o working correctly.
I think seeing a simple example is probably all you really need. I'll do something really simple: invoking bc as a subprocess to do some trivial computations, and then printing the answers to my stdout. My goal is to do something like this (but from Scala rather than from my shell):
$ printf "1+2\n3+4\n" | bc
3
7
Here's how I'd do it in Scala:
import scala.io.Source
import scala.sys.process._
object SimpleProcessExample extends App {
def out = (output: java.io.OutputStream) => {
output.flush()
output.close()
}
def in = (input: java.io.InputStream) => {
println("Stdout: " + Source.fromInputStream(input).mkString)
input.close()
}
// limit scope of any temporary variables
locally {
val calcCommand = "bc"
// strings are implicitly converted to ProcessBuilder
// via scala.sys.process.ProcessImplicits.stringToProcess(_)
val calcProc = calcCommand.run(new ProcessIO(
// Handle subprocess's stdin
// (which we write via an OutputStream)
in => {
val writer = new java.io.PrintWriter(in)
writer.println("1 + 2")
writer.println("3 + 4")
writer.close()
},
// Handle subprocess's stdout
// (which we read via an InputStream)
out => {
val src = scala.io.Source.fromInputStream(out)
for (line <- src.getLines()) {
println("Answer: " + line)
}
src.close()
},
// We don't want to use stderr, so just close it.
_.close()
))
// Using ProcessBuilder.run() will automatically launch
// a new thread for the input/output routines passed to ProcessIO.
// We just need to wait for it to finish.
val code = calcProc.exitValue()
println(s"Subprocess exited with code $code.")
}
}
Notice that you don't actually call any of the methods of the ProcessIO object directly because they're automatically called by the ProcessBuilder.
Here's the result:
$ scala SimpleProcessExample
Answer: 3
Answer: 7
Subprocess exited with code 0.
If you wanted interaction between the input and output handlers to the subprocess, you can use standard thread communication tools (e.g., have both close over an instance of BlockingQueue).
Here is an example of obtaining input and output streams from a process, which you can write to and read from after the process starts:
object demo {
import scala.sys.process._
def getIO = {
// create piped streams that can attach to process streams:
val procInput = new java.io.PipedOutputStream()
val procOutput = new java.io.PipedInputStream()
val io = new ProcessIO(
// attach to the process's internal input stream
{ in =>
val istream = new java.io.PipedInputStream(procInput)
val buf = Array.fill(100)(0.toByte)
var br = 0
while (br >= 0) {
br = istream.read(buf)
if (br > 0) { in.write(buf, 0, br) }
}
in.close()
},
// attach to the process's internal output stream
{ out =>
val ostream = new java.io.PipedOutputStream(procOutput)
val buf = Array.fill(100)(0.toByte)
var br = 0
while (br >= 0) {
br = out.read(buf)
if (br > 0) { ostream.write(buf, 0, br) }
}
out.close()
},
// ignore stderr
{ err => () }
)
// run the command with the IO object:
val cmd = List("awk", "{ print $1 + $2 }")
val proc = cmd.run(io)
// wrap the raw streams in formatted IO objects:
val procO = new java.io.BufferedReader(new java.io.InputStreamReader(procOutput))
val procI = new java.io.PrintWriter(procInput, true)
(procI, procO)
}
}
Here's a short example of using the input and output objects. Note that it's hard to guarantee that the process will receive it's input until you close the input streams/objects, since everything is piped, buffered, etc.
scala> :load /home/eje/scala/input2proc.scala
Loading /home/eje/scala/input2proc.scala...
defined module demo
scala> val (procI, procO) = demo.getIO
procI: java.io.PrintWriter = java.io.PrintWriter#7e809b79
procO: java.io.BufferedReader = java.io.BufferedReader#5cc126dc
scala> procI.println("1 2")
scala> procI.println("3 4")
scala> procI.println("5 6")
scala> procI.close()
scala> procO.readLine
res4: String = 3
scala> procO.readLine
res5: String = 7
scala> procO.readLine
res6: String = 11
scala>
In general, if you are managing both input and output simultaneously in the same thread, there is the potential for deadlock, since either read or write can block waiting for the other. It is safest to run input logic and output logic in their own threads. With these threading concerns in mind, it is also possible to just put the input and output logic directly into the definitions { in => ... } and { out => ... }, as these are both run in separate threads automatically
I haven't actually tried this, but the documentation says that you can use a instance of ProcessIO to handle the Process's input and output in a manner similar to what you would do in Java.
var outPutStream: Option[OutputStream] = None
val io = new ProcessIO(
{ outputStream =>
outPutStream = Some(outputStream)
},
Source.fromInputStream(_).getLines().foreach(println),
Source.fromInputStream(_).getLines().foreach(println)
)
command run io
val out = outPutStream.get
out.write("test" getBytes())
You can get an InputStream in the same way.
How to call an external process and read both of its exit code and standard out, once it finished?
Using sys.Process will result in an exception being thrown on different exit code than 0 for success.
Try this:
import sys.process._
val stdout = new StringBuilder
val stderr = new StringBuilder
val logger = ProcessLogger(stdout append _, stderr append _)
val status = "ls -al " ! logger
println(status)
println("stdout: " + stdout)
println("stderr: " + stderr)
Then you got both of them: status, stdout and stderr.
Have you looked at Process.exitValue?
Returns the exit value for the subprocess.
(I've asked this question on freenode #java and was requested to post here if I found a solution, so here goes)
Simple approach is to use sys.ProcessBuilder:
def RunExternal(executableName: String, executableDir: String) : (Int, List[String]) = {
val startExecutionTime = System.currentTimeMillis()
val pb : ProcessBuilder = new ProcessBuilder (executableName)
pb.directory(new java.io.File(executableDir))
val proc = pb.start()
proc.waitFor()
val exitCode = proc.exitValue()
val output = scala.io.Source.fromInputStream(proc.getInputStream).getLines.toList
val executionTime = System.currentTimeMillis() - startExecutionTime
logger.info(String.format(s"Process exited with exit code: ${exitCode}."))
logger.info(String.format(s"Process took ${executionTime} milliseconds."))
(exitCode, output)
}