I just had a look at the new scala.sys and scala.sys.process packages to see if there is something helpful here. However, I am at a complete loss.
Has anybody got an example on how to actually start a process?
And, which is most interesting for me: Can you detach processes?
A detached process will continue to run when the parent process ends and is one of the weak spots of Ant.
UPDATE:
There seem to be some confusion what detach is. Have a real live example from my current project. Once with z-Shell and once with TakeCommand:
Z-Shell:
if ! ztcp localhost 5554; then
echo "[ZSH] Start emulator"
emulator \
-avd Nexus-One \
-no-boot-anim \
1>~/Library/Logs/${PROJECT_NAME}-${0:t:r}.out \
2>~/Library/Logs/${PROJECT_NAME}-${0:t:r}.err &
disown
else
ztcp -c "${REPLY}"
fi;
Take-Command:
IFF %#Connect[localhost 5554] lt 0 THEN
ECHO [TCC] Start emulator
DETACH emulator -avd Nexus-One -no-boot-anim
ENDIFF
In both cases it is fire and forget, the emulator is started and will continue to run even after the script has ended. Of course having to write the scripts twice is a waste. So I look into Scala now for unified process handling without cygwin or xml syntax.
First import:
import scala.sys.process.Process
then create a ProcessBuilder
val pb = Process("""ipconfig.exe""")
Then you have two options:
run and block until the process exits
val exitCode = pb.!
run the process in background (detached) and get a Process instance
val p = pb.run
Then you can get the exitcode from the process with (If the process is still running it blocks until it exits)
val exitCode = p.exitValue
If you want to handle the input and output of the process you can use ProcessIO:
import scala.sys.process.ProcessIO
val pio = new ProcessIO(_ => (),
stdout => scala.io.Source.fromInputStream(stdout)
.getLines.foreach(println),
_ => ())
pb.run(pio)
I'm pretty sure detached processes work just fine, considering that you have to explicitly wait for it to exit, and you need to use threads to babysit the stdout and stderr. This is pretty basic, but it's what I've been using:
/** Run a command, collecting the stdout, stderr and exit status */
def run(in: String): (List[String], List[String], Int) = {
val qb = Process(in)
var out = List[String]()
var err = List[String]()
val exit = qb ! ProcessLogger((s) => out ::= s, (s) => err ::= s)
(out.reverse, err.reverse, exit)
}
Process was imported from SBT. Here's a thorough guide on how to use the process library as it appears in SBT.
https://github.com/harrah/xsbt/wiki/Process
Has anybody got an example on how to
actually start a process?
import sys.process._ // Package object with implicits!
"ls"!
And, which is most interesting for me:
Can you detach processes?
"/path/to/script.sh".run()
Most of what you'll do is related to sys.process.ProcessBuilder, the trait. Get to know that.
There are implicits that make usage less verbose, and they are available through the package object sys.process. Import its contents, like shown in the examples. Also, take a look at its scaladoc as well.
The following function will allow easy use if here documents:
def #<<< (command: String) (hereDoc: String) =
{
val process = Process (command)
val io = new ProcessIO (
in => {in.write (hereDoc getBytes "UTF-8"); in.close},
out => {scala.io.Source.fromInputStream(out).getLines.foreach(println)},
err => {scala.io.Source.fromInputStream(err).getLines.foreach(println)})
process run io
}
Sadly I was not able to (did not have the time to) to make it an infix operation. Suggested calling convention is therefore:
#<<< ("command") {"""
Here Document data
"""}
It would be call if anybody could give me a hint on how to make it a more shell like call:
"command" #<<< """
Here Document data
""" !
Documenting process a little better was second on my list for probably two months. You can infer my list from the fact that I never got to it. Unlike most things I don't do, this is something I said I'd do, so I greatly regret that it remains as undocumented as it was when it arrived. Sword, ready yourself! I fall upon thee!
If I understand the dialog so far, one aspect of the original question is not yet answered:
how to "detach" a spawned process so it continues to run independently of the parent scala script
The primary difficulty is that all of the classes involved in spawning a process must run on the JVM, and they are unavoidably terminated when the JVM exits. However, a workaround is to indirectly achieve the goal by leveraging the shell to do the "detach" on your behalf. The following scala script, which launches the gvim editor, appears to work as desired:
val cmd = List(
"scala",
"-e",
"""import scala.sys.process._ ; "gvim".run ; System.exit(0);"""
)
val proc = cmd.run
It assumes that scala is in the PATH, and it does (unavoidably) leave a JVM parent process running as well.
Related
When I write the following code (in ammonite but i don't think it matters)
("tail -f toTail.txt" lineStream) foreach(println(_)), the program give me the last line as intend but then hang, and even if i write more in the file, nothing come out.
How does the API support process that have unbounded output ?
I try to write val myStream = ("tail -f toTail.txt" lineStream_!)
but it still does not return write away
Here is what the scala doc says:
lineStream: returns immediately like run, and the output being generated is provided through a Stream[String]. Getting the next element of that Stream may block until it becomes available.
Hence i don't understand why it blocks
By the way i am having exactly the same behavior with the Ammonite API
If type %%("tail", "-f", "toTail.txt") once again the method just hang and does not return immediately.
There is no issue with the ProcessBuilder (at least not one that stems from your use case). From the ProcessBuilder documentation:
Starting Processes
To execute all external commands associated with a ProcessBuilder, one may use one of four groups of methods. Each of these methods have various overloads and variations to enable further control over the I/O. These methods are:
run: the most general method, it returns a scala.sys.process.Process immediately, and the external command executes concurrently.
!: blocks until all external commands exit, and returns the exit code of the last one in the chain of execution.
!!: blocks until all external commands exit, and returns a String with the output generated.
lineStream: returns immediately like run, and the output being generated is provided through a Stream[String]. Getting the next element of that Stream may block until it becomes available. This method will throw an exception if the return code is different than zero -- if this is not desired, use the lineStream_! method.
The documentation clearly states that lineStream might block until the next line becomes available. Since the nature of tail -f is an infinite stream of lines lineBreak the program will block waiting for the next line to appear.
For the following assume I have a file: /Users/user/tmp/sample.txt and it's contents are:
boom
bar
cat
Why lineStream_! isn't wrong
import scala.language.postfixOps
import scala.sys.process._
object ProcessBuilder extends App {
val myStream: Stream[String] = ("tail /Users/user/tmp/sample.txt" lineStream_!)
println("I'm after tail!")
myStream.filter(_ != null).foreach(println)
println("Finished")
System.exit(0)
}
Outputs:
I'm after tail!
boom
bar
cat
Finished
So you see that the lineStream_! returned immediately. Because the nature of the command is finite.
How to return immediately from a command that produces infinite output:
Let's try this with tail -f. You need more control over your process. Again, as the documentation states:
If one desires full control over input and output, then a scala.sys.process.ProcessIO can be used with run.
So for an example:
import java.io.{BufferedReader, InputStreamReader}
import scala.language.postfixOps
import scala.sys.process._
object ProcessBuilder extends App {
var reader: BufferedReader = _
try {
var myStream: Stream[String] = Stream.empty
val processIO = new ProcessIO(
(os: java.io.OutputStream) => ??? /* Send things to the process here */,
(in: java.io.InputStream) => {
reader = new BufferedReader(new InputStreamReader(in))
myStream = Stream.continually(reader.readLine()).takeWhile(_ != "ff")
},
(in: java.io.InputStream) => ???,
true
)
"tail -f /Users/user/tmp/sample.txt".run(processIO)
println("I'm after the tail command...")
Thread.sleep(2000)
println("Such computation performed while tail was active!")
Thread.sleep(2000)
println("Such computation performed while tail was active again!")
println(
s"Captured these lines while computing: ${myStream.print(System.lineSeparator())}")
Thread.sleep(2000)
println("Another computation!")
} finally {
Option(reader).foreach(_.close())
}
println("Finished")
System.exit(0)
}
Outputs:
I'm after the tail command...
Such computation performed while tail was active!
Such computation performed while tail was active again!
boom
bar
cat
It still returns immediately and now it just hangs there waiting for more input. If I do echo 'fff' >> sample.txt from the tmp directory, the program outputs:
Another computation!
Finished
You now have the power to perform any computation you want after issuing a tail -f command and the power to terminate it depending on the condition you pass to the takeWhile method (or other methods that close the input stream).
For more details on ProcessIO check the documentation here.
I think, this has to do with how you are adding data to the file, not with the ProcessBuilder. If you are using it in an editor for example to add data, it rewrites the entire file every time you save, with a different inode, and tail isn't detecting that.
Try doing tail -F instead of tail -f, that should work.
I am new to this Scala world and I am trying some exercises from a book. So, I have an example that print a vector in sequential and parallel fashion. The former works perfectly and the later hangs the console.
Code
val v = Vector.range(0, 10)
v.foreach(println)
Code output
0123456789
But if I use the same code, but instead of using foearch, use par, it freezes the console
val v = Vector.range(0,10)
v.par.foreach(println)
The book I am using says that the output should be something like:
5678901234
But it hangs and the program never finishes.
Can someone explain me why?
It is better to post whole the program wich is hanged.
I've just tested it with scala 2.12.8 and jvm 1.8.0_161:
object MainClassss {
def main(args: Array[String]): Unit = {
val v = Vector.range(0,10)
(0 to 999999).foreach(_ => v.par.foreach(println))
}
}
the program has executed well with output and hasn't hanged.
If your program reproduced the aforementioned issue you need to take a thread dump by:
$ jstack <PID>
where PID is the process id.
Or you can take by jvisualvm jvm tool.
You can analyze why your program is hanged owing to caught thread dump if the program has some code blocking it from execution.
So, I have the following kind of code running in each map tasks on Spark.
#volatile var res = (someProgram + fileName) !
var cmdRes = ("rm " + fileName) !;
The filenames for each map tasks are unique. The basic idea is that once the first command finishes, the second command deletes the file. However, I notice that the program sometimes complain that the file does not exist. It seems that the subprocess call is not synchronous, that is, it does not wait for the subprocess to complete. Is that correct. And if it indeed the case, how can we correct that?
As you can see in the docs, the ! method blocks until exit. The docs say that it:
Starts the process represented by this builder, blocks until it exits, and returns the exit code.
It's possible that you should be checking the exit code to interpret the result and to deal with exceptional cases.
When creating process commands by concatenation, you are often better off using the Seq extensions (as opposed to the String ones) to create a ProcessBuilder. The docs even include this helper, which might help you:
// This uses ! to get the exit code
def fileExists(name: String) = Seq("test", "-f", name).! == 0
I am trying to build a git command parser in sbt.
The goal of the parser is not so much to validate the actual git command but rather to provide auto-completion within the sbt console.
The parser relies on bash completion scripts, so it's fair to say that generating the completions is fairly expensive as a process has to spawn every time. That's why I'd like to minimize the number of call made to the bash-completion process.
I have a working solution, that looks like this:
def autoCompleteParser(state: State) = {
val extracted = Project.extract(state)
import extracted._
val dir = extracted.get(baseDirectory)
def suggestions(args: Seq[String]): Seq[String] = {
// .. calling Process and collecting the completions into a Seq[String]
}
val gitArgsParser: Parser[Seq[String]] = {
def loop(previous: Seq[String]): Parser[Seq[String]] =
token(Space) ~> NotSpace.examples(suggestions(previous): _*).flatMap(res => loop(previous :+ res))
loop(Vector())
}
gitArgsParser
}
val test = Command("git-auto-complete")(autoCompleteParser _)(autoCompleteAction)
However I have two problems:
the completion process is called for every character, which is more than I'd like
the potential completions seems to be passed as a parameter to another round of completions, which means even more calls to the external process.
My question is the following: how do I tell sbt to reuse/cache the completions he has got for the rest of an argument without calling the process for each character? For example:
completions for 'git a' are:
dd m nnotate pply rchive
Then completion for 'git ad' are:
d
without the need to call the suggestions method again. I have tried to implement an ExampleSource, but I could not obtain the behavior I was looking for from it.
Any pointer would be welcome. And if someone understands why the potential completions seems to be passed into another completions round, that would help me a lot too.
I am trying to retrieve the AST from scala souce file. I have simplified the code (only relevant code) to following.
trait GetAST {
val settings = new Settings
val global = new Global(settings, new ConsoleReporter(settings))
def getSt = "hello" //global.typedTree(src, true)
}
object Tre extends GetAST {
def main(args:Array[String])
{
println(getSt.getClass)
println("exiting program")
}
}
The above code compiles fine and runs fine. But the problem is the program does not exit. The prompt is not displayed after printing "exiting program". I have to use ^c to exit.
Any idea what the problem might be
I believe Michael is correct, the compiler uses Threads and therefore the JVM doesn't just exit.
The good news is that interactive.Global mixes in interactive.CompilerControl trait whose askShutdown method you can call at the end of your main to let the program exit.
Without knowing what Settings, Global and ConsoleReporter are nobody can give you an exact answer. I would guess that at least one of them is creating a thread. The JVM waits until all threads are done (or all running are deamon threads). See here.
I would bet if you comment out the settings and global lines it will exit as expected.