Earlier I runned val pb = Process("""java -version""") and it gave me an exitValue of 0 as expected but code below runs process without exiting or blocking, so how can I get exitValue, my requirement actually is how to get status of a process that runs in background without stopping.
object Sample extends App {
import scala.sys.process.Process
val pb = Process("""java -jar common-api_2.11-1.3-one-jar.jar""")
val x = pb.run
print( "Exit value :" + x.exitValue )
}
You can get all the std output of a running process by passing a ProcesLogger to the run method:
e.g.
val logger = ProcessLogger((msg: String) ⇒ println(msg))
val x = pb.run(logger)
will print all output to the System.out. -but you can pass a function that would parse and evaluate the output of the process to extract some kind of state meaningful to your application. You can also pass a different function for statndard and error output. Have a look at ProcessLogger.apply variants.
Related
I wait for a Future to complete and print the content on the console. Even when everything is finished, the main application doesn't exit and I have to kill it manually.
def main(args: Array[String]): Unit {
val req = HttpRequest(GET, myURL)
val res = Http().singleRequest(req)
val resultsFutures = Future {
val resultString = Await.result(HttpRequests.unpackResponse(res), Duration.Inf)
JsonMethods.parse(resultString).extract[List[Results]]
}
val results = Await.result(resultsFutures, Duration.Inf)
println(results)
}
So results gets printed on the console with the expected contend, but the application still doesn't end.
Is there something I can do to exit the application? Is there still something running, that the main is waiting for?
I'm using:
scala 2.12.10
akka 2.5.26
akkaHttp 10.1.11
As you are using Akka, you likely have an ActorSystem instantiated somehow under the hood that will keep the process running.
Either you are able to get a hand on it and call its actorSystem.terminate() method, or you can also use an explicit sys.exit(0) at the end of your main method (0 being the exit code you want).
Edit: you should also wrap the Awaits in Try and make sure to call sys.exit in case of failures as well.
I am new to Scala and Play Framework so I am not quite sure what is wrong. I am trying to unpack a Future[Option[MyType]] given by a Slick DB controler (Play Framework). MyType is called BoundingBox in the code:
def getBoundingBoxByFileName(name: String) = {
val selectByName = boundingBoxTableQuery.filter{ boundingBoxTable =>
boundingBoxTable.name === name
}
db.run(selectByName.result.headOption)
}
BoundingBox type has a field called product_name. To retrieve this field I do the following:
val boundingBoxFutOpt = BoundingBoxQueryActions.getBoundingBoxByFileName("some_file")
val res = for {
optBb : Option[db.BoundingBox] <- boundingBoxFutOpt
} yield{
for(bb : db.BoundingBox <- optBb) yield {
println(s"${bb.product_name}")
}
}
This code does not yield anything on the output, though I have no compilation errors. If I change the println statement for some random text (not using the bb reference), it is also not printed on the console. To me it seems that the println statement is never executed.
I'll appreciate some directions on this problem.
It's likely that your program is terminating before the future has a chance to run the println. I think this will get you what you want:
import scala.concurrent.Await
import scala.concurrent.duration.Duration
// your code here
Await.result(res, Duration.Inf)
In your above example you're running a thread but then not giving it a chance to finish execution. The above will block until the future is complete.
It's worth nothing that you shouldn't use Await in production code as the blocking done negates the value of having code run in a separate thread.
I am running a spark scala program for performing text scanning in input file. I am trying to achieve parallelism by using rdd.mappartition. Inside the mappartition section i am performing few checks and calling the map function to achieve parallel execution for each partition. Inside the map function i am calling a custom method where i am performing the scanning and sending the results back.
Now, the code is working fine when i submit the code using --master local[*] but the same is not working when i submit using --master yarn-cluster. It is working without any error but the call is not getting inside the mappartition itself.I verified this by placing few println statements.
Please help me with your suggestions.
Here is the sample code:
def main(args: Array[String]) {
val inputRdd = sc.textFile(inputFile,2)
val resultRdd = inputRdd.mapPartitions{ iter =>
println("Inside scanning method..")
var scanEngine = ScanEngine.getInstance();
...
....
....
var mapresult = iter.map { y =>
line = y
val last = line.lastIndexOf("|");
message = line.substring(last + 1, line.length());
getResponse(message)
}
}
val finalRdd = sc.parallelize(resultRdd.map(x => x.trim()))
finalRdd.coalesce(1, true).saveAsTextFile(hdfsOutpath)
}
def getResponse(input: String): String = {
var result = "";
val rList = new ListBuffer[String]();
try {
//logic here
}
return result;
}
If your evidence of it working is seeing Inside scanning method.. printed out, it won't show up when run on the cluster because that code is executed by the workers, not the driver.
You're going to have to go over the code in forensic detail, with an open mind and try to find why the job has no output. Usually when a job works on local mode but not on a cluster it is because of some subtlety in where the code is executed, or where output is recorded.
There's too much clipped code to provide a more specific answer.
Spark achieves parallelism using the map function as well as mapPartitions. The number of partitions determines the amount of parallelism, but each partition will execute independently whether or not you use the mapPartitions function.
There are only a few reasons to use mapPartitions over map; e.g. there isa high initialization cost for a function, but then can call it multiple times such as doing some NLP task on text
I'm very new to both scalaz-stream and specifically scalaz.stream.tcp. I'm trying to do a very simple server for my own educational purposes. I parse the requests into commands, execute them to produce responses, and write the responses back to the client. The part I am having issues with is that I want to log each received command to stdout.
Here is my inner Process that I am passing to tcp.server:
def inner: Process[tcp.Connection, Unit] = {
val requests: Process[Connection, String] = tcp.reads(1024) pipe text.utf8Decode
val cmds: Process[Connection, Command] = requests.map(parseRequest)
val header: Process[Task, ByteVector] = Process("HEADER\n").pipe(text.utf8Encode)
val loggedCmds: Process[Connection, Command] = cmds.map { cmd =>
println(cmd.toString)
cmd
}
val results: Process[Connection, Process[Task, ByteVector]] = loggedCmds.map(_.execute)
val processedRequests: Process[Connection, Unit] = results.flatMap(result => tcp.writes(tcp.lift(header ++ result)))
processedRequests
}
(I am not in the habit of specifying the types everywhere; I just did that to try to get a handle on things. I plan to remove those.)
The above code actually compiles and runs correctly, but I do not feel it is very clean or idiomatic. Specifically I am unhappy with the loggedCmds part. I wanted to use io.stdOutLines, either through .observer or using writer.logged/mapW/drainW, but no matter what I tried I could not seem to get the types to line up correctly. I was always getting type conflicts between Task and Connection. tcp.lift seems to help with an input stream, but it does not seem to work for a Sink. Is there a cleaner/better way to do the loggedCmds part (FWIW: I'm open to corrections or improvements to any of the above code).
I should note that if I just have the results go to stdout via io.stdOutLines I do not have an issue ("through" seems to work in that case, which I have seen in examples), it's just when I want to send the stream to io.stdOutLines and also continue using the stream to respond to the client.
Figured it out on my own (finally). Using ".toChannel" I was able to do it:
val cmdFormatter = process1.id[Command].map(_.toString)
val cmdPrinter = io.stdOutLines.pipeIn(cmdFormatter)
...
val cmds: Process[Connection, Command] = requests.map(parseRequest) through
cmdPrinter.toChannel
A shorter solution is
val log = stdOutLines contramap { (_: Any).toString }
requests map(parseRequest) observe log
when I run the following test it throws an exception directly in the description clause, befor e it reaches the in clause:
"when the message send actor receives a reference to an address" - {
val unusedInThisTestActor = mock[ActorRef]
val serviceFacade = testActor
val keys = Array("password")
val values = Array("cheese")
val addressRef = TestActorRef[ServiceEndpoint]
val messageSender = system.actorOf(new Props(() => new MessageSendActor(unusedInThisTestActor, serviceFacade, unusedInThisTestActor)))
messageSender ! ReferenceToAddress(addressRef, ("command", keys, values))
"it tells the service facade to send the command to the address reference" in {
expectMsg(SendCommandToService(addressRef,("command", keys, values)))
}
}
ScalaTest then ignores this test, rather than saying it failed which leaves everyone thinking the code is working fine. The only way to notice something went wrong is to look carefully at the output:
[ERROR] [06/16/2013 10:31:05.722] [main] [akka://testactorsystems/user/$$a] error while processing Create(622541251)
d7565b2b-b0a4-4af6-83c7-ed67f7bf0302akka.actor.ActorInitializationException: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:170)
And use this to work out which tests aren't being run.
This is not very helpful, and I would prefer scala test to mark this as a failure somehow. Is that possible?