I'm trying to create a test case for the controller below but I'm encountering some issues that I believe to be related to mocking methods correctly.
Issue:
Every time my test runs through this line:
file.ref.moveTo(new File(s"${configuration.uploadFileLocation}/$filename").getCanonicalFile,true)
It throws the following error:
Error:
[info] - run test *** FAILED ***
[info] java.lang.NullPointerException:
[info] at java.nio.file.Files.provider(Files.java:97)
[info] at java.nio.file.Files.move(Files.java:1392)
[info] at play.api.libs.Files$TemporaryFile.moveTo(Files.scala:100)
Controller:
def run = Action.async(parse.multipartFormData) { implicit request =>
request.body.file("file").map { file =>
import java.io.File
val filename = randomFileName
val contentType = file.contentType
if (contentType == Some("text/csv")) {
file.ref.moveTo(new File(s"${configuration.uploadFileLocation}/$filename").getCanonicalFile, true)
val result = apiClient.match (filename)
Future.successful("ok")
} else {
Future.successful("ok")
}
}.getOrElse {
Future.successful("ok")
}
}
Test case:
test("run() test") {
val mockMessageApi = mock(classOf[MessagesApi])
val mockApiClient = mock(classOf[apiClient])
val mockConfiguration = mock(classOf[Configuration])
val mockFile = mock(classOf[File])
val buildFakeRequest = FakeRequest(POST, "", FakeHeaders(), {
val tempFile = new TemporaryFile(mockFile)
val filePart = FilePart("file", "test.file", Some("text/csv"), tempFile)
MultipartFormData(Map(), List(filePart), List())
}
)
val controller = new MultipleMatch(mockApiClient, mockMessageApi, mockConfiguration)
val response = controller.run()(buildFakeRequest)
}
Question:
How can I effectively mock the ref and the moveTo method? What is causing this to happen? How to avoid the same issue again in the future?
Related
I am trying to process some files asynchronously, with the ability to choose the number of threads the program should use. But I want to wait till processFiles() is completed processing all the files. So, I am searching for ways to stop function from returning until all the Futures are done executing. And it would be very helpful if anyone gives any ideas to approach this problem. Here is my sample code.
object FileReader{
def processFiles(files: Array[File]) = {
val execService = Executors.newFixedThreadPool(5)
implicit val execContext = ExecutionContext.fromExecutorService(execService)
val processed = files.map { f =>
Future {
val name = f.getAbsolutePath()
val fp = Source.fromFile(name)
var data = ""
fp.getLines().foreach(x => {
data = data ++ s"$x\n"
})
fp.close()
// process the data.
println("Processing ....")
data
}
}
execContext.shutdown()
}
def main(args: Array[String]): Unit = {
println("Start")
val tmp = new File("/path/to/files")
val files = tmp.listFiles()
val result = processFiles(files)
println("done processing")
println("done work")
}
}
I am thinking if my usage of Future here is wrong, please correct me if I am wrong.
My expected output :
Start
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
done processing
done work
My current output:
Start
done processing
done work
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
Processing ....
You need to use Future.traverse to combine all the Future's for individual file processing and Await.result on them after:
import java.io.File
import java.util.concurrent.Executors
import scala.io.Source
import scala.concurrent._
import scala.concurrent.duration._
import scala.language.postfixOps
object FileReader {
def processFiles(files: Array[File]) = {
val execService = Executors.newFixedThreadPool(5)
implicit val execContext = ExecutionContext.fromExecutorService(execService)
//Turn `Array[Future[String]]` to `Future[Array[String]]`
val processed = Future.traverse(files.toList) { f =>
Future {
val name = f.getAbsolutePath()
val fp = Source.fromFile(name)
var data = ""
fp.getLines()
.foreach(x => {
data = data ++ s"$x\n"
})
fp.close()
// process the data.
println("Processing ....")
data
}
}
//TODO: Put proper timeout here
//Execution will be blocked until all futures completed
Await.result(processed, 30 minute)
execContext.shutdown()
}
def main(args: Array[String]): Unit = {
println("Start")
val tmp = new File(
"/path/to/file"
)
val files = tmp.listFiles()
val result = processFiles(files)
println("done processing")
println("done work")
}
}
Thanks to #Ivan Kurchenko. The solution worked. I am posting my final version of the code that worked.
object FileReader {
def processFiles(files: Seq[File]) = {
val execService = Executors.newFixedThreadPool(5)
implicit val execContext = ExecutionContext.fromExecutorService(execService)
//Turn `Array[Future[String]]` to `Future[Array[String]]`
val processed = Future.traverse(files) {
f =>
Future {
val name = f.getAbsolutePath()
val fp = Source.fromFile(name)
var data = ""
fp.getLines()
.foreach(x => {
data = augmentString(data) ++ s"$x\n"
})
fp.close()
// process the data.
println("Processing ....")
f
}
}
// TODO: Put proper timeout here
// Execution will be blocked until all futures completed
Await.result(processed, 30.minute)
execContext.shutdown()
}
def main(args: Array[String]): Unit = {
println("Start")
val tmp = new File(
"/path/to/file"
)
val files =tmp.listFiles.toSeq
val result = processFiles(files)
println("done processing")
println("done work")
}
}
I have an akka-gRPC service BiDirectional stream and I am testing it on a unit test. The service has uses akka-stream and I use the TestSink.probe to test the reply message. I am receiving back the messages from the service, but there is an error related to timeout that I cannot figure out what is the reason. This is the test:
object GreeterServiceConf {
// important to enable HTTP/2 in server ActorSystem's config
val configServer = ConfigFactory.parseString("akka.http.server.preview.enable-http2 = on")
.withFallback(ConfigFactory.defaultApplication())
val configString2 =
"""
|akka.grpc.client {
| "helloworld.GreeterService" {
| host = 127.0.0.1
| port = 8080
| }
|}
|""".stripMargin
val configClient = ConfigFactory.parseString(configString2)
}
class GreeterServiceImplSpec extends TestKit(ActorSystem("GreeterServiceImplSpec", ConfigFactory.load(GreeterServiceConf.configServer)))
with AnyWordSpecLike
with BeforeAndAfterAll
with Matchers
with ScalaFutures {
implicit val patience: PatienceConfig = PatienceConfig(scaled(5.seconds), scaled(100.millis))
// val testKit = ActorTestKit(conf)
val serverSystem: ActorSystem = system
val bound = new GreeterServer(serverSystem).run()
// make sure server is bound before using client
bound.futureValue
implicit val clientSystem: ActorSystem = ActorSystem("GreeterClient", ConfigFactory.load(GreeterServiceConf.configClient))
val client = GreeterServiceClient(
GrpcClientSettings
.fromConfig("helloworld.GreeterService")
.withTls(false)
)
override def afterAll: Unit = {
TestKit.shutdownActorSystem(system)
TestKit.shutdownActorSystem(clientSystem)
}
"GreeterService" should {
"reply to multiple requests" in {
import GreeterServiceData._
val names = List("John", "Michael", "Simone")
val expectedReply: immutable.Seq[HelloReply] = names.map { name =>
HelloReply(s"Hello, $name -> ${mapHelloReply.getOrElse(name, "this person does not exist =(")}")
}
val requestStream: Source[HelloRequest, NotUsed] = Source(names).map(name => HelloRequest(name))
val responseStream: Source[HelloReply, NotUsed] = client.sayHelloToAll(requestStream)
val sink = TestSink.probe[HelloReply]
val replyStream = responseStream.runWith(sink)
replyStream
.requestNext(HelloReply(s"Hello, John -> I killed Java"))
.requestNext(HelloReply(s"Hello, Michael -> We are the Jacksons 5"))
.requestNext(HelloReply(s"Hello, Simone -> I have found a job to work with Scala =)")) // THIS IS THE LINE 122 ON THE ERROR
// .request(3)
// .expectNextUnorderedN(expectedReply) // I also tested this but it did not work
.expectComplete()
}
}
}
The error is:
assertion failed: timeout (3 seconds) during expectMsg while waiting
for OnComplete java.lang.AssertionError: assertion failed: timeout (3
seconds) during expectMsg while waiting for OnComplete at
scala.Predef$.assert(Predef.scala:223) at
akka.testkit.TestKitBase.expectMsg_internal(TestKit.scala:459) at
akka.testkit.TestKitBase.expectMsg(TestKit.scala:436) at
akka.testkit.TestKitBase.expectMsg$(TestKit.scala:436) at
akka.testkit.TestKit.expectMsg(TestKit.scala:969) at
akka.stream.testkit.TestSubscriber$ManualProbe.expectComplete(StreamTestKit.scala:479)
at
com.example.helloworld.GreeterServiceImplSpec.$anonfun$new$5(GreeterServiceImplSpec.scala:121)
I got it to work based on the project akka-grpc-quickstart-scala.g8. I am executing runForeach to run the graph and have a materialized Sink on the response stream. Then, when the response is done I am doing an assert inside the Future[Done].
"reply to multiple requests" in {
import GreeterServiceData._
import system.dispatcher
val names = List("John", "Martin", "Michael", "UnknownPerson")
val expectedReplySeq: immutable.Seq[HelloReply] = names.map { name =>
HelloReply(s"Hello, $name -> ${mapHelloReply.getOrElse(name, "this person does not exist =(")}")
}
// println(s"expectedReplySeq: ${expectedReplySeq.foreach(println)}")
val requestStream: Source[HelloRequest, NotUsed] = Source(names).map(name => HelloRequest(name))
val responseStream: Source[HelloReply, NotUsed] = client.sayHelloToAll(requestStream)
val done: Future[Done] = responseStream.runForeach { reply: HelloReply =>
// println(s"got streaming reply: ${reply.message}")
assert(expectedReplySeq.contains(reply))
}
// OR USING Sink.foreach[HelloReply])(Keep.right)
val sinkHelloReply = Sink.foreach[HelloReply] { e =>
println(s"element: $e")
assert(expectedReplySeq.contains(e))
}
responseStream.toMat(sinkHelloReply)(Keep.right).run().onComplete {
case Success(value) => println(s"done")
case Failure(exception) => println(s"exception $exception")
}
}
Just to keep the reference of the whole code, the GreeterServiceImplSpec class is here.
I am using WebsocketClient and would like to test against the received message. I've chosen the Scalatest framework and I know, that the test has be carry out asynchronously.
The websocket client looks as the following:
import akka.{Done}
import akka.http.scaladsl.Http
import akka.stream.scaladsl._
import akka.http.scaladsl.model.ws._
import io.circe.syntax._
import scala.concurrent.Future
object WsClient {
import Trigger._
private val convertJson: PreMsg => String = msg =>
msg.asJson.noSpaces
val send: PreMsg => (String => Unit) => RunnableGraph[Future[Done]] = msg => fn =>
Source.single(convertJson(msg))
.map(TextMessage(_))
.via(Http().webSocketClientFlow(WebSocketRequest(s"ws://{$Config.host}:{$Config.port}/saprs")))
.map(_.asTextMessage.getStrictText)
.toMat(Sink.foreach(fn))(Keep.right)
}
and the test:
feature("Process incoming messages") {
info("As a user, I want that incoming messages is going to process appropriately.")
info("A message should contain the following properties: `sap_id`, `sap_event`, `payload`")
scenario("Message is not intended for the server") {
Given("A message with `sap_id:unknown`")
val msg = PreMsg("unknown", "unvalid", "{}")
When("the message gets validated")
val ws = WsClient.send(msg)
Then("it should has the `status: REJECT` in the response content")
ws { msg =>
//Would like test against the msg here
}.run()
.map(_ => assert(1 == 1))
}
I would to test against the content of msg, but I do not know, how to do it.
I followed the play-scala-websocket-example
They use a WebSocketClient as a helper, see WebSocketClient.java
Then a test looks like:
Helpers.running(TestServer(port, app)) {
val myPublicAddress = s"localhost:$port"
val serverURL = s"ws://$myPublicAddress/ws"
val asyncHttpClient: AsyncHttpClient = client.underlying[AsyncHttpClient]
val webSocketClient = new WebSocketClient(asyncHttpClient)
val queue = new ArrayBlockingQueue[String](10)
val origin = serverURL
val consumer: Consumer[String] = new Consumer[String] {
override def accept(message: String): Unit = queue.put(message)
}
val listener = new WebSocketClient.LoggingListener(consumer)
val completionStage = webSocketClient.call(serverURL, origin, listener)
val f = FutureConverters.toScala(completionStage)
// Test we can get good output from the websocket
whenReady(f, timeout = Timeout(1.second)) { webSocket =>
val condition: Callable[java.lang.Boolean] = new Callable[java.lang.Boolean] {
override def call(): java.lang.Boolean = webSocket.isOpen && queue.peek() != null
}
await().until(condition)
val input: String = queue.take()
val json:JsValue = Json.parse(input)
val symbol = (json \ "symbol").as[String]
List(symbol) must contain oneOf("AAPL", "GOOG", "ORCL")
}
}
}
See here: FunctionalSpec.scala
Given the following slightly modified Load Balancer ZeroMQ code:
package net.broker
import org.zeromq.ZMQ
object LruQueue2 {
class ClientTask(name: String) extends Runnable {
override def run(): Unit = {
val context = ZMQ.context(1)
val client = context.socket(ZMQ.REQ)
client.setIdentity(name.getBytes)
client.connect("tcp://localhost:5555")
// send request, get reply
client.send("HELLO".getBytes, 0)
val reply = client.recv(0)
println(s"${new String(client.getIdentity)} received: ${new String(reply)}")
}
}
class WorkerTask(name: String) extends Runnable {
override def run(): Unit = {
val context = ZMQ.context(1)
val worker = context.socket(ZMQ.REQ)
worker.connect("tcp://localhost:5556")
worker.setIdentity(name.getBytes)
worker.send("READY".getBytes, 0)
while(true) {
val clientAddr = worker.recv(0)
val empty = worker.recv(0)
val clientMsg = worker.recv(0)
worker.send(clientAddr, ZMQ.SNDMORE)
worker.send("".getBytes, ZMQ.SNDMORE)
worker.send("WORLD".getBytes, 0)
println(s"${new String(worker.getIdentity)}: 3-frames to client: ${new String(clientAddr)}")
}
}
}
def main(args: Array[String]): Unit = {
val NOFLAGS = 0
// worker using REQ socket to do LRU routing
val NBR_CLIENTS = 1
val NBR_WORKERS = 1
val context = ZMQ.context(1)
val frontend = context.socket(ZMQ.ROUTER)
val backend = context.socket(ZMQ.ROUTER)
frontend.bind("tcp://*:5555")
backend.bind("tcp://*:5556")
val clients = (1 to NBR_CLIENTS).toList.map{ i => new Thread(new ClientTask(s"CLIENT$i"))}
val workers = (1 to NBR_CLIENTS).toList.map{ i => new Thread(new WorkerTask(s"WORKER$i"))}
clients.foreach(_.start)
workers.foreach(_.start)
val workerQueue = scala.collection.mutable.Queue[Array[Byte]]()
val poller = context.poller(2)
poller.register(backend, ZMQ.Poller.POLLIN)
poller.register(frontend, ZMQ.Poller.POLLIN)
var clientNbr = NBR_CLIENTS
while(true) {
println("begin to poll")
poller.poll()
println("done polling")
println("clientNbr:" + clientNbr)
println("workerQueue.length: " + workerQueue.length)
if(clientNbr == 0) {
sys.exit(0)
}
else if(poller.pollin(0) && clientNbr > 0) {
val workerAddr = backend.recv(NOFLAGS)
val empty = backend.recv(NOFLAGS)
val clientAddrOrReadyMsg = backend.recv(NOFLAGS)
workerQueue.enqueue(workerAddr)
if(new String(clientAddrOrReadyMsg) == "READY") {
// nothing to do - worker is letting us know that he's ready to work
}
else {
// retrieve remaining 2 frames of client message
// [Empty][Client Message of "HELLO"]
val empty = backend.recv(0)
val workerResponse = backend.recv(0)
frontend.send(clientAddrOrReadyMsg, ZMQ.SNDMORE)
frontend.send("".getBytes, ZMQ.SNDMORE)
frontend.send(workerResponse, NOFLAGS)
clientNbr -= 1
}
}
else if (poller.pollin(1) && workerQueue.nonEmpty) {
val clientAddr = frontend.recv(0)
val empty = frontend.recv(0)
val clientMsg = frontend.recv(0)
backend.send(workerQueue.dequeue(), ZMQ.SNDMORE)
backend.send("".getBytes, ZMQ.SNDMORE)
backend.send(clientAddr, ZMQ.SNDMORE)
backend.send("".getBytes, ZMQ.SNDMORE)
backend.send(clientMsg, NOFLAGS)
}
else {}
}
}
}
When I run the above code, I see the following output:
[info] Running net.broker.LruQueue2
[info] begin to poll
[info] done polling
[info] clientNbr:1
[info] workerQueue.length: 0
[info] begin to poll
[info] done polling
[info] clientNbr:1
[info] workerQueue.length: 1
[info] begin to poll
[info] WORKER1: 3-frames to client: CLIENT1
[info] done polling
[info] clientNbr:1
[info] workerQueue.length: 0
[info] begin to poll
[info] CLIENT1 received: WORLD
It appears that it's stuck on poller.poll(), per the output print statements.
How can I debug this behavior?
I'm using the new mapWithState function in spark streaming (1.6) with a timing out state. I want to use the timing out state and add it to another rdd in order to use it for calculations further down the road:
val aggedlogs = sc.emptyRDD[MyLog];
val mappingFunc = (key: String, newlog: Option[MyLog], state: State[MyLog]) => {
val _newLog = newlog.getOrElse(null)
if ((state.exists())&&(_newLog!=null))
{
val stateLog = state.get()
val combinedLog = LogUtil.CombineLogs(_newLog, stateLog);
state.update(combinedLog)
}
else if (_newLog !=null) {
state.update(_newLog);
}
if (state.isTimingOut())
{
val stateLog = state.get();
aggedlogs.union(sc.parallelize(List(stateLog), 1))
}
val stateLog = state.get();
(key,stateLog);
}
val stateDstream = reducedlogs.mapWithState(StateSpec.function(mappingFunc).timeout(Seconds(10)))
but when I try to add it to an rdd in the StateSpec function, I get an error that the function is not serializable. Any thoughts on how I can get pass this?
EDIT:
After drilling deeper i found that my approach was wrong. before trying this solution i tried to get the timing out logs from the statesnapeshot(), but they were not there anymore, changing the mapping function to :
def mappingFunc(key: String, newlog: Option[MyLog], state: State[KomoonaLog]) : Option[(String, MyLog)] = {
val _newLog = newlog.getOrElse(null)
if ((state.exists())&&(_newLog!=null))
{
val stateLog = state.get()
val combinedLog = LogUtil.CombineLogs(_newLog, stateLog);
state.update(combinedLog)
Some(key,combinedLog);
}
else if (_newLog !=null) {
state.update(_newLog);
Some(key,_newLog);
}
if (state.isTimingOut())
{
val stateLog = state.get();
stateLog.timinigOut = true;
System.out.println("timinigOut : " +key );
Some(key, stateLog);
}
val stateLog = state.get();
Some(key,stateLog);
}
i managed to filter the mapedwithstatedstream for the logs that are timing out in each batch:
val stateDstream = reducedlogs.mapWithState(
StateSpec.function(mappingFunc _).timeout(Seconds(60)))
val tiningoutlogs= stateDstream.filter (filtertimingout)