File writing iteratee does not receive EOF for WS.get - scala

I have created a simple iteratee to download a file using WS as explained in this link.
Consider the following snippet:
import java.nio.ByteBuffer
import java.nio.channels.FileChannel
import org.specs2.mutable.Specification
import org.specs2.time.NoTimeConversions
import play.api.libs.Files.TemporaryFile
import play.api.libs.iteratee.{Done, Input, Cont, Iteratee}
import play.api.libs.ws.WS
import scala.concurrent.Await
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
class DummySpec extends Specification with NoTimeConversions {
def fileChannelIteratee(channel: FileChannel): Iteratee[Array[Byte], Unit] = Cont {
case Input.EOF =>
println("Input.EOF")
channel.close()
Done(Unit, Input.EOF)
case Input.El(bytes) =>
println("Input.El")
val buf = ByteBuffer.wrap(bytes)
channel.write(buf)
fileChannelIteratee(channel)
case Input.Empty =>
println("Input.Empty")
fileChannelIteratee(channel)
}
"fileChannelIteratee" should {
"work" in {
val file = TemporaryFile.apply("test").file
val channel = FileChannel.open(file.toPath)
val future = WS.url("http://www.example.com").get(_ => fileChannelIteratee(channel)).map(_.run)
Await.result(future, 10.seconds)
file.length !== 0
}
}
}
Calling .map(_.run) after WS.get seems to have no effect here as the iteratee does not seem to receive Input.EOF. It prevents me from being able to close the channel. This is the output I get:
Input.El
[info] DummySpec
[info]
[info] fileChannelIteratee should
[info] x work (941 ms)
[error] '0' is equal to '0' (DummySpec.scala:37)
[info]
[info]
[info] Total for specification DummySpec
[info] Finished in 948 ms
[info] 1 example, 1 failure, 0 error
What am I doing wrong?
I am using Play Framework 2.2.2.
Thanks in advance.

I was opening the FileChannel in a wrong way. It seems to default to read mode according to this link when no parameters are given.
The exception thrown from channel.write was being swallowed by map operation as the return type of the whole operation is Future[Future[Unit]]. The outer Future is in a successful state even if the internal one fails in this case. flatMap should be used instead.

Related

How should I test akka-streams RestartingSource usage

I'm working on an application that has a couple of long-running streams going, where it subscribes to data about a certain entity and processes that data. These streams should be up 24/7, so we needed to handle failures (network issues etc).
For that purpose, we've wrapped our sources in RestartingSource.
I'm now trying to verify this behaviour, and while it looks like it functions, I'm struggling to create a test where I push in some data, verify that it processes correctly, then send an error, and verify that it reconnects after that and continues processing.
I've boiled that down to this minimal case:
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.{RestartSource, Sink, Source}
import akka.stream.testkit.TestPublisher
import org.scalatest.concurrent.Eventually
import org.scalatest.{FlatSpec, Matchers}
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext
class MinimalSpec extends FlatSpec with Matchers with Eventually {
"restarting a failed source" should "be testable" in {
implicit val sys: ActorSystem = ActorSystem("akka-grpc-measurements-for-test")
implicit val mat: ActorMaterializer = ActorMaterializer()
implicit val ec: ExecutionContext = sys.dispatcher
val probe = TestPublisher.probe[Int]()
val restartingSource = RestartSource
.onFailuresWithBackoff(1 second, 1 minute, 0d) { () => Source.fromPublisher(probe) }
var last: Int = 0
val sink = Sink.foreach { l: Int => last = l }
restartingSource.runWith(sink)
probe.sendNext(1)
eventually {
last shouldBe 1
}
probe.sendNext(2)
eventually {
last shouldBe 2
}
probe.sendError(new RuntimeException("boom"))
probe.expectSubscription()
probe.sendNext(3)
eventually {
last shouldBe 3
}
}
}
This test consistently fails on the last eventually block with Last failure message: 2 was not equal to 3. What am I missing here?
Edit: akka version is 2.5.31
I figured it out after having had a look at the TestPublisher code. Its subscription is a lazy val. So when RestartSource detects the error, and executes the factory method () => Source.fromPublisher(probe) again, it gets a new Source, but the subscription of the probe is still pointing to the old Source. Changing the code to initialize both a new Source and TestPublisher works.

Null pointer exception with GenericWriteAheadSink and FileCheckpointCommitter in Flink 1.8

I'm implementing a custom NSQ sink for Flink. I have it working as a subclass of RichSinkFunction, but I'd like to get the write-ahead log implementation working for extra data integrity.
Using O'Reilly's WriteAheadSinkExample available here, I attempted to implement my own:
package com.wistia.analytics
import java.net.{InetSocketAddress, SocketAddress}
import com.github.mitallast.nsq._
import org.apache.flink.api.scala.createTypeInformation
import java.lang.Iterable
import java.nio.file.{Files, Paths}
import java.util.UUID
import org.apache.commons.lang3.StringUtils
import org.apache.flink.api.common.ExecutionConfig
import org.apache.flink.streaming.runtime.operators.{CheckpointCommitter, GenericWriteAheadSink}
import scala.collection.mutable
class WALNsqSink(val topic: String) extends GenericWriteAheadSink[String](
// CheckpointCommitter that commits checkpoints to the local filesystem
new FileCheckpointCommitter(System.getProperty("java.io.tmpdir")),
// Serializer for records
createTypeInformation[String]
.createSerializer(new ExecutionConfig),
// Random JobID used by the CheckpointCommitter
UUID.randomUUID.toString) {
var client: NSQClient = _
var producer: NSQProducer = _
override def open(): Unit = {
val lookup = new NSQLookup {
def nodes(): List[SocketAddress] = List(new InetSocketAddress("127.0.0.1",4150))
def lookup(topic: String): List[SocketAddress] = List(new InetSocketAddress("127.0.0.1",4150))
}
client = NSQClient(lookup)
producer = client.producer()
}
def sendValues(readings: Iterable[String], checkpointId: Long, timestamp: Long): Boolean = {
val arr = mutable.Seq()
readings.forEach{ reading =>
arr :+ reading
}
producer.mpubStr(topic=topic, data=arr)
true
}
}
reusing FileCheckpointCommitter at the bottom of the class, but I get a null pointer exception inside GenericWriteAheadSink:
Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:638)
at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:123)
at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
at com.wistia.analytics.NsqProcessor$.main(NsqProcessor.scala:24)
at com.wistia.analytics.NsqProcessor.main(NsqProcessor.scala)
Caused by: java.lang.NullPointerException
at org.apache.flink.streaming.runtime.operators.GenericWriteAheadSink.processElement(GenericWriteAheadSink.java:277)
at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:202)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:748)
Nonzero exit code returned from runner: 1
(Compile / run) Nonzero exit code returned from runner: 1
Total time: 45 s, completed Feb 10, 2020 6:41:06 PM
I have no idea where to go from here. Any help appreciated
The issue here is most certainly the fact that You never call the open() method of the superclass. This will cause some of the variables to be uninitialized.
This should be solved by calling the super.open() inside Your open() method.

Cannot implement .extraInfoExtractor in Gatling Script

I am trying to log extra response information in gatling result through, http.extraInfoExtractor. Below is my code, I am failing to execute the same, also mentioned the error. Please help.
Code :
package cloudnative
import scala.concurrent.duration._
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.jdbc.Predef._
class cloudnativems extends Simulation {
val nbUsers = Integer.getInteger("users", 1)
val myRamp = java.lang.Long.getLong("ramp", 0L)
val varPipelineId = sys.env.get("CI_PIPELINE_ID")
println(varPipelineId)
val httpProtocol = http
.baseUrl("https://lXXXXXXXXXX")
.inferHtmlResources()
.contentTypeHeader("application/json")
val scn = scenario("cloudnativems")
.exec(http("account_movement_post")
.post("/XXXXXXX/gatling-poc/demo/movement/account")
.extraInfoExtractor(extraInfo => List(extraInfo.response.statusCode.get))
.body(StringBody("""{
"movementId": "m0001",
"accountId": "a0001",
"amount": 2000,
"movementDate": "2019-02-26T09:34:50.301Z",
"counterparty": "c0001"
}""")))
.exec(http("account_movement_get")
.get("/XXXXXXX/gatling-poc/demo/movement/account/m0001")
)
setUp(scn.inject(atOnceUsers(1)).protocols(httpProtocol))
}
Error
C:\Sopra Project\Tools\gatling-charts-highcharts-bundle-3.0.3\bin>gatling.bat -s cloudnative.cloudnativems
GATLING_HOME is set to "C:\Sopra Project\Tools\gatling-charts-highcharts-bundle-3.0.3"
JAVA = ""C:\Program Files\Java\jdk1.8.0_201\bin\java.exe""
11:56:14.849 [ERROR] i.g.c.ZincCompiler$ - C:\Sopra Project\Tools\gatling-charts-highcharts-bundle-3.0.3\user-files\simulations\cloudnative\cloudnativems.scala:25:8: value extraInfoExtractor is not a member of io.gatling.http.request.builder.HttpRequestBuilder
possible cause: maybe a semicolon is missing before `value extraInfoExtractor'?
.extraInfoExtractor(extraInfo => List(extraInfo.response.statusCode.get))
^
11:56:15.190 [ERROR] i.g.c.ZincCompiler$ - one error found
11:56:15.193 [ERROR] i.g.c.ZincCompiler$ - Compilation crashed
sbt.internal.inc.CompileFailed: null
extraInfoExtractor has been dropped with version 3.0 according to https://gatling.io/docs/current/migration_guides/2.3-to-3.0/:
extraInfoExtractor was dropped as it wasn’t used in any Gatling component
I am not aware of a replacement construct in Gatling 3.0
Greetings,
Matthias

Writing ScalaTest for ReactiveMongoApi in Playframework 2.4?

I am trying to start writing test for mongodb in my play application.
It seems that I can not get it working because the connection is shutdown before the save is executed.
Here are the results
[info] UserDaoMongoSpec:
[info] UserDao
[info] - application - ReactiveMongoApi starting...
[info] - application - ReactiveMongoApi successfully started with DB 'test'! Servers:
[localhost:27017]
[info] - should save users and find them by userId
[info] - application - ReactiveMongoApi stopping...
[info] - application - ReactiveMongoApi connections stopped. [Success(Closed)]
[INFO] [12/24/2015 15:36:43.961] [reactivemongo-akka.actor.default-dispatcher-4] [akka://reactivemongo/user/Monitor-3] Message [reactivemongo.core.actors.Close$] from Actor[akka://reactivemongo/deadLetters] to Actor[akka://reactivemongo/user/Monitor-3#1192398481] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[info] ScalaTest
[info] Run completed in 3 seconds, 989 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
This is the code of the test, which is not testing anything, but I am trying first to write the database.
package test.daos
import scala.concurrent.ExecutionContext.Implicits.global
import daos.impl.UserDaoMongo
import org.scalatest._
import play.api.test._
import play.api.test.Helpers._
import org.scalatestplus.play._
class UserDaoMongoSpec extends DaosApplicationSpecOneAppPerTest {
"UserDao" should {
"save users and find them by userId" in {
val userDao = new UserDaoMongo
val future = for {
_ <- userDao.save(credentialsTestUser)
maybeUser <- userDao.find(credentialsTestUser.id)
} yield maybeUser.map(_ == credentialsTestUser)
}
}
}
And the dao implementation
package daos.impl
import java.util.UUID
import scala.concurrent.Future
import play.api.Play.current
import play.api.libs.concurrent.Execution.Implicits.defaultContext
import play.api.libs.json.Json
import play.modules.reactivemongo.ReactiveMongoApi
import play.modules.reactivemongo.json._
import play.modules.reactivemongo.json.collection.JSONCollection
import com.mohiva.play.silhouette.api.LoginInfo
import models.{User, Profile}
import models.User._
import daos.api.UserDao
import play.api.Logger
class UserDaoMongo extends UserDao {
lazy val reactiveMongoApi = current.injector.instanceOf[ReactiveMongoApi]
val users = reactiveMongoApi.db.collection[JSONCollection]("users")
def find(loginInfo:LoginInfo):Future[Option[User]] =
users.find(Json.obj("profiles.loginInfo" -> loginInfo)).one[User]
def find(userId:UUID):Future[Option[User]] ={
Logger.debug("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA")
users.find(Json.obj("id" -> userId)).one[User]
}
def save(user:User):Future[User] =
users.insert(user).map(_ => user)
def confirm(loginInfo:LoginInfo):Future[User] = for {
_ <- users.update(Json.obj(
"profiles.loginInfo" -> loginInfo
), Json.obj("$set" -> Json.obj("profiles.$.confirmed" -> true)))
user <- find(loginInfo)
} yield user.get
def link(user:User, profile:Profile) = for {
_ <- users.update(Json.obj(
"id" -> user.id
), Json.obj("$push" -> Json.obj("profiles" -> profile)))
user <- find(user.id)
} yield user.get
def update(profile:Profile) = for {
_ <- users.update(Json.obj(
"profiles.loginInfo" -> profile.loginInfo
), Json.obj("$set" -> Json.obj("profiles.$" -> profile)))
user <- find(profile.loginInfo)
} yield user.get
}
What could be that I am doing wrong?
Thank you
As says in the comments, it turns out that the mistake was that I was not waiting for the future completion. I decided to use ScalaTest specific function to make tests wait for the futures completions
Here is the code
"UserDao" should {
"save users and find them by userId" in withUserDao { userDao =>
val future = for {
user <- userDao.save(credentialsTestUser)
maybeUser <- userDao.find(credentialsTestUser.id)
} yield {
maybeUser.map(_ == credentialsTestUser)
}
whenReady (future) { result =>
result.get must be (true)
}
}
}
and just in case, I override the default behaviour because the default time was not enough
implicit override val patienceConfig =
PatienceConfig(timeout = Span(2, Seconds), interval = Span(5, Millis))
Thank you

Not all akka stream Sinks receive the emitted data

When running the following akka streaming FlowGraph not all the emitted Chars are received by all Sinks.
package sample.stream
import java.io.{ FileOutputStream, PrintWriter }
import akka.actor.ActorSystem
import akka.stream.ActorFlowMaterializer
import akka.stream.scaladsl.{ Broadcast, FlowGraph, Sink, Source }
import scala.concurrent.forkjoin.ThreadLocalRandom
import scala.util.{ Failure, Success, Try }
object Sample {
def main(args: Array[String]): Unit = {
println("start")
implicit val system = ActorSystem("Sys")
import system.dispatcher
implicit val materializer = ActorFlowMaterializer()
var counter = -1
val countSource: Source[Char, Unit] = Source(() => Iterator.continually { counter += 1; (counter + 'A').toChar }.take(11))
var counter1 = 0
val consoleSink1 = Sink.foreach[Char] { counter =>
println("sink1:" + counter1 + ":" + counter)
counter1 += 1
Thread.sleep(100)
//Thread.sleep(300)
}
var counter2 = 0
val consoleSink2 = Sink.foreach[Char] { counter =>
println("sink2:" + counter2 + ":" + counter)
counter2 += 1
Thread.sleep(200)
}
val materialized = FlowGraph.closed(consoleSink1, consoleSink2)((x1, x2) => x1) { implicit builder =>
(console1, console2) =>
import FlowGraph.Implicits._
val broadcast = builder.add(Broadcast[Char](2))
countSource ~> broadcast ~> console1
broadcast ~> console2
}.run()
// ensure the output file is closed and the system shutdown upon completion
materialized.onComplete {
case Success(_) =>
system.shutdown()
case Failure(e) =>
println(s"Failure: ${e.getMessage}")
system.shutdown()
}
println("waiting the remaining ones")
//scala.concurrent.Await.ready(materialized, scala.concurrent.duration.DurationInt(100).seconds)
//system.shutdown()
println("end")
}
}
After running the following output is generated
[info] Running sample.stream.Sample
[info] start
[info] waiting the remaining ones
[info] end
[info] sink2:0:A
[info] sink1:0:A
[info] sink1:1:B
[info] sink1:2:C
[info] sink2:1:B
[info] sink1:3:D
[info] sink2:2:C
[info] sink1:4:E
[info] sink1:5:F
[info] sink2:3:D
[info] sink1:6:G
[info] sink1:7:H
[info] sink2:4:E
[info] sink2:5:F
[info] sink1:8:I
[info] sink1:9:J
[info] sink2:6:G
[info] sink2:7:H
[info] sink1:10:K
The second sink doesn't receive the 8th, 9th and 10th values: IJK but still the entire flow is ended.
What should I do to wait for both Sinks to consume all the data?
I discovered that if I change the (x1,x2)=>x1 to (x1,x2)=>x2 this will wait. That is the same with sleeping 300ms in the first sink.
The function that you pass to a second parameter list of FlowGraph.closed determines what materialized value is returned when you run the flow. So when you pass in (x1,x2)=>x1 you return a future which is completed when the first sink gets all elements and then the callback on that future shuts down the actor system without the second sink having a chance receiving all of the elements.
Instead, you should get both futures out and shutdown the system only when both futures are completed.
You can actually see how this approach is used in some of the akka-stream tests here.