Spray Client for Jenkins hangs waiting for large response - scala

I'm writing a Spray client for the Jenkins build server. I can query a project with up to 18 build results but a project with 31 has the behaviour that my client never receives the response. I can do this successfully with curl so I'm at a loss for what to look at next.
I do not believe this is about a chunked transfer encoding as I don't see that header in the successful response - although I may be mistaken in my understand of what should happen.
object JenkinsBuildStatus extends App {
implicit val system = ActorSystem("JenkinsBuildStatus")
import system.dispatcher // execution context for futures
val log = Logging(system, getClass)
val pipeline: HttpRequest => Future[HttpResponse] = (
addCredentials(BasicHttpCredentials("username","foobar"))
~> addHeader("Accept", MediaRanges.`*/*`.value)
~> addHeader("User-Agent", "curl/7.37.1")
~> logRequest(log)
~> sendReceive
~> logResponse(log)
)
val response: Future[HttpResponse] = pipeline(Get("https://jenkinsbuilds.corp.com/jenkins/job/Project-Java1.8/api/json/"))
response onComplete {
case Success(data) => {
log.info (data.toString())
shutdown()
}
case Failure(error) => {
log.error(error, "Failure with web client")
shutdown()
}
}
def shutdown(): Unit = {
IO(Http).ask(Http.CloseAll)(1.second).await
system.shutdown()
}
}

Related

Why am I getting this timeout during unit test of akka-stream?

I have an akka-gRPC service BiDirectional stream and I am testing it on a unit test. The service has uses akka-stream and I use the TestSink.probe to test the reply message. I am receiving back the messages from the service, but there is an error related to timeout that I cannot figure out what is the reason. This is the test:
object GreeterServiceConf {
// important to enable HTTP/2 in server ActorSystem's config
val configServer = ConfigFactory.parseString("akka.http.server.preview.enable-http2 = on")
.withFallback(ConfigFactory.defaultApplication())
val configString2 =
"""
|akka.grpc.client {
| "helloworld.GreeterService" {
| host = 127.0.0.1
| port = 8080
| }
|}
|""".stripMargin
val configClient = ConfigFactory.parseString(configString2)
}
class GreeterServiceImplSpec extends TestKit(ActorSystem("GreeterServiceImplSpec", ConfigFactory.load(GreeterServiceConf.configServer)))
with AnyWordSpecLike
with BeforeAndAfterAll
with Matchers
with ScalaFutures {
implicit val patience: PatienceConfig = PatienceConfig(scaled(5.seconds), scaled(100.millis))
// val testKit = ActorTestKit(conf)
val serverSystem: ActorSystem = system
val bound = new GreeterServer(serverSystem).run()
// make sure server is bound before using client
bound.futureValue
implicit val clientSystem: ActorSystem = ActorSystem("GreeterClient", ConfigFactory.load(GreeterServiceConf.configClient))
val client = GreeterServiceClient(
GrpcClientSettings
.fromConfig("helloworld.GreeterService")
.withTls(false)
)
override def afterAll: Unit = {
TestKit.shutdownActorSystem(system)
TestKit.shutdownActorSystem(clientSystem)
}
"GreeterService" should {
"reply to multiple requests" in {
import GreeterServiceData._
val names = List("John", "Michael", "Simone")
val expectedReply: immutable.Seq[HelloReply] = names.map { name =>
HelloReply(s"Hello, $name -> ${mapHelloReply.getOrElse(name, "this person does not exist =(")}")
}
val requestStream: Source[HelloRequest, NotUsed] = Source(names).map(name => HelloRequest(name))
val responseStream: Source[HelloReply, NotUsed] = client.sayHelloToAll(requestStream)
val sink = TestSink.probe[HelloReply]
val replyStream = responseStream.runWith(sink)
replyStream
.requestNext(HelloReply(s"Hello, John -> I killed Java"))
.requestNext(HelloReply(s"Hello, Michael -> We are the Jacksons 5"))
.requestNext(HelloReply(s"Hello, Simone -> I have found a job to work with Scala =)")) // THIS IS THE LINE 122 ON THE ERROR
// .request(3)
// .expectNextUnorderedN(expectedReply) // I also tested this but it did not work
.expectComplete()
}
}
}
The error is:
assertion failed: timeout (3 seconds) during expectMsg while waiting
for OnComplete java.lang.AssertionError: assertion failed: timeout (3
seconds) during expectMsg while waiting for OnComplete at
scala.Predef$.assert(Predef.scala:223) at
akka.testkit.TestKitBase.expectMsg_internal(TestKit.scala:459) at
akka.testkit.TestKitBase.expectMsg(TestKit.scala:436) at
akka.testkit.TestKitBase.expectMsg$(TestKit.scala:436) at
akka.testkit.TestKit.expectMsg(TestKit.scala:969) at
akka.stream.testkit.TestSubscriber$ManualProbe.expectComplete(StreamTestKit.scala:479)
at
com.example.helloworld.GreeterServiceImplSpec.$anonfun$new$5(GreeterServiceImplSpec.scala:121)
I got it to work based on the project akka-grpc-quickstart-scala.g8. I am executing runForeach to run the graph and have a materialized Sink on the response stream. Then, when the response is done I am doing an assert inside the Future[Done].
"reply to multiple requests" in {
import GreeterServiceData._
import system.dispatcher
val names = List("John", "Martin", "Michael", "UnknownPerson")
val expectedReplySeq: immutable.Seq[HelloReply] = names.map { name =>
HelloReply(s"Hello, $name -> ${mapHelloReply.getOrElse(name, "this person does not exist =(")}")
}
// println(s"expectedReplySeq: ${expectedReplySeq.foreach(println)}")
val requestStream: Source[HelloRequest, NotUsed] = Source(names).map(name => HelloRequest(name))
val responseStream: Source[HelloReply, NotUsed] = client.sayHelloToAll(requestStream)
val done: Future[Done] = responseStream.runForeach { reply: HelloReply =>
// println(s"got streaming reply: ${reply.message}")
assert(expectedReplySeq.contains(reply))
}
// OR USING Sink.foreach[HelloReply])(Keep.right)
val sinkHelloReply = Sink.foreach[HelloReply] { e =>
println(s"element: $e")
assert(expectedReplySeq.contains(e))
}
responseStream.toMat(sinkHelloReply)(Keep.right).run().onComplete {
case Success(value) => println(s"done")
case Failure(exception) => println(s"exception $exception")
}
}
Just to keep the reference of the whole code, the GreeterServiceImplSpec class is here.

Akka streams Source.repeat stops after 100 requests

I am working on the below stream processing system to grab frames from one source, process, and send to another. I'm using a combination of akka-streams and akka-http through their scapa api. The pipeline is very short but I can't seem to locate where the system decides to stop after precisely 100 requests to the endpoint.
object frameProcessor extends App {
implicit val system: ActorSystem = ActorSystem("VideoStreamProcessor")
val decider: Supervision.Decider = _ => Supervision.Restart
implicit val materializer: ActorMaterializer = ActorMaterializer()
implicit val dispatcher: ExecutionContextExecutor = system.dispatcher
val http = Http(system)
val sourceConnectionFlow: Flow[HttpRequest, HttpResponse, Future[Http.OutgoingConnection]] = http.outgoingConnection(sourceUri)
val byteFlow: Flow[HttpResponse, Future[ByteString], NotUsed] =
Flow[HttpResponse].map(_.entity.dataBytes.runFold(ByteString.empty)(_ ++ _))
Source.repeat(HttpRequest(uri = sourceUri))
.via(sourceConnectionFlow)
.via(byteFlow)
.map(postFrame)
.runWith(Sink.ignore)
.onComplete(_ => system.terminate())
def postFrame(imageBytes: Future[ByteString]): Unit = {
imageBytes.onComplete{
case Success(res) => system.log.info(s"post frame. ${res.length} bytes")
case Failure(_) => system.log.error("failed to post image!")
}
}
}
Fore reference, I'm using akka-streams version 2.5.19 and akka-http version 10.1.7. No error is thrown, no error codes on the source server where the frames come from, and the program exits with error code 0.
My application.conf is as follows:
logging = "DEBUG"
Always 100 units processed.
Thanks!
Edit
Added logging to the stream like so
.onComplete{
case Success(res) => {
system.log.info(res.toString)
system.terminate()
}
case Failure(res) => {
system.log.error(res.getMessage)
system.terminate()
}
}
Received a connection reset exception but this is inconsistent. The stream completes with Done.
Edit 2
Using .mapAsync(1)(postFrame) I get the same Success(Done) after precisely 100 requests. Additionally, when I check the nginx server access.log and error.log there are only 200 responses.
I had to modify postFrame as follows to run mapAsync
def postFrame(imageBytes: Future[ByteString]): Future[Unit] = {
imageBytes.onComplete{
case Success(res) => system.log.info(s"post frame. ${res.length} bytes")
case Failure(_) => system.log.error("failed to post image!")
}
Future(Unit)
}
I believe I have found the answer on on the Akka docs using delayed restarts with a backoff operator. Instead of sourcing direct from an unstable remote connection, I use RestartSource.withBackoff and not RestartSource.onFailureWithBackoff. The modified stream looks like;
val restartSource = RestartSource.withBackoff(
minBackoff = 100.milliseconds,
maxBackoff = 1.seconds,
randomFactor = 0.2
){ () =>
Source.single(HttpRequest(uri = sourceUri))
.via(sourceConnectionFlow)
.via(byteFlow)
.mapAsync(1)(postFrame)
}
restartSource
.runWith(Sink.ignore)
.onComplete{
x => {
println(x)
system.terminate()
}
}
I was not able to find the source of the problem but it seems this will work.

Play framework akka stream websocket handling message get sent to deadletters

I'm trying to wrap my head around akka streams and the way to handle web sockets, but some things are quite clear to me.
For starters, I'm trying to accomplish one-way communication from some client to the server and communication between the same server and some other client.
client1 -----> Server <------> client2
I was looking at the example provided here.
The resulting code looks something like this:
1) starting with the controller
class Test #Inject()(#Named("connManager") myConnectionsManager: ActorRef, cc: ControllerComponents)
(implicit val actorSystem: ActorSystem,
val mat: Materializer,
implicit val executionContext: ExecutionContext)
extends AbstractController(cc) {
private def wsFutureFlow(id: String): Future[Flow[String, String, NotUsed]] = {
implicit val timeout: Timeout = Timeout(5.seconds)
val future = myConnectionsManager ? CreateRemote(id)
val futureFlow = future.mapTo[Flow[String, String, NotUsed]]
futureFlow
}
private def wsFutureLocalFlow: Future[Flow[String, String, NotUsed]] = {
implicit val timeout: Timeout = Timeout(5.seconds)
val future = myConnectionsManager ? CreateLocal
val futureFlow = future.mapTo[Flow[String, String, NotUsed]]
futureFlow
}
def ws: WebSocket = WebSocket.acceptOrResult[String, String] {
rh =>
wsFutureFlow(rh.id.toString).map { flow =>
Right(flow)
}
}
def wsLocal: WebSocket = WebSocket.acceptOrResult[String, String] {
_ =>
wsFutureLocalFlow.map { flow =>
Right(flow)
}
}
}
As for the connection manager actor. That would be the equivalent of the UserParentActor from the example.
class MyConnectionsManager #Inject()(childFactory: MyTestActor.Factory)
(implicit ec: ExecutionContext, mat: Materializer) extends Actor with InjectedActorSupport {
import akka.pattern.{ask, pipe}
implicit val timeout: Timeout = Timeout(2.seconds)
override def receive: Receive = {
case CreateRemote(x) =>
val child = injectedChild(childFactory(), s"remote-$x")
context.watch(child)
privatePipe(child)
case CreateLocal =>
val child = injectedChild(childFactory(), "localConnection")
context.become(onLocalConnected(child))
privatePipe(child)
case Terminated(child) =>
println(s"${child.path.name} terminated...")
}
def onLocalConnected(local: ActorRef): Receive = {
case CreateRemote(x) =>
val child = injectedChild(childFactory(), s"remote-$x")
context.watch(child)
privatePipe(child)
case x: SendToLocal => local ! x
}
private def privatePipe(child: ActorRef) = {
val future = (child ? Init).mapTo[Flow[String, String, _]]
pipe(future) to sender()
() // compiler throws exception without this: non-unit value discarded
}
}
And the MyTestActor looks like this:
class MyTestActor #Inject()(implicit mat: Materializer, ec: ExecutionContext) extends Actor {
val source: Source[String, Sink[String, NotUsed]] = MergeHub.source[String]
.recoverWithRetries(-1, { case _: Exception => Source.empty })
private val jsonSink: Sink[String, Future[Done]] = Sink.foreach { json =>
println(s"${self.path.name} got message: $json")
context.parent ! SendToLocal(json)
}
private lazy val websocketFlow: Flow[String, String, NotUsed] = {
Flow.fromSinkAndSourceCoupled(jsonSink, source).watchTermination() { (_, termination) =>
val name = self.path.name
termination.foreach(_ => context.stop(self))
NotUsed
}
}
def receive: Receive = {
case Init =>
println(s"${self.path.name}: INIT")
sender ! websocketFlow
case SendToLocal(x) =>
println(s"Local got from remote: $x")
case msg: String => sender ! s"Actor got message: $msg"
}
}
What I don't understand, apart from how sinks and sources actually connect to the actors, is the following. When I start up my system, I send a few messages to the actor. However, after I close the connection to an actor named remote, and continue sending messages to the one called "localConnection", the messages get sent to DeadLetters:
[info] Done compiling.
[info] 15:49:20.606 - play.api.Play - Application started (Dev)
localConnection: INIT
localConnection got message: test data
Local got from remote: test data
localConnection got message: hello world
Local got from remote: hello world
remote-133: INIT
remote-133 got message: hello world
Local got from remote: hello world
remote-133 got message: hello from remote
Local got from remote: hello from remote
[error] 15:50:24.449 - a.a.OneForOneStrategy - Monitored actor [Actor[akka://application/user/connManager/remote-133#-998945083]] terminated
akka.actor.DeathPactException: Monitored actor [Actor[akka://application/user/connManager/remote-133#-998945083]] terminated
deadLetters got message: hello local
I assume this is because of the exception thrown... Can anyone explain to me as to why the message gets sent to DeadLetters?
Apart from that, I would like to know why I keep getting a compiler exception without the "()" returned at the end of privatePipe?
Also, should I be doing anything differently?
I realised that the exception was being thrown because I forgot to handle the Terminated message in the new behaviour of the MyConnectionsManager actor.
def onLocalConnected(local: ActorRef): Receive = {
case CreateRemote(x) =>
val child = injectedChild(childFactory(), s"remote-$x")
context.watch(child)
privatePipe(child)
case Terminated(child) => println(s"${child.path.name} terminated...")
case x: SendToLocal => local ! x
}
It seems to be working now.

Akka-http: connect to websocket on localhost

I am trying to connect to some server through websocket on localhost. When I try to do it in JS by
ws = new WebSocket('ws://localhost:8137');
it succeeds. However, when I use akka-http and akka-streams I get "connection failed" error.
object Transmitter {
implicit val system: ActorSystem = ActorSystem()
implicit val materializer: ActorMaterializer = ActorMaterializer()
import system.dispatcher
object Rec extends Actor {
override def receive: Receive = {
case TextMessage.Strict(msg) =>
Log.info("Recevied signal " + msg)
}
}
// val host = "ws://echo.websocket.org"
val host = "ws://localhost:8137"
val sink: Sink[Message, NotUsed] = Sink.actorRef[Message](system.actorOf(Props(Rec)), PoisonPill)
val source: Source[Message, NotUsed] = Source(List("test1", "test2") map (TextMessage(_)))
val flow: Flow[Message, Message, Future[WebSocketUpgradeResponse]] =
Http().webSocketClientFlow(WebSocketRequest(host))
val (upgradeResponse, closed) =
source
.viaMat(flow)(Keep.right) // keep the materialized Future[WebSocketUpgradeResponse]
.toMat(sink)(Keep.both) // also keep the Future[Done]
.run()
val connected: Future[Done.type] = upgradeResponse.flatMap { upgrade =>
if (upgrade.response.status == StatusCodes.SwitchingProtocols) {
Future.successful(Done)
} else {
Future.failed(new Exception(s"Connection failed: ${upgrade.response.status}")
}
}
def test(): Unit = {
connected.onComplete(Log.info)
}
}
It works completely OK with ws://echo.websocket.org.
I think attaching code of my server is reasonless, because it works with JavaScript client and problem is only with connection, however if you would like to look at it I may show it.
What am I doing wrong?
I have tested your client implementation with a websocket server from akka documentation,
and I did not get any connection error. Your websocket client connects successfully. That is why I am guessing the problem is with your server implementation.
object WebSocketServer extends App {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
import Directives._
val greeterWebSocketService = Flow[Message].collect {
case tm: TextMessage => TextMessage(Source.single("Hello ") ++ tm.textStream)
}
val route =
get {
handleWebSocketMessages(greeterWebSocketService)
}
val bindingFuture = Http().bindAndHandle(route, "localhost", 8137)
println(s"Server online at http://localhost:8137/\nPress RETURN to stop...")
StdIn.readLine()
import system.dispatcher // for the future transformations
bindingFuture
.flatMap(_.unbind()) // trigger unbinding from the port
.onComplete(_ => system.terminate()) // and shutdown when done
}
By the way, I noticed that your actor's receive method does not cover all possible messages. According to that akka issue,
every message, even very small, can end up as Streamed. If you want to print all text messages a better implementation of the actor would be:
object Rec extends Actor {
override def receive: Receive = {
case TextMessage.Strict(text) ⇒ println(s"Received signal $text")
case TextMessage.Streamed(textStream) ⇒ textStream.runFold("")(_ + _).foreach(msg => println(s"Received streamed signal: $msg"))
}
}
Please find a working project on my github.
I found the solution: the server I used was running on IPv6 (as ::1), but akka-http treats localhost as 127.0.0.1 and ignores ::1. I had to rewrite server to force it to use IPv4 and it worked.

Echo simple HTTP server with Akka Http in Scala

I am developing a simple HTTP server using Akka-Http in Scala.
My code is as given below:
object HttpServer extends App {
override def main(args: Array[String]): Unit = {
implicit val system = ActorSystem("my-system")
implicit val materializer = ActorMaterializer()
implicit val executionContext = system.dispatcher
val route : Route = post {
path("echo") {
val json = ???
complete((StatusCodes.OK, json))
}
}
val bindingFuture = Http().bindAndHandle(route, "localhost", 8080)
println(s"Server online at http://localhost:8080/\nPress RETURN to stop...")
StdIn.readLine()
bindingFuture.flatMap(_.unbind())
port.onComplete(_ => system.terminate())
}
}
I do not know Scala enough yet. For that, I need some help.
I do not know how I can get JSON from Http POST body to give back that json to client.
You only need to add an extractor to your route definition:
val route : Route = post {
path("echo") {
entity(as[String]) { json =>
complete(json)
}
}
Note that you don't need to set the status code explicitly, as akka-http will automatically set status 200 OK for you when you pass a value to complete