I have a java jar that generates the tcp stream on particular port.
I cam run the using java command like java -jar runner.jar and this start producing the stream of message on port 8888.
When I do nc -l 8888 I can see the messages.
I want to read this stream using scala and another framework or tool like akka, akka-stream.
Can anyone help me understand the best tool, framework or any other technique to read this tcp stream.
I tried using akka stream with following code :-
implicit val system = ActorSystem()
implicit val mater = ActorMaterializer() val ss = Tcp().outgoingConnection("127.0.0.1", 8888)
.to(Sink.foreach(println(_)))
Source.empty.to(ss).run()
I also tried
Tcp().outgoingConnection(new InetSocketAddress("127.0.0.1", 8888))
.runWith(Source.maybe[ByteString], Sink.foreach(bs => println(bs.utf8String)))
This doesn't work.
I only need to read the messages and process on my own.
Thanks
As I understand you want to setup TCP server, here is example of TCP Echo using akka streams
def server(system: ActorSystem, address: String, port: Int): Unit = {
implicit val sys = system
import system.dispatcher
implicit val materializer = ActorMaterializer()
val handler = Sink.foreach[Tcp.IncomingConnection] { conn =>
println("Client connected from: " + conn.remoteAddress)
conn handleWith Flow[ByteString]
}
val connections = Tcp().bind(address, port)
val binding = connections.to(handler).run()
binding.onComplete {
case Success(b) =>
println("Server started, listening on: " + b.localAddress)
case Failure(e) =>
println(s"Server could not bind to $address:$port: ${e.getMessage}")
system.terminate()
}
}
Related
I'm quite new in Akka Stream and I'd like to learn how handle a TCP socket for a project of mine. I took this piece of code from the Akka Stream official documentation.
import akka.stream.scaladsl.Framing
val connections: Source[IncomingConnection, Future[ServerBinding]] =
Tcp().bind(host, port)
connections.runForeach { connection =>
println(s"New connection from: ${connection.remoteAddress}")
val echo = Flow[ByteString]
.via(Framing.delimiter(ByteString("\n"), maximumFrameLength = 256, allowTruncation = true))
.map(_.utf8String)
.map(_ + "!!!\n")
.map(ByteString(_))
connection.handleWith(echo)
}
If I connect from the terminal using netcat I can see that the Akka Stream TCP socket works as expected. I also found out that If I need to close the connection using an user message, I can use a takeWhile as follow
import akka.stream.scaladsl.Framing
val connections: Source[IncomingConnection, Future[ServerBinding]] =
Tcp().bind(host, port)
connections.runForeach { connection =>
println(s"New connection from: ${connection.remoteAddress}")
val echo = Flow[ByteString]
.via(Framing.delimiter(ByteString("\n"), maximumFrameLength = 256, allowTruncation = true))
.map(_.utf8String)
.takeWhile(_.toLowerCase.trim != "exit") // < - - - - - - HERE
.map(_ + "!!!\n")
.map(ByteString(_))
connection.handleWith(echo)
}
What I can't find is how to manage a socket closed by a CMD + C action. Akka Stream use Akka.io to manage the TCP connection internally, so it must send some of its PeerClose messages when the socket is closed. So, my understanding of Akka.io tells me that I should receive a feedback from the socket closing but I can't find how to do that with Akka Stream. Is there a way to manage that ?
connection.handleWith(echo) is syntactic sugar for connection.flow.joinMat(echo)(Keep.right).run() which will have the materialized value of echo, which is generally not useful. Flow.via.map.takeWhile has NotUsed as a materialized value, so that's also basically useless. However, you can attach stages to echo which will materialize differently.
One of these is .watchTermination:
connections.runForeach { connection =>
println(s"New connection from: ${connection.remoteAddress}")
val echo: Flow[ByteString, ByteString, Future[Done]] = Flow[ByteString]
.via(Framing.delimiter(ByteString("\n"), maximumFrameLength = 256, allowTruncation = true))
.map(_.utf8String)
.takeWhile(_.toLowerCase.trim != "exit") // < - - - - - - HERE
.map(_ + "!!!\n")
.map(ByteString(_))
// change the materialized value to a Future[Done]
.watchTermination()(Keep.right)
// you may need to have an implicit ExecutionContext in scope, e.g. system.dispatcher,
// if you don't already
connection.handleWith(echo).onComplete {
case Success(_) => println("stream completed successfully")
case Failure(e) => println(e.getMessage)
}
}
This will not distinguish between your side or the remote side closing the connection normally; it will distinguish the stream failing.
I have a very simple Akka WebSocket server that pushes lines from a file to a connected client with an interval of 400ms per line. Everything works fine, except for the fact that the web server seems to buffer messages for about a minute before broadcasting them.
So when a client connects, I see at the server end that every 400ms a line is read and pushed to the Sink, but on the client side I get nothing for a minute and then a burst of about 150 messages (corresponding to a minute of messages).
Is there a setting that I'm overlooking?
object WebsocketServer extends App {
implicit val actorSystem = ActorSystem("WebsocketServer")
implicit val materializer = ActorMaterializer()
implicit val executionContext = actorSystem.dispatcher
val file = Paths.get("websocket-server/src/main/resources/EURUSD.txt")
val fileSource =
FileIO.fromPath(file)
.via(Framing.delimiter(ByteString("\n"), Int.MaxValue))
val delayedSource: Source[Strict, Future[IOResult]] =
fileSource
.map { line =>
Thread.sleep(400)
println(line.utf8String)
TextMessage(line.utf8String)
}
def route = path("") {
extractUpgradeToWebSocket { upgrade =>
complete(upgrade.handleMessagesWithSinkSource(
Sink.ignore,
delayedSource)
)
}
}
val bindingFuture = Http().bindAndHandle(route, "localhost", 8080)
bindingFuture.onComplete {
case Success(binding) ⇒
println(s"Server is listening on ws://localhost:8080")
case Failure(e) ⇒
println(s"Binding failed with ${e.getMessage}")
actorSystem.terminate()
}
}
So the approach with Thread.sleep(400) was wrong. I should've used the .throttle mechanic on sources:
val delayedSource: Source[Strict, Future[IOResult]] =
fileSource
.throttle(elements = 1, per = 400.millis)
.map { line =>
println(line.utf8String)
TextMessage(line.utf8String)
}
This fixed the issue.
I am trying to connect to some server through websocket on localhost. When I try to do it in JS by
ws = new WebSocket('ws://localhost:8137');
it succeeds. However, when I use akka-http and akka-streams I get "connection failed" error.
object Transmitter {
implicit val system: ActorSystem = ActorSystem()
implicit val materializer: ActorMaterializer = ActorMaterializer()
import system.dispatcher
object Rec extends Actor {
override def receive: Receive = {
case TextMessage.Strict(msg) =>
Log.info("Recevied signal " + msg)
}
}
// val host = "ws://echo.websocket.org"
val host = "ws://localhost:8137"
val sink: Sink[Message, NotUsed] = Sink.actorRef[Message](system.actorOf(Props(Rec)), PoisonPill)
val source: Source[Message, NotUsed] = Source(List("test1", "test2") map (TextMessage(_)))
val flow: Flow[Message, Message, Future[WebSocketUpgradeResponse]] =
Http().webSocketClientFlow(WebSocketRequest(host))
val (upgradeResponse, closed) =
source
.viaMat(flow)(Keep.right) // keep the materialized Future[WebSocketUpgradeResponse]
.toMat(sink)(Keep.both) // also keep the Future[Done]
.run()
val connected: Future[Done.type] = upgradeResponse.flatMap { upgrade =>
if (upgrade.response.status == StatusCodes.SwitchingProtocols) {
Future.successful(Done)
} else {
Future.failed(new Exception(s"Connection failed: ${upgrade.response.status}")
}
}
def test(): Unit = {
connected.onComplete(Log.info)
}
}
It works completely OK with ws://echo.websocket.org.
I think attaching code of my server is reasonless, because it works with JavaScript client and problem is only with connection, however if you would like to look at it I may show it.
What am I doing wrong?
I have tested your client implementation with a websocket server from akka documentation,
and I did not get any connection error. Your websocket client connects successfully. That is why I am guessing the problem is with your server implementation.
object WebSocketServer extends App {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
import Directives._
val greeterWebSocketService = Flow[Message].collect {
case tm: TextMessage => TextMessage(Source.single("Hello ") ++ tm.textStream)
}
val route =
get {
handleWebSocketMessages(greeterWebSocketService)
}
val bindingFuture = Http().bindAndHandle(route, "localhost", 8137)
println(s"Server online at http://localhost:8137/\nPress RETURN to stop...")
StdIn.readLine()
import system.dispatcher // for the future transformations
bindingFuture
.flatMap(_.unbind()) // trigger unbinding from the port
.onComplete(_ => system.terminate()) // and shutdown when done
}
By the way, I noticed that your actor's receive method does not cover all possible messages. According to that akka issue,
every message, even very small, can end up as Streamed. If you want to print all text messages a better implementation of the actor would be:
object Rec extends Actor {
override def receive: Receive = {
case TextMessage.Strict(text) ⇒ println(s"Received signal $text")
case TextMessage.Streamed(textStream) ⇒ textStream.runFold("")(_ + _).foreach(msg => println(s"Received streamed signal: $msg"))
}
}
Please find a working project on my github.
I found the solution: the server I used was running on IPv6 (as ::1), but akka-http treats localhost as 127.0.0.1 and ignores ::1. I had to rewrite server to force it to use IPv4 and it worked.
I try to redirect/forward a TCP stream to another Sink with Akka 2.4.3.
The program should open a server socket, listen for incoming connections and then consume the tcp stream. Our sender does not expect/accept replies from us so we never send back anything - we just consume the stream. After framing the tcp stream we need to transform the bytes into something more useful and send it to the Sink.
I tried the following so far but i struggle especially with the part how to not sending tcp packets back to the sender and to properly connect the Sink.
import scala.util.Failure
import scala.util.Success
import akka.actor.ActorSystem
import akka.event.Logging
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.Sink
import akka.stream.scaladsl.Tcp
import akka.stream.scaladsl.Framing
import akka.util.ByteString
import java.nio.ByteOrder
import akka.stream.scaladsl.Flow
object TcpConsumeOnlyStreamToSink {
implicit val system = ActorSystem("stream-system")
private val log = Logging(system, getClass.getName)
//The Sink
//In reality this is of course a real Sink doing some useful things :-)
//The Sink accept types of "SomethingMySinkUnderstand"
val mySink = Sink.ignore;
def main(args: Array[String]): Unit = {
//our sender is not interested in getting replies from us
//so we just want to consume the tcp stream and never send back anything to the sender
val (address, port) = ("127.0.0.1", 6000)
server(system, address, port)
}
def server(system: ActorSystem, address: String, port: Int): Unit = {
implicit val sys = system
import system.dispatcher
implicit val materializer = ActorMaterializer()
val handler = Sink.foreach[Tcp.IncomingConnection] { conn =>
println("Client connected from: " + conn.remoteAddress)
conn handleWith Flow[ByteString]
//this is neccessary since we use a self developed tcp wire protocol
.via(Framing.lengthField(4, 0, 65532, ByteOrder.BIG_ENDIAN))
//here we want to map the raw bytes into something our Sink understands
.map(msg => new SomethingMySinkUnderstand(msg.utf8String))
//here we like to connect our Sink to the Tcp Source
.to(mySink) //<------ NOT COMPILING
}
val tcpSource = Tcp().bind(address, port)
val binding = tcpSource.to(handler).run()
binding.onComplete {
case Success(b) =>
println("Server started, listening on: " + b.localAddress)
case Failure(e) =>
println(s"Server could not bind to $address:$port: ${e.getMessage}")
system.terminate()
}
}
class SomethingMySinkUnderstand(x:String) {
}
}
Update: Add this to your build.sbt file to get necessary deps
libraryDependencies += "com.typesafe.akka" % "akka-stream_2.11" % "2.4.3"
handleWith expects a Flow, i.e. a box with an unconnected inlet and an unconnected outlet. You effectively provide a Source, because you connected the Flow with a Sink by using the to operation.
I think you could do the following:
conn.handleWith(
Flow[ByteString]
.via(Framing.lengthField(4, 0, 65532, ByteOrder.BIG_ENDIAN))
.map(msg => new SomethingMySinkUnderstand(msg.utf8String))
.alsoTo(mySink)
.map(_ => ByteString.empty)
.filter(_ => false) // Prevents sending anything back
)
Alternative (and in my view cleaner) way to code it (AKKA 2.6.x), that will also emphasize of the fact, that you do not do any outbound flow, would be:
val receivingPipeline = Flow
.via(framing)
.via(decoder)
.to(mySink)
val sendingNothing = Source.never[ByteString]()
conn.handleWith(Flow.fromSinkAndSourceCoupled(receivingPiline, sendingNothing))
Here is the setup: I want to be able to stream messages (jsons converted to bytestrings) from a publisher to a remote server subscriber over a tcp connection.
Ideally, the publisher would be an actor that would receive internal messages, queue them and then stream them to the subscriber server if there is outstanding demand of course. I understood that what is necessary for this is to extend ActorPublisher class in order to onNext() the messages when needed.
My problem is that so far I am able just to send (receive and decode properly) one shot messages to the server opening a new connection each time. I did not manage to get my head around the akka doc and be able to set the proper tcp Flow with the ActorPublisher.
Here is the code from the publisher:
def send(message: Message): Unit = {
val system = Akka.system()
implicit val sys = system
import system.dispatcher
implicit val materializer = ActorMaterializer()
val address = Play.current.configuration.getString("eventservice.location").getOrElse("localhost")
val port = Play.current.configuration.getInt("eventservice.port").getOrElse(9000)
/*** Try with actorPublisher ***/
//val result = Source.actorPublisher[Message] (Props[EventActor]).via(Flow[Message].map(Json.toJson(_).toString.map(ByteString(_))))
/*** Try with actorRef ***/
/*val source = Source.actorRef[Message](0, OverflowStrategy.fail).map(
m => {
Logger.info(s"Sending message: ${m.toString}")
ByteString(Json.toJson(m).toString)
}
)
val ref = Flow[ByteString].via(Tcp().outgoingConnection(address, port)).to(Sink.ignore).runWith(source)*/
val result = Source(Json.toJson(message).toString.map(ByteString(_))).
via(Tcp().outgoingConnection(address, port)).
runFold(ByteString.empty) { (acc, in) ⇒ acc ++ in }//Handle the future
}
and the code from the actor which is quite standard in the end:
import akka.actor.Actor
import akka.stream.actor.ActorSubscriberMessage.{OnComplete, OnError}
import akka.stream.actor.{ActorPublisherMessage, ActorPublisher}
import models.events.Message
import play.api.Logger
import scala.collection.mutable
class EventActor extends Actor with ActorPublisher[Message] {
import ActorPublisherMessage._
var queue: mutable.Queue[Message] = mutable.Queue.empty
def receive = {
case m: Message =>
Logger.info(s"EventActor - message received and queued: ${m.toString}")
queue.enqueue(m)
publish()
case Request => publish()
case Cancel =>
Logger.info("EventActor - cancel message received")
context.stop(self)
case OnError(err: Exception) =>
Logger.info("EventActor - error message received")
onError(err)
context.stop(self)
case OnComplete =>
Logger.info("EventActor - onComplete message received")
onComplete()
context.stop(self)
}
def publish() = {
while (queue.nonEmpty && isActive && totalDemand > 0) {
Logger.info("EventActor - message published")
onNext(queue.dequeue())
}
}
I can provide the code from the subscriber if necessary:
def connect(system: ActorSystem, address: String, port: Int): Unit = {
implicit val sys = system
import system.dispatcher
implicit val materializer = ActorMaterializer()
val handler = Sink.foreach[Tcp.IncomingConnection] { conn =>
Logger.info("Event server connected to: " + conn.remoteAddress)
// Get the ByteString flow and reconstruct the msg for handling and then output it back
// that is how handleWith work apparently
conn.handleWith(
Flow[ByteString].fold(ByteString.empty)((acc, b) => acc ++ b).
map(b => handleIncomingMessages(system, b.utf8String)).
map(ByteString(_))
)
}
val connections = Tcp().bind(address, port)
val binding = connections.to(handler).run()
binding.onComplete {
case Success(b) =>
Logger.info("Event server started, listening on: " + b.localAddress)
case Failure(e) =>
Logger.info(s"Event server could not bind to $address:$port: ${e.getMessage}")
system.terminate()
}
}
thanks in advance for the hints.
My first recommendation is to not write your own queue logic. Akka provides this out-of-the-box. You also don't need to write your own Actor, Akka Streams can provide it as well.
First we can create the Flow that will connect your publisher to your subscriber via Tcp. In your publisher code you only need to create the ActorSystem once and connect to the outside server once:
//this code is at top level of your application
implicit val actorSystem = ActorSystem()
implicit val actorMaterializer = ActorMaterializer()
import actorSystem.dispatcher
val host = Play.current.configuration.getString("eventservice.location").getOrElse("localhost")
val port = Play.current.configuration.getInt("eventservice.port").getOrElse(9000)
val publishFlow = Tcp().outgoingConnection(host, port)
publishFlow is a Flow that will input ByteString data that you want to send to the external subscriber and outputs ByteString data that comes from subscriber:
// data to subscriber ----> publishFlow ----> data returned from subscriber
The next step is the publisher Source. Instead of writing your own Actor you can use Source.actorRef to "materialize" the Stream into an ActorRef. Essentially the Stream will become an ActorRef for us to use later:
//these values control the buffer
val bufferSize = 1024
val overflowStrategy = akka.stream.OverflowStrategy.dropHead
val messageSource = Source.actorRef[Message](bufferSize, overflowStrategy)
We also need a Flow to convert Messages into ByteString
val marshalFlow =
Flow[Message].map(message => ByteString(Json.toJson(message).toString))
Finally we can connect all of the pieces. Since you aren't receiving any data back from the external subscriber we'll ignore any data coming in from the connection:
val subscriberRef : ActorRef = messageSource.via(marshalFlow)
.via(publishFlow)
.runWith(Sink.ignore)
We can now treat this stream as if it were an Actor:
val message1 : Message = ???
subscriberRef ! message1
val message2 : Message = ???
subscriberRef ! message2