I'm working with a Camel flow that uses a Netty TCP socket consumer to receive messages from a client program (which is outside of my control). The client should be opening a socket, sending us one message, then closing the socket, but we've been seeing cases where instead of one message Camel is "splitting" the text stream into two parts and trying to process them separately.
So I'm trying to figure out, since you can re-use the same socket for multiple Camel messages, but TCP sockets don't have a built-in concept of "frames" or a standard for message delimiters, how does Camel decide that a complete message has been received and is ready to process? I haven't been able to find a documented answer to this in the Netty component docs (https://camel.apache.org/components/3.15.x/netty-component.html), although maybe I'm missing something.
From playing around with a test script, it seems like one answer is "Camel assumes a message is complete and should be processed if it goes more than 1ms without receiving any input on the socket". Is this a correct statement, and if so is this behavior documented anywhere? Is there any way to change or configure this behavior? Really what I would prefer is for Camel to wait for an ETX character (or a much longer timeout) before processing a message, is that possible to set up?
Here's my test setup:
Camel flow:
from("netty:tcp://localhost:3003")
.log("Received: ${body}");
Python snippet:
DELAY_MS = 3
def send_msg(sock, msg):
print("Sending message: <{}>".format(msg))
if not sock.sendall(msg.encode()) is None:
print("Message failed to send")
time.sleep(DELAY_MS / 1000.0)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
print("Using DELAY_MS: {}".format(str(DELAY_MS)))
s.connect((args.hostname, args.port))
cutoff = int(math.floor(len(args.msg) / 2))
msg1 = args.msg[:cutoff]
send_msg(s, msg1)
msg2 = args.msg[cutoff:]
send_msg(s, msg2)
response = s.recv(1024)
except Exception as e:
print(e)
finally:
s.close()
I can see that with DELAY_MS=1 Camel logs one single message:
2022-02-21 16:54:40.689 INFO 19429 --- [erExecutorGroup] route1 : Received: a long string sent over the socket
But with DELAY_MS=2 it logs two separate messages:
2022-02-21 16:56:12.899 INFO 19429 --- [erExecutorGroup] route1 : Received: a long string sen
2022-02-21 16:56:12.899 INFO 19429 --- [erExecutorGroup] route1 : Received: t over the socket
After doing some more research, it seems like what I need to do is add a delimiter-based FrameDecoder to the decoders list.
Setting it up like this:
from("netty:tcp://localhost:3003?sync=true"
+ "&decoders=#frameDecoder,#stringDecoder"
+ "&encoders=#stringEncoder")
where frameDecoder is provided by
#Bean
ChannelHandlerFactory frameDecoder() {
ByteBuf[] ETX_DELIM = new ByteBuf[] { Unpooled.wrappedBuffer(new byte[] { (char)3 }) };
return ChannelHandlerFactories.newDelimiterBasedFrameDecoder(1024, ETX_DELIM,
false, "tcp");
}
seems to do the trick.
On the flip side though, it seems like this will hang indefinitely (or until lower-level TCP timeouts kick in?) if an ETX frame is not received, and I can't figure out any way to set a timeout on the decoder, so would still be eager for input if anyone knows how to do that.
I think the default "timeout" behavior I was seeing might've just been an artifact of Netty's read loop speed -- How does netty determine when a read is complete?
Related
I'm trying to create a server that sets up a Unix socket and listens for clients which send/receive data. I've made a small repository to recreate the problem.
The server runs and it can receive data from the clients that connect, but I can't get the server response to be read from the client without an error on the server.
I have commented out the offending code on the client and server. Uncomment both to recreate the problem.
When the code to respond to the client is uncommented, I get this error on the server:
thread '' panicked at 'called Result::unwrap() on an Err value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }', src/main.rs:77:42
MRE Link
Your code calls set_read_timeout to set the timeout on the socket. Its documentation states that on Unix it results in a WouldBlock error in case of timeout, which is precisely what happens to you.
As to why your client times out, the likely reason is that the server calls stream.read_to_string(&mut response), which reads the stream until end-of-file. On the other hand, your client calls write_all() followed by flush(), and (after uncommenting the offending code) attempts to read the response. But the attempt to read the response means that the stream is not closed, so the server will wait for EOF, and you have a deadlock on your hands. Note that none of this is specific to Rust; you would have the exact same issue in C++ or Python.
To fix the issue, you need to use a protocol in your communication. A very simple protocol could consist of first sending the message size (in a fixed format, perhaps 4 bytes in length) and only then the actual message. The code that reads from the stream would do the same: first read the message size and then the message itself. Even better than inventing your own protocol would be to use an existing one, e.g. to exchange messages using serde.
My client side cannot recv the two messages if the sender sends too quickly.
sender.py
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', int(port)))
sock.listen(1)
conn, addr = sock.accept()
#conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
# sends message 1 and message 2
conn.send(pickle.dumps(message1))
#time.sleep(1)
conn.send(pickle.dumps(message2))
Where both message 1 and message 2 are pickled objects.
client.py
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((ip,int(port)))
message1 = pickle.loads(sock.recv(1024))
print(message1)
message2 = pickle.loads(sock.recv(1024))
When i run this code as it is, i am able to print out message1 but i am unable to receive message2 from the sender. The socket blocks at message2.
Also, if i uncomment time.sleep(1) in my sender side code, i am able to receive both messages just fine. Not sure what the problem is. I tried to flush my TCP buffer everytime by setting TCP_NODELAY but that didnt work. Not sure what is actually happening ? How would i ensure that i receive the two messages
Your code assumes that each send on the server side will match a recv on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv will already contain data from the second send which might be simply discarded by pickle.loads as junk after the pickled data. The second recv will only receive the remaining data (or just block since all data where already received) so pickle.loads will fail.
The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)) when sending and for reading first read the fixed-length size value and then read the message with the given size.
i'm new to Netty and intend to create a tcp socket server which reads the info of each client and replies back towards client before processing requests immediately ,i.e. sort of an acknowledgement towards client as and when the message enters overriden channelRead method of ChannelInboundHandlerAdapter class.
Please guide me in the above specified objective.
i'm currently trying the basic netty 4.1.4 echo server example however i wanted server to send back acknowledgement to the client so i updated channelread method as follows :
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
ctx.write(msg);
ChannelFuture cf = ctx.channel().write("FROM SERVER");
System.out.println("Channelfuture is "+cf);
}
and the output obtained was as follows:
Channelfuture is DefaultChannelPromise#3f4ee9dd(failure: java.lang.UnsupportedOperationException: unsupported message type: String (expected: ByteBuf, FileRegion))
I understand the error that it is expecting bytebuf but how do i achieve it? also, whether this method would be able to send out acknowledgement towards client
You can use String.getBytes(Charset) and Unpooled.wrappedBuffer(byte[]) to convert to ByteBuf.
ChannelFuture cf = ctx.channel()
.write(Unpooled.wrappedBuffer("FROM SERVER".getBytes(CharsetUtil.UTF_8)));
Also note that ctx.channel().write(...); may not be what you want. Consider ctx.write(...); instead. The difference is that if your handler is a ChannelDuplexHandler it would receive a write event when you do channel().write(). Using ctx instead of channel will send the write out from your handlers point in the pipeline instead of from the end of the pipeline, which is usually what you want.
I having the following code to create a client socket to send/receive data:
val socket:Socket = new Socket(InetAddress.getByName("127.0.0.1"), 7777)
val inputStream = socket.getInputStream()
val bufferSource = new BufferedSource(inputStream)
val out = new PrintStream(socket.getOutputStream())
var data = "Hello Everyone"
out.println(data)
out.flush()
***socket.shutdownOutput()***
val in = bufferSource.getLines()
if (in.hasNext) {
println(in.next())
}
If I don't run socket.shutdownOutput(), I won't get the data from server,
because Server side is still waiting the input. Therefore I have to shutdown the outputStream.
But if shutdown the output, it can not be reopen. So I have to create a new socket for sending new data.
That caused sending one record needs to create a new socket. This is really awkward.
Is there any other way to tell the server that the output already finished without shutting down the output please.
Thanks in advance!
The problem is that the server doesn't know when to stop reading and process and reply.
What you need here is an application-level protocol that would dictate how server and clients are to communicate - what is a command, how a response to be formatted, etc.
This could be a line-oriented protocol - each new line represents a message (in general the message delimiter could be any other character sequence not appearing in the messages).
Or it could be fixed length messages; or messages pre-pended with message length (or type) to let the other side know how much data yo expect.
When I get one message from a non akka client through TCP socket, I need to reply as three messages. In the following sample given below, only the first one goes through properly to the sender (the TCP client which is non AKKA). The rest of the two goes to dead letter. Any idea? Thanks in advance.
object TcpExample {
def main(args: Array[String]): Unit = {
val system = ActorSystem("some-system")
val tcpConsumer = system.actorOf(Props[TcpConsumer])
}
class TcpConsumer extends Consumer {
def endpointUri = "mina2:tcp://localhost:6200?textline=true"
def receive = {
case msg: CamelMessage => {
sender ! msg.bodyAs[String]
sender ! msg.bodyAs[String] // This goes to dead letter
sender ! msg.bodyAs[String] // This goes to dead letter
}
case msg: Failure => sender ! msg
}
}
Without knowing too much about the internals of the akka/camel integration, let me try and demonstrate what's happening here. First, as I mentioned in my comment, the sender in your actor does not directly refer to the TCP client that is on the other side of the system. It's lower level than that; it's whatever ActorRef sent your Consumer the CamelMessage in the first place. So what actor is that? Let me try and explain what I think is happening.
When you set up tcp based a camel consumer, based on the endpointUri there will be a piece of code (from Camel) that is going to bind to the host and port from the endpointUri.
When a new connection request comes in (based on an external client opening a connection to that socket), some sort of actor is probably spun up to handle that individual connection. So there will be 1-n "connection handler" actor instances matching the number of open connections.
When a message comes inbound, it more than likely goes through that connection handler actor. From there, it is either being sent to your consumer via ask (?), or another short lived actor is being spun up to handle that individual message.
Either way, next stop is your consumer, where it's receive function gets hit with a CamelMessage representing the payload from the message sent from the remove client. When this happens, the actors sender is still whatever sent the message in step 3.
Your consumer will now send a message back to the sender and then from there it will eventually be routed back to the connection handler for that connection. In there, it will write back to the socket, in a conversational state. One message in, one message out.
I think your problem is that you are breaking the "one in, one out" paradigm here. When you get your CamelMessage, you are only supposed to respond to that message once, which will evantually trickle back up to the TCP client on the other end of the socket. I don't think the framework expects another response, and that's why you see deadletters for the other two responses.
So this begs the question, what scenario do you have that requires a "1 in, 3 out" paradigm vs the expected "1 in, one out" one that the framework seems to expect?