I am using InfiniBand/RDMA for this. My client machine is sending a block to the server (via RDMA Send), and I want the client to get back a special 8 byte acknowledgement. I know the server could send it along separately, but that would duplicate work since my understanding for RDMA Send, the NIC is already sending some form of ACK via the completion queue. However, I was looking at InfiniBand docs, the completion queue doesn't seem to have an immediate along with it, just an id (which is itself less than 8 bytes, so there aren't any funny tricks I could play). I was wondering if I could get suggestions on how to do this?
Thank you!
I'm not sure I fully understand the question, but on the client side, a completion queue entry will be generated when the send work request is fully executed. "Fully executed" means that the RDMA-level ACK has been received from the server in your case. The work request ID in that completion entry is indeed 64 bits, and it is common to use it for a pointer to an auxiliary structure holding whatever info you want for the work request.
If you want the server to actively send back some data that it chooses after it receives the send from the client, then I don't think there's any way around having the server post a work request to do that.
Related
Our system generates some messages (unsolicited cancel for example) it needs to send to the other party after a disconnect/connection lost, as soon as the connection recovers.
The problem is that we trigger sending those in onLogon(), but if there's a Resend Request that's too early and we had problems (maybe just because of how is implemented on the other end) when we had too many messages to send (hundreds).
I'm aware that ResendRequest may not come and it is impossible to figure that out without simply waiting, but what would be the best approach for us using QuickFIX/J to send our messages as soon as possible but after sequence numbers are synchronized?
EDIT: I'm trying to solve this using FIX 4.2. FIX 4.4 actually introduced http://www.onixs.biz/fix-dictionary/4.4/tagNum_789.html which would solve my problem (as long as the other party sends this optional tag too).
Thanks
My 10 cents is it sounds like you're trying to treat 2 scenarios in 1 go, and that's difficult. Do 1 thing at a time. For example, if it's your network that causes you to disconnect, before your client knows you've disconnected, your clients will send resend requests, right? Meanwhile, if a client disconnects but you don't then when they reconnect you gap fill. You've got to look carefully at the scenarios. Yes, a resend request may not come at all, it all depends how the client configures things their side. Maybe, per this question you want to send sequence resets because actually, the messages you're trying to send are quotes, right? I mean, what kind of messages are you trying to resend after a disco?
This question is not for a concrete implementation of how this is done. It is more about the concept and design of sending information over Internet with some kind of protocol - either TCP or UDP. I know only that sockets are needed, but I am wondering about the rest. For example after a connection is made and you send the information through that, but how does the other end listen for a specific port and does it listen constantly?
Is listening done in a background thread waiting for information to be received? (In order to be able to do other things/processing while waiting for information)
So in essence, I think a real world example of how such an application works on a high level would be enough to explain the data flow. For example sending files in Skype or something similar.
P.S. Most other questions on similar topics are about a concrete implementation or a bug that someone has.
What I currently do in an application is the following using POSIX sockets with the TCP Protocol:
Most important thing is: The most function are blocking functions. So when you tell your server to wait for client connection, the function will block until a connection is established (if you need a server that handles multiple clients at once, you need to use threading!)
Server listens for specific port until a client connects. After the connect, you will get a new socket file descriptor to communicate with the client whilst the initial socket can listen to new connections. My server then creats a new thread to handle that client whilst waiting for new connections on the initial socket. In the new thread the server waits for a request command from the Client (e.g. Request Login Token). After a request was received by the server, the server will gather its informations, packs it together using Googles Protocol Buffers and sends it to the client. The client now either tells the server to terminate the session (if every data is received by the client that it needs) or send another request.
Thats basically the idea in my server. The bigger problem is the way you transmit and receive data. E.g. you cant send structs or classes (at least not via C++) over the wire, you need some kind of serializer and you have to make sure the other part knows how much to receive. So what i do is, first send a 4byte integer over the wire containing the size of the incomming package, then send the package itself using a serializer (in my case Googles Protocol buffers). The other side waits for 4 byte to be available, knowing that this will be the size of the incomming package. After 4 bytes are received, the program waits for exact that amount of data being available on the socket, when available, read the data out of the buffer and deserialize it. When the socket is not receiving data for 30 seconds, trigger a timeout and terminate the connection.
What you always need to be aware of is the endianess of the systems. E.g. a big endian system (e.g. PowerPC) and a little endian system (e.g. x86) will have problems when you send an integer directly over the wire. For example a
0001
on the x86, is a
1000
on the Power PC, thus making a 8 out of a 1. So you should always use functions like ntohl, an htonl, which will convert data from and to host byte order from and to network byte order (network byte order is always big endian).
Hope this kind of helps. I could also provide some code to you if that would help.
I'm currently developing a bot with POE::Component::IRC whose job, amongst other things, is to post a notice to a list of channels on a schedule, for one week.
I can't seem to find a way to check that the message has been successfully sent to a channel though. The old Net::IRC package would fire a message received event for every message sent to a channel, including ones it itself had sent. POE seems not to do this - at least, the irc_public event is not firing when the bot's own message is published on the channel.
Is there a flag I can pass to the event handler to say "I'd really like to receive all messages please, even my own"? Or is there a way to do this with some sort of RAW event handler?
The IRC protocol does not echo your PRIVMSGs back to you, so you just have to trust that the server received your message and handled it the way it should.
If you just want to receive POE events for messages you send, there's a plug-in for that: POE::Component::IRC::Plugin::BotTraffic. It doesn't actually do anything to verify that the messages ever reach the server, though.
Fortunately, IRC runs on top of TCP, which provides guaranteed in-order delivery. Thus, as long as the connection doesn't drop or hang indefinitely, you can pretty safely assume that your commands will reach the server.
If you wanted to be absolutely sure, you could always follow your PRIVMSG with some command, such as TIME or PING, that you know the server will reply to; if it does, you'll know that it received your PRIVMSG too. Of course, even then there's still no guarantee that the server actually passed the message on to the intended recipient(s); things like netsplits do occur from time to time, and can cause messages to be dropped.
This is a rather general question about TCP sockets. I got a client/server application setup where messages are sent over the wire via TCP. The implementation is done via C++ POCO, however the question is not related to a certain technology.
A message can be a request (initiated by the client) or a response (initiated by the server).
A request has the structure:
Message Header
Request Header
Parameters
A response has the structure
Message Header
Response Header
Parameters
I know TCP guarantees that sent packages will be delivered in the order they have been sent. However, nothing can be assumed about the timespan a delivery might need.
On both sides I have a read/send timeout configured. Now I wonder how to have a clean set up on the transmitted data after a timeout. Don't know how to express this in the right terms, so let me describe an example:
Server S sends a response to the client (Message Header, Response Header, Parameters are put into the stream)
Client C receives the message header partially (e.g. the first 4 bytes of 12)
After these 4 bytes have been received, the reception timeout occurs
On client-side, an appropriate exception is thrown, the reception will be stopped.
The client considers the package as invalid.
Now the problem is, when the client tries to receive another package, he might receive the lasting part of the "old" response message header. From the point of view of the currently processed transaction (send request/get response), the client receives garbage.
So it seems that after a timeout has occured (no matter whether it has been on client or server-side), the communication should continue with a "clean setup", meaning that none of the communication partners will try to send some old package data and that no old package data is stored within the stream buffer of the respective socket.
So how are such situations commonly handled? Is there some kind of design pattern / idiomatic way to solve this?
How are such situations handled within other TCP-based protocols, e.g. HTTP?
In all the TCP samples around the net I've never seen an implementation that deals with those kind of problems...
Thank you in advance
when the client tries to receive another package, he might receive the lasting part of the "old" response message header
He will receive the rest of the failed message, if he receives anything at all. He can't receive anything else, and specifically data that was sent later can't be received before or instead of data that was sent earlier. It is a reliable byte-stream. You can code accordingly.
the communication should continue with a "clean setup", meaning that none of the communication partners will try to send some old package data
You can't control that. If a following message has been written to the TCP socket send buffer, which is all that send() actually does, it will be sent, and there is no way of preventing it short of resetting the connection.
So you either need to code your client to cope with the entire bytestream as it arrives or possibly close the connection on a timeout and start again.
Is there any way of checking if data sent using winsock's send() or WSASend() are really delivered to destination?
I'm writing an application talking with third party server, which sometimes goes down after working for some time, and need to be sure if messages sent to that server are delivered or not. The problem is sometimes calling send() finishes without error, even if server is already down, and only next send() finishes with error - so I have no idea if previous message was delivered or not.
I suppose on TCP layer there is information if certain (or all) packets sent were acked or not, but it is not available using socket interface (or I cannot find a way).
Worst of all, I cannot change the code of the server, so I can't get any delivery confirmation messages.
I'm sorry, but given what you're trying to achieve, you should realise that even if the TCP stack COULD give you an indication that a particular set of bytes has been ACK'd by the remote TCP stack it wouldn't actually mean anything different to what you know at the moment.
The problem is that unless you have an application level ACK from the remote application which is only sent once the remote application has actioned the data that you have sent to it then you will never know for sure if the data has been received by the remote application.
'but I can assume its close enough'
is just delusional. You may as well make that assumption if your send completes as it's about as valid.
The issue is that even if the TCP stack could tell you that the remote stack had ACK'd the data (1) that's not the same thing as the remote application receiving the data (2) and that is not the same thing as the remote application actually USING the data (3).
Given that the remote application COULD crash at any point, 1, 2 OR 3 the only worthwhile indication that the data has arrived is one that is sent by the remote application after it has used the data for the intended purpose.
Everything else is just wishful thinking.
Not from the return to send(). All send() says is that the data was pushed into the send buffer. Connected socket streams are not guarenteed to send all the data, just that the data will be in order. So you can't assume that your send() will be done in a single packet or if it will ever occur due to network delay or interruption.
If you need a full acknowledgement, you'll have to look at higher application level acks (server sending back a formatted ack message, not just packet ACKs).