Hello there I am getting mad with this issue since I'm using a simple pattern.
Ok, I have a infinite while in which I use epoll_wait on server socket and already connected sockets. All went ok if a new socket send a connection request; my problem is when connected sockets (right now I am just using one socket sending 390k packet) sends data: doesn't matter if I use EPOLLONESHOT or EPOLLET, after consumed all request buffer on that socket, rearming socket or receiving EAGAIN on recv(), epoll_wait always wakes up again with wrong buffer!
My server works with threadpool, but right now is just one thread that does all work (to simplify testing):
while (TRUE) {
int num = epoll_wait(efd, events, MAX_EVENTS, -1);
for (int i = 0; i < num; i++) { // ciclo epoll()
if (events[i].events & EPOLLERR || events[i].events & EPOLLHUP || !(events[i].events & EPOLLIN)) {
fprintf (stderr, "epoll error on socket: closed\n");
s = epoll_ctl(efd, EPOLL_CTL_DEL, events[i].data.fd, NULL);
}
else if (events[i].data.fd == serverSocket) {
while (TRUE) {
newInfoClient = server->m_bgServer->AcceptClient(&newSocketClient);
if (newInfoClient == NULL) { // nessun client
break;
}
else {
printf("\nSocket accettato: %d", newSocketClient);
s = CSocket::MakeNonBlocking(newSocketClient);
if (s == -1)
abort();
event.data.fd = newSocketClient;
event.events = EPOLLIN | EPOLLET;
s = epoll_ctl(efd, EPOLL_CTL_ADD, newSocketClient, &event);
if (s == -1) {
abort();
}
}
}
else {
AbstractTcpServerGame::DownloadTcpRequest(client);
}
}
}
I have just omitted some checks and other internal codes.
AbstractTcpServerGame::DownloadTcpRequest(...)
This function is just a recv in loop to rescues my own header, get buffer body and just to verify empty buffer outside loop I call a simple recv() that returns -1 (errno=EAGAIN or EWOULDBLOCK).
After this when I rearm the socket with epoll_ctr() in DownloadTcpRequest() in EPOLLONESHOT case, when it returns epoll_wait() wakes up again on the same socket!! this is my execution log:
New socket (6) request (errno 11) <--- when epoll_wait() emits EPOLLIN on socket 6
Download of 18 bytes (socket 6) <-- inside AbstractTcpServerGame::DownloadTcpRequest()
Download of 380k (socket 6) <-- another recv() loop to rescue body request
------------------- empty buffer on socket 6 ----------- <-- dummy recv to show empty buffer
New socket (6) request (errno 11)
Download of 18 bytes (socket 6)
Download of -1556256155 (socket 6)
Error on socket 6 (bad::alloc exception)
Client sends 398k (18 header + body) and all data are correctly received as shown above, but rearming socket or using EPOLLET, epoll_wait() generates another request and I don't know where those are taken infact are not correct!
Related
I have a simple XMLHttpRequest handler written in C. It reads and processes requests coming from a JavaScript XMLHttpRequest send() running in a browser.
The parent process accepts incoming connections and forks a child process for each incoming connection to read and process the data.
It works perfectly for most requests, but fails in some cases (apparently related to the network infrastructure between the client and the server) if the request is over about 2 kB in length. I'm assuming that the request is being broken into multiple packets somewhere between the browser and my socket server.
I can't change the request format, but I can see the request being sent and verify the content. The data is a 'GET' with an encoded URI that contains a 'type' field. If the type is 'file', the request could be as long as 3 kB, otherwise it's a couple of hundred bytes at most. 'File' requests are rare - the user is providing configuration data to be written to a file on the server. All other requests work fine, and any 'file' requests shorter than about 2 kB work fine.
What's the preferred technique for ensuring that I have all of the data in this situation?
Here's the portion of the parent that accepts the connection and forks the child (non-blocking version):
for (hit = 1;; hit++) {
length = sizeof(cli_addr);
if ((socketfd = accept4(listensd, (struct sockaddr *) &cli_addr, &length, SOCK_NONBLOCK)) < 0){
//if ((socketfd = accept(listensd, (struct sockaddr *) &cli_addr, &length)) < 0){
exit(3);
}
if ((pid = fork()) < 0) {
exit(3);
} else {
if (pid == 0) { /* child */
//(void) close(listensd);
childProcess(socketfd, hit); /* never returns. Close listensd when done*/
} else { /* parent */
(void) close(socketfd);
}
}
}
Here's the portion of the child process that performs the initial recv(). In the case of long 'file' requests, the child's first socket recv() gets about 1700 bytes of payload followed by the browser-supplied connection data.
ret = recv(socketfd, recv_data, BUFSIZE, 0); // read request
if (ret == 0 || ret == -1) { // read failure stop now
sprintf(sbuff, "failed to read request: %d", ret);
logger(&shm, FATAL, sbuff, socketfd);
}
recv_data[ret] = 0;
len = ret;
If the type is 'file', there could be more data. The child process never gets the rest of the data. If the socket is blocking, a second read attempt simply hangs. If the socket is non-blocking (as in the snippet below) all subsequent reads return -1 with error 'Resource temporarily unavailable' until it times out:
// It's a file. Could be broken into multiple blocks. Try second read
sleep(1);
ret = recv(socketfd, &recv_data[len], BUFSIZE, 0); // read request
while (ret != 0){
if (ret > 0){
recv_data[len+ret] = 0;
len += ret;
} else {
sleep(1);
}
ret = recv(socketfd, &recv_data[len], BUFSIZE, 0); // read request
}
I expected that read() would return 0 when the client closes the connection, but that doesn't happen.
A GET request only has a head and no body (well, almost always), so you have everything the client has sent as soon as you have the request head, and you know when you have read the whole request head when you read a blank line i.e. two returns (and no sooner or later).
If the client sends just a part, without the blank line, you are supposed to wait for the rest. I would put a time-out on that and reject the whole request if it takes too long.
BTW there are still browsers out there, and maybe some proxies as well, with a URL length limit of about 2000 characters.
I have a server written in plain-old C accepting TCP connections using kqueue on FreeBSD.
Incoming connections are accepted and added to a simple connection pool to keep track of the file handle.
When data is received (on EVFILT_READ), I call recv() and then I put the payload in a message queue for a different thread to process it.
Receiving and processing data this way works perfect.
When the processing thread is done, it may need to send something back to the client. Since the processing thread has access to the connection pool and can easily get the file handle, I'm simply calling send() from the processing thread.
This works 99% of the time, but every now and then kqueue gives me a EV_EOF flag, and the connection is dropped.
There is a clear correlation between the frequency of the calls to send() and the number of EV_EOF errors, so I have a feeling the EV_EOF due to some race condition between my kqueue thread and the processing thread.
The calls to send() always returns the expected byte count, so I'm not filling up the tx buffer.
So my question; Is it acceptable to call send() from a separate thread as described here? If not, what would be the right way to send data back to the clients asynchronously?
All the examples I find calls send() in the same context as the kqueue loop, but my processing threads may need to send back data at any time - even minutes after the last received data from the client - so obviously I can't block the kqueue loop for that time..
Relevant code snippets:
void *tcp_srvthread(void *arg)
{
[[...Bunch of declarations...]]
tcp_serversocket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
...
setsockopt(tcp_serversocket, SOL_SOCKET, SO_REUSEADDR, &i, sizeof(int));
...
err = bind(tcp_serversocket, (const struct sockaddr*)&sa, sizeof(sa));
...
err = listen(tcp_serversocket, 10);
...
kq = kqueue();
EV_SET(&evSet, tcp_serversocket, EVFILT_READ | EV_CLEAR, EV_ADD, 0, 0, NULL);
...
while(!fTerminated) {
timeout.tv_sec = 2; timeout.tv_nsec = 0;
nev = kevent(kq, &evSet, 0, evList, NLIST, &timeout);
for (i=0; i<nev; i++) {
if (evList[i].ident == tcp_serversocket) { // new connection?
socklen = sizeof(addr);
fd = accept(evList[i].ident, &addr, &socklen); // accept it
if(fd > 0) { // accept ok?
uidx = conn_add(fd, (struct sockaddr_in *)&addr); // Add it to connected controllers
if(uidx >= 0) { // add ok?
EV_SET(&evSet, fd, EVFILT_READ | EV_CLEAR, EV_ADD, 0, 0, (void*)(uint64_t)(0x00E20000 | uidx)); // monitor events from it
if (kevent(kq, &evSet, 1, NULL, 0, NULL) == -1) { // monitor ok?
conn_delete(uidx); // ..no, so delete it from my list also
}
} else { // no room on server?
close(fd);
}
}
else Log(0, "ERR: accept fd=%d", fd);
}
else
if (evList[i].flags & EV_EOF) {
[[ ** THIS IS CALLED SOMETIMES AFTER CALLING SEND - WHY?? ** ]]
uidx = (uint32_t)evList[i].udata;
conn_delete( uidx );
}
else
if (evList[i].filter == EVFILT_READ) {
if((nr = recv(evList[i].ident, buf, sizeof(buf)-2, 0)) > 0) {
uidx = (uint32_t)evList[i].udata;
recv_data(uidx, buf, nr); // This will queue the message for the processing thread
}
}
}
else {
// should not get here.
}
}
}
The processing thread looks something like this (obviously there's a lot of data manipulation going on in addition to what's shown) :
void *parsethread(void *arg)
{
int len;
tmsg_Queue mq;
char is_ok;
while(!fTerminated) {
if((len = msgrcv(msgRxQ, &mq, sizeof(tmsg_Queue), 0, 0)) > 0) {
if( process_message(mq) ) {
[[ processing will find the uidx of the client and build the return data ]]
send( ctl[uidx].fd, replydata, replydataLen, 0 );
}
}
}
}
Appreciate any ideas or nudges in the right direction. Thanks.
EV_EOF
If you write to a socket after the peer closed the reading part of it, you will receive a RST, which triggered EVFILT_READ with EV_EOF set.
Async
You should try aio_read and aio_write.
I have an application that sends a GET request using winsock on port 80 using a TCP socket. A few users have reported an issue where no response is received, looking at network logs and seeing the network device is getting the data just the app isn't it was clear that the firewall was blocking it.
Having disabled the firewall it then worked fine but what I don't understand is why it was getting blocked. The connection is created from the users computer, it connects fine and sends (which I assumes automatically opens a port) so how can data be lost on the same connection when received? Should I be providing additional winsock settings? Or is there simply no way around stopping the firewall blocking an already active connection?
Here is a stripped down version of the winsock code
SOCKET sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if (sock == INVALID_SOCKET)
return -1;
struct sockaddr_in client;
memset(&client, 0, sizeof(client));
client.sin_family = AF_INET;
client.sin_port = htons(80);
client.sin_addr.s_addr = inet_addr(inet_ntoa(*addr_list[0]));
if (connect(sock, (struct sockaddr *)&client, sizeof(client)) < 0){
closesocket(sock);
return -1;
}
if (send(sock, buffer, buflength, 0) != buflength){
closesocket(sock);
return -1;
}
//get response
response = "";
int resp_leng = BUFFERSIZE;
while (resp_leng == BUFFERSIZE)
{
resp_leng = recv(sock, (char*)&buffer, BUFFERSIZE, 0);
if (resp_leng > 0)
response += std::string(buffer).substr(0, resp_leng);
else
return -1;
}
closesocket(sock);
Your while loop exits if a recv() returns less than BUFFERSIZE. This is wrong -- you must always assume that recv() can return any amount of data from 1 byte up to and including the supplied buffer size.
I'm writing a bare bone ftp client just using sockets on VxWorks and I now would like to receive directory contents.
For that I need to send a Request: LIST and following a [SYN] which initiates the data transfer back to me but I'm wondering how I do this with simple sockets?
My code to send the LIST just looks like this:
char lst[6] = "LIST";
lst[4] = 0x0d; // add empty characters on back
lst[5] = 0x0a;
if (write (sFd, (char *) &lst, 6) == ERROR)
{
perror ("write");
close (sFd);
return ERROR;
}
if (read (sFd, replyBuf, REPLY_MSG_SIZE) < 0) {
perror ("read");
close (sFd);
return ERROR;
}
printf ("MESSAGE FROM SERVER:\n%s\n", replyBuf);
but it actually gets stuck in the read() until it times out as the server doesn't respond unless i send a 'SYNC` to initiate the connection.
edit
Upon suggestion, I replaced the addtion of 0x0d and 0x0a at the end of my string with \r\n directly top the string which changed my code to:
char lst[6] = "LIST\r\n";
if (write (sFd, (char *) &lst, strlen(lst)) == ERROR)
{
perror ("write");
close (sFd);
return ERROR;
}
if (read (sFd, replyBuf, REPLY_MSG_SIZE) < 0) {
perror ("read");
close (sFd);
return ERROR;
}
printf ("MESSAGE FROM SERVER:\n%s\n", replyBuf);
but I get exactly the same result, my client does not send a SYNC message - why not I am wondering...?
For that I need to send a Request: LIST and following a [SYN] which initiates the data transfer back to me
No you don't. The [SYN] is sent automatically when you connect a TCP socket.
char lst[6] = "LIST";
The problem is here. It should be
char[] lst = "LIST\r\n";
All FTP commands are terminated by a line terminator, which is defined as \r\n.
I'm using socket with O_NONBLOCK, select, keep alive connection and something like HTTP.
each server connection and client side uses a buffer to get all sent data until complete message be received
How to working:
client send data "A"
client try receive response from server
server receive but don't reply immediately
client gets timeout
server send response "A" (but client don't receive now)
another request:
client send data "B"
server send response "B"
client receive "AB" response instead only "B"
the problem is that client receives previous buffer message
source code bellow:
Server.cpp base class:
bool Server::start()
{
struct sockaddr_in client_addr;
SOCKET client_socket, max_sock;
Connection* conn;
int addrlen = sizeof(struct sockaddr_in);
std::list<Connection*>::iterator it, itr;
int response;
fd_set fdset, read_fds;
max_sock = m_socket;
FD_ZERO(&fdset);
FD_SET(m_socket, &fdset);
onStart();
while(true)
{
// make a copy of set
read_fds = fdset;
// wait for read event
response = select(max_sock + 1, &read_fds, NULL, NULL, NULL);
if(response == -1)
break;
// check for new connections
if(FD_ISSET(m_socket, &read_fds))
{
response--;
// accept connections
client_socket = accept(m_socket, (struct sockaddr*)&client_addr, &addrlen);
if(client_socket != INVALID_SOCKET)
{
conn = new Connection(*this, client_socket, &client_addr);
m_connections.push_front(conn);
// add connection to set for wait for read event
FD_SET(client_socket, &fdset);
// allow select new sock from select funcion
if(max_sock < client_socket)
max_sock = client_socket;
}
}
// check for received data from clients
it = m_connections.begin();
while(it != m_connections.end() && response > 0)
{
conn = *it;
// verify if connection can be readed
if(FD_ISSET(conn->getSocket(), &read_fds))
{
response--;
conn->receive();
if(!conn->isConnected())
{
FD_CLR(conn->getSocket(), &fdset);
// remove connection from list
itr = it;
it++;
m_connections.erase(itr);
delete conn;
continue;
}
}
it++;
}
}
onFinish(response >= 0);
return response >= 0;
}
main.cpp Server implementation:
void ClientConnection::onReceive(const void * data, size_t size)
{
const char *str, *pos = NULL;
HttpParser* p;
buffer->write(data, size);
do
{
str = (const char*)buffer->data();
if(contentOffset == 0)
{
pos = strnstr(str, buffer->size(), "\r\n\r\n");
if(pos != NULL)
{
contentOffset = pos - str + 4;
p = new HttpParser((const char*)buffer->data(), contentOffset);
contentLength = p->getContentLength();
delete p;
}
}
if(buffer->size() - contentOffset < contentLength || contentOffset == 0)
return;
proccessRequest();
keepDataStartingOf(contentOffset + contentLength);
}
while(buffer->size() > 0);
}
client side code is a simple recv send with timeout
any idea how to solve?
The first thing that comes to mind is to make the client's timeout large enough that the client won't timeout unless the server is actually dead... but I'm sure you've already thought of that. :)
So assuming that's not a good enough fix, the next thing to try is to have the client send an ID number with each request it sends. The ID number can be generated with a simple counter (e.g. for the client's first request, it tags the request with 0, for the second it tags it with 1, etc). The server, when sending its reply, will include that same ID number with the reply.
When the client receives a reply, it compares the ID number in the reply data against the current value of its counter. If the two numbers are the same, it processes the data. If not, it ignores the data. Et voila!