I'm using python socket programming and zmq to transfer high-resolution images from server to client to process them.
there is a problem here .when the receiver is processing the received image this process takes a while and there is a ram limit on the server, in this case when the receiver will try to process an image, no image will be received by the receiver and I guess the image will accumulate in server RAM to client ask to receive them from the server.
I wanna know if there is any way to find out how many images are accumulated in RAM from the server or client???
Related
Is it possible that a websocket client connection, while receiving larger number of data chunks, can send data to server at the same time?
I am working on a project in which I have two raspberry pis, I want to share process load between them, I am very much comfortable with sockets and have implemented program in which one pi is the host and sends the process request to another pi the other pi acknowledges the request then waits till the host sends the data stream for processing completes the task and sends the response back.
Is this method efficient or do I have to shift to MPI based libraries? I am programming it in C and I have to send bulk data for processing to the second raspberry pi.
Thanks
Ok, I had this argument with a friend about on how do we tackle data process in a restful environment.
The main idea is the application sends a data which is a position of the device (lat & long) to the server and expected to ping back the nearest device.
My argument was the process of determining the "nearest device" in the latlong position should be done in the server and just ping back the latlong of the nearest device.
Friend said that the process should be like this.. the server will pingback a json of every device with its latlong and the client itself will then determine the which one is the nearest.
Look at it this way:
A server is a datahub where all data is stored and processed.
A client is something that fetches data from a server, presents data to the user, and takes input from him.
Now you have to think about Seperation of Concerns. A client's job is to view data to a user, and take his input. A server's job is to process all data and transform it in such a way that a client needs to do minimal processing on it.
I.e. a client fetches some data from the server, all he should have to do now is show it on the screen. There should be minimal data processing on the client side. This is the server's job. The server's job is to make the client's job as easy as possible.
A client also has other things to do on the background (maybe also for other applications). It's CPU is not dedicated to your application, while your server's is.
What does a client need? The nearest device. What does the client ask the server? "Hey server, tell me what is the nearest device.". What should the server return? Exactly, the nearest device. Not a list of all devices which the client has to dig through himself.
No, I think that all information should be managed by the server...
If I understand you describe following schenario:
client sends it coords;
server send to client coords of any device
client receive data and compute nearest device
My questions is: why should a client compute the distance for anyone of (possible) thousands of devices?
Its elaboration power is not so high;
you consume most bandwidth
any information is already in charge of server
In my mind the best idea is a RESTful service where:
any client periodically sends its position (POST)
when a client need a certain number of neighbors then perform a request (GET)
server will contain most updated data, server compute distances, server create and send only small JSON as response.
This in my opinion. :)
This question is not for a concrete implementation of how this is done. It is more about the concept and design of sending information over Internet with some kind of protocol - either TCP or UDP. I know only that sockets are needed, but I am wondering about the rest. For example after a connection is made and you send the information through that, but how does the other end listen for a specific port and does it listen constantly?
Is listening done in a background thread waiting for information to be received? (In order to be able to do other things/processing while waiting for information)
So in essence, I think a real world example of how such an application works on a high level would be enough to explain the data flow. For example sending files in Skype or something similar.
P.S. Most other questions on similar topics are about a concrete implementation or a bug that someone has.
What I currently do in an application is the following using POSIX sockets with the TCP Protocol:
Most important thing is: The most function are blocking functions. So when you tell your server to wait for client connection, the function will block until a connection is established (if you need a server that handles multiple clients at once, you need to use threading!)
Server listens for specific port until a client connects. After the connect, you will get a new socket file descriptor to communicate with the client whilst the initial socket can listen to new connections. My server then creats a new thread to handle that client whilst waiting for new connections on the initial socket. In the new thread the server waits for a request command from the Client (e.g. Request Login Token). After a request was received by the server, the server will gather its informations, packs it together using Googles Protocol Buffers and sends it to the client. The client now either tells the server to terminate the session (if every data is received by the client that it needs) or send another request.
Thats basically the idea in my server. The bigger problem is the way you transmit and receive data. E.g. you cant send structs or classes (at least not via C++) over the wire, you need some kind of serializer and you have to make sure the other part knows how much to receive. So what i do is, first send a 4byte integer over the wire containing the size of the incomming package, then send the package itself using a serializer (in my case Googles Protocol buffers). The other side waits for 4 byte to be available, knowing that this will be the size of the incomming package. After 4 bytes are received, the program waits for exact that amount of data being available on the socket, when available, read the data out of the buffer and deserialize it. When the socket is not receiving data for 30 seconds, trigger a timeout and terminate the connection.
What you always need to be aware of is the endianess of the systems. E.g. a big endian system (e.g. PowerPC) and a little endian system (e.g. x86) will have problems when you send an integer directly over the wire. For example a
0001
on the x86, is a
1000
on the Power PC, thus making a 8 out of a 1. So you should always use functions like ntohl, an htonl, which will convert data from and to host byte order from and to network byte order (network byte order is always big endian).
Hope this kind of helps. I could also provide some code to you if that would help.
I have written one client socket program using linux sockets only. Here is the information giving picture what I am doing in my program
Creating the socket
Making connection with server socket
assigning that socket to read set and exception set for select.
using the select method giving the timeout value NULL in a separate thread
Server is running in one external device.
this program is working fine for reading and all.. Now I am facing problem when I unplug the power cable of that device.
I assumed that when we remove the power cable of the device all the sockets will abruptly closed and connected client sockets will get read event. when we try to read we receive number of bytes read as zero that means connection closed by server.
But in my program when I unplug the power cable of the device, Here in my client program select is not coming out means client socket is not getting any event. I don't understand why..
Any suggestion will be appreciated on how we can come to know that connection is closed by server or any information on whats the sockets behaviour when shutting down the power supply.
I need your help, its very critical.
thank you.
When a remote machine is suddenly cut off the network (network cable unplug or power loss), there is no way it can inform the other side of the connection about that. What is more the client side that performs only reads from a half-open socket (like in your case) won't be able to detect this either.
The only way to know about a connection loss is to to send a packet. Since all data being sent should be acknowledged by the other side, TCP on a client computer will keep retrying to send an unconfirmed portion of data till the number of attempts is exhausted. Then a ETIMEDOUT error should be returned (via a socket that is expecting read events). You can create one more socket for sending these messages periodically to detect a peer disappearance (heart beat connection) on the client side. But all this retries might still take some time.
Another option could be to use SO_KEEPALIVE socket option. After some time a connection has been idle, TCP starts sending probe messages to the server and can detect its disappearance. The default values for idle item are usually enormously huge, so they need to be modified. Some of other parameters that might be related (TCP_KEEPCNT, TCP_KEEPINTVL, TCP_KEEPIDLE). It appears, this option might be implemented differently on different systems or can be simply absent.
I've never personally tried to solve this problem so all this is just a bunch of thoughts that might give some ideas. Here is one more source of ideas.