MESSAGE TO DOWN VOTERS: Please read the question, I am working on a small embedded device. If you are not familar with the limitations of such a device, then please move onto another question instead of down voting!!!!
I am working with a small embedded device that has limited memory and I need to send a large file to a server from this device. Hence I cannot easily use HTTP POST which requires me to load the entire file into memory before sending.
The embedded device has UDP and TCP sockets, but to send a HTTP POST for example, I need to create a string that contains the HTTP HEADERS and the Data. As the device does not have the HTTP Protocol or other protocols available as APIs.
Can someone recommend a protocol I could use to perform the process of "streaming" or sending the data in parts to the server?
The protocol needs to be relatively simple and not use up many memory resources, and if you know of a library designed for small embedded device that would be good also. The protocol should also be simple to implement on the receiving server, preferable running .Net
I am working with a small embedded device that has limited memory and I need to send a large file to a server from this device. Hence I cannot easily use HTTP POST which requires me to load the entire file into memory before sending.
No, POST does not require that. All it requires is that the HTTP Content-Length header that you send matches the number of bytes that you send for the actual file data. Or you can use HTTP 1.1's chunked transfer encoding, which does not use the Content-Length header (so you don't need to know the file size ahead of time). POST (or HTTP, for that matter) has no concept of HOW you send the bytes in your code. So all would have to do is read the file data in a loop, using an appropriate-sized memory buffer, sending the content of that buffer over the socket after each read, until you hit EOF.
For example (pseudo-code):
sckt = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)
connect(sckt, "hostname", 80)
send(sckt, "POST /resource HTTP/1.0\r\n")
send(sckt, "Content-Type: application/octet-stream\r\n"); // or the actual file type
send(sckt, "Content-Length: " + string(the file size) + "\r\n")
send(sckt, "\r\n")
byte buffer[256] // use whatever buffer size is appropriate for your device
do
{
numread = read(file, buffer, sizeof(buffer));
if (numread <= 0) break;
send(sckt, buffer, numread);
}
while (true);
read HTTP response from sckt ...
Or:
sckt = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)
connect(sckt, "hostname", 80)
send(sckt, "POST /resource HTTP/1.1\r\n")
send(sckt, "Content-Type: application/octet-stream\r\n"); // or the actual file type
send(sckt, "Transfer-Encoding: chunked\r\n")
send(sckt, "\r\n")
byte buffer[256] // use whatever buffer size is appropriate for your device
char hex[12]
do
{
numread = read(file, buffer, sizeof(buffer));
if (numread <= 0) break;
sprintf(hex, "%x", numread);
send(sckt, string(hex) + "\r\n")
send(sckt, buffer, numread)
send(sckt, "\r\n")
}
while (true);
send(sckt, "0\r\n");
send(sckt, "\r\n");
read HTTP response from sckt ...
Even powerful desktop PCs have to do it this way, since an entire file usually cannot be put into the kernel buffer at one time anyway, so sending has to be looped accordingly.
The embedded device has UDP and TCP sockets, but to send a HTTP POST for example, I need to create a string that contains the HTTP HEADERS and the Data.
You DO NOT have to send everything at one time in a single string. You can break it up into multiple strings/sends as needed. TCP is a streaming transport, it doesn't care how many sends you perform, as long as the bytes you send are in the correct order. You could send 1 byte at a time for all it cares (though that would not be very efficient, but it would work).
As the device does not have the HTTP Protocol or other protocols available as APIs.
It doesn't need to. Since HTTP sits on top of TCP, and you have access to a TCP socket API, you can implement HTTP manually.
Can someone recommend a protocol I could use to perform the process of "streaming" or sending the data in parts to the server?
HTTP already does exactly that.
The protocol needs to be relatively simple and not use up many memory resources, and if you know of a library designed for small embedded device that would be good also. The protocol should also be simple to implement on the receiving server, preferable running .Net
HTTP is perfectly fine for that.
Related
What I want to do is make my own transport layer protocol in C++. I can't figure out how to create a raw socket that that automatically resolves IP headers, and leaves it up to me to set the payload.
I managed to receive packets on the server using
socket(AF_PACKET, SOCK_RAW, htons(ETH_P_IP))
but didn't manage to create a client that can send data to the server. (I'm not even sure if the above socket is L2 or L3)
From what I understand from reading about raw sockets, a L3 socket would look like
socket(AF_INET, SOCK_RAW, protocol)
Thing is, I don't know what to fill in for the protocol, if my intention is to create my own and not to use existing ones. (I have tried many of the iana numbers, including the range 143-252)
So the question is: how to create a socket, server and client sided, on top of the Internet Protocol such that two computers can communicate in an arbitrary protocol (or send data to each other)? In other words, I want to specify the end IP address and a payload and have the socket take care of the IP header.
What I have now:
server.cpp: https://pastebin.com/yLMFLDmJ
client.cpp: https://pastebin.com/LWuNdqPT
For those who are searching, here is the solution I found: http://www.pdbuchan.com/rawsock/rawsock.html
In the file tcp4.c on the above mentioned page, there is a client implementation using a raw socket. The code adds both IP and TCP headers, but you can simply remove the lines where the TCP headers are added and replace them with your own protocol. You also need to change this line: iphdr.ip_p = IPPROTO_TCP to iphdr.ip_p = 200 (200 or any number in the range 143-252; see https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml) and set the name of the interface you are using, as well as change the addresses.
So here is a stripped down version with the mentioned changes that sends an IP packet only containing IP headers: https://pastebin.com/z2sGmtQd
And here is a very simple server that can receive these packets: https://pastebin.com/jJgZUv5p
I've been working in a socket tcp connection to a game server. The big problem here is that the game server send the data without any separators - since it sends the packet lenght inside the data -, making impossible to use socket:receive("*a") or "*l". The data received from the server does not have a static size and are sent in HEX format. I'm using this solution:
while true do
local rect, r, st = socket.select({_S.sockets.main, _S.sockets.bulle}, nil, 0.2)
for i, con in ipairs(rect) do
resp, err, part = con:receive(1)
if resp ~= nil then
dataRecv = dataRecv..resp
end
end
end
As you can see, I can only get all the data from the socket by reading one byte and appending it to a string, not a good way since I have two sockets to read. Is there a better way to receive data from this socket?
I don't think there is any other option; usually in a situation like this the client reads a packet of specific length to figure out how much it needs to read from the rest of the stream. Some protocols combine new line and the length; for example HTTP uses line separators for headers, with one of the headers specifying the length of the content that follows the headers.
Still, you don't need to read the stream one-by-one character as you can switch to non-blocking read and request any number of characters. If there is not enough to read, you'll get partially read content plus "timeout" signaled, which you can handle in your logic; from the documentation:
In case of error, the method returns nil followed by an error message
which can be the string 'closed' in case the connection was closed
before the transmission was completed or the string 'timeout' in case
there was a timeout during the operation. Also, after the error
message, the function returns the partial result of the transmission.
I am wondering what is the recommended way to upload a large file using REST.
Giving that the time taken from the server to write the file on disk can be large, should I wait for the writing operation to end before sending the 201 reply to the client?
Is it maybe better to reply to the user on the fly and process the writing operation later? In this second case what http status should i return? In fact even though i return a 201 no one can ensure that the writing operation was successful.
It sounds as if you're not starting to write to disk until you've read the last byte from HTTP. That has two undesirable effects:
You're putting a load of data into your server's memory footprint
You're pushing all the disk IO to the end of the request handling, causing a pause that looks bad from the client perspective. In extreme cases, the client may time-out.
Instead, you need to be streaming. As soon as you have some data from the client, start writing to disk. When the client sends its final chunk, you only need to handle that small chunk before sending the 201 response.
If the API you're using doesn't allow you to consume HTTP entities in a streaming manner, I'm afraid you need to look for a different API.
For example, in the Java Servlet API, you can get an InputStream for the request content:
InputStream content = request.getInputStream();
// do something better than this
OutputStream out = new FileOutputStream("content.txt");
byte[] buffer = new byte[1024];
int len;
while ((len = in.read(buffer)) != -1) {
// whatever processing you want here
out.write(buffer, 0, len);
}
// now you can send the 201
It's important to understand that when you get the InputStream in this example, your server hasn't started reading the content off the TCP stack yet (*). Only the headers have been read. If it's a large file, it's very likely that the client is still sending at this moment. You read a chunk, write a chunk, in a loop.
(*) That's a simplification. It may be in a buffer. But conceptually, you can ignore that
I wrote a TCP socket program,and define a text protocol format like: "length|content",
to make it simple, the "length" is always 1-byte-long and it define the number of bytes of "content"
My problem is:
when attackers send packets like "1|a51",it will stay in tcp's receive buffer
the program will parse it wrong and the next packet would start like "5|1XXXX"，
then the rest of the packets remain in the buffer would all parsed wrong，
how to solve this problem?
If you get garbage, just close the connection. It's not your problem to figure out what they meant, if anything.
instead of length|content only, you also need to provide a checksum, if the checksum is not correct, you should drop the connection to avoid partial receive.
this is a typical problem in tcp protocol, since the tcp is stream based. but just as http, which is an application of tcp protocol, it has a structure of request / response to make sure each end of the connection knows when the data has been fully transferred.
but your scenario is a little bit tricky, since the hacker can only affect the connection of his own. while it cannot change the data from other connections, only if he can control the route / switcher between your application and the users.
I am working with client-server programming I am referring this link and my server is successfully running.
I need to send data continuously to the server.
I don't want to connect() every time before sending each packet. So for first time I just created a socket and send the first packet, the rest of the data I just used write() function to write data to the socket.
But my problem is while sending data continuously if the server is not there or my Ethernet is disabled still it successfully write data to socket.
Is there any method by which I can create socket only at once and send data continuously with knowing server failure?.
The main reason for doing like this that, on the server side I am using GPRS modem and on each time when call connect() function for each packet the modem get hanged.
For creating socket I using below code
Gprs_sockfd = socket(AF_INET, SOCK_STREAM, 0);
if (Gprs_sockfd < 0)
{
Display("ERROR opening socket");
return 0;
}
server = gethostbyname((const char*)ip_address);
if (server == NULL)
{
Display("ERROR, no such host");
return 0;
}
bzero((char *) &serv_addr, sizeof(serv_addr));
serv_addr.sin_family = AF_INET;
bcopy((char *)server->h_addr,(char *)&serv_addr.sin_addr.s_addr,server->h_length);
serv_addr.sin_port = htons(portno);
if (connect(Gprs_sockfd,(struct sockaddr *) &serv_addr,sizeof(serv_addr)) < 0)
{
Display("ERROR connecting");
return 0;
}
And each time I writing to the socket using the below code
n = write(Gprs_sockfd,data,length);
if(n<0)
{
Display("ERROR writing to socket");
return 0;
}
Thanks in advance.............
TCP was designed to tolerate temporary failures. It does byte sequencing, acknowledgments, and, if necessary, retransmissions. All unacknowledged data is buffered inside the kernel network stack. If I remember correctly the default is three re-transmission attempts (somebody correct me if I'm wrong) with exponential back-off timeouts. That quickly adds up to dozens of seconds, if not minutes.
My suggestion would be to design application-level acknowledgments into your protocol, meaning the server would send a short reply saying that it received that much data up to now, say every second. If the client does not receive suck ack in say 3 seconds, the client knows the connection is unusable and can close it. By the way, this is easier done with non-blocking sockets and polling functions like select(2) or poll(2).
Edit 0:
I think this would be very relevant here - "The ultimate SO_LINGER page, or: why is my tcp not reliable".
Nikolai is correct here, the behaviour you experience here is desirable as basically you could continue transfering data after network outage without any logic in your application. If your application should detect outages longer that specified amount of time, you need to add heartbeating into your protocol. This is standard way of solving the problem. It can also allow you for detect situation when network is all right, receiver is alive, but it has deadlocked (due to to a software bug).
Heartbeating could be as simple as mentioned by Nikolai -- sending a small packet every X seconds; if the server can't see the packet for N*X seconds, the connection would be dropped.