I've got a ESP8266 NodeMCU device which acts as a LAN Node server using ports 80(html) and 81(websocket server).
It works fine with a browser webpage I designed for a control application, but I'm trying to port it over to Excel using Winsock. In place of the webpage, I just use a form and need to access cells on the worksheet. This has value for other IOT apps where I want to import data to Excel and take advantage of Excel's graphs, export data, etc.
Here's the issue I can get port 81 to upgrade to TCP but as soon as I try to send a small text message, the NodeMCU disconnects with no error message. The code follows:
oString = "GET / HTTP/1.1" & vbCrLf
oString = oString & "Host: ws://192.168.0.9:81/" & vbCrLf
oString = oString & "Connection: Upgrade" & vbCrLf
oString = oString & "Pragma: no-cache" & vbCrLf
oString = oString & "cache-Control: no-cache" & vbCrLf
oString = oString & "Upgrade: WebSocket" & vbCrLf
oString = oString & "Sec-WebSocket-Version: 13" & vbCrLf
oString = oString & "Sec-WebSocket-Key: yaVX9fn9sKEiN2YbqvF/fg==" & vbCrLf
oString = oString & "Sec-WebSocket-Protocol: arduino " & vbCrLf
oString = oString & vbCrLf
ws81.SendData (oString)
NodeMCU1.0 Response:
HTTP/1.1 101 Switching Protocols
Server: arduino-WebSocketsServer
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Version: 13
Sec-WebSocket-Accept: xaeFOTHpxO8e54UJtOnwdL+odwo=
Sec-WebSocket-Protocol: arduino
NodeMCU (via serial monitor) tells me I'm connected and starts pinging and Winsock pongs back: reading but not doing anything with the Accept key(Don't think it's necessary).
Now the problem is as soon as I try to send text the NodeMCU disconnects and there's no specific error message.
I've looked at RFC 6455 and I think that the header protocol is part of MSWinsock(otherwise what's it existing for?). Looking at wireShark, the header is there for sure, but larger than I would have expected by 20 bytes. I would have pegged it at 16 bytes given the short message(about 20B) that I am sending.
It does appear to have the mask bytes in place but the Extended Payload Size bytes are populated. I would think that they would be zeros.
One important note
My client is sending unmasked and the message is clear text. Why wouldn't the Xor algorithm be being applied since RFC 6455 says ALL client uploads to server are masked? But still, that would just mean the NodeMCU would just receive garbage and react badly, but not disconnect. I've already duplicated the masking algorithm and I could apply it to the Payload but I have no way to set the MASK bit or the mask bytes in the header(at least not that I know of). If it has anything to do with Big vs Little Endian then I'm doomed because even if I swapped the payload data endian I couldn't change the header.
Anyone have any ideas?
Related
I'm working on a project related with some naval radar and how to use the data that its internal "server" sends while rotating about other ships.
the documentation for the radar protocol: here
The main idea is to create a client application to receive the data sent from the radar server, which are ascii sentences containing other ship's information, like speed, position, etc. those are always being sent on a tcp connection.
at the same time, the client must be able to send command in the same format, something like:
<command>[,<argument#1>][,<argument#2>][,...][,<argument#n>]<eol>
to ask for a specific ship information for example.
so all I have until now is this basic client (please dismiss the nonsense #coments on it)
import socket
import time
#try:
#ip = raw_input('IP: ')
#except:
#ip = input('IP: ')
#puerto = input("PUERTO??")
#MESSAGE=b'#000\r'
ip = '127.0.0.1'
puerto = 1235
try:
srvsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error:
print('Failed to create socket')
sys.exit()
print('Socket Created')
srvsock.connect((ip, puerto))
srvsock.setblocking(0)
print ('Conectado a ' + ip + ':' + str(puerto))
while True:
#listening and receiving ascii strings (buf) from radar
try:
buf = srvsock.recv(1024)
if buf == 'stop':
break
elif len(buf) > 0:
print(buf)
except:
print('error de cliente')
break
# how to be able to send commands while listening the radar??
#try:
#msg = raw_input ('Mensage: ')
#msgencode = msg.encode ('ascii')
#srvsock.send(msgencode)
#except socket.error:
#print('Failed to send data')
#break
srvsock.close()
So the idea is to be listening all the time the incoming ascii strings sent by the server... and also being able to send commands from the client app to the sever to request more data.
I'm still working on this but any better or professional way of doing this would be really appreciated. I'm a little bit confuse with the implementation of a non blocking code to do this.
This question already has answers here:
TCP Connection Seems to Receive Incomplete Data
(5 answers)
Closed 3 years ago.
I'm attempting to implement the Remote Frame Buffer protocol using Ada's Sockets library and I'm having trouble controlling the length of the packets that I'm sending.
I'm following the RFC 6143 specification (https://tools.ietf.org/pdf/rfc6143.pdf), see comments in the code for section numbers...
-- Section 7.1.1
String'Write (Comms, Protocol_Version);
Put_Line ("Server version: '"
& Protocol_Version (1 .. 11) & "'");
String'Read (Comms, Client_Version);
Put_Line ("Client version: '"
& Client_Version (1 .. 11) & "'");
-- Section 7.1.2
-- Server sends security types
U8'Write (Comms, Number_Of_Security_Types);
U8'Write (Comms, Security_Type_None);
-- client replies by selecting a security type
U8'Read (Comms, Client_Requested_Security_Type);
Put_Line ("Client requested security type: "
& Client_Requested_Security_Type'Image);
-- Section 7.1.3
U32'Write (Comms, Byte_Reverse (Security_Result));
-- Section 7.3.1
U8'Read (Comms, Client_Requested_Shared_Flag);
Put_Line ("Client requested shared flag: "
& Client_Requested_Shared_Flag'Image);
Server_Init'Write (Comms, Server_Init_Rec);
The problem seems to be (according to wireshark) that my calls to the various 'Write procedures are causing bytes to queue up on the socket without getting sent.
Consequently two or more packet's worth of data are being sent as one and causing malformed packets. Sections 7.1.2 and 7.1.3 are being sent consecutively in one packet instead of being broken into two.
I had wrongly assumed that 'Reading from the socket would cause the outgoing data to be flushed out, but that does not appear to be the case.
How do I tell Ada's Sockets library "this packet is finished, send it right now"?
To enphasize https://stackoverflow.com/users/207421/user207421 comment:
I'm not a protocols guru, but from my own experience, the usage of TCP (see RFC793) is often misunderstood.
The problem seems to be (according to wireshark) that my calls to the various 'Write procedures are causing bytes to queue up on the socket without getting sent.
Consequently two or more packet's worth of data are being sent as one and causing malformed packets. Sections 7.1.2 and 7.1.3 are being sent consecutively in one packet instead of being broken into two.
In short, TCP is not message-oriented.
Using TCP, sending/writing to socket results only append data to the TCP stream. The socket is free to send it in one exchange or several, and if you have lengthy data to send and message oriented protocol to implement on top of TCP, you may need to handle message reconstruction. Usually, an end of message special sequence of characters is added at the end of the message.
Processes transmit data by calling on the TCP and passing buffers of data as arguments. The TCP packages the data from these buffers into segments and calls on the internet module to transmit each segment to the destination TCP. The receiving TCP places the data from a segment into the receiving user's buffer and notifies the receiving user. The TCPs include control information in the segments which they use to ensure reliable ordered data transmission.
See also https://stackoverflow.com/a/11237634/7237062, quoting:
TCP is a stream-oriented connection, not message-oriented. It has no
concept of a message. When you write out your serialized string, it
only sees a meaningless sequence of bytes. TCP is free to break up
that stream up into multiple fragments and they will be received at
the client in those fragment-sized chunks. It is up to you to
reconstruct the entire message on the other end.
In your scenario, one would typically send a message length prefix.
This way, the client first reads the length prefix so it can then know
how large the incoming message is supposed to be.
or TCP Connection Seems to Receive Incomplete Data, quoting:
The recv function can receive as little as 1 byte, you may have to call it multiple times to get your entire payload. Because of this, you need to know how much data you're expecting. Although you can signal completion by closing the connection, that's not really a good idea.
Update:
I should also mention that the send function has the same conventions as recv: you have to call it in a loop because you cannot assume that it will send all your data. While it might always work in your development environment, that's the kind of assumption that will bite you later.
MESSAGE TO DOWN VOTERS: Please read the question, I am working on a small embedded device. If you are not familar with the limitations of such a device, then please move onto another question instead of down voting!!!!
I am working with a small embedded device that has limited memory and I need to send a large file to a server from this device. Hence I cannot easily use HTTP POST which requires me to load the entire file into memory before sending.
The embedded device has UDP and TCP sockets, but to send a HTTP POST for example, I need to create a string that contains the HTTP HEADERS and the Data. As the device does not have the HTTP Protocol or other protocols available as APIs.
Can someone recommend a protocol I could use to perform the process of "streaming" or sending the data in parts to the server?
The protocol needs to be relatively simple and not use up many memory resources, and if you know of a library designed for small embedded device that would be good also. The protocol should also be simple to implement on the receiving server, preferable running .Net
I am working with a small embedded device that has limited memory and I need to send a large file to a server from this device. Hence I cannot easily use HTTP POST which requires me to load the entire file into memory before sending.
No, POST does not require that. All it requires is that the HTTP Content-Length header that you send matches the number of bytes that you send for the actual file data. Or you can use HTTP 1.1's chunked transfer encoding, which does not use the Content-Length header (so you don't need to know the file size ahead of time). POST (or HTTP, for that matter) has no concept of HOW you send the bytes in your code. So all would have to do is read the file data in a loop, using an appropriate-sized memory buffer, sending the content of that buffer over the socket after each read, until you hit EOF.
For example (pseudo-code):
sckt = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)
connect(sckt, "hostname", 80)
send(sckt, "POST /resource HTTP/1.0\r\n")
send(sckt, "Content-Type: application/octet-stream\r\n"); // or the actual file type
send(sckt, "Content-Length: " + string(the file size) + "\r\n")
send(sckt, "\r\n")
byte buffer[256] // use whatever buffer size is appropriate for your device
do
{
numread = read(file, buffer, sizeof(buffer));
if (numread <= 0) break;
send(sckt, buffer, numread);
}
while (true);
read HTTP response from sckt ...
Or:
sckt = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)
connect(sckt, "hostname", 80)
send(sckt, "POST /resource HTTP/1.1\r\n")
send(sckt, "Content-Type: application/octet-stream\r\n"); // or the actual file type
send(sckt, "Transfer-Encoding: chunked\r\n")
send(sckt, "\r\n")
byte buffer[256] // use whatever buffer size is appropriate for your device
char hex[12]
do
{
numread = read(file, buffer, sizeof(buffer));
if (numread <= 0) break;
sprintf(hex, "%x", numread);
send(sckt, string(hex) + "\r\n")
send(sckt, buffer, numread)
send(sckt, "\r\n")
}
while (true);
send(sckt, "0\r\n");
send(sckt, "\r\n");
read HTTP response from sckt ...
Even powerful desktop PCs have to do it this way, since an entire file usually cannot be put into the kernel buffer at one time anyway, so sending has to be looped accordingly.
The embedded device has UDP and TCP sockets, but to send a HTTP POST for example, I need to create a string that contains the HTTP HEADERS and the Data.
You DO NOT have to send everything at one time in a single string. You can break it up into multiple strings/sends as needed. TCP is a streaming transport, it doesn't care how many sends you perform, as long as the bytes you send are in the correct order. You could send 1 byte at a time for all it cares (though that would not be very efficient, but it would work).
As the device does not have the HTTP Protocol or other protocols available as APIs.
It doesn't need to. Since HTTP sits on top of TCP, and you have access to a TCP socket API, you can implement HTTP manually.
Can someone recommend a protocol I could use to perform the process of "streaming" or sending the data in parts to the server?
HTTP already does exactly that.
The protocol needs to be relatively simple and not use up many memory resources, and if you know of a library designed for small embedded device that would be good also. The protocol should also be simple to implement on the receiving server, preferable running .Net
HTTP is perfectly fine for that.
I am having problems reading a text file content via winsock on C , does anyone have any idea how it should work? actually when I try to GET HTTP header from google am able to, but when I try on my xampp machine,
it just gives me 400 bad request.
HTTP/1.1 400 Bad Request
char *message = "GET / HTTP/1.1\r\n\r\n";
Ok the problem that I was receiving 400 bad request on my localhost via winsock was the my HTTP request, i just changed the 1.1 to 1.0 .. and it worked!!! what I am wanting now is printing nothing the content of the text file and not the whole banner?! :)
Read RFC 2616, in particular sections 5.2 and 14.23. An HTTP 1.1 request is required to include a Host header, and an HTTP 1.1 server is required to send a 400 reply if the header is missing and no host is specified in the request line.
char *message = "GET / HTTP/1.1\r\nHost: hostnamehere\r\n\r\n";
As for the text content, you need to read from the socket until you encounter a \r\n\r\n sequence (which terminates the response headers), then process the headers, then read the text content accordingly. The response headers tell you how to read the raw bytes of the text content and when to stop reading (refer to RFC 2616 section 4.4 for details). Once you have the raw bytes, the Content-Type header tells you how to interpret the raw bytes (data type, charset, etc).
I'm trying to implement a server side script for sending push notifications to apple push notification server. I create the ssl connection, I send the payload - but am unable to get a response from the APNs. Here is my code:
import socket, ssl, pprint, struct, time, binascii
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# require a certificate from the server
ssl_sock = ssl.wrap_socket( s,
keyfile="/Users/Jeff/Desktop/pickmeup-key2-noenc.pem",
certfile="/Users/Jeff/Desktop/pickmeup-cert2.pem",
server_side=False,
do_handshake_on_connect=True,
cert_reqs=ssl.CERT_REQUIRED,
ca_certs="/Users/Jeff/Desktop/entrustrootcert.pem",)
#ciphers="ALL")
ssl_sock.connect(('gateway.sandbox.push.apple.com', 2195))
print repr(ssl_sock.getpeername())
print ssl_sock.cipher()
print pprint.pformat(ssl_sock.getpeercert())
command = '\x00'
identifier = 1987
expiry = time.time()
deviceToken = "9858d81caa236a86cc67d01e1a07ba1df0982178dd7c95aae115d033b93cb3f5"
alert = "This is a test message"
sound = "UILocalNotificationDefaultSoundName"
payload = "{\"aps\":{\"alert\":\"%s\",\"sound\":\"%s\"}}" %(alert, sound)
packetFormat = "!cIIH%dsH%ds" %(32, len(payload))
packet = struct.pack(packetFormat,
command,
identifier,
int(expiry),
32,
binascii.unhexlify(deviceToken),
len(payload),
payload)
nBytesWritten = ssl_sock.write(packet)
print "nBytesWritten = %d" %(nBytesWritten)
data = ssl_sock.read(1024)
print len(data)
ssl_sock.close()
Running this script, I generate the following output:
('17.149.34.132', 2195)
('AES256-SHA', 'TLSv1/SSLv3', 256)
{'notAfter': 'May 31 00:04:27 2012 GMT',
'subject': ((('countryName', u'US'),),
(('stateOrProvinceName', u'California'),),
(('localityName', u'Cupertino'),),
(('organizationName', u'Apple Inc'),),
(('organizationalUnitName', u'Internet Services'),),
(('commonName', u'gateway.sandbox.push.apple.com'),))}
nBytesWritten = 133
0
Any ideas on what might be going wrong? (I am sending enhanced push notifications so I am expecting a response from apple push notification server)
The key thing to note is that read() is returning no data. In Python, read() is supposed to block until data is available or the connection closes. Apple is closing your connection.
Why? Well, probably because you sent a malformed request. command=0 is a normal push notification; command=1 is enhanced. The big-endian 1987 will be interpreted as a 0-byte device token and a 1987-byte payload, neither of which are valid.
(And FWIW, I'd use B instead of c for the command ID; it seems to make more sense.)
you may consider https://github.com/djacobs/PyAPNs that wrapped lot of useful features, including:
error handling
support enhanced message format and auto resend messages which are sent before error response
non-blocking ssl socket connection with great performance
Apple Push notification server doesn't give a response, it's a one-way binary socket.
Rather than rolling your own solution you could try apns-python-wrapper or apns