Receiving data with sockets adding characters - sockets

I'm writing a mod for a game to allow me to use a second screen with a set of stats. Now I send the data and receive, however there's extra characters when I receive the data, such as , ' ). This is causing issue's preventing me to create an int variable. I've tried stripping these characters but still having problems. This is the error spat out: ValueError: invalid literal for int(*) with base 10: "(0'"
Here's what i'm sending:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = '192.168.0.2'
port = 9000
s.bind((host,port))
s.listen(5)
conn, addr = s.accept()
print ('Got connection from', addr)
while True:
#--- RPM --
rpm = info.physics.rpms
max_rpm = info.static.maxRpm
if rpm != 0:
rpm_percent = int(rpm/max_rpm*100)
else:
rpm_percent = 0
#--- SPEED ---
speed = int(info.physics.speedKmh/1.609344)
#--- GEAR ---
gear = info.physics.gear - 1
if gear == 0:
gear = str("N")
elif gear < 0:
gear = str("R")
#--- FUEL ---
fuel = info.physics.fuel
max_fuel = info.static.maxFuel
if fuel != 0:
fuel_percent = int(fuel/max_fuel*100)
else:
fuel_percent = 0
#--- BRAKING ---
brake = info.physics.brake
braking_lvl = int(brake/100*10000)
#--- LAP ---
current_time = info.graphics.currentTime
#--- SEND DATA ---
time.sleep(.1)
#print_data = 'Speed: ', speed, 'RPM: ', rpm_percent, 'Gear: ', gear, 'Braking: ', braking_lvl, 'Fuel: ', fuel, fuel_percent,'Lap Time: ', current_time
send_data = (speed, ';', rpm, ';', rpm_percent, ';', gear, ';', braking_lvl, ';', fuel, ';', fuel_percent)
conn.send(str(send_data).encode('ascii'))
print(send_data)
s.close()
And on the client, which has the errors:
data = s.recv(1024).decode('ascii').replace(' ', '').replace("'", '').replace(')', '').replace(',', '')
data = data.split(';')
speed = int(data[0])
rpm = int(data[1])
rpm_percentage = int(data[2])
gear = str(data[3])
breaking_lvl = int(data[4])
fuel = int(data[5])
fuel_percentage= int(data[6])
Why is it seeing these extra characters? And what's the most effective method to remove them / prevent them?
Cheers.

send_data is a tuple. If you call str(); it returns its string representation '(item, item)'. To read it back you would need eval() or ast.literal_eval() (in simple cases). You could use json format instead.
On the server:
sock.sendall(json.dumps(data).encode('utf-8'))
On the client (assuming the connection is closed after receiving the response from the server):
response = []
while True:
chunk = sock.recv(1024)
if not chunk: # EOF
break
response.append(chunk)
data = json.loads(b"".join(response).decode('utf-8'))

You need to check how much data you receive, and ensure that the string is null terminated at the end of the data that was received, otherwise you will be parsing random data.
For reliability, you should send a fixed length message that tells the client how much data to expect. Then the client can ensure that it receives all the data sent, and no more.

Related

Socket recvfrom function hanging, not recognizing ICMP packet

So I'm trying to do a mock Traceroute function where a UDP packet is sent to an IP address. I'm trying to design the program in such a way where a packet is sent each time the packet makes it to a router. I am trying to do this by making a very short TTL. However, the recvfrom function is stalling.
Here's the code:
host_addr = gethostbyname(host)
port = 33434
max_hops = 30
ttl = 1
while True:
recv_socket = socket(AF_INET, SOCK_RAW, IPPROTO_ICMP)
send_socket = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)
send_socket.setsockopt(0, 4, ttl)
recv_socket.bind(("", port))
send_socket.sendto("waddup".encode(), (host_addr, port))
cur_addr = None
cur_name = None
path_end = False
cur_bytes = None
attempts_left = 3
timeReceived = 0
pingStartTime = 0
while not path_end and attempts_left > 0:
try:
pingStartTime = time.time()
cur_bytes, cur_addr = recv_socket.recvfrom(1024)
timeReceived = time.time()
path_end = True
cur_addr = cur_addr[0]
try:
cur_name = gethostbyaddr(cur_addr)[0]
except error:
cur_name = cur_addr
except error:
attempts_left -= 1
send_socket.close()
recv_socket.close()
if not path_end:
pass
if cur_addr is not None:
cur_host = "%s (%s) " % (cur_name, cur_addr)
else:
cur_host = ""
print("%d: %.0f ms " %
(
ttl,
(timeReceived - pingStartTime) * 1000,
) + cur_host
)
ttl += 1
if cur_addr == host_addr or ttl > max_hops:
break
I have set up the receiving socket for an ICMP packet as far as I can tell but it just hangs on recvfrom. I've allowed all ICMP connections on Windows Defender and when I run Wireshark, an appropriate ICMP packet is sent to my router.
The packet with the TTL expired message is the one I want to receive
I'm new to networking but all the code I've seen online has this exact setup. Would anyone be able to tell me why it's stalling and what I can do to fix it? I want the program to read the ICMP packet with the TTL expiration message.

Most simple network protocol for a remote function call?

I am looking for the most simple protocol to program a remote function call, e.g. from Matlab to Julia.
[out1, out2, ...] = jlcall(socket, fname, arg1, arg2, ...);
fname is a string, all other input and output variables are numerical arrays (or other structures known to both sides) on Linux, Windows as option.
Client: open connection, block, pack and transmit
Server: receive, unpack, process, pack and transmit back
Client: receive, unpack, close connection and continue
The solutions I've seen (tcp, zmq) were built with old versions and do no longer work.
Protocol could (should?) be limited to do the pack/transmit - receive/unpack work.
UPDATE
Here is what I have come up with using pipes:
function result = jlcall(varargin)
% result = jlcall('fname', arg1, arg2, ...)
% call a Julia function with arguments from Matlab
if nargin == 0 % demo
task = {'foo', 2, 3}; % demo fun, defined in jsoncall.jl
else
task = varargin;
end
% create pipe and write function and parameter(s) to pipe
pipename = tempname;
pipe = java.io.FileOutputStream(pipename);
pipe.write(uint8(jsonencode(task)));
pipe.close;
% run Julia and read result back
system(sprintf('julia jsoncall.jl %s', unixpath(pipename)))
fid = fopen(pipename, 'r');
c = fread(fid);
result = jsondecode(char(c'));
fclose(fid);
function path_unix = unixpath(path_pc)
%convert path to unix version
path_unix = path_pc;
path_unix(strfind(path_unix,'\'))='/';
# jsoncall.jl
using JSON3 # get JSON3.jl from repository first
function foo(a,b) # demo function
a+b, a*b
end
jsonfile = ARGS[1] # called as > julia jsoncall.jl <json_cmdfile>
io = open(jsonfile, "r") # open IOStream for read
data = read(io) # read UTF8 data from stream
close(io) # close stream
cmd = JSON3.read(String(data)) # unpack stream into [fun, farg] array
fun = Symbol(cmd[1]) # first element is Julia function name,
result = #eval $fun(cmd[2:end]...) # others are function arguments
io = open(jsonfile, "w") # open IOStream for write
write(io, JSON3.write(result)) # (over-)write result back to stream
close(io) # close stream
Open points:
my first use of pipes/streams
output formatting: where Julia outputs a tuple of two, Matlab creates an an nx2 array.
replace json by msgpack for performance, might help with type formatting as well.
Your comments are welcome!
Here is a stripped down way of doing it. If you are going to vary your functions and arguments, a REST as in the comments server is going to be more flexible and less likely to pose a security risk (as you are eval()ing arbitrary code in some cases).
#server code
using Sockets
const port = 6001
const addr = ip"127.0.0.1"
const server = listen(addr, port)
while true
try
#info "Server on $port awaiting request..."
sock = accept(server)
#info "Server connected."
msg = strip(readline(sock))
#info "got message $msg"
fstr, argstr = split(msg, limit=2)
x = parse(Float64, argstr) # or other taint checks here...
ans = eval(Meta.parse(fstr * "($x)"))
#info "server answer: $ans"
write(sock, "$ans\n")
catch y
#info "exiting on condition: $y"
end
end
# client code
using Sockets
port = 6001
server = ip"127.0.0.1"
sock = connect(server, port)
#info "Client connected to $server"
func = "sinpi"
x = 0.5
#info "starting send"
write(sock, "$func $x\n")
flush(sock)
#info "flushed send"
msg = strip(readline(sock)) # read one line of input and \n, remove \n
ans = parse(Float64, msg)
println("answer is $ans")
close(sock)

Issue sending/receiving data over serial connection using MATLAB

I recently connected a reactor control tower through a serial 'COMMS' port to my computer (serial to USB). It seems to create a connection on the COM4 port (as indicated on the control panel 'devices and printers' section). However, it always gives me the following message when i try to 'fwrite(s)' or 'fscanf(s)'.
Warning: Unsuccessful read: The specified amount of data was not returned within the Timeout period..
%My COM4
s = serial('COM4');
s.BaudRate = 9600;
s.DataBits = 8;
s.Parity ='none';
s.StopBits = 1;
s.FlowControl='none';
s.Terminator = ';';
s.ByteOrder = 'LittleEndian';
s.ReadAsyncMode = 'manual';
% Building write message.
devID = '02'; % device ID
cmd = 'S'; % command read or write; S for write
readM = cell(961,3);% Read at most 961-by-3 values filling a 961–by–3 matrix in column order
strF = num2str(i);
strF = '11'; %pH parameter
strP = '15'; %pH set point
val = '006.8'; %pH set value
msg_ = strcat('!', devID, cmd, strF, strP, val);%output the string
chksum = dec2hex(mod(sum(msg_),256)); %conversion to hexdec
msg = strcat(msg_,':', char(chksum), ';');
fopen(s); %connects s to the device using fopen , writes and reads text data
fwrite(s, uint8(msg)); %writes the binary data/ Convert to 8-bit unsigned integer (unit8) to the instrument connected to s.
reply=fscanf(s); %reads ASCII data from the device connected to the serial port object and returns it to reply, for binary data use fread
fclose(s); %Disconnect s from the scope, and remove s from memory and the workspace.
This leads me to believe that the device is connected but is not sending or receiving information and I am unsure as to how I can configure this or even really check if there is communication occurring between the tower and my computer.

Incorrect throughput value in python

I am writing a python code to find throughput between server and client. It is based on speedtest.net functionality where I am sending a dummy file to calculate the speed. The problem I am facing is unreliable throughput output. I will appreciate your suggestions on the same. Here is the code.
server.py
import socket
import os
port = 60000
s = socket.socket()
host = socket.gethostname()
s.bind((host, port))
s.listen(5)
print 'Server listening....'
while True:
conn, addr = s.accept()
print 'Got connection from', addr
data = conn.recv(1024)
print('Server received', repr(data))
filename='akki.txt'
b = os.path.getsize(filename)
f = open(filename,'rb')
l = f.read(b)
while (l):
conn.send(l)
l = f.read(b)
f.close()
print('Done sending')
conn.send('Thank you for connecting')
conn.close()
Client.py
import socket
import time
import os
s = socket.socket()
host = socket.gethostname()
port = 60000
t1 = time.time()
s.connect((host, port))
s.send("Hello server!")
with open('received_file', 'wb') as f:
print 'file opened'
t2 = time.time()
while True:
data = s.recv(1024)
if not data:
break
f.write(data)
t3 = time.time()
print data
print 'Total:', t3 - t1
print 'Throughput:', round((1024.0 * 0.001) / (t3 - t1), 3),
print 'K/sec.'
f.close()
print('Successfully received the file')
s.close()
print('connection closed')
Output when sending akki.txt
Server Output
Server listening....
Got connection from ('10.143.47.165', 60902)
('Server received', "'Hello server!'")
Done sending
Client output
file opened
Raw timers: 1503350568.11 1503350568.11 1503350568.11
Total: 0.00499987602234
**Throughput: 204.805 K/sec.**
Successfully received the file
connection closed
Output for ak.zip ( which is bigger file)
Client output
file opened
Total: 0.0499999523163
**Throughput: 20.48 K/sec.**
Successfully received the file
connection closed
Short Answer: you need to take the file size into consideration.
More Details:
Throughput is data/time. Your calculation:
round((1024.0 * 0.001) / (t3 - t1), 3)
Doesn't take the file size into account. Since sending a large file takes more time, 't3-t1' is bigger so your throughput is smaller (same numerator with larger denominator). Try adding the file size to the formula and you should get much more constant results.
Hope this helps.

Python 3: Server socket not closing properly, infinite loop of blank messages received

EDIT: I solved this question myself, look at the first answer if you have a similar issue
I am new to sockets and made a client-server test modifying one example, it sends and receives data ok but when closing the server get caught in a loop receiving blank messages and I can't find out why even though I use shutdown(socket.SHUT_RDWR) to close the connection immediately according to the documentation.
I had to use two cmd of windows for each one (server and client) because when running first the server in idle and then the client I got in client_example ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it here is the code. Anyways, here it is the sequence I tried:
(in the client console):
c:\python33\python client_example.py
SEND( TYPE q or Q to Quit):Hi
SEND( TYPE q or Q to Quit):q
(in the server console):
c:\python33\python server_example3.py
TCPServer Waiting for client on port 7000
I got a connection from ('127.0.0.1', 49263)
RECEIVED: b'Hi'
RECEIVED: b''
RECEIVED: b''
RECEIVED: b''
RECEIVED: b''
RECEIVED: b''
.... <---After I did CTRL+C
Traceback (most recent call last):
File "server_example.py", line 19, in <module>
print("RECEIVED:",data)
File "c:\python33\lib\encodings\cp850.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_map)[0]
KeyboardInterrupt
Here is the code of the server and client:
server_example.py:
#TCP server example
import socket
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(("", 7000))
server_socket.listen(5)
print("TCPServer Waiting for client on port 7000")
while 1:
client_socket, address = server_socket.accept()
print("I got a connection from ", address)
while 1:
data = client_socket.recv(32)
if (data == 'q' or data == 'Q'):
client_socket.shutdown(socket.SHUT_RDWR)
client_socket.close()
break;
else:
print("RECEIVED:",data)
client_example.py:
# TCP client example
import socket
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(("localhost", 7000))
while 1:
data = input( "SEND( TYPE q or Q to Quit):" )
if (data != 'Q' and data != 'q'):
my_bytes = bytearray()
for c in data:
my_bytes.append(ord(c))
client_socket.send(my_bytes)
else:
client_socket.shutdown(socket.SHUT_RDWR)
client_socket.close()
break;
Well, I feel dumb that I solved this thing after 11 minutes of posting the question when I was unable to solve it for 2 hours...
The problem was that I was comparing data with the string 'q' which was correct in python 2 but in python 3 what is being sent are bytes, b'q' <> 'q' ...
Here is the corrected code:
server_example.py
#TCP server example
import socket
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(("", 7000))
server_socket.listen(5)
print("TCPServer Waiting for client on port 7000")
while 1:
client_socket, address = server_socket.accept()
print("I got a connection from ", address)
while 1:
data = client_socket.recv(32)
print("RECEIVED:",data)
if (data == b'q' or data == b'Q'):
## client_socket.shutdown(socket.SHUT_RDWR)
client_socket.close()
break;
break;
client_example.py
# TCP client example
import socket
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(("localhost", 7000))
while 1:
data = input( "SEND( TYPE q or Q to Quit):" )
if (data != 'Q' and data != 'q'):
my_bytes = bytearray()
for c in data:
my_bytes.append(ord(c))
client_socket.send(my_bytes)
else:
my_bytes = bytearray()
my_bytes.append(ord(data))
client_socket.send(my_bytes)
## client_socket.shutdown(socket.SHUT_RDWR)
client_socket.close()
break;
You're assuming you're going to get the Q. You should also engage in the same processing when you get EOS from read, whatever form that takes in Python.
#EJP : By EOS you mean end of stream? If so you are right, this example was a very simple one to get started in which I overlooked details like these.
Anyway I found that using this check in the server it is solved:
data = input( "SEND( TYPE q or Q to Quit):" )
if (data == b''):
break;