Sniffing client to server socket communication - sockets

I was developing an IoT application with a BBB and a Quectel GSM modem to send data over internet.
I configured the modem on the debian OS with PPP. Then with something similar to this example I stablished communication between my server and my client. Data was trivial, I send just a simple "Close!", and receive a simple "OK!". Sniffing the serial port, I was expecting something simpler, but here is what I saw:
BBB >> GSM modem
ATZ
ATQ0 V1 E1 S0=0 &C1 &D2
AT+CGDCONT=1,"IP","internet.comcel.com.co"
ATD*99***1#
~}#!}!}!} }4}"}&} } } } }%}&"}'}"}(}"0~~}#!}!}"} }4}"}&} }*} } }%}&"}'}"}(}"}>~~}#!}$}!} }(}#}$#,~~}#!}"}"} }.}"}&} }*} } }'}"}(}"P~! "?~xx/2~!-/~!
!~!
j~!
_3~~!E<##_
_-7`99
}3BT~~!E<##^
_-7`99
}3D~!E4##e
_-7`:l5s
}3FGcE~!E:##^
_-7`:l5d
}3FGcEClose!=~~!E4##c
_-7`#l5
}3HcEh$~!E4##b
_-7`#l5
}3HcE~!E4##a
_-7`#l8
}3IycG~!E4##`
_-7`#l8}1
}3Iyc~!E4##_
_-7`Al9
}3I|c%~~!E4##^
_-7`#l9}1
}3Sc
!~~! "=~~!E4##\
_-7`#l9}1
}3cws~
Modem >> BBB
+CFUN: 1
+CPIN: READY
ATZ
OK
ATQ0 V1 E1 S0=0 &C1 &D2
OK
AT+CGDCONT=1,"IP","internet.comcel.com.co"
OK
ATD*99***1#
CONNECT
~}#!}!}!} }2}"}&} }*} } }#}$#}'}"}(}"U~~}#!}#}!} }*}"}&} }*} } lP~~}#!}"}"} }4}"}&} }*} } }%}&gHT}'}"}(}"Q~~}#!}!}!} }2}"}&} }*} } }#}$#}'}"}(}"U~~}#!}!}"} }.}"}&} }*} } }'}"}(}"~~!
H~~!
k~~!xx/~~!
-0~~!
xP`~~!
xP~~!E<#1-7
xPyCdOq x
d/2mQ~~!E<#1-7
xPyCdOq _x
d0N2~~!E4"#1Bw-7
xPzCdUrN
d0ev ~~!E7##1Bs-7
xPzCdUrd
d0eOK!3~~!E4$#1Bu-7
xP}]CdU}1rN
d0e~
I'm familiar with those AT commands, the "Close!" is there, and on the other side the "OK! is there too, but I didn't expect all those strange characters. I must say that I'm not very familiar with low level development for the TCP/IP network model, but I've always thought that sockets communication was the lowest layer (not counting physical layers).
What am I missing and where can I find more info related to this topic?
Thank you.

Related

Why does Rasp Pi Pico can not connect to TCP Server after some point?

I use Raspberry Pi Pico with ESP8266 WiFi module, and I am trying to write a TCP client. Rasp Pi Pico is able to send AT commands and receive responses and send data through UART. Also the TCP client is able to send data to the TCP server, which runs in my laptop. However the problem is that the client is not able to connect to the server after some point.
Let me first show the server-side code. In server, I am trying to receive data basically. ConnectionResetError was a problem for me so I wrote the following except block. I am not sure it is buggy or not, since I'm kind of a noob in this area.
import socket
HOST = ""
PORT = 8080
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
mysocket.bind((HOST, PORT))
mysocket.listen(1)
while True:
print("Waiting...")
conn, addr = mysocket.accept()
print('[SERVER] - Connected from: %s' % str(addr))
while True:
try:
request = conn.recv(1024)
if not request:
break
conn.sendall(request.upper())
print('[SERVER] - Received Data: %s' % str(request))
except ConnectionResetError as cr_err:
break
conn.close()
print("[SERVER] - Disconnected")
Here is my client-side code. In client, I wrote two helper classes called ESP8266 and Sensor, in which I control the WiFi module and read analog value from a sensor. Pico first tries to start WiFi module, afterwards it tries to connect to TCP server and send data. After some point it does not connect to the TCP server, so it restart the WiFi module and reconnects.
class EndDevice:
def __init__(self, sensor_id):
self.__wifi_module = ESP8266(UART_PIN, BAUDRATE)
self.__sensor = Sensor(sensor_id, SENSOR_PIN)
def start(self):
self.__wifi_module.start()
self.__wifi_module.set_mode(STATION_MODE)
self.__wifi_module.join_access_point(AP_NAME, AP_PWD)
def reconnect(self):
self.__wifi_module.restart()
self.__wifi_module.set_mode(STATION_MODE)
self.__wifi_module.join_access_point(AP_NAME, AP_PWD)
def run(self):
retry_count = 0
while True:
if self.__wifi_module.start_connection("TCP", SERVER_HOST, SERVER_PORT):
self.__wifi_module.send_data(
str(self.__sensor.generate_package()))
self.__wifi_module.close_connection()
else:
retry_count += 1
if retry_count == MAX_RETRY:
break
if __name__ == "__main__":
pico = EndDevice("SM-0")
pico.start()
while True:
pico.run()
pico.reconnect()
Finally I will share some of the methods in classes ESP8266 and UARTHandler (which is used in ESP8266), so you can see if I do anything non-sense.
start_connection method in ESP8266 is as follows. In this method, I tried to send the corresponding AT command to connect to a TCP server. In the method self.__uart_handler.send_receive_cmd timeout duration is 2000ms, and other parameters are AT command, connection_type (TCP), server IP address and server port, in order.
def start_connection(self, conn_type, remote_ip, remote_port):
conn_type, remote_ip = "\"{}\"".format(conn_type), "\"{}\"".format(remote_ip)
response = self.__uart_handler.send_receive_cmd(2000, CONN_START, conn_type, remote_ip, str(remote_port))
if "OK" in response:
self.__log("Connected to {} at port {}.".format(remote_ip, remote_port))
return True
else:
self.__log("Failed to create a connection with {} at port {}.".format(remote_ip, remote_port))
return False
send_receive_cmd method in UARTHandler is as follows. In this method I use lots of helper methods as you can see, however they are just formatting and writing to UART or reading from UART. I also insert a timeout between UART-read and UART-write
def __generate_cmd(self, cmd, *args):
if len(args) != 0:
cmd += "="
for idx, each in enumerate(args):
cmd += str(each)
if idx != len(args)-1:
cmd += ","
cmd += "\r\n"
return cmd
def __send_cmd(self, cmd, *args):
sent_cmd = self.__generate_cmd(cmd, *args)
self.__uart.write(sent_cmd)
def __receive_response(self, cmd):
response = self.__uart.read()
try: return response.decode('utf-8')
except: return response
def send_receive_cmd(self, timeout, cmd, *args):
self.__send_cmd(cmd, *args)
utime.sleep_ms(timeout)
return self.__receive_response(self.__generate_cmd(cmd, *args))
Let me ask my question again. This codes are working properly in starting-restarting and sending data for (let me say) 3 connections. However, after some connect-disconnect later, TCP client is not able to make a connection with TCP server. Again after some failed connection attempt, WiFi module is restarted and TCP connection is made and working properly again.

I need help to get data sent from an instrument via udp protocol

i have an instrument that measures so2 and i have a standard program that makes me see the data, but i need to connect the instrument to a server and send all the data to a database. I used wireshark to see how it comunicate with the software, but i don't understand what method to use to make my program. Here i have the wireshark dump:
the red circle indicates the measure that the instrument made.
P.S. sorry for my bad eng
So i solved my question as i'm about to show:
i made a listener and a caller;
caller:
`from socket import socket, AF_INET, SOCK_DGRAM
SERVER_IP = '192.168.1.99'
PORT_NUMBER = 53700
SIZE = 1024
print ("Test client sending packets to IP {0}, via port {1}\n".format(SERVER_IP, PORT_NUMBER))
mySocket = socket( AF_INET, SOCK_DGRAM )
mySocket.bind(('192.168.1.100', 57806))
while True:
data = bytes.fromhex('014630303430335230303102313103')
mySocket.sendto(data,(SERVER_IP,PORT_NUMBER))
exit()`
listener:
`from socket import socket, gethostbyname, AF_INET, SOCK_DGRAM
import sys
PORT_NUMBER = 57806
SIZE = 1024
hostName = gethostbyname( '' )
mySocket = socket( AF_INET, SOCK_DGRAM )
mySocket.bind( (hostName, PORT_NUMBER) )
print ("Test server listening on port {0}\n".format(PORT_NUMBER))
while True:
(data,addr) = mySocket.recvfrom(SIZE)
print (data)
sys.ext()`
i saw on wireshark that if i sent a request packet copied from the ones that i already have, the instrument would give me back the response; so i setupped a listener on the a choosen port, and now i get all the data!
the highlighted packet is the one i sent from the caller script.

Connecting two computers in different networks using socket and Port forwarding

I have set up a simple client-server communication code and it works well in my computer when my computer itself acts as a server and the client.
Now I am trying to run this same code on two different computers in different networks( different locations) where my computer will act as a server and my friend's computer as a client.
I have done port forwarding in my router as well as in my friend's router for the port which we are trying to communicate. We both have set up a static IP in our internal network behind the router. We both had shutdown the firewall while running the code.
I am running my code on Jupiter notebook and the same is my friend too.
here is my server code:
import socket
import threading
HEADER = 64
PORT = 5064
SERVER = '0.0.0.0'
ADDR = (SERVER, PORT)
FORMAT = 'utf-8'
DISCONNECT_MESSAGE = "quit"
Receive_from_client = "get info"
server = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server.bind(ADDR)
def handle_client(conn, addr):
print("\n" + f"[NEW Connection] detected from IP: {addr[0]} & Port:{addr[1]} ")
conn.send(f"connected to server {ADDR}".encode(FORMAT))
connected =True
while connected:
msg_length = conn.recv(HEADER).decode(FORMAT) # decode the msg from byte to utf-8 format
if msg_length:
msg_length = int(msg_length)
msg = conn.recv(msg_length).decode(FORMAT)
if msg == DISCONNECT_MESSAGE:
connected = False
print(f" [Client][{addr}] {msg}")
print("Your session is disconnected")
break
if msg == Receive_from_client:
print("\n" + f"Send your msg to client with IP: {addr[0]}")
thread = threading.Thread(target = send2client, args = (conn, addr))
thread.start()
print(f" [Client][{addr}] {msg}")
conn.send(f"Msg received by server with IP:{addr[0]}".encode(FORMAT))
conn.close()
server.close()
def start():
server.listen()
print("\n"+ f"[LISTENING] Server is listening from IP: {SERVER} ")
while True:
conn, addr = server.accept()
thread = threading.Thread(target = handle_client, args = (conn, addr))
thread.start()
Here is the client code
import socket
import threading
HEADER = 64
PORT = 5064
FORMAT = 'utf-8'
DISCONNECT_MESSAGE = "quit"
SERVER = '103.192.207.250' # SERVERS public IP
ADDR = (SERVER, PORT)
client = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
client.connect(ADDR)
def send2server():
while True:
msg = input()
message = msg.encode(FORMAT)
msg_lenght = len(message)
send_length = str(msg_lenght).encode(FORMAT)
send_length += b' '*(HEADER - len(send_length))
client.send(send_length)
client.send(message)
print(client.recv(2048).decode(FORMAT))
if msg == DISCONNECT_MESSAGE:
print("session closed")
client.close()
def start():
print("\n"+ f"[LISTENING] client is listening from IP: {ADDR} ")
send2server()
I have opened the port by going on windows firewall defender and selecting new inbound and outbound rules to open 5064 TCP port.
but still, the code doesn't works..
my server keeps waiting for connection and the client-side after few seconds of running gives this error:
TCP error code 10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
Even while my server is listening from port 5064 when I scan this port to check if the port is open or not it says closed.
How do I check if the port I have forwarded is for sure open and also how do I get this thing work?
I have tried all of this and if there is any other thing I am missing please tell. I am struggling to get this work for the past 3 days.

TCP data sometimes not received by java (or python) server

I'm developing a system that consists of an arduino mkr1000 that I want to send data via wifi to a java server program running in my local network.
Everything works except the main part: data sent by the arduino is sometimes not received by the server...
I'm using the arduino Wifi101 library to connect to my wifi, get a WiFiClient and send data.
The following code is just a example to demonstrate the problem:
for (int i = 0; i < 3; ++i) {
Serial.println(F("Connecting to wifi"));
const auto status = WiFi.begin("...", "...");
if (status != WL_CONNECTED) {
Serial.print(F("Could not connect to WiFi: "));
switch (status) {
case WL_CONNECT_FAILED:
Serial.println(F("WL_CONNECT_FAILED"));
break;
case WL_DISCONNECTED:
Serial.println(F("WL_DISCONNECTED"));
break;
default:
Serial.print(F("Code "));
Serial.println(status, DEC);
break;
}
} else {
Serial.println(F("WiFi status: WL_CONNECTED"));
WiFiClient client;
if (client.connect("192.168.0.102", 1234)) {
delay(500);
client.print(F("Test "));
client.println(i, DEC);
client.flush();
Serial.println(F("Data written"));
delay(5000);
client.stop();
} else {
Serial.println(F("Could not connect"));
}
WiFi.end();
}
delay(2000);
}
The java server is based on Netty but the same thing with manually creating and reading from a Socket yields the same result.
The testing code is pretty standard with only a simple output (note: in Kotlin):
val bossGroup = NioEventLoopGroup(1)
val workerGroup = NioEventLoopGroup(6)
val serverFuture = ServerBootstrap().run {
group(bossGroup, workerGroup)
channel(NioServerSocketChannel::class.java)
childHandler(object : ChannelInitializer<NioSocketChannel>() {
override fun initChannel(ch: NioSocketChannel) {
ch.pipeline()
.addLast(LineBasedFrameDecoder(Int.MAX_VALUE))
.addLast(StringDecoder())
.addLast(object : ChannelInboundHandlerAdapter() {
override fun channelRead(ctx: ChannelHandlerContext, msg: Any) {
println("msg = $msg")
ctx.close()
}
})
}
})
bind(port).sync()
}
The arduino tells that everything is OK (i.e. writing Data written to the serial console for each iteration) but the server sometimes skips individual messages.
Adding the LoggingHandler from Netty tells me in these cases:
11:28:48.576 [nioEventLoopGroup-3-1] WARN i.n.handler.logging.LoggingHandler - [id: 0x9991c251, L:/192.168.0.20:1234 - R:/192.168.0.105:63845] REGISTERED
11:28:48.577 [nioEventLoopGroup-3-1] WARN i.n.handler.logging.LoggingHandler - [id: 0x9991c251, L:/192.168.0.20:1234 - R:/192.168.0.105:63845] ACTIVE
In the cases where the message is received it tells me:
11:30:01.392 [nioEventLoopGroup-3-6] WARN i.n.handler.logging.LoggingHandler - [id: 0xd51b7bc3, L:/192.168.0.20:1234 - R:/192.168.0.105:59927] REGISTERED
11:30:01.394 [nioEventLoopGroup-3-6] WARN i.n.handler.logging.LoggingHandler - [id: 0xd51b7bc3, L:/192.168.0.20:1234 - R:/192.168.0.105:59927] ACTIVE
11:30:01.439 [nioEventLoopGroup-3-6] WARN i.n.handler.logging.LoggingHandler - [id: 0xd51b7bc3, L:/192.168.0.20:1234 - R:/192.168.0.105:59927] READ: 8B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 54 65 73 74 20 32 0d 0a |Test 2.. |
+--------+-------------------------------------------------+----------------+
11:30:01.449 [nioEventLoopGroup-3-6] WARN i.n.handler.logging.LoggingHandler - [id: 0xd51b7bc3, L:/192.168.0.20:1234 - R:/192.168.0.105:59927] CLOSE
11:30:01.451 [nioEventLoopGroup-3-6] WARN i.n.handler.logging.LoggingHandler - [id: 0xd51b7bc3, L:/192.168.0.20:1234 ! R:/192.168.0.105:59927] READ COMPLETE
11:30:01.453 [nioEventLoopGroup-3-6] WARN i.n.handler.logging.LoggingHandler - [id: 0xd51b7bc3, L:/192.168.0.20:1234 ! R:/192.168.0.105:59927] INACTIVE
11:30:01.464 [nioEventLoopGroup-3-6] WARN i.n.handler.logging.LoggingHandler - [id: 0xd51b7bc3, L:/192.168.0.20:1234 ! R:/192.168.0.105:59927] UNREGISTERED
With my understanding this means that the TCP packets are indeed received but in the faulty cases the IO thread from Netty is waiting to read the TCP data but does never continue...
The same problem exists when trying with a rudimentary python server (just waiting for a connection and printing the received data).
I confirmed the data is sent by using tcpflow on Arch Linux with the arguments -i any -C -g port 1234.
I even tried the server on a Windows 7 machine with the same results (TCP packets confirmed with SmartSniff).
Strangely using a java server to send the data always and reproducibly is received...
Does anybody have any idea to solve the problem or at least how to diagnose?
PS: Maybe it is important to note that with tcpflow (i.e. on linux) I could watch the TCP packets being resent to the server.
Does this mean the server is receiving the packets but not sending an ACK?
SmartSniff didn't show the same behavior (but maybe I used wrong options to display the resent packets).
In the meantime I send messages to acknowledge receiving another message. If the acknowledgement is not received the message is sent again.
For anyone with the same problem:
While testing something different I updated the wifi firmware of the board to the latest version 19.5.2. Since then I haven't noticed any lost data. Maybe this was the problem.
See Check WiFi101 Firmware Version and Firmware and certificates Updater.
Note: I couldn't get the sketches to run with the Arduino IDE but with PlatformIO.

wvdial NO CARRIER error

I've been trying to connect Huawei E3131 to my Raspberry pi.
So far I've installed ppp, usb_modeswitch and wvdial. Here is my wvdial.conf partially generated by wvdialconf and partially changed by myself:
[Dialer Defaults]
Modem = /dev/modem
Username = { }
Password = { }
Baud = 9600
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0
Init3 = ATX3
Carrier Check = no
Stupid Mode = 1
Modem Type = Analog Modem
ISDN = 0
Phone = *+88690xxxxxxx#
New PPPD = yes
In the end the output I get looks like that:
--> Initializing modem.
--> Sending: ATZ
ATZ
OK
--> Sending: ATQ0 V1 E1 S0=0
ATQ0 V1 E1 S0=0
OK
--> Sending: ATX3
ATX3
OK
--> Modem initialized.
--> Sending: ATDT*+88690xxxxxxx#
--> Waiting for carrier.
ATDT*+88690xxxxxxx#
NO CARRIER
--> No Carrier! Trying again.
--> Sending: ATDT*+88690xxxxxxx#
--> Waiting for carrier.
ATDT*+88690xxxxxxx#
NO CARRIER
--> No Carrier! Trying again.
The simcard in the dongle is from Chunghwa Telecom (Taiwan) if that is any help.
Ok, apparently you don't need to change the basic *99# to your phone number. So the final config file looks like
[Dialer Defaults]
Modem = /dev/modem
Username = { }
Password = { }
Baud = 9600
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0
Init3 = ATX3
Carrier Check = no
Stupid Mode = 1
Modem Type = Analog Modem
ISDN = 0
Phone = *99#
New PPPD = yes
run it with sudo for creation of ppp and everything works fine. Also you may run wvdialconf to generate the config file with suitable commands. However parameteres are not affected by it, except Baud. Hope someone will find it useful.