ESP32 MicroPython SSL WebSocket server fail - sockets

I'm trying to set up a secure socket server on esp32 with micropython. I used/tried the latest bulid (esp32-idf3-20200117-v1.12-68-g3032ae115.bin) with a self-signed certificate.
I saw a lot of memory leak related problem with ssl.wrap_socket() on esp32/esp8266 but what I got is different:
mbedtls_ssl_handshake error: -4310
Traceback (most recent call last):
File "boot.py", line 100, in <module>
OSError: [Errno 5] EIO
and the connection fails of course. I try to connect from my laptop. The exact same code of client side works if I start a secure socket server on the laptop itself (127.0.0.1) so I suppose that the client side is OK and the problem is on the server side.
I could not find any solution for this problem yet. I tried certificate and key both in 'der' and 'pem' format the result is the same.
The toy server code I tried:
import esp
esp.osdebug(None)
import gc
gc.collect()
import usocket as socket
import ssl
import machine
import network
KEY_PATH = 'server_key.der'
CERT_PATH = 'server_cert.der'
ssid= "AP_local"
pw="passwd"
ap = network.WLAN(network.AP_IF) # create access-point interface
ap.config(essid=ssid, password=pw, authmode=4) # set the ESSID of the access point
ap.ifconfig(('192.168.3.1', '255.25.255.0', '192.168.3.1', '8.8.8.8'))
ap.active(True)
with open(KEY_PATH, 'rb') as f:
key = f.read()
print(key)
print(" ")
with open(CERT_PATH, 'rb') as f:
cert = f.read()
#s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
ad = ap.ifconfig()
addr = socket.getaddrinfo(ad[0], 8443)[0][-1]
s.bind(addr)
s.listen(5)
import gc
gc.collect()
while True:
print("ssl connection started")
cl, addr = s.accept()
scl = ssl.wrap_socket(cl, server_side=True, cert=cert, key=key)
print(gc.mem_free())
l = 0
while True:
req = scl.read(1024)
print(req)
if not req or b'\r\n' in req:
break
response = '\r\n'.join(['HTTP/1.1 200 OK',
'Content-Type: text/plain',
'OK',
'Connection: close', '\r\n'])
scl.write(response.encode("utf-8"))
scl.close()
I hope someone could help me with this, thanks!

OK, after spending a few days on searching and reading and testing I got a solution and a possible reason for this issue. I hope it will help others a little.
I want to put forward that I didn't dig into the source code of mbedtls so the reasoning is somewhat phenomenological.
First, I tried to connect from a python program with default settings which failed as shown above. Then I tried with the openssl cli, which also failed.
I tried with the example given at https://github.com/micropython/micropython/blob/master/examples/network/http_server_ssl.py and failed again but with different error codes.
Finally I found a very useful page:
https://tls.mbed.org/api/ssl_8h.html#a55ed67b6e414f9b381ff536d9ea6b9c0
which helped to understand where the problem occures.
Now depending on requirements different errors occured from -7900,-7780, -7380, -7d00.
It turned out that although in the documentation the cipher suit being used is automatically agreed during handshake but there is some bug in it or the implementation of some of them in micropython is different.
I didn't tested all the available ciphers on my laptop but e.g.: AES_256_GCM_SHA384
cipher works.
For me it is enough now.

Related

OSError micropython

I get OSError on esp8266. First request is successful but second and more are failed with OSError, I dont know why. Can you help me?
Edit: I solved it. I wrote the solution at the end of the codes.
import network
name, password="wifiname", "passwordd"
wlan = network.WLAN(network.STA_IF)
wlan.active(True) # activate the interface
wlan.scan() # scan for access points
wlan.isconnected() # check if the station is connected to an AP
wlan.connect(name, password) # connect to an AP
wlan.config('mac') # get the interface's MAC address
wlan.ifconfig() # get the interface's IP/netmask/gw/DNS addresses
ap = network.WLAN(network.AP_IF) # create access-point interface
ap.active(True) # activate the interface
ap.config(essid='ESP-AP') # set the ESSID of the access point
print('Wifi connected! My IP:', wlan.ifconfig()[0])
import urequests
import time
while 1:
try:
t1=time.time()
r=urequests.get('https://saitamatechnoo.web.app/')
t2=time.time()
print(r.status_code, 'Time:', t2-t1)
except OSError:
print('error')
Guys, I solved it. It works with socket and we should close every socket after using it.
For example:
r=urequests.get('https://saitamatechnoo.web.app/')
print(r.status_code)
r.close()

Pydev showing 'undefined variable' for Jython code analysis

I have used Eclipse, Jython & Pydev for a long time. Upgrading as and when new releases arrived. All has worked very well until recently when the Pydev code completion started marking common items such as dir or print as 'undefined variable'. But the program ran correctly.
The error log showed:
'The python client still hasn't connected back to the eclipse java vm (will retry..)'
'Attempt: 2 of 5 failed, trying again...(socket connected still null)'
and more attempts to 5 out of 5
'Error connecting to python process(most likely cause for failure is firewall blocking...misconfigured network)'
Also, attempting to create a live jython console CTRL+ALT+ENTER gives the following error:
'Create interactive Console' has encountered a problem
Error initializing console.
Unexpected error connecting to console.
Failed to receive suitable Hello response from pydevconsole. Last msg received: Console already exited with value: 1 while waiting for an answer.
I have spent a lot of time looking for answers here and elsewhere that have included suggestions to check:
Mixed 32/64 bit installs; Firewall problems; IPV4 preference; localhost entries; path issues and others: all I've checked out with no success so far.
Software is Windows 10, Eclipse 4.21.0, Pydev 9.1.0.2021, Java JDK 11.0.13, Jython 2.7.2
I should be most grateful for any further help on this problem.
Many thanks
Well, you get that message because PyDev does spawn a shell and communicate with it for collecting code-completion results and the same happens for the interactive console.
Now, in both cases it seems like the socket communication is being prevented in your use case (as the messages states, the usual culprit is some firewall -- or possibly an anti-virus -- or some network misconfiguration).
Unfortunately, it's really hard for me to diagnose this as it's pretty much machine-dependent (all I can say is that I checked things here and Jython is working well, so, the issue lies in some misconfiguration on your machine).
I actually have plans to stop requiring that socket communication (for getting the completions from a shell) and using stdin/stdout (https://www.brainwy.com/tracker/PyDev/1183), but this still isn't done.
As Jython does run for you, you can try to create a simple server to verify a connection works.
i.e.: Create an echo_server.py with:
import socket
if __name__ == '__main__':
HOST = '127.0.0.1'
PORT = 65432
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print('Connected by', addr)
data = conn.recv(1024)
conn.sendall(data)
print('Echo server finished')
and a client.py with:
import socket
if __name__ == '__main__':
HOST = '127.0.0.1'
PORT = 65432
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.sendall(b'Hello, world')
data = s.recv(1024)
print('Received %r' % (data,))
Then run the server part and then the client part and check if that does work properly for you.

Download Sharepoint Online file using Python

I am trying to automate some work processes by using Python to download an Excel file from our SharePoint site. Some background is that I'm fairly new to Python and am mostly self-taught. I have read over many examples here using sharepy, office365, requests, etc. and none seem to work. Below is what I have tried (basically copied from another example) with the error below that:
from office365.runtime.auth.authentication_context import AuthenticationContext
from office365.sharepoint.client_context import ClientContext
from office365.sharepoint.file import File
#Inputs
url_shrpt = "https://company.sharepoint.com/sites/teamsite"
username_shrpt = "user#company.com"
password_shrpt = "password"
folder_url_shrpt = "sites/teamsite/library"
#Authentication
ctx_auth = AuthenticationContext(url_shrpt)
if ctx_auth.acquire_token_for_user(username_shrpt, password_shrpt):
ctx = ClientContext(url_shrpt, ctx_auth)
web = ctx.web
ctx.load(web)
ctx.execute_query()
print("Authenticated into sharepoint as: ",web.properties['Title'])
else:
print(ctx_auth.get_last_error())
#Function for extracting the file names from a folder
global print_folder_contents
def print_folder_contents(ctx, folder_url):
try:
folder = ctx.web.get_folder_by_server_relative_url(folder_url)
fold_names = []
sub_folders = folder.files
ctx.load(sub_folders)
ctx.execute_query()
for s_folder in sub_folders:
fold_names.append(s_folder.properties["Name"])
return fold_names
except Exception as e:
print('Problem printing out library contents: ', e)
######################################################
# Call the function by giving your folder URL as input
filelist_shrpt=print_folder_contents(ctx,folder_url_shrpt)
#Print the list of files present in the folder
print(filelist_shrpt)
Here is the error I receive:
Error: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /extSTS.srf (Caused by NewConnectionError(': Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))
After collaborating with some more knowledgeable coworkers, I was told it might be a permissions issue; but I'm not sure how to verify that. Any help and/or advice would be appreciated.

Incorrect SO_REUSEADDR behavior on Linux

There are two uses for SO_REUSEADDR:
binding two servers on the same address (for server performance)
binding a client then a server (for example for hole punching)
It seems that the second one doesn't work on linux (I tested on RedHat and Chromium OS) although it works on macOS.
I made this little code:
import socket
conn = socket.create_connection(("google.fr", 80))
if len(conn.getsockname()) == 2:
family = socket.AF_INET
else:
family = socket.AF_INET6
s = socket.socket(family)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
s.bind(conn.getsockname())
This code works on macOS but fails with OSError: [Errno 98] Address already in use otherwise.
Is there any way to make it work? If not, where does this behavior come from?

Trouble authenticating Tor with python

Probably doing something very silly here, but I'm having some trouble authenticating automatically through Tor.
I'm using 32 bit ubuntu 12.04 with obfuscated bridges.
This should be all the relevant code, but let me know if there's something else that would be useful in debugging this issue:
import socket
import socks
import httplib
def connectTor():
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050, True)
#9050 is the Tor proxy port
socket.socket = socks.socksocket
def newIdentity():
socks.setdefaultproxy() #Disconnect from Tor network
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", 46594))
s.send("AUTHENTICATE\r\n")
response = s.recv(128)
#128 bytes of data for now, just to see how Tor responds
print response
if response.startswith("250"): #250 is the code for a positive response from Tor
s.send("SIGNAL NEWNYM\r\n") #Use a new identity
s.close()
connectTor() #Just to make sure we're still connected to Tor
Whenever I run this I get the following error:
515 Authentication failed: Password did not match HashedControlPassword value from configuration. Maybe you tried a plain text password
I tried using the --hash-password option and pasting that in the place of the AUTHENTICATE string, but that just caused the script to hang. Thoughts?
That error means that you set the HashedControlPassword option in your torrc. I would suggest option for CookieAuthentication 1 instead then using a controller library rather than doing this from scratch.
What you're trying to do here (issue a NEWNYM) is a very, very common request (1, 2) so I just added a FAQ entry for it. Here's an example using stem...
from stem import Signal
from stem.control import Controller
with Controller.from_port(port = 9051) as controller:
controller.authenticate()
controller.signal(Signal.NEWNYM)