Extracting a body of an email using [Python 3.6] - email

I have an inbox and I need to write a code that triggers every time a new message arrives, and extracts a body of that email.
I have this code so far:
import poplib, email
from email import parser
pop_conn = poplib.POP3_SSL('pop.gmail.com')
email_user = 'email#gmail.com'
email_pass = 'password'
pop_conn.user(email_user)
pop_conn.pass_(email_pass)
#Get messages from server:
messages = [pop_conn.retr(i) for i in range(1,
len(pop_conn.list()[1]) + 1)]
#Parse message intom an email object:
messages = [parser.Parser().parsestr(mssg) for mssg in messages]
for message in messages:
print(message.get_payload(None, True))
pop_conn.quit()
The error I am getting is:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-6-2a08d6a8ccf9> in <module>()
2 #Parse message intom an email object:
3 #messages = [email.message_from_bytes(mssg) for mssg in messages]
----> 4 messages = [parser.Parser().parsestr(mssg) for mssg in messages]
5 for message in messages:
6 # print(message['body'])
<ipython-input-6-2a08d6a8ccf9> in <listcomp>(.0)
2 #Parse message intom an email object:
3 #messages = [email.message_from_bytes(mssg) for mssg in messages]
----> 4 messages = [parser.Parser().parsestr(mssg) for mssg in messages]
5 for message in messages:
6 # print(message['body'])
/Applications/anaconda3/lib/python3.6/email/parser.py in parsestr(self, text, headersonly)
66 the file.
67 """
---> 68 return self.parse(StringIO(text), headersonly=headersonly)
69
70
TypeError: initial_value must be str or None, not tuple
TypeError: initial_value must be str or None, not tuple
Why am I getting a tuple? How do I extract a body from the messages[n]?

The return value from retr() is a tuple. The second value in the tuple is a list of lines which comprise the actual message. See the Python poplib documentation for details.
# Get messages from server
popped = ['\n'.join(pop_conn.retr(i)[1])
for i in range(1, len(pop_conn.list()[1]) + 1)]
# Parse message into an email object
messages = [parser.Parser().parsestr(mssg) for mssg in popped]
Notice also how we avoid reusing the same variable name for objects of different types.

Related

python - extract numbers from a string coming from serial port

i receive this type of messages in my Raspberry Pi from Serial Port ( in this case usb1) , and i want to execute some instrutions in this case os.system(instrutions) when i receive some numeric values but from
some strings and not from other strings.
whe start with Light: ..... i want to receive value
example message receive in my serial interface:
"Light: 16 lx , quality: low" -> i just want number 16
http : 200 ok -> i don need this number 200
I have already one python cript in place but give error many times:
error messages
Light: 13 lx , quality: toolow
Traceback (most recent call last):
File "receive_serial_data_from_usb1.py", line 11, in
if number [0] > 400 :
IndexError: list index out of range
//python script:
#!/usr/bin/env python3
import serial
import os
ser = serial.Serial('/dev/ttyUSB1', 115200, timeout=1)
ser.flush()
while True:
if ser.in_waiting > 0:
line = ser.readline().decode('utf-8').rstrip()
number = [int (s) for s in line.split () if s.isdigit ()]
if number [0] > 400 :
print ('HIGH ATTENTION!!!')
os.system('sudo python control.py -t DA:74:56:99:71:DE -c move_down')
if number [0] < 10 :
print ('LOW ATTENTION!!!')
os.system('sudo python control.py -t DA:74:56:99:71:DE -c move_up')
else :
print(line)
Can you help me?

Databricks UDF calling an external web service cannot be serialised (PicklingError)

I am using Databricks and have a column in a dataframe that I need to update for every record with an external web service call. In this case it is using the Azure Machine Learning Service SDK and does a service call. This code works fine when not run as a UDF in spark (ie. just python) however it throws a serialization error when I try to call it as a UDF. The same happens if I use a lambda and a map with an rdd.
The model uses fastText and can be invoked fine from Postman or python via a normal http call or using the WebService SDK from AMLS - it's just when it is a UDF that it fails with this message:
TypeError: can't pickle _thread._local objects
The only workaround I can think of is to loop through each record in the dataframe sequentially and update the record with a call, however this is not very efficient. I don't know if this is a spark error or because the service is loading a fasttext model. When I use the UDF and mock a return value it works though.
Error at bottom...
from azureml.core.webservice import Webservice, AciWebservice
from azureml.core import Workspace
def predictModelValue2(summary, modelName, modelLabel):
raw_data = '[{"label": "' + modelLabel + '", "model": "' + modelName + '", "as_full_account": "' + summary + '"}]'
prediction = service.run(raw_data)
return prediction
from pyspark.sql.types import FloatType
from pyspark.sql.functions import udf
predictModelValueUDF = udf(predictModelValue2)
DVIRCRAMFItemsDFScored1 = DVIRCRAMFItemsDF.withColumn("Result", predictModelValueUDF("Summary", "ModelName", "ModelLabel"))
TypeError: can't pickle _thread._local objects
During handling of the above exception, another exception occurred:
PicklingError Traceback (most recent call
last) in
----> 2 x = df.withColumn("Result", predictModelValueUDF("Summary",
"ModelName", "ModelLabel"))
/databricks/spark/python/pyspark/sql/udf.py in wrapper(*args)
194 #functools.wraps(self.func, assigned=assignments)
195 def wrapper(*args):
--> 196 return self(*args)
197
198 wrapper.name = self._name
/databricks/spark/python/pyspark/sql/udf.py in call(self, *cols)
172
173 def call(self, *cols):
--> 174 judf = self._judf
175 sc = SparkContext._active_spark_context
176 return Column(judf.apply(_to_seq(sc, cols, _to_java_column)))
/databricks/spark/python/pyspark/sql/udf.py in _judf(self)
156 # and should have a minimal performance impact.
157 if self._judf_placeholder is None:
--> 158 self._judf_placeholder = self._create_judf()
159 return self._judf_placeholder
160
/databricks/spark/python/pyspark/sql/udf.py in _create_judf(self)
165 sc = spark.sparkContext
166
--> 167 wrapped_func = _wrap_function(sc, self.func, self.returnType)
168 jdt = spark._jsparkSession.parseDataType(self.returnType.json())
169 judf = sc._jvm.org.apache.spark.sql.execution.python.UserDefinedPythonFunction(
/databricks/spark/python/pyspark/sql/udf.py in _wrap_function(sc,
func, returnType)
33 def _wrap_function(sc, func, returnType):
34 command = (func, returnType)
---> 35 pickled_command, broadcast_vars, env, includes = _prepare_for_python_RDD(sc, command)
36 return sc._jvm.PythonFunction(bytearray(pickled_command), env, includes, sc.pythonExec,
37 sc.pythonVer, broadcast_vars, sc._javaAccumulator)
/databricks/spark/python/pyspark/rdd.py in _prepare_for_python_RDD(sc,
command) 2461 # the serialized command will be compressed by
broadcast 2462 ser = CloudPickleSerializer()
-> 2463 pickled_command = ser.dumps(command) 2464 if len(pickled_command) >
sc._jvm.PythonUtils.getBroadcastThreshold(sc._jsc): # Default 1M
2465 # The broadcast will have same life cycle as created
PythonRDD
/databricks/spark/python/pyspark/serializers.py in dumps(self, obj)
709 msg = "Could not serialize object: %s: %s" % (e.class.name, emsg)
710 cloudpickle.print_exec(sys.stderr)
--> 711 raise pickle.PicklingError(msg)
712
713
PicklingError: Could not serialize object: TypeError: can't pickle
_thread._local objects
I am not expert in DataBricks or Spark, but pickling functions from the local notebook context is always problematic when you are touching complex objects like the service object. In this particular case, I would recommend removing the dependency on the azureML service object and just use requests to call the service.
Pull the key from the service:
# retrieve the API keys. two keys were generated.
key1, key2 = service.get_keys()
scoring_uri = service.scoring_uri
You should be able to use these strings in the UDF directly without pickling issues -- here is an example of how you would call the service with just requests. Below applied to your UDF:
import requests, json
def predictModelValue2(summary, modelName, modelLabel):
input_data = json.dumps({"summary": summary, "modelName":, ....})
headers = {'Content-Type':'application/json', 'Authorization': 'Bearer ' + key1}
# call the service for scoring
resp = requests.post(scoring_uri, input_data, headers=headers)
return resp.text[1]
On a side node, though: your UDF will be called for each row in your data frame and each time it will make a network call -- that will be very slow. I would recommend looking for ways to batch the execution. As you can see from your constructed json service.run will accept an array of items, so you should call it in batches of 100s or so.

How to convert mnist dataset in array

Hello consider following code
# load the mnist training data CSV file into a list
training_data_file = open("Training_Set/mnist_train_100.csv", 'r')
training_data_list = training_data_file.readlines()
training_data_file.close()
for record in training_data_list:
all_values = record.split(',')
x_inputs = (np.asfarray(all_values[1:]) / 255.0 * 0.99) + 0.01
print("xinput=" + str(x_inputs))
print(len(training_data_list))
MyCompleteInput = np.array(x_inputs,len(training_data_list))
I want to put x_inputs and len(training_data_list) into an array so if I print the shape of the array I get an output of (784,100).
But if I run my code I get following error:
TypeError Traceback (most recent call last)
<ipython-input-38-b0f129f57bcb> in <module>()
11 print("xinput=" + str(x_inputs))
12 print(len(training_data_list))
---> 13 MyCompleteInput = np.array(x_inputs,len(training_data_list))
14
15
TypeError: data type not understood
Can somebody help me out? tnx
The line will be
MyCompleteInput = np.array((x_inputs,len(training_data_list)))
Do this and your error will be gone. You need to add another set of parantheses for specifying the size.

'str' does not support the buffer interface - Python 3 [duplicate]

This question already has an answer here:
TypeError: str does not support buffer interface [duplicate]
(1 answer)
Closed 6 years ago.
import socket
serverName = "hostname"
serverPort = 12000
clientSocket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
message = input('input lowercase sentence:')
clientSocket.sendto(message, (serverName, serverPort))
modifiedMessage, serverAddress = clientSocket.recvfrom(2048)
print (modifiedMessage)
clientSocket.close()
gives this error
Traceback (most recent call last):
File "C:/Python34/server1.py", line 11, in <module>
clientSocket.sendto(message, (serverName, serverPort))
TypeError: 'str' does not support the buffer interface
What do I do?
I've encode this code
import socket
serverName = "hostname"
serverPort = 12000
clientSocket = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
message = input('input lowercase sentence:').encode('ascii')
clientSocket.sendto(message,(serverName, serverPort))
modifiedMessage, serverAddress = clientSocket.recvfrom(2048)
print (modifiedMessage.decode('ascii'))
clientSocket.close()
But it still error
Traceback (most recent call last):
File "J:\Sistem Jaringan\Task I\client.py", line 11, in <module>
clientSocket.sendto(message,(serverName, serverPort))
socket.gaierror: [Errno 11004] getaddrinfo failed
How can i fix it?
Sockets sent bytes to the other end of the connection. Encode strings to bytes before trying to send. In the sendto method, change message to message.encode(). Add argument encoding='xyz' if you do not want the default utf-8 encoding. The modifiedMessage you receive will also be bytes, and you may want to decode it to a string.

Need help in Python3

I have the below code that i am trying the run using python3.2 interpreter.
import socket #for sockets
import sys #for exit
#from UserString import MutableString
#create an INET, STREAMing socket
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error:
print ('Failed to create socket')
sys.exit()
print ('Socket Created')
host = 'www.google.com';
port = 80;
try:
remote_ip = socket.gethostbyname( host )
except socket.gaierror:
#could not resolve
print ('Hostname could not be resolved. Exiting')
sys.exit()
#Connect to remote server
s.connect((remote_ip , port))
print ('Socket Connected to ' + host + ' on ip ' + remote_ip)
#Send some data to remote server
message = "GET / HTTP/1.0\r\n\r\n"
try :
#Set the whole string
s.sendall(message.encode())
except socket.error:
#Send failed
print ('Send failed')
sys.exit()
print ('Message send successfully')
#Now receive data
messageParts = []
remaining = 4096
chunk = s.recv(remaining)
messageParts.append(chunk)
while (len(chunk) > 0):
chunk = s.recv(remaining)
messageParts.append(chunk.decode())
finalMessage = b"".join(messageParts)
print('Printing the html contents ...')
print(finalMessage)
Upon running the above code, with python version 3.2, i get the below error:
Socket Created
Socket Connected to www.google.com on ip 74.125.201.147
Message send successfully
Traceback (most recent call last):
File "TestMainServerV2.py", line 73, in <module>
finalMessage = b"".join(messageParts)
TypeError: sequence item 1: expected bytes, str found
Could anybody let me know what is the issue?
Thanks!
messageParts.append(chunk.decode())
is appending strs to messageParts.
chunk = s.recv(remaining)
messageParts.append(chunk)
is appending bytes to messageParts. (Everything that comes through a socket is bytes so chunk is a bytes object.)
Especially in Python3, one should never mix strs and bytes.
b"".join(messageParts)
raises a TypeError since b"".join expects only bytes.
You can avoid the problem by not decoding chunk. Use
messageParts.append(chunk)
instead of
messageParts.append(chunk.decode())