Read message body of an email using Apache Nifi - email

Question

Is it possible to retrieve the body content of email, email header details and email attachments in Single step using Apache Nifi.
If so Please help me how to achieve this.

Answer 1

It is not possible in a single step unless you write your own processor or script (using ExecuteScript or InvokeScriptedProcessor). However it is possible in a single flow with something like the following:
ConsumePOP3 -> ExtractEmailHeaders -> ExtractEmailAttachments -> ...
At the end of the flow above, you will have one flow file per attachment, each flow file containing the email headers as attributes and the attachment as the content.

Answer 2

You can use the processor "ExecuteScript", not developing custom processor.
import email
import mimetypes
from email.parser import Parser
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from java.io import BufferedReader, InputStreamReader
from org.apache.nifi.processors.script import ExecuteScript
from org.apache.nifi.processor.io import InputStreamCallback
from org.apache.nifi.processor.io import StreamCallback
class PyInputStreamCallback(InputStreamCallback):
_text = None
def __init__(self):
pass
def getText(self) :
return self._text
def process(self, inputStream):
self._text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
flowFile = session.get()
if flowFile is not None :
reader = PyInputStreamCallback()
session.read(flowFile, reader)
msg = email.message_from_string(reader.getText())
body = ""
if msg.is_multipart():
for part in msg.walk():
ctype = part.get_content_type()
cdispo = str(part.get('Content-Disposition'))
if ctype == 'text/plain' and 'attachment' not in cdispo:
body = part.get_payload(decode=True) # decode
break
else:
body = msg.get_payload(decode=True)
flowFile = session.putAttribute(flowFile, 'msgbody', body.decode('utf-8', 'ignore'))
session.transfer(flowFile, ExecuteScript.REL_SUCCESS)
Screenshot

Answer 3

The gist above posted by #Thomas mostly worked for me.
However, there were cases where the multipart mail was multi-level, i.e. one of the parts was also multipart. To address that, you could use this version of the getTextFromMessage method instead:
private String getTextFromMessage(Part part) throws MessagingException, IOException {
String result = null;
if (part.isMimeType("text/plain")){
// If the part is a plaintext message, just return the content as a String
Object content = part.getContent();
if (content instanceof String) {
result = (String)content;
}
} else if (part.isMimeType("multipart/*")) {
// If the part is a multi-part message, iterate over the sub-parts
MimeMultipart mimeMultipart = (MimeMultipart)part.getContent();
int count = mimeMultipart.getCount();
for (int i = 0; i < count; i ++){
final BodyPart bodyPart = mimeMultipart.getBodyPart(i);
// Inserted from here...
final String text = getTextFromMessage( bodyPart );
if (text != null) {
result = text;
break;
}
}
}
// Just to be safe...
result = (result != null) ? result : "";
return result;
}
Note that it is possible that there is not even a "html/plain" part at all in the email, so you may want to think about how to handle that case as well. For instance, you could loop over the parts again to extract the "text/html" part (if it exists) and deal with that, and so on.