Ingest email attachments on ElasticSearch - email

Question

I'm trying to use ELK pipeline to read an email (IMAP), extract generic attachments (mainly PDF, eventually doc or ppt) and put them on ElasticSearch.
This is what I was able to do:
Loading directly to ElasticSearch from file some base64 data using Logstash, using the Ingest Attachment Processor on ElasticSearch to read the base64 content.
Loading data from IMAP (exchange email) I can correctly load all email information on ElasticSearch except the attachment (what I need).
The first solution works fine and does what I am looking for, except that it doesn't extract attachments directly from the email and that I have hardcoded base64 data inside the files.
With the second solution I have a field x-ms-has-attach: yes on Kibana, but there isn't anywhere the attachment itself. The imap plugin is intended to load only the content of the email without the attachment?
What am I missing? Could you suggest me a pipeline to achieve what I am looking for?
This is my logstash configuration for the first example:
input {
file {
path => "/my/path/to/data/*"
start_position => "beginning"
# sincedb_path => "/my/path/to/sincedb"
sincedb_path => "/dev/null"
close_older => 0
tags => ["attachment"]
}
}
output {
elasticsearch {
index => "email-attachment"
hosts => [ "localhost:9200" ]
}
}
This is the pipeline:
PUT _ingest/pipeline/email-attachment
{
"description": "Pipeline to parse an email and its attachments",
"processors": [
{
"attachment" : {
"field" : "message"
}
},
{
"remove" : {
"field" : "message"
}
},
{
"date_index_name" : {
"field" : "#timestamp",
"index_name_prefix" : "email-attachment-",
"index_name_format": "yyyy-MM",
"date_rounding" : "M"
}
}
]
}
This is my logstash configuration for the second example:
input {
imap {
host => "my.domain.it"
password => "mypassword"
user => "myuser"
port => 12345
type => "imap"
secure => true
strip_attachment => true
}
}
output {
elasticsearch {
index => "email-attachment"
hosts => [ "localhost:9200" ]
}
}
UPDATE
I'm using version 5.2.2

Answer 1

In the end I defined a totally different pipeline.
I read emails using a Ruby application with the mail library (you can find it on github), where it's quite easy to extract attachments.
Then I put the base64 encoding of those attachments directly on ElasticSearch, using Ingest Attachment Processor.
I filter on content_type just to be sure to load only "real" attachments, as the multiparts emails treat any multimedial content in the body (ie: images) as attachment.
P.S.
Using the mail library, you should do something like:
Mail.defaults do
retriever_method :imap, { :address => address,
:port => port,
:user_name => user_name,
:password => password,
:enable_ssl => enable_ssl,
:openssl_verify_mode => openssl_verify_mode }
and new_messages = Mail.find(keys: ['NOT','SEEN']) to retrieve unseen messages.
Then iterate over new_messages. After, you can encode a message simply using encoded = Base64.strict_encode64(attachment.body.to_s). Please inspect new_messages to check the exact field names to use.

Answer 2

Your problem might come from strip_attachment => true in the imap input plugin.