I use EWS GetItem endpoint in order to get MIME content of e-mail message in Outlook. As a rule several 'Received' header fields are present at the beginning of message content. For example:
Received: from AM6PR04CA0035.eurprd04.prod.outlook.com
(2603:10a6:20b:92::48) by AM6PR10MB2789.EURPRD10.PROD.OUTLOOK.COM
(2603:10a6:20b:ac::25) with Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.15; Thu,
10 Sep 2020 11:36:56 +0000 Received-SPF: Pass
(protection.outlook.com: domain of alterdomus.com designates
51.137.107.73 as permitted sender) receiver=protection.outlook.com; client-ip=51.137.107.73;
helo=smtpworker-in-9.xware-europe-west-1.o365.crossware.co.nz;
Received: from VI1PR10MB3616.EURPRD10.PROD.OUTLOOK.COM
([fe80::204a:2512:2ac2:d5ef]) by VI1PR10MB3616.EURPRD10.PROD.OUTLOOK.COM
([fe80::204a:2512:2ac2:d5ef%3]) with mapi id 15.20.3305.031; Tue, 25 Aug 2020
17:51:53 +0000
Received: from EUR02-AM5-obe.outbound.protection.outlook.com
(104.47.4.57) by
smtpworker-in-9.xware-europe-west-1.o365.crossware.co.nz with
Crossware for Office365; Thu, 10 Sep 2020 11:36:46 +0000
However in some cases those 'Received' fields are missed for some messages. As far as I understand this field is very important for spam detection. It contains information about the sender, the receiver and the received time of a message. This header is added by the receiving mail server as the top line. Depending on the number of mail servers involved, several 'Received' header fields may be included.
What can be the reason of missed 'Received' header fields inside e-mail MIME content? What does it mean?
Not all messages that are in a Office365 mailbox will have been routed via the Transport stack (most will). One example is the MyAnalytics Messages, the Office365 substrate generates these messages and they are created directly in the Mailbox by the substrate rather then being delivered. You can use something like the Message Header analyzer to check https://appsource.microsoft.com/en-us/product/office/WA104005406?src=office&corrid=f7f9ed61-d151-4fcf-b0e2-9e3dec4a92d2&omexanonuid=034f54c1-a89b-429c-adc3-526f6f024f80&referralurl=https%3a%2f%2fwww.google.com%2f to check the headers it grabs the PidtagTransportheaders property https://learn.microsoft.com/en-us/office/client-developer/outlook/mapi/pidtagtransportmessageheaders-canonical-property which is a more reliable way of getting the headers then trying to parse it back out of the Mime Message
Related
There is a problem accessing the To alias from a o365 account IF the from account is also o365. If the from account is say, gmail, it works.
If I send an email to alias#mycompany.com which is an alias to realAccount#mycompany.com, if I examine the To header in Outlook, it will always show me the original alias. If I view the header progrmatically, it will NOT show the alias if it was sent from an o365 account. Instead, it shows the real account. If I do this same test with a gmail instead of an o365 email it works -- shows the alias in the To: header as expected.
How does Outlook access this data? The number of headers are different too. Outlook contains more data. Has anyone experienced this? Any ideas on how to access the alias like Outlook does?
Header when accessing from Outlook:
From: o365Account#somecompany.com
To: ***************** alias#mycompany.com ****************
Subject: shdaKJSDHA
Thread-Topic: shdaKJSDHA
Thread-Index: AQHUSTkz1fQhzI5SG0ie26mNIvHmmQ==
Date: Mon, 10 Sep 2018 19:05:12 +0000
Message-ID: <---#-----.prod.outlook.com>
Header when accessed programatically:
From: o365Account#company.com
To: *****************realAccount#mycompany.com ****************
Subject: shdaKJSDHA
Thread-Topic: shdaKJSDHA
Thread-Index: AQHUSTkz1fQhzI5SG0ie26mNIvHmmQ==
X-MS-Exchange-MessageSentRepresentingType: 1
Date: Mon, 10 Sep 2018 19:05:12 +0000
Message-ID: <----#-----.prod.outlook.com>
Keep in mind that Exchange always resolves the sender and recipients to their primary addresses both when sending and when receiving the messages. This is just the way it works.
Are you sending through SMTP?
You know what I ended up doing? I made a Distro list with 1 recipient. The distro list takes the place of the alias. It always shows the To: as the distro list. That way it doesn't get lost. Thank you for helping me understand Exchange Server dmitry-streblechenko.
as I read on the official rfc2822
The "Message-ID:" field provides a unique message identifier that
refers to a particular version of a particular message. The
uniqueness of the message identifier is guaranteed by the host that
generates it (see below). This message identifier is intended to be
machine readable and not necessarily meaningful to humans. A message
identifier pertains to exactly one instantiation of a particular
message; subsequent revisions to the message each receive new message
identifiers.
Note: There are many instances when messages are "changed", but those
changes do not constitute a new instantiation of that message, and
therefore the message would not get a new message identifier. For
example, when messages are introduced into the transport system, they
are often prepended with additional header fields such as trace
fields (described in section 3.6.7) and resent fields (described in
section 3.6.6). The addition of such header fields does not change
the identity of the message and therefore the original "Message-ID:"
field is retained. In all cases, it is the meaning that the sender
of the message wishes to convey (i.e., whether this is the same
message or a different message) that determines whether or not the
"Message-ID:" field changes, not any particular syntactic difference
that appears (or does not appear) in the message.
An email message can hold the same ID in some particular cases.
For example, take a look at the two following message header:
Return-Path: <xxx.xxx#xxx.it>
Received: from relaypsm.eng.it (xxxx.xxx.it [xxx.xxx.xxx.xxx])
by xxxxx-xxx.xxxxx.it (8.14.4/8.14.4) with ESMTP id v0PJ6wOD014924
(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL)
for <xxxxx#xxx.xxxxx.it>; Wed, 25 Jan 2017 20:06:59 +0100
X-Icontrol: Sent by Inrete Icontrol
Received: from MailAV (unknown [xxx.xxx.xxx.xxx])
by deliver (Postfix) with ESMTP id 52B743A4
for <xxxxx#xxx.xxxxx.it>; Wed, 25 Jan 2017 20:06:58 +0100 (CET)
Received: from LBURRINIW (unknown [192.168.63.9])
by xxxxx.xxx.it (Postfix) with ESMTPSA id 8D80339F
for <xxxxx#xxx.xxxxx.it>; Wed, 25 Jan 2017 20:06:51 +0100 (CET)
From: "Luca Burrini" <luca.burrini#eng.it>
To: <xxxxx#xxx.xxxxx.it>
References:
In-Reply-To:
Subject: fatture xxxxx
Date: Wed, 25 Jan 2017 20:06:51 +0100
Message-ID: <!&!AAAAAAAAAAAYAAAAAAAAAHCgtCXXdtBNl+uXnP8eDHPCgAAAEAAAAGutlHZJGkpAhmPV6f+ofh8BAAAAAA==#eng.it>
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="----=_NextPart_000_00D6_01D27746.91533090"
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AdCOWOCYZlK03GM4S5Cd5xo9GuRjGAAorxQwBlBUrvAFQ/QDgAew8EzACjCMyCAGU/qYMAV4vr+QBuilgWAKzjrI4AagZVDwBqrsgiAABfH4YAXsTykQBafYYmAEIBNvwBSv7j/wBgnvoHAB0wKvkAWE5BHA
Content-Language: it
X-KLMS-Rule-ID: 1
X-KLMS-Message-Action: clean
X-KLMS-AntiSpam-Status: not scanned, disabled by settings
X-KLMS-AntiPhishing: not scanned, disabled by settings
X-KLMS-AntiVirus: Kaspersky Security 8.0 for Linux Mail Server, version 8.0.1.721, bases: 2017/01/25 14:07:00 #8842754
X-KLMS-AntiVirus-Status: Clean, skipped
X-IControl-Milter-Checked: yes (IControlServer: xxxxx-NEW SrcIpType: UnTrusted SrcIpHeaderType: Undefined Authenticated: no)
X-IControl-Milter-SPF-Checked: yes (IControlServer: CEDACRIPEC1-NEW HeloSPFType: none FromSPFType: pass HeloHeaderSPFType: Undefined FromHeaderSPFType: pass)
X-IControl-Milter-MD5SUM: c803ab2c6498f91967e8cb2f5954f43a
and
Return-Path: <xxx.xxx#xxx.it>
Received: from xxxxx.xxx.it (xxxxx.xxx.it [xxx.xxx.xxx.xxx])
by xxxxx-new.xxxxx.it (8.14.4/8.14.4) with ESMTP id uBSGqOAP023446
(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL)
for <xxxxx#xxx.xxxxx.it>; Wed, 28 Dec 2016 17:52:24 +0100
X-Icontrol: Sent by Inrete Icontrol
Received: from MailAV (unknown [161.27.15.23])
by deliver (Postfix) with ESMTP id A0FD33AC
for <xxxxx#xxx.xxxxx.it>; Wed, 28 Dec 2016 17:52:23 +0100 (CET)
Received: from LBURRINIW (unknown [xxx.xxx.xxx.xxx])
by relaypsm.eng.it (Postfix) with ESMTPSA id 7D2D83CC
for <xxxxx#xxx.xxxxx.it>; Wed, 28 Dec 2016 17:52:17 +0100 (CET)
From: "Luca Burrini" <xxx.xxxxx#xxx.it>
To: <xxxxx#xxx.xxxxx.it>
References:
In-Reply-To:
Subject: fatture xxx
Date: Wed, 28 Dec 2016 17:52:16 +0100
Message-ID: <!&!AAAAAAAAAAAYAAAAAAAAAHCgtCXXdtBNl+uXnP8eDHPCgAAAEAAAAGutlHZJGkpAhmPV6f+ofh8BAAAAAA==#eng.it>
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="----=_NextPart_000_0092_01D26133.2135B540"
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AdCOWOCYZlK03GM4S5Cd5xo9GuRjGAAorxQwBlBUrvAFQ/QDgAew8EzACjCMyCAGU/qYMAV4vr+QBuilgWAKzjrI4AagZVDwBqrsgiAABfH4YAXsTykQBafYYmAEIBNvwBSv7j/wBgnvoHAB0wKvkA==
Content-Language: it
X-KLMS-Rule-ID: 1
X-KLMS-Message-Action: clean
X-KLMS-AntiSpam-Status: not scanned, disabled by settings
X-KLMS-AntiPhishing: not scanned, disabled by settings
X-KLMS-AntiVirus: Kaspersky Security 8.0 for Linux Mail Server, version 8.0.1.721, bases: 2016/12/28 09:52:00 #8723401
X-KLMS-AntiVirus-Status: Clean, skipped
X-IControl-Milter-Checked: yes (IControlServer: CEDACRIPEC1-NEW SrcIpType: UnTrusted SrcIpHeaderType: Undefined Authenticated: no)
X-IControl-Milter-SPF-Checked: yes (IControlServer: CEDACRIPEC1-NEW HeloSPFType: none FromSPFType: pass HeloHeaderSPFType: Undefined FromHeaderSPFType: pass)
X-IControl-Milter-MD5SUM: 30378f3a90c4e4051ecd9eab79feaa02
They hold the same ID but the two emails, trust me, belong to different transmissions. In addition they hold different attachments.
Can you tell me why the email IDs are the same?
You can get different messages with the same message-Id. Here's how:
Message-Id 1 is generated when user A sends message1 to B and C.
B is a final destination, which accepts the message and stores it.
C is a mailing list. The mail message is held for moderation. At some later point in time, the mail list moderator accepts the message. The message is modified with the header and footer of the mailing list and sent out. B is on the mailing list and receives a copy.
Now B has two copies of message1, one without the mailing list info, one with it.
I'll take a shot at this.
RFC 822 specifies that:
"The uniqueness of the message identifier is guaranteed by the host which generates it."
It also specifies under section 4.2:
Some systems permit mail recipients to forward a message,
retaining the original headers, by adding some new fields. This
standard supports such a service, through the "Resent-" prefix to
field names.
So:
It sounds like the host email program is not changing the said ID.
message-id is optional, but ESMTP indicates that servers should add
it if missing during a transaction. (https://www.rfc-editor.org/rfc/rfc2821)
forwarding messages SHOULD add "resent-", but that's up to the code within the client/server.
The ESMTP ID change seems to indicate different email servers... and the subject's typo feels contrived, even spam-like. (isn't "fatture Engineering" a mixture of Italian and English for "Engineering Invoice"?)
If it's spam, I'd expect message IDs to be similar even if they're coming from the same server since they are, in fact, the same message.
Do you have more information about the purpose/content of this email?
Here is an example of how I have configured sieve to forward any mail sent to [nameA|nameB|nameC]#example.org to my private email address.
if address :localpart :is ["To","Cc","Bcc"]
["nameA", "nameB", "nameC"] {
redirect "<my private email address>";
stop;
}
Sometimes though, email is not forwarded because the address that it was sent to is tucked away somewhere in a "Received" header.
Received: from ###server### ([###ip_address###])
by ###server### with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256)
(Exim 4.84)
(envelope-from <###email_address###>)
id 1alDM0-0000yT-60
for nameA#example.org; Wed, 30 Mar 2016 12:28:00 +0200
Is there an effective way to catch these emails in the sieve rule, too?
You have an XY Problem here. What you actually want to do here is filter based on the address being delivered to, not the address in the headers. (As unintuitive as it may be, the address in the headers may have nothing to do with the address it's being delivered to, which is how Bcc can work at all.)
The command to test against the actual SMTP envelope is envelope.
require "envelope";
if envelope :localpart :is "to" ["nameA", "nameB", "nameC"] {
redirect "<my private email address>";
stop;
}
This will handle all mail being delivered to those names, regardless of whether they show up in the mail headers or not.
With the help of the sieves' index feature you can parse the recipient address out of the Received headers.
For BCC sorting I typically do something like this:
require ["fileinto", "envelope", "variables", "mailbox", "index", "subaddress"];
...
if header :index 3 :matches "Received" "*<*#example.com>*" {
set :lower "foldername" "${2}";
fileinto :create "inbox.${foldername}";
} elsif header :index 2 :matches "Received" "*<*#example.com>*" {
set :lower "foldername" "${2}";
fileinto :create "inbox.${foldername}";
}
...
In the Received headers of the mails I receive the adress is set in angle brackets and that's why I've chosen the pattern in the example above.
Additionally, sometimes the number of Received headers varies thus I test at least for two different ones.
Currently I am filtering incoming mails by a .forward in the following way:
if $header_to: matches "(office|info)#domain.com" then
save Maildir/.office/
endif
if $header_to: matches "whatever#domain.com" then
save Maildir/.whatever/
endif
So I have a mail account, which receives mails for different addresses. Basically I want them to land in different subdirs based on the address the mail was sent to.
This works for mails where the recipient is in the to-header, but does not work if the recipient was in the bcc.
When a mail is received which was sent with the bcc-header, only the envelope-to-header matches the real address the mail is delivered to and it is mentioned in a Received-header
Envelope-to: office#domain.com
Received: from mail.other.domain ([1.1.1.1])
by mail.domain.com with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
(Exim 4.71)
(envelope-from <sender#other.domain>)
id 1RO5xc-0001TF-Qj
for office#domain.com; Wed, 09 Nov 2011 12:04:57 +0100
...
To: can_be_anything#whatever.com
I already tried:
if $header_envelope-to: matches ...
but this does not work, mails are not filtered at all even when sent with To-header (looks like the Envelope-To-header is not available in forward-files). Should I try to parse the (multiple) Received-headers?
How can I move mails into a subdir of the recipient based on the real recipient address?
looks like I finally found the answer.
if $original_local_part matches "office|info" then
save Maildir/.office/
endif
This checks only the local_part, but afaik could be extended to use the domain, too, with $original_domain (see the doc)
The variable $recipients contains all (to, cc and bcc) recipients. Have you tried it?
I'm trying to sort out an opt in mailing list system. I understand the basic principles and design required but i'm having an issue with it being picked up as spam.
If i send a html email through outlook through email#domain.com it works fine and is not treated as spam. When i use the Zend_mail object to send mail it sends but is treated as spam on the test emails accounts i'm sending it too.
This is the code im using to send an email item.
//send an email
$mail = new Zend_Mail();
$config = array('auth' => 'login','username' => 'email#domain.com','password' => 'mypassword');
$transport = new Zend_Mail_Transport_Smtp('mail.domain.com', $config);
$mail->setSubject($item->title);
$mail->setFrom("email#domain.com");
$mail->addTo($item->email, $item->forename);
//$mail->setBodyText($item->contentPlain);
$mail->setBodyHtml($item->contentHTML);
$mail->send($transport);
As you can see im using the smtp transport object to authenticate but it still seems to treat this as spam. Anyone with pointers or tips is very much appreciated!!
Header info from the email that is treated as spam:
It seems to contain a couple of client domain names in the header info that i host for people any ideas why that would be the case? I use a shared IP address with about 10 domains on it
Received: (qmail 1436 invoked from network); 14 Aug 2009 16:02:10 +0100
Received: from clientdomain1.co.uk (HELO localhost) (91.192.***.196)
by clientdomain2.info with SMTP; 14 Aug 2009 16:02:10 +0100
Subject: Manchester 2 Day Seminar: Dealing with difficult people
From: events#domain.com
To: Andi <subscriber1#domain.com>
Date: Fri, 14 Aug 2009 15:02:10 +0000
Content-Type: text/html; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
MIME-Version: 1.0
"Roll your own mail" is often treated as spam by large hosted email systems. When you use a paid service to send out mass emails, you are paying for those company's agreements with the major email vendors to keep them white-listed.
One thing you can do, though, is to ensure that the account you are sending from exists and the email is being sent from a matching domain (e.g. #foo.com sent from foo.com's smtp server). That is a big red flag for spam filters.
Compare your email and email from outlook. Are any headers missing? Which? Do they seem significant?
Try this to get rid of the last localhost reference:
$protocol = new Zend_Mail_Protocol_Smtp('localhost');
$protocol->connect();
$protocol->helo('mail.yourserver.com'); //**DO THIS**
$transport->setConnection($protocol);