when using Zend_mail my emails seem to be treated as spam, send through outlook and there not? - zend-framework

I'm trying to sort out an opt in mailing list system. I understand the basic principles and design required but i'm having an issue with it being picked up as spam.
If i send a html email through outlook through email#domain.com it works fine and is not treated as spam. When i use the Zend_mail object to send mail it sends but is treated as spam on the test emails accounts i'm sending it too.
This is the code im using to send an email item.
//send an email
$mail = new Zend_Mail();
$config = array('auth' => 'login','username' => 'email#domain.com','password' => 'mypassword');
$transport = new Zend_Mail_Transport_Smtp('mail.domain.com', $config);
$mail->setSubject($item->title);
$mail->setFrom("email#domain.com");
$mail->addTo($item->email, $item->forename);
//$mail->setBodyText($item->contentPlain);
$mail->setBodyHtml($item->contentHTML);
$mail->send($transport);
As you can see im using the smtp transport object to authenticate but it still seems to treat this as spam. Anyone with pointers or tips is very much appreciated!!
Header info from the email that is treated as spam:
It seems to contain a couple of client domain names in the header info that i host for people any ideas why that would be the case? I use a shared IP address with about 10 domains on it
Received: (qmail 1436 invoked from network); 14 Aug 2009 16:02:10 +0100
Received: from clientdomain1.co.uk (HELO localhost) (91.192.***.196)
by clientdomain2.info with SMTP; 14 Aug 2009 16:02:10 +0100
Subject: Manchester 2 Day Seminar: Dealing with difficult people
From: events#domain.com
To: Andi <subscriber1#domain.com>
Date: Fri, 14 Aug 2009 15:02:10 +0000
Content-Type: text/html; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
MIME-Version: 1.0

"Roll your own mail" is often treated as spam by large hosted email systems. When you use a paid service to send out mass emails, you are paying for those company's agreements with the major email vendors to keep them white-listed.
One thing you can do, though, is to ensure that the account you are sending from exists and the email is being sent from a matching domain (e.g. #foo.com sent from foo.com's smtp server). That is a big red flag for spam filters.

Compare your email and email from outlook. Are any headers missing? Which? Do they seem significant?

Try this to get rid of the last localhost reference:
$protocol = new Zend_Mail_Protocol_Smtp('localhost');
$protocol->connect();
$protocol->helo('mail.yourserver.com'); //**DO THIS**
$transport->setConnection($protocol);

Related

Why 'Received' header fields are missed in Outlook message?

I use EWS GetItem endpoint in order to get MIME content of e-mail message in Outlook. As a rule several 'Received' header fields are present at the beginning of message content. For example:
Received: from AM6PR04CA0035.eurprd04.prod.outlook.com
(2603:10a6:20b:92::48) by AM6PR10MB2789.EURPRD10.PROD.OUTLOOK.COM
(2603:10a6:20b:ac::25) with Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.15; Thu,
10 Sep 2020 11:36:56 +0000 Received-SPF: Pass
(protection.outlook.com: domain of alterdomus.com designates
51.137.107.73 as permitted sender) receiver=protection.outlook.com; client-ip=51.137.107.73;
helo=smtpworker-in-9.xware-europe-west-1.o365.crossware.co.nz;
Received: from VI1PR10MB3616.EURPRD10.PROD.OUTLOOK.COM
([fe80::204a:2512:2ac2:d5ef]) by VI1PR10MB3616.EURPRD10.PROD.OUTLOOK.COM
([fe80::204a:2512:2ac2:d5ef%3]) with mapi id 15.20.3305.031; Tue, 25 Aug 2020
17:51:53 +0000
Received: from EUR02-AM5-obe.outbound.protection.outlook.com
(104.47.4.57) by
smtpworker-in-9.xware-europe-west-1.o365.crossware.co.nz with
Crossware for Office365; Thu, 10 Sep 2020 11:36:46 +0000
However in some cases those 'Received' fields are missed for some messages. As far as I understand this field is very important for spam detection. It contains information about the sender, the receiver and the received time of a message. This header is added by the receiving mail server as the top line. Depending on the number of mail servers involved, several 'Received' header fields may be included.
What can be the reason of missed 'Received' header fields inside e-mail MIME content? What does it mean?
Not all messages that are in a Office365 mailbox will have been routed via the Transport stack (most will). One example is the MyAnalytics Messages, the Office365 substrate generates these messages and they are created directly in the Mailbox by the substrate rather then being delivered. You can use something like the Message Header analyzer to check https://appsource.microsoft.com/en-us/product/office/WA104005406?src=office&corrid=f7f9ed61-d151-4fcf-b0e2-9e3dec4a92d2&omexanonuid=034f54c1-a89b-429c-adc3-526f6f024f80&referralurl=https%3a%2f%2fwww.google.com%2f to check the headers it grabs the PidtagTransportheaders property https://learn.microsoft.com/en-us/office/client-developer/outlook/mapi/pidtagtransportmessageheaders-canonical-property which is a more reliable way of getting the headers then trying to parse it back out of the Mime Message

Encrypting Headers S/MIME message/rfc822

I am looking to encrypt certain mail headers (Subject and Reply-To) which are being sent in an encrypted mail.
I am taking an entire MIME (Headers included) and successfully encrypting it.
I can send this S/MIME encrypted mail to my mail client (Thunderbird) successfully. It will be successfully decrypted and verified as signed.
However, any headers that are sent in the inner encrypted MIME are not being used by my mail client.
According to RFC-5751 I should be wrapping my mail in a message/rfc822 message but I am at a loss at how to achieve this.
Below are examples of my messages that I am creating.
My first question is, is the last MIME that I am creating the message/rfc822 correctly structured?
Is this possibly an issue with the mail client?
Can I event encrypt the Reply-To Header?
If I could get an example of a mesage/rfc822 encapsulated message that would be really helpful.
Mail to be encrypted
This will successfully result in a received mail that is signed and the Subject / Reply-To headers are interpreted correctly by the mail client.
Content-Type: multipart/signed; protocol="application/pkcs7-signature";
micalg=sha256; boundary="--_NmP-d017e0e3556f7bbc-Part_1"
From: sender#domain.com
Sender: senderdomain.com
To: recipient#domain.com
Reply-To: keepsecret#domain.com
Subject: A Secret Subject
Message-ID: <400b1383-362b-eed7-0719-6b2a2e231143>
Date: Mon, 24 Feb 2020 15:59:19 +0000
MIME-Version: 1.0
----_NmP-d017e0e3556f7bbc-Part_1
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
My Message that will be encrypted
----_NmP-d017e0e3556f7bbc-Part_1
Content-Type: application/pkcs7-signature; name=smime.p7s
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename=smime.p7s
MIIOCAYJKoZIhvcNAQcCoIIN+TCCDfUCAQExDzANBglghkgBZQMEAgEFADALBgkqhkiG9w0BBwGg
gguTMIIFCDCCA/CgAwIBAgIQVz2HAGYJcTJNsPiWLx1f/TANBgkqhkiG9w0BAQsFADCBjTELMAkG
.
.
.
17p13e02JxfyCqltdb6lkOdpRZ6ZlHHuQZyBCuRtJhRN83gvcJ4d7WCxKI349NEa2/tOb8ziFGat
gzvgu+o=
----_NmP-d017e0e3556f7bbc-Part_1--
My Encrypted Mail
This encrypted mail will be received and successfully decrypted and verified (signature verified) by my mail client. Reply-To and Subject are still working as expected as they are still visible. Note: all the headers from the unencrypted mail are all still present inside the encrypted body of this message.
Sender: sender#domain.com
From: sender#domain.com
To: recipient#domain.com
Subject: A Secret Subject
Reply-To: keepsecret#domain.com
Message-ID: <400b1383-362b-eed7-0719-6b2a2e231143>
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename=smime.p7m
Content-Type: application/pkcs7-mime; smime-type=enveloped-data;
name=smime.p7m
Date: Mon, 24 Feb 2020 16:03:38 +0000
MIME-Version: 1.0
MIIYbwYJKoZIhvcNAQcDoIIYYDCCGFwCAQAxggG/MIIBuwIBADCBojCBjTELMAkG
.
.
.
O+EPVCh1fGDFwiFpDtY/z1Lv8g==
My Encapsulated message/rfc822
This message will be decrypted correctly but my client does not recognise that it was an encrypted message or verify that it was signed (Not worried about that so much). The decrypted mail is interpreted as forwarded and attached as an .eml file. However, no Subject or Reply-To headers found (they are in the encrypted mail). If I add dummy values as recommended by the RFC, those dummy values will be used by my mail client, not the encrypted ones.
Content-Type: message/rfc822; forwarded=false; boundary="--_NmP-07c15c542cedfe74-Part_1"
From: sender#domain.com
Sender: sender#domain.com
To: recipient#domain.com
Date: Mon, 24 Feb 2020 15:28:07 +0000
Message-ID: <400b1383-362b-eed7-0719-6b2a2e231143>
MIME-Version: 1.0
----_NmP-07c15c542cedfe74-Part_1
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename=smime.p7m
Content-Type: application/pkcs7-mime; smime-type=enveloped-data;
name=smime.p7m
MIIYbwYJKoZIhvcNAQcDoIIYYDCCGFwCAQAxggG/MIIBuwIBADCBojCBjTELMAkG
.
.
.
fYU1LuhSBEyymSVRzwWr2T3lrhUe5BZBoY996epZtOPdIYrz2jqUglii1+AUBpUP
UUnpr8+cHTMk/50LHdy3MqMeYA==
----_NmP-07c15c542cedfe74-Part_1
Edit: add excerpt from RFC
In RFC-8551 it states the following
In order to protect outer, non-content-related message header fields (for instance, the "Subject", "To", "From", and "Cc" fields), the
sending client MAY wrap a full MIME message in a message/rfc822
wrapper in order to apply S/MIME security services to these header
fields. It is up to the receiving client to decide how to present
this "inner" header along with the unprotected "outer" header. Given
the security difference between headers, it is RECOMMENDED that the
receiving client provide a distinction between header fields,
depending on where they are located.
When an S/MIME message is received, if the top-level protected MIME
entity has a Content-Type of message/rfc822, it can be assumed that
the intent was to provide header protection. This entity SHOULD be
presented as the top-level message, taking into account
header-merging issues as previously discussed.
RFC 822 provides a generalized description of how message headers of an email are composed and should be treated by systems they are transmitted through. RFC 5751 S/MIME 3.2 (btw, obsoleted by it successor RFC 8551 S/MIME 4.0) describes details how to use that standard to create encrypted emails.
So your approach to encrypt an email as described under My Encrypted Mail is valid and correct.
However, your approach as described under My Encapsulated message/rfc822 is not quite correct. You have obviously misinterpreted the RFC with regard to how to apply the rfc822 wrapper. The wrapper needs to be around your message before it gets encrypted, so it's going to be inside the encrypted part.
In your example, the unencrypted message (a slightly modified version Mail to be encrypted) would have to look like this:
MIME-Version: 1.0
Content-type: message/rfc822
From: sender#domain.com
Sender: senderdomain.com
To: recipient#domain.com
Reply-To: keepsecret#domain.com
Subject: A Secret Subject
Message-ID: <400b1383-362b-eed7-0719-6b2a2e231143>
Date: Mon, 24 Feb 2020 15:59:19 +0000
MIME-Version: 1.0
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha256; boundary="--_NmP-d017e0e3556f7bbc-Part_1"
----_NmP-d017e0e3556f7bbc-Part_1
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
My Message that will be encrypted
[...]
So you basically add the message/rfc822 to the message before it gets encrypted.
I have been able to verify this approach and tested the resulting message in two receiving mail clients with different results. With the macOS Mail application, the encrypted subject was not used to replace the unprotected "outer" subject, but at least, it was displayed prominently below the original headers. This is compliant with the RFC which is not very specific about the presentation:
It is up to the receiving client to decide how to present this "inner" header along with the unprotected "outer" header. Given the security difference between headers, it is RECOMMENDED that the receiving client provide a distinction between header fields, depending on where they are located.
An encrypted Reply-To header is displayed similarly, but it's email address is not honored when replying to that email.
Client Support
The support for encrypted headers in clients is somewhere between weak and non-existent. The results of some tests:
No client supports replacing the "outer" headers by the "inner" encrypted ones
Apple Mail (macOS) displays the inner headers prominently within the message
Thunderbird displays the encrypted part including its headers as a forwarded message
Outlook does not display the encrypted part, but instead confusingly displays just an empty message with an attachment (which is the encrypted message)
Alternative approaches
There is a seemingly promising approach proposed in this draft for Protected Headers for Cryptographic E-mail (work in progress). The idea is to include the protected headers as a separate part in a multipart message. This part will be rendered inline by agnostic clients, while at the same time, it can be properly processed by supporting clients.

o365 email account: Programatically accessing To Alias in Internet Headers

There is a problem accessing the To alias from a o365 account IF the from account is also o365. If the from account is say, gmail, it works.
If I send an email to alias#mycompany.com which is an alias to realAccount#mycompany.com, if I examine the To header in Outlook, it will always show me the original alias. If I view the header progrmatically, it will NOT show the alias if it was sent from an o365 account. Instead, it shows the real account. If I do this same test with a gmail instead of an o365 email it works -- shows the alias in the To: header as expected.
How does Outlook access this data? The number of headers are different too. Outlook contains more data. Has anyone experienced this? Any ideas on how to access the alias like Outlook does?
Header when accessing from Outlook:
From: o365Account#somecompany.com
To: ***************** alias#mycompany.com ****************
Subject: shdaKJSDHA
Thread-Topic: shdaKJSDHA
Thread-Index: AQHUSTkz1fQhzI5SG0ie26mNIvHmmQ==
Date: Mon, 10 Sep 2018 19:05:12 +0000
Message-ID: <---#-----.prod.outlook.com>
Header when accessed programatically:
From: o365Account#company.com
To: *****************realAccount#mycompany.com ****************
Subject: shdaKJSDHA
Thread-Topic: shdaKJSDHA
Thread-Index: AQHUSTkz1fQhzI5SG0ie26mNIvHmmQ==
X-MS-Exchange-MessageSentRepresentingType: 1
Date: Mon, 10 Sep 2018 19:05:12 +0000
Message-ID: <----#-----.prod.outlook.com>
Keep in mind that Exchange always resolves the sender and recipients to their primary addresses both when sending and when receiving the messages. This is just the way it works.
Are you sending through SMTP?
You know what I ended up doing? I made a Distro list with 1 recipient. The distro list takes the place of the alias. It always shows the To: as the distro list. That way it doesn't get lost. Thank you for helping me understand Exchange Server dmitry-streblechenko.

Test "Received" email header in sieve

Here is an example of how I have configured sieve to forward any mail sent to [nameA|nameB|nameC]#example.org to my private email address.
if address :localpart :is ["To","Cc","Bcc"]
["nameA", "nameB", "nameC"] {
redirect "<my private email address>";
stop;
}
Sometimes though, email is not forwarded because the address that it was sent to is tucked away somewhere in a "Received" header.
Received: from ###server### ([###ip_address###])
by ###server### with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256)
(Exim 4.84)
(envelope-from <###email_address###>)
id 1alDM0-0000yT-60
for nameA#example.org; Wed, 30 Mar 2016 12:28:00 +0200
Is there an effective way to catch these emails in the sieve rule, too?
You have an XY Problem here. What you actually want to do here is filter based on the address being delivered to, not the address in the headers. (As unintuitive as it may be, the address in the headers may have nothing to do with the address it's being delivered to, which is how Bcc can work at all.)
The command to test against the actual SMTP envelope is envelope.
require "envelope";
if envelope :localpart :is "to" ["nameA", "nameB", "nameC"] {
redirect "<my private email address>";
stop;
}
This will handle all mail being delivered to those names, regardless of whether they show up in the mail headers or not.
With the help of the sieves' index feature you can parse the recipient address out of the Received headers.
For BCC sorting I typically do something like this:
require ["fileinto", "envelope", "variables", "mailbox", "index", "subaddress"];
...
if header :index 3 :matches "Received" "*<*#example.com>*" {
set :lower "foldername" "${2}";
fileinto :create "inbox.${foldername}";
} elsif header :index 2 :matches "Received" "*<*#example.com>*" {
set :lower "foldername" "${2}";
fileinto :create "inbox.${foldername}";
}
...
In the Received headers of the mails I receive the adress is set in angle brackets and that's why I've chosen the pattern in the example above.
Additionally, sometimes the number of Received headers varies thus I test at least for two different ones.

Exim: Forward Based on Recipient in bcc

Currently I am filtering incoming mails by a .forward in the following way:
if $header_to: matches "(office|info)#domain.com" then
save Maildir/.office/
endif
if $header_to: matches "whatever#domain.com" then
save Maildir/.whatever/
endif
So I have a mail account, which receives mails for different addresses. Basically I want them to land in different subdirs based on the address the mail was sent to.
This works for mails where the recipient is in the to-header, but does not work if the recipient was in the bcc.
When a mail is received which was sent with the bcc-header, only the envelope-to-header matches the real address the mail is delivered to and it is mentioned in a Received-header
Envelope-to: office#domain.com
Received: from mail.other.domain ([1.1.1.1])
by mail.domain.com with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
(Exim 4.71)
(envelope-from <sender#other.domain>)
id 1RO5xc-0001TF-Qj
for office#domain.com; Wed, 09 Nov 2011 12:04:57 +0100
...
To: can_be_anything#whatever.com
I already tried:
if $header_envelope-to: matches ...
but this does not work, mails are not filtered at all even when sent with To-header (looks like the Envelope-To-header is not available in forward-files). Should I try to parse the (multiple) Received-headers?
How can I move mails into a subdir of the recipient based on the real recipient address?
looks like I finally found the answer.
if $original_local_part matches "office|info" then
save Maildir/.office/
endif
This checks only the local_part, but afaik could be extended to use the domain, too, with $original_domain (see the doc)
The variable $recipients contains all (to, cc and bcc) recipients. Have you tried it?