I am sending a newsletter via Gmail's SMTP (actually, the SMTP depends on the mail's content, but let's use Gmail for this question).
My problem is, that bounced mails shall not return to the GMail account, but to another address. This address includes some VERP data to automatically detect outdated addresses.
What I know so far:
Gmail restricts the FROM header (which is perfectly fine for me, we do not want to spoof anything).
According to the SMTP specs, Gmail changes the Return-Path (or better: adds another one).
The Error-to header is deprecated
Using PHPMailer, I already have set the envelope sender ($mail->Sender property), which is correctly sent during SMTP communication
2014-06-11 13:59:51 CLIENT -> SERVER: MAIL FROM:<mailer+P=49276#myserver.de>
2014-06-11 13:59:51 SERVER -> CLIENT: 250 2.1.0 OK ey16sm6579150wid.14 - gsmtp
So ... is there any other option to tell the receiving mailserver to send bounces to another address? And/or tell the Gmail SMTP server to send the bounces there, if they are directly generated by the SMTP server?
UPDATE
It seems that disregarding my envelope sender goes along with the known Gmail protocol violation of changing the From header (http://lee-phillips.org/gmailRewriting/). When trying the same with GMX, the email is simply refused until the envelope sender matches the GMX mail address corresponding to the login.
Does that mean, I have no chance to alter the address that bounces are sent to?
Related
I am creating a webservice with Mailgun to send out emails. It will BCC my own domain's email for every email sent out. Assuming my domain is "example.com". For every email sent out to a customer, say, customer1#gmail.com, I will BCC its content to sales#example.com.
Currently, the domain example.com and its email is hosted on a server with CPanel.
In Mailgun, I have added and verified the domain example.com. Using this domain, I've sent a mail to customer1#gmail.com and sales#example.com. The email is sent without issues to Gmail, however when sending to sales#example.com, I keep getting the error Server response: 550 550 Verification failed for <bounce+e0f051.e0179a-sales=example.com#example.com> No Such User Here.
What's baffling here is that if i send the email via Mailgun with another verified domain such as anotherexample.com, and then using this, I send my mail to sales#example.com. The email arrives perfectly fine without errors.
So far, the things I've tried:
Added Mailgun suggested SPF and DKIM
Modified SPF to include my CPanel server's IP (together with Mailgun SPF)
Deleted both the SPF and DKIM (one at a time and both at once)
Verified that the email sales#example.com exists. Using the CPanel webmail's interface, I can send and receive emails just fine.
Tried updating the CPanel MX entries Email routing from Local -> Automatic -> Remote. ("Local" works the best. If its set to "Remote", email sending and receiving doesnt work at all, even if mails are sent through Gmail/Hotmail).
My current MX settings are:
Priority 0: mail.example.com
My current Zone file records on CPanel:
example.com A <some ip>
mail.example.com A <same ip as above>
The code I am using to send mails via Mailgun (Ruby):
mg_client = Mailgun::Client.new 'xxxxxxxxxxx'
message_params = {:from => from_email,
:to => customer.email,
:bcc => bcc_email,
:subject => MessageTemplate.email_subject,
:text => message}
result = mg_client.send_message('example.com', message_params).to_h!
I currently do not have the SPF and DKIM records in the zone files. I've added and removed them and they had no effect on the error (still delivers fine to Gmail too).
I've spend the whole on this, scouring forums and whatnot but can't seem to find a solution.
If at all relevant, I have a 301 redirect of example.com to www.example.com(Which has a CNAME pointing to another server). But I've researched and found out that 301 redirect does not affect emails.
I don't think this is a send-side problem. You're sending to sales#example.com, but you're getting errors relating to bounce+e0f051.e0179a-sales=example.com#example.com, which is a typical VERP address. Now, VERP addresses are fine, so long as you're expecting them. Given that you are not apparently providing that explicit address to MailGun, I assume that they are generating that address automatically. I would check their documentation for how they generate return-path (envelope sender) addresses, and either override the sender address (with just sales#example.com), or configure handling of those VERP addresses on your own inbound mail server.
Here is a mailgun explanation
This error occurs due to what is termed Sender Address Verification (SAV). During SAV, an email server performs an MX lookup upon the domain (example.com) listed within the message envelope's Mail-From field. SAV typically rejects the message if,
the sender's (in this case, Mailgun's) MX records are not configured for that domain AND
the domain of the message envelope's Mail-From field does not match the domain of the message header's From field.
https://help.mailgun.com/hc/en-us/articles/360011804533-Sender-Verification-Error
Is it a problem if 127.0.0.1 appears in email headers?
Example: Received: from baobabsmail.baobab.fi ([127.0.0.1])
I ask because emails sent from my server to #outlook.com addresses end up in the spam folder and this is the last thing I can think of. I have properly configured HELO, DKIM, Reverse DNS, SenderID, SPF and DMARC. I don't send out mass emails. My IP is from AWS, but it isn't on any publicly available blacklists. I have verified that everything is set up correctly using DKIMvalidator, MxToolBox and mail-tester.
Edit: for what it's worth, I finally got rid of the 127.0.0.1's in my headers and it did not resolve the issue for me.
Unfortunately, it depends...
Mail systems vary in how they are configured, and it is perfectly legitimate for an MUA (e.g. Thunderbird) to send outgoing mail to an MTA / mail server (e.g. exim) running on the same machine using the localhost address. Unusual, these days, but not "bad by definition".
When you say 'end up in the spam folder', what is it that puts it there: are you using a local mail server? if so is it that server that junks the mail (on send) or outlook.com itself (on receipt). Either way, what error messages or other failure information have you found?
Some random thoughts:
DKIM is a pain to set up correctly. Try disabling it entirely and see if that changes things in interesting ways.
Ditto DMARC.
Have you got SPF set up separately? If so, disable SPF and retry.
Is IPv6 involved in the mix at all? Various things are subtly different if so.
If outlook.com were to do sender verify callbacks (i.e. on receipt, check that mail from address was an acceptable recipient to your server) would it pass?
Is your email system sending RFC-conformant mail: that is, does it have a From: address, To: or Sender: address, Message-ID:, Date: headers and, if using MIME, Content-* headers (and probably a couple I forget!).
If changing DKIM / DMARC / SPF changes things (and remembering DNS timeouts, leave it a while between attempts), re-add SPF first - it is the simplest to get right.
127.0.0.1 can be flagged by Spam filters because it fails to provide an identity trace of the sender. Most common e-mail systems will show the IP address or the host name. The next item will be the recipient e-mail server.
For example:
Received: from [127.0.0.1] (81.27.148.196) by TAE1.agent.com.pk
What is funny about this one is that the top-level domain says it is received initially by a domain in Pakistan, but the IP address is registered to an entity in St. Petersburg, Russia.
I have a client which sends email using the smtp protocol. The client can be configured against any mail server which supports smtp (for example: gmail, yahoo).
Now, while sending the mail from client, I am setting the "message-id" in the header. Now, my question is does mail server guarantees that the message id set by me in the header will "not" be reset (or replaced) by the mail server, before actual forwarding the mail.
For example: Let say my client is configured against "abc#gmail.com". Now I want to send mail to "abc2#yahoo.com". So client set message-id as "message123456644#abc.gmal.com" and send the mail using smtp.
So, now when gmail sends this message to yahoo, does it guarantee that my message-id will be retained.
Thanks
Kunal
The Message-ID is a unique string assigned by the mail system when the message is first created. This is also forgeable in most cases, but requires a little more specialized knowledge than forging the From: line. Also, the Message-ID: often identifies the system from which the sender is logged in, rather than the actual system where the message originated.
Long story short: Your client can set it, but there is not guarantee it will remain that way.
Message ID is used to understand the originating server of the message. Technically this is set by the first message server that handles your message. Once set this is not meddled by subsequent server. In your case your smtp client acts as the originating server and the server will not meddle with it.
(What are you doing with message id anyways.)
On our mailing application we are sending emails with the following header:
FROM: marketing#customer.com
TO: subscriber1#domain1.example
Return-PATH: bouncemgmt#ourcompany.example
The problem that we are facing is that some email servers will bounce back a message immediately and use the from or reverse path (marketing#customer.example) instead to our bounce mgmt server. We want to know if we modify in the header the reply-to to be the same as the return-path if we will be able to catch all bounces.
Any other ideas are welcome?
We are using the following documents as references:
VERP
RFC
Bounce Messages
SMTP Log Parsing to get Bounces
EDIT 1: A few more bits of information to see if we can get this resolve.
We want to know at what point the email server relaying the message will choose to use the reply-to versus the return-path. We have notice that when the first SMTP server relaying the message gets rejected it sends it to the reply-to, but when it happens after one hop it sends it to the return-path.
Let's start with a simple example. Let's say you have an email list, that is going to send out the following RFC2822 content.
From: <coolstuff#mymailinglist.example>
To: <you#example.com>
Subject: Super simple email
Reply-To: <coolstuff-threadId=123#mymailinglist.example>
This is a very simple body.
Now, let's say you are going to send it from a mailing list, that implements VERP (or some other bounce tracking mechanism that uses a different return-path). Lets say it will have a return-path of coolstuff-you=yourcompany.com#mymailinglist.example. The SMTP session might look like:
{S}220 workstation1 Microsoft ESMTP MAIL Service
{C}HELO workstation1
{S}250 workstation1 Hello [127.0.0.1]
{C}MAIL FROM:<coolstuff-you=yourcompany.com#mymailinglist.example>
{S}250 2.1.0 me#mycompany.com....Sender OK
{C}RCPT TO:<you#example.com>
{S}250 2.1.5 you#example.com
{C}DATA
{S}354 Start mail input; end with <CRLF>.<CRLF>
{C}From: <coolstuff#mymailinglist.example>
To: <you#example.com>
Subject: Super simple email
Reply-To: <coolstuff-threadId=123#mymailinglist.example>
This is a very simple body.
.
{S}250 Queued mail for delivery
{C}QUIT
{S}221 Service closing transmission channel
Where {C} and {S} represent Client and Server commands, respectively.
The recipient's mail would look like:
Return-Path: coolstuff-you=yourcompany.com#mymailinglist.example
From: <coolstuff#mymailinglist.example>
To: <you#example.com>
Subject: Super simple email
Reply-To: <coolstuff-threadId=123#mymailinglist.example>
This is a very simple body.
Now, let's describe the different "FROM"s.
The return path (sometimes called the reverse path, envelope sender, or envelope from — all of these terms can be used interchangeably) is the value used in the SMTP session in the MAIL FROM command. As you can see, this does not need to be the same value that is found in the message headers. Only the recipient's mail server is supposed to add a Return-Path header to the top of the email. This records the actual Return-Path sender during the SMTP session. If a Return-Path header already exists in the message, then that header is removed and replaced by the recipient's mail server.
All bounces that occur during the SMTP session should go back to the Return-Path address. Some servers may accept all email, and then queue it locally, until it has a free thread to deliver it to the recipient's mailbox. If the recipient doesn't exist, it should bounce it back to the recorded Return-Path value.
Note, not all mail servers obey this rule; Some mail servers will bounce it back to the FROM address.
The FROM address is the value found in the FROM header. This is supposed to be who the message is FROM. This is what you see as the "FROM" in most mail clients. If an email does not have a Reply-To header, then all human (mail client) replies should go back to the FROM address.
The Reply-To header is added by the sender (or the sender's software). It is where all human replies should be addressed too. Basically, when the user clicks "reply", the Reply-To value should be the value used as the recipient of the newly composed email. The Reply-To value should not be used by any server. It is meant for client-side (MUA) use only.
However, as you can tell, not all mail servers obey the RFC standards or recommendations.
Hopefully this should help clear things up. However, if I missed anything, let me know, and I'll try to answer.
Another way to think about Return-Path vs Reply-To is to compare it to snail mail.
When you send an envelope in the mail, you specify a return address. If the recipient does not exist or refuses your mail, the postmaster returns the envelope back to the return address. For email, the return address is the Return-Path.
Inside of the envelope might be a letter and inside of the letter it may direct the recipient to "Send correspondence to example address". For email, the example address is the Reply-To.
In essence, a Postage Return Address is comparable to SMTP's Return-Path header and SMTP's Reply-To header is similar to the replying instructions contained in a letter.
for those who got here because the title of the question:
I use Reply-To: address with webforms. when someone fills out the form, the webpage sends an automatic email to the page's owner. the From: is the automatic mail sender's address, so the owner knows it is from the webform. but the Reply-To: address is the one filled in in the form by the user, so the owner can just hit reply to contact them.
I had to add a Return-Path header in emails send by a Redmine instance.
I agree with greatwolf only the sender can determine a correct (non default) Return-Path.
The case is the following:
E-mails are send with the default email address: admin#example.com
But we want that the real user initiating the action receives the bounce emails, because he will be the one knowing how to fix wrong recipients emails (and not the application adminstrators that have other cats to whip :-) ).
We use this and it works perfectly well with exim on the application server and zimbra as the final company mail server.
For the past few weeks, I've been trying to learn about just how email works. I understand the process of a client receiving mail from a server using POP pretty well. I also understand how a client computer can use SMTP to ask an SMTP server to send a message. However, I'm still missing something...
The way I understand it, outgoing mail has to make three trips:
Client (gmail user using Thunderbird) to a server (Gmail)
First server (Gmail) to second server (Hotmail)
Second server (Hotmail) to second client (hotmail user using OS X Mail)
As I understand it, step one uses SMTP for the client to communicate. The client authenticates itself somehow (say, with USER and PASS), and then sends a message to the gmail server.
However, I don't understand how gmail server transfers the message to the hotmail server.
For step three, I'm pretty sure, the hotmail server uses POP to send the message to the hotmail client (using authentication, again).
So, the big question is: when I click send Mail sends my message to my gmail server, how does my gmail server forward the message to, say, a hotmail server so my friend can recieve it?
Thank you so much!
~Jason
Thanks, that's been helpful so far.
As I understand it, the first client sends the message to the first server using SMTP, often to an address such as smtp.mail.SOMESERVER.com on port 25 (usually).
Then, SOMESERVER uses SMTP again to send the message to RECEIVESERVER.com on port 25 (not smtp.mail.RECEIVESERVER.com or anything fancy).
Then, when the recipient asks RECEIVESERVER for its mail, using POP, s/he recieves the message... right?
Thanks again (especially to dr-jan),
Jason
The SMTP server at Gmail (which accepted the message from Thunderbird) will route the message to the final recipient.
It does this by using DNS to find the MX (mail exchanger) record for the domain name part of the destination email address (hotmail.com in this example). The DNS server will return an IP address which the message should be sent to. The server at the destination IP address will hopefully be running SMTP (on the standard port 25) so it can receive the incoming messages.
Once the message has been received by the hotmail server, it is stored until the appropriate user logs in and retrieves their messages using POP (or IMAP).
Jason - to answer your follow up...
Then, SOMESERVER uses SMTP again to send the message to RECEIVESERVER.com on port 25 (not smtp.mail.RECEIVESERVER.com or anything fancy).
That's correct - the domain name to send to is taken as everything after the '#' in the email address of the recipient. Often, RECEIVESERVER.com is an alias for something more specific, say something like incoming.RECEIVESERVER.com, (or, indeed, smtp.mail.RECEIVESERVER.com).
You can use nslookup to query your local DNS servers (this works in Linux and in a Windows cmd window):
nslookup
> set type=mx
> stackoverflow.com
Server: 158.155.25.16
Address: 158.155.25.16#53
Non-authoritative answer:
stackoverflow.com mail exchanger = 10 aspmx.l.google.com.
stackoverflow.com mail exchanger = 20 alt1.aspmx.l.google.com.
stackoverflow.com mail exchanger = 30 alt2.aspmx.l.google.com.
stackoverflow.com mail exchanger = 40 aspmx2.googlemail.com.
stackoverflow.com mail exchanger = 50 aspmx3.googlemail.com.
Authoritative answers can be found from:
aspmx.l.google.com internet address = 64.233.183.114
aspmx.l.google.com internet address = 64.233.183.27
>
This shows us that email to anyone at stackoverflow.com should be sent to one of the gmail servers shown above.
The Wikipedia article mentioned (http://en.wikipedia.org/wiki/Mx_record) discusses the priority numbers shown above (10, 20, ..., 50).
You're looking for the Mail Transfer Agent, Wikipedia has a nice article on the topic.
Within Internet message handling services (MHS), a message transfer agent or mail transfer agent (MTA) or mail relay is software that transfers electronic mail messages from one computer to another using a client–server application architecture. An MTA implements both the client (sending) and server (receiving) portions of the Simple Mail Transfer Protocol.
The terms mail server, mail exchanger, and MX host may also refer to a computer performing the MTA function. The Domain Name System (DNS) associates a mail server to a domain with mail exchanger (MX) resource records containing the domain name of a host providing MTA services.
You might also be interested to know why the GMail to HotMail link uses SMTP, just like your Thunderbird client. In other words, since your client can send email via SMTP, and it can use DNS to get the MX record for hotmail.com, why doesn't it just send it there directly, skipping gmail.com altogether?
There are a couple of reasons, some historical and some for security. In the original question, it was assumed that your Thunderbird client logs in with a user name and password. This is often not the case. SMTP doesn't actually require a login to send a mail. And SMTP has no way to tell who's really sending the mail. Thus, spam was born!
There are, unfortunately, still many SMTP servers out there that allow anyone and everyone to connect and send mail, trusting blindly that the sender is who they claim to be. These servers are called "open relays" and are routinely black-listed by smarter administrators of other mail servers, because of the spam they churn out.
Responsible SMTP server admins set up their server to accept mail for delivery only in special cases 1) the mail is coming from "its own" network, or 2) the mail is being sent to "its own" network, or 3) the user presents credentials that identifies him as a trusted sender. Case #1 is probably what happens when you send mail from work; your machine is on the trusted network, so you can send mail to anyone. A lot of corporate mail servers still don't require authentication, so you can impersonate anyone in your office. Fun! Case #2 is when someone sends you mail. And case #3 is probably what happens with your GMail example. You're not coming from a trusted network, you’re just out on the Internet with the spammers. But by using a password, you can prove to GMail that you are who you say you are.
The historical aspect is that in the old days, the link between gmail and hotmail was likely to be intermittent. By queuing your mail up at a local server, you could wash your hands of it, knowing that when a link was established, the local server could transfer your messages to the remote server, which would hold the message until the recipient's agent picked it up.
The first server will look at DNS for a MX record of Hotmail server. MX is a special record that defines a mail server for a certain domain. Knowing IP address of Hotmail server, GMail server will sent the message using SMTP protocol and will wait for an answer. If Hotmail server goes down, GMail server wiil try to resend the message (it will depend on server software configuration). If the process terminates ok, then ok, if not, GMail server will notify you that he wasn´t able to deliver the message.
If you really want to know how email works you could read the SMTP RFC or the POP3 RFC.
All emails are transferred using SMTP (or ESMTP).
The important thing to understand is that the when you send message to someguy#hotmail.com this message's destination is not his PC. The destination is someguy's inbox folder at hotmail.com server.
After the message arrives at it's destination. The user can check if he has any new messages on his account at hotmail server and retrieve them using POP3
Also it would be possible to send the message without using gmail server, by sending it directly from your PC to hotmail using SMTP.
Step 2 to 3 (i.e. Gmail to Hotmail) would normally happen through SMTP (or ESMTP - extended SMTP).
Hotmail doesn't send anything to a client via POP3. It's important to understand some of the nuances here. The client contacts Hotmail via POP3 and requests its mail. (i.e. the client initiates the discussion).