Emails very delayed getting from mandrill to gmail - email

For the past 4 months we have been seeing large delays when sending emails through mandrill to gmail addresses. Sometimes it takes 15 minutes but other times it can be up to an hour. When i check the mandrill outbound section shortly after the email is sent it shows the email was delivered, but it usually takes a while before it actually shows up in my inbox. We are using this service for welcome emails and password resets so waiting long periods of time isn't acceptable.
It has been very hard to find any information on this issue. Has anyone seen this issue? Any recommendations on what i could do to fix it?

I had similar issues with delays on emails sent via Mandrill to gmail.
To fix the issue I viewed the "Sending Domains" page under "Settings" in Mandrill. I discovered the DKIM and SPF DNS records were either missing or not valid. Mandrill will provide you with new values by clicking on the "View... settings" link. After updating these settings we no longer experience the delay.

I've run into this issue a number of times. Our DNS settings were all good (DKIM and SPF confirmed my Mandrill) and after some investigation (looking at the headers of the delayed emails) the delay appeared to be entirely on Mandrill's side (once it was handed off to Gmail or Yahoo the delivery occurred within a second). When I contact Mandrill support they explained why we were seeing these delays:
In looking over the logs for your account we are seeing intermittent
delays for some of your recipients. Generally, the speed of delivery
in most cases depends largely on the receiving domain, and how quickly
they will receive and process emails. Most of the major email
providers limit how much email they'll receive in a certain period of
time, and will restrict delivery—Mandrill's sending servers are
designed to queue and back off sending if this occurs. In these cases,
the receiving mail server or ISP will return a specific kind of SMTP
response telling Mandrill's servers to 'back off' and 'try again
later,' which ultimately results in the message lingering on our mail
servers longer than expected (and since the message isn't passed off
to the receiving server at that point, and we're only getting a 'try
again' response, you won't see that information in the message headers
of the final email you receive. You'll only see that the email stayed
on our servers for a longer time period which can be confusing).
Additionally, even though we may hand the messages off to ISPs for
delivery almost immediately, it's still up to that ISP, like Gmail or
Yahoo, to actually to process that email and place it in the inbox.
Each receiving server is different though, so it may take a different
amount of time for Yahoo to process the mail than Gmail, for example.
In many cases, things like the time of day and overall email traffic
to that recipient server can affect how quickly they're able to
receive and process email.
All that said, the delays you're seeing generally aren't expected, and
while we see that messages are ultimately delivering, we are detecting
factors on our end where we may need to make some changes to help
mitigate further delays. Our delivery team is continuing to monitor
traffic to major ISPs and will make necessary adjustments as needed.
We still periodically see these delays, though they've improved is so the delays are rarely longer than 10 minutes or so, but it still can cause issues with things like password resets or confirmations that are time-sensitive. Bottom line: Mandrill is awesome for bulk mailing, but if you need instantaneous delivery you may want to rely on a different or self-hosted service.

I also had gmail showing emails sent through mandrill around 10 minutes later. And that is unacceptable to register confirmations and password resets.
I had configured my DKIM and SPF dns records and mandrill reported all green in this records.
But mail delivery to gmail was always delayed with no aparent reason.
After a while I decided do test/use my own email server to do this, instead of mandrill. Now there are no delays in gmail. I'm happy :)
After this I think I will only use mandrill for massive email delivery / marketing, where delays are not important. Time will tell.
Would like to hear other people about this subject.

In mandrillapp.com > Settings > Domains > Sending domains, verify these 3 points:
DKIM is valid,
SPF is valid,
domain is verified.

My experience has been that the Google SMTP servers are causing the delay (not Mandrill). Verify this by looking at the original email headers (in gmail, with email opened, in the top right More > Show Original) and pasting the email header into the google Message header analyzer will show you the path your email took and how long it was delayed at each server. This report will also tell you if you DKIM / SPF is invalid.
Why the delay is occurring is still a mystery to me. I suspect however that because the domain I am using to send is new, perhaps the gmail spam filters are grey listing the emails until enough users have opened emails and not clicked the spam button? I don't know.

Related

Determining maximum accepable email lag

This question is not 100% technical. I've looked online and I couldn't find this being discussed.
We have a forgot username feature in our website which mails the username to the user's email address, using our own mail server.
We also put a message in the web-page asking the user to wait for 20 minutes for the mail to arrive since there can be occasional delays.
Our business tester raised an interesting point saying that 20 minutes seems an unacceptable time to ask the user to wait for. He said our technology should send emails immediately and the maximum lag should be 1 minute at worst.
Firstly, emails are usually received immediately by the recipient so there isn't a problem there.
But in our experience in using other websites, sometimes emails do take a while to arrive. I also remember reading somewhere that emails (at a network level) use a lower quality of service QoS unlike voip services. I can't seem to find it now.
Users can also experience delays in receiving emails because of issues in their own mail server.
Now, all we can do is send the mail using our mail server and ensure that the load and resources on the server is well managed.
1) Is there anything else we can do to ensure that our mails are sent quickly at all times.
2) What is the acceptable time, we can ask the user to wait until he logs a call with help-desk? I believe that there can be lags at a network/protocol level and the user's mail server which we can't do anything about.
Thanks.
All you can do is to inform user that your server "handed over" responsibility for email delivery to named SMTP server "beyond your control/responsibility". You may expect it would take a few (<5s) in most cases (>50%).
Your smtp-client may try initial delivery attempt.
On delivery success => inform the user that email delivery/delay is now beyond your control
On (initial) delivery failure => pass message to your SMTP server (with initial delivery attempt skipped).

SendGrid Emails Getting Rejected as Spam

I'm making a user management system for my app, and I need to send users a "forgot my password" email with a token that lets them reset their account password. I signed up for SendGrid through Azure (to get the 25,000 emails per month free, which sounded like a great deal) and wrote some code to use it, but after testing my program a bit I was dismayed to find that only a couple of my emails actually went through.
After going onto the SG control panel, I found that 4 out of the 6 test emails I sent went through, and all of the others were rejected as being spam. I sent an email to mail-tester.com to see what it though my spam score was and it gave me a 4.3/10.
The email in question was a single sentence with a link to the password reset, without any images or other elements. I only sent those 6 emails out, so the volume of my emails definitely wasn't the issue. Still, I'm very puzzled as to why my messages are getting flagged as spam.
Without going to the trouble of making an elaborate authentication setup, are there any basic changes I can make to my system to make it get through to users?
In this case it's most likely because you are sending such a short message, with a link to 'reset your password' from a non-whitelabelled email address (the email address you're sending from cannot be verified against the actual domain), and the link may also be a different URL. It's probably getting pulled up as a potential phishing email.
You can rectify this by white labeling your domain and email links via the SendGrid dashboard, it's easy to do and should improve your deliverability.
Also check out this article from the SendGrid support team about White Labeling.
A question from 2015 which is sadly still relevant today as usage of SendGrid increases.
My organization has blocked all SendGrid mails except for those on the paid tier using fixed IP addresses with resolvable public DNS names (such as sendgrid1.sampledomain.tld) which we then whitelist.
There are now far too many domain impersonation, phishing and other spam mails coming in from SendGrid for us to allow everything from them - roughly 10 000 mails over a seven day period, which is far too many to manually report to SendGrids abuse department.
So my answer would be that switching to the paid tier of SendGrid is the better option if you like a better chance of your mails arriving intact at their destination.
I receive only Spam Mails from Sendgrid.
Goes direct to Spam folder and try to report Sendgrid everywhere I can. Maybe they get blocked by most mail servers and make them think about their policy in "hosting" all these Spammers.
In my case my emails are marked as spam because of the anchor label different to the href being actually called.
And that's because of the 'click tracking' setting of sendgrid.
So, if you have something like
yourdomain.com
sendgrid may replace the href and you end up with something like:
yourdomain.com
The sendgrid page being called tracks the click and then redirects the user to the url you originally set. But this sometimes results in your email being marked as spam.
Try to set 'click tracking' in sendgrid dashboard to off: settings | tracking | click tracking.
details here: https://sendgrid.com/docs/ui/account-and-settings/tracking/
Always start by setting up Domain Authentication, formerly known as domain whitelabel as #MartynDavies says. Found under Settings -> Sender Authentication in the UI. Should look like this:
https://sendgrid.com/docs/ui/account-and-settings/how-to-set-up-domain-authentication/
To identify problems have a look at Activity and choose to see deferred, drops, bounces, blocks and spam reports.
https://app.sendgrid.com/email_activity
Under Suppressions you can see details for Blocks and Bounces among others:
https://app.sendgrid.com/suppressions/blocks
https://app.sendgrid.com/suppressions/bounces
There you can see errors like:
550 5.7.1 SPF check failed. em1234.mydomain.com does not declare 11.222.33.44 as a valid sender
If it says Verified but you see errors like this then contact SendGrid support.
One thing that has worked is to upgrade from the Free plan to Essentials or Bronze via the Azure Portal. This made a lot of the emails marked as spam pass through.
I had a similar issue when trying to send a user verification email using SendGrid.
In my case, using a custom domain as the sender identity solved the issue.
Make sure to also verify the domain before using it.

Mandrill Emails not reaching any mailbox, but Mandrill showing status 'Delivered'

Mandrill does not offer any support. I've sent them many tickets, but still no answer. I hope someone here will help me out.
I'm sending emails through SMTP. I'm able to send few thousand emails in start, but after few thousands, no email reaching any mailbox. Mandrill activity showing that the email is delivered, but it's not and there's no email in backlog. I have limits around 50K/hour.
I tried making another account after a day, and did some deposit too, but same thing happen with other account too. No error in SMTP client, no error in logs, nothing.
Other thing to notice is, when I hover over 'Delivered' it says "No SMTP event", but emails which are actually got delivered, showing some stats on hover over.
The Mandrill Delivered-status in the UI doesn't actually mean that it is sent, only that Mandrill have received the message for processing. This is of course extremely confusing.
The only way of seeing if an email is actually sent (i.e. successfully delivered to the receiving mail server) is to see if the message has smtp-events. Do note that it can take some time before the SMTP-events are available in the GUI/API (I have experienced a delay of between 2 minutes and 24 hours).
To see all emails that not currently confirmed delivered you can search for "NOT smtp_events.diag:250" in the search field.
After some research, one of the reason, I came to know that if TEST API KEY gets used to send emails, no mail is actually sent to inbox, but webhooks trigger normally and it shows status as delivered on the Mandrill UI. In order to check actual delivery of email to your inbox, please try to use different API KEY other than test account
It turns out to have been a delay in the emails being received by the mailserver (gmail). It appears as if they were accepted and not processed for a few hours.
I had run in to the same issue and this explanation given by OakHosting_James helped me a lot in understanding what is going on:
It turns out the message was sent from IP that is on an RBL (it happens - I get that). So the receiving server rejected the message at SMTP time.
They replied to say two things about that:
(i) Anyone can set up a blacklist and put any IP on for no good reason.
(ii) Some messages bounce in such a way that Mandrill is not able to detect that it's bounced - which is why their website said "delivered".
Let's take those issues:
(i) It's true, but this was UCEPROTECT-Level 1. They're not a pleasant blacklist to work with, but they're not a backyard project for someone with an axe to grind. I still get the fact that it's impossible for any sending network to remain 100% clean. I'm not frustrated that one of their IPs was listed temporarily. But I did feel fobbed off by being told that there are some tiny blacklists out there that no-one in their right mind would use. UCEPROTECT is not one of those, and they should have come clean: "Even with the best spam protection, we get blacklisted occasionally; we detect this very quickly and switch to other IPs."
(ii) I'm sure there are some after-the-fact bounces that Mandrill's system can miss. But this was rejected at SMTP time. How can they mark a message that never left their sending server as "delivered"?
So the solution (to some degree) to the RBL IP problem (i) could be an "Dedicated IP $29.95 / month" for your account in Mandrill. But using a dedicated IP can be a problem on it's own and is for most cases not advised.
In your mail.rb file you need to do the following :
ActionMailer::Base.smtp_settings = {
address: "smtp.mandrillapp.com",
port: 587,
enable_starttls_auto: true,
user_name: "yourname#gmail.com",
password: "apipassword",
authentication: "login"
}

Mandrill: How long do I wait before assuming deliverability?

I've set up a Mandrill webhook which will update my app whenever an email hard-bounces or is rejected, so I don't keep that particular email address in my database. The way it works is this: a user gives me an address, I send him a confirmation, and if I don't hear from Mandrill's webhook in 30 minutes, I assume it's OK.
So I ran a some tests with non-existing addresses, and they didn't go too well. Most of them appeared as delivered for hours, long after I'd assumed they were fine.
Also, I didn't account for the delay in receiving webhook batches. One mail bounced at 2:01pm, according to the outbound activity logs, but the webhook history shows a batch being sent only at 2:52pm.
My questions is: how long should I delay my app's assumption of deliverability in order to give Mandrill enough time to detect hard-bounces / rejections and then send me webhook batches? I can live with letting some 5% bad emails going by because of delayed in processing on peak-time or other extraordinary events, but it seems like my 30 minutes isn't enough to catch anything at all...
Not the answer you're looking for, but Mandrill doesn't let you do this. The only way of checking if an email has been delivered is to poll Mandrill (with the message/info.json API). To see if the message is delivered you have to check the smtp_events and look for an event with diag starting with 250. As you've already experienced it may take a long time between delivery and when a message is accessible through the API. In my experience the normal case is around 10 minutes, but it might take many, many hours (this is the case for bounced emails as well as emails that got delivered immediately).
If it is important to you to know when an email is delivered, I would recommend you to switch to another email provider. There are plenty of different ones out there. I've personally used Amazon SES. They're cheaper than Mandrill, and you can expect a delivery notification after a second or so. Do note that Amazon SES is a bit more bare-bone than Mandrill (they don't have support for open/click-tracking, templating, dedicated IP, etc.), so it might not be the right provider for you.

Limits you hit when sending lots of email from your server

What kind of practical issues are there concerning sending tons of e-mail from a server? Will the likelihood of that e-mail being received be just the same as if it had been sent from g-mail or a personal e-mail account if I for example just blindly call the mail() function in PHP tens of thousands of times a day?
(note: you are not helping a spammer here, this relates to a notify feature I'm thinking about for a future link sharing site)
While you may technically be able to send thousands of mails per minute, in reality you must be carefull.
Say you send out 500 emails to yahoo for example. if enough people mark your message as spam, soon, ANY email you send to yahoo will be marked as spam, or [BULK]. Many isp's routinely tar-pit or outright reject email from servers on lists such as RBL (the real-time black hole list). If your mail IP gets put on one of these lists, you can kiss sending email normally from that ip ever again goodbye. Users are very finicky and it doesn't take many complaints to get your mail ip blocked at many domains.
Also since you are sending automated messages, there are heuristics used to determine if the same message is being sent to many users on the same domain. This also increases the chance your mail will be marked as spam.
This is why clean emails from some addresses always go into the spam box. Their company may have not been careful enough when sending what could be perceived as spam. Proceed with caution.
http://wiki.apache.org/spamassassin/AvoidingFpsForSenders
http://support.microsoft.com/kb/842851
http://www.blacklistedip.com/rbl_list.php
It helps to set a 'x-mailer' and ('X-MimeOLE' if your pretending to be outlook) of a real mail client.
It also helps to send it from a server that is a mail server for the domain in the from address, with forward & reverse DNS records setup.
No issues. Once a server is correctly configured as a mail server (SMTP) for a particular domain, there is no difference if the mail it sends out came to it from Outlook, or from the mail() function in PHP - both are getting the SMTP server to do all the heavy lifting
I always make sure to set my X-Mailer headers correctly (identifying that the message was sent from within PHP) to ensure that any overzealous anti-spam services recognize it as an automatated notification as opposed to bulk/junk email. e.g.
$headers .= "X-Mailer: PHP/".phpversion();
All the configuration and limits you'll encounter are with the SMTP server, not from PHP. You can configure SMTP to rate-limit to 2 messages per second for example, this means if you queue up 1,200 messages they'll be drip fed out over the next hour rather than all at once (two is a really low number, 5-25 is more realistic).
SMTP is the backbone of email and some SMTP servers can happily handle tens of thousands of messages per minute (or more!) - the only limitation you'll likely face is bandwdith ;)
Check with your hosting provider, especially if you're on shared hosting. For example: GoDaddy limits shared hosting accounts to sending 1000 emails per day on their server (http://support.godaddy.com/groups/web-hosting/forum/topic/e-mail-sending-limit/). I'm sure other providers have their own limits (I believe the provider one of the companies I worked for used limited outgoing emails to 250 per minute or something along those lines).
Edit: In my case the solution was to contact our hosting provider. They provided info to route outgoing emails through a server they had dedicated to sending outgoing emails. Solved the problem right away.