I have been working on AWS for the last month trying to scale our application email sending, we were using email chimp and decided to migrate our servers to amazon. Our application currently generates between 3000-4000 emails a day (not all at once and in different time span). The issue i'm trying to solve is deliver the emails in the least time possible (SES send is 14 mails/s.)
What i been able to do is: Application -> SQS -> Lambda pull (schedule 1 per minute that pull 10 messages) -> SES -> SNS -> Application.
The Lambda schedule is generated with the cloudwatch rule, i've seen that you can target events but i haven't been able to do it =(
I'm trying to find the correct approach but i haven't been able to put all my thoughts together.
Can anyone help me? =)
Firstly if you want to increase the maximum send rate, you may open a case in the Support Center.
Then you could setup a CloudWatch alarm for your SQS NumberOfMessagesSent metric and call a SNS topic that triggers a Lambda. You can trigger this Lambda if your NumberOfMessagesSent is greater than a certain value. eg: 1, 10 or maximum SES send rate. Lambda could call SES and send emails for the newly added messages. The method that I'm proposing is SNS -> Lambda -> SES. In this method, you may not want to rely on a schedule.
Related
In testing code that uses the SendGrid Email Activity API, I have received "too many messages" errors. I have examined the "rate limit" response headers and it appears that I am being limited to 10 requests per 5 minute block in the day. That is, the first 5 minutes of every hour can have 10 requests, the next 5 minutes can have 10 requests, etc.
I asked SendGrid support about this. The first response was pretty generic, but seems to indicate that the threshold is correct and says I really should be using webhooks to get the status. I haven't found anything in the documentation saying this and I haven't seen anything the specifies what the rate limits are.
For those of you using the Email Activity API, are you limited to 10 requests per 5 minutes? If yes, what do you do with the API?
Here's an snippet of what I ended up using with requests, tenacity and ratelimit:
from ratelimit import limits, sleep_and_retry
import requests
import tenacity
#sleep_and_retry
#limits(calls=2, period=60)
#tenacity.retry(
retry=tenacity.retry_if_exception_type(requests.exceptions.HTTPError),
stop=tenacity.stop_after_attempt(10),
wait=tenacity.wait.wait_fixed(60),
)
def _call_api(headers, params):
response = requests.get(
"https://api.sendgrid.com/v3/messages",
json={},
headers={},
params={},
)
try:
response.raise_for_status()
except requests.exceptions.HTTPError:
logger.info(f"Request failed {response.headers}, retrying in 1 minute")
raise
return response
I received a response from SendGrid support that says:
Your findings are correct in that we do limit this endpoint to 10 requests per 5 minutes. This is a hard limit that we do not have the means of raising. The Email Activity Feed as well as the Email Activity API endpoint are meant for troubleshooting specific issues and attaining detailed message metadata.
I previously found the rate limit to be 10/5min but it appears that SendGrid have changed the rate limit to 2 requests every 60 seconds sometime in the past week. Can anyone confirm this?
I'm using the webhook to report non-delivery back to my application but I also need to use the activity API to resolve async bounce notifications. Async bounces are when a destination mail server accepts a message during the smtp session but subsequently sends a bounce notification email. When this occurs, SendGrid do not provide the detail of the bounced message in the webhook and the message is incorrectly reported as delivered in the SendGrid app. When asked, they respond that there's nothing they can do about it, even though I have explained to them how I use their activity api to resolve this.
I pay extra to use the activity API to fix a problem that they should address themselves, so I'm very frustrated that they apply such restrictive rate limits, then change them without notice.
I have some a script with trigger set up to send emails daily with GSuite account (about 10 triggers to send about 15 email in total per day, average 6 recipients/email.
However, last few days, I got the alert:
Exception: Service invoked too many times for one day: email
when running the script by trigger or by manually.
I do not think I hit the quota of sending email daily of Google. This morning, when I check the quota remaining by function MailApp.getRemainingDailyQuota(), I only get 4 email remaining. I do not know what happened.
Is there anyone could please help me to solve this problem?
This is the alert I receive:
Your script, AutomaticSendingEmail, has recently failed to finish
successfully. A summary of the failure(s) is shown below. To configure
the triggers for this script, or change your setting for receiving
future failure notifications, click here.
Start Function Error Message Trigger End
6/13/20 3:59 PM send_overtimerequest_email Exception: Service invoked too many times for one day: email. time-based 6/13/20 3:59 PM
Sincerely,
Looking at the page for Apps Script quotas it does specify that the limitations per day is counting recipient (100 recipient/day), so in this case you have no much option. Make sure that you are using the correct account because the limitation for G Suite accounts are 1500 recipient/day. Also check out if this may be applying to you right now:
Note: Newly created G Suite domains are subject to the consumer limit for the first billing cycle if they have six or more users, or several billing cycles if they have fewer users. For more information, see the Help Center page on sending limits.
A workaround that I would suggest is to have a service account and use Domain Wide Delegation to impersonate and user then make use of the Gmal API
We have a CloudFormation stack that we want to provide to our clients. When they run the stack, we want to receive some output values directly, i.e. we don't want them to need them to send us the output. Our first thought was to use SNS and the notification capabilities of CF but it seems that the topic must be in the account running the template and can't be in another account. We also considered subscribing to the existing SNS topic as part of the template but that doesn't get a message sent.
We realize that CF is a resource creation tool but we think there must be a way to get the info relayed to us automatically. Doesn't have to be SNS. Any ideas on how we might be able to do this?
Update your CF script to contain a lambda and cloud watch rule which runs every 5 minutes on a cron.
Give the lambda IAM permissions to query the stack/get any output values you require.
When the lambda triggers you query the data you need and can send it to yourself however you see fit. E.g http POST to an API you own.
To finish up your lambda should call the cloud watch API to disable the cloud watch rule so this code doesn't run again.
You should consider if all this offsets the effort of saying to your client "please send us details of x y z". If you have 10 clients, probably not, if you have 1000 clients then possibly.
The Current Quotas table on the Quotas for Google Services page shows a feature called "Email read/write (excluding send)," which is limited to 50,000 / day for G Suite Business customers.
I have several Google Apps Scripts that use MailApp to send emails, and today users of my scripts started getting the error: "Service invoked too many times for one day: email"
When I ran MailApp.getRemainingDailyQuota() it showed -1, confirming that the quota had been exceeded.
When I checked Google Vault to see how many messages my account had sent between yesterday and today, it showed about 3,294.
When I reached out to G Suite Support to ask about this, they directed me to the G Suite Admin Email sending limits page, which shows that G Suite accounts are limited to sending 2,000 messages per rolling 24-hour period.
3,294 is greater than 2,000, but both are well below 50,000, so I'm wondering what actually counts against the 50,000 quota.
What mail-related operation does "read/write" pertain to?
We also use gsuite and Google script to send lots of mails. And also have lots of questions about quotas. The results of my observation is that limits applied with some lag. I can send over limit mails (some amount), before limitation will apply. Amount is vary and bit unpredictable, but almost always if I do sime pause before sending over limit mails - I got quota errors :(
I can't comment, thats why updating answer. Just try to add few mails into bcc :) and check metrics
How can I implement a RateLimit for sending emails via play.mvc.Mailer?
I'm implementing SES for my play framework 1.2.5 application to replace GMAIL. The application runs into situations where it may send out 200 emails at a time (notifications) so I need to RateLimit them to 5 recipients/second. Amazon recommends Google Guava's RateLimit class (http://sesblog.amazon.com/post/TxKR75VKOYDS60/How-to-handle-a-quot-Throttling-Maximum-sending-rate-exceeded-quot-error). I'm currently extending Play's Mailer class. If I try to call my own send method such as sendWithRateLimit(Object...args) I lose the contexts which were acquired using play's LocalvariablesNamesEnhancer class so it doesn't know which template to invoke.
Any help is greatly appreciated.
Too late now, but the way to do this would have been to:
create a model that represents your emails, lets call it Email, save them to the DB
create a job that fetches the next X Email records, and is scheduled to run at the rate allowed
for each Email record, send an email using the mailer
I went down this road for about 6 hours and realized that I was writing way too much code in order to utilize SES. We're going to utilize Mandrill's Heroku add-on instead: https://addons.heroku.com/mandrill