Recently we found that someone has sent out unsolicited emails from our server. This has resulted in the server being blacklisted. I assume this is hackers using forms that have not escaped data correctly, or could it be something else as well?
We have a number of sites with their own 'contact us' type forms. I am going through all the forms and making sure the post data is being escaped. I found one form adding POST data to message without validating it first. I have just added a check before sending the email. Do you think the following will suffice, or is it better practice to escape the email post value before running it through the filter_var?
if(filter_var($_POST['email'], FILTER_VALIDATE_EMAIL))
{
$message = $email." says hello";
$headers = "From: me#example.com";
mail('to#example.com', 'Subject', $message, $headers);
}
Should I be checking the transfer logs for header injection attacks/other attacks, if so what would I be looking for?
The best thing you can do is to check your logs to see who's sending those emails, as you probably aren't gonna check all your users' scripts.
Also, it doesn't make any sense to filter/escape/encode your own forms input (although you should definitely do it) as any of your users can user your smtp server.
Here's how you can trail your mail logs:
tail -f /var/mail/exim_mainlog
tail -f /var/log/exim_mainlog
tail -f /var/log/exim_paniclog
tail -f /var/log/exim_rejectlog
Related
Background
We're building an application that will process & send emails via Mailgun. These are sometimes one-off messages, initiated by a transaction. Some emails, though, will be sent to 30k+ at once.
Eg, a newsletter to all members.
Considerations
Mailgun offers a Batch Sending option with their API. Using "Recipient Variables", you can include dynamic values that are paired with a particular user.
This Batch Sending functionality is limited, however. You cannot send more than 1,000 recipients per request, which means we have to iterate through a recipient list (on our database) for each set of 1,000. Mailgun provides an example of how this might work, using Python (scroll about 2/3 down).
Question
Are there any advantages to batch sending (ie, sending an email to a group of recipients through a single API call, using recipient variables) as opposed to making our own loop, variable substitutions and individual API calls?
I assume this is more taxing on our server, as it would be processing each message itself, instead of just offloading all that data to Mailgun's server for heavy-lifting on their end. But I also like the flexibility & simplicity of handling that on our end and sending a "fully-rendered" message to Mailgun, one at a time, without having to iterate 1k at a time.
Any thoughts on best practices, or considerations we should take into account?
Stumbled onto this today, and felt it provided a pretty good summary/answer for my original question. I wanted to post this as an answer, in case anybody else has this question and hasn't found this Mailgun post. Straight from the horse's mouth, too. The nutshell version:
For PHP, at least, the SDK has a Mailgun class, with a BatchMessage() method. This actually handles the counting of recipients for you, so you can just queue up as many email addresses as you want (ie, more than 1k) and Mailgun will fire off to the API endpoint as needed. Pretty slick!
Here's their original wording, plus a link to the page.
Sending a message with Mailgun PHP SDK + Batch Message:
Batch Message
In addition to Message Builder, we have Batch Message. This class
allows you to build a message and submit a template message in
batches, up to 1,000 recipients per post. The benefit of using this
class is that the recipients tally is monitored and will automatically
submit the message to the endpoint when you've added the 1,000th
recipient. This means you can build your message and begin iterating
through your database. Forget about sending the message, the SDK will
keep track of posting to the API when necessary.
// First, instantiate the SDK with your API credentials and define your domain.
$mgClient = new Mailgun("key-example");
$domain = "example.com";
// Next, instantiate a Message Builder object from the SDK, pass in your sending domain.
$batchMsg = $mgClient->BatchMessage($domain);
// Define the from address.
$batchMsg->setFromAddress("dwight#example.com",
array("first"=>"Dwight", "last" => "Schrute"));
// Define the subject.
$batchMsg->setSubject("Help!");
// Define the body of the message.
$batchMsg->setTextBody("The printer is on fire!");
// Next, let's add a few recipients to the batch job.
$batchMsg->addToRecipient("pam#example.com",
array("first" => "pam", "last" => "Beesly"));
$batchMsg->addToRecipient("jim#example.com",
array("first" => "Jim", "last" => "Halpert"));
$batchMsg->addToRecipient("andy#example.com",
array("first" => "Andy", "last" => "Bernard"));
// ...etc...etc...
// After 1,000 recipeints,
// Batch Message will automatically post your message to the messages endpoint.
// Call finalize() to send any remaining recipients still in the buffer.
$batchMsg->finalize();
The answer of #cdwyer and #nikoshr is very helpful, but bit legacy. Used methods in the example are deprecated. Here is current usage of lib:
$batchMessage = $this->mailgun->messages()->getBatchMessage('mydomain.com');
$batchMessage->setFromAddress('user#domain.com');
$batchMessage->setReplyToAddress('user2#domain.com');
$batchMessage->setSubject('Contact form | Company');
$batchMessage->setHtmlBody('<html>...</html>');
foreach ($recipients as $recipient) {
$batchMessage->addToRecipient($recipient);
}
$batchMessage->finalize();
More info at documentation.
I have a raw email, (MIME multipart), and I want to display this on a website (e.g. in an iframe, with tabs for the HTML part and the plain text part, etc.). Are there any CPAN modules or Template::Toolkit plugins that I can use to help me achieve this?
At the moment, it's looking like I'll have to parse the message with Email::MIME, then iterate over all the parts, and write a handler for all the different mime types.
It's a long shot, but I'm wondering if anyone has done all this already? It's going to be a long and error prone process writing handlers if I attempt it myself.
Thanks for any help.
I actually just dealt with this problem just a few months ago. I added an email feature to the product I work for, both sending and receiving. The first part was sending reminders to users, but we didn't want to manage the bounce backs for our customer admins, we decided to have a message inbox that the admins could see bounces and replies without us, and the admins can deal with adjusting email addresses if they needed to.
Because of this, we accept all email that is sent to an inbox we watch. We use VERP to associate an email with a user, and store the entire email as is in the database. Then, when the admin requests to see the email, we have to parse the email.
My first attempt was very similar to an earlier answer. If one of the parts is html, show it. If it's text, show it. Otherwise, show the original, raw email. This broke down real fast with a few emails not generated by sendmail. Outlook, Exchange, and a few other email systems don't do that, they use multiparts to send the email. After a lot of digging and cussing, I discovered that the problem doesn't appear to be well documented. With the help of looking through MHonArc and reading the RFC's (RFC2045 and RFC2046), I settled on the solution below. I decided on not using MHonArc, since I couldn't easily resuse the parsing and display functionality. I wouldn't say this is perfect, but it's been good enough that we used it.
First, take the message and use Email::MIME to parse it. Then call a function called get_part with the array of parts Email::MIME gives you with ->parts().
get_part, for each part it was passed, decodes the content type, looks it up in a hash, and if it exists, call the function associated with that content type. If the decoder was able to give us something, put it on a result array.
The last piece of the puzzle is this decoder array. Basically, it defines the content types I can deal with:
text/html
text/plain
message/delivery-status, which is actually also plain text
multipart/mixed
multipart/related
multipart/alternative
The non-multipart sections I return as is. With mixed, related and alternative, I merely call get_parts on that MIME node and returns the results. Because alternative is special, it has some extra code after calling get_parts. It will only return html if it has an html part, or it will return only the text part of it has a text part. If it has neither, it won't return anything valid.
The advantage with the hash of valid content types is that I can easily add logic for more parts as needed. And by the time you get_parts is done, you should have an array of all content you care about.
One more item I should mention. As a part of this, we created a separate domain that actually serves these messages. The main domain that an admin works on will refuse to serve the message and redirect the browser to our user content domain. This second domain will only serve user content. This is to help the browser properly sandbox the content away from our main domain. See same origin policy (http://en.wikipedia.org/wiki/Same_origin_policy)
It doesn't sound like a difficult job to me:
use Email::MIME;
my $parsed = Email::MIME->new($message);
my #parts = $parsed->parts; # These will be Email::MIME objects, too.
print <<EOF;
<html><head><title>!</title></head><body>
EOF
for my $part (#parts) {
my $content_type = $parsed->content_type;
if ($content_type eq "text/plain") {
print "<pre>", $part->body (), "</pre>\n";
}
elsif ($content_type eq "text/html") {
print $part->body ();
}
# Handle some more cases here
}
print <<EOF;
</body></html>
EOF
Reuse existing complete software. The MHonArc mail-to-HTML converter has excellent MIME support.
I am using the below code to send an email
#!/usr/bin/perl
sub BEGIN {
unshift (#INC,'/opt/dev/common/mds/perlLib');
}
use Mail::Sender;
$sender = new Mail::Sender
{smtp => 'xxx.xxx.x.xx', from => 'abc#xyz.xom'};
$sender->MailFile({to => 'abc#xyz.xom',
subject => 'Here is the file',
msg => "I'm sending you the list you wanted."});
$sender->Close;
But, it is not sending the mail at all. What is wrong in my code?
I don't use that module because MIME::Entity works so much better, but from working with mail, I can tell you that you are getting ahead of yourself, and making assumptions rather than deductions.
perldoc on that module shows several methods -- and MailFile is for attaching and sending files. If it fails when sending a message without an attached file, I'd have to say I'm not surprised. Has that worked anywhere else?
Were you able to make MailMsg() work? If you cannot, (the syntax is very similar but with fewer things to go wrong), then you may have a problem with connecting to server. From the xxx.xxx.xxx.xxx bit, I'd have to assume you're using an ip address. Try a hostname. Also, set on_errors to die() or maybe warn(), and see if you can trap the error. Most mail attempts fail during the connect to the server/creation of the object -- then the sending attempt will by definition fail, but often without telling you why.
If you cannot connect, make sure that you aren't needing to authenticate to your server.
I have a simple email sender for user account activation. Depending on which email address I use, I get significantly different response times: University email - 1 minute, Gmail - 3-4 hours, Yahoo - 1 or 2 days -- which seems bizarre. Has anyone else seen this phenomenon?
EDIT:
There weren't many responses (even for a bounty), but I'll try to explain my problem more clearly.
This probably isn't greylsting -- If I so a simple:
php mail ($to, $subject, $body) // this delivers instantly.
My cakephp code:
function __sendActivationEmail($id) {
$User = $this->User->read ( null, $id );
$this->set ( 'suffix_url', $User ['User'] ['id'] . '/' . $this->User->getActivationHash () );
$this->set ( 'username', $User ['User'] ['username'] );
$this->Email->to = $User ['User'] ['email'];
$this->Email->subject = 'Test.com - ' . __ ( 'please confirm your email address', true );
$this->Email->from = 'noreply#test.com';
$this->Email->template = 'user_confirm';
$this->Email->sendAs = 'text';
$this->Email->delivery = 'mail';
$this->Email->send ();
}
Causes delays from 13 minutes (ok; we'll deal with it) to 5-6 hours (less okay, since this is an activation email). For some of my users, it works instantly, but for other users (of the same service provider, i.e., gmail, it sees these delays).
Any clues?
The code looks fine, but it of course doesn't tell anything about the mail server's configuration.
3-4 hours I would put down to Greylisting, but 1-2 days is definitely too much. Is this reproducible? How many addresses have you tried this with?
What do the full headers of the (received) mails look like? The "received from: .... "path should tell you at which point it took 1-2 days to deliver.
Maybe you can install PHPMailer as a Vendor and create a Component called "Mail"...
And don't forget to authenticate with your SMTP server! :)
Ignore the whole PHP element of it for a moment.
If its a linux server for example, send a mail from the command line e.g. mail myemail#me.com
see if the same thing is happening that way. Its quite likely its a server configuration issue not a php or cakePHP issue.
Look up a few basics like having a FQDN and maybe look into setting up SPF records for your email. Make sure the emails are coming from your domain name not someone elses e.g. not the users email.
Also check if you have email spam software set up that could be grey listing you email on the way out (unlikely but possible). the mostly like thing is the destination spam filter is delaying it. Try send to a gmail account and see if it gets through fine or goes into spam.
Do all this without touching PHP, if all is going fine there then set up a basic php script to do a basic email not using CakePHP, if that works fine then you know its CakePHP etc but I doubt it.
So after further digging, I realized that it was our server host's problem. We use Slicehost, and it just so happens that a range of ips that had been blacklisted included our own ip. We got our name off the list, and we're good to go.
I have a bot that replies to users. But sometimes when my bot sends its reply, the user or their email provider will auto-respond (vacation message, bounce message, error from mailer-daemon, etc). That is then a new message from the user (so my bot thinks) that it in turn replies to. Mail loop!
I'd like my bot to only reply to real emails from real humans. I'm currently filtering out email that admits to being bulk precedence or from a mailing list or has the Auto-Submitted header equal to "auto-replied" or "auto-generated" (see code below). But I imagine there's a more comprehensive or standard way to deal with this. (I'm happy to see solutions in other languages besides Perl.)
NB: Remember to have your own bot declare that it is autoresponding! Include
Auto-Submitted: auto-reply
in the header of your bot's email.
My original code for avoiding mail loops follows. Only reply if realmail returns true.
sub realmail {
my($email) = #_;
$email =~ /\nSubject\:\s*([^\n]*)\n/s;
my $subject = $1;
$email =~ /\nPrecedence\:\s*([^\n]*)\n/s;
my $precedence = $1;
$email =~ /\nAuto-Submitted\:\s*([^\n]*)\n/s;
my $autosub = $1;
return !($precedence =~ /bulk|list|junk/i ||
$autosub =~ /(auto\-replied|auto\-generated)/i ||
$subject =~ /^undelivered mail returned to sender$/i
);
}
(The Subject check is surely unnecessary; I just added these checks one at a time as problems arose and the above now seems to work so I don't want to touch it unless there's something definitively better.)
RFC 3834 provides some guidance for what you should do, but here are some concrete guidelines:
Set your envelope sender to a different email address than your auto-responder so bounces don't feed back into the system.
I always store in a database a key of when an email response was sent from a specific address to another address. Under no circumstance will I ever respond to the same address more than once in a 10 minute period. This alone stopped all loops, but doesn't ensure nice behavior (auto-responses to mailing lists are annoying).
Make sure you add any permutation of header that other people are matching on to stop loops. Here's the list I use:
X-Loop: autoresponder
Auto-Submitted: auto-replied
Precedence: bulk (autoreply)
Here are some header regex's I use to avoid loops and to try to play nice:
/^precedence:\s+(?:bulk|list|junk)/i
/^X-(?:Loop|Mailing-List|BeenThere|Mailman)/i
/^List-/i
/^Auto-Submitted:/i
/^Resent-/i
I also avoid responding if any of these are the envelop senders:
if ($sender eq ""
|| $sender =~ /^(?:request|owner|admin|bounce|bounces)-|-(?:request|owner|admin|bounce|bounces)\#|^(?:mailer-daemon|postmaster|daemon|majordomo|ma
ilman|bounce)\#|(?:listserv|listsrv)/i) {
That really sounds like something that's probably available as a module from CPAN, but I didn't find anything clearly relevant in five minutes of searching. Mail::Lite::Mbox::Processor looks like it might do what you want:
Mail::Lite::Message::Matcher is a
framework for automated mail
processing. For example you have a
mail server and you have a need to
process some types of incoming mail
messages automatically. For example,
you can extract automated
notifications, invoices, alerts etc.
from your mail flow and perform some
tasks based on content of those
messages.
but its docs are sparse enough that it isn't immediately obvious whether it provides those example functions itself or if you have to provide the code to drive them.
In any case, though, if you haven't already checked CPAN, that's where I would start if I wanted to do something like this.
My answer here only deals with bounces which is more straightforward.
Using DSN (Delivery Status Notification) identifier will help you detect a DSN/bounced message. It should go to Return-Path and not Reply-To.
Here's a sample of a typical DSN message. The header information includes the message id, content type has specific values (delivery-status) etc.
Not able to provide you any codes in perl, just my 2 cents of idea.
PS: Do note that not all mail servers or MTA conforms to this, but I guess most do.
There should be a standard way of dealing with this, but the problem is that you'd have to assume that systems that send auto-replies comply to that standard, when most the time, they just don't.
How do you get the address that you reply to? I hope you aren't using the From: header. Check the Reply-to: header first and if that doesn't exist, use the Return-path:.
But whatever you do, you will simply have to keep a log of what you sent to whom and throttle your bot to some sensible value of messages per time.