I host a domain for a client; the domain's A records point to my IP address. When email arrives addressed to someone in his domain, he just wants to forward it to his ISP (charter.net). This works about 90-99% of the time, but a few times a day charter.net rejects the message with 'Service not available'. We tried forwarding to his gmail account instead, and same thing -- it usually works but sometimes gmail returns 'Service not available'. When this happens, my sendmail apparently gives up.
My suspicion is it is not the recipient's server uptime that is at issue; it rejects email that it sees as suspicious.
I have recently added SPF and DKIM records for my server's canonical name but I still get 'Service not available' from charter.net. Now sendmail DKIM-signs any mail that originates on my server, but it is not signing email that it forwards. I have not found a way to configure opendkim to do that. But I have seen that mandrillapp.com resigns email that it forwards; the headers include a DKIM-signature with d=originaldomain.com, and a 2nd DKIM-signature with d=mandrillapp.com.
So I guess my questions are,
1) Does anyone really know why my server gets 'Service not available' from ultimate recipient?
2) Can I configure opendkim to sign email that my server forwards?
3) Might it do any good to set up SPF and/or DKIM records for my client's domain?
Thanks,
Bob
ADDENDUM:
For forwarding, I have entries like this in /etc/mail/virtusertable:
a#clientdomain.com a#charter.net
For DKIM, I have this in my sendmail.mc:
INPUT_MAIL_FILTER(`opendkim', `S=inet:8891#127.0.0.1')
I also have clientdomain.com in /etc/mail/local-host-names, and as I said above clientdomain.com's A records resolve to the same IP as mydomain.com.
Classic sendmail redirects via aliases, ~/.forward file and virtusertable keep unchanged original envelope address (It is used in MAIL FROM command during SMTP session). It may make your forwards break restrictions set by SPF record of the original sender domain.
Possible fixes for "many addresses" redirect:
SRS
custom sendmail.cf fixes
For more details ask at more suited siter site serverfault.com.
Related
Is it a problem if 127.0.0.1 appears in email headers?
Example: Received: from baobabsmail.baobab.fi ([127.0.0.1])
I ask because emails sent from my server to #outlook.com addresses end up in the spam folder and this is the last thing I can think of. I have properly configured HELO, DKIM, Reverse DNS, SenderID, SPF and DMARC. I don't send out mass emails. My IP is from AWS, but it isn't on any publicly available blacklists. I have verified that everything is set up correctly using DKIMvalidator, MxToolBox and mail-tester.
Edit: for what it's worth, I finally got rid of the 127.0.0.1's in my headers and it did not resolve the issue for me.
Unfortunately, it depends...
Mail systems vary in how they are configured, and it is perfectly legitimate for an MUA (e.g. Thunderbird) to send outgoing mail to an MTA / mail server (e.g. exim) running on the same machine using the localhost address. Unusual, these days, but not "bad by definition".
When you say 'end up in the spam folder', what is it that puts it there: are you using a local mail server? if so is it that server that junks the mail (on send) or outlook.com itself (on receipt). Either way, what error messages or other failure information have you found?
Some random thoughts:
DKIM is a pain to set up correctly. Try disabling it entirely and see if that changes things in interesting ways.
Ditto DMARC.
Have you got SPF set up separately? If so, disable SPF and retry.
Is IPv6 involved in the mix at all? Various things are subtly different if so.
If outlook.com were to do sender verify callbacks (i.e. on receipt, check that mail from address was an acceptable recipient to your server) would it pass?
Is your email system sending RFC-conformant mail: that is, does it have a From: address, To: or Sender: address, Message-ID:, Date: headers and, if using MIME, Content-* headers (and probably a couple I forget!).
If changing DKIM / DMARC / SPF changes things (and remembering DNS timeouts, leave it a while between attempts), re-add SPF first - it is the simplest to get right.
127.0.0.1 can be flagged by Spam filters because it fails to provide an identity trace of the sender. Most common e-mail systems will show the IP address or the host name. The next item will be the recipient e-mail server.
For example:
Received: from [127.0.0.1] (81.27.148.196) by TAE1.agent.com.pk
What is funny about this one is that the top-level domain says it is received initially by a domain in Pakistan, but the IP address is registered to an entity in St. Petersburg, Russia.
I'm implementing an application in ruby on rails (although this is a minor detail) that needs to send emails. Everything works fine, but I'm surprised cause when the app sends a mail, as long as it's authenticated against the smtp server, the sender it's going to be whatever it's specified in mail from. My point is, I can authenticate in a smtp server with "myuser#mydomain.com" and "mypassword" and be able to send a mail in which the recipient would see whichever address I want as sender.
This doesn't make any sense for me, and I'm sure it can't be so easy. I know it's surely a completely silly question for most of you, but would really appreciate anyone could make it clear for me.
Cheers!!
You authenticate as a generic user, it is only an accident that your login name is similar to a mail address. And yes, usually you can send mail with any reverse-path (reverse-path is the address in the MAIL FROM SMTP command). This may be necessary, for example a list manager uses VERP and sends mails with many unique reverse-paths.
However, some mail servers and mail administrators do place restrictions on the reverse path and the address in the From header. Especially, if the server signs the message.
Moreover, if the receiving server checks SPF records, it is useless to impersonate a remote domain, even if it is allowed, because the mail will be rejected anyway (if the domain publishes an SPF DNS record).
Need to implement the email validation to find if the email really exists. After some search found that the this is not possible due to the spam control techniques used (e.g. grey listing).
So moved with MX and A attribute check (as we are not allowed to send confirmation mails directly):
DirContext ictx = new InitialDirContext();
Attributes attrs = ictx.getAttributes("gmail.com",
new String[] {"MX"});
I am getting the results but not sure how the dns lookup works here (the complete background of the request and response i.e how the request is forwarded to the mail exchange server and also which port is used for the contacting the Mail Exchange server)
This will immensely help in moving into the production env without any doubts about the network behavior as we are having strick firewall restrictions.
Production Env : Weblogic Server on Solaris OS.
Similar dicussion is at Email validation MX Lookup but it doesnt have enough detail
It sounds like you're confusing two different techniques:
1) Perform a DNS lookup to ensure that an MX record exists for the domain. This will eliminate all email addresses at fake (i.e., not registered) domains. Note however that having an MX record is not a requirement for legitimate mail reception. In the absence of an MX record, the mail software will simply use the A record for the base domain.
2) Make a TCP connection to the recipient mail server to see if anything is listening. This is always port 25. You don't have to send a test message, you can just connect to the port and then immediately disconnect. A successful connection implies a valid email address. (Or at least, a valid domain, which is as much as you can verify.)
When developing an application that sends out notification email messages, what are the best practices for
not getting flagged as a spammer by your hosting company. (Cover any of:)
best technique for not flooding a mail server
best mail server products, if you were to set up your own
sending messages as if from a specific user but still clearly from your application (to ensure complaints, etc come back to you) without breaking good email etiquette
any other lessons learned
not getting flagged as spam by the receiver's client? (Cover any of:)
configuring and using sender-id, domain-keys, SPF, reverse-dns, etc to make sure your emails are properly identified
best SMTP header techniques to avoid getting flagged as spam when sending emails for users (for example, using Sender and From headers together)
any other lessons learned
An additional requirement: this application would be sending a single message to a single recipient based upon an event. So, techniques for sending the same messages to multiple recipients will not apply.
best technique for not flooding a mail server
not a lot you can do about this beyond checking with your mail server admin (if it's a shared hosting account / not in your control). but if the requirement is one email to a single recipient per event, that shouldn't be too much of an issue. the things that tend to clog mail systems are emails with hundreds (or more) of recipients.
if you have events firing off all the time, perhaps consider consolidating them and having an email sent that summarizes them periodically.
sending messages as if from a specific user but still clearly from your application (to ensure complaints, etc come back to you) without breaking good email etiquette
you can accomplish this by using the "Reply-To" header, which will then have clients use that address instead of the From address when an email message is being composed.
you should also set the "Return-Path" header of any email, as email without this will often get filtered off.
ex.
From: me#me.com
Return-Path: me#me.com
Reply-To: auto#myapp.com
configuring and using sender-id, domain-keys, SPF, reverse-dns, etc to make sure your emails are properly identified
this is all highly dependent on how much ownership you have of your mail and DNS servers. spf/sender-id etc... are all DNS issues, so you would need to have access to DNS.
in your example this could present quite the problem. as you are setting mail to be from a specific user, that user would have to have SPF (for example) set in their DNS to allow your mail server as a valid sender. you can imagine how messy (if not outright impossible) this would get with a number of users with various domain names.
as for reverse DNS and the like, it really depends. most client ISP's, etc... will just check to see that reverse DNS is set. (ie, 1.2.3.4 resolves to host.here.domain.com, even if host.here.domain.com doesn't resolve back to 1.2.3.4). this is due to the amount of shared hosting out there (where mail servers will often report themselves as the client's domain name, and not the real mail server).
there are a few stringent networks that require matching reverse DNS, but this requires that you have control over the mail server if it doesn't match in the first place.
if you can be a bit more specific i may be able to provide a bit more advice, but generally, for people who need to send application mail, and don't have a pile of control over their environment, i'd suggest the following:
make sure to set a "Return-Path"
it's nice to add your app and abuse info as well in headers ie: "X-Mailer" and "X-Abuse-To" (these are custom headers, for informational purposes only really)
make sure reverse DNS is set for the IP address of your outgoing mail server
first a quick correction to the previous
return-path: is a header added by recieving system based on the envelope-sender of the incomming message
for spf to work the return-path/envelope-sender needs to be yourapp#yourdomain.com
and ensure the spf record for yourdomain.com {or if per-user spf} for yourapp#yourdomain.com allows mails to originate on the server that hosts the app/sends the email
this envelope-sender is the address that will recieve all bounces/errors
now sender-id is different entirely it checks the return-path/envelope-sender
and the
from: address {stored inside the message}
if sending
from: hisname yourapp#yourdomain.com
reply-to: hisname hisaddres#hisdomain.com
this will be a non-issue
if sending
from: hisname hisaddres#hisdomain.com
it will be and you must add a
Resent-From: hisname yourapp#yourdomain.com
as this specifies to ignore the from: for sender-id checks use this instead as it has been sent by you on his behalf
now for the other bits that are worthwhile
ip's mentioned are your mailservers
a have your ip's ptr point to a name that also resolves to the same ip
FQDNS
b have your server helo/ehlo with whatever.domain.com where domain.com is the same as the domain of the name in step A {not the same name for resons below}
c have that helo/ehlo servername also resolve to the ip of your server
d add the following spf record to that helo/ehlo name "v=spf1 a -all"
{meaning allow helo/ehlo with this name from ip's this name points to only}
e add the following sender-id lines to the helo/ehlo name {purely for completeness
"spf2.0/mfrom,pra -all" {ie there are no users#this-domain}
f add the following spf to the FQDNS-name and any other hostnames for your server
"v=spf1 -all" {ie no machines will ever helo/ehlo as this name and no users#this-domain}
{as the fqdns name can be determined by bots/infections its better to never allow this name to be used in helo/ehlo greetings directly it is enough that it be from the same domain as the helo/ehlo identity to prove the validity of both}
For the past few weeks, I've been trying to learn about just how email works. I understand the process of a client receiving mail from a server using POP pretty well. I also understand how a client computer can use SMTP to ask an SMTP server to send a message. However, I'm still missing something...
The way I understand it, outgoing mail has to make three trips:
Client (gmail user using Thunderbird) to a server (Gmail)
First server (Gmail) to second server (Hotmail)
Second server (Hotmail) to second client (hotmail user using OS X Mail)
As I understand it, step one uses SMTP for the client to communicate. The client authenticates itself somehow (say, with USER and PASS), and then sends a message to the gmail server.
However, I don't understand how gmail server transfers the message to the hotmail server.
For step three, I'm pretty sure, the hotmail server uses POP to send the message to the hotmail client (using authentication, again).
So, the big question is: when I click send Mail sends my message to my gmail server, how does my gmail server forward the message to, say, a hotmail server so my friend can recieve it?
Thank you so much!
~Jason
Thanks, that's been helpful so far.
As I understand it, the first client sends the message to the first server using SMTP, often to an address such as smtp.mail.SOMESERVER.com on port 25 (usually).
Then, SOMESERVER uses SMTP again to send the message to RECEIVESERVER.com on port 25 (not smtp.mail.RECEIVESERVER.com or anything fancy).
Then, when the recipient asks RECEIVESERVER for its mail, using POP, s/he recieves the message... right?
Thanks again (especially to dr-jan),
Jason
The SMTP server at Gmail (which accepted the message from Thunderbird) will route the message to the final recipient.
It does this by using DNS to find the MX (mail exchanger) record for the domain name part of the destination email address (hotmail.com in this example). The DNS server will return an IP address which the message should be sent to. The server at the destination IP address will hopefully be running SMTP (on the standard port 25) so it can receive the incoming messages.
Once the message has been received by the hotmail server, it is stored until the appropriate user logs in and retrieves their messages using POP (or IMAP).
Jason - to answer your follow up...
Then, SOMESERVER uses SMTP again to send the message to RECEIVESERVER.com on port 25 (not smtp.mail.RECEIVESERVER.com or anything fancy).
That's correct - the domain name to send to is taken as everything after the '#' in the email address of the recipient. Often, RECEIVESERVER.com is an alias for something more specific, say something like incoming.RECEIVESERVER.com, (or, indeed, smtp.mail.RECEIVESERVER.com).
You can use nslookup to query your local DNS servers (this works in Linux and in a Windows cmd window):
nslookup
> set type=mx
> stackoverflow.com
Server: 158.155.25.16
Address: 158.155.25.16#53
Non-authoritative answer:
stackoverflow.com mail exchanger = 10 aspmx.l.google.com.
stackoverflow.com mail exchanger = 20 alt1.aspmx.l.google.com.
stackoverflow.com mail exchanger = 30 alt2.aspmx.l.google.com.
stackoverflow.com mail exchanger = 40 aspmx2.googlemail.com.
stackoverflow.com mail exchanger = 50 aspmx3.googlemail.com.
Authoritative answers can be found from:
aspmx.l.google.com internet address = 64.233.183.114
aspmx.l.google.com internet address = 64.233.183.27
>
This shows us that email to anyone at stackoverflow.com should be sent to one of the gmail servers shown above.
The Wikipedia article mentioned (http://en.wikipedia.org/wiki/Mx_record) discusses the priority numbers shown above (10, 20, ..., 50).
You're looking for the Mail Transfer Agent, Wikipedia has a nice article on the topic.
Within Internet message handling services (MHS), a message transfer agent or mail transfer agent (MTA) or mail relay is software that transfers electronic mail messages from one computer to another using a client–server application architecture. An MTA implements both the client (sending) and server (receiving) portions of the Simple Mail Transfer Protocol.
The terms mail server, mail exchanger, and MX host may also refer to a computer performing the MTA function. The Domain Name System (DNS) associates a mail server to a domain with mail exchanger (MX) resource records containing the domain name of a host providing MTA services.
You might also be interested to know why the GMail to HotMail link uses SMTP, just like your Thunderbird client. In other words, since your client can send email via SMTP, and it can use DNS to get the MX record for hotmail.com, why doesn't it just send it there directly, skipping gmail.com altogether?
There are a couple of reasons, some historical and some for security. In the original question, it was assumed that your Thunderbird client logs in with a user name and password. This is often not the case. SMTP doesn't actually require a login to send a mail. And SMTP has no way to tell who's really sending the mail. Thus, spam was born!
There are, unfortunately, still many SMTP servers out there that allow anyone and everyone to connect and send mail, trusting blindly that the sender is who they claim to be. These servers are called "open relays" and are routinely black-listed by smarter administrators of other mail servers, because of the spam they churn out.
Responsible SMTP server admins set up their server to accept mail for delivery only in special cases 1) the mail is coming from "its own" network, or 2) the mail is being sent to "its own" network, or 3) the user presents credentials that identifies him as a trusted sender. Case #1 is probably what happens when you send mail from work; your machine is on the trusted network, so you can send mail to anyone. A lot of corporate mail servers still don't require authentication, so you can impersonate anyone in your office. Fun! Case #2 is when someone sends you mail. And case #3 is probably what happens with your GMail example. You're not coming from a trusted network, you’re just out on the Internet with the spammers. But by using a password, you can prove to GMail that you are who you say you are.
The historical aspect is that in the old days, the link between gmail and hotmail was likely to be intermittent. By queuing your mail up at a local server, you could wash your hands of it, knowing that when a link was established, the local server could transfer your messages to the remote server, which would hold the message until the recipient's agent picked it up.
The first server will look at DNS for a MX record of Hotmail server. MX is a special record that defines a mail server for a certain domain. Knowing IP address of Hotmail server, GMail server will sent the message using SMTP protocol and will wait for an answer. If Hotmail server goes down, GMail server wiil try to resend the message (it will depend on server software configuration). If the process terminates ok, then ok, if not, GMail server will notify you that he wasn´t able to deliver the message.
If you really want to know how email works you could read the SMTP RFC or the POP3 RFC.
All emails are transferred using SMTP (or ESMTP).
The important thing to understand is that the when you send message to someguy#hotmail.com this message's destination is not his PC. The destination is someguy's inbox folder at hotmail.com server.
After the message arrives at it's destination. The user can check if he has any new messages on his account at hotmail server and retrieve them using POP3
Also it would be possible to send the message without using gmail server, by sending it directly from your PC to hotmail using SMTP.
Step 2 to 3 (i.e. Gmail to Hotmail) would normally happen through SMTP (or ESMTP - extended SMTP).
Hotmail doesn't send anything to a client via POP3. It's important to understand some of the nuances here. The client contacts Hotmail via POP3 and requests its mail. (i.e. the client initiates the discussion).