When I send an email to somebody, (I think) my email is sent to my home server, then the email is sent to target server, then it is finally sent to the people I want to send to.
Sometimes, this needs multiple hops, which is the point that confuses me. Why does this need multiple hops? Why can't the email be sent to the target server directly? XMPP doesn't need multiple hops, for example.
A typical email transaction uses an outbound mail server, which looks up your recipient's incoming server, delivers it there, then that server hands it off to your recipient's mailbox server. That's 3 hops:
When you send a message, you connect from your email client (more properly: mail user agent, MUA) to your outbound SMTP server. This is the first hop, and your SMTP server creates a Received header denoting it for auditing later on. This will include the IP it sees you connecting on, a message ID internal to that specific server, and sometimes some other information.
Your outbound SMTP server then looks up the MX (mail exchange) record for each recipient's domain in DNS and connects to the highest priority relay it can find (the numerically lowest MX priority) for them. This is the second hop. As with any hop, a Received header is added.
The MX relay typically does anti-spam filtering, potentially rejecting the message during the SMTP connection. This is what generates a bounce message, which your outbound SMTP server sends you directly (if it's sent across the internet, it risks being backscatter, a form of spam, because the bounce cannot be proven to be legitimate). Often, anti-spam filtering is done later and the message is dropped silently or else delivered with markup.
The MX relay then hands the message off internally to your mailbox (IMAP/Exchange) server (hop #3). Sometimes, local mailbox server filters are used to handle patterns (or markup from the MX server) and facilitate various actions such as filing suspected spam into your Junk folder.
If you use a mail forwarding service, the MX relay instead passes your message to the forwarding target (another MX relay). This can theoretically go on for quite a while or even loop infinitely, but it should eventually end in a mailbox server.
See also the formal SMTP specification, RFC 5321.
If you use webmail, the web mail server is your email client. Some web mail servers add headers resembling Received headers to track the connection so spam filters can extend their reputation checks to the user rather than to the webmail system, which could be characterized as yet another hop.
In enterprise-grade environments, it is common for multiple layers of servers to be used for various purposes. These extra hops happen between the MX relay and the mailbox server and can be related to extra anti-spam or anti-virus handling, advanced mail aliasing, email archiving, internal mail routing between campuses, or any number of other things. These can for example ease transitions from one mail system to another (like migrating between versions of Exchange, which may require lots of testing and a slow migration so a problem doesn't affect the whole company).
Fundamentally that has to do with the original purpose of the Internet, or rather it's predecessor ARPANET, was designed to survive the loss of portions of the network (popular to popular misconception, this design goal existed not in order to survive nuclear attack, but because network components were rather unreliable at the time).
The packet switching network is designed to find one or more routes from where the packet starts (your email client) to where it needs to be (the email client of the person you are sending to).
In order to achieve that flexibility, the system is designed not just to route directly from point A to point B, but to find an available route (from a possibly very large set of possible routes) that is currently functioning.
If your email client had to know how to directly send to every other device in the world, without intervening subnets managing traffic, your computer would need a direct connection to every other computer in the world. That would be prohibitively expensive. Instead, routers inbetween accept packets from you and other people in your area and figure out how to route each packet closer to its destination. That's similar to the method that the airlines use to have regional flights feed hubs.
When a packet is being routed through the internet it will make multiple hops to get from source to destination.
Related
A standard email delivery uses at least 2 SMTP connections:
From the sender to the sender's MTA
From the sender's MTA to the receiver's MTA
and requires at least two SMTP servers, the first one being a relay.
Instead of this routing / relay infrastructure, the client could directly send its messages to the receiver's MTA, e.g. bypassing the relay step.
I get that there are some advantages, such as :
authentication delegation and greylisting: The receiver can trust the sender's MTA to authenticate the sender, so it is not necessary to be able to authenticate the sender.
advantages of the client-server architecture: the client's MTA being a server, it can provide easily more functions (be more easily secured etc...) than a client application. For example, back when being connected to the Internet was costly, the MTA server could mutualise the Internet connection.
But it seems these are only advantages, and not a design constraint today (embedding a library on the client is not a problem anymore).
Are there some reasons that require using a client MTA today, or is it because of the technological inertia of these protocols (e.g. the difficulty to change a standard that is well established and implemented)?
There are several compelling reasons to use a client-side MTA. Here are two:
1) The sending MTA does the DNS queries to figure out which mail server it needs to contact to reach the receiver's MTA.
2) If the sending MTA can't reach the receiver's mail server, it can retry delivery later on. Or, it could fall back to using one of the receiving domain's higher-distance MX records (if any exist).
You could always write this functionality in your mail client, but if you do that, you are well on your way to making your mail client an MTA.
There are other reasons. For example, if the sending MTA handles mail for multiple senders, it can batch emails: if it has multiple emails destined for gmail addresses, for example, it can send them all over the same connection. That way it doesn't have to open and shut down a connection for every single email. This becomes even more advantageous if it needs to negotiate TLS and/or AUTH.
Your suggestion is far too complicated for most users in "spam era".
It may be worth the effort required for some but sadly mostly for spammers.
Sending via "sender MTA/SMTP server" grossly simplifies email sending, both troubleshooting by sender and email client implementation.
Am developing and newsletter email software in vb.net for a friend, and he has asked me to add option to send email using dns servers,
How do i send an emai using dns server without smtp or imap?
Send email "using DNS servers" still uses SMTP. And it's generally a bad idea. Here's what it's about:
Usually, an email app (your own, Outlook, etc.) send mail using SMTP or similar protocol to a configured mail server. It could be a company's own server, or an ISP's server, or something like that. The mail may pass through various outgoing mail servers on its way out of an organisation, but at some point it hits a "boundary server" (There are pretty names for all these different types of server, but they're not important right now.)
This boundary server uses DNS to find the recipient's mail server and then sends the message (using SMTP) to that server. (I have omitted the details on purpose)
Sending email "using DNS" means that your app acts as a boundary server. It uses DNS to find the recipient's mail server and then sends the mail directly - rather than using any outgoing servers.
It's a bad idea for two reasons.
More and more ISPs block SMTP traffic that doesn't pass through their outgoing servers
More and more incoming servers validate the boundary server using various techniques. Your app is not likely to be accepted by these servers
Both of these issues are anti-spam countermeasures, so using this technique makes you look like a spammer, leaves many of your messages blocked, and is likely to get your IP address blacklisted.
Hopefully you can explain this to your friend so that they will understand that the "normal" way of sending mail is the only way to ensure that they don't get listed as a spammer.
Need to implement the email validation to find if the email really exists. After some search found that the this is not possible due to the spam control techniques used (e.g. grey listing).
So moved with MX and A attribute check (as we are not allowed to send confirmation mails directly):
DirContext ictx = new InitialDirContext();
Attributes attrs = ictx.getAttributes("gmail.com",
new String[] {"MX"});
I am getting the results but not sure how the dns lookup works here (the complete background of the request and response i.e how the request is forwarded to the mail exchange server and also which port is used for the contacting the Mail Exchange server)
This will immensely help in moving into the production env without any doubts about the network behavior as we are having strick firewall restrictions.
Production Env : Weblogic Server on Solaris OS.
Similar dicussion is at Email validation MX Lookup but it doesnt have enough detail
It sounds like you're confusing two different techniques:
1) Perform a DNS lookup to ensure that an MX record exists for the domain. This will eliminate all email addresses at fake (i.e., not registered) domains. Note however that having an MX record is not a requirement for legitimate mail reception. In the absence of an MX record, the mail software will simply use the A record for the base domain.
2) Make a TCP connection to the recipient mail server to see if anything is listening. This is always port 25. You don't have to send a test message, you can just connect to the port and then immediately disconnect. A successful connection implies a valid email address. (Or at least, a valid domain, which is as much as you can verify.)
We have set up a system where notifications get sent to a user with the following From address format: user-{0}#aol.com (replace {0} with an ID)
This way we can track what user we sent the message to originally. This format is not likely to change for various reasons. The issue we are running into is this: every email we send out with a dynamic address is creating a new SMTP relay.
My knowledge of relays is very limited. Our host (GoDaddy) limits SMTP relays to 250 per day. This is an application that potentially will be sending out thousands of emails per day.
Is my assumption that every 'unregistered' email address we use opens a new relay, correct? If so, are there any services or hosts that provide unlimited relays for a reasonable price?
Thanks.
so my assumption that every 'unregistered' email address we use opens a new relay, correct?
Relay is basically a classification. every email sent via relay opens an SMTP connection on the mail server and not "a new relay"...that said, It might be cheaper, if you can, to set up your own sending SMPT server. It comes as a component of IIS. If you need any additional information on setting this up, let me know.
Godaddy, yes, due to the different plans. You can try any other email providers like google and yahoo as well. Just make sure to setup a "catch" all mailbox for undelivered mails and any responses.
What kind of practical issues are there concerning sending tons of e-mail from a server? Will the likelihood of that e-mail being received be just the same as if it had been sent from g-mail or a personal e-mail account if I for example just blindly call the mail() function in PHP tens of thousands of times a day?
(note: you are not helping a spammer here, this relates to a notify feature I'm thinking about for a future link sharing site)
While you may technically be able to send thousands of mails per minute, in reality you must be carefull.
Say you send out 500 emails to yahoo for example. if enough people mark your message as spam, soon, ANY email you send to yahoo will be marked as spam, or [BULK]. Many isp's routinely tar-pit or outright reject email from servers on lists such as RBL (the real-time black hole list). If your mail IP gets put on one of these lists, you can kiss sending email normally from that ip ever again goodbye. Users are very finicky and it doesn't take many complaints to get your mail ip blocked at many domains.
Also since you are sending automated messages, there are heuristics used to determine if the same message is being sent to many users on the same domain. This also increases the chance your mail will be marked as spam.
This is why clean emails from some addresses always go into the spam box. Their company may have not been careful enough when sending what could be perceived as spam. Proceed with caution.
http://wiki.apache.org/spamassassin/AvoidingFpsForSenders
http://support.microsoft.com/kb/842851
http://www.blacklistedip.com/rbl_list.php
It helps to set a 'x-mailer' and ('X-MimeOLE' if your pretending to be outlook) of a real mail client.
It also helps to send it from a server that is a mail server for the domain in the from address, with forward & reverse DNS records setup.
No issues. Once a server is correctly configured as a mail server (SMTP) for a particular domain, there is no difference if the mail it sends out came to it from Outlook, or from the mail() function in PHP - both are getting the SMTP server to do all the heavy lifting
I always make sure to set my X-Mailer headers correctly (identifying that the message was sent from within PHP) to ensure that any overzealous anti-spam services recognize it as an automatated notification as opposed to bulk/junk email. e.g.
$headers .= "X-Mailer: PHP/".phpversion();
All the configuration and limits you'll encounter are with the SMTP server, not from PHP. You can configure SMTP to rate-limit to 2 messages per second for example, this means if you queue up 1,200 messages they'll be drip fed out over the next hour rather than all at once (two is a really low number, 5-25 is more realistic).
SMTP is the backbone of email and some SMTP servers can happily handle tens of thousands of messages per minute (or more!) - the only limitation you'll likely face is bandwdith ;)
Check with your hosting provider, especially if you're on shared hosting. For example: GoDaddy limits shared hosting accounts to sending 1000 emails per day on their server (http://support.godaddy.com/groups/web-hosting/forum/topic/e-mail-sending-limit/). I'm sure other providers have their own limits (I believe the provider one of the companies I worked for used limited outgoing emails to 250 per minute or something along those lines).
Edit: In my case the solution was to contact our hosting provider. They provided info to route outgoing emails through a server they had dedicated to sending outgoing emails. Solved the problem right away.