Need to implement the email validation to find if the email really exists. After some search found that the this is not possible due to the spam control techniques used (e.g. grey listing).
So moved with MX and A attribute check (as we are not allowed to send confirmation mails directly):
DirContext ictx = new InitialDirContext();
Attributes attrs = ictx.getAttributes("gmail.com",
new String[] {"MX"});
I am getting the results but not sure how the dns lookup works here (the complete background of the request and response i.e how the request is forwarded to the mail exchange server and also which port is used for the contacting the Mail Exchange server)
This will immensely help in moving into the production env without any doubts about the network behavior as we are having strick firewall restrictions.
Production Env : Weblogic Server on Solaris OS.
Similar dicussion is at Email validation MX Lookup but it doesnt have enough detail
It sounds like you're confusing two different techniques:
1) Perform a DNS lookup to ensure that an MX record exists for the domain. This will eliminate all email addresses at fake (i.e., not registered) domains. Note however that having an MX record is not a requirement for legitimate mail reception. In the absence of an MX record, the mail software will simply use the A record for the base domain.
2) Make a TCP connection to the recipient mail server to see if anything is listening. This is always port 25. You don't have to send a test message, you can just connect to the port and then immediately disconnect. A successful connection implies a valid email address. (Or at least, a valid domain, which is as much as you can verify.)
Related
I am trying to get the SMTP of an email from the MX records. For instance, nslookup if I type in outlook.com what I get is outlook-com.olc.protection.outlook.com but actually the SMTP is smtp-mail.outlook.com. How can I get this record smtp-mail.outlook.com from the MX records rather than google for each domain what their SMTP is?
Even for 163.com the mx record I get is 163mx03.mxmail.netease.com now I have to google what is the smtp for netease.com and I figured out from internet that it is smtp.ym.163.com.
Is there a simpler or a standardised procedure to get this output? I even tried sending a mail from that email and in email header also there is no SMTP provided like this.
In any mail delivery from a#domain1.com to b#domain2.com, there are generally* two SMTP servers used. The first is a#domain1.com’s submission server, used for a to submit mail to other people. The second is domain2.com’s destination** server, used to receive messages for domain2.com’s users. So an email goes from a’s email client to a’s submission server. The submission server relays it to domain2.com’s destination server, where b can then retrieve it.
The MX record represents the destination SMTP server, that other providers should use to send mail for that server’s users. For example, the MX for outlook.com is the SMTP server your server would use to relay messages to email addresses xyz#outlook.com. It is not the server outlook.com’s submission users use to send mail to other people.
Submission servers can theoretically be stored in DNS SRV records, but this isn’t widely deployed. Mail Clients usually guess using heuristics (try smtp and mail.domain2.com) or using databases that have been collected (eg, thunderbirds ispdb), or configured by the end user.
For example, thunderbird documents their autoconfiguration methods. It is a combination of a database, a special web server at autconfig.domain.com, admin provided configuration file, and guessing.
*: in more complicated setups, even more can be used (like internal submission and edge servers), but this is the baseline.
**: this is more precisely called a relay server, I use destination as a more precise name for the final relay server.
When I send an email to somebody, (I think) my email is sent to my home server, then the email is sent to target server, then it is finally sent to the people I want to send to.
Sometimes, this needs multiple hops, which is the point that confuses me. Why does this need multiple hops? Why can't the email be sent to the target server directly? XMPP doesn't need multiple hops, for example.
A typical email transaction uses an outbound mail server, which looks up your recipient's incoming server, delivers it there, then that server hands it off to your recipient's mailbox server. That's 3 hops:
When you send a message, you connect from your email client (more properly: mail user agent, MUA) to your outbound SMTP server. This is the first hop, and your SMTP server creates a Received header denoting it for auditing later on. This will include the IP it sees you connecting on, a message ID internal to that specific server, and sometimes some other information.
Your outbound SMTP server then looks up the MX (mail exchange) record for each recipient's domain in DNS and connects to the highest priority relay it can find (the numerically lowest MX priority) for them. This is the second hop. As with any hop, a Received header is added.
The MX relay typically does anti-spam filtering, potentially rejecting the message during the SMTP connection. This is what generates a bounce message, which your outbound SMTP server sends you directly (if it's sent across the internet, it risks being backscatter, a form of spam, because the bounce cannot be proven to be legitimate). Often, anti-spam filtering is done later and the message is dropped silently or else delivered with markup.
The MX relay then hands the message off internally to your mailbox (IMAP/Exchange) server (hop #3). Sometimes, local mailbox server filters are used to handle patterns (or markup from the MX server) and facilitate various actions such as filing suspected spam into your Junk folder.
If you use a mail forwarding service, the MX relay instead passes your message to the forwarding target (another MX relay). This can theoretically go on for quite a while or even loop infinitely, but it should eventually end in a mailbox server.
See also the formal SMTP specification, RFC 5321.
If you use webmail, the web mail server is your email client. Some web mail servers add headers resembling Received headers to track the connection so spam filters can extend their reputation checks to the user rather than to the webmail system, which could be characterized as yet another hop.
In enterprise-grade environments, it is common for multiple layers of servers to be used for various purposes. These extra hops happen between the MX relay and the mailbox server and can be related to extra anti-spam or anti-virus handling, advanced mail aliasing, email archiving, internal mail routing between campuses, or any number of other things. These can for example ease transitions from one mail system to another (like migrating between versions of Exchange, which may require lots of testing and a slow migration so a problem doesn't affect the whole company).
Fundamentally that has to do with the original purpose of the Internet, or rather it's predecessor ARPANET, was designed to survive the loss of portions of the network (popular to popular misconception, this design goal existed not in order to survive nuclear attack, but because network components were rather unreliable at the time).
The packet switching network is designed to find one or more routes from where the packet starts (your email client) to where it needs to be (the email client of the person you are sending to).
In order to achieve that flexibility, the system is designed not just to route directly from point A to point B, but to find an available route (from a possibly very large set of possible routes) that is currently functioning.
If your email client had to know how to directly send to every other device in the world, without intervening subnets managing traffic, your computer would need a direct connection to every other computer in the world. That would be prohibitively expensive. Instead, routers inbetween accept packets from you and other people in your area and figure out how to route each packet closer to its destination. That's similar to the method that the airlines use to have regional flights feed hubs.
When a packet is being routed through the internet it will make multiple hops to get from source to destination.
We are building a system that is, effectively, an email/calendar/contact client.
Users will provide us with their email address password (or other auth, eg oauth) and we will connect to their underlying email system.
"underlying systems" include:
Microsoft Exchange / Office365
GMail
Yahoo Mail
Apple email
Generic IMAP
Each of these systems have subtle (and not so subtle) differences in their APIs, especially to access calendar & contact data. Thus we need to know what provider the user is using.
But we would rather not ASK the user. We would like to figure it out (and at, least reduce the choices) automatically.
I've looked around for something that already implements this but have not found anything. I know it's mostly possible because Windows Phone does it pretty well (just enter username/pw and it does the right thing).
Before I dive in and start writing my own I want to ensure I'm not wasting my time if someone's already done it in an excellent way.
Know of anything like this? For this project prefer C#/.NET.
[EDIT: Adding potential algo]
Potential Algorithm:
Given email address & password
Extract domain name from email address
Try Exchange autodiscover. If successful done.
Use DNS MX records to find smtp host.
Do SMTP EHLO
Gmail responds with "250-mx.google.com at your service"
So if we see a "google.com" we are done.
Yahoo responds with 250-mta1257.mail.sk1.yahoo.com
So if we see a "yahoo.com" in the response we are done.
Apple responds with 250-xxxxxx-mac.com
So if we see a "mac.com" in the response we are done.
If none of the above
IMAP?
...
[EDIT: 5/18]
I built a prototype that uses methods that don't require auth (e.g. just MX/SMTP sluthing). Give it a try: http://bit.ly/KLZKxD
Algorithm seems reasonable. You will get best results running from unfiltered server (meaning it has direct SMTP outbound / doesn't run through a proxy). If running from client (mobile/tablet/desktop), then no guarantees as some ISP's pass SMTP through a proxy relay hence EHLO response is only for proxy.
You may want to do a port check for servers to verify expected protocol support (just a TCP connect may be sufficient but protocol handshake is preferable). Additionally, best to build up a database of verified SMTP and IMAP server mappings as there can be split names (e.g. smtp.domain.com and mail.domain.com) - discovering SMTP is easy, discovering outbound server(s) which usually also means calendar/contacts server, not so much (except for Exchange but only if autodiscovery is configured correctly).
If you can get your users to approve and assuming you have their username/password, you could try connecting to SMTP via MX record and sending an email back to your own address then checking through headers for useful info about the server (needs to be authenticated to relay). Users could alternatively reply to an email you generate in order to get the same server info.
Also ensure that you do your own DNS query and try each MX record or all primaries -- if the principal MX is down or DNS is poorly configured for equal weighting, you could end up hitting a smarthost / backup which may just be a dumb SMTP relay / store-and-forward and not give you the correct response.
TL;DR: No quick solution but a cascaded algorithm that trys and fails different solutions until one works / gives an expected result should work.
Most devices can auto detect the service by parsing the Whole e-mail address. Xyz#gmail.com would obviously be a gmail account. So for Apple, Gmail, Yahoo, Live, Hotmail etc you can easily program for.
For other domains, including custom, you can try this: http://www.exclamationsoft.com/exclamationsoft/netmailbot/help/website/HowToFindTheSMTPMailServerForAnEmailAddress.html
You can detect Google Apps For Your Domain accounts by examining the domain's MX records. If the primary MX record is ASPMX.L.GOOGLE.com, then it's GMail.
I created an implementation of this that has been used widely with some success: https://github.com/tig/Email2Calendar
This is used by both milelogr.com and freebusy.io.
Am developing and newsletter email software in vb.net for a friend, and he has asked me to add option to send email using dns servers,
How do i send an emai using dns server without smtp or imap?
Send email "using DNS servers" still uses SMTP. And it's generally a bad idea. Here's what it's about:
Usually, an email app (your own, Outlook, etc.) send mail using SMTP or similar protocol to a configured mail server. It could be a company's own server, or an ISP's server, or something like that. The mail may pass through various outgoing mail servers on its way out of an organisation, but at some point it hits a "boundary server" (There are pretty names for all these different types of server, but they're not important right now.)
This boundary server uses DNS to find the recipient's mail server and then sends the message (using SMTP) to that server. (I have omitted the details on purpose)
Sending email "using DNS" means that your app acts as a boundary server. It uses DNS to find the recipient's mail server and then sends the mail directly - rather than using any outgoing servers.
It's a bad idea for two reasons.
More and more ISPs block SMTP traffic that doesn't pass through their outgoing servers
More and more incoming servers validate the boundary server using various techniques. Your app is not likely to be accepted by these servers
Both of these issues are anti-spam countermeasures, so using this technique makes you look like a spammer, leaves many of your messages blocked, and is likely to get your IP address blacklisted.
Hopefully you can explain this to your friend so that they will understand that the "normal" way of sending mail is the only way to ensure that they don't get listed as a spammer.
When developing an application that sends out notification email messages, what are the best practices for
not getting flagged as a spammer by your hosting company. (Cover any of:)
best technique for not flooding a mail server
best mail server products, if you were to set up your own
sending messages as if from a specific user but still clearly from your application (to ensure complaints, etc come back to you) without breaking good email etiquette
any other lessons learned
not getting flagged as spam by the receiver's client? (Cover any of:)
configuring and using sender-id, domain-keys, SPF, reverse-dns, etc to make sure your emails are properly identified
best SMTP header techniques to avoid getting flagged as spam when sending emails for users (for example, using Sender and From headers together)
any other lessons learned
An additional requirement: this application would be sending a single message to a single recipient based upon an event. So, techniques for sending the same messages to multiple recipients will not apply.
best technique for not flooding a mail server
not a lot you can do about this beyond checking with your mail server admin (if it's a shared hosting account / not in your control). but if the requirement is one email to a single recipient per event, that shouldn't be too much of an issue. the things that tend to clog mail systems are emails with hundreds (or more) of recipients.
if you have events firing off all the time, perhaps consider consolidating them and having an email sent that summarizes them periodically.
sending messages as if from a specific user but still clearly from your application (to ensure complaints, etc come back to you) without breaking good email etiquette
you can accomplish this by using the "Reply-To" header, which will then have clients use that address instead of the From address when an email message is being composed.
you should also set the "Return-Path" header of any email, as email without this will often get filtered off.
ex.
From: me#me.com
Return-Path: me#me.com
Reply-To: auto#myapp.com
configuring and using sender-id, domain-keys, SPF, reverse-dns, etc to make sure your emails are properly identified
this is all highly dependent on how much ownership you have of your mail and DNS servers. spf/sender-id etc... are all DNS issues, so you would need to have access to DNS.
in your example this could present quite the problem. as you are setting mail to be from a specific user, that user would have to have SPF (for example) set in their DNS to allow your mail server as a valid sender. you can imagine how messy (if not outright impossible) this would get with a number of users with various domain names.
as for reverse DNS and the like, it really depends. most client ISP's, etc... will just check to see that reverse DNS is set. (ie, 1.2.3.4 resolves to host.here.domain.com, even if host.here.domain.com doesn't resolve back to 1.2.3.4). this is due to the amount of shared hosting out there (where mail servers will often report themselves as the client's domain name, and not the real mail server).
there are a few stringent networks that require matching reverse DNS, but this requires that you have control over the mail server if it doesn't match in the first place.
if you can be a bit more specific i may be able to provide a bit more advice, but generally, for people who need to send application mail, and don't have a pile of control over their environment, i'd suggest the following:
make sure to set a "Return-Path"
it's nice to add your app and abuse info as well in headers ie: "X-Mailer" and "X-Abuse-To" (these are custom headers, for informational purposes only really)
make sure reverse DNS is set for the IP address of your outgoing mail server
first a quick correction to the previous
return-path: is a header added by recieving system based on the envelope-sender of the incomming message
for spf to work the return-path/envelope-sender needs to be yourapp#yourdomain.com
and ensure the spf record for yourdomain.com {or if per-user spf} for yourapp#yourdomain.com allows mails to originate on the server that hosts the app/sends the email
this envelope-sender is the address that will recieve all bounces/errors
now sender-id is different entirely it checks the return-path/envelope-sender
and the
from: address {stored inside the message}
if sending
from: hisname yourapp#yourdomain.com
reply-to: hisname hisaddres#hisdomain.com
this will be a non-issue
if sending
from: hisname hisaddres#hisdomain.com
it will be and you must add a
Resent-From: hisname yourapp#yourdomain.com
as this specifies to ignore the from: for sender-id checks use this instead as it has been sent by you on his behalf
now for the other bits that are worthwhile
ip's mentioned are your mailservers
a have your ip's ptr point to a name that also resolves to the same ip
FQDNS
b have your server helo/ehlo with whatever.domain.com where domain.com is the same as the domain of the name in step A {not the same name for resons below}
c have that helo/ehlo servername also resolve to the ip of your server
d add the following spf record to that helo/ehlo name "v=spf1 a -all"
{meaning allow helo/ehlo with this name from ip's this name points to only}
e add the following sender-id lines to the helo/ehlo name {purely for completeness
"spf2.0/mfrom,pra -all" {ie there are no users#this-domain}
f add the following spf to the FQDNS-name and any other hostnames for your server
"v=spf1 -all" {ie no machines will ever helo/ehlo as this name and no users#this-domain}
{as the fqdns name can be determined by bots/infections its better to never allow this name to be used in helo/ehlo greetings directly it is enough that it be from the same domain as the helo/ehlo identity to prove the validity of both}