We are building a system that is, effectively, an email/calendar/contact client.
Users will provide us with their email address password (or other auth, eg oauth) and we will connect to their underlying email system.
"underlying systems" include:
Microsoft Exchange / Office365
GMail
Yahoo Mail
Apple email
Generic IMAP
Each of these systems have subtle (and not so subtle) differences in their APIs, especially to access calendar & contact data. Thus we need to know what provider the user is using.
But we would rather not ASK the user. We would like to figure it out (and at, least reduce the choices) automatically.
I've looked around for something that already implements this but have not found anything. I know it's mostly possible because Windows Phone does it pretty well (just enter username/pw and it does the right thing).
Before I dive in and start writing my own I want to ensure I'm not wasting my time if someone's already done it in an excellent way.
Know of anything like this? For this project prefer C#/.NET.
[EDIT: Adding potential algo]
Potential Algorithm:
Given email address & password
Extract domain name from email address
Try Exchange autodiscover. If successful done.
Use DNS MX records to find smtp host.
Do SMTP EHLO
Gmail responds with "250-mx.google.com at your service"
So if we see a "google.com" we are done.
Yahoo responds with 250-mta1257.mail.sk1.yahoo.com
So if we see a "yahoo.com" in the response we are done.
Apple responds with 250-xxxxxx-mac.com
So if we see a "mac.com" in the response we are done.
If none of the above
IMAP?
...
[EDIT: 5/18]
I built a prototype that uses methods that don't require auth (e.g. just MX/SMTP sluthing). Give it a try: http://bit.ly/KLZKxD
Algorithm seems reasonable. You will get best results running from unfiltered server (meaning it has direct SMTP outbound / doesn't run through a proxy). If running from client (mobile/tablet/desktop), then no guarantees as some ISP's pass SMTP through a proxy relay hence EHLO response is only for proxy.
You may want to do a port check for servers to verify expected protocol support (just a TCP connect may be sufficient but protocol handshake is preferable). Additionally, best to build up a database of verified SMTP and IMAP server mappings as there can be split names (e.g. smtp.domain.com and mail.domain.com) - discovering SMTP is easy, discovering outbound server(s) which usually also means calendar/contacts server, not so much (except for Exchange but only if autodiscovery is configured correctly).
If you can get your users to approve and assuming you have their username/password, you could try connecting to SMTP via MX record and sending an email back to your own address then checking through headers for useful info about the server (needs to be authenticated to relay). Users could alternatively reply to an email you generate in order to get the same server info.
Also ensure that you do your own DNS query and try each MX record or all primaries -- if the principal MX is down or DNS is poorly configured for equal weighting, you could end up hitting a smarthost / backup which may just be a dumb SMTP relay / store-and-forward and not give you the correct response.
TL;DR: No quick solution but a cascaded algorithm that trys and fails different solutions until one works / gives an expected result should work.
Most devices can auto detect the service by parsing the Whole e-mail address. Xyz#gmail.com would obviously be a gmail account. So for Apple, Gmail, Yahoo, Live, Hotmail etc you can easily program for.
For other domains, including custom, you can try this: http://www.exclamationsoft.com/exclamationsoft/netmailbot/help/website/HowToFindTheSMTPMailServerForAnEmailAddress.html
You can detect Google Apps For Your Domain accounts by examining the domain's MX records. If the primary MX record is ASPMX.L.GOOGLE.com, then it's GMail.
I created an implementation of this that has been used widely with some success: https://github.com/tig/Email2Calendar
This is used by both milelogr.com and freebusy.io.
Related
I wondered how email is secured (encryption and signing), then googled this and got the following knowledge.
Normally, end-to-end encryption is not performed. If it should be encrypted, another protocol such as S/MIME is needed.
Inter mail server connection mostly be encrypted using StartTLS.
Though, I couldn't find out whether inter mail server connection uses mutual TLS or not.
In other words, when the sender's mail server (e.g., owned by Google) sends emails to the receiver's mail server (e.g., owned by Microsoft), the latter check the former's certificate and verify the sender is actually who he says he is?
First of all, email is much less secure than one would think and hope. Because I didn't understand email and had similar questions as you do, I spent several months researching all technical aspects of modern email, which resulted in this article.
As you have already figured out, end-to-end encryption is totally independent from everything else. Encrypting email in transit is optional, and the percentage of TLS-secured connections increased from around 40% in 2013 to around 90% in 2020 as reported by Google/Gmail.
ESMTP itself provides only opportunistic security, which is useful only against passive attackers. As a result, the identity of the receiving server is often not even verified.
There are two standards to provide confidentiality even in the presence of an active attacker, which aren't widely deployed yet:
DNS-Based Authentication of Named Entities (DANE)
Mail Transfer Agent Strict Transport Security (MTA-STS)
Authenticating that an email is coming from the sender's domain is solved separately with the following standards:
Sender Policy Framework (SPF)
DomainKeys Identified Mail (DKIM)
Domain-based Message Authentication, Reporting, and Conformance (DMARC)
These standards are more widely deployed than the previous ones, but we're still far away from "universal" adoption and enforcement. One of the reasons for this is that not everyone sees email spoofing as a bug. Mailing lists use this as a feature.
Nope, no mutual SSL is used by default
emails totally can't be trusted
You are right. The same way as you cannot trust the name of the sender. However there are additional measures (not always used or enforced unfortunately) in the email header
routing information (can be spoofed
Domain Keys Identified Mail
see: https://blog.mailfence.com/email-header/
I'm currently running a SAAS application and mails are being sent from our application using Mailjet.
Some of the larger customers have been asking to allow the emails to be sent from their domain (e.g. info#largehotel.com) instead of our system (notifications#saasapp.com).
Are there any initial pointers I will need to look at? I'm guessing they will need to add our SPF records to their SPF records too and that they will need add a DKIM key that we generate for them to add to their records too? Then do some validation on them on the DNS level and mark them as validated?
I have some understanding to have customers run their own domain against our SAAS domain but a bit lost on the sending from their email domain requirement.
First, for the record, my SaaS platform does this (vía option 2b). It’s an e-commerce marketplace and I need the receipts to be sent from the email address of the product seller, not from me (the marketplace)
You have two(ish) options
Send email through your client’s mail servers (instead of mailjet)
Verify the client’s domain on your Mailjet (or similar email) service
option 1
With option 1, you’ll need to ask your client’s IT team to setup a username and password for you to access their SMTP server. This is essentially just like them creating an email account for you to use. This may seem like the easiest path available for you, but there are potential pitfalls and disadvantages:
Doing this, you will lose the mail open/click/bounce tracking functionality you get with mailjet; because you’ll be using the company’s SMTP server instead.
If you’re sending out as a fairly common email address (eg info#your-client.com) the client may already have that account active on their mail servers. That would allow them to receive replies into the existing infrastructure but make them wary of the security issues with sharing a password to their mail server with you.
You might find that they don’t even have the ability to give you a username and password. Modern mail services don’t allow for SMTP access (which is what your web app will need); and security conscious companies require 2 factor authentication on mail accounts (which your web app can’t answer)
Option 2
For this, you will need to ask their IT team to configure some DNS records to prove to mailjet, and to the email recipient, that you’re allowed to send on behalf of your client.
You did this for your own domain when you first setup mailjet. See https://app.mailjet.com/support/how-to-add-a-sender-address,96.htm for what this involves, but it’s a case of asking the client to configure a DNS record.
That tells mailjet that you’re allowed to send on behalf of that domain; but you’ll also want the client to adjust their SPF and DKIM records so as the recipient of the emails knows to trust Mailjet’s servers with emails sent from your client’s domain name. Normally, recipients only trust email sent from your client’s mail server (which you have as option 1) and distrust email sent from SAAS providers.
You will (or should) have done this on mailjet for your own domain already as well. https://app.mailjet.com/docs/spf-dkim-guide
So for this, you’ll need your client to setup 3 DNS records.
If you go this way, you could setup a separate Mailjet account which they and you have access to. That way they ca see their dashboard directly and feel a sense of ownership and security around it. But you won’t be able to markup the price of it 😜
Conclusion
How important is the tracking? If you can’t lose that you need to go with option 2.
How technically savvy is the client? Are they going to be able to have those DNS records changed? Are they going to be (rightly) security conscious around giving you an account on their main mail sever.
Option 2 would be my preference. You might need to hold their hand through the DNS setup so get it configured on Mailjet (And ask about SPF in here to make sure you get it right) so you can provide them with clear instructions of the specific 3 DNS records to create/update.
Whatever approach you take make sure you’re talking to the right people at your clients side soon. Their marketing team may be keen to do this with you, but if their IT feels left out of the conversation they will be difficult to get on board when you need them to make the changes. Us IT folk can be grumpy and obstinate 😀
your web app
This is going to need some adjustment. You probably already store your Mailjet credentials in a file or environment variables; these might need to move these to a dB table so you can relate credentials with specific accounts. But we’d need more info on the web app to be able to speak more to that side of the challenge.
option 2b
just as a note instead of a real suggestion. Be aware that some email service provers allow the sending verification part to be done by sending an email to someone on that domain (eg admin#yourclient.com) and then allowing sending vía the API if the recipient clicks on the approve link on that email. But, even with that setup you still need the client to configure SPF and DKIM on their DNS, so the extra one record isn’t a big ask. AWS’s SES allows for this. This works for me; but I have different requirements around deliver ability, and a large number of non-tech users (as opposed to your one or two big clients)
you can ask your client to generate programmatic(app key/password) user for email need to use for example info#largehotel.com and some other info like (host:gmail, protocol: smtp,...) all basic info needed then in your saas retrieve all this info to create object with client info that you stored before to send email for the target (from developer prospective non network engineering )
The SPF is the most important think to do. In most cases you have to be very careful about the IP reputation, but since you are using Mailjet it's up to them to manage this part.
Be attentive to the overall quality of the email, text/image ratio... Also offers a text body version of the content and dont forget the unsubscribe link. Since you already send emails with your service, I guess it's points are already correct.
The fact that I access a certain website is certainly visible to the admin in my office - i.e. at work - and to the ISPs at work, home or wherever I access that website from. But if I send an e-mail from Yahoo/Google/etc. are the sender and receiver visible to the said admin, considering the fact that the connection is thru https: ?
There's a few things to consider here:
Connecting to your service prover If you log on to https://mail.google.com or https://www.hotmail.com then you are over https and so should be secure.
However most corporates only have internet access through a proxy and like to virus scan traffic. As HTTPS traffic is encrypted they use so called man in the middle (MITM) proxies so when you connect to Gmail over https you actual connect to the proxy over https (and it provides a HTTPS certificate in Google's name) and then that proxy connects to Google.
This is allowed by web browsers if the proxy certificate is set up as a local CA on your computer (which if in such a corporate environment they will be able to install this), but is very difficult to do without access to your machine (e.g.on coffee shop wifi as you will get a HTTPS certificate error). While this sounds dodgy (and is not well liked by security professionals as can introduce more risks), it is quite common in larger companies - though often companies that do this also block access to web mail providers as well. Virus scanners installed on your machine work in the exact same way.
You can see if this is the case by looking at the HTTPS certificate change in your browser and see if it is the same as at home and chains up to a real certificate authority (CA) or to your company name.
Once a party is performing a MITM and, while most companies have neither the resources nor the inclination to read this traffic, other than to run it through an automatic virus scanner, in theory they can.
Encryption to the sender address. Once you are on your mail provider (via web mail or from an email client on your device) and send a mail, for example from your gmail account to a hotmail account you've got to consider whether that connection is encrypted. In 2026 Gmail started warning if the sender address is for a service that doesn't allow Gmail to use HTTPS to send it the message. Unfortunately that was pretty much the norm for a long time and in 2014 Google said 50% of mail it sends is unencrypted, though (partially in response to Gmail marking this as insecure I guess) that has increased to 80% at the time of writing.
Either way this is from Gmail servers so local admins (e.g. in your office) should not be able to intercept this traffic as it's not sent on the local network (unless you work for a network telco or a spy agency!), but in theory unencrypted messages could be read here.
User encrypted mail only encrypted the body. Technologies like PGP or SMIME allow you to Encrypt your email message separately before you send it via a provider but this is only for the message contents so the To Address, From Address and Subject are still visible over plain text SMIME does allow you to have different subjects but that has its own problems - see the answer here for more details). And it's a massive of a pain to use these to be honest.
Authentication technologies like do not help here When researching encrypted mail you may see mention of DMARC, DKIM and SPF but they are authentication methods (to ensure you sent the message) rather than tools to hide data so ignore any mention of those when asking about hiding data. For example the earlier Google Blog link talks about authentication as the other thing they will warn against.
Quote a broad topic, and not a great fit for a Stack Overflow question to be honest (perhaps would be OK on the sister site http://security.stackexchange.com), but hopefully that gives you some understanding anyway.
Another, more detailed answer, can be found here: Using SSL/TSL to secure your communications. What I understand summing up both answers is that the e-mail content should be encrypted separately to be completely safe. I still am not sure what the answer is re: e-mail adresses of the sender and the receiver(s).
Am developing and newsletter email software in vb.net for a friend, and he has asked me to add option to send email using dns servers,
How do i send an emai using dns server without smtp or imap?
Send email "using DNS servers" still uses SMTP. And it's generally a bad idea. Here's what it's about:
Usually, an email app (your own, Outlook, etc.) send mail using SMTP or similar protocol to a configured mail server. It could be a company's own server, or an ISP's server, or something like that. The mail may pass through various outgoing mail servers on its way out of an organisation, but at some point it hits a "boundary server" (There are pretty names for all these different types of server, but they're not important right now.)
This boundary server uses DNS to find the recipient's mail server and then sends the message (using SMTP) to that server. (I have omitted the details on purpose)
Sending email "using DNS" means that your app acts as a boundary server. It uses DNS to find the recipient's mail server and then sends the mail directly - rather than using any outgoing servers.
It's a bad idea for two reasons.
More and more ISPs block SMTP traffic that doesn't pass through their outgoing servers
More and more incoming servers validate the boundary server using various techniques. Your app is not likely to be accepted by these servers
Both of these issues are anti-spam countermeasures, so using this technique makes you look like a spammer, leaves many of your messages blocked, and is likely to get your IP address blacklisted.
Hopefully you can explain this to your friend so that they will understand that the "normal" way of sending mail is the only way to ensure that they don't get listed as a spammer.
I need a cross platform (at least windows and mac) development utility that runs as an SMTP server that acts as an SMTP server but will redirect all mail to a single address that's configurable. It would also be helpful if it wrote the contents out to a file or gui. Long ago I configured Apache James to do this but it wasn't that straight forward to figure out. Hoping there's something really simple out there.
I need the emails to be forwarded (to the single address) so I can see how they are rendered on different clients (gmail, outlook, etc.)
Thanks! -Mike
We used Mailtrap for this. It give you remote smtp server account and direct access to all mails in it. So you just enter given smtp credential in your application and after that all email sent by your system will be visible on mailtrap.
On mailtrap you can have as many smtp account as you want( different account for different application environments, or different application) Also you can manage access to your account ( so only trusted people will se your emails) and you can forward some emails to real email addresses.
It doesn't do the forwarding you are looking for, but for most of my testing I make use of fakemail (http://www.lastcraft.com/fakemail.php) it's simply a script that listens on a port an acts as an smtp server, writing any incoming mail to a directory.
You could use Python's smtpd library and override the process_message function to only send to the desired address -- i.e. replace the "to" field with your desired constant email address.
Here's a page with some examples of using smtpd (with asyncore) to actually send out mail.