I wondered how email is secured (encryption and signing), then googled this and got the following knowledge.
Normally, end-to-end encryption is not performed. If it should be encrypted, another protocol such as S/MIME is needed.
Inter mail server connection mostly be encrypted using StartTLS.
Though, I couldn't find out whether inter mail server connection uses mutual TLS or not.
In other words, when the sender's mail server (e.g., owned by Google) sends emails to the receiver's mail server (e.g., owned by Microsoft), the latter check the former's certificate and verify the sender is actually who he says he is?
First of all, email is much less secure than one would think and hope. Because I didn't understand email and had similar questions as you do, I spent several months researching all technical aspects of modern email, which resulted in this article.
As you have already figured out, end-to-end encryption is totally independent from everything else. Encrypting email in transit is optional, and the percentage of TLS-secured connections increased from around 40% in 2013 to around 90% in 2020 as reported by Google/Gmail.
ESMTP itself provides only opportunistic security, which is useful only against passive attackers. As a result, the identity of the receiving server is often not even verified.
There are two standards to provide confidentiality even in the presence of an active attacker, which aren't widely deployed yet:
DNS-Based Authentication of Named Entities (DANE)
Mail Transfer Agent Strict Transport Security (MTA-STS)
Authenticating that an email is coming from the sender's domain is solved separately with the following standards:
Sender Policy Framework (SPF)
DomainKeys Identified Mail (DKIM)
Domain-based Message Authentication, Reporting, and Conformance (DMARC)
These standards are more widely deployed than the previous ones, but we're still far away from "universal" adoption and enforcement. One of the reasons for this is that not everyone sees email spoofing as a bug. Mailing lists use this as a feature.
Nope, no mutual SSL is used by default
emails totally can't be trusted
You are right. The same way as you cannot trust the name of the sender. However there are additional measures (not always used or enforced unfortunately) in the email header
routing information (can be spoofed
Domain Keys Identified Mail
see: https://blog.mailfence.com/email-header/
Related
I'm currently running a SAAS application and mails are being sent from our application using Mailjet.
Some of the larger customers have been asking to allow the emails to be sent from their domain (e.g. info#largehotel.com) instead of our system (notifications#saasapp.com).
Are there any initial pointers I will need to look at? I'm guessing they will need to add our SPF records to their SPF records too and that they will need add a DKIM key that we generate for them to add to their records too? Then do some validation on them on the DNS level and mark them as validated?
I have some understanding to have customers run their own domain against our SAAS domain but a bit lost on the sending from their email domain requirement.
First, for the record, my SaaS platform does this (vía option 2b). It’s an e-commerce marketplace and I need the receipts to be sent from the email address of the product seller, not from me (the marketplace)
You have two(ish) options
Send email through your client’s mail servers (instead of mailjet)
Verify the client’s domain on your Mailjet (or similar email) service
option 1
With option 1, you’ll need to ask your client’s IT team to setup a username and password for you to access their SMTP server. This is essentially just like them creating an email account for you to use. This may seem like the easiest path available for you, but there are potential pitfalls and disadvantages:
Doing this, you will lose the mail open/click/bounce tracking functionality you get with mailjet; because you’ll be using the company’s SMTP server instead.
If you’re sending out as a fairly common email address (eg info#your-client.com) the client may already have that account active on their mail servers. That would allow them to receive replies into the existing infrastructure but make them wary of the security issues with sharing a password to their mail server with you.
You might find that they don’t even have the ability to give you a username and password. Modern mail services don’t allow for SMTP access (which is what your web app will need); and security conscious companies require 2 factor authentication on mail accounts (which your web app can’t answer)
Option 2
For this, you will need to ask their IT team to configure some DNS records to prove to mailjet, and to the email recipient, that you’re allowed to send on behalf of your client.
You did this for your own domain when you first setup mailjet. See https://app.mailjet.com/support/how-to-add-a-sender-address,96.htm for what this involves, but it’s a case of asking the client to configure a DNS record.
That tells mailjet that you’re allowed to send on behalf of that domain; but you’ll also want the client to adjust their SPF and DKIM records so as the recipient of the emails knows to trust Mailjet’s servers with emails sent from your client’s domain name. Normally, recipients only trust email sent from your client’s mail server (which you have as option 1) and distrust email sent from SAAS providers.
You will (or should) have done this on mailjet for your own domain already as well. https://app.mailjet.com/docs/spf-dkim-guide
So for this, you’ll need your client to setup 3 DNS records.
If you go this way, you could setup a separate Mailjet account which they and you have access to. That way they ca see their dashboard directly and feel a sense of ownership and security around it. But you won’t be able to markup the price of it 😜
Conclusion
How important is the tracking? If you can’t lose that you need to go with option 2.
How technically savvy is the client? Are they going to be able to have those DNS records changed? Are they going to be (rightly) security conscious around giving you an account on their main mail sever.
Option 2 would be my preference. You might need to hold their hand through the DNS setup so get it configured on Mailjet (And ask about SPF in here to make sure you get it right) so you can provide them with clear instructions of the specific 3 DNS records to create/update.
Whatever approach you take make sure you’re talking to the right people at your clients side soon. Their marketing team may be keen to do this with you, but if their IT feels left out of the conversation they will be difficult to get on board when you need them to make the changes. Us IT folk can be grumpy and obstinate 😀
your web app
This is going to need some adjustment. You probably already store your Mailjet credentials in a file or environment variables; these might need to move these to a dB table so you can relate credentials with specific accounts. But we’d need more info on the web app to be able to speak more to that side of the challenge.
option 2b
just as a note instead of a real suggestion. Be aware that some email service provers allow the sending verification part to be done by sending an email to someone on that domain (eg admin#yourclient.com) and then allowing sending vía the API if the recipient clicks on the approve link on that email. But, even with that setup you still need the client to configure SPF and DKIM on their DNS, so the extra one record isn’t a big ask. AWS’s SES allows for this. This works for me; but I have different requirements around deliver ability, and a large number of non-tech users (as opposed to your one or two big clients)
you can ask your client to generate programmatic(app key/password) user for email need to use for example info#largehotel.com and some other info like (host:gmail, protocol: smtp,...) all basic info needed then in your saas retrieve all this info to create object with client info that you stored before to send email for the target (from developer prospective non network engineering )
The SPF is the most important think to do. In most cases you have to be very careful about the IP reputation, but since you are using Mailjet it's up to them to manage this part.
Be attentive to the overall quality of the email, text/image ratio... Also offers a text body version of the content and dont forget the unsubscribe link. Since you already send emails with your service, I guess it's points are already correct.
The fact that I access a certain website is certainly visible to the admin in my office - i.e. at work - and to the ISPs at work, home or wherever I access that website from. But if I send an e-mail from Yahoo/Google/etc. are the sender and receiver visible to the said admin, considering the fact that the connection is thru https: ?
There's a few things to consider here:
Connecting to your service prover If you log on to https://mail.google.com or https://www.hotmail.com then you are over https and so should be secure.
However most corporates only have internet access through a proxy and like to virus scan traffic. As HTTPS traffic is encrypted they use so called man in the middle (MITM) proxies so when you connect to Gmail over https you actual connect to the proxy over https (and it provides a HTTPS certificate in Google's name) and then that proxy connects to Google.
This is allowed by web browsers if the proxy certificate is set up as a local CA on your computer (which if in such a corporate environment they will be able to install this), but is very difficult to do without access to your machine (e.g.on coffee shop wifi as you will get a HTTPS certificate error). While this sounds dodgy (and is not well liked by security professionals as can introduce more risks), it is quite common in larger companies - though often companies that do this also block access to web mail providers as well. Virus scanners installed on your machine work in the exact same way.
You can see if this is the case by looking at the HTTPS certificate change in your browser and see if it is the same as at home and chains up to a real certificate authority (CA) or to your company name.
Once a party is performing a MITM and, while most companies have neither the resources nor the inclination to read this traffic, other than to run it through an automatic virus scanner, in theory they can.
Encryption to the sender address. Once you are on your mail provider (via web mail or from an email client on your device) and send a mail, for example from your gmail account to a hotmail account you've got to consider whether that connection is encrypted. In 2026 Gmail started warning if the sender address is for a service that doesn't allow Gmail to use HTTPS to send it the message. Unfortunately that was pretty much the norm for a long time and in 2014 Google said 50% of mail it sends is unencrypted, though (partially in response to Gmail marking this as insecure I guess) that has increased to 80% at the time of writing.
Either way this is from Gmail servers so local admins (e.g. in your office) should not be able to intercept this traffic as it's not sent on the local network (unless you work for a network telco or a spy agency!), but in theory unencrypted messages could be read here.
User encrypted mail only encrypted the body. Technologies like PGP or SMIME allow you to Encrypt your email message separately before you send it via a provider but this is only for the message contents so the To Address, From Address and Subject are still visible over plain text SMIME does allow you to have different subjects but that has its own problems - see the answer here for more details). And it's a massive of a pain to use these to be honest.
Authentication technologies like do not help here When researching encrypted mail you may see mention of DMARC, DKIM and SPF but they are authentication methods (to ensure you sent the message) rather than tools to hide data so ignore any mention of those when asking about hiding data. For example the earlier Google Blog link talks about authentication as the other thing they will warn against.
Quote a broad topic, and not a great fit for a Stack Overflow question to be honest (perhaps would be OK on the sister site http://security.stackexchange.com), but hopefully that gives you some understanding anyway.
Another, more detailed answer, can be found here: Using SSL/TSL to secure your communications. What I understand summing up both answers is that the e-mail content should be encrypted separately to be completely safe. I still am not sure what the answer is re: e-mail adresses of the sender and the receiver(s).
I am build a tool that initiates an SMTP transaction with a domain to see if (a) that domain can receive emails and (b) the desired address exists on that domain. I will be batching large groups of email addresses (10,000+ at a time), but I don't want to bombard the server and get blacklisted. Are there guidelines for how often is it safe to communicate with an SMTP server?
I know about the VRFY command, but it is not implemented across the board. I plan to attempt to use the VRFY command and fall back to using,
MAIL From:<user#example.com>
RCPT To:<first.last#example.org>
QUIT
to see if the message will be deliverable. Again, are there guidelines on how often I can initiate an SMTP transaction like this on a domain?
Edit:
The purpose of this is to create a tool that my organization can use to (a) clean some bad emails from several largely inactive lists so that we do not have to pay our email delivery system to send potentially thousands of emails that will bounce, and (b) check an email when a user subscribes to a list so that we reject emails like aoghuifdgsiuvb#gmail.com.
First of all, spamming is bad. Always ask user wheter she wants to receive newsletters.
"Unsort" mail addresses by domain, leaving the "distance" between e-mail addresses with same domain as big possible.
I think it's not the programmer's decision. There should be a config value which tells a minimum amount of time between two mail sending to the same domain. You should set up a limit also for that config value, avoid setting it to zero or low value.
The only universal guideline I believe can be offered is "don't do this". If you behave like a spammer, you will be treated like a spammer. In the optimistic scenario, sites will already have controls in place, and silently throttle or block you. In less ideal scenarios, they will initiate actions against you on the (reasonable) assumption that you are collecting addresses for a spam list.
A better soluton would be to actually follow through the whole SMTP session, sending a user an email with a verification code/link. This has the advantage of showing that the user actually has control of the address in question and it keeps you from looking like a spam bot.
Volume is not as much the issue as reputation. Let the user know you're about to send them an email in your web flow. This means they're much less likely to mark it as spam.
some hosts have clear and defined guidelines as to how many emails can be sent per hour.
So i guess this would depned onyour hostng service provider, UNLESS your hosting your own mail server off course.
We are building a system that is, effectively, an email/calendar/contact client.
Users will provide us with their email address password (or other auth, eg oauth) and we will connect to their underlying email system.
"underlying systems" include:
Microsoft Exchange / Office365
GMail
Yahoo Mail
Apple email
Generic IMAP
Each of these systems have subtle (and not so subtle) differences in their APIs, especially to access calendar & contact data. Thus we need to know what provider the user is using.
But we would rather not ASK the user. We would like to figure it out (and at, least reduce the choices) automatically.
I've looked around for something that already implements this but have not found anything. I know it's mostly possible because Windows Phone does it pretty well (just enter username/pw and it does the right thing).
Before I dive in and start writing my own I want to ensure I'm not wasting my time if someone's already done it in an excellent way.
Know of anything like this? For this project prefer C#/.NET.
[EDIT: Adding potential algo]
Potential Algorithm:
Given email address & password
Extract domain name from email address
Try Exchange autodiscover. If successful done.
Use DNS MX records to find smtp host.
Do SMTP EHLO
Gmail responds with "250-mx.google.com at your service"
So if we see a "google.com" we are done.
Yahoo responds with 250-mta1257.mail.sk1.yahoo.com
So if we see a "yahoo.com" in the response we are done.
Apple responds with 250-xxxxxx-mac.com
So if we see a "mac.com" in the response we are done.
If none of the above
IMAP?
...
[EDIT: 5/18]
I built a prototype that uses methods that don't require auth (e.g. just MX/SMTP sluthing). Give it a try: http://bit.ly/KLZKxD
Algorithm seems reasonable. You will get best results running from unfiltered server (meaning it has direct SMTP outbound / doesn't run through a proxy). If running from client (mobile/tablet/desktop), then no guarantees as some ISP's pass SMTP through a proxy relay hence EHLO response is only for proxy.
You may want to do a port check for servers to verify expected protocol support (just a TCP connect may be sufficient but protocol handshake is preferable). Additionally, best to build up a database of verified SMTP and IMAP server mappings as there can be split names (e.g. smtp.domain.com and mail.domain.com) - discovering SMTP is easy, discovering outbound server(s) which usually also means calendar/contacts server, not so much (except for Exchange but only if autodiscovery is configured correctly).
If you can get your users to approve and assuming you have their username/password, you could try connecting to SMTP via MX record and sending an email back to your own address then checking through headers for useful info about the server (needs to be authenticated to relay). Users could alternatively reply to an email you generate in order to get the same server info.
Also ensure that you do your own DNS query and try each MX record or all primaries -- if the principal MX is down or DNS is poorly configured for equal weighting, you could end up hitting a smarthost / backup which may just be a dumb SMTP relay / store-and-forward and not give you the correct response.
TL;DR: No quick solution but a cascaded algorithm that trys and fails different solutions until one works / gives an expected result should work.
Most devices can auto detect the service by parsing the Whole e-mail address. Xyz#gmail.com would obviously be a gmail account. So for Apple, Gmail, Yahoo, Live, Hotmail etc you can easily program for.
For other domains, including custom, you can try this: http://www.exclamationsoft.com/exclamationsoft/netmailbot/help/website/HowToFindTheSMTPMailServerForAnEmailAddress.html
You can detect Google Apps For Your Domain accounts by examining the domain's MX records. If the primary MX record is ASPMX.L.GOOGLE.com, then it's GMail.
I created an implementation of this that has been used widely with some success: https://github.com/tig/Email2Calendar
This is used by both milelogr.com and freebusy.io.
Encrypted email - sounds like a great thing, right? Problem already solved, right? Well... I don't think so and I'm hoping I'm wrong!
To understand what I'm asking, please understand what I'm NOT asking: I am not asking how I encrypt and sign messages sent over public networks. This is a bit different.
I want to set up a mail server where email clients can both read and post and their messages to the mail server encrypted in both directions, preferably using standard clients available at internet cafes, etc, without requirement for specialized software or encrypted port-forwarding a-la SSH.
The key here is that there is a trusted mail server which can deliver email to a community using a local-delivery-agent. One can then send email to and receive from anyone and everyone who also uses that same system without concern for the security of their communications. There's no need to encrypt every message using the public key of every recipient - what pain THAT would be! - and instead all we're talking about here is a bidirectional communication that's encrypted from clients to this system.
Of course, public messages come in unencrypted via the usual port 25 process to all participants of the email server. They may or may not be encrypted as they were - we're not worried about them. Email Clients connect from anywhere and the responding code on the server system encrypts those messages for reading, even though they were sent through the public network in plain text already... This much I can already get fairly easily with encrypting IMAP servers like Dovecot.
What I want to add to this is that connected email clients can send email, encrypted, back to the system of which they are a client wherein that system forwards externally unencrypted, wherever it needs to go. If it's a local mailbox, the message is delivered via a local delivery agent. No keys involved there. The advantage of this design is that there's NO exposure of the email to external, untrusted systems or networks, and if the delivery is local, it's effectively protected end-to-end WITHOUT having a point-to-point hassle of encrypting individual messages in the more typical use.
This would be "god send" because as it is now, it's impossible to send secured mail through clients on the public internet to groups of people within an internal network.
I guess another way to phrase what I’m asking for is: Has someone created a package that gives us the other half of the encryption that IMAP (and POP?) servers already do – that distant clients on untrusted networks can hand-off out-bound unencrypted email through an encrypted link to the server on the other end?
ANOTHER ALTERNATIVE occurred to me: Encrypting SMTP / ESMTP servers that talk mail server to mail server in an encrypted form. (Similarly, clients should be able to hand off unencrypted email through an encrypted link, much as https works.) Anybody know of such a package? This is not quite as good, but an important part of email architecture...
If this doesn’t exist today, it should!
Thanks for your thoughts, pointers, etc.
Most existing email clients support message submission (either using SMTP or MSA) via TLS/SSL. I just checked in Outlook and Opera and both support it.
And I know for a fact that Courier's email suite supports TLS/SSL for both SMTP and MSA (and IMAP), so it's not an obscure setup; just a little uncommon. And it supports requiring TLS/SSL for any or all protocols.
Here is an example client SSL config screen.
Outlook Express: