I am thinking of creating a mail client as a side project and wanted a brief overview of how they work. A few issues I was thinking of are:
How do they connect to SMTP servers (My guess is by using built in libraries)?
Where are mails stored?
How are passwords encrypted?
How do they check if computer is connected to internet?
Your question is too broad to answer here. But you may check the following titles to get an initial idea.
First of all you should learn how the SMTP protocol works. It is old and stable, so this is easy to manage. You may start by reading [wikipedia's SMTP title] (http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol) . In fact there exists a full SMTP transport example, which you can try with a working smtp server after establishing a telnet connection.
Also, as you want to implement a mail client, you should learn about IMAP and POP3 by reading related wikipedia titles and its references.
Related
I have a basic question about mail servers. Usually any general server (not necessarily mail servers) handle all kinds of related requests i.e. both direction interaction with client (like sending and receiving msg to and from client) but in case of mail servers, there are 2 different servers -- one for outgoing messages called outgoing server following SMTP protocol and another for incoming messages called incoming server following POP3/IMAP protocol. Why so? . For that matter couldn't the 2 protocols be accommodated in one single protocol to handle both direction message flow. Also, typically are these 2 servers hosted on same machine in a general business?
Because the protocols are old. In the old Unix tradition, and ignoring UUCP for now:
SMTP came first, to transfer mail between sites, and mail was read locally on the server/network using a text mode client when logged in. There was no need to fetch mail, you used the 'mail' command, and it accessed your local mail spool (a file containing your mail, sitting on the file system, that the SMTP server appended to).
Later on, came the development that people wanted to read mail on their own hosts so a protocol was invented to serve that. The POP server would read that same spool file, and allow you to download all your messages to your intermittently connected client computer. SMTP was reused for sending mail because it already existed and was easily adapted for that purpose.
These days, there are usually three servers of note: SMTP submission server, SMTP transmission server, and IMAP. The submission server is where end users of the service submit their e-mail to be forwarded onto the final host, for example the Google server a Gmail user submits their email to (on port 465 or port 587, usually, with authentication). The transmission server is responsible for delivering and receiving email between sites (eg. when Yahoo and Gmail exchange mail for their customers with each other, on port 25). And IMAP is used by an end user to fetch their email.
These three services, on large sites, are generally served by separate servers on separate hosts. On very large services, like Gmail, they are separate pools of servers.
On a small business host, they were often just one machine.
There are newer and more integrated protocols. For example, both EAS and EWS have mail fetch and submission contained in the same protocol.
Personally, I regret that almost nobody uses UUCP anymore :)
The separation of shipping, management and collection of mail has a historical, philosophical and design basis.
This is in line with the spirit of the Unix world: "do one thing but do it right" and "make everything as simple as possible".
How complex the issue of e-mail is and how much technology it covers, read here:
https://en.wikipedia.org/wiki/Email
and then see the relevant RFCs in the footnotes for details.
It is difficult to implement a monolithic program that takes into account all the nuances of e-mail and at the same time
keeping it simple (Simple in SMTP). Such a monolith would be a nightmare to develop, maintain and administer,
and the post office has changed many times since the 1970s.
As for the second question, there is no reason to use the same physical machine, but there are also no contraindications other than the amount of resources available.
I have some ideas about the next email services. So I would like to build a world-class email server.
(The kind of mail server that could compete with Gmail and Hotmail in a near futur - say, two years.)
How should I start ? I mean, what language and what kind of repository ? Ruby, Java ? Oracle, MySQL ?
Any open source suggestion (I already know JAMES) ?
Thank you.
Yave you looked at the source for the most widely used email server to get an idea of what to use and what their strengths and weaknesses are?
Some MTAs to look at are Postfix, QMail, Exim and Sendmail. Here is a good comparison of those: http://shearer.org/MTA_Comparison
I recently developed an email hosting solution, http://fijisoftware.com/ based on top of Postfix and must say it is a very robust server and can handle hundreds of thousands of users per server. I chose it because it's used by the largest open source email solution, http://www.zimbra.com/. I'd suggest studing the architecture of that server.
Usually mail servers are part MTA to handle SMTP and the other part that handles POP and IMAP are usually separate programs integrated, such as Courier, Dovecot etc. So you'd have to compare and study those as well.
I am looking for a SMTP server that we can use for outgoing mail that permits a high volume (1,000+ per hour) without getting blocked. It could be downloaded on our machine or run online, but cannot cost over $25. Isn't there a good open source solution? I haven't been able to find one yet. Thanks.
Well, there are even free SMTP servers that can send a high volume of e-mail.
The problem is not in sending them, the problem is in delivering. You see, SMTP is a best-effort store-and-forward protocol - the e-mails are passed from one SMTP server to another in hope that it gets closer to the adressee's mail server (and usually it works). However, this also means that your message can be rejected or dropped anywhere in this chain of servers - or by the recipient. Once the e-mail leaves your network, there's not much you can do about it. In other words, it's not your server blocking the e-mails, but the servers it will be passing the mails to.
This question may be a useful reference for your problem; see also this article for some tips that will help your e-mail get delivered.
For that ammount of email, probably any server will do just fine. The problem is not to be blacklisted as a spammer. Jeff Atwood has a good blog post on the subject.
I personnaly would recommend Postfix or qmail, but that just me ...
I've got a situation where I want to connect to an email server whose only exposed service for mail/calendar is Activesync (it's an Exchange server). The only web interface exposed is "OWA light" which is horrific.
I'd like to figure out a way to synchronize the mail from this Exchange so that it appears in a desktop mail client (or a webmail client if there any that work this way - gmail doesn't support activesync for subscriptions -- it publishes its mail as activesync which is not what I want).
Basically the ascii art of what I want is:
Exchange => Activesync <=> Client application
There are lots of mobile client applications that do this, but I am looking for a desktop or webmail client that can handle activesync as a subscriber (kind of the way IMAP works).
Thanks for any references or ideas!
Steve
Ok - of course 20 seconds after posting this, I find an answer on superuser. Probably where I should have posted to begin with (I swear I googled before posting but apparently not enough).
SU post: https://superuser.com/questions/43238/activesync-owa-desktop-client
Answer: http://davmail.sourceforge.net/
This appears to be a POP3/IMAP and SMTP bridge to activesync which is exactly what I want. If it works as claimed, I can point any mail client to it and it will sync my mail in two directions so I can send messages that will pass through the system and see all the messages that come in.
Update 2013: I've been using this DavMail OSS software for a few years now and it's really great. For retrieving mail from relatively locked-down Exchange servers, that only permit OWA access, it very easily allows me to "break out" of the box the IT guys want to put me in, and use rational tools and protocols like IMAP, POP, SMTP and Thunderbird to send/retrieve mail from the Exchange server. If I were really motivated, I could get this running on a public server, and serve my mail so that Gmail could poll it, and bridge Exchange and Gmail, but of course I'd have to pay more attention to security.
I haven't had a ton of luck getting the Calendar portions of the system working, primarily because I can't find a caldav client (on Windows) that I like - I think the DavMail part works as well as can be expected.
I tried "TouchDown" just today. I am able to fetch my email and calendar etc.. to my Mac from my Corporate ActiveSync account.
Which operating system are you using? They have support for various OS, including Windows 8.1 and Mac.
Here is the list of all major activeSync clients - http://social.technet.microsoft.com/wiki/contents/articles/1150.exchange-activesync-client-comparison-table.aspx
I am writing a program that will be emailing reports out many (~100) clients which I want to test before I spam everyone.
I want to do a test run against my production data and actually send the messages to a SMTP server, but I don't want the SMTP server to actually deliver the messages. I want the server to act like a real SMTP server from the perspective of my application, but instead of delivering messages, I just want it to store the messages, and log what happened.
Is there a SMTP server specifically designed for testing purposes?
Does anyone know of a way to configure exim or postfix to behave like I have described above
What do you use to test a mass-email delivery?
In java you can use dumbster
Its easy to use and you can validate every aspect of the email you are intercepting.
It's a Java SMTP server implementation meant for unit testing. (Just make sure you redirect your email to the machine running dumbster...)
I just found another alternative that do almost the same: Greenmail
Greenmail also support POP3, IMAP with SSL so you can test your client against it.
For .NET I set the config file to deliver mail to a folder, then you can have the automated test inspect the directory and files.
<system.net>
<mailSettings>
<smtp deliveryMethod="SpecifiedPickupDirectory">
<specifiedPickupDirectory pickupDirectoryLocation="c:\pickupDirectory"/>
</smtp>
</mailSettings>
</system.net>
While searching for options I found the following that may be useful.
DevNullSmtp
Use a null SMTP server for testing
Fake SMTP Service
twisted examples/emailserver.tac
"The Wiser framework for unit testing mail"
I've heard of a few other developers moving from Dumbster to Wiser and have migrated my testing code as well. One of the Java components that I've worked on sends thousands of emails a day and I've written unit tests for the different email templates and scenarios using Dumbster and Wiser. I prefer Wiser.
Snips from the Wiser website (http://code.google.com/p/subethasmtp/wiki/Wiser):
Wiser is a smart replacement for Dumbster and is built on top of the SubEtha SMTP Java library which allows your Java application to receive SMTP mail with a simple, easy-to-understand API.
A good program for email testing is smtp4dev (Windows only).
It's a dummy SMTP server. Sits in the system tray and does not deliver the received messages. The received messages can be quickly viewed, saved and the source/structure inspected.
http://smtp4dev.codeplex.com/
http://skaraarslan.blogspot.com/2008/02/how-to-check-email-works-without-using.html
(this presumes you are using .net to send emails)
Given that you mention exim and postfix (which I'm taking to be some kind of unix stuff), this answer might not be as useful as it could be, but check out Neptune. It's a fake SMTP server designed for automated testing. If you've got a spare windows box floating around, you could put Neptune on that then configure your app to send "through" the Neptune server.
Exim can be configured to accept incoming mails but not deliver them. Look for the keywords queue_only and queue_only_file in the documentation.
I personally modify the e-mail addresses to test, I send them to a dummy account of mine, that way I can validate not only that they sent, but that they appear in the proper format.
At my office, we have a server that is set up to always send all incoming mail to one address, regardless of who it's actually addressed to. We just point all our testing environments at that server and watch the QA mailbox fill up. I don't know what server it is, but it's probably some open source thing someone found.
Sendmail has a Test Mode.
You just invoke sendmail with the -bt parameter. As an example:
/usr/lib/sendmail -bt -Ciu-testconfiguration.cf
Please be aware that in this method, Sendmail requires an special configuration on rewrite rules. You need to understand how Sendmail rewrites addresses in order to properly create a .cf file for Test Mode.
Edit: See this article: http://ussg.iu.edu/usail/mail/debugging/
After not beeing happy with the solutions I found, I ended up writing developmentSMTP, easy to use, 100% Java --> cross platform.
Supports writing emails to file, forwarding emails or simply printing them on stdout.
Post Hoc is a pure Java application that looks exactly like an SMTP server to the application you are testing, but it simply collects all the email messages and allows you to inspect them using a web interface.
Freely available at: Post Hoc GitHub Site
For more information: PostHoc: Testing Apps that Send Email
If you're looking to manually test that the email sends and that the email template has the right kind of html and css that you're expecting, then I would recommend maildev https://www.npmjs.com/package/maildev. You can install and run it as a node module and also as a docker container! I've found it extremely handy for basic sanity testing of emails.