Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
This question got me thinking, and I now realize that I don't know anything about the internals of MTAs.
What exactly does an MTA do? Everything after the SMTP protocol seems like dark magic to me. Let's say that I wanted to code a minimalistic MTA (or MDA) just for sending emails, what would I need to learn/do?
Edit: I don't actually plan on writing an MTA, I just want to understand how it works internally.
--- edit after somehow noticing you talked about possibly writing a MTA ---
To write a MTA, you need to open a server socket. When someone connects, you need to send and receive text (ascii) data on that socket in compliance with the SMTP protocol. SMTP is very chatty, so you can expect a few rounds of communication.
The initial round of communication typically tells you whether SMTP is supported or ESMTP is supported. The second (optional) round of communication is to determine security / encryption / feature support. Eventually the "client" side will ask to send a message to a particular address / set of addresses. When done, the server will indicate that it's ready to get the body of the email message. When the body of the message (and it's optinal attachments) have all been transmitted, the MTA will tell you it received the message fine. At that point in time, the MTA will act as a client to other MTAs discovered via DNS MX records to get your email closer to it's destination MTA which will copy it into someone's inbox.
So an MTA is needed because mail delivery on the client side is the equivalent to handing a physical letter to a post office. Post offices are responsible for inter-postoffice routing (which parallels to MTA-to-MTA transmission). The destination Post office is then responsible for delivery of the letter to the post office box or home address (which parallels one's computer inbox).
They don't call it e-mail for nothing.
--- original post follows ---
A MTA will accept a mail message, see if it can forward or deliver it, respond if it can be forwarded or delivered, and then forward or deliver it if it indicated it could.
How the message gets closer to it's final destination usually has a bit to do with DNS. MX (mail exchange) records in DNS indicate servers which are responsible (or at least closer to the responsible server) for particular email domain names. It is not possible to fully understand how a mail message gets closer to it's destination without understanding how DNS works.
A MTA typically looks at the delivery address, and either is configured to be the "end point" of the email address's mail domain, or knows that server XYZ is one hop closer to the email address's mail domain. If it's an endpoint, it will copy the message from the wire into someone's inbox. If it's relaying it will "forward" the message to the next MTA.
Here ya go: http://en.wikipedia.org/wiki/Message_transfer_agent
Quickly, the MTA receives the raw message, decides where it's ultimate destination is, and then forwards the message on to that destination.
A very simple MTA can be written the delivers only to local inboxes. The MTA is an "easier" part of the system to write because you can behave badly but still be functional, so your interoperability with other systems is less of an issue (that's where much of the complexities of email lie nowadays, that and spam/virus checking).
The real contract of an MTA is simply that if you accept the message from the system sending it to you, you accept responsibility to deliver that message. Thus, when that socket closes with an acknowledgement of acceptance, the delivering systems job is done and it's all in your hands.
If you happen to do a crummy job, mail is lost, and it's your problem. But it's still fun to play around with.
Edit: The original tutorial I linked to has gone 404. Here's another that's ok: https://troubleshootguru.wordpress.com/2014/07/06/mail-server-components-mta-mda-mua/
In short, a MUA is a user client that uses SMTP to send an email to an MTA. The MTA is a server that is responsible for routing the MTA to its destination. If that destination is another server, the MTA hands the email to an MDA. The MDA is a client on the server that uses SMTP to forward the email to the other server, which is also an MTA.
So what do you need to learn? If you want to write an MUA or MDA, you need to learn how to open a socket to another computer, send SMTP commands, and receive SMTP responses. If you want to write an MTA, you need to learn how to listen for socket connections on a port, receive SMTP commands, and send SMTP responses.
If you like Java, try the code on this page as a starting point for a client.
Related
I have a basic question about mail servers. Usually any general server (not necessarily mail servers) handle all kinds of related requests i.e. both direction interaction with client (like sending and receiving msg to and from client) but in case of mail servers, there are 2 different servers -- one for outgoing messages called outgoing server following SMTP protocol and another for incoming messages called incoming server following POP3/IMAP protocol. Why so? . For that matter couldn't the 2 protocols be accommodated in one single protocol to handle both direction message flow. Also, typically are these 2 servers hosted on same machine in a general business?
Because the protocols are old. In the old Unix tradition, and ignoring UUCP for now:
SMTP came first, to transfer mail between sites, and mail was read locally on the server/network using a text mode client when logged in. There was no need to fetch mail, you used the 'mail' command, and it accessed your local mail spool (a file containing your mail, sitting on the file system, that the SMTP server appended to).
Later on, came the development that people wanted to read mail on their own hosts so a protocol was invented to serve that. The POP server would read that same spool file, and allow you to download all your messages to your intermittently connected client computer. SMTP was reused for sending mail because it already existed and was easily adapted for that purpose.
These days, there are usually three servers of note: SMTP submission server, SMTP transmission server, and IMAP. The submission server is where end users of the service submit their e-mail to be forwarded onto the final host, for example the Google server a Gmail user submits their email to (on port 465 or port 587, usually, with authentication). The transmission server is responsible for delivering and receiving email between sites (eg. when Yahoo and Gmail exchange mail for their customers with each other, on port 25). And IMAP is used by an end user to fetch their email.
These three services, on large sites, are generally served by separate servers on separate hosts. On very large services, like Gmail, they are separate pools of servers.
On a small business host, they were often just one machine.
There are newer and more integrated protocols. For example, both EAS and EWS have mail fetch and submission contained in the same protocol.
Personally, I regret that almost nobody uses UUCP anymore :)
The separation of shipping, management and collection of mail has a historical, philosophical and design basis.
This is in line with the spirit of the Unix world: "do one thing but do it right" and "make everything as simple as possible".
How complex the issue of e-mail is and how much technology it covers, read here:
https://en.wikipedia.org/wiki/Email
and then see the relevant RFCs in the footnotes for details.
It is difficult to implement a monolithic program that takes into account all the nuances of e-mail and at the same time
keeping it simple (Simple in SMTP). Such a monolith would be a nightmare to develop, maintain and administer,
and the post office has changed many times since the 1970s.
As for the second question, there is no reason to use the same physical machine, but there are also no contraindications other than the amount of resources available.
Is there a way to tell if an RV inbox has a valid, active endpoint?
I have a system where clients create RV Inboxes. These are then passed to other components in the system which can use the Inbox to send messages to the client.
I would like a Monitor process in my system to know if a client has died. The Monitor will have the client Inbox.
I can implement a heartbeating mechanism, but I wonder if there is a mechanism within RV which can tell me whether the Inbox is still valid - ie, whether messages sent on it will be routed to an active client.
I guess that RV itself must know this - as it will know whether it can send a message to the Inbox or not. Is there a way for my code to be able to access this information, or to make a test about whether an Inbox is valid at a given time?
Inboxes use direct connections. From the documentation:
Transport objects can create inbox names,
designating a destination that is unique
to that transport object and its process.
Rendezvous software
uses point-to-point
techniques to deliver messages with inbox subject names.
And, direct communication is really just UDP
Direct communication uses RPTP over a UDP channel.
So you have two problems - first, mapping the inbox to the ip address and UDP port, and two, telling if it's what you want.
I haven't been able to find any way to solve the first one - those inboxes are opaque, though presumably someone clever could figure it out.
For the second one, it seems like it's not so easy a problem to solve in general - from our sister site
UDP ports only have two states: listening or not. That usually translates to "having a socket open on it by a process" or "not having any socket open". The latter case should be easy to detect since the system should respond with an ICMP Destination Unreachable packet with code=3 (Port unreachable). Unfortunately many firewalls could cut those packets so if you don't get anything back you don't know for sure if the port is in this state or not. And let's not forget that ICMP is session less too and doesn't do retransmissions: the Port Unreachable packet could very well be lost somewhere on the net.
I would guess, the easiest way to do this is with a heartbeating mechanism. In the past, we had a library around RV that would publish all of the clients and any subjects subscribed to (including inboxes) out for various management processes.
(Not really answering my own question - this is from a work colleague.)
I came across this when I was working on an RV bridge that we had to write last year, as it was proxying and rewriting inboxes. It turns out RV creates “throwaway” inboxes for certain operations (the blocking ones), and no message is sent (on that multicast, anyway) to say that the inbox is gone. The way inboxes work seems to be that:
The inbox encodes the IP address of the host, a unique process ID for the client on that host/transport, and an incrementing counter
The sender just sends a unicast packet to the right IP and port (service)
The daemon (if it has clients on that service) checks if it has any active subscriptions on that subject (just like for multicast), and passes it on to the client
In other words, it’s the receiving daemon which has to discard messages sent on closed inboxes, and there is no equivalent to the “LISTEN.STOP” messages that accompany regular subscription ends.
I am looking for a SMTP server that we can use for outgoing mail that permits a high volume (1,000+ per hour) without getting blocked. It could be downloaded on our machine or run online, but cannot cost over $25. Isn't there a good open source solution? I haven't been able to find one yet. Thanks.
Well, there are even free SMTP servers that can send a high volume of e-mail.
The problem is not in sending them, the problem is in delivering. You see, SMTP is a best-effort store-and-forward protocol - the e-mails are passed from one SMTP server to another in hope that it gets closer to the adressee's mail server (and usually it works). However, this also means that your message can be rejected or dropped anywhere in this chain of servers - or by the recipient. Once the e-mail leaves your network, there's not much you can do about it. In other words, it's not your server blocking the e-mails, but the servers it will be passing the mails to.
This question may be a useful reference for your problem; see also this article for some tips that will help your e-mail get delivered.
For that ammount of email, probably any server will do just fine. The problem is not to be blacklisted as a spammer. Jeff Atwood has a good blog post on the subject.
I personnaly would recommend Postfix or qmail, but that just me ...
I am relatively new to arena of emails. Just drilling out tutorial by tutorial about email internals. My requirement is I need to send emails and receive emails from my webhost. For this to happen, I need an smtp server(daemon) running right? I recently studied about MTAs, which are responsible for transferring emails from one host to the other. So this smtp server(daemon) acts as an MTA. And I also studied sendmail is an MTA, which boils down to sendmail runs a smtp daemon in the background. Right?
It's probably fairer to say that sendmail is an SMTP daemon than to say it runs one, since sendmail is pretty monolithic. But basically you have it right - in order to receive email, you need an MTA listening on port 25 when an incoming connection comes. There are many choices for an MTA. I prefer postfix because it's not monolithic, it has a very easy to read config file, and it has a good security model. Other good choices are exim and qmail.
For outgoing mail, you need a program that can figure out where the mail is supposed to go, and make a connection to that receiver's port 25. Once again, just about any MTA will do that for you, but some programs that want to send mail will attempt to do it directly instead of invoking the local MTA. The problem with that is that they then have to duplicate all the things MTAs give you, such as knowing how to fall back and retry when the mail receiver can't be reached.
You need to run daemon to receive emails and to resend emails if they failed to be sent for some reason.
To send them, you just invoke sendmail which will connect to the destination's sendmail (which, of course, runs as a daemon), send you mail and exit.
If you sending this from a website, you may use mail functions of your scripting language, because spawning a process is quite costly thing under a heavy load.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Closed 4 years ago.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Improve this question
My VPS provider (Slicehost) doesn't provide an SMTP server. I use Google Apps to send and receive mail for my domains, but I want to be able to programmatically send e-mail.
I've been Googling this issue on and off for many months, and I just can't seem to get a clear picture of what I need to do. Do I just need an MTA like postfix? Do I configure it as a "satellite", or an "internet" site? Do I need to make any DNS changes? If I want to programmatically receive e-mail someday, are there any configuration options I should be careful about tinkering with? Also, if my VPS host did provide an SMTP server, what wouldn't I need to do, that I'm doing now?
Thank you!
Edit: A quick synopsis of my findings:
Update /etc/hosts and /etc/hostname to reflect the FQDN (for example, mail.domain.com or sub.domain.com instead of slicename)
Reboot and verify with hostname -f
Set up reverse DNS (which is owned by whoever owns the IP pool) to point at the same FQDN
Check that the RDNS has propogated with dig -x server.ip.address
sudo aptitude install postfix telnet mailx
Use internet site setting
Set FQDN to whatever was chosen above
Test with mail whateveryouremailis#gmail.com
Have you looked here, seems to me that there is plenty of information to get started. I found mail-server-slice-setup to be quite informative.
To my experience, when email providers check for spams, they check 5 main things other than content:
(When I say domain, I'm referring to the domain of the sender's email address.)
"A" record (IP) of the domain name exists
MX records of the domain name exist
PTR record (Reverse lookup) of the sender's IP address exists (Sometimes, I think it is also checked to see if the PTR record has the same A record as the original domain)
SPF record of that domain matches the IP number (This is a list of IP numbers allowed to send emails of that domain. Although this would be a great solution for spam filtering, this is not an official record, so it is not always checked. Check out the SPF setup wizard.)
The IP segment of the server is not a blacklisted segment.
The fifth one is the one that was a problem for me. Apparently email providers such as Hotmail, Gmail, Yahoo drop all incoming email from widely used server providers such as Amazon EC2. In that case, there's simply no way to guarantee the delivery of all outgoing emails, so you would have to relay all outgoing email from some other SMTP server outside the blacklisted segment. The easiest way to do this is configuring Postfix (or whatever MTA you're using) to automatically relay all emails. You can find plenty of articles on this if you do some googling.
If that is not the case, then all you need is a simple MTA like Postfix, as long as all the necessary DNS records listed above are configured properly, you should have no trouble sending emails.
Other answers have provided information about the mechanics of sending email, but one thing you need to consider is what sender address you're going to use, and how you're going to deal with bounce messages.
Let's say you're sending emails from sender#example.com; then (as Murat Ayfer noted) you need to make sure that your host is allowed to send mail according to example.com's SPF record. (This may or may not be configured by IP address -- see the SPF site for more details.) If you're in charge of the example.com domain, then this shouldn't be a problem. Being listed as an acceptable sender for the domain is an important step towards your emails getting through.
The next thing you need to worry about is how to deal with bounced emails. If some other host receives emails for example.com (i.e. the MX record points elsewhere), then it's mainly someone else's problem. (Note however that your application has to be prepared for a failure during the SMTP session to the mailserver on your host, due to syntactically invalid email addresses, for example.) But if you decide that you'll receive mail for example.com on your host, then you need to worry about dealing with retry messages, bounce messages, double bounce messages, and a whole lot of other general mail server administration. There's some info on handling undelivered emails in web applications which might be helpful, but I also suggest getting a good book on Postfix or whatever mail server you decide to use.
Just install something like postfix as "internet site" and you'll be able to send mails directly from your VPS
it should work with default settings but it would be recommended to also read a few docs/how-to's to secure it properly