Based on a what I found on the internet, MIME (Multipurpose Internet Mail Extensions, now Internet Media Type (?)) is a way to describe file types (a header used by several protocols).
So, MIME itself is not a protocol, rather an extension used by other protocols, right ?
This means that the extension is used at the application layer by the applications with no protocol doing anything other than carrying the MIME header.
So, if I send a mail with a mp3 attachment, SMTP/other application layer protocol recognizes that this is an mp3 attachment or it is the duty of the application solely to recognize the file? In that sense, MIME cannot be called as an extension to SMTP but rather a feature to be used by applications.
If SMTP does not recognize that this is a different kind of file, how will it properly store it at the mail server ? (e.g. a MPEG video file needs a particular format to be stored, how will mail server store it without giving it any special treatment ? )
Sorry if my questions sound a bit vague but I want to get an idea of how different protocols (especially, SMTP) use MIME.
Thanks for your help.
RFC 822 email was originally purely plain-text, 7-bit US-ASCII. MIME specifies a facility for encapsulating other media types in email containers. It does not specify any changes to SMTP (although e.g. the 8BITMIME ESMTP extension is useful for simplifying transport of MIME messages). Thus, it is an extension of an existing protocol, not a distinct protocol in its own right. This is also demonstrated by the fact that other protocols -- notably, HTTP -- have incorporated (parts of) MIME for tagging of content types and encodings.
An Internet Media Type is only one aspect of what MIME used to codify; the mechanisms for specifying character sets and encodings are still defined in MIME proper.
Traditionally, the mail server simply stores the bare RFC822 message in its message store; it is the responsibility of the mail client to parse and possibly manipulate any MIME structure in the body for display and interaction. (The fact that RFC 822 has been superseded by 2282 and then 5322 has not fundamentally changed the actual mail message format.)
Some servers deviate from this model; for example, Microsoft Exchange seems to parse all incoming messages in order to borg them into its internal format, somewhat to the detriment of its interoperability with standard tools, and the sanity of those few of us who require reliable, felicitous access to our actual email.
The SMTP protocol itself knows nothing about the MIME format, but the SMTP server itself has to at least implement basic rfc0822 support in order to ad the Received headers, however, it does not need to implement MIME.
How does the server save the file to disk? The same way it received it from the client over the TCP/IP stream. It just saves the raw bytes sent (with the addition of the addition of a Received header I mentioned).
In other words, you are way over-thinking this. The SMTP server doesn't have to know anything about mp3 file attachments or anything else because the MIME format (it's not a protocol) is just a way to serialize the mp3 data in a message.
Related
I have been doing some research/tests on the standardized email format. Ultimately I am looking to develop an email parser for an application. I am noticing some differences in the format of the email, mainly between email clients (gmail, mac mail, etc) and email marketing services (Constant Contact, Mail Chimp, etc).
My understanding of the format (RFC2822) is that a \n\n separates the headers from the body. These appears to be consistent with emails received from email marketing services. Email clients, however, appear to have an extra set of header(s) or instructions for the message. See examples of email strings below. Note that I pulled these strings via an email pipe. Also note, these are only snippets of the header/body split.
Email Marketing Service:
Content-Type: text/html;
charset="utf-8"
Content-Transfer-Encoding: 8bit
<html>
<head>
<title>Welcome to Banana Republic. Enjoy 25% off! </title>
<STYLE type="text/css">
.ReadMsgBody
{ width: 100%;}
.ExternalClass
{width: 100%;}
Here you will see the line break separating the headers from the body. All good according to the format. Now look at the email client.
Email client:
Mime-Version: 1.0 (Mac OS X Mail 7.0 (1816))
X-Mailer: Apple Mail (2.1816)
--Apple-Mail=_28DD752B-7960-488D-994F-DA9408FCA880
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=windows-1252
Testing Mac Mail. This is the body.
You see that in this case, there is an additional set of "headers" which appear to be instructions about how, in this case, Mac Mail has formatted the email.
I guess my question is, is this a valid format? Is there any specification on it? Is there any well known/documented ways to check for and parse this type of format without knowing which type of format is being received?
[extending points made in comments]
is this a valid format?
Yes. The overall framework for mail messages more complex than strict 7-bit ASCII text is known as MIME. It includes the specification of the "Content-Type" header in your first example that informs a client that the whole message is HTML rather than plain text. Many (possibly most) messages these days are of type "multipart/alternative" at the outermost level, encapsulating 2 (or more!) versions of the message body, most often a text/plain representation and text/html version, which is itself often inside a multipart/mixed container including embedded images.
Is there any specification on it?
Yes. The basics of MIME are described in RFC's 2045-2049 and there have been many extensions and corrections described in many later RFC's and type registration docs. MIME also provides the core components for the specification of HTTP documents, so many of the extensions are almost irrelevant for email.
Is there any well known/documented ways to check for and parse this
type of format without knowing which type of format is being received?
Yes. While nearly all modern email is in MIME format, formally you can detect it by looking for the "MIME-Version" header. See RFC2045 for specifics. Note that your first example doesn't show that header but it must have existed in the full original because otherwise the headers you showed would be meaningless.
This demonstrates why you probably should reconsider the idea of writing your own mail parser. What you saw as 2 formats are not that in fact, rather they are just different applications of the MIME format framework. MIME is significantly older than RFC2822 (which, incidentally, is itself obsoleted by RFC5322) and has many mature and robust parsers available. It is easy to write a MIME parser that will work for most mail, a little harder to write one that will work for nearly all valid mail, and sanity-challenging to write one that will safely handle the real world of mail which often isn't exactly correct and in some cases is designed to break naive parsers in malicious ways. Take advantage of the torn-out hair of decades of coders who have preceded you: use an existing parser.
I am currently working on xmppframework, Requirements are to transfer the file between two iPhones. I searched for XEPs and found 0065 and 0096
XEP-0065 says:
XMPP is designed for sending relatively small chunks of XML between
network entities and is not designed for sending binary data. However,
sometimes it is desirable to send binary data to another entity that
one has discovered on the XMPP network (e.g., to send a file).
Therefore it is valuable to have a generic protocol for streaming
binary data between any two entities on an XMPP network. The main
application for such a bytestreaming technology is file transfer as
specified in SI File Transfer [1] and Jingle File Transfer [2].
However, other applications are possible, which is why it is important
to develop a generic protocol rather than one that is specialized for
a particular application such as file transfer.
Please see the line in bold, its confusing me if file transfer XEPs are SI File Transfer(0096) and Jingle File Transfer(0234), then what is the purpose of this 0065 XEP? why people on net referring sep-0065 for file transfer?
In XMPP there are different protocols (XEPS) for file transfer. Jingle, Bytestreams, OOB, IBB...
The purpose of XEP-0096 is stream initiation. So its build on top of the other file transfer protocols to enable seamless file transfers.
So its used to agree on one of the above file transfer protocols between 2 clients for a transfer, and also for finding a fallback method if this fails for any reason.
Alex
XEP-0065 is for proxied file transfers: you will need such a proxy in your infrastructure, unless you use a public one.
XEP-0096 is much more complex, I wouldn't recommend that for a start, although I would recommend it if you later extensively use large binary transfers/exchanges, as Jingle is used for VoIP at least.
Why webmails (like Gmail) sends MIME messages using multipart/alternative subtype (when composing in HTML) while others send HTML as MIME with text/html parts inside (without using alternative subtype)?
The section 5.1.4 of RFC 2046 defines multipart/alternative MIME type to allow the sender to provide different, interchangeable representations of the same message and to leave it up to the receiver to chose the form of presentation most suitable for its capabilities. Note that while the general meaning of each representation for the user should be retained, there usually is some information loss from one representation to the other (e.g. text/plain is missing the formatting information with respect to text/html). The alternatives should generally be ordered from the plainest to the richest, i.e. if the alternatives are again text/html and text/plain then text/plain should come first. This helps the users of non-MIME-conformant viewers in which the easiest to interpret part will show up first. Generally, a a MIME-conformant viewer should display the last representation it is capable of viewing since it is the most preferable.
This content type is often contrasted with multipart/mixed where a number of different resources are combined in a single message.
The main reason some mail services provide messages as multipart/alternative is to support different types of viewing applications on the receiving end. For example, some viewers lack the ability to render HTML and require text/plain representation for the message to be at all readable. At the same time, other viewers do have the ability to render HTML and can provide much better user experience when message is delivered as text/html. The most flexible solution to the trade-off between supporting wide range of viewers and enhancing user experience for the more capable ones is afforded by delivering both representations wrapped in a multipart/alternative message.
For details see RFC 2046.
multipart/alternative indicates that each part is an "alternative" version of the same (or similar) content, each in a different format denoted by its "Content-Type" header. The formats are ordered by how faithful they are to the original, with the least faithful first and the most faithful last.
Mail-agents like Gmail know what they are doing, and convert the text/html to text/plain and put both alternatives into there emails and let the receiving end decide which alternative to use.
There are also mail-agents that don't know how to extract a text-only version from the html content, just because the developer did not bother to implement it, so they only send text/html with out any alternatives.
And sometimes - i call them the crazy ones - send multipart/alternative, but actually only put text/html without any alternatives. Which is not really nice, but it is not against any spec.
The problem is this: I'm using a third-party Email delivery service that doesn't accept mail addresses with non-ASCII characters in the name part, like müller#example.com .
Encoding such an address with Punycode:
http://en.wikipedia.org/wiki/Punycode
http://idnaconv.phlymail.de/index.php?decoded=m%C3%BCller%40example.com&idn_version=2008&encode=Encode+%3E%3E&lang=de
yields this address:
xn--mller-kva#example.com
And sending mail to it via the service seems to work.
However, I'm not sure if someone couldn't register "xn--mller-kva#example.com" directly, thus receiving Emails meant for "müller#example.com".
Is this clashing possible ? Are there other solutions for this problem ?
UPDATE
Thanks for the answers. Here's a summary of what we learned:
Punycoding the local part of the email address works, and you can send and receive from such an encoded address (of course)
However, there are no guarantees at all that providers or mail clients will understand the encoding, or do it automatically. Clashes are therefore possible, and the whole idea not a good one :)
One should simply do what everyone else does, which is to not allow or accept non-ASCII name parts, as per specification
And finally, it turns out the third-party service prohibits such shenanigans anyway.
Non-ASCII characters are not allowed in the local part of email addresses. Period. Punycode is ONLY FOR DOMAINS, not for local parts of email addresses.
However, it is very likely that the IETF adopts a standard that makes internationalized local parts possible. This standard, however, will probably not be based on punycode.
I got bored and was researching this tonight, and apparently this is now codified in the Extended SMTP standard, specifically SMTPUTF8 as per RFC 6531. See http://en.wikipedia.org/wiki/Extended_SMTP#SMTPUTF8
My brief experiment using emoji mailbox names returned the following error when sending via Gmail:
local-part of envelope contains utf8 but remote server did not offer SMTPUTF8
This is the same regardless whether I used the emoji or punycode version of the address.
You can encode sections of mail header fields into different character encodings using a format like the following: =?UTF-8?B?w6HDq8O0?= This allows you to embed things like umlauts but I'm pretty sure it doesn't work for the actual address part.
There's not reason why you cannot use these characters to form your address. RFC5322 defines the characters that may appear in the address part in Section 3.4 and all the characters you use above are valid. However as the other comment added it's all a little fruitless if the mail clients that you are sending to cannot parse this format.
Some SMTP servers might 'accidentally' allow umlauts but since they're not within the supported character ranges I wouldn't risk it.
The only standard way to send non us-ascii characters in the local-part of a email address is through rfc6532 (Internationalized email headers) and rfc6531 (SMTP Extension for SMTPUTF8).
As far as I know there is no standard way to encode non us-ascii chars in a local part of a email address notably:
Puny code is for domain names only, not the local part. But you can have a local part which happens to look like the puny encoding of some string but it should be displayed in it's puny encoded form. If a mail program decides to display it after puny decoding it it's non standard behavior.
The encoded word encoding mechanism mentioned in one of the answers (the =?utf-8?Q?foobar?= thing) is not applicable to the local part of a mail address, only to the display name of a mailbox (which is something different, but related i.e. the thing your mail program might display instead of the mail address).
In the end this means that müler#example.com and xn--mler-0ra#example.com
are two completely unrelated email addresses which just would have
the same meaning if they would have been domains (but they are not
so they can collide).
Theoretically you could hope that by now (2019) all mail servers support
SMTPUTF8 and all client support internationalized mails, but sadly I would
not count on it if it's important.
Btw. it happens that the local part of a email address is the only thing in
the mail standard(s) where you might want to have non us-ascii chars and there
is no way to encode it (as far as I know). All other parts either have encoded word, puny, percent, base64, quoted-printable or some other form of encoding mechanism.
did a few tests.. umlauts in the local part seem to work in certain setups. neither my MUA (claws) nor the outbound relay (exim) nor the receiving MTA (postfix) complained or did any punycode conversion. providers like gmail and hotmail however don't allow the umlauts at all ( tested webmail and direct incoming and outgoing smtp). I didn't find any documentation about this case that suggests punycoding local parts.so, since it's not documented and no one does it there is no clashing problem :-)
conclusion: you probably shouldn't accept umlauts in the local part in the first place and not even try to send an email to those addresses. (if the big players don't do it and it's not documented/supported by RFC, why should you?)
Allow me to be more specific...
If I want to send files, but am required to wrap them in SOAP, wouldn't I use http? I am seeing a surprisng lack of info on this online.
Sending files via SOAP doesn't have anything specifically to do with FTP. To send a file through a SOAP interface, you might base64 encode the file and stuff the whole thing into a SOAP string parameter. However, this might only be appropriate if your file size has a reasonable upper bound.
If your files can be an unbounded size, you might investigate using a different transport protocol to transfer the actual file data (eg. HTTP or even FTP), then use SOAP to transfer a pointer to the file (such as its URL). Some implementations of SOAP cannot handle arbitrary large messages.
Pretty vague question but if you're using web services you can use MTOM http://en.wikipedia.org/wiki/MTOM (SOAP Message Transmission Optimization Mechanism)
I don't know your environment but there are examples of this using .NET / WCF if you Google it.
Two standard ways of sending files along with SOAP messages are:
SOAP with Attachments
MTOM
MTOM supports either using MIME attachments or base64 encoding the file into the body, where as SOAP with Attachments only supports MIME attachments.