When parsing a collection of MBox files, I came across a surprising number of Date header fields of the form:
"Date:" date-time "<" addr-spec ">"
Reading the available RFCs I find no matching syntax. The valid form appears to be:
"Date:" date-time [CFWS]
CFWS represents comments plus folding whitespace as described in Section 3.3 (Date and Time Specification) of RFC5322.
Reading Proper Mail Date Header Formatting where the author analyses 132k date headers, yet the form above never shows in the linked dataset.
Is this an MBox artefact, IMF attribute, or corruption from a mail agent and/or mail relay?
The Internet Message Format has been evolving since the 1980s, it's a bit of a mess and has been interpreted a number of ways à la HTTP. I'm leaning toward this being a vendor-specific modification that results in a malformed Date header field? IDK.
Example MBox
From ???#???
From: from#mail.com
To: to#mail.com
Date: Sat, 02 Feb 2002 00:55:01 +0000 <name#mail.com>
Subject: Email One
This is email 1.
Specifications
RFC 5322
RFC 2822
RFC 822
RFC 4021
MBox Format
MBOXO Format
MBOXRD Format
MBOXCL Format
MBOXCL2 Format
RFC 5321
RFC 2821
Related
One email provider rejected an email containing special characters (e.g. umlaute). They say that they are RFC-5321 and RFC-5322 compliant. Now I browsed those standards however they are not supporting international emails (thus no umlaute). Only ASCII-127 is supported.
Now there is an extension called RFC-6532 which standardizes international emails. Our emails are UTF-8 (quoted-printable) encoded and sent like this:
"=?UTF-8?Q?B=C3=B6rge_M=C3=B6ller?="<boerge.moeller#foo.org>
Is this an RFC-6532 compliant address? Or is it some other/older RFC (like RFC-2054)? After all there are so many mail related RFCs that I might have missed 10 or 20 ;-)
It's on the right track, but it's wrong.
"=?UTF-8?Q?B=C3=B6rge_M=C3=B6ller?="<boerge.moeller#foo.org>
There are 2 problems with the above form:
The encoded-word (the =?UTF-8?Q?...?= bit) is quoted and shouldn't be. Mail software that parse this address won't decode that token if they are standards-compliant.
The "name" is butted up against the angle brackets and should not be. There MUST be a space in order to be standards compliant.
In other words, this is what it should look like:
=?UTF-8?Q?B=C3=B6rge_M=C3=B6ller?= <boerge.moeller#foo.org>
The RFCs that you need to look at are:
RFC5322 - this defines the modern Message syntax that is implemented by the server you are trying to interoperate with.
RFC2047 - this defines the methods and syntax of the encoded-words that are needed to represent non-ASCII characters in headers like Subject and address headers (e.g. To/From/Cc/Reply-To/etc). (This is the =?UTF-8?Q?B=C3=B6rge_M=C3=B6ller?= part)
RFC822 - this defines the grammar used by RFC2047 and is an older version of RFC5322.
It may also be helpful to read RFC2822 which is newer than RFC822 but older than RFC5322. My guess, however, is that you can skip it because it won't have a lot of value. The only reason RFC822 still has value is because of its much older grammar definitions that are referenced by RFC2047 (such as atom, dot-atom, phrase, angle-addr, addr-spec, tspecials, etc).
RFC6532 is even newer than RFC5322. The purpose of which is to remove the need to encode headers altogether by allowing the use of UTF-8 as an alternative.
Before RFC6532, there was no standard for the character encoding to use for headers other than ASCII (which was what RFC822 used) and so everything was always supposed to conform to ASCII.
A lot of software doesn't follow the standards, however, and so there was a lot of mail in the real world that used ISO-8859-1 and every other character encoding under the sun, all depending on what region the user(s) were in and what character encoding(s) were in wide use in those regions (e.g. Big5 and GB2312 are popular in various parts of China, Shift-JIS being popular in Japan, EUC-KR/KS-C-5601-1987 are popular in Korea, etc).
This caused major interoperability problems, though, not least of which because not every mail client could handle every character encoding under the sun, but also because there was no way for a client to figure out which character encoding was even being used! It's all just binary gobbeldy-gook.
UTF-8, however, has existed for a long time and it can represent all characters in all languages, so it was only logical for it to eventually win out as the standard character encoding to use for international email.
It's about sending emails with non ASCII chars in the email address.
When I use send the TO /RCPT stuff to the SMTP server I know that I need to use punycode here.
But what about the To: and From: Header. Again I know that if the User friendly part contains a non ascii char I con use the standard header encoding that I also use for the subject. But this encoding is only used for the user friendly part.
But what if the email address contains a non ascii char? How must the To header be formatted.
So how to encode "Tüst" ?
This is the encoding as far as I know.
"=?iso-8859-1?Q?T=FCst?="<tüst#domain.de>
But what with the email address.
In fact: I don't understand the RFC's. I tried hard but failed.
The answer is: UTF-8 is the correct way to encode the header.
After some more research I found the answer hidden inside this article:
https://en.wikipedia.org/wiki/International_email
Although the traditional format for email header section allows
non-ASCII characters to be included in the value portion of some of
the header fields using MIME-encoded words (e.g. in display names or
in a Subject header field), MIME-encoding must not be used to encode
other information in a header, such as an email address, or header
fields like Message-ID or Received. Moreover, the MIME-encoding
requires extra processing of the header to convert the data to and
from its MIME-encoded word representation, and harms readability of a
header section.
The 2012 standards RFC 6532 and RFC 6531 allow the inclusion of
Unicode characters in a header content using UTF-8 encoding, and their
transmission via SMTP - but in practice support is only slowly rolling
out.[5]
I have to make a regular expression for the email addresses (RFC 822) and I want to know which characters are allowed in the local part and in the domain.
I found this https://www.rfc-editor.org/rfc/rfc822#section-6.1 but I don't see that it says which are the valid characters.
According to RFC 822, the local part may contain any ASCII character, since local-part is defined using word, which is defined as atom / quoted-string; atom covers most ASCII characters, and the rest can be written in a quoted-string. There are syntactic restrictions, but obeying them, any ASCII character can be used.
On similar grounds, RFC 822 allows any ASCII character in the domain part.
On the other hand, RFC 822 was obsoleted in 2001 by RFC 2822, which in turn was obsoleted in 2008 by RFC 5322. The status of RFCs can be checked from the RFC Editor’s RFC database.
I have a raw email with headers that look like this:
From xxxx#xxxx Fri Apr 25 22:46:08 2003
>From xxxx#mxxxx Wed Feb 19 20:06:07 2003
Envelope-to: yyyy#xxxx
...
Date: Wed, 19 Feb 2003 22:05:59 +0500
From: "Actual Author" <xxxx#xxxx>
I don't know how to interpret the first two lines, and the initial reading of RFC2822 has left me without a clue. They don't look like normal headers and manage to confuse Python 2.7 email parser (fine if I remove the > sign at the start of the second line). I have the same email body in Apple mail's cache, and it seems fine, so the input is clearly correct.
What's that header format? (From <email> <date>\r\n)
Why is the second one prefixed with > (greater sign)?
What you have is a mail in mbox format, where the first "From" line marks the start of the message. The second line (>From) seems to be caused by the escaping strategy of mbox known as From quoting - has this message been double-encoded as mbox?
Are german umlauts (ä, ö, ü) and the sz-character (ß) valid in the local part of an email-address?
For example take this email-address: björn.nußbaum#trouble.org
RFC 5322 quite clearly says, that umlauts (and other international characters) aren't allowed. If I take a look at chapter 3.4.1, there's the following regarding the local part:
local-part = dot-atom / quoted-string / obs-local-part
So what means dot-atom? It's described in chapter 3.2.3: Well, long story short: Printable US-ASCII characters not including specials
So in the whole RFC 5322 I can't see anything regarding international characters.
Or is RFC 5322 already obsolete? (RFC 822 -> RFC 2822 -> RFC 5322)
Update:
The important point for me is: What's the current standard? International characters allowed or not?
RFC 5322 is marked as DRAFT STANDARD. So I think that's the most recent source to rely on, isn't it?
Efran mentioned, that RFC 5336 allows international characters. But RFC 5336 is marked as EXPERIMENTAL, so that's not interesting for me.
Yes, they are valid characters as long as the mail exchanger responsible for the email address supports the UTF8SMTP extension, discussed in RFC 5336. Beware that just a small portion of the mail exchangers out there supports internationalized email addresses.
Both our email validation component for Microsoft .NET and our REST email validation service, for example, allow UTF8 characters in the local part of an email address but will mark it as invalid if its related mail exchanger does not support the aforementioned extension.
https://www.rfc-editor.org/rfc/rfc5322#section-3.4.1 is your latest standards track reference. Generally it is not advisable to use characters which require quoting due to the outrageously high amount of standards unconformant MTAs out there. Such email are bound to get lost in the long run.
As a friendly advice this table is pretty useful too (from Jochen Topf, titled "Characters in the local part of an email address"): https://www.jochentopf.com/email/chars.html
It looks like rfc6531 replaces 5336 and it is "PROPOSED STANDARD"
https://www.rfc-editor.org/rfc/rfc6531