When using 8BITMIME smtp, you can set Content-Transfer-Encoding: 8bit in Mime messages and send text without encoding.
Except, there is still a line limit of 1000 octets (plus the line endings should all be <CR><LF>)
When my library gets arbitrary UTF-8 data from a user, how should I go about splitting lines? Is there any way to split a 1002 octets line in a safe way? And what about a 1002 octet word (without whitespace).
In Quoted-Printable you can do =<CR><LF>, is there something similar for 8bit?
There is no way for 8bit to have longer lines, just like there is no way for 7bit to (legitimately) contain 8-bit characters. If you want arbitrarily long lines, the binary content type is available, but the standard, robust approach is to use a content-transfer-encoding such as quoted-printable or base64. Then the content within the encoding can be completely free-form.
Related
While sending email content, it is required to set "Content Transfer Encoding" header. I observed many headers of emails that I received. Some emails using "7bit" and some are using "8bit".
What is the difference between these two? Which is recommended? Is there any special encoding required for email body in order to set these headers?
It can be a bit dense to read, but the "Content-Transfer-Encoding" section of RFC 1341 has all of the details:
http://www.w3.org/Protocols/rfc1341/5_Content-Transfer-Encoding.html
The situation kinda goes from bad to worse. Here's my summary:
Background
SMTP, by definition (RFC 821), limits mail to lines of 1000 characters of 7 bits each. That means that none of the bytes you send down the pipe can have the most significant ("highest-order") bit set to "1".
The content that we want to send will often not obey this restriction inherently. Think of an image file, or a text file that contains Unicode characters: the bytes of these files will often have their 8th bit set to "1". SMTP doesn't allow this, so you need to use "transfer encoding" to describe how you've worked around the mismatch.
The values for the Content-Transfer-Encoding header describe the rule that you've chosen to solve this problem.
7Bit Encoding
7bit simply means "My data consists only of US-ASCII characters, which only use the lower 7 bits for each character." You're basically guaranteeing that all of the bytes in your content already adhere to the restrictions of SMTP, and so it needs no special treatment. You can just read it as-is.
Note that when you choose 7bit, you're agreeing that all of the lines in your content are less than 1000 characters in length.
As long as your content adheres to these rule, 7bit is the best transfer encoding, since there's no extra work necessary; you just read/write the bytes as they come off the pipe. It's also easy to eyeball 7bit content and make sense of it. The idea here is that if you're just writing in "plain English text" you'll be fine. But that wasn't true in 2005 and it isn't true today.
8Bit Encoding
8bit means "My data may include extended ASCII characters; they may use the 8th (highest) bit to indicate special characters outside of the standard US-ASCII 7-bit characters." As with 7bit, there's still a 1000-character line limit.
8bit, just like 7bit, does not actually do any transformation of the bytes as they're written to or read from the wire. It just means that you're not guaranteeing that none of the bytes will have the highest bit set to "1".
This seems like a step up from 7bit, since it gives you more freedom in your content. However, RFC 1341 contains this tidbit:
As of the publication of this document, there are no standardized Internet transports for which it is legitimate to include unencoded 8-bit or binary data in mail bodies. Thus there are no circumstances in which the "8bit" or "binary" Content-Transfer-Encoding is actually legal on the Internet.
RFC 1341 came out over 20 years ago. Since then we've gotten 8bit MIME Extensions in RFC 6152. But even then, line limits still may apply:
Note that this extension does NOT eliminate the possibility of an SMTP server limiting line length; servers are free to implement this extension but nevertheless set a line length limit no lower than 1000 octets.
Binary Encoding
binary is the same as 8bit, except that there's no line length restriction. You can still include any characters you want, and there's no extra encoding. Similar to 8bit, RFC 1341 states that it's not really a legitimate encoding transfer encoding. RFC 3030 extended this with BINARYMIME.
Quoted Printable
Before the 8BITMIME extension, there needed to be a way to send content that couldn't be 7bit over SMTP. HTML files (which might have more than 1000-character lines) and files with international characters are good examples of this. The quoted-printable encoding (Defined in Section 5.1 of RFC 1341) is designed to handle this. It does two things:
Defines how to escape non-US-ASCII characters so that they can be represented in only 7-bit characters. (Short version: they get displayed as an equals sign plus two 7-bit characters.)
Defines that lines will be no greater than 76 characters, and that line breaks will be represented using special characters (which are then escaped).
Quoted Printable, because of the escaping and short lines, is much harder to read by a human than 7bit or 8bit, but it does support a much wider range of possible content.
Base64 Encoding
If your data is largely non-text (ex: an image file), you don't have many options. 7bit is off the table. 8bit and binary were unsupported prior to the MIME extension RFCs. quoted-printable would work, but is really inefficient (every byte is going to be represented by 3 characters).
base64 is a good solution for this type of data. It encodes 3 raw bytes as 4 US-ASCII characters, which is relatively efficient. RFC 1341 further limits the line length of base64-encoded data to 76 characters to fit within an SMTP message, but that's relatively easy to manage when you're just splitting or concatenating arbitrary characters at fixed lengths.
The big downside is that base64-encoded data is pretty much entirely unreadable by humans, even if it's just "plain" text underneath.
With content-transfer-encoding: 7bit the bytes that are used in body (or more correct within part's boundaries) should represent ascii characters but not extended-ascii characters. This means 0-127 decimal (8th bit not used).
Since 8th bit is not used it means that you cannot encode your text using utf-8 or iso8859-7 bytes because they use the 8th bit. Nor you can add binary content.
With content-transfer-encoding: 8bit you can use any possible byte which means that you can encode your text using utf-8 bytes or iso8859-7 bytes (both assuming that 8BITMIME extension is used in SMTP). You are however still unsafe adding binary content due to the max line-restriction that still applies which could break your bytes with newlines.
Now even with 7bit content-transfer-encoding you can still set content-type's charset param to utf-8 as long as you still keep your bytes between the boundaries of 0-127.
e.g. A possible way to represent characters outside ascii using the 7bit content-transfer-encoding could be to use html code characters (with content-type: text/html)
Many email clients will set content-transfer-encoding to 7bit or 8bit depending on the case. E.g. 7bit when sending english text, 8bit when sending multilingual text. And there are always the options of quoted-printable and base64 whose characters are also not using 8th bit, but this is out of scope of the
question.
I have an app that sends e-mails and for many months, it was working fine. I recently had problems with utf-8 encoded emails sent to iPhone Exchange account (i.e. not IMAP).
All the receiver had to see was a big bunch of meaningless characters like LS0tLS0tPV9QYXJ0XzE0N18[....].
Comparing my email headers with Gmail, I noticed that I had an extra Content-Transfer-Encoding associated with the Content-Type: multipart/alternative;.
My email would look like
Delivered-To: ...
Received: ...
...
MIME-Version: ...
Content-Type: multipart/alternative;
boundary="----=_boundary"
Content-Transfer-Encoding: Base64 # <= the extra setting
----=_boundary
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: Base64
YmVu0Cg==
----=_boundary
Content-type: text/html; charset=utf-8
Content-Transfer-Encoding: Base64
PGh0bWwgeG1sbnM6bz0iIj48aGVhZD48dGl0bGU+PC90aXRsZT48L2hlYWQ+PGJvZHk+YmVub2l0
PC9ib2R5PjwvaHRtbD4NCjx9IjAiIC8+Cg==
----=_boundary
If I remove the extra setting, my email is received and display properly.
My questions:
Is the Encoding setting basically needed with Content-Type: multipart/alternative;, even for specific cases ?
Is it safe to remove it and just keep using my app as I used to ?
Edit
I found on IETF:
Encoding considerations: Multipart content-types cannot have encodings.
But I also found on Wikipedia:
The content-transfer-encoding of a multipart type must always be
"7bit", "8bit" or "binary" to avoid the complications that would be
posed by multiple levels of decoding.
Isn't it contradictory ?
The statements from IETF and wikipedia aren't really contradictory. 7bit, 8bit, or binary aren't really content encodings in that they don't specify any transformation of the content. They simply state that the content hasn't been encoded. In the case of 7bit it also specifies that the content doesn't need to be encoded even if the message needs to be sent over a transport that isn't 8-bit clean.
Only the bottom-most layers of messages should have an actual Content-Transfer-Encoding such as base64 or quoted-printable. In the message that you quote from the outer portion certainly isn't base64 encoded, so stating that it is was not only violating the standard but also incorrect. That would certainly be expected to cause problems with display of that message.
In practice, each part of a multipart has its own encoding, and it doesn't make sense to declare one for the multipart container. I cannot make sense of the Wikipedia quote anyway; in any event, it is hardly authoritative.
I parsing email message and i found part with encoding: 7bit
how a can convert text of this part to plain text?
i use perl
Content-Transfer-Encoding: 7bit
means that the text is already plain old ASCII text. No conversion is necessary. (Well, unless the Content-Type header indicates a non-ASCII-based charset, but those are pretty rare, especially with 7bit text.)
It sounds like you have UU-encoded data (older method) or MIME-encoded. To deal with that, you can use Convert::UU and MIME::Base64 CPAN modules respectively.
To use MIME::Base64 (or its pure-Perl implementation, MIME::Base64::Perl):
use MIME::Base64::Perl;
my $decoded = decode_base64($encoded);
How do you know the difference?
The modern MIME encoded text looks like this (Especially pay attention to MIME-Version: header which tells you it's MIME-encoded as well as Content-Transfer-Encoding header which tells you the encoding base - if it's not base64, you need a different CPAN module:
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="frontier"
This is a message with multiple parts in MIME format.
--frontier
Content-Type: text/plain
This is the body of the message.
--frontier
Content-Type: application/octet-stream
Content-Transfer-Encoding: base64
UU-encoded text would look something like:
begin 644 cat.txt
#0V%T
`
end
If the encoded data looks differently than either of the above samples, please post the exact format so we can determine what it is.
Some of the mails contents fetched from imap server looks like =C3=B6=C3=BC=C3=B6=C3=BC=C3=B6=C3=BC= what kind of encoding is this? Mail header encoding is UTF-8 but decoding with UTF-8 i got scrambled msg.
Any help is much appreciated.
Quoted-Printable
It is used to transmit 8-bit data over a 7-bit medium.
Characters are converted from 8-bit to three 7-bit characters in the form =XX where XX is the hexadecimal character code for the 8-bit character, the = character will become =3D.
The length of a line is restricted to 76 characters, soft line breaks are added to comply with this rule, this is done by ending with a = to indicate that the line should continue.
https://www.rfc-editor.org/rfc/rfc2045
http://en.wikipedia.org/wiki/Quoted-printable
Online Decoder
Is there any practical difference between "7bit" and "quoted-printable" as Content-Transfer-Encoding in email? From all I could gather the encoding schemes are virtually identical.
For example, in 7bit, you can have a space at the end of a line, but in quoted-printable, you have to write it as =20 (which would be interpreted literally by 7bit).