Download rfc1642.pdf

Contents : Network Working Group Request for Comments: 1642 Category: Experimental D. Goldsmith M. Davis Taligent Inc. July 1994 UTF-7 A Mail-Safe Transformation Format of Unicode Status of this Memo This memo defines an Experimental Protocol for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited. Abstract The Unicode Standard version 1.1 and ISO/IEC 10646-1:1993(E) jointly define a 16 bit character set (hereafter referred to as Unicode) which encompasses most of the world's writing systems. However Internet mail (STD 11 RFC 822) currently supports only 7-bit US ASCII as a character set. MIME (RFC 1521 and RFC 1522) extends Internet mail to support different media types and character sets and thus could support Unicode in mail messages. MIME neither defines Unicode as a permitted character set nor specifies how it would be encoded although it does provide for the registration of additional character sets over time. This document describes a new transformation format of Unicode that contains only 7-bit ASCII characters and is intended to be readable by humans in the limiting case that the document consists of characters from the US-ASCII repertoire. It also specifies how this transformation format is used in the context of RFC 1521 RFC 1522 and the document "Using Unicode with MIME". Motivation Although other transformation formats of Unicode exist and could conceivably be used in this context (most notably UTF-1 and UTF-8 also known as UTF-2 or UTF-FSS) they suffer the disadvantage that they use octets in the range decimal 128 through 255 to encode Unicode characters outside the US-ASCII range. Thus in the context of mail those octets must themselves be encoded. This requires putting text through two successive encoding processes and leads to a significant expansion of characters outside the US-ASCII range putting non-English speakers at a disadvantage. For example using UTF-FSS together with the Quoted-Printable content transfer encoding of MIME represents US-ASCII characters in one octet but other characters may require up to nine octets. Goldsmith & Davis Page 1 RFC 1642 UTF-7 July 1994 Overview UTF-7 encodes Unicode characters as US-ASCII together with shift sequences to encode characters outside that range. For this purpose one of the characters in the US-ASCII repertoire is reserved for use as a shift character. Many mail gateways and systems cannot handle the entire US-ASCII character set (those based on EBCDIC for example) and so UTF-7 contains provisions for encoding characters within US-ASCII in a way that all mail systems can accomodate. UTF-7 should normally be used only in the context of 7 bit transports such as mail and news. In other contexts straight Unicode or UTF-8 is preferred. See the document "Using Unicode with MIME" for the overall specification on usage of Unicode transformation formats with MIME. Definitions First the definition of Unicode: The 16 bit character set Unicode is defined by "The Unicode Standard Version 1.1". This character set is identical with the character repertoire and coding of the international standard ISO/IEC 10646-1:1993(E) Coded Representation Form UCS-2 Subset 300 Implementation Level 3. Note . Unicode 1.1 further specifies the use and interaction of these character codes beyond the ISO standard. However any valid 10646 BMP (Basic Multilingual Plane) sequence is a valid Unicode sequence and vice versa Unicode supplies interpretations of sequences on which the ISO standard is silent as to interpretation. Next some handy definitions of US-ASCII character subsets: Set D (directly encoded characters) consists of the following characters (derived from RFC 1521 Appendix B): the upper and lower case letters A through Z and a through z the 10 digits 0-9 and the following nine special characters (note that "+" and " " are omitted): Goldsmith & Davis Page 2 RFC 1642 Character ' ( ) . / : UTF-7 ASCII & Unicode Value (decimal) 39 40 41 44 45 46 47 58 63 July 1994 Set O (optional direct characters) consists of the following characters (note that " " and " " are omitted): Character ! " # $ % & * @ ASCII & Unicode Value (decimal) 33 34 35 36 37 38 42 59 60 61 62 64 91 93 94 95 96 123 124 125 Rationale. The characters " " and " " are omitted because they are often redefined in variants of ASCII. Set B (Modified Base 64) is the set of characters in the Base64 alphabet defined in RFC 1521 excluding the pad character " " (decimal value 61). Goldsmith & Davis Page 3 RFC 1642 UTF-7 July 1994 Rationale. The pad character " " is excluded because UTF-7 is designed for use within header fields as set forth in RFC 1522. Since the only readable encoding in RFC 1522 is "Q" (based on RFC 1521's Quoted-Printable) the " " character is not available for use (without a lot of escape sequences). This was very unfortunate but unavoidable.The " " character could otherwise have been used as the UTF-7 escape character as well (rather than using "+"). Note that all characters in US-ASCII have the same value in Unicode when zero-extended to 16 bits. UTF-7 Definition A UTF-7 stream represents 16-bit Unicode characters in 7-bit US-ASCII as follows: Rule 1: (direct encoding) Unicode characters in set D above may be encoded directly as their ASCII equivalents. Unicode characters in Set O may optionally be encoded directly as their ASCII equivalents bearing in mind that many of these characters are illegal in header fields or may not pass correctly through some mail gateways. Rule 2: (Unicode shifted encoding) Any Unicode character sequence may be encoded using a sequence of characters in set B when preceded by the shift character "+" (USASCII character value decimal 43). The "+" signals that subsequent octets are to be interpreted as elements of the Modified Base64 alphabet until a character not in that alphabet is encountered. Such characters include control characters such as carriage returns and line feeds thus a Unicode shifted sequence always terminates at the end of a line. As a special case if the sequence terminates with the character " -" (US-ASCII decimal 45) then that character is absorbed other terminating characters are not absorbed and are processed normally. Rationale. A terminating character is necessary for cases where the next character after the Modified Base64 sequence is part of character set B. It can also enhance readability by delimiting encoded sequences. Also as a special case the sequence " +-" may be used to encode the character "+". A "+" character followed immediately by any character other than members of set B or "-" is an ill-formed sequence. Unicode is encoded using Modified Base64 by first converting Unicode 16-bit quantities to an octet stream (with the most significant octet first). Text with an odd number of octets is ill-formed. Rationale. ISO/IEC 10646-1:1993(E) specifies that when characters in the UCS-2 form are serialized as octets that the most significant octet appear first. This is also in keeping with common network practice of choosing a canonical format for transmission. Goldsmith & Davis Page 4 RFC 1642 UTF-7 July 1994 Next the octet stream is encoded by applying the Base64 content transfer encoding algorithm as defined in RFC 1521 modified to omit the " " pad character. Instead when encoding zero bits are added to pad to a Base64 character boundary. When decoding any bits at the end of the Modified Base64 sequence that do not constitute a complete 16bit Unicode character are discarded. If such discarded bits are non-zero the sequence is ill-formed. Rationale. The pad character " " is not used when encoding Modified Base64 because of the conflict with its use as an escape character for the Q content transfer encoding in RFC 1522 header fields as mentioned above. Rule 3: The space (decimal 32) tab (decimal 9) carriage return (decimal 13) and line feed (decimal 10) characters may be directly represented by their ASCII equivalents. However note that MIME content transfer encodings have rules concerning the use of such characters. Usage that does not conform to the restrictions of RFC 822 for example would have to be encoded using MIME content transfer encodings other than 7bit or 8bit such as quoted-printable binary or base64. Given this set of rules Unicode characters which may be encoded via rules 1 or 3 take one octet per character and other Unicode characters are encoded on average with 2 2/3 octets per character plus one octet to switch into Modified Base64 and an optional octet to switch out. Example. The Unicode sequence "A NOT IDENTICAL TO ALPHA ." (hexadecimal 0041 2262 0391 002E) may be encoded as follows: A+ImIDkQ. Example. The Unicode sequence "Hi Mom WHITE SMILING FACE !" (hexadecimal 0048 0069 0020 004D 006F 004D 0020 263A 0021) may be encoded as follows: Hi Mom +Jjo-! Example. The Unicode sequence representing the Han characters for the Japanese word "nihongo" (hexadecimal 65E5 672C 8A9E) may be encoded as follows: +ZeVnLIqe- Use of Character Set UTF-7 Within MIME Character set UTF-7 is safe for mail transmission and therefore may be used with any content transfer encoding in MIME (except where line length and line break restrictions are violated). Specifically the 7 bit encoding for bodies and the Q encoding for headers are both acceptable. The MIME character set identifier is UNICODE-1-1-UTF-7. Goldsmith & Davis Page 5

INFO HASH : 27d533f3bee0f196fbe2641ee03c753731f2446c

Download now

File Size: 28.9 kb