「帶 BOM 的 UTF-8」和「無 BOM 的 UTF-8」有什麼區別？網頁代碼一般使用哪個？

Where the byte order is explicitly specified, such as in UTF-16BE or UTF-16LE, then all

U+FEFF characters—even at the very beginning of the text—are to be interpreted as zero

width no-break spaces. Similarly, where Unicode text has known byte order, initial U+FEFF

characters are not required, but for backward compatibility are to be interpreted as zero

width no-break spaces. For example, for strings in an API, the memory architecture of the

processor provides the explicit byte order. For databases and similar structures, it is much

more efficient and robust to use a uniform byte order for the same field (if not the entire

database), thereby avoiding use of the byte order mark.

Systems that use the byte order mark must recognize when an initial U+FEFF signals the

byte order. In those cases, it is not part of the textual content and should be removed before

processing, because otherwise it may be mistaken for a legitimate zero width no-break space.

To represent an initial U+FEFF zero width no-break space in a UTF-16 file, use

U+FEFF twice in a row. The first one is a byte order mark; the second one is the initial zero

width no-break space. See Table 16-4 for a summary of encoding scheme signatures.

Table 16-4. Unicode Encoding Scheme Signatures

Encoding Scheme Signature

UTF-8 EF BB BF

UTF-16 Big-endian FE FF

UTF-16 Little-endian FF FE

UTF-32 Big-endian 00 00 FE FF

UTF-32 Little-endian FF FE 00 00

If U+FEFF had only the semantics of a signature code point, it could be freely deleted from

text without affecting the interpretation of the rest of the text. Carelessly appending files

together, for example, can result in a signature code point in the middle of text. Unfortu-

nately, U+FEFF also has significance as a character. As a zero width no-break space, it indi-

cates that line breaks are not allowed between the adjoining characters. Thus U+FEFF

affects the interpretation of text and cannot be freely deleted. The overloading of semantics

The Unicode Standard 5.0 – Electronic edition Copyright ? 1991–2007 Unicode, Inc.

552 Special Areas and Format Characters

for this code point has caused problems for programs and protocols. The new character

U+2060 word joiner has the same semantics in all cases as U+FEFF, except that it cannot

be used as a signature. Implementers are strongly encouraged to use word joiner in those

circumstances whenever word joining semantics are intended.

An initial U+FEFF also takes a characteristic form in other charsets designed for Unicode

text. (The term 「charset」 refers to a wide range of text encodings, including encoding

schemes as well as compression schemes and text-specific transformation formats.) The

characteristic sequences of bytes associated with an initial U+FEFF can serve as signatures

in those cases, as shown in Table 16-5.

Table 16-5. U+FEFF Signature in Other Charsets

Charset Signature

SCSU 0E FE FF

BOCU-1 FB EE 28

UTF-7 2B 2F 76 38 or

2B 2F 76 39 or

2B 2F 76 2B or

2B 2F 76 2F

UTF-EBCDIC DD 73 66 73

Most signatures can be deleted either before or after conversion of an input stream into a

Unicode encoding form. However, in the case of BOCU-1 and UTF-7, the input byte

sequence must be converted before the initial U+FEFF can be deleted, because stripping

the signature byte sequence without conversion destroys context necessary for the correct

interpretation of subsequent bytes in the input sequence.

Specials: U+FFF0–U+FFF8

The nine unassigned Unicode code points in the range U+FFF0.. U+FFF8 are reserved for

special character definitions.

Annotation Characters: U+FFF9–U+FFFB

An interlinear annotation consists of annotating text that is related to a sequence of anno-

tated characters. For all regular editing and text-processing algorithms, the annotated char-

acters are treated as part of the text stream. The annotating text is also part of the content,

but for all or some text processing, it does not form part of the main text stream. However,

within the annotating text, characters are accessible to the same kind of layout, text-pro-

cessing, and editing algorithms as the base text. The annotation characters delimit the

annotating and the annotated text, and identify them as part of an annotation. See

Figure 16-4.

The annotation characters are used in internal processing when out-of-band information is

associated with a character stream, very similarly to the usage of U+FFFC object replace-

Copyright ? 1991-2007, Unicode, Inc. The Unicode Standard 5.0 – Electronic edition

Figure 16-4. Annotation Characters

Felix

Text display

Text stream

Annotated

text

Annotating

text

Annotation

characters

Annotated

text

Annotating

text

Annotation

characters

16.8 Specials 553

ment character. However, unlike the opaque objects hidden by the latter character, the

annotation itself is textual.

Conformance. A conformant implementation that supports annotation characters inter-

prets the base text as if it were part of an unannotated text stream. Within the annotating

text, it interprets the annotating characters with their regular Unicode semantics.

U+FFF9 interlinear annotation anchor is an anchor character, preceding the interlin-

ear annotation. The exact nature and formatting of the annotation depend on additional

information that is not part of the plain text stream. This situation is analogous to that for

U+FFFC object replacement character.

U+FFFA interlinear annotation separator separates the base characters in the text

stream from the annotation characters that follow. The exact interpretation of this charac-

ter depends on the nature of the annotation. More than one separator may be present.

Additional separators delimit parts of a multipart annotating text.

U+FFFB interlinear annotation terminator terminates the annotation object (and

returns to the regular text stream).

Use in Plain Text. Usage of the annotation characters in plain text interchange is strongly

discouraged without prior agreement between the sender and the receiver, because the con-

tent may be misinterpreted otherwise. Simply filtering out the annotation characters on

input will produce an unreadable result or, even worse, an opposite meaning. On input, a

plain text receiver should either preserve all characters or remove the interlinear annota-

tion characters as well as the annotating text included between the interlinear annota-

tion separator and the interlinear annotation terminator.

When an output for plain text usage is desired but the receiver is unknown to the sender,

these interlinear annotation characters should be removed as well as the annotating text

included between the interlinear annotation separator and the interlinear anno-

tation terminator.

This restriction does not preclude the use of annotation characters in plain text inter-

change, but it requires a prior agreement between the sender and the receiver for correct

interpretation of the annotations.

The Unicode Standard 5.0 – Electronic edition Copyright ? 1991–2007 Unicode, Inc.

554 Special Areas and Format Characters

Lexical Restrictions. If an implementation encounters a paragraph break between an

anchor and its corresponding terminator, it shall terminate any open annotations at this

point. Anchor characters must precede their corresponding terminator characters.

Unpaired anchors or terminators shall be ignored. A separator occurring outside a pair of

delimiters, shall be ignored. Annotations may be nested.

Formatting. All formatting information for an annotation is provided by higher-level pro-

tocols. The details of the layout of the annotation are implementation-defined. Correct for-

matting may require additional information that is not present in the character stream, but

rather is maintained out-of-band. Therefore, annotation markers serve as placeholders for

an implementation that has access to that information from another source. The format-

ting of annotations and other special line layout features of Japanese is discussed in JIS X

4501.

Input. Annotation characters are not normally input or edited directly by end users. Their

insertion and management in text are typically handled by an application, which will

present a user interface for selecting and annotating text.

Collation. With the exception of the special case where the annotation is intended to be

used as a sort key, annotations are typically ignored for collation or optionally preprocessed

to act as tie breakers only. Importantly, annotation base characters are not ignored, but

rather are treated like regular text.

Replacement Characters: U+FFFC–U+FFFD

U+FFFC. The U+FFFC object replacement character is used as an insertion point for

objects located within a stream of text. All other information about the object is kept out-

side the character data stream. Internally it is a dummy character that acts as an anchor

point for the object』s formatting information. In addition to assuring correct placement of

an object in a data stream, the object replacement character allows the use of general

stream-based algorithms for any textual aspects of embedded objects.

U+FFFD. The U+FFFD replacement character is the general substitute character in

the Unicode Standard. It can be substituted for any 「unknown」 character in another

encoding that cannot be mapped in terms of known Unicode characters (see Section 5.3,

Unknown and Missing Characters).

為了各自的利益把UTF-8搞得烏煙瘴氣