diff options
| -rw-r--r-- | pod/perlunicode.pod | 5 | ||||
| -rw-r--r-- | utf8.h | 4 |
2 files changed, 9 insertions, 0 deletions
diff --git a/pod/perlunicode.pod b/pod/perlunicode.pod index ad3637f567..26e704a63e 100644 --- a/pod/perlunicode.pod +++ b/pod/perlunicode.pod @@ -737,6 +737,11 @@ The following table is from Unicode 3.2. Note the A0..BF in U+0800..U+0FFF, the 80..9F in U+D000...U+D7FF, the 90..BF in U+10000..U+3FFFF, and the 80...8F in U+100000..U+10FFFF. +The "gaps" are caused by legal UTF-8 avoiding non-shortest encodings: +it is technically possible to UTF-8-encode a single code point in different +ways, but that is explicitly forbidden, and the shortest possible encoding +should always be used (and that is what Perl does). + Or, another way to look at it, as bits: Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte @@ -80,6 +80,10 @@ END_EXTERN_C Note the A0..BF in U+0800..U+0FFF, the 80..9F in U+D000...U+D7FF, the 90..BF in U+10000..U+3FFFF, and the 80...8F in U+100000..U+10FFFF. +The "gaps" are caused by legal UTF-8 avoiding non-shortest encodings: +it is technically possible to UTF-8-encode a single code point in different +ways, but that is explicitly forbidden, and the shortest possible encoding +should always be used (and that is what Perl does). */ |