Re: [PATCH] (Re: [PATCH] unicode/utf8 pod)

Message-ID: <20070304150019.GN4723@c4.convolution.nl>

p4raw-id: //depot/perl@30493

1 files changed, 51 insertions, 65 deletions

diff --git a/lib/utf8.pm b/lib/utf8.pm
index 5ff900d8f9..f8c1c1033a 100644
--- a/lib/utf8.pm
+++ b/lib/utf8.pm
@@ -50,22 +50,18 @@ program text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based
 platforms).  The C<no utf8> pragma tells Perl to switch back to treating
 the source text as literal bytes in the current lexical scope.
 
-This pragma is primarily a compatibility device.  Perl versions
-earlier than 5.6 allowed arbitrary bytes in source code, whereas
-in future we would like to standardize on the UTF-8 encoding for
-source text.
-
 B<Do not use this pragma for anything else than telling Perl that your
 script is written in UTF-8.> The utility functions described below are
-useful for their own purposes, but they are not really part of the
-"pragmatic" effect.
+directly usable without C<use utf8;>.
+
+Because it is not possible to reliably tell UTF-8 from native 8 bit
+encodings, you need either a Byte Order Mark at the beginning of your
+source code, or C<use utf8;>, to instruct perl.
 
-Until UTF-8 becomes the default format for source text, either this
-pragma or the L<encoding> pragma should be used to recognize UTF-8
-in the source.  When UTF-8 becomes the standard source format, this
-pragma will effectively become a no-op.  For convenience in what
-follows the term I<UTF-X> is used to refer to UTF-8 on ASCII and ISO
-Latin based platforms and UTF-EBCDIC on EBCDIC based platforms.
+When UTF-8 becomes the standard source format, this pragma will
+effectively become a no-op.  For convenience in what follows the term
+I<UTF-X> is used to refer to UTF-8 on ASCII and ISO Latin based
+platforms and UTF-EBCDIC on EBCDIC based platforms.
 
 See also the effects of the C<-C> switch and its cousin, the
 C<$ENV{PERL_UNICODE}>, in L<perlrun>.
@@ -93,21 +89,6 @@ UTF-X.  If you want to have such bytes under C<use utf8>, you can disable
 this pragma until the end the block (or file, if at top level) by
 C<no utf8;>.
 
-If you want to automatically upgrade your 8-bit legacy bytes to Unicode,
-use the L<encoding> pragma instead of this pragma.  For example, if
-you want to implicitly upgrade your ISO 8859-1 (Latin-1) bytes to Unicode
-as used in e.g. C<chr()> and C<\x{...}>, try this:
-
-    use encoding "latin-1";
-    my $c = chr(0xc4);
-    my $x = "\x{c5}";
-
-In case you are wondering: C<use encoding 'utf8';> is mostly the same as
-C<use utf8;>, except that C<use encoding> marks all string literals in the
-source code as Unicode, regardless of whether they contain any high-bit bytes.
-Moreover, C<use encoding> installs IO layers on C<STDIN> and C<STDOUT> to work
-with Unicode strings; see L<encoding> for details.
-
 =head2 Utility functions
 
 The following functions are defined in the C<utf8::> package by the
@@ -118,64 +99,69 @@ you should not say that  unless you really want to have UTF-8 source code.
 
 =item * $num_octets = utf8::upgrade($string)
 
-Converts in-place the octet sequence in the native encoding
+Converts in-place the internal octet sequence in the native encoding
 (Latin-1 or EBCDIC) to the equivalent character sequence in I<UTF-X>.
-I<$string> already encoded as characters does no harm.
-Returns the number of octets necessary to represent the string as I<UTF-X>.
-Can be used to make sure that the UTF-8 flag is on,
-so that C<\w> or C<lc()> work as Unicode on strings
-containing characters in the range 0x80-0xFF (on ASCII and
-derivatives).
+I<$string> already encoded as characters does no harm.  Returns the
+number of octets necessary to represent the string as I<UTF-X>.  Can be
+used to make sure that the UTF-8 flag is on, so that C<\w> or C<lc()>
+work as Unicode on strings containing characters in the range 0x80-0xFF
+(on ASCII and derivatives).
 
 B<Note that this function does not handle arbitrary encodings.>
-Therefore I<Encode.pm> is recommended for the general purposes.
-
-Affected by the encoding pragma.
+Therefore Encode is recommended for the general purposes; see also
+L<Encode>.
 
 =item * $success = utf8::downgrade($string[, FAIL_OK])
 
-Converts in-place the character sequence in I<UTF-X>
-to the equivalent octet sequence in the native encoding (Latin-1 or EBCDIC).
-I<$string> already encoded as octets does no harm.
-Returns true on success. On failure dies or, if the value of
-C<FAIL_OK> is true, returns false.
-Can be used to make sure that the UTF-8 flag is off,
-e.g. when you want to make sure that the substr() or length() function
-works with the usually faster byte algorithm.
+Converts in-place the internal octet sequence in I<UTF-X> to the
+equivalent octet sequence in the native encoding (Latin-1 or EBCDIC).
+I<$string> already encoded as native 8 bit does no harm.  Can be used to
+make sure that the UTF-8 flag is off, e.g. when you want to make sure
+that the substr() or length() function works with the usually faster
+byte algorithm.
 
-B<Note that this function does not handle arbitrary encodings.>
-Therefore I<Encode.pm> is recommended for the general purposes.
+Fails if the original I<UTF-X> sequence cannot be represented in the
+native 8 bit encoding. On failure dies or, if the value of C<FAIL_OK> is
+true, returns false. 
 
-B<Not> affected by the encoding pragma.
+Returns true on success.
+
+B<Note that this function does not handle arbitrary encodings.>
+Therefore Encode is recommended for the general purposes; see also
+L<Encode>.
 
-B<NOTE:> this function is experimental and may change
-or be removed without notice.
+B<NOTE:> this function is experimental and may change or be removed
+without notice.
 
 =item * utf8::encode($string)
 
-Converts in-place the character sequence to the corresponding octet sequence
-in I<UTF-X>.  The UTF-8 flag is turned off.  Returns nothing.
+Converts in-place the character sequence to the corresponding octet
+sequence in I<UTF-X>.  The UTF8 flag is turned off, so that after this
+operation, the string is a byte string.  Returns nothing.
 
 B<Note that this function does not handle arbitrary encodings.>
-Therefore I<Encode.pm> is recommended for the general purposes.
+Therefore Encode is recommended for the general purposes; see also
+L<Encode>.
 
-=item * utf8::decode($string)
+=item * $success = utf8::decode($string)
 
-Attempts to convert in-place the octet sequence in I<UTF-X>
-to the corresponding character sequence.  The UTF-8 flag is turned on
-only if the source string contains multiple-byte I<UTF-X> characters.
-If I<$string> is invalid as I<UTF-X>, returns false; otherwise returns true.
+Attempts to convert in-place the octet sequence in I<UTF-X> to the
+corresponding character sequence.  The UTF-8 flag is turned on only if
+the source string contains multiple-byte I<UTF-X> characters.  If
+I<$string> is invalid as I<UTF-X>, returns false; otherwise returns
+true.
 
 B<Note that this function does not handle arbitrary encodings.>
-Therefore I<Encode.pm> is recommended for the general purposes.
+Therefore Encode is recommended for the general purposes; see also
+L<Encode>.
 
-B<NOTE:> this function is experimental and may change
-or be removed without notice.
+B<NOTE:> this function is experimental and may change or be removed
+without notice.
 
 =item * $flag = utf8::is_utf8(STRING)
 
-(Since Perl 5.8.1)  Test whether STRING is in UTF-8.  Functionally
-the same as Encode::is_utf8().
+(Since Perl 5.8.1)  Test whether STRING is in UTF-8 internally.
+Functionally the same as Encode::is_utf8().
 
 =item * $flag = utf8::valid(STRING)
 
@@ -213,6 +199,6 @@ portable answers.
 
 =head1 SEE ALSO
 
-L<perluniintro>, L<encoding>, L<perlrun>, L<bytes>, L<perlunicode>
+L<perlunitut>, L<perluniintro>, L<perlrun>, L<bytes>, L<perlunicode>
 
 =cut