From b3419ed8e52ed491b665f8ffe8367e7a3ced7c6e Mon Sep 17 00:00:00 2001 From: Prymmer/Kahn Date: Sun, 8 Apr 2001 17:08:58 -0700 Subject: [PATCH] documentation tweaks for UTF-EBCDIC support Message-ID: p4raw-id: //depot/perl@9655 --- lib/utf8.pm | 24 +++++++++++++----------- pod/perlunicode.pod | 17 ++++++++++------- 2 files changed, 23 insertions(+), 18 deletions(-) diff --git a/lib/utf8.pm b/lib/utf8.pm index f32c0bc..402127a 100644 --- a/lib/utf8.pm +++ b/lib/utf8.pm @@ -25,7 +25,7 @@ __END__ =head1 NAME -utf8 - Perl pragma to enable/disable UTF-8 in source code +utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code =head1 SYNOPSIS @@ -38,9 +38,9 @@ WARNING: The implementation of Unicode support in Perl is incomplete. See L for the exact details. The C pragma tells the Perl parser to allow UTF-8 in the -program text in the current lexical scope. The C pragma -tells Perl to switch back to treating the source text as literal -bytes in the current lexical scope. +program text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based +platforms). The C pragma tells Perl to switch back to treating +the source text as literal bytes in the current lexical scope. This pragma is primarily a compatibility device. Perl versions earlier than 5.6 allowed arbitrary bytes in source code, whereas @@ -48,9 +48,9 @@ in future we would like to standardize on the UTF-8 encoding for source text. Until UTF-8 becomes the default format for source text, this pragma should be used to recognize UTF-8 in the source. When UTF-8 becomes the standard source format, this pragma will -effectively become a no-op. This pragma already is a no-op on -EBCDIC platforms (where it is alright to code perl in EBCDIC -rather than UTF-8). +effectively become a no-op. For convenience in what follows the +term UTF-X is used to refer to UTF-8 on ASCII and ISO Latin based +platforms and UTF-EBCDIC on EBCDIC based platforms. Enabling the C pragma has the following effects: @@ -61,16 +61,18 @@ Enabling the C pragma has the following effects: Bytes in the source text that have their high-bit set will be treated as being part of a literal UTF-8 character. This includes most literals such as identifiers, string constants, constant regular expression patterns -and package names. +and package names. On EBCDIC platforms, characters in the C1 control group +and the Latin 1 character set are treated as being part of a literal +UTF-EBCDIC character. =item * -In the absence of inputs marked as UTF-8, regular expressions within the +In the absence of inputs marked as UTF-X, regular expressions within the scope of this pragma will default to using character semantics instead of byte semantics. @bytes_or_chars = split //, $data; # may split to bytes if data - # $data isn't UTF-8 + # $data isn't UTF-X { use utf8; # force char semantics @chars = split //, $data; # splits characters @@ -100,7 +102,7 @@ representing it in perl's UTF-X encoding. =item * $flag = utf8::decode($string) -Attempts to converts I<$string> in-place from perl's UTF-X encoding into logical characters. +Attempts to convert I<$string> in-place from perl's UTF-X encoding into logical characters. =back diff --git a/pod/perlunicode.pod b/pod/perlunicode.pod index 5b8d5be..64a14ab 100644 --- a/pod/perlunicode.pod +++ b/pod/perlunicode.pod @@ -47,7 +47,8 @@ normally be used. However, as a compatibility measure, this pragma must be explicitly used to enable recognition of UTF-8 encoded literals and identifiers in the -source text. +source text on ASCII based machines or recognize UTF-EBCDIC encoded literals +and identifiers on EBCDIC based machines. =back @@ -55,7 +56,7 @@ source text. Beginning with version 5.6, Perl uses logically wide characters to represent strings internally. This internal representation of strings -uses the UTF-8 encoding. +uses either the UTF-8 or the UTF-EBCDIC encoding. In future, Perl-level operations can be expected to work with characters rather than bytes, in general. @@ -84,7 +85,7 @@ Regardless of the above, the C pragma can always be used to force byte semantics in a particular lexical scope. See L. The C pragma is primarily a compatibility device that enables -recognition of UTF-8 in literals encountered by the parser. It may also +recognition of UTF-(8|EBCDIC) in literals encountered by the parser. It may also be used for enabling some of the more experimental Unicode support features. Note that this pragma is only required until a future version of Perl in which character semantics will become the default. This pragma may @@ -104,6 +105,8 @@ bytes change to operating on characters. For ASCII data this makes no difference, because UTF-8 stores ASCII in single bytes, but for any character greater than C, the character may be stored in a sequence of two or more bytes, all of which have the high bit set. +For C1 controls or Latin 1 characters on an EBCDIC platform the character +may be stored in a UTF-EBCDIC multi byte sequence. But by and large, the user need not worry about this, because Perl hides it from the user. A character in Perl is logically just a number ranging from 0 to 2**32 or so. Larger characters encode to longer @@ -122,9 +125,9 @@ Strings and patterns may contain characters that have an ordinal value larger than 255. Presuming you use a Unicode editor to edit your program, such characters -will typically occur directly within the literal strings as UTF-8 +will typically occur directly within the literal strings as UTF-(8|EBCDIC) characters, but you can also specify a particular character with an -extension of the C<\x> notation. UTF-8 characters are specified by +extension of the C<\x> notation. UTF-X characters are specified by putting the hexadecimal code within curlies after the C<\x>. For instance, a Unicode smiley face is C<\x{263A}>. @@ -233,8 +236,8 @@ And finally, C reverses by character rather than by byte. =head1 CAVEATS As of yet, there is no method for automatically coercing input and -output to some encoding other than UTF-8. This is planned in the near -future, however. +output to some encoding other than UTF-8 or UTF-EBCDIC. This is planned +in the near future, however. Whether an arbitrary piece of data will be treated as "characters" or "bytes" by internal operations cannot be divined at the current time. -- 2.7.4