1package utf8; 2 3$utf8::hint_bits = 0x00800000; 4 5our $VERSION = '1.03'; 6 7sub import { 8 $^H |= $utf8::hint_bits; 9 $enc{caller()} = $_[1] if $_[1]; 10} 11 12sub unimport { 13 $^H &= ~$utf8::hint_bits; 14} 15 16sub AUTOLOAD { 17 require "utf8_heavy.pl"; 18 goto &$AUTOLOAD if defined &$AUTOLOAD; 19 Carp::croak("Undefined subroutine $AUTOLOAD called"); 20} 21 221; 23__END__ 24 25=head1 NAME 26 27utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code 28 29=head1 SYNOPSIS 30 31 use utf8; 32 no utf8; 33 34 # Convert a Perl scalar to/from UTF-8. 35 $num_octets = utf8::upgrade($string); 36 $success = utf8::downgrade($string[, FAIL_OK]); 37 38 # Change the native bytes of a Perl scalar to/from UTF-8 bytes. 39 utf8::encode($string); 40 utf8::decode($string); 41 42 $flag = utf8::is_utf8(STRING); # since Perl 5.8.1 43 $flag = utf8::valid(STRING); 44 45=head1 DESCRIPTION 46 47The C<use utf8> pragma tells the Perl parser to allow UTF-8 in the 48program text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based 49platforms). The C<no utf8> pragma tells Perl to switch back to treating 50the source text as literal bytes in the current lexical scope. 51 52This pragma is primarily a compatibility device. Perl versions 53earlier than 5.6 allowed arbitrary bytes in source code, whereas 54in future we would like to standardize on the UTF-8 encoding for 55source text. 56 57B<Do not use this pragma for anything else than telling Perl that your 58script is written in UTF-8.> The utility functions described below are 59useful for their own purposes, but they are not really part of the 60"pragmatic" effect. 61 62Until UTF-8 becomes the default format for source text, either this 63pragma or the L</encoding> pragma should be used to recognize UTF-8 64in the source. When UTF-8 becomes the standard source format, this 65pragma will effectively become a no-op. For convenience in what 66follows the term I<UTF-X> is used to refer to UTF-8 on ASCII and ISO 67Latin based platforms and UTF-EBCDIC on EBCDIC based platforms. 68 69See also the effects of the C<-C> switch and its cousin, the 70C<$ENV{PERL_UNICODE}>, in L<perlrun>. 71 72Enabling the C<utf8> pragma has the following effect: 73 74=over 4 75 76=item * 77 78Bytes in the source text that have their high-bit set will be treated 79as being part of a literal UTF-8 character. This includes most 80literals such as identifier names, string constants, and constant 81regular expression patterns. 82 83On EBCDIC platforms characters in the Latin 1 character set are 84treated as being part of a literal UTF-EBCDIC character. 85 86=back 87 88Note that if you have bytes with the eighth bit on in your script 89(for example embedded Latin-1 in your string literals), C<use utf8> 90will be unhappy since the bytes are most probably not well-formed 91UTF-8. If you want to have such bytes and use utf8, you can disable 92utf8 until the end the block (or file, if at top level) by C<no utf8;>. 93 94If you want to automatically upgrade your 8-bit legacy bytes to UTF-8, 95use the L</encoding> pragma instead of this pragma. For example, if 96you want to implicitly upgrade your ISO 8859-1 (Latin-1) bytes to UTF-8 97as used in e.g. C<chr()> and C<\x{...}>, try this: 98 99 use encoding "latin-1"; 100 my $c = chr(0xc4); 101 my $x = "\x{c5}"; 102 103In case you are wondering: yes, C<use encoding 'utf8';> works much 104the same as C<use utf8;>. 105 106=head2 Utility functions 107 108The following functions are defined in the C<utf8::> package by the 109Perl core. You do not need to say C<use utf8> to use these and in fact 110you should not say that unless you really want to have UTF-8 source code. 111 112=over 4 113 114=item * $num_octets = utf8::upgrade($string) 115 116Converts (in-place) internal representation of string to Perl's 117internal I<UTF-X> form. Returns the number of octets necessary to 118represent the string as I<UTF-X>. Can be used to make sure that the 119UTF-8 flag is on, so that C<\w> or C<lc()> work as expected on strings 120containing characters in the range 0x80-0xFF (oon ASCII and 121derivatives). Note that this should not be used to convert a legacy 122byte encoding to Unicode: use Encode for that. Affected by the 123encoding pragma. 124 125=item * $success = utf8::downgrade($string[, FAIL_OK]) 126 127Converts (in-place) internal representation of string to be un-encoded 128bytes. Returns true on success. On failure dies or, if the value of 129FAIL_OK is true, returns false. Can be used to make sure that the 130UTF-8 flag is off, e.g. when you want to make sure that the substr() 131or length() function works with the usually faster byte algorithm. 132Note that this should not be used to convert Unicode back to a legacy 133byte encoding: use Encode for that. B<Not> affected by the encoding 134pragma. 135 136=item * utf8::encode($string) 137 138Converts in-place the octets of the I<$string> to the octet sequence 139in Perl's I<UTF-X> encoding. Returns nothing. B<Note that this does 140not change the "type" of I<$string> to UTF-8>, and that this handles 141only ISO 8859-1 (or EBCDIC) as the source character set. Therefore 142this should not be used to convert a legacy 8-bit encoding to Unicode: 143use Encode::decode() for that. In the very limited case of wanting to 144handle just ISO 8859-1 (or EBCDIC), you could use utf8::upgrade(). 145 146=item * utf8::decode($string) 147 148Attempts to convert I<$string> in-place from Perl's I<UTF-X> encoding 149into octets. Returns nothing. B<Note that this does not change the 150"type" of <$string> from UTF-8>, and that this handles only ISO 8859-1 151(or EBCDIC) as the destination character set. Therefore this should 152not be used to convert Unicode back to a legacy 8-bit encoding: 153use Encode::encode() for that. In the very limited case of wanting 154to handle just ISO 8859-1 (or EBCDIC), you could use utf8::downgrade(). 155 156=item * $flag = utf8::is_utf8(STRING) 157 158(Since Perl 5.8.1) Test whether STRING is in UTF-8. Functionally 159the same as Encode::is_utf8(). 160 161=item * $flag = utf8::valid(STRING) 162 163[INTERNAL] Test whether STRING is in a consistent state regarding 164UTF-8. Will return true is well-formed UTF-8 and has the UTF-8 flag 165on B<or> if string is held as bytes (both these states are 'consistent'). 166Main reason for this routine is to allow Perl's testsuite to check 167that operations have left strings in a consistent state. You most 168probably want to use utf8::is_utf8() instead. 169 170=back 171 172C<utf8::encode> is like C<utf8::upgrade>, but the UTF8 flag is 173cleared. See L<perlunicode> for more on the UTF8 flag and the C API 174functions C<sv_utf8_upgrade>, C<sv_utf8_downgrade>, C<sv_utf8_encode>, 175and C<sv_utf8_decode>, which are wrapped by the Perl functions 176C<utf8::upgrade>, C<utf8::downgrade>, C<utf8::encode> and 177C<utf8::decode>. Note that in the Perl 5.8.0 and 5.8.1 implementation 178the functions utf8::is_utf8, utf8::valid, utf8::encode, utf8::decode, 179utf8::upgrade, and utf8::downgrade are always available, without a 180C<require utf8> statement-- this may change in future releases. 181 182=head1 BUGS 183 184One can have Unicode in identifier names, but not in package/class or 185subroutine names. While some limited functionality towards this does 186exist as of Perl 5.8.0, that is more accidental than designed; use of 187Unicode for the said purposes is unsupported. 188 189One reason of this unfinishedness is its (currently) inherent 190unportability: since both package names and subroutine names may need 191to be mapped to file and directory names, the Unicode capability of 192the filesystem becomes important-- and there unfortunately aren't 193portable answers. 194 195=head1 SEE ALSO 196 197L<perluniintro>, L<encoding>, L<perlrun>, L<bytes>, L<perlunicode> 198 199=cut 200