1package utf8; 2 3$utf8::hint_bits = 0x00800000; 4 5our $VERSION = '1.02'; 6 7sub import { 8 $^H |= $utf8::hint_bits; 9 $enc{caller()} = $_[1] if $_[1]; 10} 11 12sub unimport { 13 $^H &= ~$utf8::hint_bits; 14} 15 16sub AUTOLOAD { 17 require "utf8_heavy.pl"; 18 goto &$AUTOLOAD if defined &$AUTOLOAD; 19 Carp::croak("Undefined subroutine $AUTOLOAD called"); 20} 21 221; 23__END__ 24 25=head1 NAME 26 27utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code 28 29=head1 SYNOPSIS 30 31 use utf8; 32 no utf8; 33 34 $num_octets = utf8::upgrade($string); 35 $success = utf8::downgrade($string[, FAIL_OK]); 36 37 utf8::encode($string); 38 utf8::decode($string); 39 40 $flag = utf8::is_utf8(STRING); # since Perl 5.8.1 41 $flag = utf8::valid(STRING); 42 43=head1 DESCRIPTION 44 45The C<use utf8> pragma tells the Perl parser to allow UTF-8 in the 46program text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based 47platforms). The C<no utf8> pragma tells Perl to switch back to treating 48the source text as literal bytes in the current lexical scope. 49 50This pragma is primarily a compatibility device. Perl versions 51earlier than 5.6 allowed arbitrary bytes in source code, whereas 52in future we would like to standardize on the UTF-8 encoding for 53source text. 54 55B<Do not use this pragma for anything else than telling Perl that your 56script is written in UTF-8.> The utility functions described below are 57useful for their own purposes, but they are not really part of the 58"pragmatic" effect. 59 60Until UTF-8 becomes the default format for source text, either this 61pragma or the L</encoding> pragma should be used to recognize UTF-8 62in the source. When UTF-8 becomes the standard source format, this 63pragma will effectively become a no-op. For convenience in what 64follows the term I<UTF-X> is used to refer to UTF-8 on ASCII and ISO 65Latin based platforms and UTF-EBCDIC on EBCDIC based platforms. 66 67See also the effects of the C<-C> switch and its cousin, the 68C<$ENV{PERL_UNICODE}>, in L<perlrun>. 69 70Enabling the C<utf8> pragma has the following effect: 71 72=over 4 73 74=item * 75 76Bytes in the source text that have their high-bit set will be treated 77as being part of a literal UTF-8 character. This includes most 78literals such as identifier names, string constants, and constant 79regular expression patterns. 80 81On EBCDIC platforms characters in the Latin 1 character set are 82treated as being part of a literal UTF-EBCDIC character. 83 84=back 85 86Note that if you have bytes with the eighth bit on in your script 87(for example embedded Latin-1 in your string literals), C<use utf8> 88will be unhappy since the bytes are most probably not well-formed 89UTF-8. If you want to have such bytes and use utf8, you can disable 90utf8 until the end the block (or file, if at top level) by C<no utf8;>. 91 92If you want to automatically upgrade your 8-bit legacy bytes to UTF-8, 93use the L</encoding> pragma instead of this pragma. For example, if 94you want to implicitly upgrade your ISO 8859-1 (Latin-1) bytes to UTF-8 95as used in e.g. C<chr()> and C<\x{...}>, try this: 96 97 use encoding "latin-1"; 98 my $c = chr(0xc4); 99 my $x = "\x{c5}"; 100 101In case you are wondering: yes, C<use encoding 'utf8';> works much 102the same as C<use utf8;>. 103 104=head2 Utility functions 105 106The following functions are defined in the C<utf8::> package by the 107Perl core. You do not need to say C<use utf8> to use these and in fact 108you should not say that unless you really want to have UTF-8 source code. 109 110=over 4 111 112=item * $num_octets = utf8::upgrade($string) 113 114Converts (in-place) internal representation of string to Perl's 115internal I<UTF-X> form. Returns the number of octets necessary to 116represent the string as I<UTF-X>. Can be used to make sure that the 117UTF-8 flag is on, so that C<\w> or C<lc()> work as expected on strings 118containing characters in the range 0x80-0xFF (oon ASCII and 119derivatives). Note that this should not be used to convert a legacy 120byte encoding to Unicode: use Encode for that. Affected by the 121encoding pragma. 122 123=item * $success = utf8::downgrade($string[, FAIL_OK]) 124 125Converts (in-place) internal representation of string to be un-encoded 126bytes. Returns true on success. On failure dies or, if the value of 127FAIL_OK is true, returns false. Can be used to make sure that the 128UTF-8 flag is off, e.g. when you want to make sure that the substr() 129or length() function works with the usually faster byte algorithm. 130Note that this should not be used to convert Unicode back to a legacy 131byte encoding: use Encode for that. B<Not> affected by the encoding 132pragma. 133 134=item * utf8::encode($string) 135 136Converts (in-place) I<$string> from logical characters to octet 137sequence representing it in Perl's I<UTF-X> encoding. Returns 138nothing. Same as Encode::encode_utf8(). Note that this should not be 139used to convert a legacy byte encoding to Unicode: use Encode for 140that. 141 142=item * utf8::decode($string) 143 144Attempts to convert I<$string> in-place from Perl's I<UTF-X> encoding 145into logical characters. Returns nothing. Same as Encode::decode_utf8(). 146Note that this should not be used to convert Unicode back to a legacy 147byte encoding: use Encode for that. 148 149=item * $flag = utf8::is_utf8(STRING) 150 151(Since Perl 5.8.1) Test whether STRING is in UTF-8. Functionally 152the same as Encode::is_utf8(). 153 154=item * $flag = utf8::valid(STRING) 155 156[INTERNAL] Test whether STRING is in a consistent state regarding 157UTF-8. Will return true is well-formed UTF-8 and has the UTF-8 flag 158on B<or> if string is held as bytes (both these states are 'consistent'). 159Main reason for this routine is to allow Perl's testsuite to check 160that operations have left strings in a consistent state. You most 161probably want to use utf8::is_utf8() instead. 162 163=back 164 165C<utf8::encode> is like C<utf8::upgrade>, but the UTF8 flag is 166cleared. See L<perlunicode> for more on the UTF8 flag and the C API 167functions C<sv_utf8_upgrade>, C<sv_utf8_downgrade>, C<sv_utf8_encode>, 168and C<sv_utf8_decode>, which are wrapped by the Perl functions 169C<utf8::upgrade>, C<utf8::downgrade>, C<utf8::encode> and 170C<utf8::decode>. Note that in the Perl 5.8.0 and 5.8.1 implementation 171the functions utf8::is_utf8, utf8::valid, utf8::encode, utf8::decode, 172utf8::upgrade, and utf8::downgrade are always available, without a 173C<require utf8> statement-- this may change in future releases. 174 175=head1 BUGS 176 177One can have Unicode in identifier names, but not in package/class or 178subroutine names. While some limited functionality towards this does 179exist as of Perl 5.8.0, that is more accidental than designed; use of 180Unicode for the said purposes is unsupported. 181 182One reason of this unfinishedness is its (currently) inherent 183unportability: since both package names and subroutine names may need 184to be mapped to file and directory names, the Unicode capability of 185the filesystem becomes important-- and there unfortunately aren't 186portable answers. 187 188=head1 SEE ALSO 189 190L<perluniintro>, L<encoding>, L<perlrun>, L<bytes>, L<perlunicode> 191 192=cut 193