1=head1 NAME 2 3perlunicode - Unicode support in Perl 4 5=head1 DESCRIPTION 6 7If you haven't already, before reading this document, you should become 8familiar with both L<perlunitut> and L<perluniintro>. 9 10Unicode aims to B<UNI>-fy the en-B<CODE>-ings of all the world's 11character sets into a single Standard. For quite a few of the various 12coding standards that existed when Unicode was first created, converting 13from each to Unicode essentially meant adding a constant to each code 14point in the original standard, and converting back meant just 15subtracting that same constant. For ASCII and ISO-8859-1, the constant 16is 0. For ISO-8859-5, (Cyrillic) the constant is 864; for Hebrew 17(ISO-8859-8), it's 1488; Thai (ISO-8859-11), 3424; and so forth. This 18made it easy to do the conversions, and facilitated the adoption of 19Unicode. 20 21And it worked; nowadays, those legacy standards are rarely used. Most 22everyone uses Unicode. 23 24Unicode is a comprehensive standard. It specifies many things outside 25the scope of Perl, such as how to display sequences of characters. For 26a full discussion of all aspects of Unicode, see 27L<https://www.unicode.org>. 28 29=head2 Important Caveats 30 31Even though some of this section may not be understandable to you on 32first reading, we think it's important enough to highlight some of the 33gotchas before delving further, so here goes: 34 35Unicode support is an extensive requirement. While Perl does not 36implement the Unicode standard or the accompanying technical reports 37from cover to cover, Perl does support many Unicode features. 38 39Also, the use of Unicode may present security issues that aren't 40obvious, see L</Security Implications of Unicode> below. 41 42=over 4 43 44=item Safest if you C<use feature 'unicode_strings'> 45 46In order to preserve backward compatibility, Perl does not turn 47on full internal Unicode support unless the pragma 48L<S<C<use feature 'unicode_strings'>>|feature/The 'unicode_strings' feature> 49is specified. (This is automatically 50selected if you S<C<use 5.012>> or higher.) Failure to do this can 51trigger unexpected surprises. See L</The "Unicode Bug"> below. 52 53This pragma doesn't affect I/O. Nor does it change the internal 54representation of strings, only their interpretation. There are still 55several places where Unicode isn't fully supported, such as in 56filenames. 57 58=item Input and Output Layers 59 60Use the C<:encoding(...)> layer to read from and write to 61filehandles using the specified encoding. (See L<open>.) 62 63=item You must convert your non-ASCII, non-UTF-8 Perl scripts to be 64UTF-8. 65 66The L<encoding> module has been deprecated since perl 5.18 and the 67perl internals it requires have been removed with perl 5.26. 68 69=item C<use utf8> still needed to enable L<UTF-8|/Unicode Encodings> in scripts 70 71If your Perl script is itself encoded in L<UTF-8|/Unicode Encodings>, 72the S<C<use utf8>> pragma must be explicitly included to enable 73recognition of that (in string or regular expression literals, or in 74identifier names). B<This is the only time when an explicit S<C<use 75utf8>> is needed.> (See L<utf8>). 76 77If a Perl script begins with the bytes that form the UTF-8 encoding of 78the Unicode BYTE ORDER MARK (C<BOM>, see L</Unicode Encodings>), those 79bytes are completely ignored. 80 81=item L<UTF-16|/Unicode Encodings> scripts autodetected 82 83If a Perl script begins with the Unicode C<BOM> (UTF-16LE, 84UTF16-BE), or if the script looks like non-C<BOM>-marked 85UTF-16 of either endianness, Perl will correctly read in the script as 86the appropriate Unicode encoding. 87 88=back 89 90=head2 Byte and Character Semantics 91 92Before Unicode, most encodings used 8 bits (a single byte) to encode 93each character. Thus a character was a byte, and a byte was a 94character, and there could be only 256 or fewer possible characters. 95"Byte Semantics" in the title of this section refers to 96this behavior. There was no need to distinguish between "Byte" and 97"Character". 98 99Then along comes Unicode which has room for over a million characters 100(and Perl allows for even more). This means that a character may 101require more than a single byte to represent it, and so the two terms 102are no longer equivalent. What matter are the characters as whole 103entities, and not usually the bytes that comprise them. That's what the 104term "Character Semantics" in the title of this section refers to. 105 106Perl had to change internally to decouple "bytes" from "characters". 107It is important that you too change your ideas, if you haven't already, 108so that "byte" and "character" no longer mean the same thing in your 109mind. 110 111The basic building block of Perl strings has always been a "character". 112The changes basically come down to that the implementation no longer 113thinks that a character is always just a single byte. 114 115There are various things to note: 116 117=over 4 118 119=item * 120 121String handling functions, for the most part, continue to operate in 122terms of characters. C<length()>, for example, returns the number of 123characters in a string, just as before. But that number no longer is 124necessarily the same as the number of bytes in the string (there may be 125more bytes than characters). The other such functions include 126C<chop()>, C<chomp()>, C<substr()>, C<pos()>, C<index()>, C<rindex()>, 127C<sort()>, C<sprintf()>, and C<write()>. 128 129The exceptions are: 130 131=over 4 132 133=item * 134 135the bit-oriented C<vec> 136 137E<nbsp> 138 139=item * 140 141the byte-oriented C<pack>/C<unpack> C<"C"> format 142 143However, the C<W> specifier does operate on whole characters, as does the 144C<U> specifier. 145 146=item * 147 148some operators that interact with the platform's operating system 149 150Operators dealing with filenames are examples. 151 152=item * 153 154when the functions are called from within the scope of the 155S<C<L<use bytes|bytes>>> pragma 156 157Likely, you should use this only for debugging anyway. 158 159=back 160 161=item * 162 163Strings--including hash keys--and regular expression patterns may 164contain characters that have ordinal values larger than 255. 165 166If you use a Unicode editor to edit your program, Unicode characters may 167occur directly within the literal strings in UTF-8 encoding, or UTF-16. 168(The former requires a C<use utf8>, the latter may require a C<BOM>.) 169 170L<perluniintro/Creating Unicode> gives other ways to place non-ASCII 171characters in your strings. 172 173=item * 174 175The C<chr()> and C<ord()> functions work on whole characters. 176 177=item * 178 179Regular expressions match whole characters. For example, C<"."> matches 180a whole character instead of only a single byte. 181 182=item * 183 184The C<tr///> operator translates whole characters. (Note that the 185C<tr///CU> functionality has been removed. For similar functionality to 186that, see C<pack('U0', ...)> and C<pack('C0', ...)>). 187 188=item * 189 190C<scalar reverse()> reverses by character rather than by byte. 191 192=item * 193 194The bit string operators, C<& | ^ ~> and (starting in v5.22) 195C<&. |. ^. ~.> can operate on bit strings encoded in UTF-8, but this 196can give unexpected results if any of the strings contain code points 197above 0xFF. Starting in v5.28, it is a fatal error to have such an 198operand. Otherwise, the operation is performed on a non-UTF-8 copy of 199the operand. If you're not sure about the encoding of a string, 200downgrade it before using any of these operators; you can use 201L<C<utf8::utf8_downgrade()>|utf8/Utility functions>. 202 203=back 204 205The bottom line is that Perl has always practiced "Character Semantics", 206but with the advent of Unicode, that is now different than "Byte 207Semantics". 208 209=head2 ASCII Rules versus Unicode Rules 210 211Before Unicode, when a character was a byte was a character, 212Perl knew only about the 128 characters defined by ASCII, code points 0 213through 127 (except for under L<S<C<use locale>>|perllocale>). That 214left the code 215points 128 to 255 as unassigned, and available for whatever use a 216program might want. The only semantics they have is their ordinal 217numbers, and that they are members of none of the non-negative character 218classes. None are considered to match C<\w> for example, but all match 219C<\W>. 220 221Unicode, of course, assigns each of those code points a particular 222meaning (along with ones above 255). To preserve backward 223compatibility, Perl only uses the Unicode meanings when there is some 224indication that Unicode is what is intended; otherwise the non-ASCII 225code points remain treated as if they are unassigned. 226 227Here are the ways that Perl knows that a string should be treated as 228Unicode: 229 230=over 231 232=item * 233 234Within the scope of S<C<use utf8>> 235 236If the whole program is Unicode (signified by using 8-bit B<U>nicode 237B<T>ransformation B<F>ormat), then all literal strings within it must be 238Unicode. 239 240=item * 241 242Within the scope of 243L<S<C<use feature 'unicode_strings'>>|feature/The 'unicode_strings' feature> 244 245This pragma was created so you can explicitly tell Perl that operations 246executed within its scope are to use Unicode rules. More operations are 247affected with newer perls. See L</The "Unicode Bug">. 248 249=item * 250 251Within the scope of S<C<use 5.012>> or higher 252 253This implicitly turns on S<C<use feature 'unicode_strings'>>. 254 255=item * 256 257Within the scope of 258L<S<C<use locale 'not_characters'>>|perllocale/Unicode and UTF-8>, 259or L<S<C<use locale>>|perllocale> and the current 260locale is a UTF-8 locale. 261 262The former is defined to imply Unicode handling; and the latter 263indicates a Unicode locale, hence a Unicode interpretation of all 264strings within it. 265 266=item * 267 268When the string contains a Unicode-only code point 269 270Perl has never accepted code points above 255 without them being 271Unicode, so their use implies Unicode for the whole string. 272 273=item * 274 275When the string contains a Unicode named code point C<\N{...}> 276 277The C<\N{...}> construct explicitly refers to a Unicode code point, 278even if it is one that is also in ASCII. Therefore the string 279containing it must be Unicode. 280 281=item * 282 283When the string has come from an external source marked as 284Unicode 285 286The L<C<-C>|perlrun/-C [numberE<sol>list]> command line option can 287specify that certain inputs to the program are Unicode, and the values 288of this can be read by your Perl code, see L<perlvar/"${^UNICODE}">. 289 290=item * When the string has been upgraded to UTF-8 291 292The function L<C<utf8::utf8_upgrade()>|utf8/Utility functions> 293can be explicitly used to permanently (unless a subsequent 294C<utf8::utf8_downgrade()> is called) cause a string to be treated as 295Unicode. 296 297=item * There are additional methods for regular expression patterns 298 299A pattern that is compiled with the C<< /u >> or C<< /a >> modifiers is 300treated as Unicode (though there are some restrictions with C<< /a >>). 301Under the C<< /d >> and C<< /l >> modifiers, there are several other 302indications for Unicode; see L<perlre/Character set modifiers>. 303 304=back 305 306Note that all of the above are overridden within the scope of 307C<L<use bytes|bytes>>; but you should be using this pragma only for 308debugging. 309 310Note also that some interactions with the platform's operating system 311never use Unicode rules. 312 313When Unicode rules are in effect: 314 315=over 4 316 317=item * 318 319Case translation operators use the Unicode case translation tables. 320 321Note that C<uc()>, or C<\U> in interpolated strings, translates to 322uppercase, while C<ucfirst>, or C<\u> in interpolated strings, 323translates to titlecase in languages that make the distinction (which is 324equivalent to uppercase in languages without the distinction). 325 326There is a CPAN module, C<L<Unicode::Casing>>, which allows you to 327define your own mappings to be used in C<lc()>, C<lcfirst()>, C<uc()>, 328C<ucfirst()>, and C<fc> (or their double-quoted string inlined versions 329such as C<\U>). (Prior to Perl 5.16, this functionality was partially 330provided in the Perl core, but suffered from a number of insurmountable 331drawbacks, so the CPAN module was written instead.) 332 333=item * 334 335Character classes in regular expressions match based on the character 336properties specified in the Unicode properties database. 337 338C<\w> can be used to match a Japanese ideograph, for instance; and 339C<[[:digit:]]> a Bengali number. 340 341=item * 342 343Named Unicode properties, scripts, and block ranges may be used (like 344bracketed character classes) by using the C<\p{}> "matches property" 345construct and the C<\P{}> negation, "doesn't match property". 346 347See L</"Unicode Character Properties"> for more details. 348 349You can define your own character properties and use them 350in the regular expression with the C<\p{}> or C<\P{}> construct. 351See L</"User-Defined Character Properties"> for more details. 352 353=back 354 355=head2 Extended Grapheme Clusters (Logical characters) 356 357Consider a character, say C<H>. It could appear with various marks around it, 358such as an acute accent, or a circumflex, or various hooks, circles, arrows, 359I<etc.>, above, below, to one side or the other, I<etc>. There are many 360possibilities among the world's languages. The number of combinations is 361astronomical, and if there were a character for each combination, it would 362soon exhaust Unicode's more than a million possible characters. So Unicode 363took a different approach: there is a character for the base C<H>, and a 364character for each of the possible marks, and these can be variously combined 365to get a final logical character. So a logical character--what appears to be a 366single character--can be a sequence of more than one individual characters. 367The Unicode standard calls these "extended grapheme clusters" (which 368is an improved version of the no-longer much used "grapheme cluster"); 369Perl furnishes the C<\X> regular expression construct to match such 370sequences in their entirety. 371 372But Unicode's intent is to unify the existing character set standards and 373practices, and several pre-existing standards have single characters that 374mean the same thing as some of these combinations, like ISO-8859-1, 375which has quite a few of them. For example, C<"LATIN CAPITAL LETTER E 376WITH ACUTE"> was already in this standard when Unicode came along. 377Unicode therefore added it to its repertoire as that single character. 378But this character is considered by Unicode to be equivalent to the 379sequence consisting of the character C<"LATIN CAPITAL LETTER E"> 380followed by the character C<"COMBINING ACUTE ACCENT">. 381 382C<"LATIN CAPITAL LETTER E WITH ACUTE"> is called a "pre-composed" 383character, and its equivalence with the "E" and the "COMBINING ACCENT" 384sequence is called canonical equivalence. All pre-composed characters 385are said to have a decomposition (into the equivalent sequence), and the 386decomposition type is also called canonical. A string may be comprised 387as much as possible of precomposed characters, or it may be comprised of 388entirely decomposed characters. Unicode calls these respectively, 389"Normalization Form Composed" (NFC) and "Normalization Form Decomposed". 390The C<L<Unicode::Normalize>> module contains functions that convert 391between the two. A string may also have both composed characters and 392decomposed characters; this module can be used to make it all one or the 393other. 394 395You may be presented with strings in any of these equivalent forms. 396There is currently nothing in Perl 5 that ignores the differences. So 397you'll have to specially handle it. The usual advice is to convert your 398inputs to C<NFD> before processing further. 399 400For more detailed information, see L<http://unicode.org/reports/tr15/>. 401 402=head2 Unicode Character Properties 403 404(The only time that Perl considers a sequence of individual code 405points as a single logical character is in the C<\X> construct, already 406mentioned above. Therefore "character" in this discussion means a single 407Unicode code point.) 408 409Very nearly all Unicode character properties are accessible through 410regular expressions by using the C<\p{}> "matches property" construct 411and the C<\P{}> "doesn't match property" for its negation. 412 413For instance, C<\p{Uppercase}> matches any single character with the Unicode 414C<"Uppercase"> property, while C<\p{L}> matches any character with a 415C<General_Category> of C<"L"> (letter) property (see 416L</General_Category> below). Brackets are not 417required for single letter property names, so C<\p{L}> is equivalent to C<\pL>. 418 419More formally, C<\p{Uppercase}> matches any single character whose Unicode 420C<Uppercase> property value is C<True>, and C<\P{Uppercase}> matches any character 421whose C<Uppercase> property value is C<False>, and they could have been written as 422C<\p{Uppercase=True}> and C<\p{Uppercase=False}>, respectively. 423 424This formality is needed when properties are not binary; that is, if they can 425take on more values than just C<True> and C<False>. For example, the 426C<Bidi_Class> property (see L</"Bidirectional Character Types"> below), 427can take on several different 428values, such as C<Left>, C<Right>, C<Whitespace>, and others. To match these, one needs 429to specify both the property name (C<Bidi_Class>), AND the value being 430matched against 431(C<Left>, C<Right>, I<etc.>). This is done, as in the examples above, by having the 432two components separated by an equal sign (or interchangeably, a colon), like 433C<\p{Bidi_Class: Left}>. 434 435All Unicode-defined character properties may be written in these compound forms 436of C<\p{I<property>=I<value>}> or C<\p{I<property>:I<value>}>, but Perl provides some 437additional properties that are written only in the single form, as well as 438single-form short-cuts for all binary properties and certain others described 439below, in which you may omit the property name and the equals or colon 440separator. 441 442Most Unicode character properties have at least two synonyms (or aliases if you 443prefer): a short one that is easier to type and a longer one that is more 444descriptive and hence easier to understand. Thus the C<"L"> and 445C<"Letter"> properties above are equivalent and can be used 446interchangeably. Likewise, C<"Upper"> is a synonym for C<"Uppercase">, 447and we could have written C<\p{Uppercase}> equivalently as C<\p{Upper}>. 448Also, there are typically various synonyms for the values the property 449can be. For binary properties, C<"True"> has 3 synonyms: C<"T">, 450C<"Yes">, and C<"Y">; and C<"False"> has correspondingly C<"F">, 451C<"No">, and C<"N">. But be careful. A short form of a value for one 452property may not mean the same thing as the short form spelled the same 453for another. 454Thus, for the C<L</General_Category>> property, C<"L"> means 455C<"Letter">, but for the L<C<Bidi_Class>|/Bidirectional Character Types> 456property, C<"L"> means C<"Left">. A complete list of properties and 457synonyms is in L<perluniprops>. 458 459Upper/lower case differences in property names and values are irrelevant; 460thus C<\p{Upper}> means the same thing as C<\p{upper}> or even C<\p{UpPeR}>. 461Similarly, you can add or subtract underscores anywhere in the middle of a 462word, so that these are also equivalent to C<\p{U_p_p_e_r}>. And white space 463is generally irrelevant adjacent to non-word characters, such as the 464braces and the equals or colon separators, so C<\p{ Upper }> and 465C<\p{ Upper_case : Y }> are equivalent to these as well. In fact, white 466space and even hyphens can usually be added or deleted anywhere. So 467even C<\p{ Up-per case = Yes}> is equivalent. All this is called 468"loose-matching" by Unicode. The "name" property has some restrictions 469on this due to a few outlier names. Full details are given in 470L<https://www.unicode.org/reports/tr44/tr44-24.html#UAX44-LM2>. 471 472The few places where stricter matching is 473used is in the middle of numbers, the "name" property, and in the Perl 474extension properties that begin or end with an underscore. Stricter 475matching cares about white space (except adjacent to non-word 476characters), hyphens, and non-interior underscores. 477 478You can also use negation in both C<\p{}> and C<\P{}> by introducing a caret 479(C<^>) between the first brace and the property name: C<\p{^Tamil}> is 480equal to C<\P{Tamil}>. 481 482Almost all properties are immune to case-insensitive matching. That is, 483adding a C</i> regular expression modifier does not change what they 484match. There are two sets that are affected. 485The first set is 486C<Uppercase_Letter>, 487C<Lowercase_Letter>, 488and C<Titlecase_Letter>, 489all of which match C<Cased_Letter> under C</i> matching. 490And the second set is 491C<Uppercase>, 492C<Lowercase>, 493and C<Titlecase>, 494all of which match C<Cased> under C</i> matching. 495This set also includes its subsets C<PosixUpper> and C<PosixLower> both 496of which under C</i> match C<PosixAlpha>. 497(The difference between these sets is that some things, such as Roman 498numerals, come in both upper and lower case so they are C<Cased>, but 499aren't considered letters, so they aren't C<Cased_Letter>'s.) 500 501See L</Beyond Unicode code points> for special considerations when 502matching Unicode properties against non-Unicode code points. 503 504=head3 B<General_Category> 505 506Every Unicode character is assigned a general category, which is the "most 507usual categorization of a character" (from 508L<https://www.unicode.org/reports/tr44>). 509 510The compound way of writing these is like C<\p{General_Category=Number}> 511(short: C<\p{gc:n}>). But Perl furnishes shortcuts in which everything up 512through the equal or colon separator is omitted. So you can instead just write 513C<\pN>. 514 515Here are the short and long forms of the values the C<General Category> property 516can have: 517 518 Short Long 519 520 L Letter 521 LC, L& Cased_Letter (that is: [\p{Ll}\p{Lu}\p{Lt}]) 522 Lu Uppercase_Letter 523 Ll Lowercase_Letter 524 Lt Titlecase_Letter 525 Lm Modifier_Letter 526 Lo Other_Letter 527 528 M Mark 529 Mn Nonspacing_Mark 530 Mc Spacing_Mark 531 Me Enclosing_Mark 532 533 N Number 534 Nd Decimal_Number (also Digit) 535 Nl Letter_Number 536 No Other_Number 537 538 P Punctuation (also Punct) 539 Pc Connector_Punctuation 540 Pd Dash_Punctuation 541 Ps Open_Punctuation 542 Pe Close_Punctuation 543 Pi Initial_Punctuation 544 (may behave like Ps or Pe depending on usage) 545 Pf Final_Punctuation 546 (may behave like Ps or Pe depending on usage) 547 Po Other_Punctuation 548 549 S Symbol 550 Sm Math_Symbol 551 Sc Currency_Symbol 552 Sk Modifier_Symbol 553 So Other_Symbol 554 555 Z Separator 556 Zs Space_Separator 557 Zl Line_Separator 558 Zp Paragraph_Separator 559 560 C Other 561 Cc Control (also Cntrl) 562 Cf Format 563 Cs Surrogate 564 Co Private_Use 565 Cn Unassigned 566 567Single-letter properties match all characters in any of the 568two-letter sub-properties starting with the same letter. 569C<LC> and C<L&> are special: both are aliases for the set consisting of everything matched by C<Ll>, C<Lu>, and C<Lt>. 570 571=head3 B<Bidirectional Character Types> 572 573Because scripts differ in their directionality (Hebrew and Arabic are 574written right to left, for example) Unicode supplies a C<Bidi_Class> property. 575Some of the values this property can have are: 576 577 Value Meaning 578 579 L Left-to-Right 580 LRE Left-to-Right Embedding 581 LRO Left-to-Right Override 582 R Right-to-Left 583 AL Arabic Letter 584 RLE Right-to-Left Embedding 585 RLO Right-to-Left Override 586 PDF Pop Directional Format 587 EN European Number 588 ES European Separator 589 ET European Terminator 590 AN Arabic Number 591 CS Common Separator 592 NSM Non-Spacing Mark 593 BN Boundary Neutral 594 B Paragraph Separator 595 S Segment Separator 596 WS Whitespace 597 ON Other Neutrals 598 599This property is always written in the compound form. 600For example, C<\p{Bidi_Class:R}> matches characters that are normally 601written right to left. Unlike the 602C<L</General_Category>> property, this 603property can have more values added in a future Unicode release. Those 604listed above comprised the complete set for many Unicode releases, but 605others were added in Unicode 6.3; you can always find what the 606current ones are in L<perluniprops>. And 607L<https://www.unicode.org/reports/tr9/> describes how to use them. 608 609=head3 B<Scripts> 610 611The world's languages are written in many different scripts. This sentence 612(unless you're reading it in translation) is written in Latin, while Russian is 613written in Cyrillic, and Greek is written in, well, Greek; Japanese mainly in 614Hiragana or Katakana. There are many more. 615 616The Unicode C<Script> and C<Script_Extensions> properties give what 617script a given character is in. The C<Script_Extensions> property is an 618improved version of C<Script>, as demonstrated below. Either property 619can be specified with the compound form like 620C<\p{Script=Hebrew}> (short: C<\p{sc=hebr}>), or 621C<\p{Script_Extensions=Javanese}> (short: C<\p{scx=java}>). 622In addition, Perl furnishes shortcuts for all 623C<Script_Extensions> property names. You can omit everything up through 624the equals (or colon), and simply write C<\p{Latin}> or C<\P{Cyrillic}>. 625(This is not true for C<Script>, which is required to be 626written in the compound form. Prior to Perl v5.26, the single form 627returned the plain old C<Script> version, but was changed because 628C<Script_Extensions> gives better results.) 629 630The difference between these two properties involves characters that are 631used in multiple scripts. For example the digits '0' through '9' are 632used in many parts of the world. These are placed in a script named 633C<Common>. Other characters are used in just a few scripts. For 634example, the C<"KATAKANA-HIRAGANA DOUBLE HYPHEN"> is used in both Japanese 635scripts, Katakana and Hiragana, but nowhere else. The C<Script> 636property places all characters that are used in multiple scripts in the 637C<Common> script, while the C<Script_Extensions> property places those 638that are used in only a few scripts into each of those scripts; while 639still using C<Common> for those used in many scripts. Thus both these 640match: 641 642 "0" =~ /\p{sc=Common}/ # Matches 643 "0" =~ /\p{scx=Common}/ # Matches 644 645and only the first of these match: 646 647 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Common} # Matches 648 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Common} # No match 649 650And only the last two of these match: 651 652 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Hiragana} # No match 653 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Katakana} # No match 654 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Hiragana} # Matches 655 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Katakana} # Matches 656 657C<Script_Extensions> is thus an improved C<Script>, in which there are 658fewer characters in the C<Common> script, and correspondingly more in 659other scripts. It is new in Unicode version 6.0, and its data are likely 660to change significantly in later releases, as things get sorted out. 661New code should probably be using C<Script_Extensions> and not plain 662C<Script>. If you compile perl with a Unicode release that doesn't have 663C<Script_Extensions>, the single form Perl extensions will instead refer 664to the plain C<Script> property. If you compile with a version of 665Unicode that doesn't have the C<Script> property, these extensions will 666not be defined at all. 667 668(Actually, besides C<Common>, the C<Inherited> script, contains 669characters that are used in multiple scripts. These are modifier 670characters which inherit the script value 671of the controlling character. Some of these are used in many scripts, 672and so go into C<Inherited> in both C<Script> and C<Script_Extensions>. 673Others are used in just a few scripts, so are in C<Inherited> in 674C<Script>, but not in C<Script_Extensions>.) 675 676It is worth stressing that there are several different sets of digits in 677Unicode that are equivalent to 0-9 and are matchable by C<\d> in a 678regular expression. If they are used in a single language only, they 679are in that language's C<Script> and C<Script_Extensions>. If they are 680used in more than one script, they will be in C<sc=Common>, but only 681if they are used in many scripts should they be in C<scx=Common>. 682 683The explanation above has omitted some detail; refer to UAX#24 "Unicode 684Script Property": L<https://www.unicode.org/reports/tr24>. 685 686A complete list of scripts and their shortcuts is in L<perluniprops>. 687 688=head3 B<Use of the C<"Is"> Prefix> 689 690For backward compatibility (with ancient Perl 5.6), all properties writable 691without using the compound form mentioned 692so far may have C<Is> or C<Is_> prepended to their name, so C<\P{Is_Lu}>, for 693example, is equal to C<\P{Lu}>, and C<\p{IsScript:Arabic}> is equal to 694C<\p{Arabic}>. 695 696=head3 B<Blocks> 697 698In addition to B<scripts>, Unicode also defines B<blocks> of 699characters. The difference between scripts and blocks is that the 700concept of scripts is closer to natural languages, while the concept 701of blocks is more of an artificial grouping based on groups of Unicode 702characters with consecutive ordinal values. For example, the C<"Basic Latin"> 703block is all the characters whose ordinals are between 0 and 127, inclusive; in 704other words, the ASCII characters. The C<"Latin"> script contains some letters 705from this as well as several other blocks, like C<"Latin-1 Supplement">, 706C<"Latin Extended-A">, I<etc.>, but it does not contain all the characters from 707those blocks. It does not, for example, contain the digits 0-9, because 708those digits are shared across many scripts, and hence are in the 709C<Common> script. 710 711For more about scripts versus blocks, see UAX#24 "Unicode Script Property": 712L<https://www.unicode.org/reports/tr24> 713 714The C<Script_Extensions> or C<Script> properties are likely to be the 715ones you want to use when processing 716natural language; the C<Block> property may occasionally be useful in working 717with the nuts and bolts of Unicode. 718 719Block names are matched in the compound form, like C<\p{Block: Arrows}> or 720C<\p{Blk=Hebrew}>. Unlike most other properties, only a few block names have a 721Unicode-defined short name. 722 723Perl also defines single form synonyms for the block property in cases 724where these do not conflict with something else. But don't use any of 725these, because they are unstable. Since these are Perl extensions, they 726are subordinate to official Unicode property names; Unicode doesn't know 727nor care about Perl's extensions. It may happen that a name that 728currently means the Perl extension will later be changed without warning 729to mean a different Unicode property in a future version of the perl 730interpreter that uses a later Unicode release, and your code would no 731longer work. The extensions are mentioned here for completeness: Take 732the block name and prefix it with one of: C<In> (for example 733C<\p{Blk=Arrows}> can currently be written as C<\p{In_Arrows}>); or 734sometimes C<Is> (like C<\p{Is_Arrows}>); or sometimes no prefix at all 735(C<\p{Arrows}>). As of this writing (Unicode 9.0) there are no 736conflicts with using the C<In_> prefix, but there are plenty with the 737other two forms. For example, C<\p{Is_Hebrew}> and C<\p{Hebrew}> mean 738C<\p{Script_Extensions=Hebrew}> which is NOT the same thing as 739C<\p{Blk=Hebrew}>. Our 740advice used to be to use the C<In_> prefix as a single form way of 741specifying a block. But Unicode 8.0 added properties whose names begin 742with C<In>, and it's now clear that it's only luck that's so far 743prevented a conflict. Using C<In> is only marginally less typing than 744C<Blk:>, and the latter's meaning is clearer anyway, and guaranteed to 745never conflict. So don't take chances. Use C<\p{Blk=foo}> for new 746code. And be sure that block is what you really really want to do. In 747most cases scripts are what you want instead. 748 749A complete list of blocks is in L<perluniprops>. 750 751=head3 B<Other Properties> 752 753There are many more properties than the very basic ones described here. 754A complete list is in L<perluniprops>. 755 756Unicode defines all its properties in the compound form, so all single-form 757properties are Perl extensions. Most of these are just synonyms for the 758Unicode ones, but some are genuine extensions, including several that are in 759the compound form. And quite a few of these are actually recommended by Unicode 760(in L<https://www.unicode.org/reports/tr18>). 761 762This section gives some details on all extensions that aren't just 763synonyms for compound-form Unicode properties 764(for those properties, you'll have to refer to the 765L<Unicode Standard|https://www.unicode.org/reports/tr44>. 766 767=over 768 769=item B<C<\p{All}>> 770 771This matches every possible code point. It is equivalent to C<qr/./s>. 772Unlike all the other non-user-defined C<\p{}> property matches, no 773warning is ever generated if this is property is matched against a 774non-Unicode code point (see L</Beyond Unicode code points> below). 775 776=item B<C<\p{Alnum}>> 777 778This matches any C<\p{Alphabetic}> or C<\p{Decimal_Number}> character. 779 780=item B<C<\p{Any}>> 781 782This matches any of the 1_114_112 Unicode code points. It is a synonym 783for C<\p{Unicode}>. 784 785=item B<C<\p{ASCII}>> 786 787This matches any of the 128 characters in the US-ASCII character set, 788which is a subset of Unicode. 789 790=item B<C<\p{Assigned}>> 791 792This matches any assigned code point; that is, any code point whose L<general 793category|/General_Category> is not C<Unassigned> (or equivalently, not C<Cn>). 794 795=item B<C<\p{Blank}>> 796 797This is the same as C<\h> and C<\p{HorizSpace}>: A character that changes the 798spacing horizontally. 799 800=item B<C<\p{Decomposition_Type: Non_Canonical}>> (Short: C<\p{Dt=NonCanon}>) 801 802Matches a character that has a non-canonical decomposition. 803 804The L</Extended Grapheme Clusters (Logical characters)> section above 805talked about canonical decompositions. However, many more characters 806have a different type of decomposition, a "compatible" or 807"non-canonical" decomposition. The sequences that form these 808decompositions are not considered canonically equivalent to the 809pre-composed character. An example is the C<"SUPERSCRIPT ONE">. It is 810somewhat like a regular digit 1, but not exactly; its decomposition into 811the digit 1 is called a "compatible" decomposition, specifically a 812"super" decomposition. There are several such compatibility 813decompositions (see L<https://www.unicode.org/reports/tr44>), including 814one called "compat", which means some miscellaneous type of 815decomposition that doesn't fit into the other decomposition categories 816that Unicode has chosen. 817 818Note that most Unicode characters don't have a decomposition, so their 819decomposition type is C<"None">. 820 821For your convenience, Perl has added the C<Non_Canonical> decomposition 822type to mean any of the several compatibility decompositions. 823 824=item B<C<\p{Graph}>> 825 826Matches any character that is graphic. Theoretically, this means a character 827that on a printer would cause ink to be used. 828 829=item B<C<\p{HorizSpace}>> 830 831This is the same as C<\h> and C<\p{Blank}>: a character that changes the 832spacing horizontally. 833 834=item B<C<\p{In=*}>> 835 836This is a synonym for C<\p{Present_In=*}> 837 838=item B<C<\p{PerlSpace}>> 839 840This is the same as C<\s>, restricted to ASCII, namely C<S<[ \f\n\r\t]>> 841and starting in Perl v5.18, a vertical tab. 842 843Mnemonic: Perl's (original) space 844 845=item B<C<\p{PerlWord}>> 846 847This is the same as C<\w>, restricted to ASCII, namely C<[A-Za-z0-9_]> 848 849Mnemonic: Perl's (original) word. 850 851=item B<C<\p{Posix...}>> 852 853There are several of these, which are equivalents, using the C<\p{}> 854notation, for Posix classes and are described in 855L<perlrecharclass/POSIX Character Classes>. 856 857=item B<C<\p{Present_In: *}>> (Short: C<\p{In=*}>) 858 859This property is used when you need to know in what Unicode version(s) a 860character is. 861 862The "*" above stands for some Unicode version number, such as 863C<1.1> or C<12.0>; or the "*" can also be C<Unassigned>. This property will 864match the code points whose final disposition has been settled as of the 865Unicode release given by the version number; C<\p{Present_In: Unassigned}> 866will match those code points whose meaning has yet to be assigned. 867 868For example, C<U+0041> C<"LATIN CAPITAL LETTER A"> was present in the very first 869Unicode release available, which is C<1.1>, so this property is true for all 870valid "*" versions. On the other hand, C<U+1EFF> was not assigned until version 8715.1 when it became C<"LATIN SMALL LETTER Y WITH LOOP">, so the only "*" that 872would match it are 5.1, 5.2, and later. 873 874Unicode furnishes the C<Age> property from which this is derived. The problem 875with Age is that a strict interpretation of it (which Perl takes) has it 876matching the precise release a code point's meaning is introduced in. Thus 877C<U+0041> would match only 1.1; and C<U+1EFF> only 5.1. This is not usually what 878you want. 879 880Some non-Perl implementations of the Age property may change its meaning to be 881the same as the Perl C<Present_In> property; just be aware of that. 882 883Another confusion with both these properties is that the definition is not 884that the code point has been I<assigned>, but that the meaning of the code point 885has been I<determined>. This is because 66 code points will always be 886unassigned, and so the C<Age> for them is the Unicode version in which the decision 887to make them so was made. For example, C<U+FDD0> is to be permanently 888unassigned to a character, and the decision to do that was made in version 3.1, 889so C<\p{Age=3.1}> matches this character, as also does C<\p{Present_In: 3.1}> and up. 890 891=item B<C<\p{Print}>> 892 893This matches any character that is graphical or blank, except controls. 894 895=item B<C<\p{SpacePerl}>> 896 897This is the same as C<\s>, including beyond ASCII. 898 899Mnemonic: Space, as modified by Perl. (It doesn't include the vertical tab 900until v5.18, which both the Posix standard and Unicode consider white space.) 901 902=item B<C<\p{Title}>> and B<C<\p{Titlecase}>> 903 904Under case-sensitive matching, these both match the same code points as 905C<\p{General Category=Titlecase_Letter}> (C<\p{gc=lt}>). The difference 906is that under C</i> caseless matching, these match the same as 907C<\p{Cased}>, whereas C<\p{gc=lt}> matches C<\p{Cased_Letter>). 908 909=item B<C<\p{Unicode}>> 910 911This matches any of the 1_114_112 Unicode code points. 912C<\p{Any}>. 913 914=item B<C<\p{VertSpace}>> 915 916This is the same as C<\v>: A character that changes the spacing vertically. 917 918=item B<C<\p{Word}>> 919 920This is the same as C<\w>, including over 100_000 characters beyond ASCII. 921 922=item B<C<\p{XPosix...}>> 923 924There are several of these, which are the standard Posix classes 925extended to the full Unicode range. They are described in 926L<perlrecharclass/POSIX Character Classes>. 927 928=back 929 930=head2 Comparison of C<\N{...}> and C<\p{name=...}> 931 932Starting in Perl 5.32, you can specify a character by its name in 933regular expression patterns using C<\p{name=...}>. This is in addition 934to the longstanding method of using C<\N{...}>. The following 935summarizes the differences between these two: 936 937 \N{...} \p{Name=...} 938 can interpolate only with eval yes [1] 939 custom names yes no [2] 940 name aliases yes yes [3] 941 named sequences yes yes [4] 942 name value parsing exact Unicode loose [5] 943 944=over 945 946=item [1] 947 948The ability to interpolate means you can do something like 949 950 qr/\p{na=latin capital letter $which}/ 951 952and specify C<$which> elsewhere. 953 954=item [2] 955 956You can create your own names for characters, and override official 957ones when using C<\N{...}>. See L<charnames/CUSTOM ALIASES>. 958 959=item [3] 960 961Some characters have multiple names (synonyms). 962 963=item [4] 964 965Some particular sequences of characters are given a single name, in 966addition to their individual ones. 967 968=item [5] 969 970Exact name value matching means you have to specify case, hyphens, 971underscores, and spaces precisely in the name you want. Loose matching 972follows the Unicode rules 973L<https://www.unicode.org/reports/tr44/tr44-24.html#UAX44-LM2>, 974where these are mostly irrelevant. Except for a few outlier character 975names, these are the same rules as are already used for any other 976C<\p{...}> property. 977 978=back 979 980=head2 Wildcards in Property Values 981 982Starting in Perl 5.30, it is possible to do something like this: 983 984 qr!\p{numeric_value=/\A[0-5]\z/}! 985 986or, by abbreviating and adding C</x>, 987 988 qr! \p{nv= /(?x) \A [0-5] \z / }! 989 990This matches all code points whose numeric value is one of 0, 1, 2, 3, 9914, or 5. This particular example could instead have been written as 992 993 qr! \A [ \p{nv=0}\p{nv=1}\p{nv=2}\p{nv=3}\p{nv=4}\p{nv=5} ] \z !xx 994 995in earlier perls, so in this case this feature just makes things easier 996and shorter to write. If we hadn't included the C<\A> and C<\z>, these 997would have matched things like C<1E<sol>2> because that contains a 1 (as 998well as a 2). As written, it matches things like subscripts that have 999these numeric values. If we only wanted the decimal digits with those 1000numeric values, we could say, 1001 1002 qr! (?[ \d & \p{nv=/[0-5]/ ]) }!x 1003 1004The C<\d> gets rid of needing to anchor the pattern, since it forces the 1005result to only match C<[0-9]>, and the C<[0-5]> further restricts it. 1006 1007The text in the above examples enclosed between the C<"E<sol>"> 1008characters can be just about any regular expression. It is independent 1009of the main pattern, so doesn't share any capturing groups, I<etc>. The 1010delimiters for it must be ASCII punctuation, but it may NOT be 1011delimited by C<"{">, nor C<"}"> nor contain a literal C<"}">, as that 1012delimits the end of the enclosing C<\p{}>. Like any pattern, certain 1013other delimiters are terminated by their mirror images. These are 1014C<"(">, C<"[>", and C<"E<lt>">. If the delimiter is any of C<"-">, 1015C<"_">, C<"+">, or C<"\">, or is the same delimiter as is used for the 1016enclosing pattern, it must be preceded by a backslash escape, both 1017fore and aft. 1018 1019Beware of using C<"$"> to indicate to match the end of the string. It 1020can too easily be interpreted as being a punctuation variable, like 1021C<$/>. 1022 1023No modifiers may follow the final delimiter. Instead, use 1024L<perlre/(?adlupimnsx-imnsx)> and/or 1025L<perlre/(?adluimnsx-imnsx:pattern)> to specify modifiers. 1026However, certain modifiers are illegal in your wildcard subpattern. 1027The only character set modifier specifiable is C</aa>; 1028any other character set, and C<-m>, and C<p>, and C<s> are all illegal. 1029Specifying modifiers like C<qr/.../gc> that aren't legal in the 1030C<(?...)> notation normally raise a warning, but with wildcard 1031subpatterns, their use is an error. The C<m> modifier is ineffective; 1032everything that matches will be a single line. 1033 1034By default, your pattern is matched case-insensitively, as if C</i> had 1035been specified. You can change this by saying C<(?-i)> in your pattern. 1036 1037There are also certain operations that are illegal. You can't nest 1038C<\p{...}> and C<\P{...}> calls within a wildcard subpattern, and C<\G> 1039doesn't make sense, so is also prohibited. 1040 1041And the C<*> quantifier (or its equivalent C<(0,}>) is illegal. 1042 1043This feature is not available when the left-hand side is prefixed by 1044C<Is_>, nor for any form that is marked as "Discouraged" in 1045L<perluniprops/Discouraged>. 1046 1047This experimental feature has been added to begin to implement 1048L<https://www.unicode.org/reports/tr18/#Wildcard_Properties>. Using it 1049will raise a (default-on) warning in the 1050C<experimental::uniprop_wildcards> category. We reserve the right to 1051change its operation as we gain experience. 1052 1053Your subpattern can be just about anything, but for it to have some 1054utility, it should match when called with either or both of 1055a) the full name of the property value with underscores (and/or spaces 1056in the Block property) and some things uppercase; or b) the property 1057value in all lowercase with spaces and underscores squeezed out. For 1058example, 1059 1060 qr!\p{Blk=/Old I.*/}! 1061 qr!\p{Blk=/oldi.*/}! 1062 1063would match the same things. 1064 1065Another example that shows that within C<\p{...}>, C</x> isn't needed to 1066have spaces: 1067 1068 qr!\p{scx= /Hebrew|Greek/ }! 1069 1070To be safe, we should have anchored the above example, to prevent 1071matches for something like C<Hebrew_Braille>, but there aren't 1072any script names like that, so far. 1073A warning is issued if none of the legal values for a property are 1074matched by your pattern. It's likely that a future release will raise a 1075warning if your pattern ends up causing every possible code point to 1076match. 1077 1078Starting in 5.32, the Name, Name Aliases, and Named Sequences properties 1079are allowed to be matched. They are considered to be a single 1080combination property, just as has long been the case for C<\N{}>. Loose 1081matching doesn't work in exactly the same way for these as it does for 1082the values of other properties. The rules are given in 1083L<https://www.unicode.org/reports/tr44/tr44-24.html#UAX44-LM2>. As a 1084result, Perl doesn't try loose matching for you, like it does in other 1085properties. All letters in names are uppercase, but you can add C<(?i)> 1086to your subpattern to ignore case. If you're uncertain where a blank 1087is, you can use C< ?> in your subpattern. No character name contains an 1088underscore, so don't bother trying to match one. The use of hyphens is 1089particularly problematic; refer to the above link. But note that, as of 1090Unicode 13.0, the only script in modern usage which has weirdnesses with 1091these is Tibetan; also the two Korean characters U+116C HANGUL JUNGSEONG 1092OE and U+1180 HANGUL JUNGSEONG O-E. Unicode makes no promises to not 1093add hyphen-problematic names in the future. 1094 1095Using wildcards on these is resource intensive, given the hundreds of 1096thousands of legal names that must be checked against. 1097 1098An example of using Name property wildcards is 1099 1100 qr!\p{name=/(SMILING|GRINNING) FACE/}! 1101 1102Another is 1103 1104 qr/(?[ \p{name=\/CJK\/} - \p{ideographic} ])/ 1105 1106which is the 200-ish (as of Unicode 13.0) CJK characters that aren't 1107ideographs. 1108 1109There are certain properties that wildcard subpatterns don't currently 1110work with. These are: 1111 1112 Bidi Mirroring Glyph 1113 Bidi Paired Bracket 1114 Case Folding 1115 Decomposition Mapping 1116 Equivalent Unified Ideograph 1117 Lowercase Mapping 1118 NFKC Case Fold 1119 Titlecase Mapping 1120 Uppercase Mapping 1121 1122Nor is the C<@I<unicode_property>@> form implemented. 1123 1124Here's a complete example of matching IPV4 internet protocol addresses 1125in any (single) script 1126 1127 no warnings 'experimental::regex_sets'; 1128 no warnings 'experimental::uniprop_wildcards'; 1129 1130 # Can match a substring, so this intermediate regex needs to have 1131 # context or anchoring in its final use. Using nt=de yields decimal 1132 # digits. When specifying a subset of these, we must include \d to 1133 # prevent things like U+00B2 SUPERSCRIPT TWO from matching 1134 my $zero_through_255 = 1135 qr/ \b (*sr: # All from same sript 1136 (?[ \p{nv=0} & \d ])* # Optional leading zeros 1137 ( # Then one of: 1138 \d{1,2} # 0 - 99 1139 | (?[ \p{nv=1} & \d ]) \d{2} # 100 - 199 1140 | (?[ \p{nv=2} & \d ]) 1141 ( (?[ \p{nv=:[0-4]:} & \d ]) \d # 200 - 249 1142 | (?[ \p{nv=5} & \d ]) 1143 (?[ \p{nv=:[0-5]:} & \d ]) # 250 - 255 1144 ) 1145 ) 1146 ) 1147 \b 1148 /x; 1149 1150 my $ipv4 = qr/ \A (*sr: $zero_through_255 1151 (?: [.] $zero_through_255 ) {3} 1152 ) 1153 \z 1154 /x; 1155 1156=head2 User-Defined Character Properties 1157 1158You can define your own binary character properties by defining subroutines 1159whose names begin with C<"In"> or C<"Is">. (The experimental feature 1160L<perlre/(?[ ])> provides an alternative which allows more complex 1161definitions.) The subroutines can be defined in any 1162package. They override any Unicode properties expressed as the same 1163names. The user-defined properties can be used in the regular 1164expression 1165C<\p{}> and C<\P{}> constructs; if you are using a user-defined property from a 1166package other than the one you are in, you must specify its package in the 1167C<\p{}> or C<\P{}> construct. 1168 1169 # assuming property Is_Foreign defined in Lang:: 1170 package main; # property package name required 1171 if ($txt =~ /\p{Lang::IsForeign}+/) { ... } 1172 1173 package Lang; # property package name not required 1174 if ($txt =~ /\p{IsForeign}+/) { ... } 1175 1176 1177Note that the effect is compile-time and immutable once defined. 1178However, the subroutines are passed a single parameter, which is 0 if 1179case-sensitive matching is in effect and non-zero if caseless matching 1180is in effect. The subroutine may return different values depending on 1181the value of the flag, and one set of values will immutably be in effect 1182for all case-sensitive matches, and the other set for all case-insensitive 1183matches. 1184 1185Note that if the regular expression is tainted, then Perl will die rather 1186than calling the subroutine when the name of the subroutine is 1187determined by the tainted data. 1188 1189The subroutines must return a specially-formatted string, with one 1190or more newline-separated lines. Each line must be one of the following: 1191 1192=over 4 1193 1194=item * 1195 1196A single hexadecimal number denoting a code point to include. 1197 1198=item * 1199 1200Two hexadecimal numbers separated by horizontal whitespace (space or 1201tabular characters) denoting a range of code points to include. The 1202second number must not be smaller than the first. 1203 1204=item * 1205 1206Something to include, prefixed by C<"+">: a built-in character 1207property (prefixed by C<"utf8::">) or a fully qualified (including package 1208name) user-defined character property, 1209to represent all the characters in that property; two hexadecimal code 1210points for a range; or a single hexadecimal code point. 1211 1212=item * 1213 1214Something to exclude, prefixed by C<"-">: an existing character 1215property (prefixed by C<"utf8::">) or a fully qualified (including package 1216name) user-defined character property, 1217to represent all the characters in that property; two hexadecimal code 1218points for a range; or a single hexadecimal code point. 1219 1220=item * 1221 1222Something to negate, prefixed C<"!">: an existing character 1223property (prefixed by C<"utf8::">) or a fully qualified (including package 1224name) user-defined character property, 1225to represent all the characters in that property; two hexadecimal code 1226points for a range; or a single hexadecimal code point. 1227 1228=item * 1229 1230Something to intersect with, prefixed by C<"&">: an existing character 1231property (prefixed by C<"utf8::">) or a fully qualified (including package 1232name) user-defined character property, 1233for all the characters except the characters in the property; two 1234hexadecimal code points for a range; or a single hexadecimal code point. 1235 1236=back 1237 1238For example, to define a property that covers both the Japanese 1239syllabaries (hiragana and katakana), you can define 1240 1241 sub InKana { 1242 return <<END; 1243 3040\t309F 1244 30A0\t30FF 1245 END 1246 } 1247 1248Imagine that the here-doc end marker is at the beginning of the line. 1249Now you can use C<\p{InKana}> and C<\P{InKana}>. 1250 1251You could also have used the existing block property names: 1252 1253 sub InKana { 1254 return <<'END'; 1255 +utf8::InHiragana 1256 +utf8::InKatakana 1257 END 1258 } 1259 1260Suppose you wanted to match only the allocated characters, 1261not the raw block ranges: in other words, you want to remove 1262the unassigned characters: 1263 1264 sub InKana { 1265 return <<'END'; 1266 +utf8::InHiragana 1267 +utf8::InKatakana 1268 -utf8::IsCn 1269 END 1270 } 1271 1272The negation is useful for defining (surprise!) negated classes. 1273 1274 sub InNotKana { 1275 return <<'END'; 1276 !utf8::InHiragana 1277 -utf8::InKatakana 1278 +utf8::IsCn 1279 END 1280 } 1281 1282This will match all non-Unicode code points, since every one of them is 1283not in Kana. You can use intersection to exclude these, if desired, as 1284this modified example shows: 1285 1286 sub InNotKana { 1287 return <<'END'; 1288 !utf8::InHiragana 1289 -utf8::InKatakana 1290 +utf8::IsCn 1291 &utf8::Any 1292 END 1293 } 1294 1295C<&utf8::Any> must be the last line in the definition. 1296 1297Intersection is used generally for getting the common characters matched 1298by two (or more) classes. It's important to remember not to use C<"&"> for 1299the first set; that would be intersecting with nothing, resulting in an 1300empty set. (Similarly using C<"-"> for the first set does nothing). 1301 1302Unlike non-user-defined C<\p{}> property matches, no warning is ever 1303generated if these properties are matched against a non-Unicode code 1304point (see L</Beyond Unicode code points> below). 1305 1306=head2 User-Defined Case Mappings (for serious hackers only) 1307 1308B<This feature has been removed as of Perl 5.16.> 1309The CPAN module C<L<Unicode::Casing>> provides better functionality without 1310the drawbacks that this feature had. If you are using a Perl earlier 1311than 5.16, this feature was most fully documented in the 5.14 version of 1312this pod: 1313L<http://perldoc.perl.org/5.14.0/perlunicode.html#User-Defined-Case-Mappings-%28for-serious-hackers-only%29> 1314 1315=head2 Character Encodings for Input and Output 1316 1317See L<Encode>. 1318 1319=head2 Unicode Regular Expression Support Level 1320 1321The following list of Unicode supported features for regular expressions describes 1322all features currently directly supported by core Perl. The references 1323to "Level I<N>" and the section numbers refer to 1324L<UTS#18 "Unicode Regular Expressions"|https://www.unicode.org/reports/tr18>, 1325version 18, October 2016. 1326 1327=head3 Level 1 - Basic Unicode Support 1328 1329 RL1.1 Hex Notation - Done [1] 1330 RL1.2 Properties - Done [2] 1331 RL1.2a Compatibility Properties - Done [3] 1332 RL1.3 Subtraction and Intersection - Experimental [4] 1333 RL1.4 Simple Word Boundaries - Done [5] 1334 RL1.5 Simple Loose Matches - Done [6] 1335 RL1.6 Line Boundaries - Partial [7] 1336 RL1.7 Supplementary Code Points - Done [8] 1337 1338=over 4 1339 1340=item [1] C<\N{U+...}> and C<\x{...}> 1341 1342=item [2] 1343C<\p{...}> C<\P{...}>. This requirement is for a minimal list of 1344properties. Perl supports these. See R2.7 for other properties. 1345 1346=item [3] 1347Perl has C<\d> C<\D> C<\s> C<\S> C<\w> C<\W> C<\X> C<[:I<prop>:]> 1348C<[:^I<prop>:]>, plus all the properties specified by 1349L<https://www.unicode.org/reports/tr18/#Compatibility_Properties>. These 1350are described above in L</Other Properties> 1351 1352=item [4] 1353 1354The experimental feature C<"(?[...])"> starting in v5.18 accomplishes 1355this. 1356 1357See L<perlre/(?[ ])>. If you don't want to use an experimental 1358feature, you can use one of the following: 1359 1360=over 4 1361 1362=item * 1363Regular expression lookahead 1364 1365You can mimic class subtraction using lookahead. 1366For example, what UTS#18 might write as 1367 1368 [{Block=Greek}-[{UNASSIGNED}]] 1369 1370in Perl can be written as: 1371 1372 (?!\p{Unassigned})\p{Block=Greek} 1373 (?=\p{Assigned})\p{Block=Greek} 1374 1375But in this particular example, you probably really want 1376 1377 \p{Greek} 1378 1379which will match assigned characters known to be part of the Greek script. 1380 1381=item * 1382 1383CPAN module C<L<Unicode::Regex::Set>> 1384 1385It does implement the full UTS#18 grouping, intersection, union, and 1386removal (subtraction) syntax. 1387 1388=item * 1389 1390L</"User-Defined Character Properties"> 1391 1392C<"+"> for union, C<"-"> for removal (set-difference), C<"&"> for intersection 1393 1394=back 1395 1396=item [5] 1397C<\b> C<\B> meet most, but not all, the details of this requirement, but 1398C<\b{wb}> and C<\B{wb}> do, as well as the stricter R2.3. 1399 1400=item [6] 1401 1402Note that Perl does Full case-folding in matching, not Simple: 1403 1404For example C<U+1F88> is equivalent to C<U+1F00 U+03B9>, instead of just 1405C<U+1F80>. This difference matters mainly for certain Greek capital 1406letters with certain modifiers: the Full case-folding decomposes the 1407letter, while the Simple case-folding would map it to a single 1408character. 1409 1410=item [7] 1411 1412The reason this is considered to be only partially implemented is that 1413Perl has L<C<qrE<sol>\b{lb}E<sol>>|perlrebackslash/\b{lb}> and 1414C<L<Unicode::LineBreak>> that are conformant with 1415L<UAX#14 "Unicode Line Breaking Algorithm"|https://www.unicode.org/reports/tr14>. 1416The regular expression construct provides default behavior, while the 1417heavier-weight module provides customizable line breaking. 1418 1419But Perl treats C<\n> as the start- and end-line 1420delimiter, whereas Unicode specifies more characters that should be 1421so-interpreted. 1422 1423These are: 1424 1425 VT U+000B (\v in C) 1426 FF U+000C (\f) 1427 CR U+000D (\r) 1428 NEL U+0085 1429 LS U+2028 1430 PS U+2029 1431 1432C<^> and C<$> in regular expression patterns are supposed to match all 1433these, but don't. 1434These characters also don't, but should, affect C<< <> >> C<$.>, and 1435script line numbers. 1436 1437Also, lines should not be split within C<CRLF> (i.e. there is no 1438empty line between C<\r> and C<\n>). For C<CRLF>, try the C<:crlf> 1439layer (see L<PerlIO>). 1440 1441=item [8] 1442UTF-8/UTF-EBDDIC used in Perl allows not only C<U+10000> to 1443C<U+10FFFF> but also beyond C<U+10FFFF> 1444 1445=back 1446 1447=head3 Level 2 - Extended Unicode Support 1448 1449 RL2.1 Canonical Equivalents - Retracted [9] 1450 by Unicode 1451 RL2.2 Extended Grapheme Clusters and - Partial [10] 1452 Character Classes with Strings 1453 RL2.3 Default Word Boundaries - Done [11] 1454 RL2.4 Default Case Conversion - Done 1455 RL2.5 Name Properties - Done 1456 RL2.6 Wildcards in Property Values - Partial [12] 1457 RL2.7 Full Properties - Partial [13] 1458 RL2.8 Optional Properties - Partial [14] 1459 1460=over 4 1461 1462=item [9] 1463Unicode has rewritten this portion of UTS#18 to say that getting 1464canonical equivalence (see UAX#15 1465L<"Unicode Normalization Forms"|https://www.unicode.org/reports/tr15>) 1466is basically to be done at the programmer level. Use NFD to write 1467both your regular expressions and text to match them against (you 1468can use L<Unicode::Normalize>). 1469 1470=item [10] 1471Perl has C<\X> and C<\b{gcb}>. Unicode has retracted their "Grapheme 1472Cluster Mode", and recently added string properties, which Perl does not 1473yet support. 1474 1475=item [11] see 1476L<UAX#29 "Unicode Text Segmentation"|https://www.unicode.org/reports/tr29>, 1477 1478=item [12] see 1479L</Wildcards in Property Values> above. 1480 1481=item [13] 1482Perl supports all the properties in the Unicode Character Database 1483(UCD). It does not yet support the listed properties that come from 1484other Unicode sources. 1485 1486=item [14] 1487The only optional property that Perl supports is Named Sequence. None 1488of these properties are in the UCD. 1489 1490=back 1491 1492=head3 Level 3 - Tailored Support 1493 1494This has been retracted by Unicode. 1495 1496=head2 Unicode Encodings 1497 1498Unicode characters are assigned to I<code points>, which are abstract 1499numbers. To use these numbers, various encodings are needed. 1500 1501=over 4 1502 1503=item * 1504 1505UTF-8 1506 1507UTF-8 is a variable-length (1 to 4 bytes), byte-order independent 1508encoding. In most of Perl's documentation, including elsewhere in this 1509document, the term "UTF-8" means also "UTF-EBCDIC". But in this section, 1510"UTF-8" refers only to the encoding used on ASCII platforms. It is a 1511superset of 7-bit US-ASCII, so anything encoded in ASCII has the 1512identical representation when encoded in UTF-8. 1513 1514The following table is from Unicode 3.2. 1515 1516 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte 1517 1518 U+0000..U+007F 00..7F 1519 U+0080..U+07FF * C2..DF 80..BF 1520 U+0800..U+0FFF E0 * A0..BF 80..BF 1521 U+1000..U+CFFF E1..EC 80..BF 80..BF 1522 U+D000..U+D7FF ED 80..9F 80..BF 1523 U+D800..U+DFFF +++++ utf16 surrogates, not legal utf8 +++++ 1524 U+E000..U+FFFF EE..EF 80..BF 80..BF 1525 U+10000..U+3FFFF F0 * 90..BF 80..BF 80..BF 1526 U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF 1527 U+100000..U+10FFFF F4 80..8F 80..BF 80..BF 1528 1529Note the gaps marked by "*" before several of the byte entries above. These are 1530caused by legal UTF-8 avoiding non-shortest encodings: it is technically 1531possible to UTF-8-encode a single code point in different ways, but that is 1532explicitly forbidden, and the shortest possible encoding should always be used 1533(and that is what Perl does). 1534 1535Another way to look at it is via bits: 1536 1537 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte 1538 1539 0aaaaaaa 0aaaaaaa 1540 00000bbbbbaaaaaa 110bbbbb 10aaaaaa 1541 ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa 1542 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa 1543 1544As you can see, the continuation bytes all begin with C<"10">, and the 1545leading bits of the start byte tell how many bytes there are in the 1546encoded character. 1547 1548The original UTF-8 specification allowed up to 6 bytes, to allow 1549encoding of numbers up to C<0x7FFF_FFFF>. Perl continues to allow those, 1550and has extended that up to 13 bytes to encode code points up to what 1551can fit in a 64-bit word. However, Perl will warn if you output any of 1552these as being non-portable; and under strict UTF-8 input protocols, 1553they are forbidden. In addition, it is now illegal to use a code point 1554larger than what a signed integer variable on your system can hold. On 155532-bit ASCII systems, this means C<0x7FFF_FFFF> is the legal maximum 1556(much higher on 64-bit systems). 1557 1558=item * 1559 1560UTF-EBCDIC 1561 1562Like UTF-8, but EBCDIC-safe, in the way that UTF-8 is ASCII-safe. 1563This means that all the basic characters (which includes all 1564those that have ASCII equivalents (like C<"A">, C<"0">, C<"%">, I<etc.>) 1565are the same in both EBCDIC and UTF-EBCDIC.) 1566 1567UTF-EBCDIC is used on EBCDIC platforms. It generally requires more 1568bytes to represent a given code point than UTF-8 does; the largest 1569Unicode code points take 5 bytes to represent (instead of 4 in UTF-8), 1570and, extended for 64-bit words, it uses 14 bytes instead of 13 bytes in 1571UTF-8. 1572 1573=item * 1574 1575UTF-16, UTF-16BE, UTF-16LE, Surrogates, and C<BOM>'s (Byte Order Marks) 1576 1577The followings items are mostly for reference and general Unicode 1578knowledge, Perl doesn't use these constructs internally. 1579 1580Like UTF-8, UTF-16 is a variable-width encoding, but where 1581UTF-8 uses 8-bit code units, UTF-16 uses 16-bit code units. 1582All code points occupy either 2 or 4 bytes in UTF-16: code points 1583C<U+0000..U+FFFF> are stored in a single 16-bit unit, and code 1584points C<U+10000..U+10FFFF> in two 16-bit units. The latter case is 1585using I<surrogates>, the first 16-bit unit being the I<high 1586surrogate>, and the second being the I<low surrogate>. 1587 1588Surrogates are code points set aside to encode the C<U+10000..U+10FFFF> 1589range of Unicode code points in pairs of 16-bit units. The I<high 1590surrogates> are the range C<U+D800..U+DBFF> and the I<low surrogates> 1591are the range C<U+DC00..U+DFFF>. The surrogate encoding is 1592 1593 $hi = ($uni - 0x10000) / 0x400 + 0xD800; 1594 $lo = ($uni - 0x10000) % 0x400 + 0xDC00; 1595 1596and the decoding is 1597 1598 $uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00); 1599 1600Because of the 16-bitness, UTF-16 is byte-order dependent. UTF-16 1601itself can be used for in-memory computations, but if storage or 1602transfer is required either UTF-16BE (big-endian) or UTF-16LE 1603(little-endian) encodings must be chosen. 1604 1605This introduces another problem: what if you just know that your data 1606is UTF-16, but you don't know which endianness? Byte Order Marks, or 1607C<BOM>'s, are a solution to this. A special character has been reserved 1608in Unicode to function as a byte order marker: the character with the 1609code point C<U+FEFF> is the C<BOM>. 1610 1611The trick is that if you read a C<BOM>, you will know the byte order, 1612since if it was written on a big-endian platform, you will read the 1613bytes C<0xFE 0xFF>, but if it was written on a little-endian platform, 1614you will read the bytes C<0xFF 0xFE>. (And if the originating platform 1615was writing in ASCII platform UTF-8, you will read the bytes 1616C<0xEF 0xBB 0xBF>.) 1617 1618The way this trick works is that the character with the code point 1619C<U+FFFE> is not supposed to be in input streams, so the 1620sequence of bytes C<0xFF 0xFE> is unambiguously "C<BOM>, represented in 1621little-endian format" and cannot be C<U+FFFE>, represented in big-endian 1622format". 1623 1624Surrogates have no meaning in Unicode outside their use in pairs to 1625represent other code points. However, Perl allows them to be 1626represented individually internally, for example by saying 1627C<chr(0xD801)>, so that all code points, not just those valid for open 1628interchange, are 1629representable. Unicode does define semantics for them, such as their 1630C<L</General_Category>> is C<"Cs">. But because their use is somewhat dangerous, 1631Perl will warn (using the warning category C<"surrogate">, which is a 1632sub-category of C<"utf8">) if an attempt is made 1633to do things like take the lower case of one, or match 1634case-insensitively, or to output them. (But don't try this on Perls 1635before 5.14.) 1636 1637=item * 1638 1639UTF-32, UTF-32BE, UTF-32LE 1640 1641The UTF-32 family is pretty much like the UTF-16 family, except that 1642the units are 32-bit, and therefore the surrogate scheme is not 1643needed. UTF-32 is a fixed-width encoding. The C<BOM> signatures are 1644C<0x00 0x00 0xFE 0xFF> for BE and C<0xFF 0xFE 0x00 0x00> for LE. 1645 1646=item * 1647 1648UCS-2, UCS-4 1649 1650Legacy, fixed-width encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit 1651encoding. Unlike UTF-16, UCS-2 is not extensible beyond C<U+FFFF>, 1652because it does not use surrogates. UCS-4 is a 32-bit encoding, 1653functionally identical to UTF-32 (the difference being that 1654UCS-4 forbids neither surrogates nor code points larger than C<0x10_FFFF>). 1655 1656=item * 1657 1658UTF-7 1659 1660A seven-bit safe (non-eight-bit) encoding, which is useful if the 1661transport or storage is not eight-bit safe. Defined by RFC 2152. 1662 1663=back 1664 1665=head2 Noncharacter code points 1666 166766 code points are set aside in Unicode as "noncharacter code points". 1668These all have the C<Unassigned> (C<Cn>) C<L</General_Category>>, and 1669no character will ever be assigned to any of them. They are the 32 code 1670points between C<U+FDD0> and C<U+FDEF> inclusive, and the 34 code 1671points: 1672 1673 U+FFFE U+FFFF 1674 U+1FFFE U+1FFFF 1675 U+2FFFE U+2FFFF 1676 ... 1677 U+EFFFE U+EFFFF 1678 U+FFFFE U+FFFFF 1679 U+10FFFE U+10FFFF 1680 1681Until Unicode 7.0, the noncharacters were "B<forbidden> for use in open 1682interchange of Unicode text data", so that code that processed those 1683streams could use these code points as sentinels that could be mixed in 1684with character data, and would always be distinguishable from that data. 1685(Emphasis above and in the next paragraph are added in this document.) 1686 1687Unicode 7.0 changed the wording so that they are "B<not recommended> for 1688use in open interchange of Unicode text data". The 7.0 Standard goes on 1689to say: 1690 1691=over 4 1692 1693"If a noncharacter is received in open interchange, an application is 1694not required to interpret it in any way. It is good practice, however, 1695to recognize it as a noncharacter and to take appropriate action, such 1696as replacing it with C<U+FFFD> replacement character, to indicate the 1697problem in the text. It is not recommended to simply delete 1698noncharacter code points from such text, because of the potential 1699security issues caused by deleting uninterpreted characters. (See 1700conformance clause C7 in Section 3.2, Conformance Requirements, and 1701L<Unicode Technical Report #36, "Unicode Security 1702Considerations"|https://www.unicode.org/reports/tr36/#Substituting_for_Ill_Formed_Subsequences>)." 1703 1704=back 1705 1706This change was made because it was found that various commercial tools 1707like editors, or for things like source code control, had been written 1708so that they would not handle program files that used these code points, 1709effectively precluding their use almost entirely! And that was never 1710the intent. They've always been meant to be usable within an 1711application, or cooperating set of applications, at will. 1712 1713If you're writing code, such as an editor, that is supposed to be able 1714to handle any Unicode text data, then you shouldn't be using these code 1715points yourself, and instead allow them in the input. If you need 1716sentinels, they should instead be something that isn't legal Unicode. 1717For UTF-8 data, you can use the bytes 0xC1 and 0xC2 as sentinels, as 1718they never appear in well-formed UTF-8. (There are equivalents for 1719UTF-EBCDIC). You can also store your Unicode code points in integer 1720variables and use negative values as sentinels. 1721 1722If you're not writing such a tool, then whether you accept noncharacters 1723as input is up to you (though the Standard recommends that you not). If 1724you do strict input stream checking with Perl, these code points 1725continue to be forbidden. This is to maintain backward compatibility 1726(otherwise potential security holes could open up, as an unsuspecting 1727application that was written assuming the noncharacters would be 1728filtered out before getting to it, could now, without warning, start 1729getting them). To do strict checking, you can use the layer 1730C<:encoding('UTF-8')>. 1731 1732Perl continues to warn (using the warning category C<"nonchar">, which 1733is a sub-category of C<"utf8">) if an attempt is made to output 1734noncharacters. 1735 1736=head2 Beyond Unicode code points 1737 1738The maximum Unicode code point is C<U+10FFFF>, and Unicode only defines 1739operations on code points up through that. But Perl works on code 1740points up to the maximum permissible signed number available on the 1741platform. However, Perl will not accept these from input streams unless 1742lax rules are being used, and will warn (using the warning category 1743C<"non_unicode">, which is a sub-category of C<"utf8">) if any are output. 1744 1745Since Unicode rules are not defined on these code points, if a 1746Unicode-defined operation is done on them, Perl uses what we believe are 1747sensible rules, while generally warning, using the C<"non_unicode"> 1748category. For example, C<uc("\x{11_0000}")> will generate such a 1749warning, returning the input parameter as its result, since Perl defines 1750the uppercase of every non-Unicode code point to be the code point 1751itself. (All the case changing operations, not just uppercasing, work 1752this way.) 1753 1754The situation with matching Unicode properties in regular expressions, 1755the C<\p{}> and C<\P{}> constructs, against these code points is not as 1756clear cut, and how these are handled has changed as we've gained 1757experience. 1758 1759One possibility is to treat any match against these code points as 1760undefined. But since Perl doesn't have the concept of a match being 1761undefined, it converts this to failing or C<FALSE>. This is almost, but 1762not quite, what Perl did from v5.14 (when use of these code points 1763became generally reliable) through v5.18. The difference is that Perl 1764treated all C<\p{}> matches as failing, but all C<\P{}> matches as 1765succeeding. 1766 1767One problem with this is that it leads to unexpected, and confusing 1768results in some cases: 1769 1770 chr(0x110000) =~ \p{ASCII_Hex_Digit=True} # Failed on <= v5.18 1771 chr(0x110000) =~ \p{ASCII_Hex_Digit=False} # Failed! on <= v5.18 1772 1773That is, it treated both matches as undefined, and converted that to 1774false (raising a warning on each). The first case is the expected 1775result, but the second is likely counterintuitive: "How could both be 1776false when they are complements?" Another problem was that the 1777implementation optimized many Unicode property matches down to already 1778existing simpler, faster operations, which don't raise the warning. We 1779chose to not forgo those optimizations, which help the vast majority of 1780matches, just to generate a warning for the unlikely event that an 1781above-Unicode code point is being matched against. 1782 1783As a result of these problems, starting in v5.20, what Perl does is 1784to treat non-Unicode code points as just typical unassigned Unicode 1785characters, and matches accordingly. (Note: Unicode has atypical 1786unassigned code points. For example, it has noncharacter code points, 1787and ones that, when they do get assigned, are destined to be written 1788Right-to-left, as Arabic and Hebrew are. Perl assumes that no 1789non-Unicode code point has any atypical properties.) 1790 1791Perl, in most cases, will raise a warning when matching an above-Unicode 1792code point against a Unicode property when the result is C<TRUE> for 1793C<\p{}>, and C<FALSE> for C<\P{}>. For example: 1794 1795 chr(0x110000) =~ \p{ASCII_Hex_Digit=True} # Fails, no warning 1796 chr(0x110000) =~ \p{ASCII_Hex_Digit=False} # Succeeds, with warning 1797 1798In both these examples, the character being matched is non-Unicode, so 1799Unicode doesn't define how it should match. It clearly isn't an ASCII 1800hex digit, so the first example clearly should fail, and so it does, 1801with no warning. But it is arguable that the second example should have 1802an undefined, hence C<FALSE>, result. So a warning is raised for it. 1803 1804Thus the warning is raised for many fewer cases than in earlier Perls, 1805and only when what the result is could be arguable. It turns out that 1806none of the optimizations made by Perl (or are ever likely to be made) 1807cause the warning to be skipped, so it solves both problems of Perl's 1808earlier approach. The most commonly used property that is affected by 1809this change is C<\p{Unassigned}> which is a short form for 1810C<\p{General_Category=Unassigned}>. Starting in v5.20, all non-Unicode 1811code points are considered C<Unassigned>. In earlier releases the 1812matches failed because the result was considered undefined. 1813 1814The only place where the warning is not raised when it might ought to 1815have been is if optimizations cause the whole pattern match to not even 1816be attempted. For example, Perl may figure out that for a string to 1817match a certain regular expression pattern, the string has to contain 1818the substring C<"foobar">. Before attempting the match, Perl may look 1819for that substring, and if not found, immediately fail the match without 1820actually trying it; so no warning gets generated even if the string 1821contains an above-Unicode code point. 1822 1823This behavior is more "Do what I mean" than in earlier Perls for most 1824applications. But it catches fewer issues for code that needs to be 1825strictly Unicode compliant. Therefore there is an additional mode of 1826operation available to accommodate such code. This mode is enabled if a 1827regular expression pattern is compiled within the lexical scope where 1828the C<"non_unicode"> warning class has been made fatal, say by: 1829 1830 use warnings FATAL => "non_unicode" 1831 1832(see L<warnings>). In this mode of operation, Perl will raise the 1833warning for all matches against a non-Unicode code point (not just the 1834arguable ones), and it skips the optimizations that might cause the 1835warning to not be output. (It currently still won't warn if the match 1836isn't even attempted, like in the C<"foobar"> example above.) 1837 1838In summary, Perl now normally treats non-Unicode code points as typical 1839Unicode unassigned code points for regular expression matches, raising a 1840warning only when it is arguable what the result should be. However, if 1841this warning has been made fatal, it isn't skipped. 1842 1843There is one exception to all this. C<\p{All}> looks like a Unicode 1844property, but it is a Perl extension that is defined to be true for all 1845possible code points, Unicode or not, so no warning is ever generated 1846when matching this against a non-Unicode code point. (Prior to v5.20, 1847it was an exact synonym for C<\p{Any}>, matching code points C<0> 1848through C<0x10FFFF>.) 1849 1850=head2 Security Implications of Unicode 1851 1852First, read 1853L<Unicode Security Considerations|https://www.unicode.org/reports/tr36>. 1854 1855Also, note the following: 1856 1857=over 4 1858 1859=item * 1860 1861Malformed UTF-8 1862 1863UTF-8 is very structured, so many combinations of bytes are invalid. In 1864the past, Perl tried to soldier on and make some sense of invalid 1865combinations, but this can lead to security holes, so now, if the Perl 1866core needs to process an invalid combination, it will either raise a 1867fatal error, or will replace those bytes by the sequence that forms the 1868Unicode REPLACEMENT CHARACTER, for which purpose Unicode created it. 1869 1870Every code point can be represented by more than one possible 1871syntactically valid UTF-8 sequence. Early on, both Unicode and Perl 1872considered any of these to be valid, but now, all sequences longer 1873than the shortest possible one are considered to be malformed. 1874 1875Unicode considers many code points to be illegal, or to be avoided. 1876Perl generally accepts them, once they have passed through any input 1877filters that may try to exclude them. These have been discussed above 1878(see "Surrogates" under UTF-16 in L</Unicode Encodings>, 1879L</Noncharacter code points>, and L</Beyond Unicode code points>). 1880 1881=item * 1882 1883Regular expression pattern matching may surprise you if you're not 1884accustomed to Unicode. Starting in Perl 5.14, several pattern 1885modifiers are available to control this, called the character set 1886modifiers. Details are given in L<perlre/Character set modifiers>. 1887 1888=back 1889 1890As discussed elsewhere, Perl has one foot (two hooves?) planted in 1891each of two worlds: the old world of ASCII and single-byte locales, and 1892the new world of Unicode, upgrading when necessary. 1893If your legacy code does not explicitly use Unicode, no automatic 1894switch-over to Unicode should happen. 1895 1896=head2 Unicode in Perl on EBCDIC 1897 1898Unicode is supported on EBCDIC platforms. See L<perlebcdic>. 1899 1900Unless ASCII vs. EBCDIC issues are specifically being discussed, 1901references to UTF-8 encoding in this document and elsewhere should be 1902read as meaning UTF-EBCDIC on EBCDIC platforms. 1903See L<perlebcdic/Unicode and UTF>. 1904 1905Because UTF-EBCDIC is so similar to UTF-8, the differences are mostly 1906hidden from you; S<C<use utf8>> (and NOT something like 1907S<C<use utfebcdic>>) declares the script is in the platform's 1908"native" 8-bit encoding of Unicode. (Similarly for the C<":utf8"> 1909layer.) 1910 1911=head2 Locales 1912 1913See L<perllocale/Unicode and UTF-8> 1914 1915=head2 When Unicode Does Not Happen 1916 1917There are still many places where Unicode (in some encoding or 1918another) could be given as arguments or received as results, or both in 1919Perl, but it is not, in spite of Perl having extensive ways to input and 1920output in Unicode, and a few other "entry points" like the C<@ARGV> 1921array (which can sometimes be interpreted as UTF-8). 1922 1923The following are such interfaces. Also, see L</The "Unicode Bug">. 1924For all of these interfaces Perl 1925currently (as of v5.16.0) simply assumes byte strings both as arguments 1926and results, or UTF-8 strings if the (deprecated) C<encoding> pragma has been used. 1927 1928One reason that Perl does not attempt to resolve the role of Unicode in 1929these situations is that the answers are highly dependent on the operating 1930system and the file system(s). For example, whether filenames can be 1931in Unicode and in exactly what kind of encoding, is not exactly a 1932portable concept. Similarly for C<qx> and C<system>: how well will the 1933"command-line interface" (and which of them?) handle Unicode? 1934 1935=over 4 1936 1937=item * 1938 1939C<chdir>, C<chmod>, C<chown>, C<chroot>, C<exec>, C<link>, C<lstat>, C<mkdir>, 1940C<rename>, C<rmdir>, C<stat>, C<symlink>, C<truncate>, C<unlink>, C<utime>, C<-X> 1941 1942=item * 1943 1944C<%ENV> 1945 1946=item * 1947 1948C<glob> (aka the C<E<lt>*E<gt>>) 1949 1950=item * 1951 1952C<open>, C<opendir>, C<sysopen> 1953 1954=item * 1955 1956C<qx> (aka the backtick operator), C<system> 1957 1958=item * 1959 1960C<readdir>, C<readlink> 1961 1962=back 1963 1964=head2 The "Unicode Bug" 1965 1966The term, "Unicode bug" has been applied to an inconsistency with the 1967code points in the C<Latin-1 Supplement> block, that is, between 1968128 and 255. Without a locale specified, unlike all other characters or 1969code points, these characters can have very different semantics 1970depending on the rules in effect. (Characters whose code points are 1971above 255 force Unicode rules; whereas the rules for ASCII characters 1972are the same under both ASCII and Unicode rules.) 1973 1974Under Unicode rules, these upper-Latin1 characters are interpreted as 1975Unicode code points, which means they have the same semantics as Latin-1 1976(ISO-8859-1) and C1 controls. 1977 1978As explained in L</ASCII Rules versus Unicode Rules>, under ASCII rules, 1979they are considered to be unassigned characters. 1980 1981This can lead to unexpected results. For example, a string's 1982semantics can suddenly change if a code point above 255 is appended to 1983it, which changes the rules from ASCII to Unicode. As an 1984example, consider the following program and its output: 1985 1986 $ perl -le' 1987 no feature "unicode_strings"; 1988 $s1 = "\xC2"; 1989 $s2 = "\x{2660}"; 1990 for ($s1, $s2, $s1.$s2) { 1991 print /\w/ || 0; 1992 } 1993 ' 1994 0 1995 0 1996 1 1997 1998If there's no C<\w> in C<s1> nor in C<s2>, why does their concatenation 1999have one? 2000 2001This anomaly stems from Perl's attempt to not disturb older programs that 2002didn't use Unicode, along with Perl's desire to add Unicode support 2003seamlessly. But the result turned out to not be seamless. (By the way, 2004you can choose to be warned when things like this happen. See 2005C<L<encoding::warnings>>.) 2006 2007L<S<C<use feature 'unicode_strings'>>|feature/The 'unicode_strings' feature> 2008was added, starting in Perl v5.12, to address this problem. It affects 2009these things: 2010 2011=over 4 2012 2013=item * 2014 2015Changing the case of a scalar, that is, using C<uc()>, C<ucfirst()>, C<lc()>, 2016and C<lcfirst()>, or C<\L>, C<\U>, C<\u> and C<\l> in double-quotish 2017contexts, such as regular expression substitutions. 2018 2019Under C<unicode_strings> starting in Perl 5.12.0, Unicode rules are 2020generally used. See L<perlfunc/lc> for details on how this works 2021in combination with various other pragmas. 2022 2023=item * 2024 2025Using caseless (C</i>) regular expression matching. 2026 2027Starting in Perl 5.14.0, regular expressions compiled within 2028the scope of C<unicode_strings> use Unicode rules 2029even when executed or compiled into larger 2030regular expressions outside the scope. 2031 2032=item * 2033 2034Matching any of several properties in regular expressions. 2035 2036These properties are C<\b> (without braces), C<\B> (without braces), 2037C<\s>, C<\S>, C<\w>, C<\W>, and all the Posix character classes 2038I<except> C<[[:ascii:]]>. 2039 2040Starting in Perl 5.14.0, regular expressions compiled within 2041the scope of C<unicode_strings> use Unicode rules 2042even when executed or compiled into larger 2043regular expressions outside the scope. 2044 2045=item * 2046 2047In C<quotemeta> or its inline equivalent C<\Q>. 2048 2049Starting in Perl 5.16.0, consistent quoting rules are used within the 2050scope of C<unicode_strings>, as described in L<perlfunc/quotemeta>. 2051Prior to that, or outside its scope, no code points above 127 are quoted 2052in UTF-8 encoded strings, but in byte encoded strings, code points 2053between 128-255 are always quoted. 2054 2055=item * 2056 2057In the C<..> or L<range|perlop/Range Operators> operator. 2058 2059Starting in Perl 5.26.0, the range operator on strings treats their lengths 2060consistently within the scope of C<unicode_strings>. Prior to that, or 2061outside its scope, it could produce strings whose length in characters 2062exceeded that of the right-hand side, where the right-hand side took up more 2063bytes than the correct range endpoint. 2064 2065=item * 2066 2067In L<< C<split>'s special-case whitespace splitting|perlfunc/split >>. 2068 2069Starting in Perl 5.28.0, the C<split> function with a pattern specified as 2070a string containing a single space handles whitespace characters consistently 2071within the scope of C<unicode_strings>. Prior to that, or outside its scope, 2072characters that are whitespace according to Unicode rules but not according to 2073ASCII rules were treated as field contents rather than field separators when 2074they appear in byte-encoded strings. 2075 2076=back 2077 2078You can see from the above that the effect of C<unicode_strings> 2079increased over several Perl releases. (And Perl's support for Unicode 2080continues to improve; it's best to use the latest available release in 2081order to get the most complete and accurate results possible.) Note that 2082C<unicode_strings> is automatically chosen if you S<C<use 5.012>> or 2083higher. 2084 2085For Perls earlier than those described above, or when a string is passed 2086to a function outside the scope of C<unicode_strings>, see the next section. 2087 2088=head2 Forcing Unicode in Perl (Or Unforcing Unicode in Perl) 2089 2090Sometimes (see L</"When Unicode Does Not Happen"> or L</The "Unicode Bug">) 2091there are situations where you simply need to force a byte 2092string into UTF-8, or vice versa. The standard module L<Encode> can be 2093used for this, or the low-level calls 2094L<C<utf8::upgrade($bytestring)>|utf8/Utility functions> and 2095L<C<utf8::downgrade($utf8string[, FAIL_OK])>|utf8/Utility functions>. 2096 2097Note that C<utf8::downgrade()> can fail if the string contains characters 2098that don't fit into a byte. 2099 2100Calling either function on a string that already is in the desired state is a 2101no-op. 2102 2103L</ASCII Rules versus Unicode Rules> gives all the ways that a string is 2104made to use Unicode rules. 2105 2106=head2 Using Unicode in XS 2107 2108See L<perlguts/"Unicode Support"> for an introduction to Unicode at 2109the XS level, and L<perlapi/Unicode Support> for the API details. 2110 2111=head2 Hacking Perl to work on earlier Unicode versions (for very serious hackers only) 2112 2113Perl by default comes with the latest supported Unicode version built-in, but 2114the goal is to allow you to change to use any earlier one. In Perls 2115v5.20 and v5.22, however, the earliest usable version is Unicode 5.1. 2116Perl v5.18 and v5.24 are able to handle all earlier versions. 2117 2118Download the files in the desired version of Unicode from the Unicode web 2119site L<https://www.unicode.org>). These should replace the existing files in 2120F<lib/unicore> in the Perl source tree. Follow the instructions in 2121F<README.perl> in that directory to change some of their names, and then build 2122perl (see L<INSTALL>). 2123 2124=head2 Porting code from perl-5.6.X 2125 2126Perls starting in 5.8 have a different Unicode model from 5.6. In 5.6 the 2127programmer was required to use the C<utf8> pragma to declare that a 2128given scope expected to deal with Unicode data and had to make sure that 2129only Unicode data were reaching that scope. If you have code that is 2130working with 5.6, you will need some of the following adjustments to 2131your code. The examples are written such that the code will continue to 2132work under 5.6, so you should be safe to try them out. 2133 2134=over 3 2135 2136=item * 2137 2138A filehandle that should read or write UTF-8 2139 2140 if ($] > 5.008) { 2141 binmode $fh, ":encoding(UTF-8)"; 2142 } 2143 2144=item * 2145 2146A scalar that is going to be passed to some extension 2147 2148Be it C<Compress::Zlib>, C<Apache::Request> or any extension that has no 2149mention of Unicode in the manpage, you need to make sure that the 2150UTF8 flag is stripped off. Note that at the time of this writing 2151(January 2012) the mentioned modules are not UTF-8-aware. Please 2152check the documentation to verify if this is still true. 2153 2154 if ($] > 5.008) { 2155 require Encode; 2156 $val = Encode::encode("UTF-8", $val); # make octets 2157 } 2158 2159=item * 2160 2161A scalar we got back from an extension 2162 2163If you believe the scalar comes back as UTF-8, you will most likely 2164want the UTF8 flag restored: 2165 2166 if ($] > 5.008) { 2167 require Encode; 2168 $val = Encode::decode("UTF-8", $val); 2169 } 2170 2171=item * 2172 2173Same thing, if you are really sure it is UTF-8 2174 2175 if ($] > 5.008) { 2176 require Encode; 2177 Encode::_utf8_on($val); 2178 } 2179 2180=item * 2181 2182A wrapper for L<DBI> C<fetchrow_array> and C<fetchrow_hashref> 2183 2184When the database contains only UTF-8, a wrapper function or method is 2185a convenient way to replace all your C<fetchrow_array> and 2186C<fetchrow_hashref> calls. A wrapper function will also make it easier to 2187adapt to future enhancements in your database driver. Note that at the 2188time of this writing (January 2012), the DBI has no standardized way 2189to deal with UTF-8 data. Please check the L<DBI documentation|DBI> to verify if 2190that is still true. 2191 2192 sub fetchrow { 2193 # $what is one of fetchrow_{array,hashref} 2194 my($self, $sth, $what) = @_; 2195 if ($] < 5.008) { 2196 return $sth->$what; 2197 } else { 2198 require Encode; 2199 if (wantarray) { 2200 my @arr = $sth->$what; 2201 for (@arr) { 2202 defined && /[^\000-\177]/ && Encode::_utf8_on($_); 2203 } 2204 return @arr; 2205 } else { 2206 my $ret = $sth->$what; 2207 if (ref $ret) { 2208 for my $k (keys %$ret) { 2209 defined 2210 && /[^\000-\177]/ 2211 && Encode::_utf8_on($_) for $ret->{$k}; 2212 } 2213 return $ret; 2214 } else { 2215 defined && /[^\000-\177]/ && Encode::_utf8_on($_) for $ret; 2216 return $ret; 2217 } 2218 } 2219 } 2220 } 2221 2222 2223=item * 2224 2225A large scalar that you know can only contain ASCII 2226 2227Scalars that contain only ASCII and are marked as UTF-8 are sometimes 2228a drag to your program. If you recognize such a situation, just remove 2229the UTF8 flag: 2230 2231 utf8::downgrade($val) if $] > 5.008; 2232 2233=back 2234 2235=head1 BUGS 2236 2237See also L</The "Unicode Bug"> above. 2238 2239=head2 Interaction with Extensions 2240 2241When Perl exchanges data with an extension, the extension should be 2242able to understand the UTF8 flag and act accordingly. If the 2243extension doesn't recognize that flag, it's likely that the extension 2244will return incorrectly-flagged data. 2245 2246So if you're working with Unicode data, consult the documentation of 2247every module you're using if there are any issues with Unicode data 2248exchange. If the documentation does not talk about Unicode at all, 2249suspect the worst and probably look at the source to learn how the 2250module is implemented. Modules written completely in Perl shouldn't 2251cause problems. Modules that directly or indirectly access code written 2252in other programming languages are at risk. 2253 2254For affected functions, the simple strategy to avoid data corruption is 2255to always make the encoding of the exchanged data explicit. Choose an 2256encoding that you know the extension can handle. Convert arguments passed 2257to the extensions to that encoding and convert results back from that 2258encoding. Write wrapper functions that do the conversions for you, so 2259you can later change the functions when the extension catches up. 2260 2261To provide an example, let's say the popular C<Foo::Bar::escape_html> 2262function doesn't deal with Unicode data yet. The wrapper function 2263would convert the argument to raw UTF-8 and convert the result back to 2264Perl's internal representation like so: 2265 2266 sub my_escape_html ($) { 2267 my($what) = shift; 2268 return unless defined $what; 2269 Encode::decode("UTF-8", Foo::Bar::escape_html( 2270 Encode::encode("UTF-8", $what))); 2271 } 2272 2273Sometimes, when the extension does not convert data but just stores 2274and retrieves it, you will be able to use the otherwise 2275dangerous L<C<Encode::_utf8_on()>|Encode/_utf8_on> function. Let's say 2276the popular C<Foo::Bar> extension, written in C, provides a C<param> 2277method that lets you store and retrieve data according to these prototypes: 2278 2279 $self->param($name, $value); # set a scalar 2280 $value = $self->param($name); # retrieve a scalar 2281 2282If it does not yet provide support for any encoding, one could write a 2283derived class with such a C<param> method: 2284 2285 sub param { 2286 my($self,$name,$value) = @_; 2287 utf8::upgrade($name); # make sure it is UTF-8 encoded 2288 if (defined $value) { 2289 utf8::upgrade($value); # make sure it is UTF-8 encoded 2290 return $self->SUPER::param($name,$value); 2291 } else { 2292 my $ret = $self->SUPER::param($name); 2293 Encode::_utf8_on($ret); # we know, it is UTF-8 encoded 2294 return $ret; 2295 } 2296 } 2297 2298Some extensions provide filters on data entry/exit points, such as 2299C<DB_File::filter_store_key> and family. Look out for such filters in 2300the documentation of your extensions; they can make the transition to 2301Unicode data much easier. 2302 2303=head2 Speed 2304 2305Some functions are slower when working on UTF-8 encoded strings than 2306on byte encoded strings. All functions that need to hop over 2307characters such as C<length()>, C<substr()> or C<index()>, or matching 2308regular expressions can work B<much> faster when the underlying data are 2309byte-encoded. 2310 2311In Perl 5.8.0 the slowness was often quite spectacular; in Perl 5.8.1 2312a caching scheme was introduced which improved the situation. In general, 2313operations with UTF-8 encoded strings are still slower. As an example, 2314the Unicode properties (character classes) like C<\p{Nd}> are known to 2315be quite a bit slower (5-20 times) than their simpler counterparts 2316like C<[0-9]> (then again, there are hundreds of Unicode characters matching 2317C<Nd> compared with the 10 ASCII characters matching C<[0-9]>). 2318 2319=head1 SEE ALSO 2320 2321L<perlunitut>, L<perluniintro>, L<perluniprops>, L<Encode>, L<open>, L<utf8>, L<bytes>, 2322L<perlretut>, L<perlvar/"${^UNICODE}">, 2323L<https://www.unicode.org/reports/tr44>). 2324 2325=cut 2326