1<section id="std.localization.facet.codecvt" xreflabel="codecvt"> 2<?dbhtml filename="codecvt.html"?> 3 4<sectioninfo> 5 <keywordset> 6 <keyword> 7 ISO C++ 8 </keyword> 9 <keyword> 10 codecvt 11 </keyword> 12 </keywordset> 13</sectioninfo> 14 15<title>codecvt</title> 16 17<para> 18The standard class codecvt attempts to address conversions between 19different character encoding schemes. In particular, the standard 20attempts to detail conversions between the implementation-defined wide 21characters (hereafter referred to as wchar_t) and the standard type 22char that is so beloved in classic <quote>C</quote> (which can now be 23referred to as narrow characters.) This document attempts to describe 24how the GNU libstdc++ implementation deals with the conversion between 25wide and narrow characters, and also presents a framework for dealing 26with the huge number of other encodings that iconv can convert, 27including Unicode and UTF8. Design issues and requirements are 28addressed, and examples of correct usage for both the required 29specializations for wide and narrow characters and the 30implementation-provided extended functionality are given. 31</para> 32 33<section id="facet.codecvt.req"> 34<title>Requirements</title> 35 36<para> 37Around page 425 of the C++ Standard, this charming heading comes into view: 38</para> 39 40<blockquote> 41<para> 4222.2.1.5 - Template class codecvt 43</para> 44</blockquote> 45 46<para> 47The text around the codecvt definition gives some clues: 48</para> 49 50<blockquote> 51<para> 52<emphasis> 53-1- The class codecvt<internT,externT,stateT> is for use when 54converting from one codeset to another, such as from wide characters 55to multibyte characters, between wide character encodings such as 56Unicode and EUC. 57</emphasis> 58</para> 59</blockquote> 60 61<para> 62Hmm. So, in some unspecified way, Unicode encodings and 63translations between other character sets should be handled by this 64class. 65</para> 66 67<blockquote> 68<para> 69<emphasis> 70-2- The stateT argument selects the pair of codesets being mapped between. 71</emphasis> 72</para> 73</blockquote> 74 75<para> 76Ah ha! Another clue... 77</para> 78 79<blockquote> 80<para> 81<emphasis> 82-3- The instantiations required in the Table ?? 83(lib.locale.category), namely codecvt<wchar_t,char,mbstate_t> and 84codecvt<char,char,mbstate_t>, convert the implementation-defined 85native character set. codecvt<char,char,mbstate_t> implements a 86degenerate conversion; it does not convert at 87all. codecvt<wchar_t,char,mbstate_t> converts between the native 88character sets for tiny and wide characters. Instantiations on 89mbstate_t perform conversion between encodings known to the library 90implementor. Other encodings can be converted by specializing on a 91user-defined stateT type. The stateT object can contain any state that 92is useful to communicate to or from the specialized do_convert member. 93</emphasis> 94</para> 95</blockquote> 96 97<para> 98At this point, a couple points become clear: 99</para> 100 101<para> 102One: The standard clearly implies that attempts to add non-required 103(yet useful and widely used) conversions need to do so through the 104third template parameter, stateT.</para> 105 106<para> 107Two: The required conversions, by specifying mbstate_t as the third 108template parameter, imply an implementation strategy that is mostly 109(or wholly) based on the underlying C library, and the functions 110mcsrtombs and wcsrtombs in particular.</para> 111</section> 112 113<section id="facet.codecvt.design"> 114<title>Design</title> 115 116<section id="codecvt.design.wchar_t_size"> 117 <title><type>wchar_t</type> Size</title> 118 119 <para> 120 The simple implementation detail of wchar_t's size seems to 121 repeatedly confound people. Many systems use a two byte, 122 unsigned integral type to represent wide characters, and use an 123 internal encoding of Unicode or UCS2. (See AIX, Microsoft NT, 124 Java, others.) Other systems, use a four byte, unsigned integral 125 type to represent wide characters, and use an internal encoding 126 of UCS4. (GNU/Linux systems using glibc, in particular.) The C 127 programming language (and thus C++) does not specify a specific 128 size for the type wchar_t. 129 </para> 130 131 <para> 132 Thus, portable C++ code cannot assume a byte size (or endianness) either. 133 </para> 134 </section> 135 136<section id="codecvt.design.unicode"> 137 <title>Support for Unicode</title> 138 <para> 139 Probably the most frequently asked question about code conversion 140 is: "So dudes, what's the deal with Unicode strings?" 141 The dude part is optional, but apparently the usefulness of 142 Unicode strings is pretty widely appreciated. Sadly, this specific 143 encoding (And other useful encodings like UTF8, UCS4, ISO 8859-10, 144 etc etc etc) are not mentioned in the C++ standard. 145 </para> 146 147 <para> 148 A couple of comments: 149 </para> 150 151 <para> 152 The thought that all one needs to convert between two arbitrary 153 codesets is two types and some kind of state argument is 154 unfortunate. In particular, encodings may be stateless. The naming 155 of the third parameter as stateT is unfortunate, as what is really 156 needed is some kind of generalized type that accounts for the 157 issues that abstract encodings will need. The minimum information 158 that is required includes: 159 </para> 160 161 <itemizedlist> 162 <listitem> 163 <para> 164 Identifiers for each of the codesets involved in the 165 conversion. For example, using the iconv family of functions 166 from the Single Unix Specification (what used to be called 167 X/Open) hosted on the GNU/Linux operating system allows 168 bi-directional mapping between far more than the following 169 tantalizing possibilities: 170 </para> 171 172 <para> 173 (An edited list taken from <code>`iconv --list`</code> on a 174 Red Hat 6.2/Intel system: 175 </para> 176 177<blockquote> 178<programlisting> 1798859_1, 8859_9, 10646-1:1993, 10646-1:1993/UCS4, ARABIC, ARABIC7, 180ASCII, EUC-CN, EUC-JP, EUC-KR, EUC-TW, GREEK-CCIcode, GREEK, GREEK7-OLD, 181GREEK7, GREEK8, HEBREW, ISO-8859-1, ISO-8859-2, ISO-8859-3, 182ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, 183ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14, 184ISO-8859-15, ISO-10646, ISO-10646/UCS2, ISO-10646/UCS4, 185ISO-10646/UTF-8, ISO-10646/UTF8, SHIFT-JIS, SHIFT_JIS, UCS-2, UCS-4, 186UCS2, UCS4, UNICODE, UNICODEBIG, UNICODELIcodeLE, US-ASCII, US, UTF-8, 187UTF-16, UTF8, UTF16). 188</programlisting> 189</blockquote> 190 191<para> 192For iconv-based implementations, string literals for each of the 193encodings (i.e. "UCS-2" and "UTF-8") are necessary, 194although for other, 195non-iconv implementations a table of enumerated values or some other 196mechanism may be required. 197</para> 198</listitem> 199 200<listitem><para> 201 Maximum length of the identifying string literal. 202</para></listitem> 203 204<listitem><para> 205 Some encodings require explicit endian-ness. As such, some kind 206 of endian marker or other byte-order marker will be necessary. See 207 "Footnotes for C/C++ developers" in Haible for more information on 208 UCS-2/Unicode endian issues. (Summary: big endian seems most likely, 209 however implementations, most notably Microsoft, vary.) 210</para></listitem> 211 212<listitem><para> 213 Types representing the conversion state, for conversions involving 214 the machinery in the "C" library, or the conversion descriptor, for 215 conversions using iconv (such as the type iconv_t.) Note that the 216 conversion descriptor encodes more information than a simple encoding 217 state type. 218</para></listitem> 219 220<listitem><para> 221 Conversion descriptors for both directions of encoding. (i.e., both 222 UCS-2 to UTF-8 and UTF-8 to UCS-2.) 223</para></listitem> 224 225<listitem><para> 226 Something to indicate if the conversion requested if valid. 227</para></listitem> 228 229<listitem><para> 230 Something to represent if the conversion descriptors are valid. 231</para></listitem> 232 233<listitem><para> 234 Some way to enforce strict type checking on the internal and 235 external types. As part of this, the size of the internal and 236 external types will need to be known. 237</para></listitem> 238</itemizedlist> 239</section> 240 241<section id="codecvt.design.issues"> 242 <title>Other Issues</title> 243<para> 244In addition, multi-threaded and multi-locale environments also impact 245the design and requirements for code conversions. In particular, they 246affect the required specialization codecvt<wchar_t, char, mbstate_t> 247when implemented using standard "C" functions. 248</para> 249 250<para> 251Three problems arise, one big, one of medium importance, and one small. 252</para> 253 254<para> 255First, the small: mcsrtombs and wcsrtombs may not be multithread-safe 256on all systems required by the GNU tools. For GNU/Linux and glibc, 257this is not an issue. 258</para> 259 260<para> 261Of medium concern, in the grand scope of things, is that the functions 262used to implement this specialization work on null-terminated 263strings. Buffers, especially file buffers, may not be null-terminated, 264thus giving conversions that end prematurely or are otherwise 265incorrect. Yikes! 266</para> 267 268<para> 269The last, and fundamental problem, is the assumption of a global 270locale for all the "C" functions referenced above. For something like 271C++ iostreams (where codecvt is explicitly used) the notion of 272multiple locales is fundamental. In practice, most users may not run 273into this limitation. However, as a quality of implementation issue, 274the GNU C++ library would like to offer a solution that allows 275multiple locales and or simultaneous usage with computationally 276correct results. In short, libstdc++ is trying to offer, as an 277option, a high-quality implementation, damn the additional complexity! 278</para> 279 280<para> 281For the required specialization codecvt<wchar_t, char, mbstate_t> , 282conversions are made between the internal character set (always UCS4 283on GNU/Linux) and whatever the currently selected locale for the 284LC_CTYPE category implements. 285</para> 286 287</section> 288 289</section> 290 291<section id="facet.codecvt.impl"> 292<title>Implementation</title> 293 294<para> 295The two required specializations are implemented as follows: 296</para> 297 298<para> 299<code> 300codecvt<char, char, mbstate_t> 301</code> 302</para> 303<para> 304This is a degenerate (i.e., does nothing) specialization. Implementing 305this was a piece of cake. 306</para> 307 308<para> 309<code> 310codecvt<char, wchar_t, mbstate_t> 311</code> 312</para> 313 314<para> 315This specialization, by specifying all the template parameters, pretty 316much ties the hands of implementors. As such, the implementation is 317straightforward, involving mcsrtombs for the conversions between char 318to wchar_t and wcsrtombs for conversions between wchar_t and char. 319</para> 320 321<para> 322Neither of these two required specializations deals with Unicode 323characters. As such, libstdc++ implements a partial specialization 324of the codecvt class with and iconv wrapper class, encoding_state as the 325third template parameter. 326</para> 327 328<para> 329This implementation should be standards conformant. First of all, the 330standard explicitly points out that instantiations on the third 331template parameter, stateT, are the proper way to implement 332non-required conversions. Second of all, the standard says (in Chapter 33317) that partial specializations of required classes are a-ok. Third 334of all, the requirements for the stateT type elsewhere in the standard 335(see 21.1.2 traits typedefs) only indicate that this type be copy 336constructible. 337</para> 338 339<para> 340As such, the type encoding_state is defined as a non-templatized, POD 341type to be used as the third type of a codecvt instantiation. This 342type is just a wrapper class for iconv, and provides an easy interface 343to iconv functionality. 344</para> 345 346<para> 347There are two constructors for encoding_state: 348</para> 349 350<para> 351<code> 352encoding_state() : __in_desc(0), __out_desc(0) 353</code> 354</para> 355<para> 356This default constructor sets the internal encoding to some default 357(currently UCS4) and the external encoding to whatever is returned by 358nl_langinfo(CODESET). 359</para> 360 361<para> 362<code> 363encoding_state(const char* __int, const char* __ext) 364</code> 365</para> 366 367<para> 368This constructor takes as parameters string literals that indicate the 369desired internal and external encoding. There are no defaults for 370either argument. 371</para> 372 373<para> 374One of the issues with iconv is that the string literals identifying 375conversions are not standardized. Because of this, the thought of 376mandating and or enforcing some set of pre-determined valid 377identifiers seems iffy: thus, a more practical (and non-migraine 378inducing) strategy was implemented: end-users can specify any string 379(subject to a pre-determined length qualifier, currently 32 bytes) for 380encodings. It is up to the user to make sure that these strings are 381valid on the target system. 382</para> 383 384<para> 385<code> 386void 387_M_init() 388</code> 389</para> 390<para> 391Strangely enough, this member function attempts to open conversion 392descriptors for a given encoding_state object. If the conversion 393descriptors are not valid, the conversion descriptors returned will 394not be valid and the resulting calls to the codecvt conversion 395functions will return error. 396</para> 397 398<para> 399<code> 400bool 401_M_good() 402</code> 403</para> 404 405<para> 406Provides a way to see if the given encoding_state object has been 407properly initialized. If the string literals describing the desired 408internal and external encoding are not valid, initialization will 409fail, and this will return false. If the internal and external 410encodings are valid, but iconv_open could not allocate conversion 411descriptors, this will also return false. Otherwise, the object is 412ready to convert and will return true. 413</para> 414 415<para> 416<code> 417encoding_state(const encoding_state&) 418</code> 419</para> 420 421<para> 422As iconv allocates memory and sets up conversion descriptors, the copy 423constructor can only copy the member data pertaining to the internal 424and external code conversions, and not the conversion descriptors 425themselves. 426</para> 427 428<para> 429Definitions for all the required codecvt member functions are provided 430for this specialization, and usage of codecvt<internal character type, 431external character type, encoding_state> is consistent with other 432codecvt usage. 433</para> 434 435</section> 436 437<section id="facet.codecvt.use"> 438<title>Use</title> 439<para>A conversions involving string literal.</para> 440 441<programlisting> 442 typedef codecvt_base::result result; 443 typedef unsigned short unicode_t; 444 typedef unicode_t int_type; 445 typedef char ext_type; 446 typedef encoding_state state_type; 447 typedef codecvt<int_type, ext_type, state_type> unicode_codecvt; 448 449 const ext_type* e_lit = "black pearl jasmine tea"; 450 int size = strlen(e_lit); 451 int_type i_lit_base[24] = 452 { 25088, 27648, 24832, 25344, 27392, 8192, 28672, 25856, 24832, 29184, 453 27648, 8192, 27136, 24832, 29440, 27904, 26880, 28160, 25856, 8192, 29696, 454 25856, 24832, 2560 455 }; 456 const int_type* i_lit = i_lit_base; 457 const ext_type* efrom_next; 458 const int_type* ifrom_next; 459 ext_type* e_arr = new ext_type[size + 1]; 460 ext_type* eto_next; 461 int_type* i_arr = new int_type[size + 1]; 462 int_type* ito_next; 463 464 // construct a locale object with the specialized facet. 465 locale loc(locale::classic(), new unicode_codecvt); 466 // sanity check the constructed locale has the specialized facet. 467 VERIFY( has_facet<unicode_codecvt>(loc) ); 468 const unicode_codecvt& cvt = use_facet<unicode_codecvt>(loc); 469 // convert between const char* and unicode strings 470 unicode_codecvt::state_type state01("UNICODE", "ISO_8859-1"); 471 initialize_state(state01); 472 result r1 = cvt.in(state01, e_lit, e_lit + size, efrom_next, 473 i_arr, i_arr + size, ito_next); 474 VERIFY( r1 == codecvt_base::ok ); 475 VERIFY( !int_traits::compare(i_arr, i_lit, size) ); 476 VERIFY( efrom_next == e_lit + size ); 477 VERIFY( ito_next == i_arr + size ); 478</programlisting> 479 480</section> 481 482<section id="facet.codecvt.future"> 483<title>Future</title> 484<itemizedlist> 485<listitem> 486 <para> 487 a. things that are sketchy, or remain unimplemented: 488 do_encoding, max_length and length member functions 489 are only weakly implemented. I have no idea how to do 490 this correctly, and in a generic manner. Nathan? 491</para> 492</listitem> 493 494<listitem> 495 <para> 496 b. conversions involving std::string 497 </para> 498 <itemizedlist> 499 <listitem><para> 500 how should operators != and == work for string of 501 different/same encoding? 502 </para></listitem> 503 504 <listitem><para> 505 what is equal? A byte by byte comparison or an 506 encoding then byte comparison? 507 </para></listitem> 508 509 <listitem><para> 510 conversions between narrow, wide, and unicode strings 511 </para></listitem> 512 </itemizedlist> 513</listitem> 514<listitem><para> 515 c. conversions involving std::filebuf and std::ostream 516</para> 517 <itemizedlist> 518 <listitem><para> 519 how to initialize the state object in a 520 standards-conformant manner? 521 </para></listitem> 522 523 <listitem><para> 524 how to synchronize the "C" and "C++" 525 conversion information? 526 </para></listitem> 527 528 <listitem><para> 529 wchar_t/char internal buffers and conversions between 530 internal/external buffers? 531 </para></listitem> 532 </itemizedlist> 533</listitem> 534</itemizedlist> 535</section> 536 537 538<bibliography id="facet.codecvt.biblio"> 539<title>Bibliography</title> 540 541 <biblioentry> 542 <title> 543 The GNU C Library 544 </title> 545 <author> 546 <surname>McGrath</surname> 547 <firstname>Roland</firstname> 548 </author> 549 <author> 550 <surname>Drepper</surname> 551 <firstname>Ulrich</firstname> 552 </author> 553 <copyright> 554 <year>2007</year> 555 <holder>FSF</holder> 556 </copyright> 557 <pagenums> 558 Chapters 6 Character Set Handling and 7 Locales and Internationalization 559 </pagenums> 560 </biblioentry> 561 562 <biblioentry> 563 <title> 564 Correspondence 565 </title> 566 <author> 567 <surname>Drepper</surname> 568 <firstname>Ulrich</firstname> 569 </author> 570 <copyright> 571 <year>2002</year> 572 <holder></holder> 573 </copyright> 574 </biblioentry> 575 576 <biblioentry> 577 <title> 578 ISO/IEC 14882:1998 Programming languages - C++ 579 </title> 580 <copyright> 581 <year>1998</year> 582 <holder>ISO</holder> 583 </copyright> 584 </biblioentry> 585 586 <biblioentry> 587 <title> 588 ISO/IEC 9899:1999 Programming languages - C 589 </title> 590 <copyright> 591 <year>1999</year> 592 <holder>ISO</holder> 593 </copyright> 594 </biblioentry> 595 596 <biblioentry> 597 <biblioid class="uri"> 598 <ulink url="http://www.opengroup.org/austin/"> 599 <citetitle> 600 System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008) 601 </citetitle> 602 </ulink> 603 </biblioid> 604 <copyright> 605 <year>2008</year> 606 <holder> 607 The Open Group/The Institute of Electrical and Electronics 608 Engineers, Inc. 609 </holder> 610 </copyright> 611 </biblioentry> 612 613 <biblioentry> 614 <title> 615 The C++ Programming Language, Special Edition 616 </title> 617 <author> 618 <surname>Stroustrup</surname> 619 <firstname>Bjarne</firstname> 620 </author> 621 <copyright> 622 <year>2000</year> 623 <holder>Addison Wesley, Inc.</holder> 624 </copyright> 625 <pagenums>Appendix D</pagenums> 626 <publisher> 627 <publishername> 628 Addison Wesley 629 </publishername> 630 </publisher> 631 </biblioentry> 632 633 634 <biblioentry> 635 <title> 636 Standard C++ IOStreams and Locales 637 </title> 638 <subtitle> 639 Advanced Programmer's Guide and Reference 640 </subtitle> 641 <author> 642 <surname>Langer</surname> 643 <firstname>Angelika</firstname> 644 </author> 645 <author> 646 <surname>Kreft</surname> 647 <firstname>Klaus</firstname> 648 </author> 649 <copyright> 650 <year>2000</year> 651 <holder>Addison Wesley Longman, Inc.</holder> 652 </copyright> 653 <publisher> 654 <publishername> 655 Addison Wesley Longman 656 </publishername> 657 </publisher> 658 </biblioentry> 659 660 <biblioentry> 661 <biblioid class="uri"> 662 <ulink url="http://www.lysator.liu.se/c/na1.html"> 663 <citetitle> 664 A brief description of Normative Addendum 1 665 </citetitle> 666 </ulink> 667 </biblioid> 668 <author> 669 <surname>Feather</surname> 670 <firstname>Clive</firstname> 671 </author> 672 <pagenums>Extended Character Sets</pagenums> 673 </biblioentry> 674 675 <biblioentry> 676 <biblioid class="uri"> 677 <ulink url="http://tldp.org/HOWTO/Unicode-HOWTO.html"> 678 <citetitle> 679 The Unicode HOWTO 680 </citetitle> 681 </ulink> 682 </biblioid> 683 <author> 684 <surname>Haible</surname> 685 <firstname>Bruno</firstname> 686 </author> 687 </biblioentry> 688 689 <biblioentry> 690 <biblioid class="uri"> 691 <ulink url="http://www.cl.cam.ac.uk/~mgk25/unicode.html"> 692 <citetitle> 693 UTF-8 and Unicode FAQ for Unix/Linux 694 </citetitle> 695 </ulink> 696 </biblioid> 697 <author> 698 <surname>Khun</surname> 699 <firstname>Markus</firstname> 700 </author> 701 </biblioentry> 702 703</bibliography> 704 705</section> 706