1<section xmlns="http://docbook.org/ns/docbook" version="5.0" 2 xml:id="std.localization.facet.codecvt" xreflabel="codecvt"> 3<?dbhtml filename="codecvt.html"?> 4 5<info><title>codecvt</title> 6 <keywordset> 7 <keyword>ISO C++</keyword> 8 <keyword>codecvt</keyword> 9 </keywordset> 10</info> 11 12 13 14<para> 15The standard class codecvt attempts to address conversions between 16different character encoding schemes. In particular, the standard 17attempts to detail conversions between the implementation-defined wide 18characters (hereafter referred to as wchar_t) and the standard type 19char that is so beloved in classic <quote>C</quote> (which can now be 20referred to as narrow characters.) This document attempts to describe 21how the GNU libstdc++ implementation deals with the conversion between 22wide and narrow characters, and also presents a framework for dealing 23with the huge number of other encodings that iconv can convert, 24including Unicode and UTF8. Design issues and requirements are 25addressed, and examples of correct usage for both the required 26specializations for wide and narrow characters and the 27implementation-provided extended functionality are given. 28</para> 29 30<section xml:id="facet.codecvt.req"><info><title>Requirements</title></info> 31 32 33<para> 34Around page 425 of the C++ Standard, this charming heading comes into view: 35</para> 36 37<blockquote> 38<para> 3922.2.1.5 - Template class codecvt 40</para> 41</blockquote> 42 43<para> 44The text around the codecvt definition gives some clues: 45</para> 46 47<blockquote> 48<para> 49<emphasis> 50-1- The class codecvt<internT,externT,stateT> is for use when 51converting from one codeset to another, such as from wide characters 52to multibyte characters, between wide character encodings such as 53Unicode and EUC. 54</emphasis> 55</para> 56</blockquote> 57 58<para> 59Hmm. So, in some unspecified way, Unicode encodings and 60translations between other character sets should be handled by this 61class. 62</para> 63 64<blockquote> 65<para> 66<emphasis> 67-2- The stateT argument selects the pair of codesets being mapped between. 68</emphasis> 69</para> 70</blockquote> 71 72<para> 73Ah ha! Another clue... 74</para> 75 76<blockquote> 77<para> 78<emphasis> 79-3- The instantiations required in the Table ?? 80(lib.locale.category), namely codecvt<wchar_t,char,mbstate_t> and 81codecvt<char,char,mbstate_t>, convert the implementation-defined 82native character set. codecvt<char,char,mbstate_t> implements a 83degenerate conversion; it does not convert at 84all. codecvt<wchar_t,char,mbstate_t> converts between the native 85character sets for tiny and wide characters. Instantiations on 86mbstate_t perform conversion between encodings known to the library 87implementor. Other encodings can be converted by specializing on a 88user-defined stateT type. The stateT object can contain any state that 89is useful to communicate to or from the specialized do_convert member. 90</emphasis> 91</para> 92</blockquote> 93 94<para> 95At this point, a couple points become clear: 96</para> 97 98<para> 99One: The standard clearly implies that attempts to add non-required 100(yet useful and widely used) conversions need to do so through the 101third template parameter, stateT.</para> 102 103<para> 104Two: The required conversions, by specifying mbstate_t as the third 105template parameter, imply an implementation strategy that is mostly 106(or wholly) based on the underlying C library, and the functions 107mcsrtombs and wcsrtombs in particular.</para> 108</section> 109 110<section xml:id="facet.codecvt.design"><info><title>Design</title></info> 111 112 113<section xml:id="codecvt.design.wchar_t_size"><info><title><type>wchar_t</type> Size</title></info> 114 115 116 <para> 117 The simple implementation detail of wchar_t's size seems to 118 repeatedly confound people. Many systems use a two byte, 119 unsigned integral type to represent wide characters, and use an 120 internal encoding of Unicode or UCS2. (See AIX, Microsoft NT, 121 Java, others.) Other systems, use a four byte, unsigned integral 122 type to represent wide characters, and use an internal encoding 123 of UCS4. (GNU/Linux systems using glibc, in particular.) The C 124 programming language (and thus C++) does not specify a specific 125 size for the type wchar_t. 126 </para> 127 128 <para> 129 Thus, portable C++ code cannot assume a byte size (or endianness) either. 130 </para> 131 </section> 132 133<section xml:id="codecvt.design.unicode"><info><title>Support for Unicode</title></info> 134 135 <para> 136 Probably the most frequently asked question about code conversion 137 is: "So dudes, what's the deal with Unicode strings?" 138 The dude part is optional, but apparently the usefulness of 139 Unicode strings is pretty widely appreciated. Sadly, this specific 140 encoding (And other useful encodings like UTF8, UCS4, ISO 8859-10, 141 etc etc etc) are not mentioned in the C++ standard. 142 </para> 143 144 <para> 145 A couple of comments: 146 </para> 147 148 <para> 149 The thought that all one needs to convert between two arbitrary 150 codesets is two types and some kind of state argument is 151 unfortunate. In particular, encodings may be stateless. The naming 152 of the third parameter as stateT is unfortunate, as what is really 153 needed is some kind of generalized type that accounts for the 154 issues that abstract encodings will need. The minimum information 155 that is required includes: 156 </para> 157 158 <itemizedlist> 159 <listitem> 160 <para> 161 Identifiers for each of the codesets involved in the 162 conversion. For example, using the iconv family of functions 163 from the Single Unix Specification (what used to be called 164 X/Open) hosted on the GNU/Linux operating system allows 165 bi-directional mapping between far more than the following 166 tantalizing possibilities: 167 </para> 168 169 <para> 170 (An edited list taken from <code>`iconv --list`</code> on a 171 Red Hat 6.2/Intel system: 172 </para> 173 174<blockquote> 175<programlisting> 1768859_1, 8859_9, 10646-1:1993, 10646-1:1993/UCS4, ARABIC, ARABIC7, 177ASCII, EUC-CN, EUC-JP, EUC-KR, EUC-TW, GREEK-CCIcode, GREEK, GREEK7-OLD, 178GREEK7, GREEK8, HEBREW, ISO-8859-1, ISO-8859-2, ISO-8859-3, 179ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, 180ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14, 181ISO-8859-15, ISO-10646, ISO-10646/UCS2, ISO-10646/UCS4, 182ISO-10646/UTF-8, ISO-10646/UTF8, SHIFT-JIS, SHIFT_JIS, UCS-2, UCS-4, 183UCS2, UCS4, UNICODE, UNICODEBIG, UNICODELIcodeLE, US-ASCII, US, UTF-8, 184UTF-16, UTF8, UTF16). 185</programlisting> 186</blockquote> 187 188<para> 189For iconv-based implementations, string literals for each of the 190encodings (i.e. "UCS-2" and "UTF-8") are necessary, 191although for other, 192non-iconv implementations a table of enumerated values or some other 193mechanism may be required. 194</para> 195</listitem> 196 197<listitem><para> 198 Maximum length of the identifying string literal. 199</para></listitem> 200 201<listitem><para> 202 Some encodings require explicit endian-ness. As such, some kind 203 of endian marker or other byte-order marker will be necessary. See 204 "Footnotes for C/C++ developers" in Haible for more information on 205 UCS-2/Unicode endian issues. (Summary: big endian seems most likely, 206 however implementations, most notably Microsoft, vary.) 207</para></listitem> 208 209<listitem><para> 210 Types representing the conversion state, for conversions involving 211 the machinery in the "C" library, or the conversion descriptor, for 212 conversions using iconv (such as the type iconv_t.) Note that the 213 conversion descriptor encodes more information than a simple encoding 214 state type. 215</para></listitem> 216 217<listitem><para> 218 Conversion descriptors for both directions of encoding. (i.e., both 219 UCS-2 to UTF-8 and UTF-8 to UCS-2.) 220</para></listitem> 221 222<listitem><para> 223 Something to indicate if the conversion requested if valid. 224</para></listitem> 225 226<listitem><para> 227 Something to represent if the conversion descriptors are valid. 228</para></listitem> 229 230<listitem><para> 231 Some way to enforce strict type checking on the internal and 232 external types. As part of this, the size of the internal and 233 external types will need to be known. 234</para></listitem> 235</itemizedlist> 236</section> 237 238<section xml:id="codecvt.design.issues"><info><title>Other Issues</title></info> 239 240<para> 241In addition, multi-threaded and multi-locale environments also impact 242the design and requirements for code conversions. In particular, they 243affect the required specialization codecvt<wchar_t, char, mbstate_t> 244when implemented using standard "C" functions. 245</para> 246 247<para> 248Three problems arise, one big, one of medium importance, and one small. 249</para> 250 251<para> 252First, the small: mcsrtombs and wcsrtombs may not be multithread-safe 253on all systems required by the GNU tools. For GNU/Linux and glibc, 254this is not an issue. 255</para> 256 257<para> 258Of medium concern, in the grand scope of things, is that the functions 259used to implement this specialization work on null-terminated 260strings. Buffers, especially file buffers, may not be null-terminated, 261thus giving conversions that end prematurely or are otherwise 262incorrect. Yikes! 263</para> 264 265<para> 266The last, and fundamental problem, is the assumption of a global 267locale for all the "C" functions referenced above. For something like 268C++ iostreams (where codecvt is explicitly used) the notion of 269multiple locales is fundamental. In practice, most users may not run 270into this limitation. However, as a quality of implementation issue, 271the GNU C++ library would like to offer a solution that allows 272multiple locales and or simultaneous usage with computationally 273correct results. In short, libstdc++ is trying to offer, as an 274option, a high-quality implementation, damn the additional complexity! 275</para> 276 277<para> 278For the required specialization codecvt<wchar_t, char, mbstate_t> , 279conversions are made between the internal character set (always UCS4 280on GNU/Linux) and whatever the currently selected locale for the 281LC_CTYPE category implements. 282</para> 283 284</section> 285 286</section> 287 288<section xml:id="facet.codecvt.impl"><info><title>Implementation</title></info> 289 290 291<para> 292The two required specializations are implemented as follows: 293</para> 294 295<para> 296<code> 297codecvt<char, char, mbstate_t> 298</code> 299</para> 300<para> 301This is a degenerate (i.e., does nothing) specialization. Implementing 302this was a piece of cake. 303</para> 304 305<para> 306<code> 307codecvt<char, wchar_t, mbstate_t> 308</code> 309</para> 310 311<para> 312This specialization, by specifying all the template parameters, pretty 313much ties the hands of implementors. As such, the implementation is 314straightforward, involving mcsrtombs for the conversions between char 315to wchar_t and wcsrtombs for conversions between wchar_t and char. 316</para> 317 318<para> 319Neither of these two required specializations deals with Unicode 320characters. As such, libstdc++ implements a partial specialization 321of the codecvt class with and iconv wrapper class, encoding_state as the 322third template parameter. 323</para> 324 325<para> 326This implementation should be standards conformant. First of all, the 327standard explicitly points out that instantiations on the third 328template parameter, stateT, are the proper way to implement 329non-required conversions. Second of all, the standard says (in Chapter 33017) that partial specializations of required classes are a-ok. Third 331of all, the requirements for the stateT type elsewhere in the standard 332(see 21.1.2 traits typedefs) only indicate that this type be copy 333constructible. 334</para> 335 336<para> 337As such, the type encoding_state is defined as a non-templatized, POD 338type to be used as the third type of a codecvt instantiation. This 339type is just a wrapper class for iconv, and provides an easy interface 340to iconv functionality. 341</para> 342 343<para> 344There are two constructors for encoding_state: 345</para> 346 347<para> 348<code> 349encoding_state() : __in_desc(0), __out_desc(0) 350</code> 351</para> 352<para> 353This default constructor sets the internal encoding to some default 354(currently UCS4) and the external encoding to whatever is returned by 355nl_langinfo(CODESET). 356</para> 357 358<para> 359<code> 360encoding_state(const char* __int, const char* __ext) 361</code> 362</para> 363 364<para> 365This constructor takes as parameters string literals that indicate the 366desired internal and external encoding. There are no defaults for 367either argument. 368</para> 369 370<para> 371One of the issues with iconv is that the string literals identifying 372conversions are not standardized. Because of this, the thought of 373mandating and or enforcing some set of pre-determined valid 374identifiers seems iffy: thus, a more practical (and non-migraine 375inducing) strategy was implemented: end-users can specify any string 376(subject to a pre-determined length qualifier, currently 32 bytes) for 377encodings. It is up to the user to make sure that these strings are 378valid on the target system. 379</para> 380 381<para> 382<code> 383void 384_M_init() 385</code> 386</para> 387<para> 388Strangely enough, this member function attempts to open conversion 389descriptors for a given encoding_state object. If the conversion 390descriptors are not valid, the conversion descriptors returned will 391not be valid and the resulting calls to the codecvt conversion 392functions will return error. 393</para> 394 395<para> 396<code> 397bool 398_M_good() 399</code> 400</para> 401 402<para> 403Provides a way to see if the given encoding_state object has been 404properly initialized. If the string literals describing the desired 405internal and external encoding are not valid, initialization will 406fail, and this will return false. If the internal and external 407encodings are valid, but iconv_open could not allocate conversion 408descriptors, this will also return false. Otherwise, the object is 409ready to convert and will return true. 410</para> 411 412<para> 413<code> 414encoding_state(const encoding_state&) 415</code> 416</para> 417 418<para> 419As iconv allocates memory and sets up conversion descriptors, the copy 420constructor can only copy the member data pertaining to the internal 421and external code conversions, and not the conversion descriptors 422themselves. 423</para> 424 425<para> 426Definitions for all the required codecvt member functions are provided 427for this specialization, and usage of codecvt<internal character type, 428external character type, encoding_state> is consistent with other 429codecvt usage. 430</para> 431 432</section> 433 434<section xml:id="facet.codecvt.use"><info><title>Use</title></info> 435 436<para>A conversions involving string literal.</para> 437 438<programlisting> 439 typedef codecvt_base::result result; 440 typedef unsigned short unicode_t; 441 typedef unicode_t int_type; 442 typedef char ext_type; 443 typedef encoding_state state_type; 444 typedef codecvt<int_type, ext_type, state_type> unicode_codecvt; 445 446 const ext_type* e_lit = "black pearl jasmine tea"; 447 int size = strlen(e_lit); 448 int_type i_lit_base[24] = 449 { 25088, 27648, 24832, 25344, 27392, 8192, 28672, 25856, 24832, 29184, 450 27648, 8192, 27136, 24832, 29440, 27904, 26880, 28160, 25856, 8192, 29696, 451 25856, 24832, 2560 452 }; 453 const int_type* i_lit = i_lit_base; 454 const ext_type* efrom_next; 455 const int_type* ifrom_next; 456 ext_type* e_arr = new ext_type[size + 1]; 457 ext_type* eto_next; 458 int_type* i_arr = new int_type[size + 1]; 459 int_type* ito_next; 460 461 // construct a locale object with the specialized facet. 462 locale loc(locale::classic(), new unicode_codecvt); 463 // sanity check the constructed locale has the specialized facet. 464 VERIFY( has_facet<unicode_codecvt>(loc) ); 465 const unicode_codecvt& cvt = use_facet<unicode_codecvt>(loc); 466 // convert between const char* and unicode strings 467 unicode_codecvt::state_type state01("UNICODE", "ISO_8859-1"); 468 initialize_state(state01); 469 result r1 = cvt.in(state01, e_lit, e_lit + size, efrom_next, 470 i_arr, i_arr + size, ito_next); 471 VERIFY( r1 == codecvt_base::ok ); 472 VERIFY( !int_traits::compare(i_arr, i_lit, size) ); 473 VERIFY( efrom_next == e_lit + size ); 474 VERIFY( ito_next == i_arr + size ); 475</programlisting> 476 477</section> 478 479<section xml:id="facet.codecvt.future"><info><title>Future</title></info> 480 481<itemizedlist> 482<listitem> 483 <para> 484 a. things that are sketchy, or remain unimplemented: 485 do_encoding, max_length and length member functions 486 are only weakly implemented. I have no idea how to do 487 this correctly, and in a generic manner. Nathan? 488</para> 489</listitem> 490 491<listitem> 492 <para> 493 b. conversions involving std::string 494 </para> 495 <itemizedlist> 496 <listitem><para> 497 how should operators != and == work for string of 498 different/same encoding? 499 </para></listitem> 500 501 <listitem><para> 502 what is equal? A byte by byte comparison or an 503 encoding then byte comparison? 504 </para></listitem> 505 506 <listitem><para> 507 conversions between narrow, wide, and unicode strings 508 </para></listitem> 509 </itemizedlist> 510</listitem> 511<listitem><para> 512 c. conversions involving std::filebuf and std::ostream 513</para> 514 <itemizedlist> 515 <listitem><para> 516 how to initialize the state object in a 517 standards-conformant manner? 518 </para></listitem> 519 520 <listitem><para> 521 how to synchronize the "C" and "C++" 522 conversion information? 523 </para></listitem> 524 525 <listitem><para> 526 wchar_t/char internal buffers and conversions between 527 internal/external buffers? 528 </para></listitem> 529 </itemizedlist> 530</listitem> 531</itemizedlist> 532</section> 533 534 535<bibliography xml:id="facet.codecvt.biblio"><info><title>Bibliography</title></info> 536 537 538 <biblioentry> 539 <citetitle> 540 The GNU C Library 541 </citetitle> 542 <author><personname><surname>McGrath</surname><firstname>Roland</firstname></personname></author> 543 <author><personname><surname>Drepper</surname><firstname>Ulrich</firstname></personname></author> 544 <copyright> 545 <year>2007</year> 546 <holder>FSF</holder> 547 </copyright> 548 <pagenums> 549 Chapters 6 Character Set Handling and 7 Locales and Internationalization 550 </pagenums> 551 </biblioentry> 552 553 <biblioentry> 554 <citetitle> 555 Correspondence 556 </citetitle> 557 <author><personname><surname>Drepper</surname><firstname>Ulrich</firstname></personname></author> 558 <copyright> 559 <year>2002</year> 560 <holder/> 561 </copyright> 562 </biblioentry> 563 564 <biblioentry> 565 <citetitle> 566 ISO/IEC 14882:1998 Programming languages - C++ 567 </citetitle> 568 <copyright> 569 <year>1998</year> 570 <holder>ISO</holder> 571 </copyright> 572 </biblioentry> 573 574 <biblioentry> 575 <citetitle> 576 ISO/IEC 9899:1999 Programming languages - C 577 </citetitle> 578 <copyright> 579 <year>1999</year> 580 <holder>ISO</holder> 581 </copyright> 582 </biblioentry> 583 584 <biblioentry> 585 <title> 586 <link xmlns:xlink="http://www.w3.org/1999/xlink" 587 xlink:href="http://www.opengroup.org/austin/"> 588 System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008) 589 </link> 590 </title> 591 592 <copyright> 593 <year>2008</year> 594 <holder> 595 The Open Group/The Institute of Electrical and Electronics 596 Engineers, Inc. 597 </holder> 598 </copyright> 599 </biblioentry> 600 601 <biblioentry> 602 <citetitle> 603 The C++ Programming Language, Special Edition 604 </citetitle> 605 <author><personname><surname>Stroustrup</surname><firstname>Bjarne</firstname></personname></author> 606 <copyright> 607 <year>2000</year> 608 <holder>Addison Wesley, Inc.</holder> 609 </copyright> 610 <pagenums>Appendix D</pagenums> 611 <publisher> 612 <publishername> 613 Addison Wesley 614 </publishername> 615 </publisher> 616 </biblioentry> 617 618 619 <biblioentry> 620 <citetitle> 621 Standard C++ IOStreams and Locales 622 </citetitle> 623 <subtitle> 624 Advanced Programmer's Guide and Reference 625 </subtitle> 626 <author><personname><surname>Langer</surname><firstname>Angelika</firstname></personname></author> 627 <author><personname><surname>Kreft</surname><firstname>Klaus</firstname></personname></author> 628 <copyright> 629 <year>2000</year> 630 <holder>Addison Wesley Longman, Inc.</holder> 631 </copyright> 632 <publisher> 633 <publishername> 634 Addison Wesley Longman 635 </publishername> 636 </publisher> 637 </biblioentry> 638 639 <biblioentry> 640 <title> 641 <link xmlns:xlink="http://www.w3.org/1999/xlink" 642 xlink:href="http://www.lysator.liu.se/c/na1.html"> 643 A brief description of Normative Addendum 1 644 </link> 645 </title> 646 647 <author><personname><surname>Feather</surname><firstname>Clive</firstname></personname></author> 648 <pagenums>Extended Character Sets</pagenums> 649 </biblioentry> 650 651 <biblioentry> 652 <title> 653 <link xmlns:xlink="http://www.w3.org/1999/xlink" 654 xlink:href="http://tldp.org/HOWTO/Unicode-HOWTO.html"> 655 The Unicode HOWTO 656 </link> 657 </title> 658 659 <author><personname><surname>Haible</surname><firstname>Bruno</firstname></personname></author> 660 </biblioentry> 661 662 <biblioentry> 663 <title> 664 <link xmlns:xlink="http://www.w3.org/1999/xlink" 665 xlink:href="http://www.cl.cam.ac.uk/~mgk25/unicode.html"> 666 UTF-8 and Unicode FAQ for Unix/Linux 667 </link> 668 </title> 669 670 671 <author><personname><surname>Khun</surname><firstname>Markus</firstname></personname></author> 672 </biblioentry> 673 674</bibliography> 675 676</section> 677