1 <section xmlns="http://docbook.org/ns/docbook" version="5.0"
2 xml:id="std.localization.facet.codecvt" xreflabel="codecvt">
3 <?dbhtml filename="codecvt.html"?>
5 <info><title>codecvt</title>
7 <keyword>ISO C++</keyword>
8 <keyword>codecvt</keyword>
15 The standard class codecvt attempts to address conversions between
16 different character encoding schemes. In particular, the standard
17 attempts to detail conversions between the implementation-defined wide
18 characters (hereafter referred to as wchar_t) and the standard type
19 char that is so beloved in classic <quote>C</quote> (which can now be
20 referred to as narrow characters.) This document attempts to describe
21 how the GNU libstdc++ implementation deals with the conversion between
22 wide and narrow characters, and also presents a framework for dealing
23 with the huge number of other encodings that iconv can convert,
24 including Unicode and UTF8. Design issues and requirements are
25 addressed, and examples of correct usage for both the required
26 specializations for wide and narrow characters and the
27 implementation-provided extended functionality are given.
30 <section xml:id="facet.codecvt.req"><info><title>Requirements</title></info>
34 Around page 425 of the C++ Standard, this charming heading comes into view:
39 22.2.1.5 - Template class codecvt
44 The text around the codecvt definition gives some clues:
50 -1- The class codecvt<internT,externT,stateT> is for use when
51 converting from one codeset to another, such as from wide characters
52 to multibyte characters, between wide character encodings such as
59 Hmm. So, in some unspecified way, Unicode encodings and
60 translations between other character sets should be handled by this
67 -2- The stateT argument selects the pair of codesets being mapped between.
73 Ah ha! Another clue...
79 -3- The instantiations required in the Table ??
80 (lib.locale.category), namely codecvt<wchar_t,char,mbstate_t> and
81 codecvt<char,char,mbstate_t>, convert the implementation-defined
82 native character set. codecvt<char,char,mbstate_t> implements a
83 degenerate conversion; it does not convert at
84 all. codecvt<wchar_t,char,mbstate_t> converts between the native
85 character sets for tiny and wide characters. Instantiations on
86 mbstate_t perform conversion between encodings known to the library
87 implementor. Other encodings can be converted by specializing on a
88 user-defined stateT type. The stateT object can contain any state that
89 is useful to communicate to or from the specialized do_convert member.
95 At this point, a couple points become clear:
99 One: The standard clearly implies that attempts to add non-required
100 (yet useful and widely used) conversions need to do so through the
101 third template parameter, stateT.</para>
104 Two: The required conversions, by specifying mbstate_t as the third
105 template parameter, imply an implementation strategy that is mostly
106 (or wholly) based on the underlying C library, and the functions
107 mcsrtombs and wcsrtombs in particular.</para>
110 <section xml:id="facet.codecvt.design"><info><title>Design</title></info>
113 <section xml:id="codecvt.design.wchar_t_size"><info><title><type>wchar_t</type> Size</title></info>
117 The simple implementation detail of wchar_t's size seems to
118 repeatedly confound people. Many systems use a two byte,
119 unsigned integral type to represent wide characters, and use an
120 internal encoding of Unicode or UCS2. (See AIX, Microsoft NT,
121 Java, others.) Other systems, use a four byte, unsigned integral
122 type to represent wide characters, and use an internal encoding
123 of UCS4. (GNU/Linux systems using glibc, in particular.) The C
124 programming language (and thus C++) does not specify a specific
125 size for the type wchar_t.
129 Thus, portable C++ code cannot assume a byte size (or endianness) either.
133 <section xml:id="codecvt.design.unicode"><info><title>Support for Unicode</title></info>
136 Probably the most frequently asked question about code conversion
137 is: "So dudes, what's the deal with Unicode strings?"
138 The dude part is optional, but apparently the usefulness of
139 Unicode strings is pretty widely appreciated. Sadly, this specific
140 encoding (And other useful encodings like UTF8, UCS4, ISO 8859-10,
141 etc etc etc) are not mentioned in the C++ standard.
145 A couple of comments:
149 The thought that all one needs to convert between two arbitrary
150 codesets is two types and some kind of state argument is
151 unfortunate. In particular, encodings may be stateless. The naming
152 of the third parameter as stateT is unfortunate, as what is really
153 needed is some kind of generalized type that accounts for the
154 issues that abstract encodings will need. The minimum information
155 that is required includes:
161 Identifiers for each of the codesets involved in the
162 conversion. For example, using the iconv family of functions
163 from the Single Unix Specification (what used to be called
164 X/Open) hosted on the GNU/Linux operating system allows
165 bi-directional mapping between far more than the following
166 tantalizing possibilities:
170 (An edited list taken from <code>`iconv --list`</code> on a
171 Red Hat 6.2/Intel system:
176 8859_1, 8859_9, 10646-1:1993, 10646-1:1993/UCS4, ARABIC, ARABIC7,
177 ASCII, EUC-CN, EUC-JP, EUC-KR, EUC-TW, GREEK-CCIcode, GREEK, GREEK7-OLD,
178 GREEK7, GREEK8, HEBREW, ISO-8859-1, ISO-8859-2, ISO-8859-3,
179 ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8,
180 ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14,
181 ISO-8859-15, ISO-10646, ISO-10646/UCS2, ISO-10646/UCS4,
182 ISO-10646/UTF-8, ISO-10646/UTF8, SHIFT-JIS, SHIFT_JIS, UCS-2, UCS-4,
183 UCS2, UCS4, UNICODE, UNICODEBIG, UNICODELIcodeLE, US-ASCII, US, UTF-8,
184 UTF-16, UTF8, UTF16).
189 For iconv-based implementations, string literals for each of the
190 encodings (i.e. "UCS-2" and "UTF-8") are necessary,
192 non-iconv implementations a table of enumerated values or some other
193 mechanism may be required.
198 Maximum length of the identifying string literal.
202 Some encodings require explicit endian-ness. As such, some kind
203 of endian marker or other byte-order marker will be necessary. See
204 "Footnotes for C/C++ developers" in Haible for more information on
205 UCS-2/Unicode endian issues. (Summary: big endian seems most likely,
206 however implementations, most notably Microsoft, vary.)
210 Types representing the conversion state, for conversions involving
211 the machinery in the "C" library, or the conversion descriptor, for
212 conversions using iconv (such as the type iconv_t.) Note that the
213 conversion descriptor encodes more information than a simple encoding
218 Conversion descriptors for both directions of encoding. (i.e., both
219 UCS-2 to UTF-8 and UTF-8 to UCS-2.)
223 Something to indicate if the conversion requested if valid.
227 Something to represent if the conversion descriptors are valid.
231 Some way to enforce strict type checking on the internal and
232 external types. As part of this, the size of the internal and
233 external types will need to be known.
238 <section xml:id="codecvt.design.issues"><info><title>Other Issues</title></info>
241 In addition, multi-threaded and multi-locale environments also impact
242 the design and requirements for code conversions. In particular, they
243 affect the required specialization codecvt<wchar_t, char, mbstate_t>
244 when implemented using standard "C" functions.
248 Three problems arise, one big, one of medium importance, and one small.
252 First, the small: mcsrtombs and wcsrtombs may not be multithread-safe
253 on all systems required by the GNU tools. For GNU/Linux and glibc,
254 this is not an issue.
258 Of medium concern, in the grand scope of things, is that the functions
259 used to implement this specialization work on null-terminated
260 strings. Buffers, especially file buffers, may not be null-terminated,
261 thus giving conversions that end prematurely or are otherwise
266 The last, and fundamental problem, is the assumption of a global
267 locale for all the "C" functions referenced above. For something like
268 C++ iostreams (where codecvt is explicitly used) the notion of
269 multiple locales is fundamental. In practice, most users may not run
270 into this limitation. However, as a quality of implementation issue,
271 the GNU C++ library would like to offer a solution that allows
272 multiple locales and or simultaneous usage with computationally
273 correct results. In short, libstdc++ is trying to offer, as an
274 option, a high-quality implementation, damn the additional complexity!
278 For the required specialization codecvt<wchar_t, char, mbstate_t> ,
279 conversions are made between the internal character set (always UCS4
280 on GNU/Linux) and whatever the currently selected locale for the
281 LC_CTYPE category implements.
288 <section xml:id="facet.codecvt.impl"><info><title>Implementation</title></info>
292 The two required specializations are implemented as follows:
297 codecvt<char, char, mbstate_t>
301 This is a degenerate (i.e., does nothing) specialization. Implementing
302 this was a piece of cake.
307 codecvt<char, wchar_t, mbstate_t>
312 This specialization, by specifying all the template parameters, pretty
313 much ties the hands of implementors. As such, the implementation is
314 straightforward, involving mcsrtombs for the conversions between char
315 to wchar_t and wcsrtombs for conversions between wchar_t and char.
319 Neither of these two required specializations deals with Unicode
320 characters. As such, libstdc++ implements a partial specialization
321 of the codecvt class with and iconv wrapper class, encoding_state as the
322 third template parameter.
326 This implementation should be standards conformant. First of all, the
327 standard explicitly points out that instantiations on the third
328 template parameter, stateT, are the proper way to implement
329 non-required conversions. Second of all, the standard says (in Chapter
330 17) that partial specializations of required classes are a-ok. Third
331 of all, the requirements for the stateT type elsewhere in the standard
332 (see 21.1.2 traits typedefs) only indicate that this type be copy
337 As such, the type encoding_state is defined as a non-templatized, POD
338 type to be used as the third type of a codecvt instantiation. This
339 type is just a wrapper class for iconv, and provides an easy interface
340 to iconv functionality.
344 There are two constructors for encoding_state:
349 encoding_state() : __in_desc(0), __out_desc(0)
353 This default constructor sets the internal encoding to some default
354 (currently UCS4) and the external encoding to whatever is returned by
355 nl_langinfo(CODESET).
360 encoding_state(const char* __int, const char* __ext)
365 This constructor takes as parameters string literals that indicate the
366 desired internal and external encoding. There are no defaults for
371 One of the issues with iconv is that the string literals identifying
372 conversions are not standardized. Because of this, the thought of
373 mandating and or enforcing some set of pre-determined valid
374 identifiers seems iffy: thus, a more practical (and non-migraine
375 inducing) strategy was implemented: end-users can specify any string
376 (subject to a pre-determined length qualifier, currently 32 bytes) for
377 encodings. It is up to the user to make sure that these strings are
378 valid on the target system.
388 Strangely enough, this member function attempts to open conversion
389 descriptors for a given encoding_state object. If the conversion
390 descriptors are not valid, the conversion descriptors returned will
391 not be valid and the resulting calls to the codecvt conversion
392 functions will return error.
403 Provides a way to see if the given encoding_state object has been
404 properly initialized. If the string literals describing the desired
405 internal and external encoding are not valid, initialization will
406 fail, and this will return false. If the internal and external
407 encodings are valid, but iconv_open could not allocate conversion
408 descriptors, this will also return false. Otherwise, the object is
409 ready to convert and will return true.
414 encoding_state(const encoding_state&)
419 As iconv allocates memory and sets up conversion descriptors, the copy
420 constructor can only copy the member data pertaining to the internal
421 and external code conversions, and not the conversion descriptors
426 Definitions for all the required codecvt member functions are provided
427 for this specialization, and usage of codecvt<internal character type,
428 external character type, encoding_state> is consistent with other
434 <section xml:id="facet.codecvt.use"><info><title>Use</title></info>
436 <para>A conversions involving string literal.</para>
439 typedef codecvt_base::result result;
440 typedef unsigned short unicode_t;
441 typedef unicode_t int_type;
442 typedef char ext_type;
443 typedef encoding_state state_type;
444 typedef codecvt<int_type, ext_type, state_type> unicode_codecvt;
446 const ext_type* e_lit = "black pearl jasmine tea";
447 int size = strlen(e_lit);
448 int_type i_lit_base[24] =
449 { 25088, 27648, 24832, 25344, 27392, 8192, 28672, 25856, 24832, 29184,
450 27648, 8192, 27136, 24832, 29440, 27904, 26880, 28160, 25856, 8192, 29696,
453 const int_type* i_lit = i_lit_base;
454 const ext_type* efrom_next;
455 const int_type* ifrom_next;
456 ext_type* e_arr = new ext_type[size + 1];
458 int_type* i_arr = new int_type[size + 1];
461 // construct a locale object with the specialized facet.
462 locale loc(locale::classic(), new unicode_codecvt);
463 // sanity check the constructed locale has the specialized facet.
464 VERIFY( has_facet<unicode_codecvt>(loc) );
465 const unicode_codecvt& cvt = use_facet<unicode_codecvt>(loc);
466 // convert between const char* and unicode strings
467 unicode_codecvt::state_type state01("UNICODE", "ISO_8859-1");
468 initialize_state(state01);
469 result r1 = cvt.in(state01, e_lit, e_lit + size, efrom_next,
470 i_arr, i_arr + size, ito_next);
471 VERIFY( r1 == codecvt_base::ok );
472 VERIFY( !int_traits::compare(i_arr, i_lit, size) );
473 VERIFY( efrom_next == e_lit + size );
474 VERIFY( ito_next == i_arr + size );
479 <section xml:id="facet.codecvt.future"><info><title>Future</title></info>
484 a. things that are sketchy, or remain unimplemented:
485 do_encoding, max_length and length member functions
486 are only weakly implemented. I have no idea how to do
487 this correctly, and in a generic manner. Nathan?
493 b. conversions involving std::string
497 how should operators != and == work for string of
498 different/same encoding?
502 what is equal? A byte by byte comparison or an
503 encoding then byte comparison?
507 conversions between narrow, wide, and unicode strings
512 c. conversions involving std::filebuf and std::ostream
516 how to initialize the state object in a
517 standards-conformant manner?
521 how to synchronize the "C" and "C++"
522 conversion information?
526 wchar_t/char internal buffers and conversions between
527 internal/external buffers?
535 <bibliography xml:id="facet.codecvt.biblio"><info><title>Bibliography</title></info>
542 <author><personname><surname>McGrath</surname><firstname>Roland</firstname></personname></author>
543 <author><personname><surname>Drepper</surname><firstname>Ulrich</firstname></personname></author>
549 Chapters 6 Character Set Handling and 7 Locales and Internationalization
557 <author><personname><surname>Drepper</surname><firstname>Ulrich</firstname></personname></author>
566 ISO/IEC 14882:1998 Programming languages - C++
576 ISO/IEC 9899:1999 Programming languages - C
586 <link xmlns:xlink="http://www.w3.org/1999/xlink"
587 xlink:href="http://www.opengroup.org/austin/">
588 System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008)
595 The Open Group/The Institute of Electrical and Electronics
603 The C++ Programming Language, Special Edition
605 <author><personname><surname>Stroustrup</surname><firstname>Bjarne</firstname></personname></author>
608 <holder>Addison Wesley, Inc.</holder>
610 <pagenums>Appendix D</pagenums>
621 Standard C++ IOStreams and Locales
624 Advanced Programmer's Guide and Reference
626 <author><personname><surname>Langer</surname><firstname>Angelika</firstname></personname></author>
627 <author><personname><surname>Kreft</surname><firstname>Klaus</firstname></personname></author>
630 <holder>Addison Wesley Longman, Inc.</holder>
634 Addison Wesley Longman
641 <link xmlns:xlink="http://www.w3.org/1999/xlink"
642 xlink:href="http://www.lysator.liu.se/c/na1.html">
643 A brief description of Normative Addendum 1
647 <author><personname><surname>Feather</surname><firstname>Clive</firstname></personname></author>
648 <pagenums>Extended Character Sets</pagenums>
653 <link xmlns:xlink="http://www.w3.org/1999/xlink"
654 xlink:href="http://tldp.org/HOWTO/Unicode-HOWTO.html">
659 <author><personname><surname>Haible</surname><firstname>Bruno</firstname></personname></author>
664 <link xmlns:xlink="http://www.w3.org/1999/xlink"
665 xlink:href="http://www.cl.cam.ac.uk/~mgk25/unicode.html">
666 UTF-8 and Unicode FAQ for Unix/Linux
671 <author><personname><surname>Khun</surname><firstname>Markus</firstname></personname></author>