1 @node Character Set Handling, Locales, String and Array Utilities, Top
2 @c %MENU% Support for extended character sets
3 @chapter Character Set Handling
11 Character sets used in the early days of computing had only six, seven,
12 or eight bits for each character: there was never a case where more than
13 eight bits (one byte) were used to represent a single character. The
14 limitations of this approach became more apparent as more people
15 grappled with non-Roman character sets, where not all the characters
16 that make up a language's character set can be represented by @math{2^8}
17 choices. This chapter shows the functionality which was added to the C
18 library to correctly support multiple character sets.
21 * Extended Char Intro:: Introduction to Extended Characters.
22 * Charset Function Overview:: Overview about Character Handling
24 * Restartable multibyte conversion:: Restartable multibyte conversion
26 * Non-reentrant Conversion:: Non-reentrant Conversion Function.
27 * Generic Charset Conversion:: Generic Charset Conversion.
31 @node Extended Char Intro
32 @section Introduction to Extended Characters
34 A variety of solutions to overcome the differences between
35 character sets with a 1:1 relation between bytes and characters and
36 character sets with ratios of 2:1 or 4:1 exist. The remainder of this
37 section gives a few examples to help understand the design decisions
38 made while developing the functionality of the @w{C library}.
40 @cindex internal representation
41 A distinction we have to make right away is between internal and
42 external representation. @dfn{Internal representation} means the
43 representation used by a program while keeping the text in memory.
44 External representations are used when text is stored or transmitted
45 through whatever communication channel. Examples of external
46 representations include files lying in a directory that are going to be
49 Traditionally there was no difference between the two representations.
50 It was equally comfortable and useful to use the same one-byte
51 representation internally and externally. This changes with more and
52 larger character sets.
54 One of the problems to overcome with the internal representation is
55 handling text which is externally encoded using different character
56 sets. Assume a program which reads two texts and compares them using
57 some metric. The comparison can be usefully done only if the texts are
58 internally kept in a common format.
60 @cindex wide character
61 For such a common format (@math{=} character set) eight bits are certainly
62 no longer enough. So the smallest entity will have to grow: @dfn{wide
63 characters} will now be used. Instead of one byte, two or four will
64 be used instead. (Three are not good to address in memory and more
65 than four bytes seem not to be necessary).
69 As shown in some other part of this manual,
70 @c !!! Ahem, wide char string functions are not yet covered -- drepper
71 there exists a completely new family of functions which can handle texts
72 of this kind in memory. The most commonly used character set for such
73 internal wide character representations are Unicode and @w{ISO 10646}.
74 The former is a subset of the later and used when wide characters are
75 chosen to by 2 bytes (@math{= 16} bits) wide. The standard names of the
78 encodings used in these cases are UCS2 (@math{= 16} bits) and UCS4
81 To represent wide characters the @code{char} type is not suitable. For
82 this reason the @w{ISO C} standard introduces a new type which is
83 designed to keep one character of a wide character string. To maintain
84 the similarity there is also a type corresponding to @code{int} for
85 those functions which take a single wide character.
89 @deftp {Data type} wchar_t
90 This data type is used as the base type for wide character strings.
91 I.e., arrays of objects of this type are the equivalent of @code{char[]}
92 for multibyte character strings. The type is defined in @file{stddef.h}.
94 The @w{ISO C89} standard, where this type was introduced, does not say
95 anything specific about the representation. It only requires that this
96 type is capable to store all elements of the basic character set.
97 Therefore it would be legitimate to define @code{wchar_t} and
98 @code{char}. This might make sense for embedded systems.
100 But for GNU systems this type is always 32 bits wide. It is therefore
101 capable to represent all UCS4 value therefore covering all of @w{ISO
102 10646}. Some Unix systems define @code{wchar_t} as a 16 bit type and
103 thereby follow Unicode very strictly. This is perfectly fine with the
104 standard but it also means that to represent all characters from Unicode
105 and @w{ISO 10646} one has to use surrogate character which is in fact a
106 multi-wide-character encoding. But this contradicts the purpose of the
112 @deftp {Data type} wint_t
113 @code{wint_t} is a data type used for parameters and variables which
114 contain a single wide character. As the name already suggests it is the
115 equivalent to @code{int} when using the normal @code{char} strings. The
116 types @code{wchar_t} and @code{wint_t} have often the same
117 representation if their size if 32 bits wide but if @code{wchar_t} is
118 defined as @code{char} the type @code{wint_t} must be defined as
119 @code{int} due to the parameter promotion.
122 This type is defined in @file{wchar.h} and got introduced in the second
123 amendment to @w{ISO C 89}.
126 As there are for the @code{char} data type there also exist macros
127 specifying the minimum and maximum value representable in an object of
132 @deftypevr Macro wint_t WCHAR_MIN
133 The macro @code{WCHAR_MIN} evaluates to the minimum value representable
134 by an object of type @code{wint_t}.
136 This macro got introduced in the second amendment to @w{ISO C89}.
141 @deftypevr Macro wint_t WCHAR_MAX
142 The macro @code{WCHAR_MIN} evaluates to the maximum value representable
143 by an object of type @code{wint_t}.
145 This macro got introduced in the second amendment to @w{ISO C89}.
148 Another special wide character value is the equivalent to @code{EOF}.
152 @deftypevr Macro wint_t WEOF
153 The macro @code{WEOF} evaluates to a constant expression of type
154 @code{wint_t} whose value is different from any member of the extended
157 @code{WEOF} need not be the same value as @code{EOF} and unlike
158 @code{EOF} it also need @emph{not} be negative. I.e., sloppy code like
164 while ((c = getc (fp)) < 0)
170 has to be rewritten to explicitly use @code{WEOF} when wide characters
177 while ((c = wgetc (fp)) != WEOF)
183 This macro was introduced in the second amendment to @w{ISO C89} and is
184 defined in @file{wchar.h}.
188 These internal representations present problems when it comes to storing
189 and transmittal, since a single wide character consists of more
190 than one byte they are effected by byte-ordering. I.e., machines with
191 different endianesses would see different value accessing the same data.
192 This also applies for communication protocols which are all byte-based
193 and therefore the sender has to decide about splitting the wide
194 character in bytes. A last (but not least important) point is that wide
195 characters often require more storage space than an customized byte
196 oriented character set.
198 @cindex multibyte character
200 For all the above reasons, an external encoding which is different
201 from the internal encoding is often used if the later is UCS2 or UCS4.
202 The external encoding is byte-based and can be chosen appropriately for
203 the environment and for the texts to be handled. There exist a variety
204 of different character sets which can be used for this external
205 encoding. Information which will not be exhaustively presented
206 here--instead, a description of the major groups will suffice. All of
207 the ASCII-based character sets [_bkoz_: do you mean Roman character
208 sets? If not, what do you mean here?] fulfill one requirement: they are
209 "filesystem safe". This means that the character @code{'/'} is used in
210 the encoding @emph{only} to represent itself. Things are a bit
211 different for character sets like EBCDIC (Extended Binary Coded Decimal
212 Interchange Code, a character set family used by IBM) but if the
213 operation system does not understand EBCDIC directly the parameters to
214 system calls have to be converted first anyhow.
218 The simplest character sets are one-byte character sets. There can be
219 only up to 256 characters (for @w{8 bit} character sets) which is not
220 sufficient to cover all languages but might be sufficient to handle a
221 specific text. Another reason to choose this is because of constraints
222 from interaction with other programs (which might not be 8-bit clean).
226 The @w{ISO 2022} standard defines a mechanism for extended character
227 sets where one character @emph{can} be represented by more than one
228 byte. This is achieved by associating a state with the text. Embedded
229 in the text can be characters which can be used to change the state.
230 Each byte in the text might have a different interpretation in each
231 state. The state might even influence whether a given byte stands for a
232 character on its own or whether it has to be combined with some more
237 In most uses of @w{ISO 2022} the defined character sets do not allow
238 state changes which cover more than the next character. This has the
239 big advantage that whenever one can identify the beginning of the byte
240 sequence of a character one can interpret a text correctly. Examples of
241 character sets using this policy are the various EUC character sets
242 (used by Sun's operations systems, EUC-JP, EUC-KR, EUC-TW, and EUC-CN)
243 or SJIS (Shift JIS, a Japanese encoding).
245 But there are also character sets using a state which is valid for more
246 than one character and has to be changed by another byte sequence.
247 Examples for this are ISO-2022-JP, ISO-2022-KR, and ISO-2022-CN.
251 Early attempts to fix 8 bit character sets for other languages using the
252 Roman alphabet lead to character sets like @w{ISO 6937}. Here bytes
253 representing characters like the acute accent do not produce output
254 themselves: one has to combine them with other characters to get the
255 desired result. E.g., the byte sequence @code{0xc2 0x61} (non-spacing
256 acute accent, following by lower-case `a') to get the ``small a with
257 acute'' character. To get the acute accent character on its on one has
258 to write @code{0xc2 0x20} (the non-spacing acute followed by a space).
260 This type of characters sets is quite frequently used in embedded
261 systems such as video text.
265 Instead of converting the Unicode or @w{ISO 10646} text used internally
266 it is often also sufficient to simply use an encoding different then
267 UCS2/UCS4. The Unicode and @w{ISO 10646} standards even specify such an
268 encoding: UTF-8. This encoding is able to represent all of @w{ISO
269 10464} 31 bits in a byte string of length one to seven.
272 There were a few other attempts to encode @w{ISO 10646} such as UTF-7
273 but UTF-8 is today the only encoding which should be used. In fact,
274 UTF-8 will hopefully soon be the only external which has to be
275 supported. It proves to be universally usable and the only disadvantage
276 is that it favor Roman languages very much by making the byte string
277 representation of other scripts (Cyrillic, Greek, Asian scripts) longer
278 than necessary if using a specific character set for these scripts.
279 Methods like the Unicode compression scheme can alleviate these
283 The question remaining is: how to select the character set or encoding
284 to use. The answer: you cannot decide about it yourself, it is decided
285 by the developers of the system or the majority of the users. Since the
286 goal is interoperability one has to use whatever the other people one
287 works with use. If there are no constraints the selection is based on
288 the requirements the expected circle of users will have. I.e., if a
289 project is expected to only be used in, say, Russia it is fine to use
290 KOI8-R or a similar character set. But if at the same time people from,
291 say, Greek are participating one should use a character set which allows
292 all people to collaborate.
294 The most widely useful solution seems to be: go with the most general
295 character set, namely @w{ISO 10646}. Use UTF-8 as the external encoding
296 and problems about users not being able to use their own language
297 adequately are a thing of the past.
299 One final comment about the choice of the wide character representation
300 is necessary at this point. We have said above that the natural choice
301 is using Unicode or @w{ISO 10646}. This is not specified in any
302 standard, though. The @w{ISO C} standard does not specify anything
303 specific about the @code{wchar_t} type. There might be systems where
304 the developers decided differently. Therefore one should as much as
305 possible avoid making assumption about the wide character representation
306 although GNU systems will always work as described above. If the
307 programmer uses only the functions provided by the C library to handle
308 wide character strings there should not be any compatibility problems
311 @node Charset Function Overview
312 @section Overview about Character Handling Functions
314 A Unix @w{C library} contains three different sets of functions in two
315 families to handle character set conversion. The one function family
316 is specified in the @w{ISO C} standard and therefore is portable even
317 beyond the Unix world.
319 The most commonly known set of functions, coming from the @w{ISO C89}
320 standard, is unfortunately the least useful one. In fact, these
321 functions should be avoided whenever possible, especially when
322 developing libraries (as opposed to applications).
324 The second family of functions got introduced in the early Unix standards
325 (XPG2) and is still part of the latest and greatest Unix standard:
326 @w{Unix 98}. It is also the most powerful and useful set of functions.
327 But we will start with the functions defined in the second amendment to
330 @node Restartable multibyte conversion
331 @section Restartable Multibyte Conversion Functions
333 The @w{ISO C} standard defines functions to convert strings from a
334 multibyte representation to wide character strings. There are a number
339 The character set assumed for the multibyte encoding is not specified
340 as an argument to the functions. Instead the character set specified by
341 the @code{LC_CTYPE} category of the current locale is used; see
342 @ref{Locale Categories}.
345 The functions handling more than one character at a time require NUL
346 terminated strings as the argument. I.e., converting blocks of text
347 does not work unless one can add a NUL byte at an appropriate place.
348 The GNU C library contains some extensions the standard which allow
349 specifying a size but basically they also expect terminated strings.
352 Despite these limitations the @w{ISO C} functions can very well be used
353 in many contexts. In graphical user interfaces, for instance, it is not
354 uncommon to have functions which require text to be displayed in a wide
355 character string if it is not simple ASCII. The text itself might come
356 from a file with translations and the user should decide about the
357 current locale which determines the translation and therefore also the
358 external encoding used. In such a situation (and many others) the
359 functions described here are perfect. If more freedom while performing
360 the conversion is necessary take a look at the @code{iconv} functions
361 (@pxref{Generic Charset Conversion})
364 * Selecting the Conversion:: Selecting the conversion and its properties.
365 * Keeping the state:: Representing the state of the conversion.
366 * Converting a Character:: Converting Single Characters.
367 * Converting Strings:: Converting Multibyte and Wide Character
369 * Multibyte Conversion Example:: A Complete Multibyte Conversion Example.
372 @node Selecting the Conversion
373 @subsection Selecting the conversion and its properties
375 We already said above that the currently selected locale for the
376 @code{LC_CTYPE} category decides about the conversion which is performed
377 by the functions we are about to describe. Each locale uses its own
378 character set (given as an argument to @code{localedef}) and this is the
379 one assumed as the external multibyte encoding. The wide character
380 character set always is UCS4, at least on GNU systems.
382 A characteristic of each multibyte character set is the maximum number
383 of bytes which can be necessary to represent one character. This
384 information is quite important when writing code which uses the
385 conversion functions. In the examples below we will see some examples.
386 The @w{ISO C} standard defines two macros which provide this information.
391 @deftypevr Macro int MB_LEN_MAX
392 This macro specifies the maximum number of bytes in the multibyte
393 sequence for a single character in any of the supported locales. It is
394 a compile-time constant and it is defined in @file{limits.h}.
400 @deftypevr Macro int MB_CUR_MAX
401 @code{MB_CUR_MAX} expands into a positive integer expression that is the
402 maximum number of bytes in a multibyte character in the current locale.
403 The value is never greater than @code{MB_LEN_MAX}. Unlike
404 @code{MB_LEN_MAX} this macro need not be a compile-time constant and in
405 fact, in the GNU C library it is not.
408 @code{MB_CUR_MAX} is defined in @file{stdlib.h}.
411 Two different macros are necessary since strictly @w{ISO C89} compilers
412 do not allow variable length array definitions but still it is desirable
413 to avoid dynamic allocation. This incomplete piece of code shows the
418 char buf[MB_LEN_MAX];
423 fread (&buf[len], 1, MB_CUR_MAX - len, fp);
424 /* @r{... process} buf */
430 The code in the inner loop is expected to have always enough bytes in
431 the array @var{buf} to convert one multibyte character. The array
432 @var{buf} has to be sized statically since many compilers do not allow a
433 variable size. The @code{fread} call makes sure that always
434 @code{MB_CUR_MAX} bytes are available in @var{buf}. Note that it isn't
435 a problem if @code{MB_CUR_MAX} is not a compile-time constant.
438 @node Keeping the state
439 @subsection Representing the state of the conversion
442 In the introduction of this chapter it was said that certain character
443 sets use a @dfn{stateful} encoding. I.e., the encoded values depend in
444 some way on the previous bytes in the text.
446 Since the conversion functions allow converting a text in more than one
447 step we must have a way to pass this information from one call of the
448 functions to another.
452 @deftp {Data type} mbstate_t
454 A variable of type @code{mbstate_t} can contain all the information
455 about the @dfn{shift state} needed from one call to a conversion
459 This type is defined in @file{wchar.h}. It got introduced in the second
460 amendment to @w{ISO C89}.
463 To use objects of this type the programmer has to define such objects
464 (normally as local variables on the stack) and pass a pointer to the
465 object to the conversion functions. This way the conversion function
466 can update the object if the current multibyte character set is
469 There is no specific function or initializer to put the state object in
470 any specific state. The rules are that the object should always
471 represent the initial state before the first use and this is achieved by
472 clearing the whole variable with code such as follows:
477 memset (&state, '\0', sizeof (state));
478 /* @r{from now on @var{state} can be used.} */
483 When using the conversion functions to generate output it is often
484 necessary to test whether the current state corresponds to the initial
485 state. This is necessary, for example, to decide whether or not to emit
486 escape sequences to set the state to the initial state at certain
487 sequence points. Communication protocols often require this.
491 @deftypefun int mbsinit (const mbstate_t *@var{ps})
492 This function determines whether the state object pointed to by @var{ps}
493 is in the initial state or not. If @var{ps} is a null pointer or the
494 object is in the initial state the return value is nonzero. Otherwise
498 This function was introduced in the second amendment to @w{ISO C89} and
499 is declared in @file{wchar.h}.
502 Code using this function often looks similar to this:
507 memset (&state, '\0', sizeof (state));
508 /* @r{Use @var{state}.} */
510 if (! mbsinit (&state))
512 /* @r{Emit code to return to initial state.} */
513 fputs ("@r{whatever needed}", fp);
519 @node Converting a Character
520 @subsection Converting Single Characters
522 The most fundamental of the conversion functions are those dealing with
523 single characters. Please note that this does not always mean single
524 bytes. But since there is very often a subset of the multibyte
525 character set which consists of single byte sequences there are
526 functions to help with converting bytes. One very important and often
527 applicable scenario is where ASCII is a subpart of the multibyte
528 character set. I.e., all ASCII characters stand for itself and all
529 other characters have at least a first byte which is beyond the range
530 @math{0} to @math{127}.
534 @deftypefun wint_t btowc (int @var{c})
535 The @code{btowc} function (``byte to wide character'') converts a valid
536 single byte character @var{c} in the initial shift state into the wide
537 character equivalent using the conversion rules from the currently
538 selected locale of the @code{LC_CTYPE} category.
540 If @code{(unsigned char) @var{c}} is no valid single byte multibyte
541 character or if @var{c} is @code{EOF} the function returns @code{WEOF}.
543 Please note the restriction of @var{c} being tested for validity only in
544 the initial shift state. There is no @code{mbstate_t} object used from
545 which the state information is taken and the function also does not use
549 This function was introduced in the second amendment of @w{ISO C89} and
550 is declared in @file{wchar.h}.
553 Despite the limitation that the single byte value always is interpreted
554 in the initial state this function is actually useful most of the time.
555 Most characters are either entirely single-byte character sets or they
556 are extension to ASCII. But then it is possible to write code like this
557 (not that this specific example is very useful):
561 itow (unsigned long int val)
563 static wchar_t buf[30];
564 wchar_t *wcp = &buf[29];
568 *--wcp = btowc ('0' + val % 10);
577 Why is it necessary to use such a complicated implementation and not
578 simply cast @code{'0' + val % 10} to a wide character? The answer is
579 that there is no guarantee that one can perform this kind of arithmetic
580 on the character of the character set used for @code{wchar_t}
581 representation. In other situations the bytes are not constant at
582 compile time and so the compiler cannot do the work. In situations like
583 this it is necessary @code{btowc}.
586 There also is a function for the conversion in the other direction.
590 @deftypefun int wctob (wint_t @var{c})
591 The @code{wctob} function (``wide character to byte'') takes as the
592 parameter a valid wide character. If the multibyte representation for
593 this character in the initial state is exactly one byte long the return
594 value of this function is this character. Otherwise the return value is
598 This function was introduced in the second amendment of @w{ISO C89} and
599 is declared in @file{wchar.h}.
602 There are more general functions to convert single character from
603 multibyte representation to wide characters and vice versa. These
604 functions pose no limit on the length of the multibyte representation
605 and they also do not require it to be in the initial state.
609 @deftypefun size_t mbrtowc (wchar_t *restrict @var{pwc}, const char *restrict @var{s}, size_t @var{n}, mbstate_t *restrict @var{ps})
611 The @code{mbrtowc} function (``multibyte restartable to wide
612 character'') converts the next multibyte character in the string pointed
613 to by @var{s} into a wide character and stores it in the wide character
614 string pointed to by @var{pwc}. The conversion is performed according
615 to the locale currently selected for the @code{LC_CTYPE} category. If
616 the conversion for the character set used in the locale requires a state
617 the multibyte string is interpreted in the state represented by the
618 object pointed to by @var{ps}. If @var{ps} is a null pointer an static,
619 internal state variable used only by the @code{mbrtowc} variable is
622 If the next multibyte character corresponds to the NUL wide character
623 the return value of the function is @math{0} and the state object is
624 afterwards in the initial state. If the next @var{n} or fewer bytes
625 form a correct multibyte character the return value is the number of
626 bytes starting from @var{s} which form the multibyte character. The
627 conversion state is updated according to the bytes consumed in the
628 conversion. In both cases the wide character (either the @code{L'\0'}
629 or the one found in the conversion) is stored in the string pointer to
630 by @var{pwc} iff @var{pwc} is not null.
632 If the first @var{n} bytes of the multibyte string possibly form a valid
633 multibyte character but there are more than @var{n} bytes needed to
634 complete it the return value of the function is @code{(size_t) -2} and
635 no value is stored. Please note that this can happen even if @var{n}
636 has a value greater or equal to @code{MB_CUR_MAX} since the input might
637 contain redundant shift sequences.
639 If the first @code{n} bytes of the multibyte string cannot possibly form
640 a valid multibyte character also no value is stored, the global variable
641 @code{errno} is set to the value @code{EILSEQ} and the function returns
642 @code{(size_t) -1}. The conversion state is afterwards undefined.
645 This function was introduced in the second amendment to @w{ISO C89} and
646 is declared in @file{wchar.h}.
649 Using this function is straight forward. A function which copies a
650 multibyte string into a wide character string while at the same time
651 converting all lowercase character into uppercase could look like this
652 (this is not the final version, just an example; it has no error
653 checking, and leaks sometimes memory):
657 mbstouwcs (const char *s)
659 size_t len = strlen (s);
660 wchar_t *result = malloc ((len + 1) * sizeof (wchar_t));
661 wchar_t *wcp = result;
664 memset (&state, '\0', sizeof (state));
666 while ((nbytes = mbrtowc (tmp, s, len, &state)) > 0)
668 if (nbytes >= (size_t) -2)
669 /* Invalid input string. */
671 *result++ = towupper (tmp[0]);
679 The use of @code{mbrtowc} should be clear. A single wide character is
680 stored in @code{@var{tmp}[0]} and the number of consumed bytes is stored
681 in the variable @var{nbytes}. In case the the conversion was successful
682 the uppercase variant of the wide character is stored in the
683 @var{result} array and the pointer to the input string and the number of
684 available bytes is adjusted.
686 The only non-obvious thing about the function might be the way memory is
687 allocated for the result. The above code uses the fact that there can
688 never be more wide characters in the converted results than there are
689 bytes in the multibyte input string. This method yields to a
690 pessimistic guess about the size of the result and if many wide
691 character strings have to be constructed this way or the strings are
692 long, the extra memory required allocated because the input string
693 contains multibzte characters might be significant. It would be
694 possible to resize the allocated memory block to the correct size before
695 returning it. A better solution might be to allocate just the right
696 amount of space for the result right away. Unfortunately there is no
697 function to compute the length of the wide character string directly
698 from the multibyte string. But there is a function which does part of
703 @deftypefun size_t mbrlen (const char *restrict @var{s}, size_t @var{n}, mbstate_t *@var{ps})
704 The @code{mbrlen} function (``multibyte restartable length'') computes
705 the number of at most @var{n} bytes starting at @var{s} which form the
706 next valid and complete multibyte character.
708 If the next multibyte character corresponds to the NUL wide character
709 the return value is @math{0}. If the next @var{n} bytes form a valid
710 multibyte character the number of bytes belonging to this multibyte
711 character byte sequence is returned.
713 If the the first @var{n} bytes possibly form a valid multibyte
714 character but it is incomplete the return value is @code{(size_t) -2}.
715 Otherwise the multibyte character sequence is invalid and the return
716 value is @code{(size_t) -1}.
718 The multibyte sequence is interpreted in the state represented by the
719 object pointer to by @var{ps}. If @var{ps} is a null pointer an state
720 object local to @code{mbrlen} is used.
723 This function was introduced in the second amendment to @w{ISO C89} and
724 is declared in @file{wchar.h}.
727 The tentative reader now will of course note that @code{mbrlen} can be
731 mbrtowc (NULL, s, n, ps != NULL ? ps : &internal)
734 This is true and in fact is mentioned in the official specification.
735 Now, how can this function be used to determine the length of the wide
736 character string created from a multibyte character string? It is not
737 directly usable but we can define a function @code{mbslen} using it:
741 mbslen (const char *s)
746 memset (&state, '\0', sizeof (state));
747 while ((nbytes = mbrlen (s, MB_LEN_MAX, &state)) > 0)
749 if (nbytes >= (size_t) -2)
750 /* @r{Something is wrong.} */
759 This function simply calls @code{mbrlen} for each multibyte character
760 in the string and counts the number of function calls. Please note that
761 we here use @code{MB_LEN_MAX} as the size argument in the @code{mbrlen}
762 call. This is OK since a) this value is larger then the length of the
763 longest multibyte character sequence and b) because we know that the
764 string @var{s} ends with a NUL byte which cannot be part of any other
765 multibyte character sequence but the one representing the NUL wide
766 character. Therefore the @code{mbrlen} function will never read invalid
769 Now that this function is available (just to make this clear, this
770 function is @emph{not} part of the GNU C library) we can compute the
771 number of wide character required to store the converted multibyte
772 character string @var{s} using
775 wcs_bytes = (mbslen (s) + 1) * sizeof (wchar_t);
778 Please note that the @code{mbslen} function is quite inefficient. The
779 implementation of @code{mbstouwcs} implemented using @code{mbslen} would
780 have to perform the conversion of the multibyte character input string
781 twice and this conversion might be quite expensive. So it is necessary
782 to think about the consequences of using the easier but imprecise method
783 before doing the work twice.
787 @deftypefun size_t wcrtomb (char *restrict @var{s}, wchar_t @var{wc}, mbstate_t *restrict @var{ps})
788 The @code{wcrtomb} function (``wide character restartable to
789 multibyte'') converts a single wide character into a multibyte string
790 corresponding to that wide character.
792 If @var{s} is a null pointer the function resets the the state stored in
793 the objects pointer to by @var{ps} (or the internal @code{mbstate_t}
794 object) to the initial state. This can also be achieved by a call like
798 wcrtombs (temp_buf, L'\0', ps)
802 since if @var{s} is a null pointer @code{wcrtomb} performs as if it
803 writes into an internal buffer which is guaranteed to be large enough.
805 If @var{wc} is the NUL wide character @code{wcrtomb} emits, if
806 necessary, a shift sequence to get the state @var{ps} into the initial
807 state followed by a single NUL byte is stored in the string @var{s}.
809 Otherwise a byte sequence (possibly including shift sequences) is
810 written into the string @var{s}. This of only happens if @var{wc} is a
811 valid wide character, i.e., it has a multibyte representation in the
812 character set selected by locale of the @code{LC_CTYPE} category. If
813 @var{wc} is no valid wide character nothing is stored in the strings
814 @var{s}, @code{errno} is set to @code{EILSEQ}, the conversion state in
815 @var{ps} is undefined and the return value is @code{(size_t) -1}.
817 If no error occurred the function returns the number of bytes stored in
818 the string @var{s}. This includes all byte representing shift
821 One word about the interface of the function: there is no parameter
822 specifying the length of the array @var{s}. Instead the function
823 assumes that there are at least @code{MB_CUR_MAX} bytes available since
824 this is the maximum length of any byte sequence representing a single
825 character. So the caller has to make sure that there is enough space
826 available, otherwise buffer overruns can occur.
829 This function was introduced in the second amendment to @w{ISO C} and is
830 declared in @file{wchar.h}.
833 Using this function is as easy as using @code{mbrtowc}. The following
834 example appends a wide character string to a multibyte character string.
835 Again, the code is not really useful (and correct), it is simply here to
836 demonstrate the use and some problems.
840 mbscatwc (char *s, size_t len, const wchar_t *ws)
843 /* @r{Find the end of the existing string.} */
844 char *wp = strchr (s, '\0');
846 memset (&state, '\0', sizeof (state));
850 if (len < MB_CUR_LEN)
852 /* @r{We cannot guarantee that the next}
853 @r{character fits into the buffer, so}
854 @r{return an error.} */
858 nbytes = wcrtomb (wp, *ws, &state);
859 if (nbytes == (size_t) -1)
860 /* @r{Error in the conversion.} */
865 while (*ws++ != L'\0');
870 First the function has to find the end of the string currently in the
871 array @var{s}. The @code{strchr} call does this very efficiently since a
872 requirement for multibyte character representations is that the NUL byte
873 never is used except to represent itself (and in this context, the end
876 After initializing the state object the loop is entered where the first
877 task is to make sure there is enough room in the array @var{s}. We
878 abort if there are not at least @code{MB_CUR_LEN} bytes available. This
879 is not always optimal but we have no other choice. We might have less
880 than @code{MB_CUR_LEN} bytes available but the next multibyte character
881 might also be only one byte long. At the time the @code{wcrtomb} call
882 returns it is too late to decide whether the buffer was large enough or
883 not. If this solution is really unsuitable there is a very slow but
884 more accurate solution.
888 if (len < MB_CUR_LEN)
890 mbstate_t temp_state;
891 memcpy (&temp_state, &state, sizeof (state));
892 if (wcrtomb (NULL, *ws, &temp_state) > len)
894 /* @r{We cannot guarantee that the next}
895 @r{character fits into the buffer, so}
896 @r{return an error.} */
904 Here we do perform the conversion which might overflow the buffer so
905 that we are afterwards in the position to make an exact decision about
906 the buffer size. Please note the @code{NULL} argument for the
907 destination buffer in the new @code{wcrtomb} call; since we are not
908 interested in the converted text at this point this is a nice way to
909 express this. The most unusual thing about this piece of code certainly
910 is the duplication of the conversion state object. But think about
911 this: if a change of the state is necessary to emit the next multibyte
912 character we want to have the same shift state change performed in the
913 real conversion. Therefore we have to preserve the initial shift state
916 There are certainly many more and even better solutions to this problem.
917 This example is only meant for educational purposes.
919 @node Converting Strings
920 @subsection Converting Multibyte and Wide Character Strings
922 The functions described in the previous section only convert a single
923 character at a time. Most operations to be performed in real-world
924 programs include strings and therefore the @w{ISO C} standard also
925 defines conversions on entire strings. However, the defined set of
926 functions is quite limited, thus the GNU C library contains a few
927 extensions which can help in some important situations.
931 @deftypefun size_t mbsrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps})
932 The @code{mbsrtowcs} function (``multibyte string restartable to wide
933 character string'') converts an NUL terminated multibyte character
934 string at @code{*@var{src}} into an equivalent wide character string,
935 including the NUL wide character at the end. The conversion is started
936 using the state information from the object pointed to by @var{ps} or
937 from an internal object of @code{mbsrtowcs} if @var{ps} is a null
938 pointer. Before returning the state object to match the state after the
939 last converted character. The state is the initial state if the
940 terminating NUL byte is reached and converted.
942 If @var{dst} is not a null pointer the result is stored in the array
943 pointed to by @var{dst}, otherwise the conversion result is not
944 available since it is stored in an internal buffer.
946 If @var{len} wide characters are stored in the array @var{dst} before
947 reaching the end of the input string the conversion stops and @var{len}
948 is returned. If @var{dst} is a null pointer @var{len} is never checked.
950 Another reason for a premature return from the function call is if the
951 input string contains an invalid multibyte sequence. In this case the
952 global variable @code{errno} is set to @code{EILSEQ} and the function
953 returns @code{(size_t) -1}.
955 @c XXX The ISO C9x draft seems to have a problem here. It says that PS
956 @c is not updated if DST is NULL. This is not said straight forward and
957 @c none of the other functions is described like this. It would make sense
958 @c to define the function this way but I don't think it is meant like this.
960 In all other cases the function returns the number of wide characters
961 converted during this call. If @var{dst} is not null @code{mbsrtowcs}
962 stores in the pointer pointed to by @var{src} a null pointer (if the NUL
963 byte in the input string was reached) or the address of the byte
964 following the last converted multibyte character.
967 This function was introduced in the second amendment to @w{ISO C} and is
968 declared in @file{wchar.h}.
971 The definition of this function has one limitation which has to be
972 understood. The requirement that @var{dst} has to be a NUL terminated
973 string provides problems if one wants to convert buffers with text. A
974 buffer is normally no collection of NUL terminated strings but instead a
975 continuous collection of lines, separated by newline characters. Now
976 assume a function to convert one line from a buffer is needed. Since
977 the line is not NUL terminated the source pointer cannot directly point
978 into the unmodified text buffer. This means, either one inserts the NUL
979 byte at the appropriate place for the time of the @code{mbsrtowcs}
980 function call (which is not doable for a read-only buffer or in a
981 multi-threaded application) or one copies the line in an extra buffer
982 where it can be terminated by a NUL byte. Note that it is not in
983 general possible to limit the number of characters to convert by setting
984 the parameter @var{len} to any specific value. Since it is not known
985 how many bytes each multibyte character sequence is in length one always
986 could do only a guess.
989 There is still a problem with the method of NUL-terminating a line right
990 after the newline character which could lead to very strange results.
991 As said in the description of the @var{mbsrtowcs} function above the
992 conversion state is guaranteed to be in the initial shift state after
993 processing the NUL byte at the end of the input string. But this NUL
994 byte is not really part of the text. I.e., the conversion state after
995 the newline in the original text could be something different than the
996 initial shift state and therefore the first character of the next line
997 is encoded using this state. But the state in question is never
998 accessible to the user since the conversion stops after the NUL byte
999 (which resets the state). Most stateful character sets in use today
1000 require that the shift state after a newline is the initial state--but
1001 this is not a strict guarantee. Therefore simply NUL terminating a
1002 piece of a running text is not always an adequate solution and therefore
1003 never should be used in generally used code.
1005 The generic conversion interface (see @xref{Generic Charset Conversion})
1006 does not have this limitation (it simply works on buffers, not
1007 strings), and the GNU C library contains a set of functions which take
1008 additional parameters specifying the maximal number of bytes which are
1009 consumed from the input string. This way the problem of
1010 @code{mbsrtowcs}'s example above could be solved by determining the line
1011 length and passing this length to the function.
1015 @deftypefun size_t wcsrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps})
1016 The @code{wcsrtombs} function (``wide character string restartable to
1017 multibyte string'') converts the NUL terminated wide character string at
1018 @code{*@var{src}} into an equivalent multibyte character string and
1019 stores the result in the array pointed to by @var{dst}. The NUL wide
1020 character is also converted. The conversion starts in the state
1021 described in the object pointed to by @var{ps} or by a state object
1022 locally to @code{wcsrtombs} in case @var{ps} is a null pointer. If
1023 @var{dst} is a null pointer the conversion is performed as usual but the
1024 result is not available. If all characters of the input string were
1025 successfully converted and if @var{dst} is not a null pointer the
1026 pointer pointed to by @var{src} gets assigned a null pointer.
1028 If one of the wide characters in the input string has no valid multibyte
1029 character equivalent the conversion stops early, sets the global
1030 variable @code{errno} to @code{EILSEQ}, and returns @code{(size_t) -1}.
1032 Another reason for a premature stop is if @var{dst} is not a null
1033 pointer and the next converted character would require more than
1034 @var{len} bytes in total to the array @var{dst}. In this case (and if
1035 @var{dest} is not a null pointer) the pointer pointed to by @var{src} is
1036 assigned a value pointing to the wide character right after the last one
1037 successfully converted.
1039 Except in the case of an encoding error the return value of the function
1040 is the number of bytes in all the multibyte character sequences stored
1041 in @var{dst}. Before returning the state in the object pointed to by
1042 @var{ps} (or the internal object in case @var{ps} is a null pointer) is
1043 updated to reflect the state after the last conversion. The state is
1044 the initial shift state in case the terminating NUL wide character was
1048 This function was introduced in the second amendment to @w{ISO C} and is
1049 declared in @file{wchar.h}.
1052 The restriction mentions above for the @code{mbsrtowcs} function applies
1053 also here. There is no possibility to directly control the number of
1054 input characters. One has to place the NUL wide character at the
1055 correct place or control the consumed input indirectly via the available
1056 output array size (the @var{len} parameter).
1060 @deftypefun size_t mbsnrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{nmc}, size_t @var{len}, mbstate_t *restrict @var{ps})
1061 The @code{mbsnrtowcs} function is very similar to the @code{mbsrtowcs}
1062 function. All the parameters are the same except for @var{nmc} which is
1063 new. The return value is the same as for @code{mbsrtowcs}.
1065 This new parameter specifies how many bytes at most can be used from the
1066 multibyte character string. I.e., the multibyte character string
1067 @code{*@var{src}} need not be NUL terminated. But if a NUL byte is
1068 found within the @var{nmc} first bytes of the string the conversion
1071 This function is a GNU extensions. It is meant to work around the
1072 problems mentioned above. Now it is possible to convert buffer with
1073 multibyte character text piece for piece without having to care about
1074 inserting NUL bytes and the effect of NUL bytes on the conversion state.
1077 A function to convert a multibyte string into a wide character string
1078 and display it could be written like this (this is not a really useful
1083 showmbs (const char *src, FILE *fp)
1087 memset (&state, '\0', sizeof (state));
1090 wchar_t linebuf[100];
1091 const char *endp = strchr (src, '\n');
1094 /* @r{Exit if there is no more line.} */
1098 n = mbsnrtowcs (linebuf, &src, endp - src, 99, &state);
1100 fprintf (fp, "line %d: \"%S\"\n", linebuf);
1105 There is no problem with the state after a call to @code{mbsnrtowcs}.
1106 Since we don't insert characters in the strings which were not in there
1107 right from the beginning and we use @var{state} only for the conversion
1108 of the given buffer there is no problem with altering the state.
1112 @deftypefun size_t wcsnrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{nwc}, size_t @var{len}, mbstate_t *restrict @var{ps})
1113 The @code{wcsnrtombs} function implements the conversion from wide
1114 character strings to multibyte character strings. It is similar to
1115 @code{wcsrtombs} but it takes, just like @code{mbsnrtowcs}, an extra
1116 parameter which specifies the length of the input string.
1118 No more than @var{nwc} wide characters from the input string
1119 @code{*@var{src}} are converted. If the input string contains a NUL
1120 wide character in the first @var{nwc} character to conversion stops at
1123 This function is a GNU extension and just like @code{mbsnrtowcs} is
1124 helps in situations where no NUL terminated input strings are available.
1128 @node Multibyte Conversion Example
1129 @subsection A Complete Multibyte Conversion Example
1131 The example programs given in the last sections are only brief and do
1132 not contain all the error checking etc. Presented here is a complete
1133 and documented example. It features the @code{mbrtowc} function but it
1134 should be easy to derive versions using the other functions.
1138 file_mbsrtowcs (int input, int output)
1140 /* @r{Note the use of @code{MB_LEN_MAX}.}
1141 @r{@code{MB_CUR_MAX} cannot portably be used here.} */
1142 char buffer[BUFSIZ + MB_LEN_MAX];
1147 /* @r{Initialize the state.} */
1148 memset (&state, '\0', sizeof (state));
1155 wchar_t outbuf[BUFSIZ];
1156 wchar_t *outp = outbuf;
1158 /* @r{Fill up the buffer from the input file.} */
1159 nread = read (input, buffer + filled, BUFSIZ);
1165 /* @r{If we reach end of file, make a note to read no more.} */
1169 /* @r{@code{filled} is now the number of bytes in @code{buffer}.} */
1172 /* @r{Convert those bytes to wide characters--as many as we can.} */
1175 size_t thislen = mbrtowc (outp, inp, filled, &state);
1176 /* @r{Stop converting at invalid character;}
1177 @r{this can mean we have read just the first part}
1178 @r{of a valid character.} */
1179 if (thislen == (size_t) -1)
1181 /* @r{We want to handle embedded NUL bytes}
1182 @r{but the return value is 0. Correct this.} */
1185 /* @r{Advance past this character.} */
1191 /* @r{Write the wide characters we just made.} */
1192 nwrite = write (output, outbuf,
1193 (outp - outbuf) * sizeof (wchar_t));
1200 /* @r{See if we have a @emph{real} invalid character.} */
1201 if ((eof && filled > 0) || filled >= MB_CUR_MAX)
1203 error (0, 0, "invalid multibyte character");
1207 /* @r{If any characters must be carried forward,}
1208 @r{put them at the beginning of @code{buffer}.} */
1210 memmove (inp, buffer, filled);
1218 @node Non-reentrant Conversion
1219 @section Non-reentrant Conversion Function
1221 The functions described in the last chapter are defined in the second
1222 amendment to @w{ISO C89}. But the original @w{ISO C89} standard also
1223 contained functions for character set conversion. The reason that they
1224 are not described in the first place is that they are almost entirely
1227 The problem is that all the functions for conversion defined in @w{ISO
1228 C89} use a local state. This implies that multiple conversions at the
1229 same time (not only when using threads) cannot be done, and that you
1230 cannot first convert single characters and then strings since you cannot
1231 tell the conversion functions which state to use.
1233 These functions are therefore usable only in a very limited set of
1234 situations. One must complete converting the entire string before
1235 starting a new one and each string/text must be converted with the same
1236 function (there is no problem with the library itself; it is guaranteed
1237 that no library function changes the state of any of these functions).
1238 @strong{For the above reasons it is highly requested that the functions
1239 from the last section are used in place of non-reentrant conversion
1243 * Non-reentrant Character Conversion:: Non-reentrant Conversion of Single
1245 * Non-reentrant String Conversion:: Non-reentrant Conversion of Strings.
1246 * Shift State:: States in Non-reentrant Functions.
1249 @node Non-reentrant Character Conversion
1250 @subsection Non-reentrant Conversion of Single Characters
1254 @deftypefun int mbtowc (wchar_t *restrict @var{result}, const char *restrict @var{string}, size_t @var{size})
1255 The @code{mbtowc} (``multibyte to wide character'') function when called
1256 with non-null @var{string} converts the first multibyte character
1257 beginning at @var{string} to its corresponding wide character code. It
1258 stores the result in @code{*@var{result}}.
1260 @code{mbtowc} never examines more than @var{size} bytes. (The idea is
1261 to supply for @var{size} the number of bytes of data you have in hand.)
1263 @code{mbtowc} with non-null @var{string} distinguishes three
1264 possibilities: the first @var{size} bytes at @var{string} start with
1265 valid multibyte character, they start with an invalid byte sequence or
1266 just part of a character, or @var{string} points to an empty string (a
1269 For a valid multibyte character, @code{mbtowc} converts it to a wide
1270 character and stores that in @code{*@var{result}}, and returns the
1271 number of bytes in that character (always at least @math{1}, and never
1272 more than @var{size}).
1274 For an invalid byte sequence, @code{mbtowc} returns @math{-1}. For an
1275 empty string, it returns @math{0}, also storing @code{'\0'} in
1276 @code{*@var{result}}.
1278 If the multibyte character code uses shift characters, then
1279 @code{mbtowc} maintains and updates a shift state as it scans. If you
1280 call @code{mbtowc} with a null pointer for @var{string}, that
1281 initializes the shift state to its standard initial value. It also
1282 returns nonzero if the multibyte character code in use actually has a
1283 shift state. @xref{Shift State}.
1288 @deftypefun int wctomb (char *@var{string}, wchar_t @var{wchar})
1289 The @code{wctomb} (``wide character to multibyte'') function converts
1290 the wide character code @var{wchar} to its corresponding multibyte
1291 character sequence, and stores the result in bytes starting at
1292 @var{string}. At most @code{MB_CUR_MAX} characters are stored.
1294 @code{wctomb} with non-null @var{string} distinguishes three
1295 possibilities for @var{wchar}: a valid wide character code (one that can
1296 be translated to a multibyte character), an invalid code, and @code{L'\0'}.
1298 Given a valid code, @code{wctomb} converts it to a multibyte character,
1299 storing the bytes starting at @var{string}. Then it returns the number
1300 of bytes in that character (always at least @math{1}, and never more
1301 than @code{MB_CUR_MAX}).
1303 If @var{wchar} is an invalid wide character code, @code{wctomb} returns
1304 @math{-1}. If @var{wchar} is @code{L'\0'}, it returns @code{0}, also
1305 storing @code{'\0'} in @code{*@var{string}}.
1307 If the multibyte character code uses shift characters, then
1308 @code{wctomb} maintains and updates a shift state as it scans. If you
1309 call @code{wctomb} with a null pointer for @var{string}, that
1310 initializes the shift state to its standard initial value. It also
1311 returns nonzero if the multibyte character code in use actually has a
1312 shift state. @xref{Shift State}.
1314 Calling this function with a @var{wchar} argument of zero when
1315 @var{string} is not null has the side-effect of reinitializing the
1316 stored shift state @emph{as well as} storing the multibyte character
1317 @code{'\0'} and returning @math{0}.
1320 Similar to @code{mbrlen} there is also a non-reentrant function which
1321 computes the length of a multibyte character. It can be defined in
1322 terms of @code{mbtowc}.
1326 @deftypefun int mblen (const char *@var{string}, size_t @var{size})
1327 The @code{mblen} function with a non-null @var{string} argument returns
1328 the number of bytes that make up the multibyte character beginning at
1329 @var{string}, never examining more than @var{size} bytes. (The idea is
1330 to supply for @var{size} the number of bytes of data you have in hand.)
1332 The return value of @code{mblen} distinguishes three possibilities: the
1333 first @var{size} bytes at @var{string} start with valid multibyte
1334 character, they start with an invalid byte sequence or just part of a
1335 character, or @var{string} points to an empty string (a null character).
1337 For a valid multibyte character, @code{mblen} returns the number of
1338 bytes in that character (always at least @code{1}, and never more than
1339 @var{size}). For an invalid byte sequence, @code{mblen} returns
1340 @math{-1}. For an empty string, it returns @math{0}.
1342 If the multibyte character code uses shift characters, then @code{mblen}
1343 maintains and updates a shift state as it scans. If you call
1344 @code{mblen} with a null pointer for @var{string}, that initializes the
1345 shift state to its standard initial value. It also returns a nonzero
1346 value if the multibyte character code in use actually has a shift state.
1350 The function @code{mblen} is declared in @file{stdlib.h}.
1354 @node Non-reentrant String Conversion
1355 @subsection Non-reentrant Conversion of Strings
1357 For convenience reasons the @w{ISO C89} standard defines also functions
1358 to convert entire strings instead of single characters. These functions
1359 suffer from the same problems as their reentrant counterparts from the
1360 second amendment to @w{ISO C89}; see @xref{Converting Strings}.
1364 @deftypefun size_t mbstowcs (wchar_t *@var{wstring}, const char *@var{string}, size_t @var{size})
1365 The @code{mbstowcs} (``multibyte string to wide character string'')
1366 function converts the null-terminated string of multibyte characters
1367 @var{string} to an array of wide character codes, storing not more than
1368 @var{size} wide characters into the array beginning at @var{wstring}.
1369 The terminating null character counts towards the size, so if @var{size}
1370 is less than the actual number of wide characters resulting from
1371 @var{string}, no terminating null character is stored.
1373 The conversion of characters from @var{string} begins in the initial
1376 If an invalid multibyte character sequence is found, this function
1377 returns a value of @math{-1}. Otherwise, it returns the number of wide
1378 characters stored in the array @var{wstring}. This number does not
1379 include the terminating null character, which is present if the number
1380 is less than @var{size}.
1382 Here is an example showing how to convert a string of multibyte
1383 characters, allocating enough space for the result.
1387 mbstowcs_alloc (const char *string)
1389 size_t size = strlen (string) + 1;
1390 wchar_t *buf = xmalloc (size * sizeof (wchar_t));
1392 size = mbstowcs (buf, string, size);
1393 if (size == (size_t) -1)
1395 buf = xrealloc (buf, (size + 1) * sizeof (wchar_t));
1404 @deftypefun size_t wcstombs (char *@var{string}, const wchar_t *@var{wstring}, size_t @var{size})
1405 The @code{wcstombs} (``wide character string to multibyte string'')
1406 function converts the null-terminated wide character array @var{wstring}
1407 into a string containing multibyte characters, storing not more than
1408 @var{size} bytes starting at @var{string}, followed by a terminating
1409 null character if there is room. The conversion of characters begins in
1410 the initial shift state.
1412 The terminating null character counts towards the size, so if @var{size}
1413 is less than or equal to the number of bytes needed in @var{wstring}, no
1414 terminating null character is stored.
1416 If a code that does not correspond to a valid multibyte character is
1417 found, this function returns a value of @math{-1}. Otherwise, the
1418 return value is the number of bytes stored in the array @var{string}.
1419 This number does not include the terminating null character, which is
1420 present if the number is less than @var{size}.
1424 @subsection States in Non-reentrant Functions
1426 In some multibyte character codes, the @emph{meaning} of any particular
1427 byte sequence is not fixed; it depends on what other sequences have come
1428 earlier in the same string. Typically there are just a few sequences
1429 that can change the meaning of other sequences; these few are called
1430 @dfn{shift sequences} and we say that they set the @dfn{shift state} for
1431 other sequences that follow.
1433 To illustrate shift state and shift sequences, suppose we decide that
1434 the sequence @code{0200} (just one byte) enters Japanese mode, in which
1435 pairs of bytes in the range from @code{0240} to @code{0377} are single
1436 characters, while @code{0201} enters Latin-1 mode, in which single bytes
1437 in the range from @code{0240} to @code{0377} are characters, and
1438 interpreted according to the ISO Latin-1 character set. This is a
1439 multibyte code which has two alternative shift states (``Japanese mode''
1440 and ``Latin-1 mode''), and two shift sequences that specify particular
1443 When the multibyte character code in use has shift states, then
1444 @code{mblen}, @code{mbtowc} and @code{wctomb} must maintain and update
1445 the current shift state as they scan the string. To make this work
1446 properly, you must follow these rules:
1450 Before starting to scan a string, call the function with a null pointer
1451 for the multibyte character address---for example, @code{mblen (NULL,
1452 0)}. This initializes the shift state to its standard initial value.
1455 Scan the string one character at a time, in order. Do not ``back up''
1456 and rescan characters already scanned, and do not intersperse the
1457 processing of different strings.
1460 Here is an example of using @code{mblen} following these rules:
1464 scan_string (char *s)
1466 int length = strlen (s);
1468 /* @r{Initialize shift state.} */
1473 int thischar = mblen (s, length);
1474 /* @r{Deal with end of string and invalid characters.} */
1479 error ("invalid multibyte character");
1482 /* @r{Advance past this character.} */
1489 The functions @code{mblen}, @code{mbtowc} and @code{wctomb} are not
1490 reentrant when using a multibyte code that uses a shift state. However,
1491 no other library functions call these functions, so you don't have to
1492 worry that the shift state will be changed mysteriously.
1495 @node Generic Charset Conversion
1496 @section Generic Charset Conversion
1498 The conversion functions mentioned so far in this chapter all had in
1499 common that they operate on character sets which are not directly
1500 specified by the functions. The multibyte encoding used is specified by
1501 the currently selected locale for the @code{LC_CTYPE} category. The
1502 wide character set is fixed by the implementation (in the case of GNU C
1503 library it always is UCS4 encoded @w{ISO 10646}.
1505 This has of course several problems when it comes to general character
1510 For every conversion where neither the source or destination character
1511 set is the character set of the locale for the @code{LC_CTYPE} category,
1512 one has to change the @code{LC_CTYPE} locale using @code{setlocale}.
1514 This introduces major problems for the rest of the programs since
1515 several more functions (e.g., the character classification functions,
1516 @xref{Classification of Characters}) use the @code{LC_CTYPE} category.
1519 Parallel conversions to and from different character sets are not
1520 possible since the @code{LC_CTYPE} selection is global and shared by all
1524 If neither the source nor the destination character set is the character
1525 set used for @code{wchar_t} representation there is at least a two-step
1526 process necessary to convert a text using the functions above. One
1527 would have to select the source character set as the multibyte encoding,
1528 convert the text into a @code{wchar_t} text, select the destination
1529 character set as the multibyte encoding and convert the wide character
1530 text to the multibyte (@math{=} destination) character set.
1532 Even if this is possible (which is not guaranteed) it is a very tiring
1533 work. Plus it suffers from the other two raised points even more due to
1534 the steady changing of the locale.
1538 The XPG2 standard defines a completely new set of functions which has
1539 none of these limitations. They are not at all coupled to the selected
1540 locales and they but no constraints on the character sets selected for
1541 source and destination. Only the set of available conversions is
1542 limiting them. The standard does not specify that any conversion at all
1543 must be available. It is a measure of the quality of the implementation.
1545 In the following text first the interface to @code{iconv}, the
1546 conversion function, will be described. Comparisons with other
1547 implementations will show what pitfalls lie on the way of portable
1548 applications. At last, the implementation is described as far as
1549 interesting to the advanced user who wants to extend the conversion
1553 * Generic Conversion Interface:: Generic Character Set Conversion Interface.
1554 * iconv Examples:: A complete @code{iconv} example.
1555 * Other iconv Implementations:: Some Details about other @code{iconv}
1557 * glibc iconv Implementation:: The @code{iconv} Implementation in the GNU C
1561 @node Generic Conversion Interface
1562 @subsection Generic Character Set Conversion Interface
1564 This set of functions follows the traditional cycle of using a resource:
1565 open--use--close. The interface consists of three functions, each of
1566 which implement one step.
1568 Before the interfaces are described it is necessary to introduce a
1569 datatype. Just like other open--use--close interface the functions
1570 introduced here work using a handles and the @file{iconv.h} header
1571 defines a special type for the handles used.
1575 @deftp {Data Type} iconv_t
1576 This data type is an abstract type defined in @file{iconv.h}. The user
1577 must not assume anything about the definition of this type, it must be
1580 Objects of this type can get assigned handles for the conversions using
1581 the @code{iconv} functions. The objects themselves need not be freed but
1582 the conversions for which the handles stand for have to.
1586 The first step is the function to create a handle.
1590 @deftypefun iconv_t iconv_open (const char *@var{tocode}, const char *@var{fromcode})
1591 The @code{iconv_open} function has to be used before starting a
1592 conversion. The two parameters this function takes determine the
1593 source and destination character set for the conversion and if the
1594 implementation has the possibility to perform such a conversion the
1595 function returns a handle.
1597 If the wanted conversion is not available the function returns
1598 @code{(iconv_t) -1}. In this case the global variable @code{errno} can
1599 have the following values:
1603 The process already has @code{OPEN_MAX} file descriptors open.
1605 The system limit of open file is reached.
1607 Not enough memory to carry out the operation.
1609 The conversion from @var{fromcode} to @var{tocode} is not supported.
1612 It is not possible to use the same descriptor in different threads to
1613 perform independent conversions. Within the data structures associated
1614 with the descriptor there is information about the conversion state.
1615 This must not be messed up by using it in different conversions.
1617 An @code{iconv} descriptor is like a file descriptor as for every use a
1618 new descriptor must be created. The descriptor does not stand for all
1619 of the conversions from @var{fromset} to @var{toset}.
1621 The GNU C library implementation of @code{iconv_open} has one
1622 significant extension to other implementations. To ease the extension
1623 of the set of available conversions the implementation allows to store
1624 the necessary files with data and code in arbitrary many directories.
1625 How this extensions have to be written will be explained below
1626 (@pxref{glibc iconv Implementation}). Here it is only important to say
1627 that all directories mentioned in the @code{GCONV_PATH} environment
1628 variable are considered if they contain a file @file{gconv-modules}.
1629 These directories need not necessarily be created by the system
1630 administrator. In fact, this extension is introduced to help users
1631 writing and using own, new conversions. Of course this does not work
1632 for security reasons in SUID binaries; in this case only the system
1633 directory is considered and this normally is
1634 @file{@var{prefix}/lib/gconv}. The @code{GCONV_PATH} environment
1635 variable is examined exactly once at the first call of the
1636 @code{iconv_open} function. Later modifications of the variable have no
1640 This function got introduced early in the X/Open Portability Guide,
1641 @w{version 2}. It is supported by all commercial Unices as it is
1642 required for the Unix branding. However, the quality and completeness
1643 of the implementation varies widely. The function is declared in
1647 The @code{iconv} implementation can associate large data structure with
1648 the handle returned by @code{iconv_open}. Therefore it is crucial to
1649 free all the resources once all conversions are carried out and the
1650 conversion is not needed anymore.
1654 @deftypefun int iconv_close (iconv_t @var{cd})
1655 The @code{iconv_close} function frees all resources associated with the
1656 handle @var{cd} which must have been returned by a successful call to
1657 the @code{iconv_open} function.
1659 If the function call was successful the return value is @math{0}.
1660 Otherwise it is @math{-1} and @code{errno} is set appropriately.
1665 The conversion descriptor is invalid.
1669 This function was introduced together with the rest of the @code{iconv}
1670 functions in XPG2 and it is declared in @file{iconv.h}.
1673 The standard defines only one actual conversion function. This has
1674 therefore the most general interface: it allows conversion from one
1675 buffer to another. Conversion from a file to a buffer, vice versa, or
1676 even file to file can be implemented on top of it.
1680 @deftypefun size_t iconv (iconv_t @var{cd}, const char **@var{inbuf}, size_t *@var{inbytesleft}, char **@var{outbuf}, size_t *@var{outbytesleft})
1682 The @code{iconv} function converts the text in the input buffer
1683 according to the rules associated with the descriptor @var{cd} and
1684 stores the result in the output buffer. It is possible to call the
1685 function for the same text several times in a row since for stateful
1686 character sets the necessary state information is kept in the data
1687 structures associated with the descriptor.
1689 The input buffer is specified by @code{*@var{inbuf}} and it contains
1690 @code{*@var{inbytesleft}} bytes. The extra indirection is necessary for
1691 communicating the used input back to the caller (see below). It is
1692 important to note that the buffer pointer is of type @code{char} and the
1693 length is measured in bytes even if the input text is encoded in wide
1696 The output buffer is specified in a similar way. @code{*@var{outbuf}}
1697 points to the beginning of the buffer with at least
1698 @code{*@var{outbytesleft}} bytes room for the result. The buffer
1699 pointer again is of type @code{char} and the length is measured in
1700 bytes. If @var{outbuf} or @code{*@var{outbuf}} is a null pointer the
1701 conversion is performed but no output is available.
1703 If @var{inbuf} is a null pointer the @code{iconv} function performs the
1704 necessary action to put the state of the conversion into the initial
1705 state. This is obviously a no-op for non-stateful encodings, but if the
1706 encoding has a state such a function call might put some byte sequences
1707 in the output buffer which perform the necessary state changes. The
1708 next call with @var{inbuf} not being a null pointer then simply goes on
1709 from the initial state. It is important that the programmer never makes
1710 any assumption on whether the conversion has to deal with states or not.
1711 Even if the input and output character sets are not stateful the
1712 implementation might still have to keep states. This is due to the
1713 implementation chosen for the GNU C library as it is described below.
1714 Therefore an @code{iconv} call to reset the state should always be
1715 performed if some protocol requires this for the output text.
1717 The conversion stops for three reasons. The first is that all
1718 characters from the input buffer are converted. This actually can mean
1719 two things: really all bytes from the input buffer are consumed or
1720 there are some bytes at the end of the buffer which possibly can form a
1721 complete character but the input is incomplete. The second reason for a
1722 stop is when the output buffer is full. And the third reason is that
1723 the input contains invalid characters.
1725 In all these cases the buffer pointers after the last successful
1726 conversion, for input and output buffer, are stored in @var{inbuf} and
1727 @var{outbuf} and the available room in each buffer is stored in
1728 @var{inbytesleft} and @var{outbytesleft}.
1730 Since the character sets selected in the @code{iconv_open} call can be
1731 almost arbitrary there can be situations where the input buffer contains
1732 valid characters which have no identical representation in the output
1733 character set. The behavior in this situation is undefined. The
1734 @emph{current} behavior of the GNU C library in this situation is to
1735 return with an error immediately. This certainly is not the most
1736 desirable solution. Therefore future versions will provide better ones
1737 but they are not yet finished.
1739 If all input from the input buffer is successfully converted and stored
1740 in the output buffer the function returns the number of conversions
1741 performed. In all other cases the return value is @code{(size_t) -1}
1742 and @code{errno} is set appropriately. In this case the value pointed
1743 to by @var{inbytesleft} is nonzero.
1747 The conversion stopped because of an invalid byte sequence in the input.
1748 After the call @code{*@var{inbuf}} points at the first byte of the
1749 invalid byte sequence.
1752 The conversion stopped because it ran out of space in the output buffer.
1755 The conversion stopped because of an incomplete byte sequence at the end
1756 of the input buffer.
1759 The @var{cd} argument is invalid.
1763 This function was introduced in the XPG2 standard and is declared in the
1764 @file{iconv.h} header.
1767 The definition of the @code{iconv} function is quite good overall. It
1768 provides quite flexible functionality. The only problems lie in the
1769 boundary cases which are incomplete byte sequences at the end of the
1770 input buffer and invalid input. A third problem, which is not really
1771 a design problem, is the way conversions are selected. The standard
1772 does not say anything about the legitimate names, a minimal set of
1773 available conversions. We will see how this negatively impacts other
1774 implementations, as is demonstrated below.
1777 @node iconv Examples
1778 @subsection A complete @code{iconv} example
1780 The example below features a solution for a common problem. Given that
1781 one knows the internal encoding used by the system for @code{wchar_t}
1782 strings one often is in the position to read text from a file and store
1783 it in wide character buffers. One can do this using @code{mbsrtowcs}
1784 but then we run into the problems discussed above.
1788 file2wcs (int fd, const char *charset, wchar_t *outbuf, size_t avail)
1792 char *wrptr = (char *) outbuf;
1796 cd = iconv_open ("UCS4", charset);
1797 if (cd == (iconv_t) -1)
1799 /* @r{Something went wrong.} */
1800 if (errno == EINVAL)
1801 error (0, 0, "conversion from `%s' to `UCS4' no available",
1804 perror ("iconv_open");
1806 /* @r{Terminate the output string.} */
1816 char *inptr = inbuf;
1818 /* @r{Read more input.} */
1819 nread = read (fd, inbuf + insize, sizeof (inbuf) - insize);
1822 /* @r{When we come here the file is completely read.}
1823 @r{This still could mean there are some unused}
1824 @r{characters in the @code{inbuf}. Put them back.} */
1825 if (lseek (fd, -insize, SEEK_CUR) == -1)
1831 /* @r{Do the conversion.} */
1832 nconv = iconv (cd, &inptr, &insize, &wrptr, &avail);
1833 if (nconv == (size_t) -1)
1835 /* @r{Not everything went right. It might only be}
1836 @r{an unfinished byte sequence at the end of the}
1837 @r{buffer. Or it is a real problem.} */
1838 if (errno == EINVAL)
1839 /* @r{This is harmless. Simply move the unused}
1840 @r{bytes to the beginning of the buffer so that}
1841 @r{they can be used in the next round.} */
1842 memmove (inbuf, inptr, insize);
1845 /* @r{It is a real problem. Maybe we ran out of}
1846 @r{space in the output buffer or we have invalid}
1847 @r{input. In any case back the file pointer to}
1848 @r{the position of the last processed byte.} */
1849 lseek (fd, -insize, SEEK_CUR);
1856 /* @r{Terminate the output string.} */
1857 *((wchar_t *) wrptr) = L'\0';
1859 if (iconv_close (cd) != 0)
1860 perror ("iconv_close");
1862 return (wchar_t *) wrptr - outbuf;
1867 This example shows the most important aspects of using the @code{iconv}
1868 functions. It shows how successive calls to @code{iconv} can be used to
1869 convert large amounts of text. The user does not have to care about
1870 stateful encodings as the functions take care of everything.
1872 An interesting point is the case where @code{iconv} return an error and
1873 @code{errno} is set to @code{EINVAL}. This is not really an error in
1874 the transformation. It can happen whenever the input character set
1875 contains byte sequences of more than one byte for some character and
1876 texts are not processed in one piece. In this case there is a chance
1877 that a multibyte sequence is cut. The caller than can simply read the
1878 remainder of the takes and feed the offending bytes together with new
1879 character from the input to @code{iconv} and continue the work. The
1880 internal state kept in the descriptor is @emph{not} unspecified after
1881 such an event as it is the case with the conversion functions from the
1884 The example also shows the problem of using wide character strings with
1885 @code{iconv}. As explained in the description of the @code{iconv}
1886 function above the function always takes a pointer to a @code{char}
1887 array and the available space is measured in bytes. In the example the
1888 output buffer is a wide character buffer. Therefore we use a local
1889 variable @var{wrptr} of type @code{char *} which is used in the
1892 This looks rather innocent but can lead to problems on platforms which
1893 have tight restriction on alignment. Therefore the caller of
1894 @code{iconv} has to make sure that the pointers passed are suitable for
1895 access of characters from the appropriate character set. Since in the
1896 above case the input parameter to the function is a @code{wchar_t}
1897 pointer this is the case (unless the user violates alignment when
1898 computing the parameter). But in other situations, especially when
1899 writing generic functions where one does not know what type of character
1900 set one uses and therefore treats text as a sequence of bytes, it might
1904 @node Other iconv Implementations
1905 @subsection Some Details about other @code{iconv} Implementations
1907 This is not really the place to discuss the @code{iconv} implementation
1908 of other systems but it is necessary to know a bit about them to write
1909 portable programs. The above mentioned problems with the specification
1910 of the @code{iconv} functions can lead to portability issues.
1912 The first thing to notice is that due to the large number of character
1913 sets in use it is certainly not practical to encode the conversions
1914 directly in the C library. Therefore the conversion information must
1915 come from files outside the C library. This is usually done in one or
1916 both of the following ways:
1920 The C library contains a set of generic conversion functions which can
1921 read the needed conversion tables and other information from data files.
1922 These files get loaded when necessary.
1924 This solution is problematic as it requires a great deal of effort to
1925 apply to all character sets (potentially an infinite set). The
1926 differences in the structure of the different character sets is so large
1927 that many different variants of the table processing functions must be
1928 developed. On top of this the generic nature of these functions make
1929 them slower than specifically implemented functions.
1932 The C library only contains a framework which can dynamically load
1933 object files and execute the therein contained conversion functions.
1935 This solution provides much more flexibility. The C library itself
1936 contains only very little code and therefore reduces the general memory
1937 footprint. Also, with a documented interface between the C library and
1938 the loadable modules it is possible for third parties to extend the set
1939 of available conversion modules. A drawback of this solution is that
1940 dynamic loading must be available.
1943 Some implementations in commercial Unices implement a mixture of these
1944 these possibilities, the majority only the second solution. Using
1945 loadable modules moves the code out of the library itself and keeps the
1946 door open for extensions and improvements. But this design is also
1947 limiting on some platforms since not many platforms support dynamic
1948 loading in statically linked programs. On platforms without his
1949 capability it is therefore not possible to use this interface in
1950 statically linked programs. The GNU C library has on ELF platforms no
1951 problems with dynamic loading in in these situations and therefore this
1952 point is mood. The danger is that one gets acquainted with this and
1953 forgets about the restrictions on other systems.
1955 A second thing to know about other @code{iconv} implementations is that
1956 the number of available conversions is often very limited. Some
1957 implementations provide in the standard release (not special
1958 international or developer releases) at most 100 to 200 conversion
1959 possibilities. This does not mean 200 different character sets are
1960 supported. E.g., conversions from one character set to a set of, say,
1961 10 others counts as 10 conversion. Together with the other direction
1962 this makes already 20. One can imagine the thin coverage these platform
1963 provide. Some Unix vendors even provide only a handful of conversions
1964 which renders them useless for almost all uses.
1966 This directly leads to a third and probably the most problematic point.
1967 The way the @code{iconv} conversion functions are implemented on all
1968 known Unix system and the availability of the conversion functions from
1969 character set @math{@cal{A}} to @math{@cal{B}} and the conversion from
1970 @math{@cal{B}} to @math{@cal{C}} does @emph{not} imply that the
1971 conversion from @math{@cal{A}} to @math{@cal{C}} is available.
1973 This might not seem unreasonable and problematic at first but it is a
1974 quite big problem as one will notice shortly after hitting it. To show
1975 the problem we assume to write a program which has to convert from
1976 @math{@cal{A}} to @math{@cal{C}}. A call like
1979 cd = iconv_open ("@math{@cal{C}}", "@math{@cal{A}}");
1983 does fail according to the assumption above. But what does the program
1984 do now? The conversion is really necessary and therefore simply giving
1985 up is no possibility.
1987 This is a nuisance. The @code{iconv} function should take care of this.
1988 But how should the program proceed from here on? If it would try to
1989 convert to character set @math{@cal{B}} first the two @code{iconv_open}
1993 cd1 = iconv_open ("@math{@cal{B}}", "@math{@cal{A}}");
2000 cd2 = iconv_open ("@math{@cal{C}}", "@math{@cal{B}}");
2004 will succeed but how to find @math{@cal{B}}?
2006 Unfortunately, the answer is: there is no general solution. On some
2007 systems guessing might help. On those systems most character sets can
2008 convert to and from UTF8 encoded @w{ISO 10646} or Unicode text.
2009 Beside this only some very system-specific methods can help. Since the
2010 conversion functions come from loadable modules and these modules must
2011 be stored somewhere in the filesystem, one @emph{could} try to find them
2012 and determine from the available file which conversions are available
2013 and whether there is an indirect route from @math{@cal{A}} to
2016 This shows one of the design errors of @code{iconv} mentioned above. It
2017 should at least be possible to determine the list of available
2018 conversion programmatically so that if @code{iconv_open} says there is
2019 no such conversion, one could make sure this also is true for indirect
2023 @node glibc iconv Implementation
2024 @subsection The @code{iconv} Implementation in the GNU C library
2026 After reading about the problems of @code{iconv} implementations in the
2027 last section it is certainly good to note that the implementation in
2028 the GNU C library has none of the problems mentioned above. What
2029 follows is a step-by-step analysis of the points raised above. The
2030 evaluation is based on the current state of the development (as of
2031 January 1999). The development of the @code{iconv} functions is not
2032 complete, but basic funtionality has solidified.
2034 The GNU C library's @code{iconv} implementation uses shared loadable
2035 modules to implement the conversions. A very small number of
2036 conversions are built into the library itself but these are only rather
2037 trivial conversions.
2039 All the benefits of loadable modules are available in the GNU C library
2040 implementation. This is especially appealing since the interface is
2041 well documented (see below) and it therefore is easy to write new
2042 conversion modules. The drawback of using loadable objects is not a
2043 problem in the GNU C library, at least on ELF systems. Since the
2044 library is able to load shared objects even in statically linked
2045 binaries this means that static linking needs not to be forbidden in
2046 case one wants to use @code{iconv}.
2048 The second mentioned problem is the number of supported conversions.
2049 Currently, the GNU C library supports more than 150 character sets. The
2050 way the implementation is designed the number of supported conversions
2051 is greater than 22350 (@math{150} times @math{149}). If any conversion
2052 from or to a character set is missing it can easily be added.
2054 Particularly impressive as it may be, this high number is due to the
2055 fact that the GNU C library implementation of @code{iconv} does not have
2056 the third problem mentioned above. I.e., whenever there is a conversion
2057 from a character set @math{@cal{A}} to @math{@cal{B}} and from
2058 @math{@cal{B}} to @math{@cal{C}} it is always possible to convert from
2059 @math{@cal{A}} to @math{@cal{C}} directly. If the @code{iconv_open}
2060 returns an error and sets @code{errno} to @code{EINVAL} this really
2061 means there is no known way, directly or indirectly, to perform the
2064 @cindex triangulation
2065 This is achieved by providing for each character set a conversion from
2066 and to UCS4 encoded @w{ISO 10646}. Using @w{ISO 10646} as an
2067 intermediate representation it is possible to @dfn{triangulate}, i.e.,
2068 converting with an intermediate representation.
2070 There is no inherent requirement to provide a conversion to @w{ISO
2071 10646} for a new character set and it is also possible to provide other
2072 conversions where neither source nor destination character set is @w{ISO
2073 10646}. The currently existing set of conversions is simply meant to
2074 cover all conversions which might be of interest.
2078 All currently available conversions use the triangulation method above,
2079 making conversion run unnecessarily slow. If, e.g., somebody often
2080 needs the conversion from ISO-2022-JP to EUC-JP, a quicker solution
2081 would involve direct conversion between the two character sets, skipping
2082 the input to @w{ISO 10646} first. The two character sets of interest
2083 are much more similar to each other than to @w{ISO 10646}.
2085 In such a situation one can easy write a new conversion and provide it
2086 as a better alternative. The GNU C library @code{iconv} implementation
2087 would automatically use the module implementing the conversion if it is
2088 specified to be more efficient.
2090 @subsubsection Format of @file{gconv-modules} files
2092 All information about the available conversions comes from a file named
2093 @file{gconv-modules} which can be found in any of the directories along
2094 the @code{GCONV_PATH}. The @file{gconv-modules} files are line-oriented
2095 text files, where each of the lines has one of the following formats:
2099 If the first non-whitespace character is a @kbd{#} the line contains
2100 only comments and is ignored.
2103 Lines starting with @code{alias} define an alias name for a character
2104 set. There are two more words expected on the line. The first one
2105 defines the alias name and the second defines the original name of the
2106 character set. The effect is that it is possible to use the alias name
2107 in the @var{fromset} or @var{toset} parameters of @code{iconv_open} and
2108 achieve the same result as when using the real character set name.
2110 This is quite important as a character set has often many different
2111 names. There is normally always an official name but this need not
2112 correspond to the most popular name. Beside this many character sets
2113 have special names which are somehow constructed. E.g., all character
2114 sets specified by the ISO have an alias of the form
2115 @code{ISO-IR-@var{nnn}} where @var{nnn} is the registration number.
2116 This allows programs which know about the registration number to
2117 construct character set names and use them in @code{iconv_open} calls.
2118 More on the available names and aliases follows below.
2121 Lines starting with @code{module} introduce an available conversion
2122 module. These lines must contain three or four more words.
2124 The first word specifies the source character set, the second word the
2125 destination character set of conversion implemented in this module. The
2126 third word is the name of the loadable module. The filename is
2127 constructed by appending the usual shared object prefix (normally
2128 @file{.so}) and this file is then supposed to be found in the same
2129 directory the @file{gconv-modules} file is in. The last word on the
2130 line, which is optional, is a numeric value representing the cost of the
2131 conversion. If this word is missing a cost of @math{1} is assumed. The
2132 numeric value itself does not matter that much; what counts are the
2133 relative values of the sums of costs for all possible conversion paths.
2134 Below is a more precise description of the use of the cost value.
2137 Returning to the example above where one has written a module to directly
2138 convert from ISO-2022-JP to EUC-JP and back. All what has to be done is
2139 to put the new module, be its name ISO2022JP-EUCJP.so, in a directory
2140 and add a file @file{gconv-modules} with the following content in the
2144 module ISO-2022-JP// EUC-JP// ISO2022JP-EUCJP 1
2145 module EUC-JP// ISO-2022-JP// ISO2022JP-EUCJP 1
2148 To see why this is sufficient, it is necessary to understand how the
2149 conversion used by @code{iconv} (and described in the descriptor) is
2150 selected. The approach to this problem is quite simple.
2152 At the first call of the @code{iconv_open} function the program reads
2153 all available @file{gconv-modules} files and builds up two tables: one
2154 containing all the known aliases and another which contains the
2155 information about the conversions and which shared object implements
2158 @subsubsection Finding the conversion path in @code{iconv}
2160 The set of available conversions form a directed graph with weighted
2161 edges. The weights on the edges are the costs specified in the
2162 @file{gconv-modules} files. The @code{iconv_open} function uses an
2163 algorithm suitable for search for the best path in such a graph and so
2164 constructs a list of conversions which must be performed in succession
2165 to get the transformation from the source to the destination character
2168 Explaining why the above @file{gconv-modules} files allows the
2169 @code{iconv} implementation to resolve the specific ISO-2022-JP to
2170 EUC-JP conversion module instead of the conversion coming with the
2171 library itself is straighforward. Since the later conversion takes two
2172 steps (from ISO-2022-JP to @w{ISO 10646} and then from @w{ISO 10646} to
2173 EUC-JP) the cost is @math{1+1 = 2}. But the above @file{gconv-modules}
2174 file specifies that the new conversion modules can perform this
2175 conversion with only the cost of @math{1}.
2177 A mysterious piece about the @file{gconv-modules} file above (and also
2178 the file coming with the GNU C library) are the names of the character
2179 sets specified in the @code{module} lines. Why do almost all the names
2180 end in @code{//}? And this is not all: the names can actually be
2181 regular expressions. At this point of time this mystery should not be
2182 revealed, unless you have the relevant spell-casting materials: ashes
2183 from an original @w{DOS 6.2} boot disk burnt in effigy, a crucifix
2184 blessed by St.@: Emacs, assorted herbal roots from Central America, sand
2185 from Cebu, etc. Sorry! @strong{The part of the implementation where
2186 this is used is not yet finished. For now please simply follow the
2187 existing examples. It'll become clearer once it is. --drepper}
2189 A last remark about the @file{gconv-modules} is about the names not
2190 ending with @code{//}. There often is a character set named
2191 @code{INTERNAL} mentioned. From the discussion above and the chosen
2192 name it should have become clear that this is the names for the
2193 representation used in the intermediate step of the triangulation. We
2194 have said that this is UCS4 but actually it is not quite right. The
2195 UCS4 specification also includes the specification of the byte ordering
2196 used. Since an UCS4 value consists of four bytes a stored value is
2197 effected by byte ordering. The internal representation is @emph{not}
2198 the same as UCS4 in case the byte ordering of the processor (or at least
2199 the running process) is not the same as the one required for UCS4. This
2200 is done for performance reasons as one does not want to perform
2201 unnecessary byte-swapping operations if one is not interested in actually
2202 seeing the result in UCS4. To avoid trouble with endianess the internal
2203 representation consistently is named @code{INTERNAL} even on big-endian
2204 systems where the representations are identical.
2206 @subsubsection @code{iconv} module data structures
2208 So far this section described how modules are located and considered to
2209 be used. What remains to be described is the interface of the modules
2210 so that one can write new ones. This section describes the interface as
2211 it is in use in January 1999. The interface will change in future a bit
2212 but hopefully only in an upward compatible way.
2214 The definitions necessary to write new modules are publically available
2215 in the non-standard header @file{gconv.h}. The following text will
2216 therefore describe the definitions from this header file. But first it
2217 is necessary to get an overview.
2219 From the perspective of the user of @code{iconv} the interface is quite
2220 simple: the @code{iconv_open} function returns a handle which can be
2221 used in calls @code{iconv} and finally the handle is freed with a call
2222 to @code{iconv_close}. The problem is: the handle has to be able to
2223 represent the possibly long sequences of conversion steps and also the
2224 state of each conversion since the handle is all which is passed to the
2225 @code{iconv} function. Therefore the data structures are really the
2226 elements to understanding the implementation.
2228 We need two different kinds of data structures. The first describes the
2229 conversion and the second describes the state etc. There are really two
2230 type definitions like this in @file{gconv.h}.
2235 @deftp {Data type} {struct gconv_step}
2236 This data structure describes one conversion a module can perform. For
2237 each function in a loaded module with conversion functions there is
2238 exactly one object of this type. This object is shared by all users of
2239 the conversion. I.e., this object does not contain any information
2240 corresponding to an actual conversion. It only describes the conversion
2244 @item struct gconv_loaded_object *shlib_handle
2245 @itemx const char *modname
2247 All these elements of the structure are used internally in the C library
2248 to coordinate loading and unloading the shared. One must not expect any
2249 of the other elements be available or initialized.
2251 @item const char *from_name
2252 @itemx const char *to_name
2253 @code{from_name} and @code{to_name} contain the names of the source and
2254 destination character sets. They can be used to identify the actual
2255 conversion to be carried out since one module might implement
2256 conversions for more than one character set and/or direction.
2259 @itemx gconv_init_fct init_fct
2260 @itemx gconv_end_fct end_fct
2261 These elements contain pointers to the functions in the loadable module.
2262 The interface will be explained below.
2264 @item int min_needed_from
2265 @itemx int max_needed_from
2266 @itemx int min_needed_to
2267 @itemx int max_needed_to;
2268 These values have to be filled in the the init function of the module.
2269 The @code{min_needed_from} value specifies how many bytes a character of
2270 the source character set at least needs. The @code{max_needed_from}
2271 specifies the maximum value which also includes possible shift
2274 The @code{min_needed_to} and @code{max_needed_to} values serve the same
2275 purpose but this time for the destination character set.
2277 It is crucial that these values are accurate since otherwise the
2278 conversion functions will have problems or not work at all.
2281 This element must also be initialized by the init function. It is
2282 nonzero if the source character set is stateful. Otherwise it is zero.
2285 This element can be used freely by the conversion functions in the
2286 module. It can be used to communicate extra information from one call
2287 to another. It need not be initialized if not needed at all. If this
2288 element gets assigned a pointer to dynamically allocated memory
2289 (presumably in the init function) it has to be made sure that the end
2290 function deallocates the memory. Otherwise the application will leak
2293 It is important to be aware that this data structure is shared by all
2294 users of this specification conversion and therefore the @code{data}
2295 element must not contain data specific to one specific use of the
2296 conversion function.
2302 @deftp {Data type} {struct gconv_step_data}
2303 This is the data structure which contains the information specific to
2304 each use of the conversion functions.
2308 @itemx char *outbufend
2309 These elements specify the output buffer for the conversion step. The
2310 @code{outbuf} element points to the beginning of the buffer and
2311 @code{outbufend} points to the byte following the last byte in the
2312 buffer. The conversion function must not assume anything about the size
2313 of the buffer but it can be safely assumed the there is room for at
2314 least one complete character in the output buffer.
2316 Once the conversion is finished and the conversion is the last step the
2317 @code{outbuf} element must be modified to point after last last byte
2318 written into the buffer to signal how much output is available. If this
2319 conversion step is not the last one the element must not be modified.
2320 The @code{outbufend} element must not be modified.
2323 This element is nonzero if this conversion step is the last one. This
2324 information is necessary for the recursion. See the description of the
2325 conversion function internals below. This element must never be
2328 @item int invocation_counter
2329 The conversion function can use this element to see how many calls of
2330 the conversion function already happened. Some character sets require
2331 when generating output a certain prolog and by comparing this value with
2332 zero one can find out whether it is the first call and therefore the
2333 prolog should be emitted or not. This element must never be modified.
2335 @item int internal_use
2336 This element is another one rarely used but needed in certain
2337 situations. It got assigned a nonzero value in case the conversion
2338 functions are used to implement @code{mbsrtowcs} et.al. I.e., the
2339 function is not used directly through the @code{iconv} interface.
2341 This sometimes makes a difference as it is expected that the
2342 @code{iconv} functions are used to translate entire texts while the
2343 @code{mbsrtowcs} functions are normally only used to convert single
2344 strings and might be used multiple times to convert entire texts.
2346 But in this situation we would have problem complying with some rules of
2347 the character set specification. Some character sets require a prolog
2348 which must appear exactly once for an entire text. If a number of
2349 @code{mbsrtowcs} calls are used to convert the text only the first call
2350 must add the prolog. But since there is no communication between the
2351 different calls of @code{mbsrtowcs} the conversion functions have no
2352 possibility to find this out. The situation is different for sequences
2353 of @code{iconv} calls since the handle allows to access the needed
2356 This element is mostly used together with @code{invocation_counter} in a
2360 if (!data->internal_use && data->invocation_counter == 0)
2361 /* @r{Emit prolog.} */
2365 This element must never be modified.
2367 @item mbstate_t *statep
2368 The @code{statep} element points to an object of type @code{mbstate_t}
2369 (@pxref{Keeping the state}). The conversion of an stateful character
2370 set must use the object pointed to by this element to store information
2371 about the conversion state. The @code{statep} element itself must never
2374 @item mbstate_t __state
2375 This element @emph{never} must be used directly. It is only part of
2376 this structure to have the needed space allocated.
2380 @subsubsection @code{iconv} module interfaces
2382 With the knowledge about the data structures we now can describe the
2383 conversion functions itself. To understand the interface a bit of
2384 knowledge about the functionality in the C library which loads the
2385 objects with the conversions is necessary.
2387 It is often the case that one conversion is used more than once. I.e.,
2388 there are several @code{iconv_open} calls for the same set of character
2389 sets during one program run. The @code{mbsrtowcs} et.al.@: functions in
2390 the GNU C library also use the @code{iconv} functionality which
2391 increases the number of uses of the same functions even more.
2393 For this reason the modules do not get loaded exclusively for one
2394 conversion. Instead a module once loaded can be used by arbitrary many
2395 @code{iconv} or @code{mbsrtowcs} calls at the same time. The splitting
2396 of the information between conversion function specific information and
2397 conversion data makes this possible. The last section showed the two
2398 data structure used to do this.
2400 This is of course also reflected in the interface and semantic of the
2401 functions the modules must provide. There are three functions which
2402 must have the following names:
2406 The @code{gconv_init} function initializes the conversion function
2407 specific data structure. This very same object is shared by all
2408 conversion which use this conversion and therefore no state information
2409 about the conversion itself must be stored in here. If a module
2410 implements more than one conversion the @code{gconv_init} function will be
2411 called multiple times.
2414 The @code{gconv_end} function is responsible to free all resources
2415 allocated by the @code{gconv_init} function. If there is nothing to do
2416 this function can be missing. Special care must be taken if the module
2417 implements more than one conversion and the @code{gconv_init} function
2418 does not allocate the same resources for all conversions.
2421 This is the actual conversion function. It is called to convert one
2422 block of text. It gets passed the conversion step information
2423 initialized by @code{gconv_init} and the conversion data, specific to
2424 this use of the conversion functions.
2427 There are three data types defined for the three module interface
2428 function and these define the interface.
2432 @deftypevr {Data type} int (*gconv_init_fct) (struct gconv_step *)
2433 This specifies the interface of the initialization function of the
2434 module. It is called exactly once for each conversion the module
2437 As explained int the description of the @code{struct gconv_step} data
2438 structure above the initialization function has to initialize parts of
2442 @item min_needed_from
2443 @itemx max_needed_from
2444 @itemx min_needed_to
2445 @itemx max_needed_to
2446 These elements must be initialized to the exact numbers of the minimum
2447 and maximum number of bytes used by one character in the source and
2448 destination character set respectively. If the characters all have the
2449 same size the minimum and maximum values are the same.
2452 This element must be initialized to an nonzero value if the source
2453 character set is stateful. Otherwise it must be zero.
2456 If the initialization function needs to communication some information
2457 to the conversion function this can happen using the @code{data} element
2458 of the @code{gconv_step} structure. But since this data is shared by
2459 all the conversion is must not be modified by the conversion function.
2460 How this can be used is shown in the example below.
2463 #define MIN_NEEDED_FROM 1
2464 #define MAX_NEEDED_FROM 4
2465 #define MIN_NEEDED_TO 4
2466 #define MAX_NEEDED_TO 4
2469 gconv_init (struct gconv_step *step)
2471 /* @r{Determine which direction.} */
2472 struct iso2022jp_data *new_data;
2473 enum direction dir = illegal_dir;
2474 enum variant var = illegal_var;
2477 if (__strcasecmp (step->from_name, "ISO-2022-JP//") == 0)
2479 dir = from_iso2022jp;
2482 else if (__strcasecmp (step->to_name, "ISO-2022-JP//") == 0)
2487 else if (__strcasecmp (step->from_name, "ISO-2022-JP-2//") == 0)
2489 dir = from_iso2022jp;
2492 else if (__strcasecmp (step->to_name, "ISO-2022-JP-2//") == 0)
2498 result = GCONV_NOCONV;
2499 if (dir != illegal_dir)
2501 new_data = (struct iso2022jp_data *)
2502 malloc (sizeof (struct iso2022jp_data));
2504 result = GCONV_NOMEM;
2505 if (new_data != NULL)
2507 new_data->dir = dir;
2508 new_data->var = var;
2509 step->data = new_data;
2511 if (dir == from_iso2022jp)
2513 step->min_needed_from = MIN_NEEDED_FROM;
2514 step->max_needed_from = MAX_NEEDED_FROM;
2515 step->min_needed_to = MIN_NEEDED_TO;
2516 step->max_needed_to = MAX_NEEDED_TO;
2520 step->min_needed_from = MIN_NEEDED_TO;
2521 step->max_needed_from = MAX_NEEDED_TO;
2522 step->min_needed_to = MIN_NEEDED_FROM;
2523 step->max_needed_to = MAX_NEEDED_FROM + 2;
2526 /* @r{Yes, this is a stateful encoding.} */
2537 The function first checks which conversion is wanted. The module from
2538 which this function is taken implements four different conversion and
2539 which one is selected can be determined by comparing the names. The
2540 comparison should always be done without paying attention to the case.
2542 Then a data structure is allocated which contains the necessary
2543 information about which conversion is selected. The data structure
2544 @code{struct iso2022jp_data} is locally defined since outside the module
2545 this data is not used at all. Please note that if all four conversions
2546 this modules supports are requested there are four data blocks.
2548 One interesting thing is the initialization of the @code{min_} and
2549 @code{max_} elements of the step data object. A single ISO-2022-JP
2550 character can consist of one to four bytes. Therefore the
2551 @code{MIN_NEEDED_FROM} and @code{MAX_NEEDED_FROM} macros are defined
2552 this way. The output is always the @code{INTERNAL} character set (aka
2553 UCS4) and therefore each character consists of exactly four bytes. For
2554 the conversion from @code{INTERNAL} to ISO-2022-JP we have to take into
2555 account that escape sequences might be necessary to switch the character
2556 sets. Therefore the @code{max_needed_to} element for this direction
2557 gets assigned @code{MAX_NEEDED_FROM + 2}. This takes into account the
2558 two bytes needed for the escape sequences to single the switching. The
2559 asymmetry in the maximum values for the two directions can be explained
2560 easily: when reading ISO-2022-JP text escape sequences can be handled
2561 alone. I.e., it is not necessary to process a real character since the
2562 effect of the escape sequence can be recorded in the state information.
2563 The situation is different for the other direction. Since it is in
2564 general not known which character comes next one cannot emit escape
2565 sequences to change the state in advance. This means the escape
2566 sequences which have to be emitted together with the next character.
2567 Therefore one needs more room then only for the character itself.
2569 The possible return values of the initialization function are:
2573 The initialization succeeded
2575 The requested conversion is not supported in the module. This can
2576 happen if the @file{gconv-modules} file has errors.
2578 Memory required to store additional information could not be allocated.
2582 The functions called before the module is unloaded is significantly
2583 easier. It often has nothing at all to do in which case it can be left
2588 @deftypevr {Data type} void (*gconv_end_fct) (struct gconv_step *)
2589 The task of this function is it to free all resources allocated in the
2590 initialization function. Therefore only the @code{data} element of the
2591 object pointed to by the argument is of interest. Continuing the
2592 example from the initialization function, the finalization function
2597 gconv_end (struct gconv_step *data)
2604 The most important function is the conversion function itself. It can
2605 get quite complicated for complex character sets. But since this is not
2606 of interest here we will only describe a possible skeleton for the
2607 conversion function.
2611 @deftypevr {Data type} int (*gconv_fct) (struct gconv_step *, struct gconv_step_data *, const char **, const char *, size_t *, int)
2612 The conversion function can be called for two basic reason: to convert
2613 text or to reset the state. From the description of the @code{iconv}
2614 function it can be seen why the flushing mode is necessary. What mode
2615 is selected is determined by the sixth argument, an integer. If it is
2616 nonzero it means that flushing is selected.
2618 Common to both mode is where the output buffer can be found. The
2619 information about this buffer is stored in the conversion step data. A
2620 pointer to this is passed as the second argument to this function. The
2621 description of the @code{struct gconv_step_data} structure has more
2622 information on this.
2625 What has to be done for flushing depends on the source character set.
2626 If it is not stateful nothing has to be done. Otherwise the function
2627 has to emit a byte sequence to bring the state object in the initial
2628 state. Once this all happened the other conversion modules in the chain
2629 of conversions have to get the same chance. Whether another step
2630 follows can be determined from the @code{is_last} element of the step
2631 data structure to which the first parameter points.
2633 The more interesting mode is when actually text has to be converted.
2634 The first step in this case is to convert as much text as possible from
2635 the input buffer and store the result in the output buffer. The start
2636 of the input buffer is determined by the third argument which is a
2637 pointer to a pointer variable referencing the beginning of the buffer.
2638 The fourth argument is a pointer to the byte right after the last byte
2641 The conversion has to be performed according to the current state if the
2642 character set is stateful. The state is stored in an object pointed to
2643 by the @code{statep} element of the step data (second argument). Once
2644 either the input buffer is empty or the output buffer is full the
2645 conversion stops. At this point the pointer variable referenced by the
2646 third parameter must point to the byte following the last processed
2647 byte. I.e., if all of the input is consumed this pointer and the fourth
2648 parameter have the same value.
2650 What now happens depends on whether this step is the last one or not.
2651 If it is the last step the only thing which has to be done is to update
2652 the @code{outbuf} element of the step data structure to point after the
2653 last written byte. This gives the caller the information on how much
2654 text is available in the output buffer. Beside this the variable
2655 pointed to by the fifth parameter, which is of type @code{size_t}, must
2656 be incremented by the number of characters (@emph{not bytes}) which were
2657 written in the output buffer. Then the function can return.
2659 In case the step is not the last one the later conversion functions have
2660 to get a chance to do their work. Therefore the appropriate conversion
2661 function has to be called. The information about the functions is
2662 stored in the conversion data structures, passed as the first parameter.
2663 This information and the step data are stored in arrays so the next
2664 element in both cases can be found by simple pointer arithmetic:
2668 gconv (struct gconv_step *step, struct gconv_step_data *data,
2669 const char **inbuf, const char *inbufend, size_t *written,
2672 struct gconv_step *next_step = step + 1;
2673 struct gconv_step_data *next_data = data + 1;
2677 The @code{next_step} pointer references the next step information and
2678 @code{next_data} the next data record. The call of the next function
2679 therefore will look similar to this:
2682 next_step->fct (next_step, next_data, &outerr, outbuf, written, 0)
2685 But this is not yet all. Once the function call returns the conversion
2686 function might have some more to do. If the return value of the
2687 function is @code{GCONV_EMPTY_INPUT} this means there is more room in
2688 the output buffer. Unless the input buffer is empty the conversion
2689 functions start all over again and processes the rest of the input
2690 buffer. If the return value is not @code{GCONV_EMPTY_INPUT} something
2691 went wrong and we have to recover from this.
2693 A requirement for the conversion function is that the input buffer
2694 pointer (the third argument) always points to the last character which
2695 was put in the converted form in the output buffer. This is trivial
2696 true after the conversion performed in the current step. But if the
2697 conversion functions deeper down the stream stop prematurely not all
2698 characters from the output buffer are consumed and therefore the input
2699 buffer pointers must be backed of to the right position.
2701 This is easy to do if the input and output character sets have a fixed
2702 width for all characters. In this situation we can compute how many
2703 characters are left in the output buffer and therefore can correct the
2704 input buffer pointer appropriate with a similar computation. Things are
2705 getting tricky if either character set has character represented with
2706 variable length byte sequences and it gets even more complicated if the
2707 conversion has to take care of the state. In these cases the conversion
2708 has to be performed once again, from the known state before the initial
2709 conversion. I.e., if necessary the state of the conversion has to be
2710 reset and the conversion loop has to be executed again. The difference
2711 now is that it is known how much input must be created and the
2712 conversion can stop before converting the first unused character. Once
2713 this is done the input buffer pointers must be updated again and the
2714 function can return.
2716 One final thing should be mentioned. If it is necessary for the
2717 conversion to know whether it is the first invocation (in case a prolog
2718 has to be emitted) the conversion function should just before returning
2719 to the caller increment the @code{invocation_counter} element of the
2720 step data structure. See the description of the @code{struct
2721 gconv_step_data} structure above for more information on how this can be
2724 The return value must be one of the following values:
2727 @item GCONV_EMPTY_INPUT
2728 All input was consumed and there is room left in the output buffer.
2729 @item GCONV_OUTPUT_FULL
2730 No more room in the output buffer. In case this is not the last step
2731 this value is propagated down from the call of the next conversion
2732 function in the chain.
2733 @item GCONV_INCOMPLETE_INPUT
2734 The input buffer is not entirely empty since it contains an incomplete
2738 The following example provides a framework for a conversion function.
2739 In case a new conversion has to be written the holes in this
2740 implementation have to be filled and that is it.
2744 gconv (struct gconv_step *step, struct gconv_step_data *data,
2745 const char **inbuf, const char *inbufend, size_t *written,
2748 struct gconv_step *next_step = step + 1;
2749 struct gconv_step_data *next_data = data + 1;
2750 gconv_fct fct = next_step->fct;
2753 /* @r{If the function is called with no input this means we have}
2754 @r{to reset to the initial state. The possibly partly}
2755 @r{converted input is dropped.} */
2760 /* @r{Possible emit a byte sequence which put the state object}
2761 @r{into the initial state.} */
2763 /* @r{Call the steps down the chain if there are any but only}
2764 @r{if we successfully emitted the escape sequence.} */
2765 if (status == GCONV_OK && ! data->is_last)
2766 status = fct (next_step, next_data, NULL, NULL,
2771 /* @r{We preserve the initial values of the pointer variables.} */
2772 const char *inptr = *inbuf;
2773 char *outbuf = data->outbuf;
2774 char *outend = data->outbufend;
2777 /* @r{This variable is used to count the number of characters}
2778 @r{we actually converted.} */
2779 size_t converted = 0;
2783 /* @r{Remember the start value for this round.} */
2785 /* @r{The outbuf buffer is empty.} */
2788 /* @r{For stateful encodings the state must be safe here.} */
2790 /* @r{Run the conversion loop. @code{status} is set}
2791 @r{appropriately afterwards.} */
2793 /* @r{If this is the last step leave the loop, there is}
2794 @r{nothing we can do.} */
2797 /* @r{Store information about how many bytes are}
2799 data->outbuf = outbuf;
2801 /* @r{Remember how many characters we converted.} */
2802 *written += converted;
2807 /* @r{Write out all output which was produced.} */
2808 if (outbuf > outptr)
2810 const char *outerr = data->outbuf;
2813 result = fct (next_step, next_data, &outerr,
2814 outbuf, written, 0);
2816 if (result != GCONV_EMPTY_INPUT)
2818 if (outerr != outbuf)
2820 /* @r{Reset the input buffer pointer. We}
2821 @r{document here the complex case.} */
2824 /* @r{Reload the pointers.} */
2828 /* @r{Possibly reset the state.} */
2830 /* @r{Redo the conversion, but this time}
2831 @r{the end of the output buffer is at}
2832 @r{@code{outerr}.} */
2835 /* @r{Change the status.} */
2839 /* @r{All the output is consumed, we can make}
2840 @r{ another run if everything was ok.} */
2841 if (status == GCONV_FULL_OUTPUT)
2845 while (status == GCONV_OK);
2847 /* @r{We finished one use of this step.} */
2848 ++data->invocation_counter;
2856 This information should be sufficient to write new modules. Anybody
2857 doing so should also take a look at the available source code in the GNU
2858 C library sources. It contains many examples of working and optimized