1 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html401/loose.dtd">
3 <!-- Created on October, 16 2022 by texi2html 1.78a -->
5 Written by: Lionel Cons <Lionel.Cons@cern.ch> (original author)
6 Karl Berry <karl@freefriends.org>
7 Olaf Bachmann <obachman@mathematik.uni-kl.de>
9 Maintained by: Many creative people.
10 Send bugs and suggestions to <texi2html-bug@nongnu.org>
14 <title>GNU libunistring: 1. Introduction</title>
16 <meta name="description" content="GNU libunistring: 1. Introduction">
17 <meta name="keywords" content="GNU libunistring: 1. Introduction">
18 <meta name="resource-type" content="document">
19 <meta name="distribution" content="global">
20 <meta name="Generator" content="texi2html 1.78a">
21 <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
22 <style type="text/css">
24 a.summary-letter {text-decoration: none}
25 pre.display {font-family: serif}
26 pre.format {font-family: serif}
27 pre.menu-comment {font-family: serif}
28 pre.menu-preformatted {font-family: serif}
29 pre.smalldisplay {font-family: serif; font-size: smaller}
30 pre.smallexample {font-size: smaller}
31 pre.smallformat {font-family: serif; font-size: smaller}
32 pre.smalllisp {font-size: smaller}
33 span.roman {font-family:serif; font-weight:normal;}
34 span.sansserif {font-family:sans-serif; font-weight:normal;}
35 ul.toc {list-style: none}
42 <body lang="en" bgcolor="#FFFFFF" text="#000000" link="#0000FF" vlink="#800080" alink="#FF0000">
44 <table cellpadding="1" cellspacing="1" border="0">
45 <tr><td valign="middle" align="left">[ << ]</td>
46 <td valign="middle" align="left">[<a href="libunistring_2.html#SEC8" title="Next chapter"> >> </a>]</td>
47 <td valign="middle" align="left"> </td>
48 <td valign="middle" align="left"> </td>
49 <td valign="middle" align="left"> </td>
50 <td valign="middle" align="left"> </td>
51 <td valign="middle" align="left"> </td>
52 <td valign="middle" align="left">[<a href="libunistring_toc.html#SEC_Top" title="Cover (top) of document">Top</a>]</td>
53 <td valign="middle" align="left">[<a href="libunistring_toc.html#SEC_Contents" title="Table of contents">Contents</a>]</td>
54 <td valign="middle" align="left">[<a href="libunistring_21.html#SEC92" title="Index">Index</a>]</td>
55 <td valign="middle" align="left">[<a href="libunistring_abt.html#SEC_About" title="About (help)"> ? </a>]</td>
59 <a name="Introduction"></a>
61 <h1 class="chapter"> <a href="libunistring_toc.html#TOC1">1. Introduction</a> </h1>
63 <p>This library provides functions for manipulating Unicode strings and
64 for manipulating C strings according to the Unicode standard.
66 <p>It consists of the following parts:
68 <dl compact="compact">
69 <dt> <code><unistr.h></code></dt>
70 <dd><p>elementary string functions
72 <dt> <code><uniconv.h></code></dt>
73 <dd><p>conversion from/to legacy encodings
75 <dt> <code><unistdio.h></code></dt>
76 <dd><p>formatted output to strings
78 <dt> <code><uniname.h></code></dt>
79 <dd><p>character names
81 <dt> <code><unictype.h></code></dt>
82 <dd><p>character classification and properties
84 <dt> <code><uniwidth.h></code></dt>
85 <dd><p>string width when using nonproportional fonts
87 <dt> <code><unigbrk.h></code></dt>
88 <dd><p>grapheme cluster breaks
90 <dt> <code><uniwbrk.h></code></dt>
93 <dt> <code><unilbrk.h></code></dt>
94 <dd><p>line breaking algorithm
96 <dt> <code><uninorm.h></code></dt>
97 <dd><p>normalization (composition and decomposition)
99 <dt> <code><unicase.h></code></dt>
102 <dt> <code><uniregex.h></code></dt>
103 <dd><p>regular expressions (not yet implemented)
109 <p>libunistring is for you if your application involves non-trivial text
110 processing, such as upper/lower case conversions, line breaking, operations
111 on words, or more advanced analysis of text. Text provided by the user can,
112 in general, contain characters of all kinds of scripts. The text processing
113 functions provided by this library handle all scripts and all languages.
115 <p>libunistring is for you if your application already uses the ISO C / POSIX
116 <a href="http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/ctype.h.html"><code><ctype.h></code></a>, <a href="http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/wctype.h.html"><code><wctype.h></code></a> functions and the text it
117 operates on is provided by the user and can be in any language.
119 <p>libunistring is also for you if your application uses Unicode strings as
120 internal in-memory representation.
124 <a name="Unicode"></a>
126 <h2 class="section"> <a href="libunistring_toc.html#TOC2">1.1 Unicode</a> </h2>
128 <p>Unicode is a standardized repertoire of characters that contains characters
129 from all scripts of the world, from Latin letters to Chinese ideographs
130 and Babylonian cuneiform glyphs. It also specifies how these characters
131 are to be rendered on a screen or on paper, and how common text processing
132 (word selection, line breaking, uppercasing of page titles etc.) is supposed
133 to behave on Unicode text.
135 <p>Unicode also specifies three ways of storing sequences of Unicode
136 characters in a computer whose basic unit of data is an 8-bit byte:
141 </p><dl compact="compact">
143 <dd><p>Every character is represented as 1 to 4 bytes.
146 <dd><p>Every character is represented as 1 to 2 units of 16 bits.
148 <dt> UTF-32, a.k.a. UCS-4</dt>
149 <dd><p>Every character is represented as 1 unit of 32 bits.
153 <p>For encoding Unicode text in a file, UTF-8 is usually used. For encoding
154 Unicode strings in memory for a program, either of the three encoding forms
155 can be reasonably used.
157 <p>Unicode is widely used on the web. Prior to the use of Unicode, web pages
158 were in many different encodings (ISO-8859-1 for English, French, Spanish,
159 ISO-8859-2 for Polish, ISO-8859-7 for Greek, KOI8-R for Russian, GB2312 or
160 BIG5 for Chinese, ISO-2022-JP-2 or EUC-JP or Shift_JIS for Japanese, and many
161 many others). It was next to impossible to create a document that contained
162 Chinese and Polish text in the same document. Due to the many encodings for
163 Japanese, even the processing of pure Japanese text was error prone.
168 The Unicode standard: <a href="https://www.unicode.org/">https://www.unicode.org/</a>
170 Definition of UTF-8: <a href="https://www.rfc-editor.org/rfc/rfc3629.txt">https://www.rfc-editor.org/rfc/rfc3629.txt</a>
172 Definition of UTF-16: <a href="https://www.rfc-editor.org/rfc/rfc2781.txt">https://www.rfc-editor.org/rfc/rfc2781.txt</a>
174 Markus Kuhn's UTF-8 and Unicode FAQ:
175 <a href="https://www.cl.cam.ac.uk/~mgk25/unicode.html">https://www.cl.cam.ac.uk/~mgk25/unicode.html</a>
179 <a name="Unicode-and-i18n"></a>
181 <h2 class="section"> <a href="libunistring_toc.html#TOC3">1.2 Unicode and Internationalization</a> </h2>
183 <p>Internationalization is the process of changing the source code of a program
184 so that it can meet the expectations of users in any culture, if culture
185 specific data (translations, images etc.) are provided.
187 <p>Use of Unicode is not strictly required for internationalization, but it
188 makes internationalization much easier, because operations that need to
189 look at specific characters (like hyphenation, spell checking, or the
190 automatic conversion of double-quotes to opening and closing double-quote
191 characters) don't need to consider multiple possible encodings of the text.
193 <p>Use of Unicode also enables multilingualization: the ability of having text
194 in multiple languages present in the same document or even in the same line
197 <p>But use of Unicode is not everything. Internationalization usually consists
201 Use of Unicode where needed for text processing. This is what this library
204 Use of message catalogs for messages shown to the user, This is what
205 GNU gettext is about.
207 Use of locale specific conventions for date and time formats, for numeric
208 formatting, or for sorting of text. This can be done adequately with the
209 POSIX APIs and the implementation of locales in the GNU C library.
211 In graphical user interfaces, adapting the GUI to the default text direction
212 of the current locale (see
213 <a href="https://en.wikipedia.org/wiki/Right-to-left">right-to-left languages</a>).
217 <a name="Locale-encodings"></a>
219 <h2 class="section"> <a href="libunistring_toc.html#TOC4">1.3 Locale encodings</a> </h2>
221 <p>A locale is a set of cultural conventions. According to POSIX, for a program,
222 at any moment, there is one locale being designated as the “current locale”.
223 (Actually, POSIX supports also one locale per thread, but this feature is not
224 yet universally implemented and not widely used.)
226 The locale is partitioned into several aspects, called the “categories”
227 of the locale. The main various aspects are:
230 The character encoding and the character properties. This is the
231 <code>LC_CTYPE</code> category.
233 The sorting rules for text. This is the <code>LC_COLLATE</code> category.
235 The language specific translations of messages. This is the
236 <code>LC_MESSAGES</code> category.
238 The formatting rules for numbers, such as the decimal separator. This is
239 the <code>LC_NUMERIC</code> category.
241 The formatting rules for amounts of money. This is the <code>LC_MONETARY</code>
244 The formatting of date and time. This is the <code>LC_TIME</code> category.
248 <p>In particular, the <code>LC_CTYPE</code> category of the current locale determines
249 the character encoding. This is the encoding of ‘<samp>char *</samp>’ strings.
250 We also call it the “locale encoding”. GNU libunistring has a function,
251 <code>locale_charset</code>, that returns a standardized (platform independent)
252 name for this encoding.
254 <p>All locale encodings used on glibc systems are essentially ASCII compatible:
255 Most graphic ASCII characters have the same representation, as a single byte,
256 in that encoding as in ASCII.
258 <p>Among the possible locale encodings are UTF-8 and GB18030. Both allow
259 to represent any Unicode character as a sequence of bytes. UTF-8 is used in
260 most of the world, whereas GB18030 is used in the People's Republic of China,
261 because it is backward compatible with the GB2312 encoding that was used in
262 this country earlier.
264 <p>The legacy locale encodings, ISO-8859-15 (which supplanted ISO-8859-1 in
265 most of Europe), ISO-8859-2, KOI8-R, EUC-JP, etc., are still in use in
268 <p>UTF-16 and UTF-32 are not used as locale encodings, because they are not
272 <a name="In_002dmemory-representation"></a>
274 <h2 class="section"> <a href="libunistring_toc.html#TOC5">1.4 Choice of in-memory representation of strings</a> </h2>
276 <p>There are three ways of representing strings in memory of a running
280 As ‘<samp>char *</samp>’ strings. Such strings are represented in locale encoding.
281 This approach is employed when not much text processing is done by the
282 program. When some Unicode aware processing is to be done, a string is
283 converted to Unicode on the fly and back to locale encoding afterwards.
285 As UTF-8 or UTF-16 or UTF-32 strings. This implies that conversion from
286 locale encoding to Unicode is performed on input, and in the opposite
287 direction on output. This approach is employed when the program does
288 a significant amount of text processing, or when the program has multiple
289 threads operating on the same data but in different locales.
291 As ‘<samp>wchar_t *</samp>’, a.k.a. “wide strings”. This approach is misguided,
292 see <a href="libunistring_18.html#SEC81">The <code>wchar_t</code> mess</a>.
295 <p>Of course, a ‘<samp>char *</samp>’ string can, in some cases, be encoded in UTF-8.
296 You will use the data type depending on what you can guarantee about how
297 it's encoded: If a string is encoded in the locale encoding, or if you
298 don't know how it's encoded, use ‘<samp>char *</samp>’. If, on the other hand,
299 you can <em>guarantee</em> that it is UTF-8 encoded, then you can use the
300 UTF-8 string type, <code>uint8_t *</code>, for it.
302 <p>The five types <code>char *</code>, <code>uint8_t *</code>, <code>uint16_t *</code>,
303 <code>uint32_t *</code>, and <code>wchar_t *</code> are incompatible types at the C
304 level. Therefore, ‘<samp>gcc -Wall</samp>’ will produce a warning if, by mistake,
305 your code contains a mismatch between these types. In the context of
306 using GNU libunistring, even a warning about a mismatch between
307 <code>char *</code> and <code>uint8_t *</code> is a sign of a bug in your code
308 that you should not try to silence through a cast.
311 <a name="char-_002a-strings"></a>
313 <h2 class="section"> <a href="libunistring_toc.html#TOC6">1.5 ‘<samp>char *</samp>’ strings</a> </h2>
315 <p>The classical C strings, with its C library support standardized by
316 ISO C and POSIX, can be used in internationalized programs with some
317 precautions. The problem with this API is that many of the C library
318 functions for strings don't work correctly on strings in locale
319 encodings, leading to bugs that only people in some cultures of the
320 world will experience.
323 <p>The first problem with the C library API is the support of multibyte
324 locales. According to the locale encoding, in general, every character
325 is represented by one or more bytes (up to 4 bytes in practice — but
326 use <code>MB_LEN_MAX</code> instead of the number 4 in the code).
327 When every character is represented by only 1 byte, we speak of an
328 “unibyte locale”, otherwise of a “multibyte locale”. It is important
329 to realize that the majority of Unix installations nowadays use UTF-8
330 or GB18030 as locale encoding; therefore, the majority of users are
331 using multibyte locales.
334 <p>The important fact to remember is:
335 </p><table class="cartouche" border="1"><tr><td>
336 <p><em>A ‘<samp>char</samp>’ is a byte, not a character.</em>
337 </p></td></tr></table>
342 The <a href="http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/ctype.h.html"><code><ctype.h></code></a> API is useless in this context; it does not work in
345 The <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strlen.html"><code>strlen</code></a> function does not return the number of characters
346 in a string. Nor does it return the number of screen columns occupied
347 by a string after it is output. It merely returns the number of
348 <em>bytes</em> occupied by a string.
350 Truncating a string, for example, with <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strncpy.html"><code>strncpy</code></a>, can have the
351 effect of truncating it in the middle of a multibyte character. Such
352 a string will, when output, have a garbled character at its end, often
353 represented by a hollow box.
355 <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strchr.html"><code>strchr</code></a> and <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strrchr.html"><code>strrchr</code></a> do not work with multibyte strings
356 if the locale encoding is GB18030 and the character to be searched is
359 <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strstr.html"><code>strstr</code></a> does not work with multibyte strings if the locale encoding
360 is different from UTF-8.
362 <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strcspn.html"><code>strcspn</code></a>, <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strpbrk.html"><code>strpbrk</code></a>, <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strspn.html"><code>strspn</code></a> cannot work
363 correctly in multibyte locales: they assume the second argument is a list of
364 single-byte characters. Even in this simple case, they do not work with
365 multibyte strings if the locale encoding is GB18030 and one of the
366 characters to be searched is a digit.
368 <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strsep.html"><code>strsep</code></a> and <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strtok_r.html"><code>strtok_r</code></a> do not work with multibyte strings
369 unless all of the delimiter characters are ASCII characters < 0x30.
371 The <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strcasecmp.html"><code>strcasecmp</code></a>, <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strncasecmp.html"><code>strncasecmp</code></a>, and <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strcasestr.html"><code>strcasestr</code></a>
372 functions do not work with multibyte strings.
375 <p>The workarounds can be found in GNU gnulib
376 <a href="https://www.gnu.org/software/gnulib/">https://www.gnu.org/software/gnulib/</a>.
379 gnulib has modules ‘<samp>mbchar</samp>’, ‘<samp>mbiter</samp>’, ‘<samp>mbuiter</samp>’ that
380 represent multibyte characters and allow to iterate across a multibyte
381 string with the same ease as through a unibyte string.
383 gnulib has functions <code>mbslen</code> and <code>mbswidth</code> that can be
384 used instead of <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strlen.html"><code>strlen</code></a> when the number of characters or the
385 number of screen columns of a string is requested.
387 gnulib has functions <code>mbschr</code> and <code>mbsrrchr</code> that are
388 like <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strchr.html"><code>strchr</code></a> and <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strrchr.html"><code>strrchr</code></a>, but work in multibyte locales.
390 gnulib has a function <code>mbsstr</code>, like <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strstr.html"><code>strstr</code></a>, but works
391 in multibyte locales.
393 gnulib has functions <code>mbscspn</code>, <code>mbspbrk</code>, <code>mbsspn</code>
394 that are like <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strcspn.html"><code>strcspn</code></a>, <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strpbrk.html"><code>strpbrk</code></a>, <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strspn.html"><code>strspn</code></a>, but
395 work in multibyte locales.
397 gnulib has functions <code>mbssep</code> and <code>mbstok_r</code> that are
398 like <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strsep.html"><code>strsep</code></a> and <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strtok_r.html"><code>strtok_r</code></a> but work in multibyte locales.
400 gnulib has functions <code>mbscasecmp</code>, <code>mbsncasecmp</code>,
401 <code>mbspcasecmp</code>, and <code>mbscasestr</code> that are like <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strcasecmp.html"><code>strcasecmp</code></a>,
402 <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strncasecmp.html"><code>strncasecmp</code></a>, and <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/strcasestr.html"><code>strcasestr</code></a>, but
403 work in multibyte locales. Still, the function <code>ulc_casecmp</code> is
404 preferable to these functions; see below.
407 <p>The second problem with the C library API is that it has some assumptions built-in that are not valid in some languages:
410 It assumes that there are only two forms of every character: uppercase
411 and lowercase. This is not true for Croatian, where the character
412 <small>LETTER DZ WITH CARON</small> comes in three forms:
413 <small>LATIN CAPITAL LETTER DZ WITH CARON</small> (DZ),
414 <small>LATIN CAPITAL LETTER D WITH SMALL LETTER Z WITH CARON</small> (Dz),
415 <small>LATIN SMALL LETTER DZ WITH CARON</small> (dz).
417 It assumes that uppercasing of 1 character leads to 1 character. This
418 is not true for German, where the <small>LATIN SMALL LETTER SHARP S</small>, when
419 uppercased, becomes ‘<samp>SS</samp>’.
421 It assumes that there is 1:1 mapping between uppercase and lowercase forms.
422 This is not true for the Greek sigma: <small>GREEK CAPITAL LETTER SIGMA</small> is
423 the uppercase of both <small>GREEK SMALL LETTER SIGMA</small> and
424 <small>GREEK SMALL LETTER FINAL SIGMA</small>.
426 It assumes that the upper/lowercase mappings are position independent.
427 This is not true for the Greek sigma and the Lithuanian i.
430 <p>The correct way to deal with this problem is
433 to provide functions for titlecasing, as well as for upper- and
436 to view case transformations as functions that operates on strings,
437 rather than on characters.
440 <p>This is implemented in this library, through the functions declared in <code><unicase.h></code>, see <a href="libunistring_14.html#SEC67">Case mappings <code><unicase.h></code></a>.
443 <a name="Unicode-strings"></a>
445 <h2 class="section"> <a href="libunistring_toc.html#TOC7">1.6 Unicode strings</a> </h2>
447 <p>libunistring supports Unicode strings in three representations:
453 UTF-8 strings, through the type ‘<samp>uint8_t *</samp>’. The units are bytes
454 (<code>uint8_t</code>).
456 UTF-16 strings, through the type ‘<samp>uint16_t *</samp>’, The units are 16-bit
457 memory words (<code>uint16_t</code>).
459 UTF-32 strings, through the type ‘<samp>uint32_t *</samp>’. The units are 32-bit
460 memory words (<code>uint32_t</code>).
463 <p>As with C strings, there are two variants:
466 Unicode strings with a terminating NUL character are represented as
467 a pointer to the first unit of the string. There is a unit containing
468 a 0 value at the end. It is considered part of the string for all
469 memory allocation purposes, but is not considered part of the string
470 for all other logical purposes.
472 Unicode strings where embedded NUL characters are allowed. These
473 are represented by a pointer to the first unit and the number of units
474 (not bytes!) of the string. In this setting, there is no trailing
475 zero-valued unit used as “end marker”.
479 <table cellpadding="1" cellspacing="1" border="0">
480 <tr><td valign="middle" align="left">[<a href="#SEC1" title="Beginning of this chapter or previous chapter"> << </a>]</td>
481 <td valign="middle" align="left">[<a href="libunistring_2.html#SEC8" title="Next chapter"> >> </a>]</td>
482 <td valign="middle" align="left"> </td>
483 <td valign="middle" align="left"> </td>
484 <td valign="middle" align="left"> </td>
485 <td valign="middle" align="left"> </td>
486 <td valign="middle" align="left"> </td>
487 <td valign="middle" align="left">[<a href="libunistring_toc.html#SEC_Top" title="Cover (top) of document">Top</a>]</td>
488 <td valign="middle" align="left">[<a href="libunistring_toc.html#SEC_Contents" title="Table of contents">Contents</a>]</td>
489 <td valign="middle" align="left">[<a href="libunistring_21.html#SEC92" title="Index">Index</a>]</td>
490 <td valign="middle" align="left">[<a href="libunistring_abt.html#SEC_About" title="About (help)"> ? </a>]</td>
494 This document was generated by <em>Bruno Haible</em> on <em>October, 16 2022</em> using <a href="https://www.nongnu.org/texi2html/"><em>texi2html 1.78a</em></a>.
projects / platform / upstream / libunistring.git / blob