+<?xml version="1.0"?>
+<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.3//EN"
+ "http://www.oasis-open.org/docbook/xml/4.3/docbookx.dtd" [
+ <!ENTITY % local.common.attrib "xmlns:xi CDATA #FIXED 'http://www.w3.org/2003/XInclude'">
+ <!ENTITY version SYSTEM "version.xml">
+]>
<chapter id="buffers-language-script-and-direction">
<title>Buffers, language, script and direction</title>
<para>
- The input to Harfbuzz is a series of Unicode characters, stored in a
+ The input to the HarfBuzz shaper is a series of Unicode characters, stored in a
buffer. In this chapter, we'll look at how to set up a buffer with
- the text that we want and then customize the properties of the
- buffer.
+ the text that we want and how to customize the properties of the
+ buffer. We'll also look at a piece of lower-level machinery that
+ you will need to understand before proceeding: the functions that
+ HarfBuzz uses to retrieve Unicode information.
+ </para>
+ <para>
+ After shaping is complete, HarfBuzz puts its output back
+ into the buffer. But getting that output requires setting up a
+ face and a font first, so we will look at that in the next chapter
+ instead of here.
</para>
<section id="creating-and-destroying-buffers">
<title>Creating and destroying buffers</title>
<para>
- As we saw in our initial example, a buffer is created and
- initialized with <literal>hb_buffer_create()</literal>. This
+ As we saw in our <emphasis>Getting Started</emphasis> example, a
+ buffer is created and
+ initialized with <function>hb_buffer_create()</function>. This
produces a new, empty buffer object, instantiated with some
default values and ready to accept your Unicode strings.
</para>
<para>
- Harfbuzz manages the memory of objects that it creates (such as
- buffers), so you don't have to. When you have finished working on
- a buffer, you can call <literal>hb_buffer_destroy()</literal>:
+ HarfBuzz manages the memory of objects (such as buffers) that it
+ creates, so you don't have to. When you have finished working on
+ a buffer, you can call <function>hb_buffer_destroy()</function>:
</para>
<programlisting language="C">
- hb_buffer_t *buffer = hb_buffer_create();
- ...
- hb_buffer_destroy(buffer);
-</programlisting>
+ hb_buffer_t *buf = hb_buffer_create();
+ ...
+ hb_buffer_destroy(buf);
+ </programlisting>
<para>
This will destroy the object and free its associated memory -
unless some other part of the program holds a reference to this
- buffer. If you acquire a Harfbuzz buffer from another subsystem
+ buffer. If you acquire a HarfBuzz buffer from another subsystem
and want to ensure that it is not garbage collected by someone
else destroying it, you should increase its reference count:
</para>
<programlisting language="C">
-void somefunc(hb_buffer_t *buffer) {
- buffer = hb_buffer_reference(buffer);
- ...
-</programlisting>
+ void somefunc(hb_buffer_t *buf) {
+ buf = hb_buffer_reference(buf);
+ ...
+ </programlisting>
<para>
And then decrease it once you're done with it:
</para>
<programlisting language="C">
- hb_buffer_destroy(buffer);
-}
-</programlisting>
+ hb_buffer_destroy(buf);
+ }
+ </programlisting>
+ <para>
+ While we are on the subject of reference-counting buffers, it is
+ worth noting that an individual buffer can only meaningfully be
+ used by one thread at a time.
+ </para>
<para>
To throw away all the data in your buffer and start from scratch,
- call <literal>hb_buffer_reset(buffer)</literal>. If you want to
+ call <function>hb_buffer_reset(buf)</function>. If you want to
throw away the string in the buffer but keep the options, you can
- instead call <literal>hb_buffer_clear_contents(buffer)</literal>.
+ instead call <function>hb_buffer_clear_contents(buf)</function>.
</para>
</section>
+
<section id="adding-text-to-the-buffer">
<title>Adding text to the buffer</title>
<para>
- Now we have a brand new Harfbuzz buffer. Let's start filling it
- with text! From Harfbuzz's perspective, a buffer is just a stream
- of Unicode codepoints, but your input string is probably in one of
- the standard Unicode character encodings (UTF-8, UTF-16, UTF-32)
+ Now we have a brand new HarfBuzz buffer. Let's start filling it
+ with text! From HarfBuzz's perspective, a buffer is just a stream
+ of Unicode code points, but your input string is probably in one of
+ the standard Unicode character encodings (UTF-8, UTF-16, or
+ UTF-32). HarfBuzz provides convenience functions that accept
+ each of these encodings:
+ <function>hb_buffer_add_utf8()</function>,
+ <function>hb_buffer_add_utf16()</function>, and
+ <function>hb_buffer_add_utf32()</function>. Other than the
+ character encoding they accept, they function identically.
+ </para>
+ <para>
+ You can add UTF-8 text to a buffer by passing in the text array,
+ the array's length, an offset into the array for the first
+ character to add, and the length of the segment to add:
</para>
+ <programlisting language="C">
+ hb_buffer_add_utf8 (hb_buffer_t *buf,
+ const char *text,
+ int text_length,
+ unsigned int item_offset,
+ int item_length)
+ </programlisting>
+ <para>
+ So, in practice, you can say:
+ </para>
+ <programlisting language="C">
+ hb_buffer_add_utf8(buf, text, strlen(text), 0, strlen(text));
+ </programlisting>
+ <para>
+ This will append your new characters to
+ <parameter>buf</parameter>, not replace its existing
+ contents. Also, note that you can use <literal>-1</literal> in
+ place of the first instance of <function>strlen(text)</function>
+ if your text array is NULL-terminated. Similarly, you can also use
+ <literal>-1</literal> as the final argument want to add its full
+ contents.
+ </para>
+ <para>
+ Whatever start <parameter>item_offset</parameter> and
+ <parameter>item_length</parameter> you provide, HarfBuzz will also
+ attempt to grab the five characters <emphasis>before</emphasis>
+ the offset point and the five characters
+ <emphasis>after</emphasis> the designated end. These are the
+ before and after "context" segments, which are used internally
+ for HarfBuzz to make shaping decisions. They will not be part of
+ the final output, but they ensure that HarfBuzz's
+ script-specific shaping operations are correct. If there are
+ fewer than five characters available for the before or after
+ contexts, HarfBuzz will just grab what is there.
+ </para>
+ <para>
+ For longer text runs, such as full paragraphs, it might be
+ tempting to only add smaller sub-segments to a buffer and
+ shape them in piecemeal fashion. Generally, this is not a good
+ idea, however, because a lot of shaping decisions are
+ dependent on this context information. For example, in Arabic
+ and other connected scripts, HarfBuzz needs to know the code
+ points before and after each character in order to correctly
+ determine which glyph to return.
+ </para>
+ <para>
+ The safest approach is to add all of the text available, then
+ use <parameter>item_offset</parameter> and
+ <parameter>item_length</parameter> to indicate which characters you
+ want shaped, so that HarfBuzz has access to any context.
+ </para>
+ <para>
+ You can also add Unicode code points directly with
+ <function>hb_buffer_add_codepoints()</function>. The arguments
+ to this function are the same as those for the UTF
+ encodings. But it is particularly important to note that
+ HarfBuzz does not do validity checking on the text that is added
+ to a buffer. Invalid code points will be replaced, but it is up
+ to you to do any deep-sanity checking necessary.
+ </para>
+
</section>
+
<section id="setting-buffer-properties">
<title>Setting buffer properties</title>
<para>
+ Buffers containing input characters still need several
+ properties set before HarfBuzz can shape their text correctly.
+ </para>
+ <para>
+ Initially, all buffers are set to the
+ <literal>HB_BUFFER_CONTENT_TYPE_INVALID</literal> content
+ type. After adding text, the buffer should be set to
+ <literal>HB_BUFFER_CONTENT_TYPE_UNICODE</literal> instead, which
+ indicates that it contains un-shaped input
+ characters. After shaping, the buffer will have the
+ <literal>HB_BUFFER_CONTENT_TYPE_GLYPHS</literal> content type.
+ </para>
+ <para>
+ <function>hb_buffer_add_utf8()</function> and the
+ other UTF functions set the content type of their buffer
+ automatically. But if you are reusing a buffer you may want to
+ check its state with
+ <function>hb_buffer_get_content_type(buffer)</function>. If
+ necessary you can set the content type with
+ </para>
+ <programlisting language="C">
+ hb_buffer_set_content_type(buf, HB_BUFFER_CONTENT_TYPE_UNICODE);
+ </programlisting>
+ <para>
+ to prepare for shaping.
+ </para>
+ <para>
+ Buffers also need to carry information about the script,
+ language, and text direction of their contents. You can set
+ these properties individually:
+ </para>
+ <programlisting language="C">
+ hb_buffer_set_direction(buf, HB_DIRECTION_LTR);
+ hb_buffer_set_script(buf, HB_SCRIPT_LATIN);
+ hb_buffer_set_language(buf, hb_language_from_string("en", -1));
+ </programlisting>
+ <para>
+ However, since these properties are often repeated for
+ multiple text runs, you can also save them in a
+ <literal>hb_segment_properties_t</literal> for reuse:
+ </para>
+ <programlisting language="C">
+ hb_segment_properties_t *savedprops;
+ hb_buffer_get_segment_properties (buf, savedprops);
+ ...
+ hb_buffer_set_segment_properties (buf2, savedprops);
+ </programlisting>
+ <para>
+ HarfBuzz also provides getter functions to retrieve a buffer's
+ direction, script, and language properties individually.
+ </para>
+ <para>
+ HarfBuzz recognizes four text directions in
+ <type>hb_direction_t</type>: left-to-right
+ (<literal>HB_DIRECTION_LTR</literal>), right-to-left (<literal>HB_DIRECTION_RTL</literal>),
+ top-to-bottom (<literal>HB_DIRECTION_TTB</literal>), and
+ bottom-to-top (<literal>HB_DIRECTION_BTT</literal>). For the
+ script property, HarfBuzz uses identifiers based on the
+ <ulink
+ url="https://unicode.org/iso15924/">ISO 15924
+ standard</ulink>. For languages, HarfBuzz uses tags based on the
+ <ulink url="https://tools.ietf.org/html/bcp47">IETF BCP 47</ulink> standard.
+ </para>
+ <para>
+ Helper functions are provided to convert character strings into
+ the necessary script and language tag types.
+ </para>
+ <para>
+ Two additional buffer properties to be aware of are the
+ "invisible glyph" and the replacement code point. The
+ replacement code point is inserted into buffer output in place of
+ any invalid code points encountered in the input. By default, it
+ is the Unicode <literal>REPLACEMENT CHARACTER</literal> code
+ point, <literal>U+FFFD</literal> "�". You can change this with
+ </para>
+ <programlisting language="C">
+ hb_buffer_set_replacement_codepoint(buf, replacement);
+ </programlisting>
+ <para>
+ passing in the replacement Unicode code point as the
+ <parameter>replacement</parameter> parameter.
</para>
- </section>
- <section id="what-about-the-other-scripts">
- <title>What about the other scripts?</title>
<para>
+ The invisible glyph is used to replace all output glyphs that
+ are invisible. By default, the standard space character
+ <literal>U+0020</literal> is used; you can replace this (for
+ example, when using a font that provides script-specific
+ spaces) with
+ </para>
+ <programlisting language="C">
+ hb_buffer_set_invisible_glyph(buf, replacement_glyph);
+ </programlisting>
+ <para>
+ Do note that in the <parameter>replacement_glyph</parameter>
+ parameter, you must provide the glyph ID of the replacement you
+ wish to use, not the Unicode code point.
+ </para>
+ <para>
+ HarfBuzz supports a few additional flags you might want to set
+ on your buffer under certain circumstances. The
+ <literal>HB_BUFFER_FLAG_BOT</literal> and
+ <literal>HB_BUFFER_FLAG_EOT</literal> flags tell HarfBuzz
+ that the buffer represents the beginning or end (respectively)
+ of a text element (such as a paragraph or other block). Knowing
+ this allows HarfBuzz to apply certain contextual font features
+ when shaping, such as initial or final variants in connected
+ scripts.
+ </para>
+ <para>
+ <literal>HB_BUFFER_FLAG_PRESERVE_DEFAULT_IGNORABLES</literal>
+ tells HarfBuzz not to hide glyphs with the
+ <literal>Default_Ignorable</literal> property in Unicode. This
+ property designates control characters and other non-printing
+ code points, such as joiners and variation selectors. Normally
+ HarfBuzz replaces them in the output buffer with zero-width
+ space glyphs (using the "invisible glyph" property discussed
+ above); setting this flag causes them to be printed, which can
+ be helpful for troubleshooting.
+ </para>
+ <para>
+ Conversely, setting the
+ <literal>HB_BUFFER_FLAG_REMOVE_DEFAULT_IGNORABLES</literal> flag
+ tells HarfBuzz to remove <literal>Default_Ignorable</literal>
+ glyphs from the output buffer entirely. Finally, setting the
+ <literal>HB_BUFFER_FLAG_DO_NOT_INSERT_DOTTED_CIRCLE</literal>
+ flag tells HarfBuzz not to insert the dotted-circle glyph
+ (<literal>U+25CC</literal>, "◌"), which is normally
+ inserted into buffer output when broken character sequences are
+ encountered (such as combining marks that are not attached to a
+ base character).
</para>
</section>
+
<section id="customizing-unicode-functions">
<title>Customizing Unicode functions</title>
<para>
+ HarfBuzz requires some simple functions for accessing
+ information from the Unicode Character Database (such as the
+ <literal>General_Category</literal> (gc) and
+ <literal>Script</literal> (sc) properties) that is useful
+ for shaping, as well as some useful operations like composing and
+ decomposing code points.
+ </para>
+ <para>
+ HarfBuzz includes its own internal, lightweight set of Unicode
+ functions. At build time, it is also possible to compile support
+ for some other options, such as the Unicode functions provided
+ by GLib or the International Components for Unicode (ICU)
+ library. Generally, this option is only of interest for client
+ programs that have specific integration requirements or that do
+ a significant amount of customization.
+ </para>
+ <para>
+ If your program has access to other Unicode functions, however,
+ such as through a system library or application framework, you
+ might prefer to use those instead of the built-in
+ options. HarfBuzz supports this by implementing its Unicode
+ functions as a set of virtual methods that you can replace —
+ without otherwise affecting HarfBuzz's functionality.
+ </para>
+ <para>
+ The Unicode functions are specified in a structure called
+ <literal>unicode_funcs</literal> which is attached to each
+ buffer. But even though <literal>unicode_funcs</literal> is
+ associated with a <type>hb_buffer_t</type>, the functions
+ themselves are called by other HarfBuzz APIs that access
+ buffers, so it would be unwise for you to hook different
+ functions into different buffers.
+ </para>
+ <para>
+ In addition, you can mark your <literal>unicode_funcs</literal>
+ as immutable by calling
+ <function>hb_unicode_funcs_make_immutable (ufuncs)</function>.
+ This is especially useful if your code is a
+ library or framework that will have its own client programs. By
+ marking your Unicode function choices as immutable, you prevent
+ your own client programs from changing the
+ <literal>unicode_funcs</literal> configuration and introducing
+ inconsistencies and errors downstream.
+ </para>
+ <para>
+ You can retrieve the Unicode-functions configuration for
+ your buffer by calling <function>hb_buffer_get_unicode_funcs()</function>:
+ </para>
+ <programlisting language="C">
+ hb_unicode_funcs_t *ufunctions;
+ ufunctions = hb_buffer_get_unicode_funcs(buf);
+ </programlisting>
+ <para>
+ The current version of <literal>unicode_funcs</literal> uses six functions:
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <function>hb_unicode_combining_class_func_t</function>:
+ returns the Canonical Combining Class of a code point.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <function>hb_unicode_general_category_func_t</function>:
+ returns the General Category (gc) of a code point.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <function>hb_unicode_mirroring_func_t</function>: returns
+ the Mirroring Glyph code point (for bi-directional
+ replacement) of a code point.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <function>hb_unicode_script_func_t</function>: returns the
+ Script (sc) property of a code point.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <function>hb_unicode_compose_func_t</function>: returns the
+ canonical composition of a sequence of two code points.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <function>hb_unicode_decompose_func_t</function>: returns
+ the canonical decomposition of a code point.
+ </para>
+ </listitem>
+ </itemizedlist>
+ <para>
+ Note, however, that future HarfBuzz releases may alter this set.
+ </para>
+ <para>
+ Each Unicode function has a corresponding setter, with which you
+ can assign a callback to your replacement function. For example,
+ to replace
+ <function>hb_unicode_general_category_func_t</function>, you can call
+ </para>
+ <programlisting language="C">
+ hb_unicode_funcs_set_general_category_func (*ufuncs, func, *user_data, destroy)
+ </programlisting>
+ <para>
+ Virtualizing this set of Unicode functions is primarily intended
+ to improve portability. There is no need for every client
+ program to make the effort to replace the default options, so if
+ you are unsure, do not feel any pressure to customize
+ <literal>unicode_funcs</literal>.
</para>
</section>
-</chapter>
\ No newline at end of file
+
+</chapter>
projects / platform / upstream / harfbuzz.git / blobdiff