Behdad Esfahbod [Fri, 20 Jul 2012 21:05:46 +0000 (17:05 -0400)]
[Uniscribe] Fix script fallback
Gurmukhi failures half now. Others changed slightly.
Behdad Esfahbod [Fri, 20 Jul 2012 20:26:55 +0000 (16:26 -0400)]
[Indic] Allow a ZWNJ after SM's
Malayalam failures go way down. Other scripts benefitted slightly too.
Sinhala had one or two test regressions, but...
Behdad Esfahbod [Fri, 20 Jul 2012 20:17:28 +0000 (16:17 -0400)]
[Indic] Improve Reph AfterMain positioning
Fixes 20 out of 48 failing Oriya tests. Failure rate down to 0.066% now.
Behdad Esfahbod [Fri, 20 Jul 2012 20:03:09 +0000 (16:03 -0400)]
[Indic] Reposition Oriya Candrabindu
Oriya failures down from 0.65% to 0.20%.
Behdad Esfahbod [Fri, 20 Jul 2012 19:47:24 +0000 (15:47 -0400)]
[Indic] Fix base search in final_reordering
Fixes most Malayalam failures. Down from 1.6% to 0.38% now. Fixes a
few more in other scripts too.
Behdad Esfahbod [Fri, 20 Jul 2012 19:10:02 +0000 (15:10 -0400)]
[Indic] Don't attach SM/VD to other characters
Behdad Esfahbod [Fri, 20 Jul 2012 18:27:38 +0000 (14:27 -0400)]
Fix compiler warnings
If x is not constant, we cannot ASSERT_STATIC on it.
Behdad Esfahbod [Fri, 20 Jul 2012 18:13:35 +0000 (14:13 -0400)]
[Indic] Form Telugu Reph out of Ra,Virama,ZWJ
Apparently this was approved in Feb 2012. No font yet.
Behdad Esfahbod [Fri, 20 Jul 2012 18:02:35 +0000 (14:02 -0400)]
[Indic] Don't split U+0AC9
Althought IndicMatraCategory.txt classifies it as Top_And_Right matra,
it does not have Unicode decomposition, and Uniscribe does not do
anything special about it either.
Gujarati failures down from 0.672% to 0.0130966%.
Behdad Esfahbod [Fri, 20 Jul 2012 17:56:32 +0000 (13:56 -0400)]
[Indic] Simplify Uniscribe cluster emulation
Now that we break syllables on Halant,ZWNJ, this code can be simplified.
Behdad Esfahbod [Fri, 20 Jul 2012 17:51:31 +0000 (13:51 -0400)]
[Indic] Minor!
Behdad Esfahbod [Fri, 20 Jul 2012 17:48:03 +0000 (13:48 -0400)]
[Indic] Break syllables at Halant,ZWNJ
That's really what Uniscribe does, and explains a lot of pecularities of
Halant,ZWNJ before the base.
Sent Telugu from 1% failures to 0.03%. Improved Kannada and Malayalam
slightly. Fixed half of Bengali, and did NOT break anything!
Behdad Esfahbod [Fri, 20 Jul 2012 17:37:48 +0000 (13:37 -0400)]
[Indic] Better check for applying 'init'
Specifically, don't apply 'init' if previous char is a joiner.
Fixes some more of Bengali.
Behdad Esfahbod [Fri, 20 Jul 2012 16:32:59 +0000 (12:32 -0400)]
[GPOS] Don't zero mark advances
Fixes more of Telugu, Kannada, and Oriya.
May break things (outside Indic...), but we cannot think of any font relying
on this immediately.
Behdad Esfahbod [Fri, 20 Jul 2012 15:42:24 +0000 (11:42 -0400)]
[Indic] Minor
Behdad Esfahbod [Fri, 20 Jul 2012 15:25:41 +0000 (11:25 -0400)]
[Indic] Implement reph positioning step 5
Not tuned, just copied from step 2. Fixes another 0.5% of Kannada
failures. 1% to go.
Behdad Esfahbod [Fri, 20 Jul 2012 15:04:15 +0000 (11:04 -0400)]
[Indic] Don't let ZWNJ at the end of syllable affect base search
Fixes a few Devanagari, half of remaining Kannada failures, quarter for
Telugu, and others slightly improved or unchanged.
Behdad Esfahbod [Fri, 20 Jul 2012 14:47:46 +0000 (10:47 -0400)]
[Indic] Apply 'cjct' globally
Fixes 5 Devanagari failures, and no regressions.
Behdad Esfahbod [Fri, 20 Jul 2012 14:30:24 +0000 (10:30 -0400)]
[Indic] Unbreak old scriptures
Brings down failures with Lohit-Telugu from 57% to 1.40%.
Behdad Esfahbod [Fri, 20 Jul 2012 14:18:23 +0000 (10:18 -0400)]
Minor
Behdad Esfahbod [Fri, 20 Jul 2012 01:17:48 +0000 (21:17 -0400)]
[Indic] Recategorize some Kannada right matras
Kannada failures down from 3.5% to 2.93%.
Behdad Esfahbod [Fri, 20 Jul 2012 01:13:04 +0000 (21:13 -0400)]
[Indic] Recategorize GURMUKHI ADDAK
It's not in IndicSyllabicCategory.txt. Fixes most of Gurmukhi failures.
Failures down from 7.7% to 0.222%!
Behdad Esfahbod [Fri, 20 Jul 2012 01:02:38 +0000 (21:02 -0400)]
Minor
Behdad Esfahbod [Fri, 20 Jul 2012 00:52:44 +0000 (20:52 -0400)]
[buffer] Make sure out_info = info during GPOS
Behdad Esfahbod [Fri, 20 Jul 2012 00:45:41 +0000 (20:45 -0400)]
Print line number during return when tracing
Behdad Esfahbod [Fri, 20 Jul 2012 00:30:22 +0000 (20:30 -0400)]
[Indic] Unbreak Khmer
For Khmer, all consonants are subjoining. No need to look in the font.
We were looking in the wrong order anyway.
Behdad Esfahbod [Fri, 20 Jul 2012 00:24:14 +0000 (20:24 -0400)]
[Indic] Apply 'akhn' globally
Fixes 1.5% more failures for Telugu, 2% for Kannada.
Breaks one test in Devanagari.
Behdad Esfahbod [Fri, 20 Jul 2012 00:03:25 +0000 (20:03 -0400)]
[Indic] Add failing test for Kannada
Behdad Esfahbod [Thu, 19 Jul 2012 23:52:19 +0000 (19:52 -0400)]
[Indic] Better position U+0CD5
Fixes another 5% of Kannada failures.
Behdad Esfahbod [Thu, 19 Jul 2012 20:20:21 +0000 (16:20 -0400)]
[Indic] Lookup consonant position in the font
Fixes most failures of Oriya, and improves others a bit.
Behdad Esfahbod [Thu, 19 Jul 2012 20:14:23 +0000 (16:14 -0400)]
[GSUB] Fix would_apply() for LigatureSubst
Behdad Esfahbod [Thu, 19 Jul 2012 19:29:13 +0000 (15:29 -0400)]
[TODO] Minor
Behdad Esfahbod [Thu, 19 Jul 2012 18:59:15 +0000 (14:59 -0400)]
Add src/test-would-substitute tool
Behdad Esfahbod [Thu, 19 Jul 2012 18:35:23 +0000 (14:35 -0400)]
Refactor / finish would_apply() operation
Untested.
Behdad Esfahbod [Thu, 19 Jul 2012 17:25:08 +0000 (13:25 -0400)]
[Indic] Implement per-script matra positioning
Following what the spec says.
Brings down Telugu failures from 40% to 3.75%, and Kannada failures from
44% to 10%. Does NOT affect other scripts' test results.
Behdad Esfahbod [Thu, 19 Jul 2012 16:59:06 +0000 (12:59 -0400)]
[Indic] More refactoring
Behdad Esfahbod [Thu, 19 Jul 2012 16:32:16 +0000 (12:32 -0400)]
[Indic] Minor refactoring
Behdad Esfahbod [Thu, 19 Jul 2012 16:23:23 +0000 (12:23 -0400)]
[Indic] Minor
Behdad Esfahbod [Thu, 19 Jul 2012 16:11:12 +0000 (12:11 -0400)]
[Indic] Start refactoring different matra positions per script
Behdad Esfahbod [Thu, 19 Jul 2012 15:30:48 +0000 (11:30 -0400)]
[test] Ignore tests with DOTTED CIRCLE in the output
Behdad Esfahbod [Thu, 19 Jul 2012 15:25:49 +0000 (11:25 -0400)]
[Indic] Disallow decomposition of a couple characters
This is a hack for now. Will be fixed when we do complex-shaper-driven
normalization properly.
The results with or without decomposition are the same, but Uniscribe
does not normalize, so this matches better.
Behdad Esfahbod [Thu, 19 Jul 2012 03:25:58 +0000 (23:25 -0400)]
[Indic] Accept a forced Rakar sequence at the end of syllable
In Sinhala, Rakar is formed by Al-Lakuna,ZWJ,Ra. If you put that at the
end of a Consonant,Matra syllable, you get a dotted-circle from
Uniscribe. Apparently adding a ZWJ before the Al-Lakuna "fixes" that.
And people have been encoding that sequence... So, allow a forced
"ZWJ,Virama,ZWJ,Ra" sequence at the of syllables.
Fixes some 100 or more of Sinhala failures. Now at 622 only (0.23%).
Behdad Esfahbod [Wed, 18 Jul 2012 21:49:19 +0000 (17:49 -0400)]
[Indic] Allow joiners on both sides of Halant at the same time
The sequence <ZWJ,Al-Lakuna,ZWJ> is used in Sinhala to explicitly ask
for Rakar. Fixes two-thousand Sinhala tests. Not many left.
Behdad Esfahbod [Wed, 18 Jul 2012 21:42:34 +0000 (17:42 -0400)]
[Indic] In final reordering, find base, even if it disappeared
POS_BASE can disappear if base ligated backward. Define base as last
with position not after base.
Fixes a few hundred of Sinhala failures with Iskoola Pota.
Behdad Esfahbod [Wed, 18 Jul 2012 21:29:10 +0000 (17:29 -0400)]
[Indic] Minor
Behdad Esfahbod [Wed, 18 Jul 2012 21:22:14 +0000 (17:22 -0400)]
[Indic] Implement Sinhala "Al Lakuna" Reph behavior
In Sinhala, Reph is formed only explicitly, by the presence of a ZWJ.
Behdad Esfahbod [Wed, 18 Jul 2012 20:50:41 +0000 (16:50 -0400)]
[Indic/Unicode] Decompose Sinhala split matras the way Uniscribe likes
Makes no visual difference.
Fixes most of the failures. Down from 15% to 1.3%!
Behdad Esfahbod [Wed, 18 Jul 2012 20:34:10 +0000 (16:34 -0400)]
Apply Tibetan (global) features.
Fixes all Tibetan failures. All 180k of them!
Merges back Hangul into the default shaper.
Behdad Esfahbod [Wed, 18 Jul 2012 20:00:49 +0000 (16:00 -0400)]
[Indic] Treat Register Shifters like Nukta
Really this time.
Fixes another 18 Khmer tests.
Behdad Esfahbod [Wed, 18 Jul 2012 20:00:20 +0000 (16:00 -0400)]
[Indic] Minor
Behdad Esfahbod [Wed, 18 Jul 2012 19:45:43 +0000 (15:45 -0400)]
[Indic] Fix Matra reordering when base is at end of syllable
For example: U+915,U+200c,U+93f
Fixes last Tamil failure!
Behdad Esfahbod [Wed, 18 Jul 2012 19:43:55 +0000 (15:43 -0400)]
Leave one extra item at the end of buffer allocation
Just in case, for the times we do out-of-bounds access.
jk
Behdad Esfahbod [Wed, 18 Jul 2012 19:41:53 +0000 (15:41 -0400)]
[Indic] Fix out-of-bounds array access
Behdad Esfahbod [Wed, 18 Jul 2012 19:28:25 +0000 (15:28 -0400)]
[Indic] Allow joiners before matras
Fixes 1 more Devanagari test!
Behdad Esfahbod [Wed, 18 Jul 2012 19:10:05 +0000 (15:10 -0400)]
[Indic] Allow halant group in Vowel and placeholder syllables
Fixes 2 out of 560 Devanagari failures. AND:
Fixes 1 out of 2 Tamil failures.
Behdad Esfahbod [Wed, 18 Jul 2012 19:05:40 +0000 (15:05 -0400)]
[Indic] Streamline halant/joiner in grammar
Behdad Esfahbod [Wed, 18 Jul 2012 18:57:28 +0000 (14:57 -0400)]
[Indic] Minor
Behdad Esfahbod [Wed, 18 Jul 2012 18:23:55 +0000 (14:23 -0400)]
[Indic] Hopefully minor!
Refactoring Indic machin. No semantic change.
Behdad Esfahbod [Wed, 18 Jul 2012 18:09:25 +0000 (14:09 -0400)]
[Indic] Minor
Behdad Esfahbod [Wed, 18 Jul 2012 17:13:03 +0000 (13:13 -0400)]
[Indic] Unbreak Tamil
Tamil has only about 150 failures now!
Behdad Esfahbod [Tue, 17 Jul 2012 22:17:30 +0000 (18:17 -0400)]
[Indic] Position Khmer Robat
It's a visual Repha.
Still not positioning logical Repha as occurs in Malayalam.
Another 200 Khmer failures fixed. 547 to go. That's better than
Devanagari!
Behdad Esfahbod [Tue, 17 Jul 2012 21:53:03 +0000 (17:53 -0400)]
[Indic] Better categorize Register Shifters and Khmer Various signs
Down another 500 or so Khmer failures!
Behdad Esfahbod [Tue, 17 Jul 2012 21:09:29 +0000 (17:09 -0400)]
Add hb_buffer_normalize_glyphs() and hb-shape --normalize-glyphs
This reorders glyphs within the cluster to a nominal order. This should
have no visible effect on the output, but helps with testing, for
getting the same hb-shape output for visually-equal glyphs for each
cluster.
Behdad Esfahbod [Tue, 17 Jul 2012 18:25:14 +0000 (14:25 -0400)]
[Indic] Minor
Behdad Esfahbod [Tue, 17 Jul 2012 18:23:28 +0000 (14:23 -0400)]
[Indic] For scripts without Half forms, always choose first consonant as base
In such scripts (ie. Khmer), a ZWJ/ZWNJ shouldn't stop the search for
base. So, instead just choose the first consonant as base directly.
Test sequence:
U+1798,200c,U+17C9,U+17D2,U+179B,U+17C1,U+17C7
Behdad Esfahbod [Tue, 17 Jul 2012 18:09:32 +0000 (14:09 -0400)]
[Indic] Treat Khmer Register Shifters more like Nuktas
Except that there may be a ZWNJ before a Register Shifter.
Behdad Esfahbod [Tue, 17 Jul 2012 18:02:28 +0000 (14:02 -0400)]
[Indic] Minor
Behdad Esfahbod [Tue, 17 Jul 2012 17:55:10 +0000 (13:55 -0400)]
[Indic] Apply 'cfar' for Khmer
Mark stuff after a pre-base reordering Ro 'cfar'. Used in Khmer.
This allows distinguishing the following cases with MS Khmer fonts:
U+1784,U+17D2,U+179A,U+17D2,U+1782
U+1784,U+17D2,U+1782,U+17D2,U+179A
Behdad Esfahbod [Tue, 17 Jul 2012 16:50:13 +0000 (12:50 -0400)]
[Indic] Position final subjoined consonants (and vowels) after matras
In Khmer, a final subjoined consonant or independent vowel can occur
after matras. This final subjoined thing should NOT be reordered to
before the matra even though it's subjoined.
Fixes another 1k of the Khmer failures. Not much left really.
Behdad Esfahbod [Tue, 17 Jul 2012 15:54:28 +0000 (11:54 -0400)]
[Indic] Recategorize Khmer coeng sign as a separate category OT_Coeng
Amend the syllable structure to allow a final subscripted consonant
(Coeng+C) and a final subscripted independent vowel (Coeng+V).
Fixes another 2k of Khmer failures.
Behdad Esfahbod [Tue, 17 Jul 2012 15:37:32 +0000 (11:37 -0400)]
[Indic] Add a separate Coeng class
No characters recategorized yet. No semantic change.
Behdad Esfahbod [Tue, 17 Jul 2012 15:16:19 +0000 (11:16 -0400)]
[Indic] Move Halant with after-base consonants
Normally, we attach the Halant to the previous character and move it
with it. For after-base consonants however, the Halant "belongs" to the
consonant after, so attach it so.
This fixes Bengali sequences involving post-base consonant Ya, which
should ligate with the Halant to form Ya Phala, but previously a
reordered matras was blocking the ligation.
Behdad Esfahbod [Tue, 17 Jul 2012 14:40:59 +0000 (10:40 -0400)]
[Indic] Apply presentation-forms features all at once
Seems like this is what Uniscribe is doing, and does not break any fonts
we tested (with Devanagari, Malayalam, Khmer, and Bengali), while fixing
some Ra Phala sequences for Bengali with Vrinda. Fixes another 2% of
Bengali failures (a couple more to go).
Behdad Esfahbod [Tue, 17 Jul 2012 02:43:17 +0000 (22:43 -0400)]
Fix MarkAttachmentType matching
Fixes issue reported by Khaled Hosny with his Hussaini Nastaleeq font
and sequences like those added in the previous commit.
Behdad Esfahbod [Tue, 17 Jul 2012 02:46:06 +0000 (22:46 -0400)]
[test/arabic] Add Arabic tests for mark skipping
Expose a bug with Khaled's Hussaini Nastaleeq font.
Behdad Esfahbod [Tue, 17 Jul 2012 02:40:21 +0000 (22:40 -0400)]
Minor
Behdad Esfahbod [Tue, 17 Jul 2012 00:31:24 +0000 (20:31 -0400)]
[Indic] Disable 'kern' in Uniscribe bug compatibility mode
Uniscribe does not apply 'kern' in the Indic module. Some of the Khmer
fonts they ship have small adjustments in the 'kern' table. Disable
'kern' in the Indic module under Uniscribe bug compatibility mode.
Fixes some 10% of the Khmer failures. Remains under 3% (excluding
dotted-circle ones).
Behdad Esfahbod [Tue, 17 Jul 2012 00:26:57 +0000 (20:26 -0400)]
Allow complex shapers overriding common features
In a new callback... Currently unused by all complex shapers.
Behdad Esfahbod [Mon, 16 Jul 2012 23:56:29 +0000 (19:56 -0400)]
[Indic] Categorize other Khmer marks
Mark them the same as the Register Shifters for now. Need to rename
that category to something more sensible after all is settled.
Fixes another percent of Khmer failures. Down to under 3%!
Behdad Esfahbod [Mon, 16 Jul 2012 21:04:46 +0000 (17:04 -0400)]
[Indic] Recognize pre-base reordering Ra anywhere in the syllable
We were doing that only immediately after base.
Fixes another percent in the Khmer failures. About three more to go...
Behdad Esfahbod [Mon, 16 Jul 2012 20:45:22 +0000 (16:45 -0400)]
[Indic] Recognizer Register Shifter marks
Fixes another 6% of the Khmer failures.
Behdad Esfahbod [Mon, 16 Jul 2012 20:13:32 +0000 (16:13 -0400)]
[GSUB/GDEF] Guess glyph classes after substitution only if no GDEF
Brings down Khmer failures with Daun Penh font from 36% to 20%.
Behdad Esfahbod [Mon, 16 Jul 2012 19:52:54 +0000 (15:52 -0400)]
[Indic] Categorize Khmer Ro
Khmer failures down from 58% to 36%.
Behdad Esfahbod [Mon, 16 Jul 2012 19:49:08 +0000 (15:49 -0400)]
[Indic] Reoder pre-base reordering Ra
Brings down Malayalam failures from 14% down to 3%.
Behdad Esfahbod [Mon, 16 Jul 2012 19:40:33 +0000 (15:40 -0400)]
[Indic] Rename
Behdad Esfahbod [Mon, 16 Jul 2012 19:30:05 +0000 (15:30 -0400)]
[Indic] Start implementing pre-base reordering
Behdad Esfahbod [Mon, 16 Jul 2012 19:26:56 +0000 (15:26 -0400)]
[Indic] Rename
No semantic change.
Behdad Esfahbod [Mon, 16 Jul 2012 19:21:12 +0000 (15:21 -0400)]
[Indic] Implement After-Main Reph positioning
Almost...
Behdad Esfahbod [Mon, 16 Jul 2012 19:20:15 +0000 (15:20 -0400)]
[Indic] Apply 'pref' to pre-base reodering Ra
No reordering yet.
Behdad Esfahbod [Mon, 16 Jul 2012 19:15:28 +0000 (15:15 -0400)]
[Indic] Minor
Should not be any semantic change. In preparation for implementing
pre-base reordering Ra.
Behdad Esfahbod [Mon, 16 Jul 2012 18:52:18 +0000 (14:52 -0400)]
Minor
Behdad Esfahbod [Mon, 16 Jul 2012 17:54:43 +0000 (13:54 -0400)]
Minor
Behdad Esfahbod [Mon, 16 Jul 2012 17:47:19 +0000 (13:47 -0400)]
[Indic] Reorder virama only for old Indic spec
Behdad Esfahbod [Mon, 16 Jul 2012 17:39:36 +0000 (13:39 -0400)]
[Indic] Categorize Khmer consonants
Behdad Esfahbod [Mon, 16 Jul 2012 17:25:17 +0000 (13:25 -0400)]
Split matras without a Unicode decomposition
This is a hack for now, to get us going with Khmer. This will be
refactored properly later to move the complex logic into complex
shapers.
Behdad Esfahbod [Mon, 16 Jul 2012 17:24:26 +0000 (13:24 -0400)]
[Indic] Adjust position for split matras
We are going to split matras without a Unicode decompositions in a way
that the second half takes the codepoint of the whole matra. So,
position them where the second half is supposed to end up.
Behdad Esfahbod [Mon, 16 Jul 2012 17:23:40 +0000 (13:23 -0400)]
[GSUB] Allow 1-to-1 ligature substitutions!
Apparently Uniscribe allows these, and they are used in some Khmer fonts
shipped with Windows, namely, Daun Penh.
Behdad Esfahbod [Mon, 16 Jul 2012 16:05:35 +0000 (12:05 -0400)]
[Indic] Apply Above Forms
Behdad Esfahbod [Sat, 14 Jul 2012 16:15:54 +0000 (12:15 -0400)]
Further simplify atomic ops on Visual Studio
Behdad Esfahbod [Fri, 13 Jul 2012 17:20:49 +0000 (13:20 -0400)]
Minor