`\u200c` and `\u200d` should be allowed in IdentifierPart, as per ES5

https://bugs.webkit.org/show_bug.cgi?id=78908

Source/JavaScriptCore:

Add additional checks for zero-width non-joiner (0x200C) and
zero-width joiner (0x200D) characters.

Patch by Mathias Bynens <mathias@qiwi.be> on 2012-02-24
Reviewed by Michael Saboff.

* parser/Lexer.cpp:
(JSC::isNonASCIIIdentPart)
* runtime/LiteralParser.cpp:
(JSC::::Lexer::lexIdentifier)

LayoutTests:

Patch by Mathias Bynens <mathias@qiwi.be> on 2012-02-24
Reviewed by Michael Saboff.

* fast/js/var-declarations-zero-width-expected.txt: Added.
* fast/js/var-declarations-zero-width.html: Added.
* fast/js/removing-Cf-characters-expected.txt: Removed the ZWJ test.
* fast/js/script-tests/removing-Cf-characters.js: Removed the ZWJ test.
* fast/js/script-tests/var-declarations-zero-width.js: Added.
* platform/chromium/test_expectations.txt: Disable the new test in Chromium, for now.

git-svn-id: http://svn.webkit.org/repository/webkit/trunk@108742 268f45cc-cd09-0410-ab3c-d52691b4dbfc

diff --git a/LayoutTests/ChangeLog b/LayoutTests/ChangeLog
index 54544ce..618c156 100644 (file)
--- a/LayoutTests/ChangeLog
+++ b/LayoutTests/ChangeLog
@@ -1,3 +1,17 @@
+2012-02-24  Mathias Bynens  <mathias@qiwi.be>
+
+        `\u200c` and `\u200d` should be allowed in IdentifierPart, as per ES5
+        https://bugs.webkit.org/show_bug.cgi?id=78908
+
+        Reviewed by Michael Saboff.
+
+        * fast/js/var-declarations-zero-width-expected.txt: Added.
+        * fast/js/var-declarations-zero-width.html: Added.
+        * fast/js/removing-Cf-characters-expected.txt: Removed the ZWJ test.
+        * fast/js/script-tests/removing-Cf-characters.js: Removed the ZWJ test.
+        * fast/js/script-tests/var-declarations-zero-width.js: Added.
+        * platform/chromium/test_expectations.txt: Disable the new test in Chromium, for now.
+
 2012-02-24  Yuzo Fujishima  <yuzo@google.com>
 
         [chromium] Unreviewed test expectation change.

diff --git a/LayoutTests/fast/js/removing-Cf-characters-expected.txt b/LayoutTests/fast/js/removing-Cf-characters-expected.txt
index bd46191..7628eaf 100644 (file)
--- a/LayoutTests/fast/js/removing-Cf-characters-expected.txt
+++ b/LayoutTests/fast/js/removing-Cf-characters-expected.txt
@@ -9,7 +9,6 @@ PASS 1 is 1
 PASS eval('""').length is 1
 PASS eval('""').charCodeAt(0) is 0xFEFF
 PASS ++1 /* BOM between the +'s */ is 1
-PASS var ZWJ_I‍nside; threw exception SyntaxError: Invalid character '\u8205'.
 PASS successfullyParsed is true
 
 TEST COMPLETE

diff --git a/LayoutTests/fast/js/script-tests/removing-Cf-characters.js b/LayoutTests/fast/js/script-tests/removing-Cf-characters.js
index 26b5eac..8d5e8ea 100644 (file)
--- a/LayoutTests/fast/js/script-tests/removing-Cf-characters.js
+++ b/LayoutTests/fast/js/script-tests/removing-Cf-characters.js
@@ -25,4 +25,3 @@ shouldBe('eval(\'"\uFEFF"\').length', '1');
 shouldBe('eval(\'"\uFEFF"\').charCodeAt(0)', '0xFEFF');
 shouldBe('+'+eval("\"\uFEFF\"")+'+1 /* BOM between the +\'s */', '1');
 
-shouldThrow('var ZWJ_I‍nside;');

diff --git a/LayoutTests/fast/js/script-tests/var-declarations-zero-width.js b/LayoutTests/fast/js/script-tests/var-declarations-zero-width.js
new file mode 100644 (file)
index 0000000..1303b82
--- /dev/null
+++ b/LayoutTests/fast/js/script-tests/var-declarations-zero-width.js
@@ -0,0 +1,5 @@
+description("This page tests if U+200C and U+200D are allowed as part of an identifier.");
+
+shouldBe("var x\u200c = 42; x\u200c", "42");
+shouldBe("var x\u200d = 43; x\u200d", "43");
+shouldBe("var x\u200c\u200d = 44; x\u200c\u200d", "44");
\ No newline at end of file

diff --git a/LayoutTests/fast/js/var-declarations-zero-width-expected.txt b/LayoutTests/fast/js/var-declarations-zero-width-expected.txt
new file mode 100644 (file)
index 0000000..3c29356
--- /dev/null
+++ b/LayoutTests/fast/js/var-declarations-zero-width-expected.txt
@@ -0,0 +1,12 @@
+This page tests if U+200C and U+200D are allowed as part of an identifier.
+
+On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
+
+
+PASS var x‌ = 42; x‌ is 42
+PASS var x‍ = 43; x‍ is 43
+PASS var x‌‍ = 44; x‌‍ is 44
+PASS successfullyParsed is true
+
+TEST COMPLETE
+

diff --git a/LayoutTests/fast/js/var-declarations-zero-width.html b/LayoutTests/fast/js/var-declarations-zero-width.html
new file mode 100644 (file)
index 0000000..6838e06
--- /dev/null
+++ b/LayoutTests/fast/js/var-declarations-zero-width.html
@@ -0,0 +1,4 @@
+<!DOCTYPE html>
+<script src="resources/js-test-pre.js"></script>
+<script src="script-tests/var-declarations-zero-width.js"></script>
+<script src="resources/js-test-post.js"></script>

diff --git a/LayoutTests/platform/chromium/test_expectations.txt b/LayoutTests/platform/chromium/test_expectations.txt
index 72b7f9a..8aeee9b 100644 (file)
--- a/LayoutTests/platform/chromium/test_expectations.txt
+++ b/LayoutTests/platform/chromium/test_expectations.txt
@@ -909,6 +909,9 @@ BUGCR10278 LINUX : fast/css/font-face-unicode-range.html = FAIL
 // what the resolution was.
 BUGCR10279 : fast/js/var-declarations-shadowing.html = FAIL
 
+// FIXME: Enable this test when V8 r10800 has rolled into WebKit.
+BUGWK79361 : fast/js/var-declarations-zero-width.html = FAIL
+
 // Implement full JSON support in V8
 BUGCR14885 SLOW : fast/js/JSON-parse.html = FAIL
 BUGCR14885 SLOW : fast/js/JSON-stringify.html = CRASH FAIL

diff --git a/Source/JavaScriptCore/ChangeLog b/Source/JavaScriptCore/ChangeLog
index 7d920f3..896e10c 100644 (file)
--- a/Source/JavaScriptCore/ChangeLog
+++ b/Source/JavaScriptCore/ChangeLog
@@ -1,3 +1,18 @@
+2012-02-24  Mathias Bynens  <mathias@qiwi.be>
+
+        `\u200c` and `\u200d` should be allowed in IdentifierPart, as per ES5
+        https://bugs.webkit.org/show_bug.cgi?id=78908
+
+        Add additional checks for zero-width non-joiner (0x200C) and
+        zero-width joiner (0x200D) characters.
+
+        Reviewed by Michael Saboff.
+
+        * parser/Lexer.cpp:
+        (JSC::isNonASCIIIdentPart)
+        * runtime/LiteralParser.cpp:
+        (JSC::::Lexer::lexIdentifier)
+
 2012-02-23  Kenichi Ishibashi  <bashi@chromium.org>
 
         Adding WebSocket per-frame DEFLATE extension

diff --git a/Source/JavaScriptCore/parser/Lexer.cpp b/Source/JavaScriptCore/parser/Lexer.cpp
index 08792d5..015c150 100644 (file)
--- a/Source/JavaScriptCore/parser/Lexer.cpp
+++ b/Source/JavaScriptCore/parser/Lexer.cpp
@@ -3,6 +3,7 @@
  *  Copyright (C) 2006, 2007, 2008, 2009 Apple Inc. All Rights Reserved.
  *  Copyright (C) 2007 Cameron Zwarich (cwzwarich@uwaterloo.ca)
  *  Copyright (C) 2010 Zoltan Herczeg (zherczeg@inf.u-szeged.hu)
+ *  Copyright (C) 2012 Mathias Bynens (mathias@qiwi.be)
  *
  *  This library is free software; you can redistribute it and/or
  *  modify it under the terms of the GNU Library General Public
@@ -379,8 +380,8 @@ static inline bool isIdentStart(int c)
 
 static NEVER_INLINE bool isNonASCIIIdentPart(int c)
 {
-    return category(c) & (Letter_Uppercase | Letter_Lowercase | Letter_Titlecase | Letter_Modifier | Letter_Other
-        | Mark_NonSpacing | Mark_SpacingCombining | Number_DecimalDigit | Punctuation_Connector);
+    return (category(c) & (Letter_Uppercase | Letter_Lowercase | Letter_Titlecase | Letter_Modifier | Letter_Other
+        | Mark_NonSpacing | Mark_SpacingCombining | Number_DecimalDigit | Punctuation_Connector)) || c == 0x200C || c == 0x200D;
 }
 
 static ALWAYS_INLINE bool isIdentPart(int c)

diff --git a/Source/JavaScriptCore/runtime/LiteralParser.cpp b/Source/JavaScriptCore/runtime/LiteralParser.cpp
index 0418737..3bde3ff 100644 (file)
--- a/Source/JavaScriptCore/runtime/LiteralParser.cpp
+++ b/Source/JavaScriptCore/runtime/LiteralParser.cpp
@@ -1,5 +1,6 @@
 /*
  * Copyright (C) 2009 Apple Inc. All rights reserved.
+ * Copyright (C) 2012 Mathias Bynens (mathias@qiwi.be)
  *
  * Redistribution and use in source and binary forms, with or without
  * modification, are permitted provided that the following conditions
@@ -294,7 +295,7 @@ ALWAYS_INLINE TokenType LiteralParser<LChar>::Lexer::lexIdentifier(LiteralParser
 template <>
 ALWAYS_INLINE TokenType LiteralParser<UChar>::Lexer::lexIdentifier(LiteralParserToken<UChar>& token)
 {
-    while (m_ptr < m_end && (isASCIIAlphanumeric(*m_ptr) || *m_ptr == '_' || *m_ptr == '$'))
+    while (m_ptr < m_end && (isASCIIAlphanumeric(*m_ptr) || *m_ptr == '_' || *m_ptr == '$' || *m_ptr == 0x200C || *m_ptr == 0x200D))
         m_ptr++;
     token.stringIs8Bit = 0;
     token.stringToken16 = token.start;