Changeset 12648 for src/Pugs/Lexer.hs

Timestamp:

08/24/06 13:08:06 (2 years ago)

Author:

audreyt

Message:

* Revert TimToady?++'s Lexer patch as isSpace is already

Unicode-aware.

* To handle BOM on UTF8 files (and UTF16, UTF32, etc),

implement my source-code-encoding detection algorithm
proposal for the next Haskell standard:

http://hackage.haskell.org/trac/haskell-prime/wiki/SourceEncodingDetection

Currently only UTF8 is handled; UTF16 and UTF32 throws
an exception.

Files:

• src/Pugs/Lexer.hs (modified) (1 diff)

src/Pugs/Lexer.hs

@@ -596 +596 @@
     return ()
         
--- XXX this should depend on real Unicode definitions
--- XXX (could probably be more efficient too, since oneOf probably scans)
-simpleSpace =
-    skipMany1 $ oneOf " \t\n\f\r\x0085\x00A0\x1680\x180E\x2000\x2001\x2002\x2003\x2004\x2005\x2006\x2007\x2008\x2009\x200A\x2028\x2029\x202F\x205F\x3000\xFEFF"
+simpleSpace = do
+    skipMany1 (satisfy (isSpace))