Implement underscores in numeric literals (NumericUnderscores extension)

Implement the proposal of underscores in numeric literals.
Underscores in numeric literals are simply ignored.

The specification of the feature is available here:
https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/000
9-numeric-underscores.rst

For a discussion of the various choices:
https://github.com/ghc-proposals/ghc-proposals/pull/76

Implementation detail:

* Added dynamic flag
  * `NumericUnderscores` extension flag is added for this feature.

* Alex "Regular expression macros" in Lexer.x
  * Add `@numspc` (numeric spacer) macro to represent multiple
    underscores.
  * Modify `@decimal`, `@decimal`, `@binary`, `@octal`, `@hexadecimal`,
    `@exponent`, and `@bin_exponent` macros to include `@numspc`.

* Alex "Rules" in Lexer.x
  * To be simpler, we have only the definitions with underscores.
    And then we have a separate function (`tok_integral` and `tok_frac`)
    that validates the literals.

* Validation functions in Lexer.x
  * `tok_integral` and `tok_frac` functions validate
    whether contain underscores or not.
    If `NumericUnderscores` extensions are not enabled,
    check that there are no underscores.
  * `tok_frac` function is created by merging `strtoken` and
    `init_strtoken`.
  * `init_strtoken` is deleted. Because it is no longer used.

* Remove underscores from target literal string
  * `parseUnsignedInteger`, `readRational__`, and `readHexRational} use
    the customized `span'` function to remove underscores.

* Added Testcase
  * testcase for NumericUnderscores enabled.
      NumericUnderscores0.hs and NumericUnderscores1.hs
  * testcase for NumericUnderscores disabled.
      NoNumericUnderscores0.hs and NoNumericUnderscores1.hs
  * testcase to invalid pattern for NumericUnderscores enabled.
      NumericUnderscoresFail0.hs and NumericUnderscoresFail1.hs

Test Plan: `validate` including the above testcase

Reviewers: goldfire, bgamari

Reviewed By: bgamari

Subscribers: carter, rwbarton, thomie

GHC Trac Issues: #14473

Differential Revision: https://phabricator.haskell.org/D4235

5 files changed, 218 insertions, 0 deletions

diff --git a/testsuite/tests/parser/should_run/NumericUnderscores0.hs b/testsuite/tests/parser/should_run/NumericUnderscores0.hs
new file mode 100644
index 0000000000..7aefce95c6
--- /dev/null
+++ b/testsuite/tests/parser/should_run/NumericUnderscores0.hs
@@ -0,0 +1,101 @@
+{-# LANGUAGE NumericUnderscores #-}
+{-# LANGUAGE BinaryLiterals #-}
+{-# LANGUAGE HexFloatLiterals #-}
+{-# LANGUAGE NegativeLiterals #-}
+
+-- Test for NumericUnderscores extension.
+-- See Trac #14473
+-- This is a testcase for boxed literals.
+
+main :: IO ()
+main = do
+    -- Each case corresponds to the definition of Lexer.x
+    --
+    -- Normal integral literals
+    -- decimal
+    print [ 1_000_000 == 1000000,
+            1__0 == 10,
+            299_792_458 == 299792458,
+            8_04_1 == 8041,
+            2017_12_31 == 20171231
+          ]
+
+    -- binary
+    print [ 0b01_0000_0000 == 0b0100000000,
+            0b1_11_01_0000_0_111 == 0b1110100000111,
+            0b1100_1011__1110_1111__0101_0011 ==
+            0b110010111110111101010011
+          ]
+
+    -- octal
+    print [ 0o1_000_000 == 0o1000000,
+            0O1__0 == 0O10
+          ]
+
+    -- hexadecimal
+    print [ 0x1_000_000 == 0x1000000,
+            0x1__0 == 0x10,
+            0xff_00_00 == 0xff0000,
+            0X3fff_ffff == 0x3fffffff
+          ]
+
+    -- negative decimal
+    print [ -1_0 == -10
+          ]
+
+    -- negative binary
+    print [ -0b1_0 == -0b10
+          ]
+
+    -- negative octal
+    print [ -0o1_0 == -0o10
+          ]
+
+    -- negative hexadecimal
+    print [ -0x1_0 == -0x10
+          ]
+
+    ---- Normal rational literals
+    -- float
+    print [ 3.141_592_653_589_793 == 3.141592653589793,
+            96_485.332_89 == 96485.33289,
+            6.022_140_857e+23 == 6.022140857e+23
+          ]
+
+    -- negative float
+    print [ -1_0.0_1 == -10.01,
+            -1_0e+2 == -10e+2,
+            -1_0.0_1e+3 == -10.01e+3
+          ]
+
+    -- hexadecimal float
+    print [ 0xF_F.1F == 0xFF.1F,
+            0xF_01p-8 == 0xF01p-8,
+            0x0.F_1p4 == 0x0.F1p4
+          ]
+
+    -- negative hexadecimal float
+    print [ -0xF_F.F == -0xFF.F,
+            -0xF_01p-1 == -0xF01p-1,
+            -0x0.F_1p1 == -0x0.F1p1
+          ]
+
+    -- Additional testcase
+    --
+    -- Validity
+    print [ 0.000_1 == 0.0001,
+            1_0.000_1 == 10.0001,
+            1e+23 == 1e+23,
+            1_e+23 == 1e+23,
+            1__e+23 == 1e+23,
+            1.0_e+23 == 1.0e+23,
+            1.0_e+2_3 == 1.0e+23,
+            1_e23 == 1e23,
+            1_e-23 == 1e-23,
+            1_0_e23 == 10e23,
+            1_0_e-23 == 10e-23,
+            0b_01 == 0b01,
+            0b__11 == 0b11,
+            0x_ff == 0xff,
+            0x__ff == 0xff
+          ]
diff --git a/testsuite/tests/parser/should_run/NumericUnderscores0.stdout b/testsuite/tests/parser/should_run/NumericUnderscores0.stdout
new file mode 100644
index 0000000000..76f19a8ad9
--- /dev/null
+++ b/testsuite/tests/parser/should_run/NumericUnderscores0.stdout
@@ -0,0 +1,13 @@
+[True,True,True,True,True]
+[True,True,True]
+[True,True]
+[True,True,True,True]
+[True]
+[True]
+[True]
+[True]
+[True,True,True]
+[True,True,True]
+[True,True,True]
+[True,True,True]
+[True,True,True,True,True,True,True,True,True,True,True,True,True,True,True]
diff --git a/testsuite/tests/parser/should_run/NumericUnderscores1.hs b/testsuite/tests/parser/should_run/NumericUnderscores1.hs
new file mode 100644
index 0000000000..b9d0dca725
--- /dev/null
+++ b/testsuite/tests/parser/should_run/NumericUnderscores1.hs
@@ -0,0 +1,88 @@
+{-# LANGUAGE NumericUnderscores #-}
+{-# LANGUAGE BinaryLiterals #-}
+{-# LANGUAGE MagicHash #-}
+{-# LANGUAGE NegativeLiterals #-}
+
+-- Test for NumericUnderscores extension.
+-- See Trac #14473
+-- This is a testcase for unboxed literals.
+
+import GHC.Types
+
+main :: IO ()
+main = do
+    -- Each case corresponds to the definition of Lexer.x
+    --
+    -- Unboxed ints and words
+    -- decimal int
+    print [ (I# 1_000_000#) == 1000000,
+            (I# 299_792_458#) == 299792458
+          ]
+
+    -- binary int
+    print [ (I# 0b01_0000_0000#) == 0b0100000000,
+            (I# 0b1_11_01_0000_0_111#) == 0b1110100000111
+          ]
+
+    -- octal int
+    print [ (I# 0o1_000_000#) == 0o1000000,
+            (I# 0O1__0#) == 0O10
+          ]
+
+    -- hexadecimal int
+    print [ (I# 0x1_000_000#) == 0x1000000,
+            (I# 0X3fff_ffff#) == 0x3fffffff
+          ]
+
+    -- negative decimal int
+    print [ (I# -1_000_000#) == -1000000
+          ]
+
+    -- negative binary int
+    print [ (I# -0b01_0000_0000#) == -0b0100000000
+          ]
+
+    -- negative octal int
+    print [ (I# -0o1_000_000#) == -0o1000000
+          ]
+
+    -- negative hexadecimal int
+    print [ (I# -0x1_000_000#) == -0x1000000
+          ]
+
+    -- decimal word
+    print [ (W# 1_000_000##) == 1000000,
+            (W# 299_792_458##) == 299792458
+          ]
+
+    -- binary word
+    print [ (W# 0b1_0##) == 0b10
+          ]
+
+    -- octal word
+    print [ (W# 0o1_0##) == 0o10
+          ]
+
+    -- hexadecimal word
+    print [ (W# 0x1_0##) == 0x10
+          ]
+
+    -- Unboxed floats and doubles
+    -- float
+    print [ (F# 3.141_592_653_589_793#) == 3.141592653589793,
+            (F# 3_14e-2#) == 314e-2,
+            (F# 96_485.332_89#) == 96485.33289,
+            (F# 6.022_140_857e+23#) == 6.022140857e+23,
+            (F# -3.141_592#) == -3.141592,
+            (F# -3_14e-2#) == -314e-2,
+            (F# -6.022_140e+23#) == -6.022140e+23
+          ]
+
+    -- double
+    print [ (D# 3_14e-2##) == 314e-2,
+            (D# 96_485.332_89##) == 96485.33289,
+            (D# 6.022_140_857e+23##) == 6.022140857e+23,
+            (D# -3.141_592##) == -3.141592,
+            (D# -3_14e-2##) == -314e-2,
+            (D# -6.022_140e+23##) == -6.022140e+23
+          ]
diff --git a/testsuite/tests/parser/should_run/NumericUnderscores1.stdout b/testsuite/tests/parser/should_run/NumericUnderscores1.stdout
new file mode 100644
index 0000000000..bddde5bccb
--- /dev/null
+++ b/testsuite/tests/parser/should_run/NumericUnderscores1.stdout
@@ -0,0 +1,14 @@
+[True,True]
+[True,True]
+[True,True]
+[True,True]
+[True]
+[True]
+[True]
+[True]
+[True,True]
+[True]
+[True]
+[True]
+[True,True,True,True,True,True,True]
+[True,True,True,True,True,True]
diff --git a/testsuite/tests/parser/should_run/all.T b/testsuite/tests/parser/should_run/all.T
index bcf0bc83f3..0c9e65fd14 100644
--- a/testsuite/tests/parser/should_run/all.T
+++ b/testsuite/tests/parser/should_run/all.T
@@ -12,3 +12,5 @@ test('BinaryLiterals2', [], compile_and_run, [''])
 test('T10807', normal, compile_and_run, [''])
 test('NegativeZero', normal, compile_and_run, [''])
 test('HexFloatLiterals', normal, compile_and_run, [''])
+test('NumericUnderscores0', normal, compile_and_run, [''])
+test('NumericUnderscores1', normal, compile_and_run, [''])