lua: a numeral ends where its digits do, so 1then is two tokens

`AdvancedTradeSkillWindow2.lua:3878` is `if Item.ReplaceWords == 1then`, with no space, and it
compiles on the reference. Its `read_numeral` (`0x6ffee0`) is 5.0's: digits, at most one `.` and its
digits, then — given an `e`/`E` — the marker, an optional sign (the `0x2b`/`0x2d` test at `0x6fff04`,
reached with the marker already saved) and its own digit loop. The only character-class test in the
whole routine is the ctype digit mask at `0x6fff47`; past it the buffer is terminated and handed to
`luaO_str2d` (`0x6f5900`). 5.1 instead ends with
`while (isalnum(ls->current) || ls->current == '_') save_and_next(ls);`, which swallowed `1then`
into the numeral and failed it — so we raised `malformed number` and lost a 4000-line file and every
global it defines, and its own XML then raised four more times calling them. The addon was dead.

The ninth hunk of the `lua-src` fork, which is where a grammar difference belongs: `BENILLA.md` is
explicit that rewriting the chunk source means parsing Lua with a regex. A replacement, not a
deletion — 5.1 had no exponent-digit loop of its own, so deleting the slurp alone would have cost
`1e2`, which is what the new test caught.

The same boundary is the deliberate half: 1.12 has **no hex literals**. The digit loop stops at the
`x`, so the reference reads `0xFF` as `0` and the name `xFF` and cannot parse the statement around
it. A comment- and string-stripping scan of the installed corpus finds hex in code in exactly two
files, `pfQuest/toolbox/pngLua/deflate.lua` and `numberlua.lua`, which no `.toc` row reaches — zero
loaded sites, the same standard the fork's three other deletions were held to. Our own shipped Lua
and his saved variables have none either. Two of our own test chunks did, and are now decimal:
no chunk of benilla's Lua may use a construct the client's parser rejects.

Verified: the whole chain still loads with no failure, the `WOW_DATA` failure set is unchanged, and
`AdvancedTradeSkillWindow2` now publishes `ATSW_OnLoad` and `ATSWTypeColor`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SgstFWtn1RxEJ5kVrpgYwn

crates/benilla-ui/src/script/compat/classic_api/spell.rs

@@ -1065,7 +1065,8 @@ mod tests {
     fn school_string_covers_the_seven_schools_and_nothing_else() {
         let s = UiScript::new().unwrap();
         assert_eq!(
-            s.eval::<String>("return C_Spell.GetSchoolString(0x4)")
+            // 4, not `0x4`: 1.12's lexer has no hex literals, so a chunk of ours must not use one.
+            s.eval::<String>("return C_Spell.GetSchoolString(4)")
                 .unwrap(),
             "SCHOOL_MASK_FIRE"
         );

crates/benilla-ui/src/script/lua50.rs

@@ -636,12 +636,13 @@ mod tests {
         }
         // Two's complement, signed out: `bnot(0)` is -1, not 4294967295.
         assert_eq!(s.eval::<i64>("return bit.bnot(0)").unwrap(), -1);
-        assert_eq!(
-            s.eval::<i64>("return bit.band(0x1234, 0xFF)").unwrap(),
-            0x34
-        );
-        assert_eq!(s.eval::<i64>("return bit.bor(0xF0, 0x0F)").unwrap(), 0xFF);
-        assert_eq!(s.eval::<i64>("return bit.bxor(0xFF, 0x0F)").unwrap(), 0xF0);
+        // Decimal in the chunks, because 1.12's lexer has no hex literals (see
+        // `a_numeral_ends_where_the_digits_do_so_a_glued_keyword_is_its_own_token`) and no chunk of
+        // benilla's own Lua may use a construct the client's parser rejects. Hex stays on the Rust
+        // side, where the masks are what they are.
+        assert_eq!(s.eval::<i64>("return bit.band(4660, 255)").unwrap(), 0x34);
+        assert_eq!(s.eval::<i64>("return bit.bor(240, 15)").unwrap(), 0xFF);
+        assert_eq!(s.eval::<i64>("return bit.bxor(255, 15)").unwrap(), 0xF0);
         assert_eq!(s.eval::<i64>("return bit.lshift(1, 4)").unwrap(), 16);
         // `rshift` is logical, `arshift` propagates the sign.
         assert_eq!(s.eval::<i64>("return bit.rshift(-1, 28)").unwrap(), 15);
@@ -721,6 +722,50 @@ mod tests {
         assert_eq!(s.eval::<i64>("return string.len('abcd')").unwrap(), 4);
     }
 
+    /// Where the client's `read_numeral` (`0x6ffee0`) ends a numeral, and where 5.1's did not.
+    /// Its only character-class test is the ctype digit mask (`0x6fff47`); the tail terminates the
+    /// buffer and calls `luaO_str2d` (`0x6f5900`) with no further slurp, so `1then` is the number
+    /// 1 and the name `then`. 5.1 ended with `while (isalnum(c) || c == '_') save_and_next(ls);`,
+    /// gave the whole run to the numeral and failed it — which killed
+    /// `AdvancedTradeSkillWindow2.lua:3878`, `if Item.ReplaceWords == 1then`, and with it a
+    /// 4000-line file and every global it defines.
+    #[test]
+    fn a_numeral_ends_where_the_digits_do_so_a_glued_keyword_is_its_own_token() {
+        let s = UiScript::new().unwrap();
+        // The addon's own line, verbatim.
+        assert_eq!(
+            s.eval::<i64>("local Item = { ReplaceWords = 1 } if Item.ReplaceWords == 1then return 7 end return 0")
+                .unwrap(),
+            7
+        );
+        for (chunk, want) in [
+            ("local n = 0 for i = 1, 3do n = n + i end return n", 6),
+            ("if 1~=2and 3==3 then return 1 end return 0", 1),
+            ("local t = { 5 } return t[1]+0", 5),
+        ] {
+            assert_eq!(
+                s.eval::<i64>(chunk)
+                    .unwrap_or_else(|e| panic!("{chunk}: {e}")),
+                want
+            );
+        }
+        // A finished exponent is still one numeral, and the name behind it still its own token.
+        assert_eq!(
+            s.eval::<f64>("if 1e2then return 1e2 end return 0").unwrap(),
+            100.0
+        );
+        // The same boundary costs 1.12 its hex literals: the digit loop stops at the `x`, so the
+        // client reads `0` and the name `xFF` and cannot parse the statement. Zero loaded sites in
+        // the corpus (only pfQuest's unlisted `toolbox/pngLua`, which no toc row reaches).
+        assert!(s
+            .eval::<bool>(r#"return loadstring("return 0xFF") == nil"#)
+            .unwrap());
+        // A malformed numeral still raises, and names the prefix the client would have named.
+        assert!(s
+            .eval::<bool>(r#"return loadstring("return 1.2.3") == nil"#)
+            .unwrap());
+    }
+
     /// The 5.0 grammar the stock files use still parses, including the three constructs the fork
     /// restores: the iterator-less generic-for, nested long strings (`LUA_COMPAT_LSTR`) and the
     /// constructor's extra semicolon.

third_party/lua-src/BENILLA.md

@@ -7,7 +7,7 @@ the version mlua 0.11 resolves to. Wired in through `[patch.crates-io]` in the w
 
 benilla's Lua has to accept exactly the grammar the 1.12.1 client's Lua accepts, no more and no
 less. Every hunk below is that one rule: three restore 5.0 constructs 5.1 deleted, three delete 5.1
-constructs 5.0 never had. A grammar difference in either direction is invisible to every instrument
+constructs 5.0 never had, and one puts a token boundary back where the client has it. A grammar difference in either direction is invisible to every instrument
 until an addon trips over it, and the addon corpus asks about the difference directly.
 
 The 1.12 addon corpus is Lua 5.0 code. It uses the iterator-less generic-for:
@@ -33,12 +33,12 @@ whole of mlua's surface, stay exactly as upstream.
 
 ## The delta, and the command that proves it
 
-Eight hunks in three files. `src/lib.rs` additionally differs by having Lua 5.2/5.3/5.4/5.5 stripped
+Nine hunks in four files. `src/lib.rs` additionally differs by having Lua 5.2/5.3/5.4/5.5 stripped
 from the `Version` enum (with their source trees deleted): benilla builds 5.1 and only 5.1, and a
 fork that still offered the other four would answer a request for one with a missing directory at
 build time instead of a compile error.
 
-Five hunks restore what 5.1 changed or removed:
+Six hunks restore what 5.1 changed or removed:
 
 | file | hunk | why |
 |---|---|---|
@@ -47,6 +47,7 @@ Five hunks restore what 5.1 changed or removed:
 | `luaconf.h` | `LUA_QL(x)` back to 5.0's `` `x' `` from 5.1's `'x'` | every Lua error message quotes a program element, and `WoW.exe`'s own `.rdata` carries all five formats in the 5.0 spelling |
 | `lparser.c` | 5.0's compat-semicolon skip restored at the top of `constructor()`'s field loop, 5.0's own line verbatim | one extra `;` after a field separator inside a table constructor (`Back_Title = AL["Factions"];;`, which AtlasLoot writes) is accepted by the client and was a parse error here |
 | `lparser.c` | `recfield`'s `cc->nh++` moved back inside its `TK_NAME` arm, 5.0's own placement | a `[expr] = value` constructor field credits neither `OP_NEWTABLE` size hint in 5.0, so the table is born on the dummy node |
+| `llex.c` | `read_numeral`'s trailing `while (isalnum(c) \|\| c == '_')` slurp replaced by 5.0's own exponent-digit loop | the client ends a numeral at its digits, so `if Item.ReplaceWords == 1then` (no space, `AdvancedTradeSkillWindow2.lua:3878`) compiles there and raised `malformed number` here, costing a 4000-line file and every global it defines |
 
 Three delete what 5.1 added, all in `lparser.c`, all byte-read out of the client's own parser
 (`simpleexp 0x6fd240`, `getunopr 0x6fe0a0`, `getbinopr 0x6fe0c0`), all landing on 5.0's own
@@ -71,7 +72,7 @@ diff -r third_party/lua-src/lua-5.1.5 \
   ~/.cargo/registry/src/*/lua-src-550.0.0/lua-5.1.5
 ```
 
-That must print exactly the eight hunks in the tables above and nothing else.
+That must print exactly the nine hunks in the tables above and nothing else.
 
 ## The generic-for, in detail
 
@@ -81,3 +82,35 @@ substitution test is a bare type-tag equality that never consults a metatable (a
 entry (an addon assigning `next = myfn` changes every later generic-for in the session); and userdata
 is not substituted. The substitution happens in `OP_TFORPREP`, an opcode 5.1 deleted, so the top of
 `OP_TFORLOOP` is where 5.1 can host it.
+
+## The numeral boundary, in detail
+
+`read_numeral` is one routine in each dialect, and they part at one line. The client's
+(`0x6ffee0`) is 5.0's: digits, at most one `.` and its digits, then — if an `e`/`E` is there — the
+marker, an optional sign (the `0x2b`/`0x2d` test at `0x6fff04`, reached with the marker already
+saved) and its own digit loop. Its only character-class test is the ctype digit mask `0x4` at
+`0x6fff47`; past the loop it NUL-terminates the buffer and calls `luaO_str2d` (`0x6f5900`), raising
+`malformed number` (`0x6fffa9`, string `0x8721c8`) when that fails. 5.1 instead ends with
+`while (isalnum(ls->current) || ls->current == '_') save_and_next(ls);`, which carries the exponent
+digits *and* swallows any word glued to the tail.
+
+Two consequences, and both are the client's:
+
+- `1then` is the number 1 and the name `then`. One corpus site —
+  `AdvancedTradeSkillWindow2.lua:3878` — and it cost the whole file: a syntax error leaves every
+  global it defines undefined, so its own XML then raised four more times calling them.
+- 1.12 has **no hex literals**. The digit loop stops at the `x`, so `0xFF` is `0` and the name
+  `xFF`, and the statement around it does not parse. A comment- and string-stripping scan of the
+  installed corpus finds hex in code in exactly two files, `pfQuest/toolbox/pngLua/deflate.lua`
+  and `numberlua.lua`, which no `.toc` row reaches (only `toolbox/extractor.lua` loads them, and it
+  is unlisted too) — so zero loaded sites, which is why this deletion is as safe as the other
+  three. Our own shipped Lua has none either.
+
+The slurp could not simply be deleted: 5.1 had no exponent-digit loop of its own, so `1e2` would
+have lexed as `1e`. The hunk is a replacement, and the test is
+`script::lua50::tests::a_numeral_ends_where_the_digits_do_so_a_glued_keyword_is_its_own_token`.
+
+One difference inside the routine is left alone: 5.1's opening `do ... while (isdigit(c) || c ==
+'.')` takes *repeated* dots where 5.0 takes at most one, so `1..2` is one malformed token here and
+`1.` followed by a `..` there. Both raise; only the message differs, and a comment- and
+string-stripping scan of the corpus finds no site, so there is nothing to tell apart.

third_party/lua-src/lua-5.1.5/llex.c

@@ -196,10 +196,22 @@ static void read_numeral (LexState *ls, SemInfo *seminfo) {
   do {
     save_and_next(ls);
   } while (isdigit(ls->current) || ls->current == '.');
-  if (check_next(ls, "Ee"))  /* `E'? */
+  /* benilla: 5.1's trailing `while (isalnum(ls->current) || ls->current == '_')
+  ** save_and_next(ls);' is replaced by 5.0's own exponent-digit loop, which is what the client's
+  ** read_numeral (0x6ffee0) has: it saves the `E', takes an optional sign (the 0x2b/0x2d test at
+  ** 0x6fff04, which runs with the marker already in the buffer) and then runs its digit loop --
+  ** the ctype digit mask 0x4 at 0x6fff47, the only character-class test in the whole routine --
+  ** and stops. Nothing slurps the following word, so the client reads `1then' as the number 1 and
+  ** the name `then', which is why `if Item.ReplaceWords == 1then' compiles there; and nothing
+  ** slurps an `x' either, so 1.12 has no hex literals, reading `0xFF' as 0 and the name `xFF'.
+  ** 5.1 gave both runs to the numeral and then failed it. Its slurp also carried the exponent's
+  ** own digits, so deleting it alone would have cost us `1e2' -- hence the loop, not a deletion.
+  */
+  if (check_next(ls, "Ee")) {  /* `E'? */
     check_next(ls, "+-");  /* optional exponent sign */
-  while (isalnum(ls->current) || ls->current == '_')
-    save_and_next(ls);
+    while (isdigit(ls->current))
+      save_and_next(ls);
+  }
   save(ls, '\0');
   buffreplace(ls, '.', ls->decpoint);  /* follow locale for decimal point */
   if (!luaO_str2d(luaZ_buffer(ls->buff), &seminfo->r))  /* format error? */