grom/benilla-octo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

comments cite the reference by address, not by private note (#30)

Browse Source 
Comments that pointed into the maintainer's private reverse-engineering notes (a note by name, a
path, a section or finding id, the analysis's own name for a client function, a process word) now
carry the fact itself, cited by the reference client's address, by vmangos file and line, or as
"the reference". Nobody outside could follow those pointers.

Comment text only, plus the test-assert and log strings that carried the same pointers. No claim
changes: a comment the notes have since corrected keeps its wording here, and correcting it is
separate work.

The two reference generators, `scripts/gen-reference-globals.py` and
`scripts/gen-reference-verb-events.py`, named the private notes' layout to find their inputs. They
now take the input files as arguments (`--capture`, `--framexml`, `--names`, `--extents`,
`--disasm`). Run on the same inputs, both write the committed tables row for row; only the two
header lines that named the old environment variable change.

Checked per file:
- the code is identical: each file with its comments stripped compares equal before and after; a
  changed string literal is compared under a mask that proves only literal text moved and every
  literal kept its line count;
- no client address the old text cited was dropped;
- a second reviewer checked every added address against its derivation;
- a detector for the notes' names, paths, ids, function names and process words finds none left;
- `scripts/gates.sh` is green.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sam Sung
2026-09-24 12:14:08 +07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent beb6cc854f
commit 7b66dac9db
887 changed files with 10649 additions and 11632 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/gen-reference-events.py
+1 -1
View File
@@ -78,7 +78,7 @@ def main():
if len(f) >= 2 and f[1]:
catalog.setdefault(int(f[0]), f[1])
if len(catalog) < 300:
sys.exit(f"event-catalog.tsv gave only {len(catalog)} names — wrong path?")
sys.exit(f"{a.catalog} gave only {len(catalog)} names — wrong path?")
# name -> {ids, producers, formats, notes}
ev = {}
scripts/gen-reference-globals.py
+26 -24
View File
@@ -1,7 +1,8 @@
#!/usr/bin/env python3
"""Regenerate `reference/1.12-globals.tsv`: the 1.12.1 client's global namespace, by origin.
WOW_RE=DIR scripts/gen-reference-globals.py [--wow-re DIR] [--out FILE]
scripts/gen-reference-globals.py --capture FILE --framexml DIR [--data DIR] [--drop NAME]...
[--out FILE]
Each name in a capture of the running reference client's in-world `_G` gets one origin:
@@ -21,8 +22,9 @@ Attribution is by definition site in the complete 1.12 shipped-UI corpus:
- LUA_5_0 wins over all of these: FrameXML's `string = getglobal(...)` would otherwise claim
the stdlib table.
The inputs are the maintainer's analysis (`--wow-re` or `$WOW_RE`), not in this repo: the
capture, the extracted FrameXML, and the install's MPQs. FrameXML alone is not the shipped UI:
The capture (`--capture`, with `--drop` naming the capturing addon's own globals) and the
extracted FrameXML (`--framexml`) are the maintainer's, not in this repo; the MPQs are the
install's (`--data`, else `$WOW_DATA`, else `WoW/Data`). FrameXML alone is not the shipped UI:
the twelve `Blizzard_*` addons live in the MPQs, and their `.lua` is reached through
`<Script file=>` in their XML, not their `.toc`. A complete corpus is 233 files; the count is
printed.
@@ -52,11 +54,6 @@ LUA_5_0 = {
"unpack", "xpcall",
}
# The capture addon's own globals (its holder frame, SavedVariables and slash token): the
# instrument, not the client. `__framescript_meta` stays in: it is the client's own shared frame
# metatable.
CAPTURE_OWN = {"W5875CaptureDB", "W5875CaptureHolder", "SLASH_W5875CAP1"}
LUA_ASSIGN = re.compile(r"^[ \t]*([A-Za-z_][A-Za-z0-9_]*)\s*=[^=]", re.M)
LUA_FUNC = re.compile(r"^[ \t]*function\s+([A-Za-z_][A-Za-z0-9_]*)\s*\(", re.M)
LUA_MULTI = re.compile(
@@ -110,9 +107,9 @@ def join_ref(base, ref):
return "/".join(out)
def build_corpus(wowre):
def build_corpus(data, framexml):
"""Every file of the 1.12 shipped UI, as {corpus-relative path: text}."""
archives = [f"{wowre}/WoW/Data/{a}.MPQ" for a in ("patch-2", "patch", "interface", "base")]
archives = [os.path.join(data, f"{a}.MPQ") for a in ("patch-2", "patch", "interface", "base")]
archives = [a for a in archives if os.path.exists(a)]
mpqcat = os.path.join(REPO, "target", "debug", "examples", "mpqcat")
if not os.path.exists(mpqcat):
@@ -129,7 +126,7 @@ def build_corpus(wowre):
return None
files = {}
fx = os.path.join(wowre, "WoW", "_extracted_framexml")
fx = framexml
for fn in sorted(os.listdir(fx)):
p = os.path.join(fx, fn)
if os.path.isfile(p):
@@ -212,25 +209,30 @@ def composed(name, roots, suffixes, depth=0):
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--wow-re", default=os.environ.get("WOW_RE"))
ap.add_argument("--capture", help="a capture of the running client's in-world _G")
ap.add_argument("--framexml", help="the shipped FrameXML, extracted to a folder")
ap.add_argument("--data", help="the install's Data folder, which holds the MPQs",
default=os.environ.get("WOW_DATA") or os.path.join(REPO, "WoW", "Data"))
# The capturing addon's own globals (its holder frame, SavedVariables and slash token) are the
# instrument, not the client. `__framescript_meta` stays in: it is the client's own shared
# frame metatable.
ap.add_argument("--drop", action="append", default=[], help="a global of the capturing addon")
ap.add_argument("--out", default=os.path.join(REPO, "reference", "1.12-globals.tsv"))
args = ap.parse_args()
if not args.wow_re:
if not args.capture or not args.framexml:
sys.exit(
"this table is derived from the RE repo's runtime capture of the reference client's "
"_G: pass --wow-re or set WOW_RE. The committed reference/1.12-globals.tsv is the "
"surface benilla tracks."
"this table is derived from a runtime capture of the reference client's _G and the "
"extracted FrameXML, which are not in this repo: pass --capture and --framexml. The "
"committed reference/1.12-globals.tsv is the surface benilla tracks."
)
fixture = os.path.join(args.wow_re, "WoW", "_w5875_fixtures", "item13", "W5875Capture.lua")
if not os.path.exists(fixture):
sys.exit(f"no capture fixture at {fixture} — is --wow-re right?")
if not os.path.exists(args.capture):
sys.exit(f"no capture at {args.capture}")
print("building the 1.12 shipped-UI corpus...")
corpus = build_corpus(args.wow_re)
corpus = build_corpus(args.data, args.framexml)
print(f" TOTAL: {len(corpus)} files" + ("" if len(corpus) == 233 else " ** expected 233 **"))
recs = [(n, t) for n, t in captured_globals(fixture) if n not in CAPTURE_OWN]
recs = [(n, t) for n, t in captured_globals(args.capture) if n not in set(args.drop)]
defined, suffixes = shipped_names(corpus)
print(f"captured globals: {len(recs)} shipped definitions: {len(defined)} "
f"$parent suffixes: {len(suffixes)}")
@@ -262,8 +264,8 @@ def main():
os.makedirs(os.path.dirname(args.out), exist_ok=True)
with open(args.out, "w") as f:
f.write("# The 1.12.1 client's global namespace. Generated by scripts/gen-reference-globals.py.\n")
f.write("# Source: a capture of the running client's in-world _G (the maintainer's analysis,\n")
f.write("# `WOW_RE`), attributed against the complete 1.12 shipped-UI corpus.\n")
f.write("# Source: a capture of the running client's in-world _G (the maintainer's),\n")
f.write("# attributed against the complete 1.12 shipped-UI corpus.\n")
f.write("# Also every name that corpus defines but the capture lacks, as type `lod`: the\n")
f.write("# twelve Blizzard_* addons are LoadOnDemand, so a live dump misses them unless\n")
f.write("# their windows were opened.\n")
scripts/gen-reference-verb-events.py
+22 -21
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
"""Regenerate `reference/1.12-verb-events.tsv`: the events 1.12 Lua verbs fire, keyed by verb.
WOW_RE=DIR scripts/gen-reference-verb-events.py [--wow-re DIR] [--out FILE]
scripts/gen-reference-verb-events.py --names FILE --extents FILE... --disasm FILE [--out FILE]
A row says the reference fires `event` on the call path of the registered Lua binding `verb`:
from the verb's body (`shape=body`), or from a helper that only registered verbs call
@@ -21,14 +21,13 @@ and the unit-field bridge and token fan-out ids below 0xb6, which are state even
A floor, not a census: a verb with no row may still fire an event, because the fire sites are
only the two signal helpers' literal call sites and `shape=helper` stops one call deep.
The fire sites and the binding shapes are the vendored `reference/` tables; the function names,
extents and disassembly are the maintainer's analysis (`--wow-re` or `$WOW_RE`), not in this
repo, so this is a manual regeneration.
The fire sites and the binding shapes are the vendored `reference/` tables; the function names
(`--names`), extents (`--extents`, the `size=` on each `fn` row) and disassembly (`--disasm`) are
the maintainer's analysis, not in this repo, so this is a manual regeneration.
"""
import argparse
import bisect
import collections
import glob
import os
import re
import sys
@@ -62,26 +61,28 @@ def rows(path):
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--wow-re", default=os.environ.get("WOW_RE"))
ap.add_argument("--names", help="the function names: `addr<TAB>kind` rows, `func` a start")
ap.add_argument("--extents", nargs="+", default=[], help="tables whose `fn` rows carry size=")
ap.add_argument("--disasm", help="the client's disassembly, one objdump line per instruction")
ap.add_argument("--out", default=os.path.join(REFERENCE, "1.12-verb-events.tsv"))
a = ap.parse_args()
if not a.wow_re:
if not (a.names and a.extents and a.disasm):
sys.exit(
"this table is derived from the RE repo's function names, ledgers and disassembly: "
"pass --wow-re or set WOW_RE. The committed reference/1.12-verb-events.tsv is the "
"surface benilla tracks."
"this table is derived from the maintainer's function names, extents and disassembly, "
"which are not in this repo: pass --names, --extents and --disasm. The committed "
"reference/1.12-verb-events.tsv is the surface benilla tracks."
)
starts = set()
for f in rows(os.path.join(a.wow_re, "re/names/out/names-5875.tsv")):
for f in rows(a.names):
if len(f) >= 2 and f[1] == "func":
starts.add(int(f[0], 16))
if len(starts) < 1000:
sys.exit(f"names-5875.tsv gave only {len(starts)} functions — wrong path?")
sys.exit(f"{a.names} gave only {len(starts)} functions — wrong file?")
# Function extents: `size=` on the `fn` rows of the analysis's ledgers.
# Function extents: `size=` on the `fn` rows of the extent tables.
extent = {} # fn start -> size in bytes
for path in glob.glob(os.path.join(a.wow_re, "system/*/ledger.tsv")):
for path in a.extents:
for line in open(path, encoding="utf-8", errors="replace"):
if not line.startswith("fn\t"):
continue
@@ -91,7 +92,7 @@ def main():
fn = int(f[1], 16)
extent[fn] = max(extent.get(fn, 0), int(m.group(1)))
if len(extent) < 5000:
sys.exit(f"the ledgers gave only {len(extent)} sized functions — wrong path?")
sys.exit(f"the extent tables gave only {len(extent)} sized functions — wrong files?")
starts |= set(extent)
verbs = {} # fn address -> name
@@ -105,7 +106,7 @@ def main():
continue
verbs.setdefault(fn, f[0])
if len(verbs) < 1000:
sys.exit(f"binding-shapes.tsv gave only {len(verbs)} verbs — wrong path?")
sys.exit(f"1.12-shapes.tsv gave only {len(verbs)} verbs — wrong path?")
starts |= set(verbs)
sites = [] # (site va, event name)
@@ -117,7 +118,7 @@ def main():
continue
sites.append((va, f[3]))
if len(sites) < 300:
sys.exit(f"event-firesites.tsv gave only {len(sites)} sites — wrong path?")
sys.exit(f"1.12-event-firesites.tsv gave only {len(sites)} sites — wrong path?")
# One pass over the disassembly: every call/jmp edge, and every padding boundary (an `int3`
# run, or a `nop` run right after a `ret`). A `nop` run after a `jmp` is a loop head's
@@ -126,9 +127,9 @@ def main():
after_pad = False
ended = False # the previous real instruction was a `ret`
n_lines = 0
disasm = os.path.join(a.wow_re, "system/ui/scratch/disasm-full.txt")
disasm = a.disasm
if not os.path.exists(disasm):
sys.exit(f"no disassembly at {disasm} — is --wow-re right?")
sys.exit(f"no disassembly at {disasm}")
for line in open(disasm, encoding="utf-8", errors="replace"):
m = LINE.match(line)
if not m:
@@ -146,7 +147,7 @@ def main():
if b:
edges.append((va, b.group(1), int(b.group(2), 16)))
if n_lines < 1_000_000:
sys.exit(f"disasm-full.txt gave only {n_lines} lines — wrong path?")
sys.exit(f"{disasm} gave only {n_lines} lines — wrong file?")
starts |= {t for _, k, t in edges if k == "call"}
ordered = sorted(starts)
@@ -192,7 +193,7 @@ def main():
fh.write(
"# The 1.12.1 client's FrameScript events fired from inside a Lua verb, keyed by the verb.\n"
"# Generated by scripts/gen-reference-verb-events.py from reference/1.12-shapes.tsv,\n"
"# reference/1.12-event-firesites.tsv and the maintainer's analysis (`WOW_RE`).\n"
"# reference/1.12-event-firesites.tsv and the maintainer's analysis.\n"
"#\n"
"# shape body = a fire site inside the verb's own extent.\n"
"# helper = a fire site inside a function that only registered verbs call\n"