comments: the generators, the tools and the reference table headers
Docstrings keep their usage and exit codes; table headers keep the column contract and the client addresses and lose the dates, the correction history and every path into another repository. A generated header changes in its generator's template, so each of the four generators still reproduces its output byte for byte, and no data row changed in any of the seven tables. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,17 +1,14 @@
|
||||
//! The 5875 message catalog, row for row.
|
||||
//! The 1.12 client's message catalog, row for row.
|
||||
//!
|
||||
//! @generated by `scripts/gen-message-catalog.py` from `reference/1.12-message-catalog.tsv`
|
||||
//! (wow-5875-re's `re/ui/message-catalog.tsv`, extracted out of `WoW.exe` there).
|
||||
//! Do not edit by hand — regenerate.
|
||||
//! @generated by `scripts/gen-message-catalog.py` from `reference/1.12-message-catalog.tsv`.
|
||||
//! Do not edit by hand; regenerate.
|
||||
|
||||
use super::{MessageRecord, MsgKind};
|
||||
|
||||
/// Every row of `0xb4b498`, indexed by message id (the ids are dense `0..465`, so
|
||||
/// `CATALOG[id]` *is* the reference's `0xb4b498 + 20*id`).
|
||||
/// Every row of `0xb4b498`, indexed by message id: the ids are dense `0..465`, so `CATALOG[id]`
|
||||
/// is the reference's `0xb4b498 + 20*id`.
|
||||
///
|
||||
/// `rustfmt` is held off so that a regeneration is a no-op diff: one row per line is the shape
|
||||
/// this script emits, and a formatter that rewrapped it would make the file drift from its own
|
||||
/// generator.
|
||||
/// `rustfmt` is off: the generator emits one row per line, and a rewrap would drift from it.
|
||||
#[rustfmt::skip]
|
||||
pub static CATALOG: [MessageRecord; 465] = [
|
||||
MessageRecord { id: 0, key: "ERR_INV_FULL", kind: MsgKind::Error, sound: None, type_tag: 0x00, chat_type: 10 },
|
||||
|
||||
@@ -1,12 +1,11 @@
|
||||
# FrameScript C->Lua event catalog — eventId -> name, extracted from WoW.exe 1.12.1.5875
|
||||
# source: the event-name pointer array at .data 0xbe1198 (stride 4, 0x225=549 slots), RE 2026-06-21,
|
||||
# BASE CORRECTED 2026-07-21 (was 0xbe11d8 -> every id was 16 too low; see
|
||||
# system/object-layer/scratch/lootroll-chat-and-lifecycle.md §3). The base is VERIFIED at
|
||||
# 0x48fea1-0x48feab: `mov edx,0x225; mov ecx,0xbe1198; call 0x703d90`, and 0x703d90 builds the
|
||||
# FrameScript C->Lua event catalog of the 1.12.1 client (WoW.exe build 5875): eventId -> name.
|
||||
# Source: the event-name pointer array at .data 0xbe1198 (stride 4, 0x225 = 549 slots), passed
|
||||
# at 0x48fea1-0x48feab (`mov edx,0x225; mov ecx,0xbe1198; call 0x703d90`). 0x703d90 builds the
|
||||
# runtime 16-byte-stride dispatch table [0xceef68] as table[i].name = strdup(nameArray[i])
|
||||
# (0x703e06-0x703e2e), so the id passed to FrameScript_SignalEvent 0x703e50 / 0x703f50 IS this index.
|
||||
# NB slots 0..15 carry no `mov imm32` name write in this scan (they are set from a register at
|
||||
# 0x51acbf and neighbours); coverage is unchanged from the pre-correction extraction (383 rows).
|
||||
# (0x703e06-0x703e2e), so the id passed to FrameScript_SignalEvent 0x703e50 / 0x703f50 is
|
||||
# this index.
|
||||
# Slots 0..15 carry no `mov imm32` name write in this scan (they are set from a register at
|
||||
# 0x51acbf and neighbours).
|
||||
0 UNIT_PET
|
||||
2 UNIT_PET
|
||||
16 UNIT_HEALTH
|
||||
|
||||
|
@@ -1,50 +1,39 @@
|
||||
# C->Lua event fire sites — every call AND TAIL-JMP to FrameScript_SignalEvent (0x703e50, ecx=eventId)
|
||||
# (the 85 `jmp 0x703e50` sites were absent until 2026-08-21 — the census had been written over `call`
|
||||
# alone, so a tail-jmp fire read as "this event has no fire site here"; see re/audit/re-sweep-status.md)
|
||||
# and SignalEvent2 (0x703f50, [esp]=eventId + printf-style arg_format). eventId resolved via event-catalog.tsv.
|
||||
# eventId "dyn" = computed (e.g. a UnitField update loop); blank name = eventId has no entry in the name table.
|
||||
# C->Lua event fire sites of the 1.12.1 client (build 5875): every call and tail-jmp to
|
||||
# FrameScript_SignalEvent (0x703e50, ecx=eventId) and SignalEvent2 (0x703f50, [esp]=eventId +
|
||||
# printf-style arg_format). eventId is resolved via 1.12-event-catalog.tsv; "dyn" = computed
|
||||
# (e.g. a UnitField update loop); a blank name = the id has no entry in the name table.
|
||||
#
|
||||
# CAVEAT (2026-09-04, staticpopup round): the `arg_format` column is the FORMAT STRING the site pushes,
|
||||
# which is a declaration, NOT a census of what the caller actually pushed. The marshaller 0x704f10
|
||||
# consumes one vararg per directive (%f eight bytes, everything else four), so a site whose format
|
||||
# declares more than its caller pushed hands Lua undefined values. `re/audit/signalevent2-fmt-args.py`
|
||||
# audits all 149 sites (0 unadjudicated) and finds exactly THREE such sites -- read that column with
|
||||
# these in mind: 0x496230 TRADE_REPLACE_ENCHANT "%s%s%d" declares 3 and pushes 2; 0x5e4527 and
|
||||
# 0x5e7960 UPDATE_TICKET "%d%s%f%f%f%d%d" declare 10 dwords and push 7 (their sibling 0x5e45e2 pushes
|
||||
# the full 12). Every other site's format matches its caller. See method.md §6.
|
||||
# arg_format is the format string the site pushes: a declaration, not a census of what the
|
||||
# caller pushed. The marshaller 0x704f10 consumes one vararg per directive (%f eight bytes,
|
||||
# everything else four), so a format that declares more than its caller pushed hands Lua
|
||||
# undefined values. Of the 149 SignalEvent2 sites exactly three do: 0x496230
|
||||
# TRADE_REPLACE_ENCHANT "%s%s%d" declares 3 and pushes 2; 0x5e4527 and 0x5e7960 UPDATE_TICKET
|
||||
# "%d%s%f%f%f%d%d" declare 10 dwords and push 7 (their sibling 0x5e45e2 pushes the full 12).
|
||||
# Every other site's format matches its caller.
|
||||
#
|
||||
# CAVEAT (2026-09-03, found while carving `UpdateSpells` for benilla; the answer was already on `main`
|
||||
# in system/glue/): the `name` column resolves every id through event-catalog.tsv, which is the
|
||||
# IN-GAME table (0xbe1198, 549 entries). 0x703d90 is called TWICE with different populations --
|
||||
# 0x46a88f with (0xb41e70, 26) for the GLUE phase and 0x48feab with (0xbe1198, 549) for the world --
|
||||
# and it rebuilds [0xceef68] wholesale from whichever it is handed, so THE SAME INTEGER NAMES A
|
||||
# DIFFERENT EVENT IN THE TWO PHASES and 0x703e50 bounds-checks nothing. 22 rows here sit in a
|
||||
# `Source\Glue\` TU (0x46aa34; 0x46bc9d-0x46c52e inside 0x46b930; 0x46e73d; 0x47461e) and their ids
|
||||
# belong to the GLUE space: name them from system/glue/scratch/glue-event-name-map.md, never from
|
||||
# event-catalog.tsv. 0x46aa34 was the one row where that produced a WRONG name (id 22 read as
|
||||
# UNIT_MAXHEALTH; it is FRAMES_LOADED -- glue-event-name-map.md's own control, with GlueParent.lua:94
|
||||
# as the independent oracle); it is blanked below. The other 21 were already blank, because those ids
|
||||
# are NULL slots in the in-game table. Owed: the glue node names them in its own artifact.
|
||||
# The name column resolves every id through the in-game table (0xbe1198, 549 entries). 0x703d90
|
||||
# is called twice with different populations, 0x46a88f with (0xb41e70, 26) for the glue phase
|
||||
# and 0x48feab with (0xbe1198, 549) for the world, and rebuilds [0xceef68] wholesale from
|
||||
# whichever it is handed, so the same integer names a different event in the two phases, and
|
||||
# 0x703e50 bounds-checks nothing. 22 rows here sit in a `Source\Glue\` TU (0x46aa34;
|
||||
# 0x46bc9d-0x46c52e inside 0x46b930; 0x46e73d; 0x47461e): their ids belong to the glue space and
|
||||
# are never named from 1.12-event-catalog.tsv; their names are blank here. 0x46aa34 fires id 22,
|
||||
# which is FRAMES_LOADED in the glue table (GlueParent.lua:94), not the in-game UNIT_MAXHEALTH.
|
||||
#
|
||||
# CAVEAT 2 (2026-09-03, carving UNIT_FLAGS for benilla): AN EVENT WITH NO ROW HERE IS NOT AN EVENT
|
||||
# WITH NO PRODUCER. This census is over the two signal helpers' own call/jmp sites, and the whole
|
||||
# per-field UNIT_* family reaches them through a WRAPPER: 0x515e50(&guid, id) fans out over the unit
|
||||
# tokens and fires 0x703f50(id, "%s", token) at 0x515e7a -- the "?" row below. Its 38 callers carry
|
||||
# the id in edx, and one of them, 0x51bd5b in the UpdateField watch callback 0x51bd50, computes it as
|
||||
# (registered field byte offset >> 2). So ids 0x00-0xb5 -- every named unit descriptor field,
|
||||
# UNIT_FLAGS 40 included -- are produced there and appear nowhere in this file.
|
||||
# SCOPE WIDENED 2026-09-07 (PLAYER_FLAGS round, benilla dispatch): "0x00-0xb5" is the COMPUTED-id
|
||||
# subset only, and reading it as the whole absent set is a trap -- most 0x515e50 sites carry a
|
||||
# LITERAL id in edx, and those are absent from this file too, at any magnitude. The literal set is
|
||||
# {0x10 0x16 0x1c 0x1d 0x1e 0x29 0xb7 0xb8 0xb9 0xba 0xbb 0x159 0x197 0x20a}. Notably 0x197 = 407
|
||||
# PLAYER_FLAGS_CHANGED (0x5eea35 mov edx,0x197 / 0x5eea3d call 0x515e50, inside the PLAYER_FLAGS
|
||||
# delta handler 0x5ee990) sits OUTSIDE 0x00-0xb5, so a reader checking 407 against the range above
|
||||
# would wrongly conclude it has no producer. It has one; it is simply not a SignalEvent call site.
|
||||
# See system/object-layer/scratch/player-flags-delta-arms.md. Mechanism, the
|
||||
# bounded four-way census that makes 0x51bd50 the SOLE producer of 40, and the fire condition
|
||||
# (a repe cmpsb of the watched span against the object's shadow, at 0x4655bb) are in
|
||||
# system/ui/scratch/unit-field-event-bridge.md. Same shape for 0x5de4b0 (id = its userdata,
|
||||
# arg1 hardcoded "player") and 0x49b199 (id from the table at 0x8062e0).
|
||||
# An event with no row here is not an event with no producer. This covers the two signal
|
||||
# helpers' own call/jmp sites, and the per-field UNIT_* family reaches them through a wrapper:
|
||||
# 0x515e50(&guid, id) fans out over the unit tokens and fires 0x703f50(id, "%s", token) at
|
||||
# 0x515e7a, the "?" row below. Its 38 callers carry the id in edx; one of them, 0x51bd5b in the
|
||||
# UpdateField watch callback 0x51bd50, computes it as (registered field byte offset >> 2), so
|
||||
# ids 0x00-0xb5 (every named unit descriptor field, UNIT_FLAGS 40 included) are produced there
|
||||
# and appear nowhere in this file. 0x51bd50 is the sole producer of 40; its fire condition is a
|
||||
# repe cmpsb of the watched span against the object's shadow, at 0x4655bb. Most other
|
||||
# 0x515e50 sites carry a literal id in edx, absent here too at any magnitude: {0x10 0x16 0x1c
|
||||
# 0x1d 0x1e 0x29 0xb7 0xb8 0xb9 0xba 0xbb 0x159 0x197 0x20a}. So 0x197 = 407
|
||||
# PLAYER_FLAGS_CHANGED (0x5eea35 mov edx,0x197 / 0x5eea3d call 0x515e50, inside the
|
||||
# PLAYER_FLAGS delta handler 0x5ee990) lies outside 0x00-0xb5 and still has a producer. Same
|
||||
# shape for 0x5de4b0 (id = its userdata, arg1 hardcoded "player") and 0x49b199 (id from the
|
||||
# table at 0x8062e0).
|
||||
fire_site_va dispatcher eventId name arg_format
|
||||
0x4018df SignalEvent2 259 TIME_PLAYED_MSG %d%d
|
||||
0x46aa34 SignalEvent 22
|
||||
|
||||
|
Can't render this file because it contains an unexpected character in line 3 and column 38.
|
@@ -1,14 +1,14 @@
|
||||
# The 1.12.1 client's FrameScript events and the ARGUMENTS their producers push.
|
||||
# GENERATED — scripts/gen-reference-events.py. Source: wow-5875-re's
|
||||
# re/events/event-catalog.tsv + event-firesites.tsv. Decision 2140.
|
||||
# The 1.12.1 client's FrameScript events and the arguments their producers push.
|
||||
# Generated by scripts/gen-reference-events.py from reference/1.12-event-catalog.tsv
|
||||
# and reference/1.12-event-firesites.tsv.
|
||||
#
|
||||
# arg_formats `|`-separated, one per distinct producer shape. `()` = zero Lua values.
|
||||
# Directives: %s string, %d/%u number, %f number. An event with two
|
||||
# producers of different shapes carries both, and which one a given
|
||||
# transition takes is a question about the transition, not the name.
|
||||
# producers of different shapes carries both; which one fires depends
|
||||
# on the transition, not the name.
|
||||
# conf exact = gate on it. advisory = a contributing site declares more
|
||||
# varargs than it pushes. none = no producer family reaches it HERE,
|
||||
# which is not a claim that it has none; never gate on it.
|
||||
# varargs than it pushes. none = no producer family reaches it here,
|
||||
# which does not mean it has none; never gate on it.
|
||||
# producers signal (0x703e50, no varargs) | signal2 (0x703f50, format-driven) |
|
||||
# unit-field-bridge (0x51bbb0/0x51bd50 -> 0x515e50, always %s) |
|
||||
# token-fanout (0x515e50 with a literal id, also %s).
|
||||
|
||||
|
@@ -1,9 +1,9 @@
|
||||
# The 1.12.1 client's global namespace. GENERATED — scripts/gen-reference-globals.py.
|
||||
# Source: wow-5875-re's W5875Capture fixture (the running client's in-world _G),
|
||||
# attributed against the complete 1.12 shipped-UI corpus. Decision 1189.
|
||||
# UNION every name that corpus defines but the capture did not contain (type
|
||||
# `lod`): the twelve Blizzard_* addons are LoadOnDemand, so a live dump misses
|
||||
# them unless the player had opened those windows. Decision 1200.
|
||||
# The 1.12.1 client's global namespace. Generated by scripts/gen-reference-globals.py.
|
||||
# Source: a capture of the running client's in-world _G (the maintainer's analysis,
|
||||
# `WOW_RE`), attributed against the complete 1.12 shipped-UI corpus.
|
||||
# Also every name that corpus defines but the capture lacks, as type `lod`: the
|
||||
# twelve Blizzard_* addons are LoadOnDemand, so a live dump misses them unless
|
||||
# their windows were opened.
|
||||
# name type origin(lua|engine|framexml)
|
||||
ABANDON_PET string framexml
|
||||
ABANDON_QUEST string framexml
|
||||
|
||||
|
Can't render this file because it is too large.
|
@@ -1,7 +1,7 @@
|
||||
# build 5875 CGGameUI message catalog (the 0xb4b498 table indexed by 0x496720)
|
||||
# 465 rows, ids 0x0..0x1d0; consumer bound `cmp id,0x1d1; jge` at 0x49672c
|
||||
# kind 0 -> 0x49a870(text, chan); 1 -> UI_INFO_MESSAGE 0xe1; 2 -> UI_ERROR_MESSAGE 0xe0; 3 -> console 0x63cd00
|
||||
# regenerate (in wow-5875-re): python3 re/ui/extract-message-catalog.py <WoW.exe> > re/ui/message-catalog.tsv
|
||||
# Vendored from the maintainer's analysis (`WOW_RE`); messages/catalog.rs is generated from it.
|
||||
id_hex id_dec row_va key kind sound type_tag chan
|
||||
0x0 0 0xb4b498 ERR_INV_FULL 2 NONE 0x0 10
|
||||
0x1 1 0xb4b4ac ERR_BANK_FULL 2 NONE 0x44 10
|
||||
|
||||
|
+37
-50
@@ -1,56 +1,43 @@
|
||||
# Lua binding SHAPE table for WoW.exe 1.12.1 build 5875 — one row per registered
|
||||
# binding: name -> (argc, return arity, return kinds), with per-column confidence.
|
||||
# WHY: a client that gates its Lua globals by NAME accepts a binding with the right
|
||||
# name and the wrong return arity, which silently breaks a stock FrameXML file.
|
||||
# Evidence + five worked examples: system/ui/scratch/binding-shape-arity-law.md.
|
||||
# Lua binding shapes of the 1.12.1 client (WoW.exe build 5875): one row per registered
|
||||
# binding, name -> (argc, return arity, return kinds), with per-column confidence.
|
||||
#
|
||||
# 82 registrar tables, 1722 bindings, 1722 analysed, 0 unresolved registrar sites.
|
||||
# Four registration routes, extents read from the REGISTRAR's own literal, never a
|
||||
# run scan (see re/audit/registrar-dump.py): 0x704120 RegisterFunction loops;
|
||||
# 0x701d80 widget method tables; the INLINED base-library loop at 0x811e28; and the
|
||||
# four luaL_openlib 0x6f4dc0 tables (bit/math/table/string).
|
||||
# 82 registrar tables, 1722 bindings, 1722 analysed, 0 unresolved registrar sites. Four
|
||||
# registration routes, extents read from the registrar's own literal: 0x704120
|
||||
# RegisterFunction loops; 0x701d80 widget method tables; the inlined base-library loop at
|
||||
# 0x811e28; and the four luaL_openlib 0x6f4dc0 tables (bit/math/table/string).
|
||||
#
|
||||
# COLUMN CONTRACT — a consumer may rely on exactly this:
|
||||
# arity / arity_conf SOUND. exact = one constant; set = a `|`-separated set of
|
||||
# possible counts; unknown = `?`. The lattice only widens on merge, so an
|
||||
# imprecise result is a set or `?` and NEVER a wrong single number. Gate on this.
|
||||
# kind `string?` STRING-OR-NIL, and this distinction has bitten a consumer.
|
||||
# lua_pushstring 0x6f3890 alone begins `test edx,edx ; jne` and TAIL-JUMPS to
|
||||
# lua_pushnil when the char* is NULL, so a push call does not prove a string
|
||||
# reached Lua. `string` is emitted only where the pointer is a constant that
|
||||
# resolves to a readable literal; every other lua_pushstring is `string?`. This
|
||||
# is why an empty/miss path can show a fixed arity with no kind alternation and
|
||||
# still answer nil in that slot (GetMapInfo -> nil,0,0; GetChannelName -> 0,nil,0).
|
||||
# kinds / kinds_conf ADVISORY unless kinds_conf == agree, which means the
|
||||
# independent lua_push* trace length matches `eax` at every ret AND every ret
|
||||
# was checked. A loop (GetTexCoord: 2 textual pushes, 8 real) -> unverified.
|
||||
# `agree` is a claim about THIS BINDING'S OWN ops. Two things it does not say.
|
||||
# (1) It used to say nothing about an op with a net delta of ZERO that rewrites
|
||||
# a slot in place: lua_gettable 0x6f3a40 and lua_rawget 0x6f3b00 pop the key
|
||||
# off the top and write the looked-up value into that same slot, so the kind
|
||||
# recorded was the KEY's. That is now modelled (REPLACE_TOP -> `any`), and it
|
||||
# was wrong on exactly three rows: getglobal, pairs, ipairs.
|
||||
# (2) It still says nothing about a CALLEE that pushes on its own behalf --
|
||||
# the trace cannot see those, so the returned window may hold slots it never
|
||||
# described. Not corrected; DECLARED, as the note token `delegated-push`.
|
||||
# Evidence for both: system/ui/scratch/binding-kinds-stack-overwrite.md.
|
||||
# argc / argc_conf NEVER GATE ON THIS. argc is a max over observed constant Lua
|
||||
# stack indices; it has no unknown state and degrades to a silent LOWER BOUND when
|
||||
# a binding delegates argument parsing to a helper. Such rows say lower-bound, but
|
||||
# the detector is one level deep, so treat EVERY argc as a floor.
|
||||
# name NOT A KEY. 60 names are registered from more than one table and 21 of
|
||||
# those DIFFER IN ARITY between registrations — GetBuildInfo returns 5 on the glue
|
||||
# screen (table 0x8373b8) and 3 in-game (0x83de68); GetAddOnInfo 8 and 7. Key on
|
||||
# `fn`, or on (table_va, name). Every such row carries the note name-not-unique.
|
||||
# notes stack-leak = the reference pushes a value and then returns 0 (or fewer).
|
||||
# A faithful client must NOT copy this; these are the disabled lottery/vendor
|
||||
# stubs. variadic = the count comes from lua_gettop, no constant exists.
|
||||
# delegated-push = the binding calls a foreign callee that itself pushes, so
|
||||
# `agree` covers this row's own ops only (see the kinds contract above).
|
||||
# cfg-truncated / indirect-branch / iteration-cap = an edge the walk could not
|
||||
# resolve; such a row's arity is not covered by the soundness claim above.
|
||||
# Column contract:
|
||||
# arity / arity_conf sound. exact = one constant; set = a `|`-separated set of possible
|
||||
# counts; unknown = `?`. An imprecise result is a set or `?`, never a wrong single
|
||||
# number. Gate on this.
|
||||
# kind `string?` string or nil. lua_pushstring 0x6f3890 begins `test edx,edx ; jne`
|
||||
# and tail-jumps to lua_pushnil when the char* is NULL, so `string` is emitted only where
|
||||
# the pointer is a constant that resolves to a readable literal; every other
|
||||
# lua_pushstring is `string?`. A miss path can show a fixed arity and still answer nil in
|
||||
# that slot (GetMapInfo -> nil,0,0; GetChannelName -> 0,nil,0).
|
||||
# kinds / kinds_conf advisory unless kinds_conf == agree: the independent lua_push* trace
|
||||
# length matches `eax` at every ret, and every ret was checked. A loop (GetTexCoord: 2
|
||||
# textual pushes, 8 real) -> unverified. `agree` is about the binding's own ops:
|
||||
# lua_gettable 0x6f3a40 and lua_rawget 0x6f3b00 replace the key on top with the looked-up
|
||||
# value (recorded as `any`), and a callee that pushes on the binding's behalf is invisible
|
||||
# to the trace, so the returned window may hold slots it never described (note
|
||||
# `delegated-push`).
|
||||
# argc / argc_conf never gate on this. argc is a max over observed constant Lua stack
|
||||
# indices, with no unknown state, and a silent lower bound when a binding delegates
|
||||
# argument parsing to a helper. Such rows say lower-bound, but the detector is one level
|
||||
# deep, so treat every argc as a floor.
|
||||
# name not a key. 60 names are registered from more than one table and 21 of those
|
||||
# differ in arity between registrations: GetBuildInfo returns 5 on the glue screen (table
|
||||
# 0x8373b8) and 3 in-game (0x83de68); GetAddOnInfo 8 and 7. Key on `fn`, or on
|
||||
# (table_va, name). Every such row carries the note name-not-unique.
|
||||
# notes stack-leak = the reference pushes a value and then returns 0 (or fewer): the
|
||||
# disabled lottery/vendor stubs, which a faithful client does not copy. variadic = the
|
||||
# count comes from lua_gettop, no constant exists. delegated-push = a foreign callee
|
||||
# pushes on the binding's behalf, so `agree` covers this row's own ops only.
|
||||
# cfg-truncated / indirect-branch / iteration-cap = an edge the walk could not resolve;
|
||||
# such a row's arity is not covered by the soundness claim above.
|
||||
#
|
||||
# regenerate (in wow-5875-re): python3 re/audit/binding-shapes.py > re/audit/binding-shapes.tsv
|
||||
# Vendored from the maintainer's analysis (`WOW_RE`); no generator for it lives in this repo.
|
||||
name fn pair_va table_va table_kind argc argc_conf arity arity_conf kinds kinds_conf notes
|
||||
AbandonQuest 0x4dfe00 0x84b128 0x84b0f0 global 0 exact 0 exact () agree
|
||||
AbandonSkill 0x4d3ab0 0x849d08 0x849cf8 global 1 exact 0 exact () agree
|
||||
|
||||
|
@@ -1,16 +1,14 @@
|
||||
# The 1.12.1 client's FrameScript events fired FROM INSIDE A LUA VERB, keyed by the verb.
|
||||
# GENERATED — scripts/gen-reference-verb-events.py. Source: wow-5875-re's
|
||||
# re/audit/binding-shapes.tsv + re/events/event-firesites.tsv + re/names/out/names-5875.tsv
|
||||
# + system/ui/scratch/disasm-full.txt.
|
||||
# The 1.12.1 client's FrameScript events fired from inside a Lua verb, keyed by the verb.
|
||||
# Generated by scripts/gen-reference-verb-events.py from reference/1.12-shapes.tsv,
|
||||
# reference/1.12-event-firesites.tsv and the maintainer's analysis (`WOW_RE`).
|
||||
#
|
||||
# shape body = a fire site inside the verb's own extent.
|
||||
# helper = a fire site inside a function that ONLY registered verbs call
|
||||
# helper = a fire site inside a function that only registered verbs call
|
||||
# (at most 8 of them); `via` is that helper. A helper with any
|
||||
# non-verb caller is a state event's producer and is NOT here.
|
||||
# sites the fire sites themselves, for the grep into wow-re.
|
||||
# A floor, not a census: no row is not a claim of no fire (see the generator's header).
|
||||
# Sites are attributed against the ledgers' verified function extents; a site past its
|
||||
# candidate's extent is left out rather than mis-filed (decision 2257).
|
||||
# non-verb caller produces a state event and is not here.
|
||||
# sites the fire sites themselves.
|
||||
# A floor, not a census: a verb with no row may still fire an event.
|
||||
# A site past its candidate function's extent is left out rather than mis-filed.
|
||||
# 43 pairs over 41 verbs and 27 events; 2 site(s) past a verified extent.
|
||||
verb event shape via sites
|
||||
AddSkillUp SKILL_LINES_CHANGED helper 0x4d32d0 0x4d346f
|
||||
|
||||
|
+13
-41
@@ -1,36 +1,16 @@
|
||||
#!/usr/bin/env python3
|
||||
"""doc-links — fail on a doc link whose target does not exist ANYWHERE in the workspace.
|
||||
"""Fail on a doc link whose target is declared nowhere in the workspace.
|
||||
|
||||
WHY THIS IS NARROW, AND WHY THE OBVIOUS VERSION IS WRONG (decision 1925). The docs are this
|
||||
project's knowledge base, so a doc link that goes nowhere is real rot — decision 1903 found a
|
||||
`UNIT_FLAG_STUNNED` comment still pointing at a `MovementState::stunned` that a later decision had
|
||||
deleted, and nothing caught it because nothing runs rustdoc.
|
||||
|
||||
The obvious fix is `-D rustdoc::broken_intra_doc_links`. Measured, it does not survive contact with
|
||||
this codebase: **941 warnings, 264 of them unambiguous `crate::`/`super::`/`self::` paths** — and
|
||||
almost none of that is rot.
|
||||
|
||||
* **195** name a target that IS declared in its crate. They do not resolve because the target is
|
||||
a *private* module named through its parent (`crate::net::apply`, where `mod apply;` is private
|
||||
to `net`) — not a rustdoc quirk but Rust's own name resolution, so the link never worked.
|
||||
Making them resolve means widening visibility for a documentation reason, trading encapsulation
|
||||
for navigation.
|
||||
* **~50** are one crate naming another's module with `crate::` — `benilla-world` writing
|
||||
`crate::entities`, which is benilla-app's and which benilla-world cannot depend on. Those can
|
||||
never resolve; the only "fix" is de-linking prose that reads fine as prose.
|
||||
|
||||
Both are conventions this codebase writes on purpose. A gate on them would be 249 edits of churn
|
||||
that improve nothing, and would then have to stay green forever.
|
||||
|
||||
What IS unambiguous rot is the residue: a link whose **leaf is declared nowhere in the workspace**
|
||||
— the item is gone, so no reading of the link can be right. That was 15 sites when this was
|
||||
written (1925 fixed them); this script keeps it at zero.
|
||||
It reads rustdoc's unresolved-link warnings and fails on a `crate::`, `super::`, `self::` or
|
||||
`Self::` path whose leaf name is declared nowhere: the item is gone. Every other warning is a
|
||||
convention here, not rot (a private module named through its parent, another crate's module
|
||||
named with `crate::`, a bare name for an unimported bevy type).
|
||||
|
||||
scripts/doc-links.py # the gate: exit 1 on any dead-target link
|
||||
scripts/doc-links.py --report # + the benign counts, for anyone re-opening the question
|
||||
scripts/doc-links.py --report # + the benign counts
|
||||
|
||||
The leaf test is a declaration grep, deliberately loose: a name declared anywhere counts, because
|
||||
the failure this catches is "the thing was deleted", not "the path is spelled right".
|
||||
The leaf test is a loose declaration grep: a name declared anywhere counts, since what it catches
|
||||
is a deleted item, not a misspelled path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -54,15 +34,10 @@ def run_rustdoc() -> str:
|
||||
|
||||
|
||||
def declared_names() -> set[str]:
|
||||
"""Every name the workspace declares, in ONE pass.
|
||||
|
||||
A grep per link is what the first version did, and at ~900 links over a 19-crate tree it does
|
||||
not finish inside a gate's patience. One scan, then set lookups.
|
||||
"""
|
||||
"""Every name the workspace declares, in one scan: a grep per link is too slow for a gate."""
|
||||
names: set[str] = set()
|
||||
# `fn` gets its own pattern: a single alternation swallows the `const` of `const fn foo` and
|
||||
# captures "fn" as the name, so `foo` is never indexed and the gate reports a live item as
|
||||
# deleted. That bug shipped for one run of this script and is the reason the two passes exist.
|
||||
# `fn` gets its own pattern: in one alternation, `const fn foo` captures "fn" and never
|
||||
# indexes `foo`.
|
||||
pats = [
|
||||
re.compile(r"\bfn\s+([A-Za-z_][A-Za-z0-9_]*)"),
|
||||
re.compile(r"\b(?:struct|enum|trait|type|const|static|union|mod)\s+([A-Za-z_][A-Za-z0-9_]*)"),
|
||||
@@ -87,11 +62,8 @@ def main() -> int:
|
||||
names = declared_names()
|
||||
dead, benign = [], 0
|
||||
for target, path, line in found:
|
||||
# ONLY paths the author asserted are in THIS crate. A bare `[`SystemParam`]` or
|
||||
# `[`KeyboardInput`]` names a bevy type that is simply not imported for linking — real,
|
||||
# not rot — and there is no honest way to tell those from a deleted local name. A
|
||||
# `crate::`/`super::`/`self::`/`Self::` path carries the assertion, so its absence is
|
||||
# unambiguous.
|
||||
# Only a `crate::`/`super::`/`self::`/`Self::` path asserts a local item; a bare name may
|
||||
# be a bevy type not imported for linking (`SystemParam`), which is not rot.
|
||||
if not re.match(r"^(crate|super|self|Self)::", target):
|
||||
benign += 1
|
||||
continue
|
||||
|
||||
@@ -1,17 +1,13 @@
|
||||
#!/usr/bin/env python3
|
||||
r"""Generate `crates/benilla-ui/src/messages/catalog.rs` from `reference/1.12-message-catalog.tsv`.
|
||||
|
||||
The TSV is wow-5875-re's byte-exact sweep of build 5875's `0xb4b498` registry -- the table
|
||||
`CGGameUI::DisplayError 0x496720` indexes by message id -- vendored under `reference/` like the
|
||||
other tables (it is regenerated there with `re/ui/extract-message-catalog.py`). This script only
|
||||
re-shapes it into a Rust const, so the table ships with benilla and is never read out of a
|
||||
sibling checkout.
|
||||
The table is the reference's message registry at `0xb4b498`, which `CGGameUI::DisplayError`
|
||||
`0x496720` indexes by message id; this re-shapes it into a Rust static that ships with benilla.
|
||||
|
||||
python3 scripts/gen-message-catalog.py # the vendored table -> the .rs, in place
|
||||
python3 scripts/gen-message-catalog.py <tsv> # another table
|
||||
|
||||
Refuses rather than emits on anything it does not understand: a non-dense id range, a duplicate key,
|
||||
an unknown kind. A generated table that quietly dropped a row would be worse than no table.
|
||||
It refuses rather than emits on a non-dense id range, a duplicate key or an unknown kind.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
@@ -23,20 +19,17 @@ OUT = os.path.join(REPO, "crates", "benilla-ui", "src", "messages", "catalog.rs"
|
||||
KINDS = {0: "Chat", 1: "Info", 2: "Error"}
|
||||
|
||||
HEADER = """\
|
||||
//! The 5875 message catalog, row for row.
|
||||
//! The 1.12 client's message catalog, row for row.
|
||||
//!
|
||||
//! @generated by `scripts/gen-message-catalog.py` from `reference/1.12-message-catalog.tsv`
|
||||
//! (wow-5875-re's `re/ui/message-catalog.tsv`, extracted out of `WoW.exe` there).
|
||||
//! Do not edit by hand — regenerate.
|
||||
//! @generated by `scripts/gen-message-catalog.py` from `reference/1.12-message-catalog.tsv`.
|
||||
//! Do not edit by hand; regenerate.
|
||||
|
||||
use super::{MessageRecord, MsgKind};
|
||||
|
||||
/// Every row of `0xb4b498`, indexed by message id (the ids are dense `0..465`, so
|
||||
/// `CATALOG[id]` *is* the reference's `0xb4b498 + 20*id`).
|
||||
/// Every row of `0xb4b498`, indexed by message id: the ids are dense `0..465`, so `CATALOG[id]`
|
||||
/// is the reference's `0xb4b498 + 20*id`.
|
||||
///
|
||||
/// `rustfmt` is held off so that a regeneration is a no-op diff: one row per line is the shape
|
||||
/// this script emits, and a formatter that rewrapped it would make the file drift from its own
|
||||
/// generator.
|
||||
/// `rustfmt` is off: the generator emits one row per line, and a rewrap would drift from it.
|
||||
#[rustfmt::skip]
|
||||
pub static CATALOG: [MessageRecord; {count}] = [
|
||||
"""
|
||||
|
||||
@@ -1,76 +1,51 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Regenerate `reference/1.12-events.tsv` — every FrameScript event the 1.12.1 client can
|
||||
dispatch, with the ARGUMENTS its producers push.
|
||||
"""Regenerate `reference/1.12-events.tsv`: each 1.12 FrameScript event and the arguments it carries.
|
||||
|
||||
scripts/gen-reference-events.py [--catalog FILE] [--firesites FILE] [--out FILE]
|
||||
|
||||
**Why this exists** (decision 2140). benilla already gates two halves of the event seam: a stock
|
||||
chain file listening for an event nothing fires
|
||||
(`reference_ui::every_event_a_chain_file_registers_has_a_producer`, 1889) and an event we fire that
|
||||
1.12 does not have (`every_event_we_fire_is_an_event_the_reference_has`, 1883). Both compare
|
||||
NAMES. Nothing compared the *arguments*, and an event name is a plain string at both ends — so a
|
||||
fire with the wrong argument count is silent on both sides, exactly as a binding with the wrong
|
||||
return arity was before 1842. It is the same class of bug one API over, and it had three live
|
||||
instances: `PLAYERBANKSLOTS_CHANGED`, `UNIT_PET_EXPERIENCE` and `UNIT_PET_TRAINING_POINTS`, the
|
||||
last of which routes through `PetPaperDollFrame_OnEvent`'s `elseif ( arg1 == "pet" )` catch-all and
|
||||
so reached nobody at all.
|
||||
|
||||
**The oracle is wow-5875-re's own two censuses**, vendored under `reference/` the way
|
||||
`1.12-shapes.tsv` is:
|
||||
The inputs are vendored under `reference/`, so this runs from any clone and reproduces the
|
||||
committed table byte for byte:
|
||||
|
||||
- `1.12-event-catalog.tsv` eventId -> name, off the name-pointer array at `.data 0xbe1198`
|
||||
- `1.12-event-firesites.tsv` every `call` AND tail-`jmp` to `FrameScript_SignalEvent`
|
||||
- `1.12-event-firesites.tsv` every `call` and tail-`jmp` to `FrameScript_SignalEvent`
|
||||
(`0x703e50`, no varargs) and `SignalEvent2` (`0x703f50`,
|
||||
printf-style), with the FORMAT STRING each site pushes
|
||||
printf-style), with the format string each site pushes
|
||||
|
||||
**Three producer families, because a literal fire site is not the whole picture.** The firesites
|
||||
census is over the two signal helpers' own call sites, and its own header says in capitals that an
|
||||
event with no row there is not an event with no producer:
|
||||
A literal fire site is not every producer; the families are:
|
||||
|
||||
signal a `0x703e50` site — `__fastcall(ecx = id)`, a plain `ret`, ZERO Lua values.
|
||||
signal2 a `0x703f50` site — the pushed format string is the argument shape.
|
||||
signal a `0x703e50` site: `__fastcall(ecx = id)`, a plain `ret`, zero Lua values.
|
||||
signal2 a `0x703f50` site: the pushed format string is the argument shape.
|
||||
unit-field-bridge the generic UpdateField -> event bridge (`0x51bbb0` registers one watch per
|
||||
named unit-window field; `0x51bd50` -> `0x515e50` fans out over the unit
|
||||
tokens and calls `0x703f50(id, "%s", token)`). Every event id below `0xb6`
|
||||
that HAS a name is produced there and appears in no firesites row.
|
||||
Cross-check: the 54 catalog ids under `0xb6` are exactly the id set
|
||||
wow-re read out of the registration loop by hand
|
||||
(`system/ui/scratch/unit-field-event-bridge.md` §2.1) — two independent
|
||||
derivations, no difference.
|
||||
token-fanout the same `0x515e50` fan-out reached with a LITERAL id in `edx` rather than a
|
||||
computed one, so also `%s` and also absent from the firesites census. The
|
||||
fourteen ids are wow-re's own list, in that file's header.
|
||||
tokens and calls `0x703f50(id, "%s", token)`). Every named event id below
|
||||
`0xb6` is produced there and has no fire-site row.
|
||||
token-fanout the same `0x515e50` fan-out reached with a literal id in `edx`: also `%s`,
|
||||
also absent from the fire sites.
|
||||
|
||||
**Confidence, and what a consumer may gate on:**
|
||||
Confidence, and what a consumer may gate on:
|
||||
|
||||
exact every contributing producer's shape is known. Gate on this.
|
||||
advisory a contributing site is one of the three whose pushed format DECLARES more varargs than
|
||||
its caller pushes (`re/audit/signalevent2-fmt-args.py` audits all 149 and finds
|
||||
exactly these): `0x496230 TRADE_REPLACE_ENCHANT`, `0x5e4527`/`0x5e7960 UPDATE_TICKET`.
|
||||
Faithful clients must not copy a bug that hands Lua undefined values.
|
||||
none the name is in the catalog and NO producer family reaches it here. **This is not a
|
||||
claim that it has none** — see the census caveat above. Never gate on it.
|
||||
advisory a contributing site's format declares more varargs than its caller pushes
|
||||
(`0x496230` TRADE_REPLACE_ENCHANT, `0x5e4527`/`0x5e7960` UPDATE_TICKET), which hands
|
||||
Lua undefined values; do not copy it.
|
||||
none no producer family reaches the name here, which does not mean it has none. Never
|
||||
gate on it.
|
||||
|
||||
Glue-space rows are dropped: `0x703d90` is called twice with different name populations, so the
|
||||
same integer names a different event on the glue screen than in the world. wow-re blanks those
|
||||
rows' names; this drops them by fire-site address as well, from the four TU ranges its header
|
||||
names, so a future re-derivation that fills a name in cannot leak one through.
|
||||
|
||||
Both inputs are in the repo, so this runs from any clone and reproduces the committed table byte
|
||||
for byte; only the two inputs themselves are regenerated in the RE repo.
|
||||
Glue-space sites are dropped by address: `0x703d90` loads a separate name table for the glue
|
||||
screen, so the same id names a different event there. Their names are blank in the input, and
|
||||
dropping by address keeps a name filled in later from leaking through.
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
|
||||
# The `Source\Glue\` TU ranges whose event ids belong to the GLUE name space, from
|
||||
# `re/events/event-firesites.tsv`'s own header.
|
||||
# The `Source\Glue\` fire-site ranges (`1.12-event-firesites.tsv`'s header): glue-space ids.
|
||||
GLUE_SITES = [(0x46AA34, 0x46AA34), (0x46BC9D, 0x46C52E), (0x46E73D, 0x46E73D), (0x47461E, 0x47461E)]
|
||||
# The three sites whose format string declares more varargs than the caller pushes.
|
||||
DECLARES_MORE_THAN_IT_PUSHES = {0x496230: "TRADE_REPLACE_ENCHANT", 0x5E4527: "UPDATE_TICKET", 0x5E7960: "UPDATE_TICKET"}
|
||||
# `0x515e50` callers carrying a LITERAL id — the token fan-out reached outside the generic bridge.
|
||||
# `0x515e50` callers with a literal id: the token fan-out reached outside the generic bridge.
|
||||
TOKEN_FANOUT_IDS = [0x10, 0x16, 0x1C, 0x1D, 0x1E, 0x29, 0xB7, 0xB8, 0xB9, 0xBA, 0xBB, 0x159, 0x197, 0x20A]
|
||||
# `0x51bbb0`'s walk bound: `cmp esi,0xb6`. Every NAMED unit-window field below it is watched.
|
||||
# `0x51bbb0`'s walk bound (`cmp esi,0xb6`): every named unit-window field below it is watched.
|
||||
UNIT_WINDOW_FIELDS = 0xB6
|
||||
# The zero-argument format, spelled so the column is never empty.
|
||||
NONE = "()"
|
||||
@@ -140,8 +115,7 @@ def main():
|
||||
if eid in catalog:
|
||||
add(catalog[eid], eid, "token-fanout", "%s")
|
||||
|
||||
# Every remaining catalog name, so the table is the whole surface and a consumer can tell
|
||||
# "no producer found" from "not an event".
|
||||
# Every remaining catalog name too, so "no producer found" differs from "not an event".
|
||||
for eid, name in catalog.items():
|
||||
if name not in ev:
|
||||
ev.setdefault(name, {"ids": set(), "producers": set(), "formats": set(), "notes": []})["ids"].add(eid)
|
||||
@@ -149,17 +123,17 @@ def main():
|
||||
out = os.path.normpath(a.out)
|
||||
with open(out, "w", encoding="utf-8") as fh:
|
||||
fh.write(
|
||||
"# The 1.12.1 client's FrameScript events and the ARGUMENTS their producers push.\n"
|
||||
"# GENERATED — scripts/gen-reference-events.py. Source: wow-5875-re's\n"
|
||||
"# re/events/event-catalog.tsv + event-firesites.tsv. Decision 2140.\n"
|
||||
"# The 1.12.1 client's FrameScript events and the arguments their producers push.\n"
|
||||
"# Generated by scripts/gen-reference-events.py from reference/1.12-event-catalog.tsv\n"
|
||||
"# and reference/1.12-event-firesites.tsv.\n"
|
||||
"#\n"
|
||||
"# arg_formats `|`-separated, one per distinct producer shape. `()` = zero Lua values.\n"
|
||||
"# Directives: %s string, %d/%u number, %f number. An event with two\n"
|
||||
"# producers of different shapes carries both, and which one a given\n"
|
||||
"# transition takes is a question about the transition, not the name.\n"
|
||||
"# producers of different shapes carries both; which one fires depends\n"
|
||||
"# on the transition, not the name.\n"
|
||||
"# conf exact = gate on it. advisory = a contributing site declares more\n"
|
||||
"# varargs than it pushes. none = no producer family reaches it HERE,\n"
|
||||
"# which is not a claim that it has none; never gate on it.\n"
|
||||
"# varargs than it pushes. none = no producer family reaches it here,\n"
|
||||
"# which does not mean it has none; never gate on it.\n"
|
||||
"# producers signal (0x703e50, no varargs) | signal2 (0x703f50, format-driven) |\n"
|
||||
"# unit-field-bridge (0x51bbb0/0x51bd50 -> 0x515e50, always %s) |\n"
|
||||
"# token-fanout (0x515e50 with a literal id, also %s).\n"
|
||||
|
||||
@@ -1,49 +1,31 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Regenerate `reference/1.12-globals.tsv` — the 1.12.1 client's global namespace, with each name
|
||||
attributed to whoever provides it.
|
||||
"""Regenerate `reference/1.12-globals.tsv`: the 1.12.1 client's global namespace, by origin.
|
||||
|
||||
WOW_RE=<the RE repo> scripts/gen-reference-globals.py [--wow-re DIR] [--out FILE]
|
||||
WOW_RE=DIR scripts/gen-reference-globals.py [--wow-re DIR] [--out FILE]
|
||||
|
||||
**Why this exists** (decision 1189): every wrong number in the addon arc came from remembering a
|
||||
list of WoW functions instead of asking the client. wow-5875-re already ships an addon
|
||||
(`W5875Capture`) that dumps the in-world `_G` of the *running* reference client, and its output is
|
||||
committed there as a fixture. This vendors that capture, attributes every name, and writes the
|
||||
result as the one artifact benilla measures its API surface against.
|
||||
Each name in a capture of the running reference client's in-world `_G` gets one origin:
|
||||
|
||||
**Three origins**, because "is this in 1.12?" and "whose job is it to provide?" are different
|
||||
questions:
|
||||
lua Lua 5.0's own runtime, which mlua provides.
|
||||
engine the C client provides it; benilla implements these in Rust.
|
||||
framexml the shipped UI defines it (its Lua, its XML, or a `$parent`-composed child of one);
|
||||
it runs off the player's own patch chain, never hardcoded in Rust.
|
||||
|
||||
lua Lua 5.0's own runtime. We get these from mlua; neither we nor FrameXML write them.
|
||||
engine The C client provides it. **This is the list benilla implements in Rust.**
|
||||
framexml Defined by the shipped UI itself — its Lua, its XML, or a `$parent`-composed child of
|
||||
one. Runs off the player's own patch chain at runtime (1751), never hardcoded in Rust.
|
||||
Attribution is by definition site in the complete 1.12 shipped-UI corpus:
|
||||
|
||||
Attribution is by *definition site*, computed from a complete 1.12 shipped-UI corpus:
|
||||
- a name assigned or `function`-declared in shipped Lua, including Lua inside a FrameXML
|
||||
document, is FrameXML's; indented assignments count, because FrameXML leaks globals out of
|
||||
function bodies (`button = getglobal(...)`);
|
||||
- a `name="..."` on any shipped XML element is FrameXML's, `virtual="true"` included: a virtual
|
||||
`<Font>` is a real font object, and 1.12 registers named virtual frames too;
|
||||
- `<a shipped name><a $parent suffix>` is a composed child (`ContainerFrame1Item16IconTexture`);
|
||||
- LUA_5_0 wins over all of these: FrameXML's `string = getglobal(...)` would otherwise claim
|
||||
the stdlib table.
|
||||
|
||||
- a name assigned or `function`-declared in shipped Lua (including a `<Script>` body inside a
|
||||
FrameXML document) is FrameXML's. Indented assignments count: vanilla FrameXML leaks globals
|
||||
out of function bodies (`button = getglobal(...)`) and those names really are in `_G`;
|
||||
- a `name="..."` on any shipped XML element is FrameXML's — including `virtual="true"`, because
|
||||
a virtual `<Font>` is a real font object and 1.12 registers named virtual frames too;
|
||||
- a name formed as `<a shipped name><a $parent suffix>` is a composed child object
|
||||
(`ContainerFrame1Item16IconTexture`), so also FrameXML's;
|
||||
- LUA_5_0 below wins over all of the above: FrameXML *clobbers* `string` in three files
|
||||
(`string = getglobal(...)`), which overwrites the stdlib table rather than defining it.
|
||||
|
||||
**Inputs live outside the repo** — a 1.12 install and the RE repo's runtime capture (`--wow-re`,
|
||||
or `$WOW_RE`) — so this is a manual regeneration, like `genmap.sh`; the committed table is the
|
||||
surface benilla tracks. It needs:
|
||||
|
||||
- `<wow-re>/WoW/_w5875_fixtures/item13/W5875Capture.lua` the captured in-world `_G`
|
||||
- `<wow-re>/WoW/_extracted_framexml/` FrameXML, already extracted there
|
||||
- `<wow-re>/WoW/Data/*.MPQ` for the twelve `Blizzard_*` addons
|
||||
|
||||
**The corpus is the part that is easy to get wrong** (decision 1188 §4 — three attempts were
|
||||
needed, and the first two produce confidently wrong answers). FrameXML alone is not the shipped
|
||||
UI: the twelve `Blizzard_*` addons live in the MPQs, and their `.lua` is *not* listed in their
|
||||
`.toc` — it is pulled in by `<Script file=>` inside their XML. A corpus that misses either half
|
||||
misattributes real FrameXML functions as the engine's. A complete one is 233 files, and this
|
||||
script prints the count so a short one is visible rather than silent.
|
||||
The inputs are the maintainer's analysis (`--wow-re` or `$WOW_RE`), not in this repo: the
|
||||
capture, the extracted FrameXML, and the install's MPQs. FrameXML alone is not the shipped UI:
|
||||
the twelve `Blizzard_*` addons live in the MPQs, and their `.lua` is reached through
|
||||
`<Script file=>` in their XML, not their `.toc`. A complete corpus is 233 files; the count is
|
||||
printed.
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
@@ -53,8 +35,7 @@ import sys
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
# The twelve addons that ship inside the MPQs. Their folders exist in a real install carrying only
|
||||
# a `.pub` signature file, so the install cannot be the source — the archives are.
|
||||
# The twelve addons shipped inside the MPQs; an install's folder for each holds only a `.pub`.
|
||||
BLIZZARD_ADDONS = [
|
||||
"Blizzard_AuctionUI", "Blizzard_BattlefieldMinimap", "Blizzard_BindingUI",
|
||||
"Blizzard_CombatText", "Blizzard_CraftUI", "Blizzard_GMSurveyUI",
|
||||
@@ -62,10 +43,8 @@ BLIZZARD_ADDONS = [
|
||||
"Blizzard_TalentUI", "Blizzard_TradeSkillUI", "Blizzard_TrainerUI",
|
||||
]
|
||||
|
||||
# Lua 5.0's own globals, as the 1.12 client actually exposes them. Note what is NOT here and is
|
||||
# absent from the capture: `_G`, `print`, `require`, `dofile`, `loadfile`, and the `io`/`os`/
|
||||
# `debug`/`coroutine` tables — the client sandboxes them away, which is a fidelity fact about the
|
||||
# VM we present to addons, not an accident of the capture.
|
||||
# Lua 5.0's globals as the 1.12 client exposes them: it removes `_G`, `print`, `require`,
|
||||
# `dofile`, `loadfile` and the `io`/`os`/`debug`/`coroutine` tables.
|
||||
LUA_5_0 = {
|
||||
"assert", "collectgarbage", "error", "gcinfo", "getfenv", "getmetatable", "ipairs",
|
||||
"loadstring", "math", "next", "pairs", "pcall", "rawequal", "rawget", "rawset",
|
||||
@@ -73,15 +52,9 @@ LUA_5_0 = {
|
||||
"unpack", "xpcall",
|
||||
}
|
||||
|
||||
# The capture addon's own footprint, each verified against its source rather than guessed at.
|
||||
# `W5875CaptureHolder` is its `CreateFrame(..., "W5875CaptureHolder", ...)`, `W5875CaptureDB` its
|
||||
# `## SavedVariables`, `SLASH_W5875CAP1` its slash token. It ran inside the client to produce the
|
||||
# fixture, so its globals are in the table it dumped; they are the instrument, not the client.
|
||||
#
|
||||
# `__framescript_meta` is deliberately NOT here despite looking like tooling: it is the *client's*
|
||||
# own shared frame metatable, published to `_G` (wow-5875-re RF-0023, `system/ui/ui.md`), and
|
||||
# benilla mirrors it at `script/object.rs`. Excluding it on the strength of its underscores made
|
||||
# our own faithful global read as a superset.
|
||||
# The capture addon's own globals (its holder frame, SavedVariables and slash token): the
|
||||
# instrument, not the client. `__framescript_meta` stays in: it is the client's own shared frame
|
||||
# metatable.
|
||||
CAPTURE_OWN = {"W5875CaptureDB", "W5875CaptureHolder", "SLASH_W5875CAP1"}
|
||||
|
||||
LUA_ASSIGN = re.compile(r"^[ \t]*([A-Za-z_][A-Za-z0-9_]*)\s*=[^=]", re.M)
|
||||
@@ -92,13 +65,9 @@ LUA_MULTI = re.compile(
|
||||
XML_ELEM = re.compile(r"<(\w+)([^>]*?)/?>", re.S)
|
||||
XML_NAME = re.compile(r'\bname\s*=\s*"([^"]*)"')
|
||||
PARENT_SUFFIX = re.compile(r'\bname\s*=\s*"\$parent([A-Za-z0-9_]*)"')
|
||||
# Lua embedded in a FrameXML document, which only this pass will ever see. Two shapes, and
|
||||
# missing either misattributes real FrameXML names to the engine: an inline `<Script>…</Script>`
|
||||
# body (Fonts.xml defines CHAT_FONT_HEIGHTS that way), and every handler element under
|
||||
# `<Scripts>` — `<OnLoad>`, `<OnEvent>`, `<PreClick>` and kin, which is where MainMenuBar.xml
|
||||
# assigns SHOW_KEYRING and PERFORMANCEBAR_LOW_LATENCY. Matched by shape rather than by a list of
|
||||
# handler names so a rarely-used one cannot be forgotten. `<Script file=>` is skipped: it
|
||||
# references a file already in the corpus.
|
||||
# Lua inside a FrameXML document: an inline `<Script>` body (Fonts.xml's CHAT_FONT_HEIGHTS) and
|
||||
# every `<Scripts>` handler element (`<OnLoad>`, `<PreClick>`, ...), matched by shape so no handler
|
||||
# name is missed. `<Script file=>` is skipped: its file is already in the corpus.
|
||||
XML_SCRIPT = re.compile(
|
||||
r"<((?:On|Pre|Post)\w+|Script)\b(?![^>]*\bfile\s*=)[^>]*>(.*?)</\1>", re.S
|
||||
)
|
||||
@@ -127,7 +96,7 @@ def captured_globals(fixture):
|
||||
|
||||
|
||||
def join_ref(base, ref):
|
||||
"""Resolve a reference against the including file's directory — `loader::join_ref`'s rule."""
|
||||
"""Resolve a reference against the including file's directory, as `loader::join_ref` does."""
|
||||
ref = ref.replace("\\", "/").strip()
|
||||
parts = ref[1:].split("/") if ref.startswith("/") else (base.split("/") if base else []) + ref.split("/")
|
||||
out = []
|
||||
@@ -189,8 +158,7 @@ def build_corpus(wowre):
|
||||
print(f" !! {addon}: missing {rel}", file=sys.stderr)
|
||||
continue
|
||||
files[rel] = data
|
||||
# The `.lua` is reached from inside the XML, never from the `.toc` — follow it, or
|
||||
# every Lua-side API in these twelve addons misattributes to the engine.
|
||||
# The addons' `.lua` is reached only from their XML, never from the `.toc`.
|
||||
if rel.lower().endswith(".xml"):
|
||||
base = rel.rsplit("/", 1)[0] if "/" in rel else ""
|
||||
for m in XML_REF.finditer(data):
|
||||
@@ -217,12 +185,7 @@ def shipped_names(corpus):
|
||||
lua_defs(body, names)
|
||||
suffixes.update(s for s in PARENT_SUFFIX.findall(text) if s)
|
||||
for tag, attrs in XML_ELEM.findall(text):
|
||||
# `<Binding name="ACTIONBUTTON1">` is a KEY-BINDING command name, not a global —
|
||||
# it never reaches `_G`. It shares the `name=` attribute with every frame element,
|
||||
# so a shape-blind harvest pulls all 228 of `Bindings.xml`'s commands in. Harmless
|
||||
# while this set was only used for *attribution* (none of them is in the capture,
|
||||
# so none could be misattributed); wrong now that the set also contributes rows
|
||||
# (decision 1200).
|
||||
# `<Binding name="ACTIONBUTTON1">` is a key-binding command, not a global.
|
||||
if tag.lower() == "binding":
|
||||
continue
|
||||
m = XML_NAME.search(attrs)
|
||||
@@ -285,24 +248,10 @@ def main():
|
||||
rows.append((name, t, origin))
|
||||
tally[(origin, t)] = tally.get((origin, t), 0) + 1
|
||||
|
||||
# ── The LoadOnDemand half the capture could not see (decision 1200) ──────────────────────
|
||||
#
|
||||
# The fixture is the in-world `_G` of a *running* client, and the twelve `Blizzard_*` addons
|
||||
# are LoadOnDemand: unless the player had opened the talent window, the trade-skill window and
|
||||
# the rest before the dump, their globals are not in it. `TalentFrame`, `MacroFrame`,
|
||||
# `TradeSkillFrame`, `CraftFrame`, `TrainerFrame`, `InspectFrame` and ~350 of their children
|
||||
# are real 1.12 names that the capture alone reports as absent — which made benilla's own
|
||||
# transcriptions of those windows look like they were inventing names.
|
||||
#
|
||||
# So the table is the capture UNION every global the shipped UI *defines*. That is the honest
|
||||
# membership test for the question we actually ask it — "is this a 1.12 name?" — and it is
|
||||
# strictly better than the capture alone, which answers "was this name live at one moment in
|
||||
# one session". The two halves stay distinguishable: a name only the corpus knows gets type
|
||||
# `lod`, so a consumer that cares can tell.
|
||||
#
|
||||
# Type inference is deliberately coarse: a `function X(` declaration is a `function`, an XML
|
||||
# `name=` is a frame (`table`), and a bare assignment could be anything, so it says `lod`
|
||||
# rather than guessing. Nothing downstream keys on the type of these rows.
|
||||
# ── The LoadOnDemand names ─────────────────────────────────────────────────────────────────
|
||||
# The twelve `Blizzard_*` addons are LoadOnDemand, so a live capture holds their globals only
|
||||
# if their windows were opened first. Every name the shipped UI defines that the capture lacks
|
||||
# joins with type `lod`: the table answers "is this a 1.12 name?", and the halves stay apart.
|
||||
lod = 0
|
||||
for name in sorted(defined - captured):
|
||||
rows.append((name, "lod", "framexml"))
|
||||
@@ -312,12 +261,12 @@ def main():
|
||||
|
||||
os.makedirs(os.path.dirname(args.out), exist_ok=True)
|
||||
with open(args.out, "w") as f:
|
||||
f.write("# The 1.12.1 client's global namespace. GENERATED — scripts/gen-reference-globals.py.\n")
|
||||
f.write("# Source: wow-5875-re's W5875Capture fixture (the running client's in-world _G),\n")
|
||||
f.write("# attributed against the complete 1.12 shipped-UI corpus. Decision 1189.\n")
|
||||
f.write("# UNION every name that corpus defines but the capture did not contain (type\n")
|
||||
f.write("# `lod`): the twelve Blizzard_* addons are LoadOnDemand, so a live dump misses\n")
|
||||
f.write("# them unless the player had opened those windows. Decision 1200.\n")
|
||||
f.write("# The 1.12.1 client's global namespace. Generated by scripts/gen-reference-globals.py.\n")
|
||||
f.write("# Source: a capture of the running client's in-world _G (the maintainer's analysis,\n")
|
||||
f.write("# `WOW_RE`), attributed against the complete 1.12 shipped-UI corpus.\n")
|
||||
f.write("# Also every name that corpus defines but the capture lacks, as type `lod`: the\n")
|
||||
f.write("# twelve Blizzard_* addons are LoadOnDemand, so a live dump misses them unless\n")
|
||||
f.write("# their windows were opened.\n")
|
||||
f.write("# name\ttype\torigin(lua|engine|framexml)\n")
|
||||
for row in sorted(rows):
|
||||
f.write("\t".join(row) + "\n")
|
||||
|
||||
@@ -1,54 +1,29 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Regenerate `reference/1.12-verb-events.tsv` — every FrameScript event the 1.12.1 client fires
|
||||
FROM INSIDE A LUA VERB, keyed by the verb.
|
||||
"""Regenerate `reference/1.12-verb-events.tsv`: the events 1.12 Lua verbs fire, keyed by verb.
|
||||
|
||||
WOW_RE=<the RE repo> scripts/gen-reference-verb-events.py [--wow-re DIR] [--out FILE]
|
||||
WOW_RE=DIR scripts/gen-reference-verb-events.py [--wow-re DIR] [--out FILE]
|
||||
|
||||
**Why this exists.** `reference/1.12-events.tsv` (2140) says what arguments an event carries and
|
||||
`reference_ui`'s two name gates (1883/1889) say whether *something* fires what a stock file listens
|
||||
for. None of them says WHO fires it. That gap is the one B389/2244 fell through: the reference's
|
||||
`SetTrainerServiceTypeFilter` commits through a helper whose whole body is "write the mask, re-run
|
||||
the finalizer, fire TRAINER_UPDATE", the stock `Blizzard_TrainerUI.lua` therefore never repaints
|
||||
the list itself, and benilla's verb set the mask and fired nothing. The name gate was green,
|
||||
because the packet arm fires the same event; the tests were green, because they asserted the mask.
|
||||
Our own retired `TrainerFrame.xml` had been repainting explicitly, so the behaviour left with the
|
||||
file that was compensating for it — and the same shape is waiting in every other window that went
|
||||
stock (1751): a verb the stock Lua calls *for its side effect of an event*.
|
||||
A row says the reference fires `event` on the call path of the registered Lua binding `verb`:
|
||||
from the verb's body (`shape=body`), or from a helper that only registered verbs call
|
||||
(`shape=helper`, `via` = the helper's address; `0x4d8c90` is the trainer's). A helper with any
|
||||
non-verb caller (a packet handler, the cursor, the tutorial signal) fires a state event and is
|
||||
left out, and so is a hub (`ClearCursor` -> CURSOR_UPDATE, 49 callers).
|
||||
|
||||
**What a row means.** `verb` is a registered Lua binding (wow-re's `re/audit/binding-shapes.tsv`)
|
||||
and `event` is fired on the verb's own call path in the reference — either from the verb's body
|
||||
(`shape=body`) or from a helper that ONLY registered verbs call (`shape=helper`, `via` = the helper's
|
||||
address; `0x4d8c90` is the trainer's). A helper with any non-verb caller — a packet handler, the
|
||||
cursor, the tutorial signal — is excluded by construction: an event fired from there is a *state*
|
||||
event, which benilla may faithfully fire from its own feeds, and demanding it of the verb would be
|
||||
the exceptions list nobody wants to maintain. Hubs (`ClearCursor` → CURSOR_UPDATE, 49 callers) fall
|
||||
out the same way.
|
||||
A site's function is the greatest start at or below it, over the union of: every function the
|
||||
analysis sizes, every named function start, every `call` target in the client's disassembly,
|
||||
every registered verb, and every address after a padding run (an `int3` run anywhere, or a `nop`
|
||||
run after a `ret`; a `nop` run after a `jmp` is a loop head, not a boundary). A site at or past
|
||||
`start + size` of a sized function is left unattributed rather than handed to it.
|
||||
|
||||
**How the enclosing function is found (2257).** A site's function is the greatest start at or
|
||||
below it, over the union of: every function wow-re's ledgers size (`size=` on the `fn` rows of
|
||||
`system/*/ledger.tsv`, ~13k verified extents), the `names-5875.tsv` starts, every `call` target in
|
||||
the full disassembly (`system/ui/scratch/disasm-full.txt`), every registered verb, and every address
|
||||
that follows a padding run — an `int3` run anywhere, or a `nop` run that follows a `ret` (MSVC's
|
||||
inter-function padding; a `nop` run after a `jmp` is a loop head and is NOT a boundary, which is
|
||||
the phantom-start trap the first derivation fell into). Then the extent check: a site that lies at
|
||||
or past `start + size` of a sized function is left UNATTRIBUTED rather than handed to that
|
||||
function. Validated against the ledgers: of ~26k padding-derived starts, 49 fall strictly inside a
|
||||
sized function, all in a handful of oversized rows. The first derivation (2251) had neither the
|
||||
extents nor the `ret`-`nop` rule, and four of its rows were wrong the same way: a fire site in an
|
||||
item-cache *callback* reached only by function pointer (`0x4ce2f0`, `0x4cba50`), a site past a
|
||||
verified extent (`0x4a7835` beyond `0x4a7620`'s 191 bytes), and a fire on a shared stop path past
|
||||
its worker's 34 bytes (`0x48f095`). The two families dropped explicitly below are unchanged:
|
||||
glue-space verbs (the GlueXML registrar; 0x46a000..0x476000, the range 2140's `GLUE_SITES` lie in)
|
||||
and the unit-field bridge / token fan-out ids below 0xb6, which are state events by definition.
|
||||
Two families are dropped: glue-space verbs and sites (the GlueXML registrar, 0x46a000..0x476000)
|
||||
and the unit-field bridge and token fan-out ids below 0xb6, which are state events.
|
||||
|
||||
**A floor, not a census.** No row for a verb is not a claim that the verb fires nothing: the
|
||||
fire-site census is over the two signal helpers' literal call sites (its header says so in
|
||||
capitals), and `shape=helper` stops one call deep on purpose. Gate on what IS here.
|
||||
A floor, not a census: a verb with no row may still fire an event, because the fire sites are
|
||||
only the two signal helpers' literal call sites and `shape=helper` stops one call deep.
|
||||
|
||||
The fire sites and the binding shapes are the vendored `reference/` tables; the function names,
|
||||
the ledgers' extents and the disassembly are the RE repo's (`--wow-re`, or `$WOW_RE`), and the
|
||||
disassembly is the client's own binary, so this is a manual regeneration there. The committed
|
||||
table is the surface benilla tracks.
|
||||
extents and disassembly are the maintainer's analysis (`--wow-re` or `$WOW_RE`), not in this
|
||||
repo, so this is a manual regeneration.
|
||||
"""
|
||||
import argparse
|
||||
import bisect
|
||||
@@ -58,10 +33,9 @@ import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
# The GlueXML registrar's verbs and fire sites (see the header): the same integer names a
|
||||
# different event on the glue screen than in the world (2140 drops these too).
|
||||
# The GlueXML registrar's address range: an id fired there names a glue-screen event.
|
||||
GLUE = (0x46A000, 0x476000)
|
||||
# `0x51bbb0`'s walk bound: every named unit-window field below it is the generic bridge (2140).
|
||||
# `0x51bbb0`'s walk bound: every named unit-window field below it fires through the generic bridge.
|
||||
UNIT_WINDOW_FIELDS = 0xB6
|
||||
# A helper "only verbs call" is trusted up to this many callers; past it, it is a hub.
|
||||
MAX_HELPER_CALLERS = 8
|
||||
@@ -105,7 +79,7 @@ def main():
|
||||
if len(starts) < 1000:
|
||||
sys.exit(f"names-5875.tsv gave only {len(starts)} functions — wrong path?")
|
||||
|
||||
# The verified extents: `size=` on the ledgers' `fn` rows (Ghidra's, via Closure.java).
|
||||
# Function extents: `size=` on the `fn` rows of the analysis's ledgers.
|
||||
extent = {} # fn start -> size in bytes
|
||||
for path in glob.glob(os.path.join(a.wow_re, "system/*/ledger.tsv")):
|
||||
for line in open(path, encoding="utf-8", errors="replace"):
|
||||
@@ -126,10 +100,8 @@ def main():
|
||||
continue
|
||||
fn = int(f[1], 16)
|
||||
if GLUE[0] <= fn < GLUE[1] or f[0].startswith("Get"):
|
||||
# A getter that fires an event is the cache-miss re-query (`GetInboxText` asking for
|
||||
# a body it does not hold yet, `GetAuctionItemInfo` for an item), announced when the
|
||||
# answer lands. That is a state event, and benilla's feeds fire it off the packet;
|
||||
# it is not a verb called FOR its event, which is the class this table is for.
|
||||
# A getter's fire is a cache-miss re-query announced when the answer lands
|
||||
# (`GetInboxText`, `GetAuctionItemInfo`): a state event, fired by benilla's feeds.
|
||||
continue
|
||||
verbs.setdefault(fn, f[0])
|
||||
if len(verbs) < 1000:
|
||||
@@ -147,10 +119,9 @@ def main():
|
||||
if len(sites) < 300:
|
||||
sys.exit(f"event-firesites.tsv gave only {len(sites)} sites — wrong path?")
|
||||
|
||||
# One pass over the disassembly: every call/jmp edge, and every padding boundary — an `int3`
|
||||
# run anywhere, or a `nop` run right after a `ret`. Not a `nop` run after a `jmp`: that is a
|
||||
# loop head's alignment inside a function, and taking it as a start manufactured phantom
|
||||
# functions nothing calls, which silently dropped a third of the table once.
|
||||
# One pass over the disassembly: every call/jmp edge, and every padding boundary (an `int3`
|
||||
# run, or a `nop` run right after a `ret`). A `nop` run after a `jmp` is a loop head's
|
||||
# alignment inside a function, not a start.
|
||||
edges = []
|
||||
after_pad = False
|
||||
ended = False # the previous real instruction was a `ret`
|
||||
@@ -180,7 +151,7 @@ def main():
|
||||
ordered = sorted(starts)
|
||||
|
||||
def enclosing(va):
|
||||
"""The function holding `va`, or None when `va` lies past its candidate's verified extent."""
|
||||
"""The function holding `va`; None when `va` lies past its candidate's extent."""
|
||||
fn = ordered[bisect.bisect_right(ordered, va) - 1]
|
||||
if fn in extent and va >= fn + extent[fn]:
|
||||
return None
|
||||
@@ -219,19 +190,17 @@ def main():
|
||||
out = os.path.normpath(a.out)
|
||||
with open(out, "w", encoding="utf-8") as fh:
|
||||
fh.write(
|
||||
"# The 1.12.1 client's FrameScript events fired FROM INSIDE A LUA VERB, keyed by the verb.\n"
|
||||
"# GENERATED — scripts/gen-reference-verb-events.py. Source: wow-5875-re's\n"
|
||||
"# re/audit/binding-shapes.tsv + re/events/event-firesites.tsv + re/names/out/names-5875.tsv\n"
|
||||
"# + system/ui/scratch/disasm-full.txt.\n"
|
||||
"# The 1.12.1 client's FrameScript events fired from inside a Lua verb, keyed by the verb.\n"
|
||||
"# Generated by scripts/gen-reference-verb-events.py from reference/1.12-shapes.tsv,\n"
|
||||
"# reference/1.12-event-firesites.tsv and the maintainer's analysis (`WOW_RE`).\n"
|
||||
"#\n"
|
||||
"# shape body = a fire site inside the verb's own extent.\n"
|
||||
"# helper = a fire site inside a function that ONLY registered verbs call\n"
|
||||
"# helper = a fire site inside a function that only registered verbs call\n"
|
||||
"# (at most 8 of them); `via` is that helper. A helper with any\n"
|
||||
"# non-verb caller is a state event's producer and is NOT here.\n"
|
||||
"# sites the fire sites themselves, for the grep into wow-re.\n"
|
||||
"# A floor, not a census: no row is not a claim of no fire (see the generator's header).\n"
|
||||
"# Sites are attributed against the ledgers' verified function extents; a site past its\n"
|
||||
"# candidate's extent is left out rather than mis-filed (decision 2257).\n"
|
||||
"# non-verb caller produces a state event and is not here.\n"
|
||||
"# sites the fire sites themselves.\n"
|
||||
"# A floor, not a census: a verb with no row may still fire an event.\n"
|
||||
"# A site past its candidate function's extent is left out rather than mis-filed.\n"
|
||||
f"# {len(table)} pairs over {len({v for v, _ in table})} verbs and"
|
||||
f" {len({e for _, e in table})} events; {unattributed} site(s) past a verified extent.\n"
|
||||
"verb\tevent\tshape\tvia\tsites\n"
|
||||
|
||||
+12
-22
@@ -1,28 +1,19 @@
|
||||
#!/usr/bin/env python3
|
||||
"""mixsum.py — read a `$WOW_MIX_TAP` capture and say, in numbers, what the mix was doing when.
|
||||
"""Read a `$WOW_MIX_TAP` capture and say, in numbers, what the mix was doing when.
|
||||
|
||||
The tap (decision 1112) records the final mix to a stereo float-32 WAV. Until now the only way to
|
||||
read one back was to listen to it — which is exactly the loop `docs/METHOD.md` step 4 forbids for a
|
||||
timing question ("timing and feel are measured, never eyeballed"), and which cannot answer the two
|
||||
questions a capture is usually taken for:
|
||||
|
||||
* **Was there a HOLE, and how long?** — "the login theme dies 2 s into a 6 s loading screen"
|
||||
(decision 1550) is a silent span with a start and an end, not an impression.
|
||||
* **Did the level do what the transition says it does?** — a 4.0 s linear fade-to-zero is a
|
||||
straight line in dBFS; a cut is a cliff; a crossfade has an overlap. All three read off the
|
||||
envelope.
|
||||
|
||||
So: a windowed RMS/peak envelope with a bar, plus a `spans` report that names every silent stretch
|
||||
and every onset/offset edge. `--json` for a machine, the default for a session log.
|
||||
The tap records the final mix to a stereo float-32 WAV. This prints a windowed RMS/peak envelope
|
||||
with a bar, and a `spans` report naming every silent stretch and every onset/offset edge: a hole
|
||||
is a silent span with a start and an end, and a linear fade-to-zero is a straight line in dBFS, a
|
||||
cut a cliff, a crossfade an overlap. `--json` for a machine, the default for a session log.
|
||||
|
||||
scripts/mixsum.py entry.wav # 250 ms windows, the envelope + the spans
|
||||
scripts/mixsum.py entry.wav --window 0.1 # finer, for a click or a declick ramp
|
||||
scripts/mixsum.py entry.wav --from 2 --to 12 # just the covered window
|
||||
scripts/mixsum.py entry.wav --quiet # the spans only, no per-window rows
|
||||
|
||||
Pure stdlib on purpose (no numpy on this machine, and an instrument that needs an install is an
|
||||
instrument nobody runs). Python's `wave` module refuses format 3 (IEEE float), so the 44-byte
|
||||
header the tap writes is parsed here directly — it is a frozen fact of `sound::mix_tap::wav_header`.
|
||||
Pure stdlib, so it runs without installing anything. Python's `wave` module refuses format 3
|
||||
(IEEE float), so the tap's 44-byte header is parsed directly; it must match
|
||||
`sound::mix_tap::wav_header`.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
@@ -32,9 +23,8 @@ import struct
|
||||
import sys
|
||||
from array import array
|
||||
|
||||
# Below this a window is "silence" for span purposes. The tap is pre-clamp and dither-free, so true
|
||||
# digital silence reads as -inf; -80 dBFS leaves room for a fade's last samples without calling an
|
||||
# audible tail silent.
|
||||
# Below this a window is silence: the tap is pre-clamp and dither-free, so true silence reads
|
||||
# -inf, and -80 dBFS still counts an audible fade tail as sound.
|
||||
SILENCE_DBFS = -80.0
|
||||
|
||||
|
||||
@@ -52,8 +42,8 @@ def read_tap(path):
|
||||
if head[36:40] != b"data":
|
||||
sys.exit("mixsum: no `data` chunk where the tap header puts it")
|
||||
raw = f.read()
|
||||
# The declared size is patched per flush; a hard kill can leave the file shorter. Trust the
|
||||
# bytes actually present, floored to whole frames.
|
||||
# The declared size is patched per flush and a hard kill can leave the file shorter: trust the
|
||||
# bytes present, floored to whole frames.
|
||||
usable = len(raw) - (len(raw) % 8)
|
||||
samples = array("f")
|
||||
samples.frombytes(raw[:usable])
|
||||
|
||||
@@ -1,18 +1,15 @@
|
||||
#!/usr/bin/env python3
|
||||
"""pass-span-lint — one `pass_span` per render pass at a time (decision 2258, bug B390).
|
||||
"""Refuse a second open `pass_span` on the same render pass (a wgpu validation error on Vulkan).
|
||||
|
||||
bevy's `DiagnosticsRecorder::pass_span` is a timestamp pair PLUS a pipeline-statistics query, and
|
||||
wgpu allows exactly one statistics query active at a time: a second `pass_span` opened on a pass
|
||||
whose own span is still open is a validation error — on the backends that expose
|
||||
`PIPELINE_STATISTICS_QUERY`, which in wgpu 27 is Vulkan alone (Metal never had it; wgpu-hal's
|
||||
DX12 arm has the line commented out). So the gates on this machine (Metal) and the lab laptop
|
||||
(pinned to DX12) run the nesting as a silent no-op, and the 09-15 sync shipped one: every Linux
|
||||
player, and every Windows player wgpu had put on Vulkan, aborted on the first world frame (B390).
|
||||
No compile, no test and no macOS run can see that shape. This check can, in a tenth of a second.
|
||||
bevy's `DiagnosticsRecorder::pass_span` is a timestamp pair plus a pipeline-statistics query, and
|
||||
wgpu allows one statistics query active at a time: a second `pass_span` on a pass whose span is
|
||||
still open is a validation error on backends with `PIPELINE_STATISTICS_QUERY`, which in wgpu 27 is
|
||||
Vulkan alone (Metal lacks it; wgpu-hal's DX12 arm has the line commented out). On Metal and DX12
|
||||
the nesting is a silent no-op, so no macOS build, test or run sees it.
|
||||
|
||||
The shape it refuses, per function: `let g = <recorder>.pass_span(&mut P, …)` followed by another
|
||||
`.pass_span(&mut P, …)` before `g.end(&mut P)`. Timestamp-only spans (`time_span`) nest fine and
|
||||
are not looked at. Lexical, per file, reset at every `fn` — a span never outlives its function.
|
||||
are not looked at. Lexical, per file, reset at every `fn`: a span never outlives its function.
|
||||
|
||||
Usage: scripts/pass-span-lint.py [path …] (default: every crates/**/*.rs)
|
||||
Exit: 0 clean, 1 findings.
|
||||
|
||||
+22
-38
@@ -1,22 +1,18 @@
|
||||
#!/usr/bin/env python3
|
||||
"""soundprobe.py — read back a `$WOW_SOUND_PROBE` capture and name the mechanism.
|
||||
"""Read back a `$WOW_SOUND_PROBE` capture and name the mechanism behind a speaker-breaking sound.
|
||||
|
||||
A measuring run (decision 1556) leaves three files: `pre.wav` (the mix as the game asked for it),
|
||||
`post.wav` (what was heard), and `timeline.jsonl` (levels, voices, deadline misses, kit starts, and
|
||||
the director's F9 marks — all keyed to the sample offset in `pre.wav`).
|
||||
|
||||
This reads them together and answers the question the capture was taken for: **of the mechanisms
|
||||
that sound like a speaker breaking, which one is actually present?** They are distinguishable, but
|
||||
only by comparing the two WAVs against each other and against the timeline — no single file
|
||||
separates them:
|
||||
A measuring run leaves `pre.wav` (the mix as the game asked for it), `post.wav` (what was heard)
|
||||
and `timeline.jsonl` (levels, voices, deadline misses, kit starts and F9 marks, keyed to the
|
||||
sample offset in `pre.wav`). Only comparing the two WAVs with each other and with the timeline
|
||||
tells the mechanisms apart:
|
||||
|
||||
over-scale sum ............ pre past 1.0, post clean, gain well under 1
|
||||
limiter not engaging ...... pre past 1.0 AND post past 1.0
|
||||
limiter not engaging ...... pre and post both past 1.0
|
||||
underrun / missed deadline both clean; `load>=1`, overruns, block-aligned steps in post
|
||||
starved stream decoder .... both clean; hard steps to zero in post
|
||||
non-finite samples ........ NaN/inf anywhere — invisible to every level meter
|
||||
non-finite samples ........ NaN/inf anywhere, invisible to every level meter
|
||||
voice refusal ............. `refused` climbing near the mark
|
||||
the limiter's own pumping . post clean but gain diving repeatedly (1551 made it worse)
|
||||
the limiter's own pumping . post clean but gain diving repeatedly
|
||||
|
||||
Usage:
|
||||
scripts/soundprobe.py # the default benilla-config/sound-probe
|
||||
@@ -24,9 +20,8 @@ Usage:
|
||||
scripts/soundprobe.py <dir> --around 2.0 # widen the window read around each mark
|
||||
scripts/soundprobe.py <dir> --json
|
||||
|
||||
Pure stdlib on purpose (no numpy on this machine — an instrument that needs an install is an
|
||||
instrument nobody runs). Windowed reductions go through `max`/`min` on array slices so the hot
|
||||
loop stays in C; the per-sample passes run only near the marks, where they are affordable.
|
||||
Pure stdlib, so it runs without installing anything. Windowed reductions use `max`/`min` on array
|
||||
slices so the hot loop stays in C; the per-sample passes run only near the marks.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
@@ -37,20 +32,16 @@ import struct
|
||||
import sys
|
||||
from array import array
|
||||
|
||||
# A sample this close to full scale was clipped, or was pushed there by a limiter that had to
|
||||
# choose. Below the renderer's own clamp of 1.0 so a limiter riding its 0.99 ceiling reads as
|
||||
# "held", not as "clipped".
|
||||
# A sample this close to full scale was clipped: below the renderer's 1.0 clamp, above the
|
||||
# limiter's 0.99 ceiling, so a limiter riding its ceiling reads as held, not clipped.
|
||||
CLIPPED = 0.999
|
||||
# Envelope resolution. 10 ms is finer than any transient we care to place and keeps a ten-minute
|
||||
# capture to ~60 000 windows.
|
||||
# Envelope resolution: 10 ms keeps a ten-minute capture to ~60 000 windows.
|
||||
WINDOW = 0.010
|
||||
# A sample-to-sample jump this large is not music. Real 44.1 kHz content moves slowly between
|
||||
# adjacent samples; a hard step is a zero-fill, a cut, or a stepped parameter — the waveform
|
||||
# signature the level meters cannot see.
|
||||
# A sample-to-sample jump this large is not music: a zero-fill, a cut or a stepped parameter,
|
||||
# which no level meter sees.
|
||||
STEP = 0.30
|
||||
# kira's internal block. A discontinuity landing on a multiple of this is a *block-level* failure
|
||||
# (a starved decoder zero-filling a whole chunk), not a one-off click — the distinction between
|
||||
# decision 1109's mechanism and a de-click bug.
|
||||
# kira's internal block: a step on a multiple of it is a block-level failure (a starved decoder
|
||||
# zero-filling a whole chunk), not a one-off click.
|
||||
BLOCK = 128
|
||||
|
||||
|
||||
@@ -80,10 +71,8 @@ def envelope(samples, rate, window=WINDOW):
|
||||
seg = samples[i : i + step]
|
||||
if not seg:
|
||||
break
|
||||
# NaN/inf first, and via a SUM rather than a comparison: `max`/`min` compare with `>`,
|
||||
# which is false for every NaN, so a NaN in the window can be silently skipped depending on
|
||||
# where it sits. A sum propagates it unconditionally — the same trap this whole capture
|
||||
# exists because of (`f32::max` discards NaN; see sound::meter).
|
||||
# Finiteness via a sum, not `max`/`min`: a comparison with NaN is false, so `max` can skip
|
||||
# a NaN depending on where it sits, while a sum always propagates it.
|
||||
if not math.isfinite(sum(seg)):
|
||||
nf = sum(1 for v in seg if not math.isfinite(v))
|
||||
finite = [v for v in seg if math.isfinite(v)]
|
||||
@@ -99,12 +88,8 @@ def envelope(samples, rate, window=WINDOW):
|
||||
def brightness(samples, rate, lo, hi):
|
||||
"""A crude high-frequency energy ratio for [lo,hi): `sum((x[n]-x[n-1])^2) / sum(x[n]^2)`.
|
||||
|
||||
NOT a spectrum — it is a one-tap difference filter, which is all that is affordable in stdlib.
|
||||
But it is enough for the question a mark actually poses: **was that moment harsher, or just
|
||||
louder?** Distortion of any origin — a clipped waveform, aliasing from a pitch-varied playback
|
||||
rate, a stepped parameter — puts energy high in the band, and this number rises with it while
|
||||
staying flat for content that merely got loud. Meaningful only against the same capture's own
|
||||
baseline, so it is always reported next to one.
|
||||
A one-tap difference filter, not a spectrum: it rises with distortion of any origin and stays
|
||||
flat for content that only got louder. Meaningful only against the same capture's baseline.
|
||||
"""
|
||||
a, b = max(0, int(lo * rate)) * 2, min(len(samples), int(hi * rate) * 2)
|
||||
num = den = 0.0
|
||||
@@ -206,8 +191,7 @@ def main():
|
||||
min_gain = min((r.get("gain", 1.0) for r in ticks), default=1.0)
|
||||
marks = [r for r in rows if r.get("ev") == "mark"]
|
||||
|
||||
# A brightness baseline from windows spread across the capture, so "harsher than usual" is a
|
||||
# statement about this session rather than about audio in general.
|
||||
# The brightness baseline: the median of windows spread across this capture.
|
||||
base = []
|
||||
if dur > 1.0:
|
||||
for k in range(1, 12):
|
||||
|
||||
+26
-37
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Read a `vpl`/`bub` overlay trace and say WHERE the overlay's motion came from — projection, solve, or snap.
|
||||
"""Read a `vpl`/`bub` overlay trace and say where the motion came from: projection, solve or snap.
|
||||
|
||||
# walk a town with friendly plates on, tracing one line per plate per frame
|
||||
WOW_WIN=1440x810 WOW_NOSOUND=1 WOW_GM=off \
|
||||
@@ -10,35 +10,28 @@
|
||||
WOW_PROBE_EXIT_AT=34 cargo run -q -p benilla
|
||||
python3 scripts/vplsum.py /tmp/vpl.trace --from 24 --to 32 # the walk leg only
|
||||
|
||||
Why it exists (decision 1341): "the plates are jittery when moving" is four different bugs wearing
|
||||
one coat — a noisy camera, a stale anchor, the anti-overlap solve relocating a plate, or the pixel
|
||||
snap quantizing its glide — and the eye cannot tell them apart. Every `vpl` line carries the world
|
||||
anchor, the camera pose, the RAW projected point, the SOLVED rect and the SNAPPED rect, so the
|
||||
question is arithmetic instead of judgement. It found phantom plates (30% of everything drawn was a
|
||||
unit off screen, dragged to the border by the seat's clamp) in one run.
|
||||
Jitter in a moving overlay has four causes the eye cannot tell apart: a noisy camera, a stale
|
||||
anchor, the anti-overlap solve relocating a plate, or the pixel snap quantizing its glide. Every
|
||||
`vpl` line carries the world anchor, the camera pose, the raw projected point, the solved rect and
|
||||
the snapped rect, so telling them apart is arithmetic.
|
||||
|
||||
Read it in this order:
|
||||
|
||||
* **projection smoothness** — the frame-to-frame acceleration of the raw projected point, per unit.
|
||||
Small (~0.05 px at 60 fps) means the camera and the anchor are innocent and the cause is
|
||||
downstream. Large means stop reading and go look at the camera or the anchor.
|
||||
* **off-screen** — plates drawn for a unit that is not on screen. Must be 0.
|
||||
* **solve jumps** — a plate moving further than its own projection did. Some is the reference's own
|
||||
anti-overlap bounce (a plate re-solving when a blocker moves); a lot, or hundreds of pixels in a
|
||||
frame, is a defect.
|
||||
* **snap** — the extra per-frame displacement the pixel snap adds to a plate's glide. It is bounded
|
||||
by half a device pixel by construction; anything larger means the snap is on the wrong grid.
|
||||
* projection smoothness: the raw projected point's frame-to-frame acceleration, per unit; small
|
||||
(~0.05 px at 60 fps) clears the camera and the anchor.
|
||||
* off-screen: plates drawn for a unit that is not on screen. Must be 0.
|
||||
* solve jumps: a plate moving further than its own projection did; some is the reference's own
|
||||
anti-overlap bounce, hundreds of pixels in a frame is a defect.
|
||||
* snap: the pixel snap's extra per-frame displacement, at most half a device pixel; more means
|
||||
the snap is on the wrong grid.
|
||||
|
||||
The `--from`/`--to` window matters as much as it does for `tracesum.py`: a run is a login, a
|
||||
teleport, a stand and a walk glued together, and their average describes none of them.
|
||||
Pick one leg with `--from`/`--to`: a run is a login, a teleport, a stand and a walk, and their
|
||||
average describes none of them.
|
||||
|
||||
**It reads the CHAT BUBBLE's `bub` lines too** (1398), auto-detected from the file — the bubble is
|
||||
the plate's sibling overlay (same `plate_basis`/`gx_px`/`device_snap` family, same projector, same
|
||||
`overhead_anchor`), so "the bubble is jittery when running" is the same four-way question and
|
||||
deserves the same arithmetic rather than a second script that drifts from this one. The bubble has
|
||||
no anti-overlap solver, so its lines carry no `solved=` stage and the SOLVE section is skipped; the
|
||||
ANCHOR section below is the one that matters for it, because the bubble's height rides the *posed*
|
||||
head attachment and therefore bobs with the run cycle:
|
||||
It reads the chat bubble's `bub` lines too, detected from the file: same projector and
|
||||
`overhead_anchor` as the plate but no anti-overlap solver, so there is no `solved=` stage and the
|
||||
SOLVE section is skipped. The ANCHOR section matters most for it, because the bubble rides the
|
||||
posed head attachment, which bobs with the run cycle:
|
||||
|
||||
WOW_MOVE_TRACE=/tmp/bub.trace WOW_MOVE_TRACE_TAGS=bub \
|
||||
WOW_PROBE_CHAT="hello" WOW_PROBE_CHAT_AT=20 WOW_PROBE_KEY="W@20:6" ... cargo run -q -p benilla
|
||||
@@ -50,19 +43,18 @@ import re
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
|
||||
# One regex for both overlays: the `solved=` stage is the plate's anti-overlap solver, which the
|
||||
# bubble has no counterpart to, so it is optional and the final rect is `plate=` or `frame=`.
|
||||
# One regex for both overlays: `solved=` (the plate's anti-overlap solve) is optional, and the
|
||||
# final rect is `plate=` or `frame=`.
|
||||
LINE = re.compile(
|
||||
r"t=\s*([\d.]+) (vpl|bub)\s+e=(\d+) vp=\((\d+),(\d+)\) anchor=\[([-\d.,e]+)\] "
|
||||
r"cam=\[([-\d.,e]+)\] fwd=\[([-\d.,e]+)\] scr=\(([-\d.e]+),([-\d.e]+)\) "
|
||||
r"(?:solved=\(([-\d.e]+),([-\d.e]+)\) )?(?:plate|frame)=\(([-\d.e]+),([-\d.e]+)\)"
|
||||
)
|
||||
|
||||
# A step this far past the plate's own projected motion is a JUMP, not a glide: about a quarter of
|
||||
# the plate's height, i.e. the smallest relocation a viewer reads as the plate moving by itself.
|
||||
# A step this far past the plate's own projected motion is a jump, not a glide: about a quarter of
|
||||
# the plate's height, the smallest relocation a viewer reads as the plate moving by itself.
|
||||
JUMP_PX = 4.0
|
||||
# Frames further apart than this are not consecutive — the plate blinked out and came back, and
|
||||
# differencing across the gap would invent a jump.
|
||||
# Frames further apart than this are a gap (the plate blinked out), never differenced.
|
||||
GAP_S = 0.1
|
||||
|
||||
|
||||
@@ -73,8 +65,7 @@ def vec3(s):
|
||||
def load(path, lo, hi):
|
||||
"""-> (tag, viewport, {entity: [(t, anchor, cam, scr, solved, rect)]}, {t: overlay count})
|
||||
|
||||
A bubble line has no solver stage; its `solved` slot mirrors the raw point, which makes the
|
||||
SOLVE arithmetic below identically zero rather than special-cased (the section is skipped).
|
||||
A bubble line has no solver stage, so its `solved` slot mirrors the raw point.
|
||||
"""
|
||||
rows, frames, vp, tag = defaultdict(list), defaultdict(int), None, None
|
||||
for line in open(path):
|
||||
@@ -173,10 +164,8 @@ def main():
|
||||
f"scr ({j[5][0]:8.1f},{j[5][1]:7.1f}) -> ({j[6][0]:8.1f},{j[6][1]:7.1f})"
|
||||
)
|
||||
|
||||
# The world anchor's own motion, split by axis. The HORIZONTAL term is the unit gliding; the
|
||||
# VERTICAL term is what the run cycle does to the posed head attachment (plus terrain). A
|
||||
# standing unit reads 0.0000 in both — which is exactly why 1341 cleared the anchor for the
|
||||
# plate and why that clearance says nothing about a unit that is RUNNING (1398).
|
||||
# The world anchor's own motion by axis: horizontal is the unit gliding, vertical is the run
|
||||
# cycle moving the posed head attachment (plus terrain). A standing unit reads 0 in both.
|
||||
print("\nANCHOR motion — the world point's own Δ per frame, yd (med p90 max):")
|
||||
for e, s in sorted(rows.items(), key=lambda kv: -len(kv[1]))[:8]:
|
||||
pairs = [(a_, b) for a_, b in zip(s, s[1:]) if b[0] - a_[0] < GAP_S]
|
||||
|
||||
Reference in New Issue
Block a user