fix(texture): keep NPOT out of the recycle caches, allow sides below 8

The dimension gate routes NPOT textures around the recycle pool on
allocation, but the free path filed them anyway: TexturePoolRelease
buckets by each side's power-of-two factor with no POT test, so a freed
48x48 landed in the 32x32 bucket and was handed to the next 32x32
request. Pool on free exactly what the allocator pools (POT, no larger
than the pool max), as 4.3.4's GxTexReusable does.

The decoded-buffer cache (TextureAllocMips / TextureFreeMips) buckets
the same way, so an NPOT BLP on the model/character cache path could be
given a smaller cached buffer and decode past it. NPOT sizes now take
the engine's own uncached alloc and free.

The create validator's 8px minimum is lowered to 1, so 1x1 and 4x4
textures load. A failed D3D create leaves the texture blank.
This commit is contained in:
Brues
2026-10-26 16:31:10 -05:00
parent cca8ff3c55
commit f5082cedee
2 changed files with 111 additions and 18 deletions
+31
View File
@@ -11654,6 +11654,12 @@ enum Offsets {
// (Texture::DimensionGate): route a texture larger than the pool can index to
// the engine's own non-pooled free FUN_TEXTURE_FREE_RAW — exactly what the
// guard's else-branch already does for too-small / too-many-mip textures.
// The same gate has no power-of-two test either (decompiled; benilla-re
// TexturePoolRelease), so an NPOT texture files by its power-of-two FACTOR —
// a 600x600 lands in the 64x64 bucket — and the allocator's hit test, which
// compares only filter bits and format, hands it to the next 64x64 request.
// 4.3.4's TextureFreeGxTex gates on GxTexReusable (target == GxTex_2d, never
// GxTex_NonPow2), so NPOT is routed to the raw free as well.
FUN_TEXTURE_POOL_FREE = 0x00448670, // __fastcall(tex); the pool free path
FUN_TEXTURE_FREE_RAW = 0x0058AD60, // __fastcall(tex); non-pooled free (device release)
FUN_TEXTURE_GET_DESC = 0x0058ADA0, // __fastcall(tex, out*) — fills width@+4, height@+8
@@ -11695,6 +11701,20 @@ enum Offsets {
// returns is precisely what the scratch slot expects — and SMemAlloc-backed,
// so the exit teardown's SMemFree on it is valid.
FUN_TEXTURE_DECODE_BUFFER_ALLOC = 0x005A4AF0,
// That recycle front and its free: a per-format cache of decoded buffers for
// sides 8..256, bucketed by `tz(w>>3)*6 + tz(h>>3)` with no power-of-two test
// on either side (verified, disassembly). Freeing an NPOT buffer files it
// under its power-of-two factor and a later request of that bucket's size
// gets it — or, the dangerous way round, a 200x100 request gets the 8x32
// buffer the bucket holds and decodes past its end. Out-of-window sizes take
// FUN_TEXTURE_DECODE_BUFFER_ALLOC / FUN_STORM_SMEM_FREE directly, so NPOT is
// routed there too (Texture::DimensionGate). Callers: the BLP IMG/cache path
// FUN_00449840 and the TGA decoder FUN_004491F0 when its slot is NULL (the
// live TGA path pre-seeds the slot with the scratch, so it never allocates);
// the free is ~CTexture with the texture's +0x118 type code and +0x144/+0x148
// size.
FUN_TEXTURE_ALLOC_MIPS = 0x0044AF90, // __fastcall(format, width, [stack] height) -> buffer, RET 4
FUN_TEXTURE_FREE_MIPS = 0x0044B110, // __fastcall(buffer, format, [stack] width, height), RET 8
// Device-relative: the caps block at +0x23C (what FUN_0058A230 returns) plus
// the type-0 (2D) max-texture-dimension slot at +0x60 — the exact field the
// create validator FUN_0058AB10 compares width and height against
@@ -11761,6 +11781,17 @@ enum Offsets {
// type-2 flags check.
PATCH_TEXVALIDATE_POT_W = 0x0058AB7C,
PATCH_TEXVALIDATE_POT_H = 0x0058AB8D,
// The imm8 of the validator's minimum side, `CMP EDI,8` (83 FF 08) /
// `CMP EBX,8` (83 FB 08), each followed by JC fail. 4.3.4 keeps the same 8
// ("Texture size must be at least 8x8"). Lowered to 1 so 0 is still refused.
// Nothing downstream needs 8: the allocator sends sides <= 16 straight here,
// the pool free takes only >= 32, the mip cache sends < 8 to the plain
// allocator, and a stock mip chain already uploads levels down to 1x1. If the
// D3D create fails anyway (a driver refusing a sub-4 DXT top level), LoadTexture
// 0x005A0D70 skips the upload while the D3D object at +0x48 is NULL, so the
// texture just stays blank.
PATCH_TEXVALIDATE_MIN_W_IMM = 0x0058AB42,
PATCH_TEXVALIDATE_MIN_H_IMM = 0x0058AB4E,
// --- Inline-texture positioning (slice 1: measure/emit integration) --------
// The 1.12 text pipeline builds a layout as a list of render "nodes" (one
+80 -18
View File
@@ -78,12 +78,21 @@
// failed" for a texture the GPU can't hold. It acts only when the slot IS
// the scratch: the IMG path decompresses into a private, right-sized
// buffer of its own and is left untouched.
// 6. The recycle-pool free follows the same bound (always on). The free path
// files a texture into a bucket with no upper size check; the cave lets
// textures larger than the pool can index exist, so freeing one overran
// the bucket table (a 2048x2048 crashed at teardown). A hook routes any
// texture the table can't index to the non-pooled free instead — the
// mirror of the allocator's cave, and the missing half of the size gate.
// 6. The recycle-pool free follows the same rule (always on). The free path
// files a texture into a bucket with no upper size check and no
// power-of-two check; the cave lets both kinds exist. Freeing an oversized
// one overran the bucket table (a 2048x2048 crashed at teardown); freeing
// an NPOT one filed it under its power-of-two factor, where the next POT
// request of that size was handed the wrong-sized texture. A hook routes
// anything the allocator would not have pooled to the non-pooled free —
// the mirror of the allocator's cave, and the missing half of the gate.
// 7. The decoded-buffer cache follows it too (always on). Its alloc and free
// bucket by power-of-two factor the same way, so an NPOT BLP on the model /
// character cache path could be handed a smaller buffer and decode past
// it. NPOT sizes take the engine's own uncached alloc and free, as sizes
// outside the cache's 8..256 window already do.
// Minimum side (always on). The create validator refuses sides below 8; it
// now refuses only 0, so a 1x1 or 4x4 texture loads.
// 5. The async fallback guard (always on). FUN_TEXTURE_FORCE_LOAD shoves a
// read that never fit the shared async buffer into a 512 KiB static
// fallback with no size check — a pre-existing engine bug the stock size
@@ -140,6 +149,9 @@ bool WriteNearJump(uintptr_t site, uint8_t cond, uintptr_t target) {
return Common::PatchBytes(reinterpret_cast<void *>(site), bytes, sizeof bytes);
}
// 0 counts as a power of two, as in every engine test of `v & (v - 1)`.
bool IsPowerOfTwo(uint32_t v) { return (v & (v - 1)) == 0; }
// ---- what the engine was actually configured with ----------------------------
// Side of the decode scratch. Read back from the two immediates FUN_00448BD0
@@ -281,8 +293,7 @@ uintptr_t Cave() { return reinterpret_cast<uintptr_t>(g_cave); }
// ---- state -------------------------------------------------------------------
uint32_t g_poolMax = 0; // what the in-function CMPs admit to the pool
uint32_t g_poolIndexMax = 0; // the largest dimension the bucket table can index
uint32_t g_poolMax = 0; // what the in-function CMPs admit to the pool
int g_poolSide = 0;
bool g_strideFixed = false;
@@ -291,7 +302,6 @@ bool g_strideFixed = false;
bool ApplySizeGate() {
g_poolSide = PoolSide();
const uint32_t indexable = PoolMaxDimension(g_poolSide);
g_poolIndexMax = indexable; // the free-path guard's bound (layer 6)
const uint32_t scratch = ScratchDimension();
// A texture that enters the pool never reaches the cave, so it must also
// fit the scratch as it stands. (A VanillaHelpers install whose scratch
@@ -414,20 +424,22 @@ using TexGetDesc_t = void(__fastcall *)(void *tex, uint32_t *out);
using TexFreeRaw_t = void(__fastcall *)(void *tex);
// The pool free files a texture into a bucket keyed by each dimension's
// power-of-two factor, with no upper bound (see Offsets.h). The allocator's cave
// lets textures larger than the pool can index exist; freeing one files it past
// the bucket table and writes through garbage. Exclude the same textures here:
// route anything the table can't index to the engine's own non-pooled free —
// exactly the guard's else-branch for too-small / too-many-mip textures. Cold
// path (texture teardown), so a MinHook here carries no per-frame concern, and
// nothing else patches this function's code.
// power-of-two factor, with no upper bound and no power-of-two test (see
// Offsets.h). The allocator's cave lets both oversized and NPOT textures exist:
// an oversized one files past the bucket table and writes through garbage, an
// NPOT one files under a POT size and is recycled for it. Pool here exactly
// what the allocator pools — a power of two on both sides, no larger than
// g_poolMax — and route the rest to the engine's own non-pooled free, the
// guard's else-branch for too-small / too-many-mip textures. Cold path (texture
// teardown), so a MinHook here carries no per-frame concern, and nothing else
// patches this function's code.
void __fastcall PoolFree_h(void *tex) {
if (g_poolIndexMax != 0 && tex != nullptr) {
if (g_poolMax != 0 && tex != nullptr) {
uint32_t desc[16] = {0};
reinterpret_cast<TexGetDesc_t>(Offsets::FUN_TEXTURE_GET_DESC)(tex, desc);
const uint32_t w = desc[Offsets::OFF_TEXDESC_WIDTH / 4];
const uint32_t h = desc[Offsets::OFF_TEXDESC_HEIGHT / 4];
if (w > g_poolIndexMax || h > g_poolIndexMax) {
if (!IsPowerOfTwo(w) || !IsPowerOfTwo(h) || w > g_poolMax || h > g_poolMax) {
reinterpret_cast<TexFreeRaw_t>(Offsets::FUN_TEXTURE_FREE_RAW)(tex);
return;
}
@@ -438,6 +450,41 @@ void __fastcall PoolFree_h(void *tex) {
const Game::HookAutoRegister _poolFreeHook{Offsets::FUN_TEXTURE_POOL_FREE, &PoolFree_h,
reinterpret_cast<void **>(&PoolFree_o)};
// ---- layer 7: the decoded-buffer cache follows the same rule -------------------
using AllocMips_t = void *(__fastcall *)(uint32_t format, uint32_t width, uint32_t height);
using FreeMips_t = void(__fastcall *)(void *buffer, uint32_t format, uint32_t width,
uint32_t height);
AllocMips_t AllocMips_o = nullptr;
FreeMips_t FreeMips_o = nullptr;
// The cache buckets by power-of-two factor (see Offsets.h), so an NPOT size is
// neither cached nor served from it: it takes the same uncached alloc and free
// the engine uses outside the cache's 8..256 window. Both halves must agree, or
// a buffer allocated here would be filed by the original free. Once per texture
// load and unload.
void *__fastcall AllocMips_h(uint32_t format, uint32_t width, uint32_t height) {
if (!IsPowerOfTwo(width) || !IsPowerOfTwo(height))
return reinterpret_cast<DecodeBufferAlloc_t>(Offsets::FUN_TEXTURE_DECODE_BUFFER_ALLOC)(
static_cast<int>(format), width, height, kAllocTag, __LINE__);
return AllocMips_o(format, width, height);
}
void __fastcall FreeMips_h(void *buffer, uint32_t format, uint32_t width, uint32_t height) {
if (!IsPowerOfTwo(width) || !IsPowerOfTwo(height)) {
if (buffer != nullptr)
reinterpret_cast<SMemFree_t>(Offsets::FUN_STORM_SMEM_FREE)(buffer, kAllocTag, __LINE__,
0);
return;
}
FreeMips_o(buffer, format, width, height);
}
const Game::HookAutoRegister _allocMipsHook{Offsets::FUN_TEXTURE_ALLOC_MIPS, &AllocMips_h,
reinterpret_cast<void **>(&AllocMips_o)};
const Game::HookAutoRegister _freeMipsHook{Offsets::FUN_TEXTURE_FREE_MIPS, &FreeMips_h,
reinterpret_cast<void **>(&FreeMips_o)};
// ---- non-power-of-two ----------------------------------------------------------
// The validator computes "is a power of two" as
@@ -461,6 +508,20 @@ bool ApplyNonPowerOfTwo() {
return ok;
}
// ---- minimum side --------------------------------------------------------------
// The validator's `CMP side,8 ; JC fail` per axis, lowered to 1 so only a 0
// side is refused. See Offsets.h for why nothing downstream depends on 8.
bool ApplyMinimumSide() {
const uint8_t one = 1;
bool ok = Common::PatchBytes(reinterpret_cast<void *>(Offsets::PATCH_TEXVALIDATE_MIN_W_IMM),
&one, sizeof one);
ok = Common::PatchBytes(reinterpret_cast<void *>(Offsets::PATCH_TEXVALIDATE_MIN_H_IMM), &one,
sizeof one) &&
ok;
return ok;
}
// Re-asserted on every registration so the code segment and our recorded state
// cannot disagree. Order matters: the cave must exist before anything is
// pointed at it.
@@ -469,6 +530,7 @@ void ApplyGates() {
return;
ApplyStrideFix();
ApplyNonPowerOfTwo();
ApplyMinimumSide();
}
// ---- Lua ---------------------------------------------------------------------