diff --git a/LIGHTING.md b/LIGHTING.md index 651e4aa7..49eaa996 100644 --- a/LIGHTING.md +++ b/LIGHTING.md @@ -20,7 +20,7 @@ behind each constant. | Synthesised lights | torches, braziers, lanterns, campfires and lampposts emit a light in the colour of their flame, with flicker | `fireLightGain`, `fireFlicker` | | Torch shadows | cube-map shadows from point lights onto buildings, models and terrain, with a contact-hardening penumbra and a shadow floor | `interiorShadows`, `exteriorShadows`, `torchShadowStrength`, `interiorShadowSoft` | | Daylight and doorways | calibrated fixtures at doors, windows and open boundaries carry daylight into a room, and doorways between rooms bleed light | `interiorDaylight`, env `WOW_DAYLIGHT`, `WOW_BLEED` | -| Terrain in torch shadows | the ground casts into an outdoor fire's cube map, so a hill or bank blocks the fire (settled exterior slots only; `torch_terrain.rs` gathers the resident MCNK chunks inside the cube's 48 yd) | `torchTerrainShadows` (default 0, High 1), env `WOW_TORCH_TERRAIN=0\|1` | +| Terrain in torch shadows | the ground casts into an outdoor fire's cube map, so a hill or bank blocks the fire (settled exterior slots only; `torch_terrain.rs` gathers the resident MCNK chunks inside the cube's 48 yd; the gather and the ground receiver both work, but stock Elwynn lamps/campfires show no visible change: the terrain-blocked ground is grazing (N·L ≈ 0, inside the 0.15 yd normal offset) or at the pool rim, below one 8-bit step — measured by the `lamp_terrain_occlusion_scan` census) | `torchTerrainShadows` (default 0, High 1), env `WOW_TORCH_TERRAIN=0\|1` | | City day floor | `interiorDaylight`'s room floor also reaches Stormwind/Ironforge rooms that the portal graph connects to the sky (an exterior-facing portal, or one hop from one; `district_sky_rooms` in `daylight.rs`); before this no city room could take it | `interiorDaylight` (default 0, suggested High 0.10) | | Spell and firework lights | fire, holy and fel effects light their surroundings for their lifetime; frost, nature, arcane and shadow do not | `spellLightGain`, env `WOW_SPELL_LIGHT=0` | | Moon shadows | at night the same shadow rig re-aims at the moon and casts a faint shadow; dims only the night sky term, never point lights | `moonShadowStrength` | @@ -31,11 +31,11 @@ behind each constant. | Sky quality | Enhanced: the five Light.dbc sky stops through a smooth monotone curve in linear light (no bands at the rings), a soft sun glow tinted by sun and fog colour (fades at night and under cloud), a procedural star field with twinkle and a faint Milky Way over the stock `Stars.m2`. High adds domain-warped cloud detail and sun-lit clouds (self-shadow, silver lining; technique from WarcraftXL, see `THIRD-PARTY.md`). Classic is the reference sky unchanged | Advanced Graphics → Sky Quality, cvar `skyQuality` 0 Classic / 1 Enhanced / 2 High, env `WOW_SKY_QUALITY` | | Modern fog | MONKEY (fog): radial distance fog with a gradual exponential curve, a daylight sun lobe, and an end colour shared by the world, WDL hull, sky horizon and volumetric haze. Each zone keeps its authored fog end up to the reference view-distance limit; optional `LightFogBand.dbc` rows provide height, sun and end-fog controls. Interior WMO fog stays classic | Advanced Graphics → Modern Fog, cvar `fogModel` 0 Classic (default) / 1 Modern (High = 1), env `WOW_FOGMODEL=0\|1`; `fog_hook.wgsl`, `lighting/fog_model.rs`, `light/fog_band.rs` | | Rain on surfaces | MONKEY (wet): rain gradually darkens and saturates exposed terrain and models, adds puddles and a sky/sun sheen, and produces rings on exterior Enhanced water. Wetness rises over roughly 90 seconds and dries over roughly four minutes; MONKEY (rainshelter): a camera-centred rain-occlusion map (128² cells of 0.75 yd, filled by vertical collision rays) keeps ground, walls and water under roofs, porches and bridges dry, with a ~1.5 yd drip margin and a soft edge; tree canopies without colliders do not shelter | Advanced Graphics → Wet Surfaces in Rain, cvar `rainSurfaces` 0/1 (default and High 1), env `WOW_RAIN_SURFACES`, `WOW_WETNESS`, `WOW_WET_T`; `weather/wetness.rs`, `weather/shelter.rs`, `wet_hook.wgsl` | -| Foliage wind | one weather-fed gust/veer field drives grass and classified tree/bush foliage; grass also parts around the player and nearby units | Advanced Graphics → Foliage Wind, cvar `foliageWind` 0 Off / 1 Grass / 2 Grass + Trees (High = 2), capture override `WOW_FOLIAGE_WIND` (dev build + `WOW_CAPTURE` only); field and benders in `benilla-world/src/wind/`. The wave phase is the CPU-integrated travel (speed integrated over time, wrapped seamlessly), and the waves' spatial term uses the fixed profile heading; the veer turns only the bend. Tree exile copies sway through material marker bit 14 (`FOLIAGE_WIND_MARKER`). Capture offset `WOW_CAPTURE_WIND_T` (s). Known: shadow casters (world-shadow proxies, `torch_depth.wgsl`) do not apply the sway, so leaves sample a static shadow; a future world-camera DepthPrepass must call the same offsets | +| Foliage wind | one weather-fed gust/veer field drives grass and classified tree/bush foliage; grass also parts around the player and nearby units | Advanced Graphics → Foliage Wind, cvar `foliageWind` 0 Off / 1 Grass / 2 Grass + Trees (High = 2), plus the slider Foliage Wind Strength, cvar `foliageWindStrength` 0.25..3 (default 1, not on the preset ladder; a gain on the sway, not the parting; a storm adds up to +60 %). Tuning at 1 (fix-wind): grass tips about ±0.2-0.3 yd on a 2.5 s wave with 1.1 s cross flutter, gust bands 55 yd wide rolling downwind; tree crowns about ±0.3 yd (lever-arm weight grows with height to 1.3x at 8.5 yd) on a 4 s sway, 5.5 s side wave, 0.8 s leaf flutter; bent tips dip slightly. capture override `WOW_FOLIAGE_WIND` (dev build + `WOW_CAPTURE` only); field and benders in `benilla-world/src/wind/`. The wave phase is the CPU-integrated travel (speed integrated over time, wrapped seamlessly), and the waves' spatial term uses the fixed profile heading; the veer turns only the bend. Tree exile copies sway through material marker bit 14 (`FOLIAGE_WIND_MARKER`). Capture offset `WOW_CAPTURE_WIND_T` (s). Known: shadow casters (world-shadow proxies, `torch_depth.wgsl`) do not apply the sway, so leaves sample a static shadow; a future world-camera DepthPrepass must call the same offsets | | Ambient occlusion | soft contact shadows under props, in wall-floor corners and around feet; depth-only SSAO at half resolution (depth-reconstructed normals, 6/12 spiral taps, 4x4 depth-aware blur, bilateral upsample) multiplied into the opaque scene after the opaque pass and before the water copy, so water, particles and UI are untouched; fades out 45-90 yd (Low) / 60-120 yd (High), spares sky and bright pixels; on alpha-tested foliage (a dense window of two-sided depth spikes) the blur widens its depth tolerance and the AO drops 40 %, which removes the leaf grain while planes, silhouettes and contact creases keep full AO (`benilla-app/src/ssao.rs`, `benilla-app/src/shaders/ssao.wgsl`) | Advanced Graphics → Ambient Occlusion (Off/Low/High; default Off, High preset 2), cvar `ambientOcclusion`, env `WOW_AO=0\|1\|2`, `WOW_AO_DEBUG=1..5` diagnostic views (factor, protection, distance, raw occlusion, foliage mask), `WOW_AO_GAIN`/`_RADIUS`/`_BIAS`/`_STRENGTH` tuning | | Lamp fog | MONKEY (lampfog): the fog fullscreen pass selects the nearest resolved exterior and interior point-table sources and analytically integrates their softened inverse-square light over the camera-to-scene-depth ray. A near-field haze floor makes lamps visible inside the ordinary volumetric haze's 10 yd clear zone; it is zero by day, stronger in rain/fog, and deliberately unshadowed. The table colour already includes intensity, flame flicker and `fireLightGain`. Halo output uses destination headroom and stays below 1.0, so a later bloom pass cannot blow it out | Advanced Graphics → Lamp Fog, cvar `lampFog` 0 Off (default, unchanged) / 1 Low (16 lamps) / 2 High (32 lamps; Graphics preset High = 2), env `WOW_LAMPFOG=0\|1\|2`; `benilla-app/src/volumetric_fog.rs` | | City window daylight | MONKEY (fix-daylight/integration): a district WMO's oversized window batch (e.g. the Cathedral of Light's stained-glass walls) is split into window-sized daylight apertures, so the nave takes daylight pane by pane. Applies to WMOs loaded after a change | Advanced Graphics → City Window Daylight, cvar `daylightWindowSplit` 0/1 (default 1, the merged behaviour); `benilla-world/src/lighting/daylight.rs` (`set_window_split`), bridge `benilla-app/src/dynamic_interior.rs` | -| Zone skyboxes | the painted sky a zone's `LightParams` names (clear-weather slot, storm slot lerped by the storm blend) for the living, fading in by the Light sphere falloff with the modern client's crossfade (ghost > WMO > zone); extended `LightSkybox.dbc` flags `0x1` day cycle, `0x2` keep sun/moon/stars/clouds, `0x4` fog-colour horizon cone, plus a celestial second model; every skybox now plays its bones, texture transforms, colour and alpha tracks (`benilla-world/src/skybox.rs`, `skybox_anim.rs`, `benilla-formats/src/light/skybox.rs`) | Advanced Graphics → Zone Skyboxes, cvar `zoneSkyboxes` (0 Off default, 1 On; High = 1), env `WOW_ZONE_SKYBOXES=0\|1` | +| Zone skyboxes | the painted sky a zone's `LightParams` names (clear-weather slot, storm slot lerped by the storm blend) for the living, fading in by the Light sphere falloff with the modern client's crossfade (ghost > WMO > zone); extended `LightSkybox.dbc` flags `0x1` day cycle, `0x2` keep sun/moon/stars/clouds, `0x4` fog-colour horizon cone, `0x8` forces the screen-space sun shafts through a low or just-set sun (the sky's weight lifts their daylight fade; `post/sun_shafts.rs`), `0x10` fades the Modern fog's sun-colour lobe out by the sky's weight (`lighting/fog_model.rs`), plus a celestial second model; every skybox now plays its bones, texture transforms, colour and alpha tracks (`benilla-world/src/skybox.rs`, `skybox_anim.rs`, `benilla-formats/src/light/skybox.rs`) | Advanced Graphics → Zone Skyboxes, cvar `zoneSkyboxes` (0 Off default, 1 On; High = 1), env `WOW_ZONE_SKYBOXES=0\|1` | | Night and interior level | global dimming of the night sky term and of interior ambient | `nightGain`, `interiorGain`, `interiorBakeFloor` | `zoneSkyboxes = 0` disables only the added living-player zone slot. WMO and ghost skyboxes are @@ -43,6 +43,8 @@ the reference 1.12 slots, so their fidelity fixes remain active in Classic: auth alpha tracks affect the layer, non-white M2 colours reach the mesh, deterministic captures pose at their pinned time, and the batch-order base reserves the lower celestial band and final fog cone. +Asset-free local builder and bilingual install guide: [`Optional/sky-and-grading/`](Optional/sky-and-grading/README.md). + Players reach all of it from **Options -> Advanced Graphics** (a Graphics Preset over everything, a Render Distance slider, a Lighting Quality preset Off / Low / Medium / High / Ultra plus the individual rows; Classic / Off is the original client look). The dev build has a panel for all of it: **Ctrl+Shift+D → Lighting & shadows**, with Dim / Default / Bright presets. diff --git a/Optional/sky-and-grading/README.md b/Optional/sky-and-grading/README.md new file mode 100644 index 00000000..c92d9b88 --- /dev/null +++ b/Optional/sky-and-grading/README.md @@ -0,0 +1,179 @@ +# Optional skyboxes and colour grading + +This asset-free kit builds an optional MPQ on your machine. It adds zone +skyboxes for Burning Steppes, Blasted Lands, and Mount Hyjal; project-authored +per-zone colour grading for Elwynn Forest, Duskwood, Westfall, and Redridge; +and, with `--wf-colours`, a moderated import of night colours from your own WoW +Forever installation. + +The repository contains only scripts and benilla's own grading LUTs/table. The +Blizzard models, textures, and lighting tables are read from installations you +own and appear only in your local build directory. + +## Requirements + +- Windows and Python 3.8 or newer. +- A World of Warcraft 1.12.1 client and its `Data` directory. +- For skyboxes, a local WoW Forever `_classic_beta_` or current retail install. + Point the command at the shared World of Warcraft root (the directory with + `.build.info` and `Data`), or at `_classic_beta_` itself. +- Git and the external `wowdev/pywowlib` checkout pinned below. It is not + redistributed here. +- 64-bit StormLib 9.40 (`StormLib.dll`) from the official + [StormLib releases](https://github.com/ladislav-zezula/StormLib/releases/tag/v9.40). + Download `stormlib_dll.zip` and use its x64 DLL. StormLib is not redistributed. +- Internet access on the first run for the public wow-listfile and public TACT + key list, unless `--listfile` and `--tact-keys` name local copies. + +## Install, one command per step + +Run these in PowerShell. Change the example paths for your machine. + +1. Clone the exact pywowlib revision used by the converter. + + ```powershell + git clone https://github.com/wowdev/pywowlib.git C:\Tools\pywowlib + ``` + +2. Pin it to the tested commit. + + ```powershell + git -C C:\Tools\pywowlib checkout 55276dc5c2195da7fe136638a2a59716622f8c65 + ``` + +3. Install the Python modules needed by the M2 reader. pywowlib remains an + external checkout; this installs dependencies only. + + ```powershell + py -3 -m pip install bidict==0.23.1 multimethod numpy + ``` + +4. Download StormLib 9.40's `stormlib_dll.zip` from the release linked above, + extract the 64-bit `StormLib.dll`, and place it at + `C:\Tools\StormLib\StormLib.dll`. + +5. Close the game and build the candidate. The tool never writes to either + input installation. + + ```powershell + py -3 .\tools\build_sky_patch.py --wow-forever "C:\Games\World of Warcraft" --client-data "C:\Games\WoW-1.12.1\Data" --pywowlib "C:\Tools\pywowlib" --stormlib "C:\Tools\StormLib\StormLib.dll" --out "C:\Temp\benilla-sky" + ``` + +6. Verify an isolated Data copy before installation. `C:\Temp\WoW-test\Data` + should be a copy or hard-link mirror in which only `patch-Z.mpq` is an + ordinary copy; replace that copy with the newly built archive first. + + ```powershell + py -3 .\tools\verify_client_chain.py "C:\Temp\WoW-test\Data" --stage "C:\Temp\benilla-sky\stage" --stormlib "C:\Tools\StormLib\StormLib.dll" + ``` + +7. Back up the live target archive if one exists. + + ```powershell + Copy-Item "C:\Games\WoW-1.12.1\Data\patch-Z.mpq" "C:\Games\WoW-1.12.1\Data\patch-Z.before-benilla.mpq.bak" + ``` + +8. Copy the verified candidate into the client. + + ```powershell + Copy-Item -Force "C:\Temp\benilla-sky\patch-Z.MPQ" "C:\Games\WoW-1.12.1\Data\patch-Z.mpq" + ``` + +If no `patch-Z.mpq` existed, skip step 7. The build report's `archive_base` +field records whether the builder preserved an existing patch-Z. + +### Variants + +Build only the project-owned grading table and LUTs; this needs no modern WoW +install or pywowlib: + +```powershell +py -3 .\tools\build_sky_patch.py --grading-only --client-data "C:\Games\WoW-1.12.1\Data" --stormlib "C:\Tools\StormLib\StormLib.dll" --out "C:\Temp\benilla-grading" +``` + +Add moderated WoW Forever night colours. LightData is extracted directly from +your installation; no derived LightData is shipped here: + +```powershell +py -3 .\tools\build_sky_patch.py --wow-forever "C:\Games\World of Warcraft" --client-data "C:\Games\WoW-1.12.1\Data" --pywowlib "C:\Tools\pywowlib" --stormlib "C:\Tools\StormLib\StormLib.dll" --wf-colours --out "C:\Temp\benilla-sky-wf" +``` + +## Enable in game + +Open **Options → Advanced Graphics** and enable **Zone Skyboxes** and +**Colour Grading**. The equivalent console commands are: + +```text +/console zoneSkyboxes 1 +/console colorGrading 1 +``` + +Use `0` instead of `1` to disable either effect without removing the MPQ. + +## MPQ load order: read before copying + +All five mutually dependent lighting DBCs must win from the same, highest +letter patch that carries any of them. The default is `patch-Z.mpq`. If the +client already has patch-Z, the builder copies that archive and overlays the +new files so unrelated members survive; install the resulting candidate as a +replacement, not as a second archive. + +Do not blindly rename the output. A later-mounted archive that also contains +`Light.dbc`, `LightParams.dbc`, `LightIntBand.dbc`, `LightFloatBand.dbc`, or +`LightSkybox.dbc` can split the table set and break the sky. Some custom clients +use punctuation after ordinary letters; in that case merge this stage into the +actual last patch or remove the conflicting DBC copies, then run +`verify_client_chain.py --require-winner ANY`. A conventional patch-Z setup +should keep the default strict check. + +The extended four-field `LightSkybox.dbc` is for benilla. Do not use this patch +with an unmodified stock 1.12 executable that expects the two-field table. + +## Uninstall + +Close the game. If the build preserved an existing patch-Z, restore the backup: + +```powershell +Move-Item -Force "C:\Games\WoW-1.12.1\Data\patch-Z.before-benilla.mpq.bak" "C:\Games\WoW-1.12.1\Data\patch-Z.mpq" +``` + +If there was no prior patch-Z and this kit created it, remove only that exact +archive: + +```powershell +Remove-Item -LiteralPath "C:\Games\WoW-1.12.1\Data\patch-Z.mpq" +``` + +Deleting a pre-existing patch-Z would remove unrelated custom content, so use +`build-report.json` and your backup to distinguish these cases. + +## Troubleshooting + +- **pywowlib revision mismatch:** run the checkout command from step 2. The + builder deliberately rejects another revision so M2 layout behaviour cannot + drift silently. +- **StormLib not found:** pass `--stormlib` or set `STORMLIB_PATH` to the x64 + DLL. A 32-bit DLL cannot load into 64-bit Python. +- **No CASC build/config:** point `--wow-forever` at the World of Warcraft root + with `.build.info` and `Data`, or at its `_classic_beta_` child. +- **Missing/encrypted CASC block:** update the public TACT key file or rerun + without a stale `--tact-keys` copy. The tool uses public keys only. +- **Unexpected source schema/clone start:** another addon has already changed + the winning lighting DBCs. Remove that conflict or merge deliberately; the + tool refuses to guess. +- **Sky does not appear:** enable `zoneSkyboxes`, fully restart the client, and + run `verify_client_chain.py` against the actual Data directory. +- **Bands fail:** run `check_bands.py --stormlib `. Correct + clone keys are `(P-1)*18+b+1` and `(P-1)*6+b+1`. +- **Output already exists:** choose a fresh `--out`, or pass `--force` only when + replacing the candidate archive in that output directory is intended. + +## Credits and rights + +WoW Forever and retail skybox art is © Blizzard Entertainment and is extracted +locally by the user from their own installation. No Blizzard art or client DBC +is distributed here. + +The colour-grading technique is based on WarcraftXL's `wxl-retail-grading` work +by iThorgrim, used with attribution. The grading LUTs and `MonkeyZoneGrade.dbc` +in `data/` are original benilla project data, licensed MIT OR Apache-2.0. diff --git a/Optional/sky-and-grading/README.ru.md b/Optional/sky-and-grading/README.ru.md new file mode 100644 index 00000000..6b8a3d3d --- /dev/null +++ b/Optional/sky-and-grading/README.ru.md @@ -0,0 +1,167 @@ +# Дополнительные небеса и цветокоррекция + +Этот комплект без игровых ресурсов собирает MPQ локально. Он добавляет небеса +зон для Burning Steppes, Blasted Lands и Mount Hyjal, собственную +цветокоррекцию benilla для Elwynn Forest, Duskwood, Westfall и Redridge, а с +ключом `--wf-colours` — умеренно смешанные ночные цвета из вашей установки WoW +Forever. + +В репозитории находятся только скрипты и созданные проектом LUT/таблица. Модели, +текстуры и исходные DBC Blizzard читаются из ваших установок и записываются +только во внешний каталог сборки. + +## Требования + +- Windows и Python 3.8 или новее. +- Клиент World of Warcraft 1.12.1 и его каталог `Data`. +- Для небес — локальная установка WoW Forever `_classic_beta_` либо retail. +- Git и внешний `wowdev/pywowlib` точной ревизии ниже. +- 64-битный StormLib 9.40 (`StormLib.dll`) из официальных + [релизов StormLib](https://github.com/ladislav-zezula/StormLib/releases/tag/v9.40). + Скачайте `stormlib_dll.zip` и используйте x64 DLL. +- Интернет при первом запуске для публичного listfile и списка публичных TACT + ключей, либо локальные файлы через `--listfile` и `--tact-keys`. + +## Установка — одна команда на шаг + +Команды выполняются в PowerShell; замените пути на свои. + +1. Клонируйте pywowlib. + + ```powershell + git clone https://github.com/wowdev/pywowlib.git C:\Tools\pywowlib + ``` + +2. Выберите проверенную ревизию. + + ```powershell + git -C C:\Tools\pywowlib checkout 55276dc5c2195da7fe136638a2a59716622f8c65 + ``` + +3. Установите зависимости Python. Сам pywowlib остаётся внешним checkout. + + ```powershell + py -3 -m pip install bidict==0.23.1 multimethod numpy + ``` + +4. Скачайте `stormlib_dll.zip` версии 9.40 по ссылке выше и распакуйте x64 + `StormLib.dll` в `C:\Tools\StormLib\StormLib.dll`. + +5. Закройте игру и соберите кандидат. Скрипт не пишет в исходные установки. + + ```powershell + py -3 .\tools\build_sky_patch.py --wow-forever "C:\Games\World of Warcraft" --client-data "C:\Games\WoW-1.12.1\Data" --pywowlib "C:\Tools\pywowlib" --stormlib "C:\Tools\StormLib\StormLib.dll" --out "C:\Temp\benilla-sky" + ``` + +6. Сначала проверьте изолированную копию `Data`. Все файлы можно сделать + жёсткими ссылками, но `patch-Z.mpq` должен быть обычной копией; замените его + собранным кандидатом и выполните: + + ```powershell + py -3 .\tools\verify_client_chain.py "C:\Temp\WoW-test\Data" --stage "C:\Temp\benilla-sky\stage" --stormlib "C:\Tools\StormLib\StormLib.dll" + ``` + +7. Если живой `patch-Z.mpq` уже есть, создайте резервную копию. + + ```powershell + Copy-Item "C:\Games\WoW-1.12.1\Data\patch-Z.mpq" "C:\Games\WoW-1.12.1\Data\patch-Z.before-benilla.mpq.bak" + ``` + +8. Установите проверенный кандидат. + + ```powershell + Copy-Item -Force "C:\Temp\benilla-sky\patch-Z.MPQ" "C:\Games\WoW-1.12.1\Data\patch-Z.mpq" + ``` + +Если patch-Z раньше не существовал, пропустите шаг 7. Поле `archive_base` в +`build-report.json` показывает, был ли старый архив сохранён внутри кандидата. + +### Варианты + +Только собственная цветокоррекция, без современной установки и pywowlib: + +```powershell +py -3 .\tools\build_sky_patch.py --grading-only --client-data "C:\Games\WoW-1.12.1\Data" --stormlib "C:\Tools\StormLib\StormLib.dll" --out "C:\Temp\benilla-grading" +``` + +Небеса и умеренные ночные цвета WoW Forever, извлечённые из вашей LightData: + +```powershell +py -3 .\tools\build_sky_patch.py --wow-forever "C:\Games\World of Warcraft" --client-data "C:\Games\WoW-1.12.1\Data" --pywowlib "C:\Tools\pywowlib" --stormlib "C:\Tools\StormLib\StormLib.dll" --wf-colours --out "C:\Temp\benilla-sky-wf" +``` + +## Включение в игре + +Включите **Zone Skyboxes** и **Colour Grading** в **Options → Advanced +Graphics**. Те же настройки через консоль: + +```text +/console zoneSkyboxes 1 +/console colorGrading 1 +``` + +Значение `0` отключает эффект без удаления MPQ. + +## Порядок загрузки MPQ + +Все пять связанных DBC освещения должны загружаться из одного самого старшего +буквенного патча, который содержит любой из них. По умолчанию это +`patch-Z.mpq`. Если он уже существует, сборщик копирует его и накладывает новые +файлы, сохраняя постороннее содержимое; результат устанавливается вместо +старого архива. + +Не переименовывайте результат вслепую. Более поздний архив с `Light.dbc`, +`LightParams.dbc`, `LightIntBand.dbc`, `LightFloatBand.dbc` или +`LightSkybox.dbc` разделит согласованный набор. Некоторые кастомные клиенты +используют символы, сортирующиеся после букв. Тогда объедините `stage` с +реально последним патчем либо удалите конфликтующие DBC и проверьте цепочку с +`--require-winner ANY`. Для обычного patch-Z оставьте строгую проверку. + +Расширенный четырёхполевой `LightSkybox.dbc` предназначен для benilla. Не +используйте его с неизменённым оригинальным exe 1.12, ожидающим два поля. + +## Удаление + +Закройте игру. Если patch-Z существовал ранее, восстановите резервную копию: + +```powershell +Move-Item -Force "C:\Games\WoW-1.12.1\Data\patch-Z.before-benilla.mpq.bak" "C:\Games\WoW-1.12.1\Data\patch-Z.mpq" +``` + +Если архив был создан этим комплектом с нуля, удалите только его: + +```powershell +Remove-Item -LiteralPath "C:\Games\WoW-1.12.1\Data\patch-Z.mpq" +``` + +Удаление старого patch-Z уничтожит другое пользовательское содержимое. +Проверяйте `archive_base` и сохраняйте резервную копию. + +## Решение проблем + +- **Неверная ревизия pywowlib:** повторите шаг 2; сборщик намеренно не принимает + другую ревизию. +- **StormLib не найден:** укажите `--stormlib` либо `STORMLIB_PATH`; DLL должна + быть 64-битной для 64-битного Python. +- **CASC build/config не найден:** укажите корень World of Warcraft с + `.build.info` и `Data` либо его дочерний `_classic_beta_`. +- **Зашифрованный/отсутствующий блок:** обновите публичный список TACT ключей. + Приватные ключи не используются. +- **Неожиданная схема или номер клона:** другой аддон уже изменил выигрывающие + DBC. Уберите конфликт или выполните осознанное объединение. +- **Небо не видно:** включите `zoneSkyboxes`, полностью перезапустите клиент и + запустите `verify_client_chain.py` на его настоящем `Data`. +- **Ошибка band rows:** запустите `check_bands.py --stormlib `. + Правильные ключи: `(P-1)*18+b+1` и `(P-1)*6+b+1`. +- **Выходной архив уже существует:** выберите новый `--out` либо используйте + `--force` только для намеренной замены кандидата. + +## Авторы и права + +Графика небес WoW Forever/retail принадлежит © Blizzard Entertainment и +извлекается пользователем локально из собственной установки. Репозиторий не +распространяет графику Blizzard или DBC клиента. + +Метод цветокоррекции основан на `wxl-retail-grading` проекта WarcraftXL автора +iThorgrim и используется с указанием авторства. LUT и `MonkeyZoneGrade.dbc` в +`data/` созданы проектом benilla и лицензированы MIT OR Apache-2.0. diff --git a/Optional/sky-and-grading/data/DBFilesClient/MonkeyZoneGrade.dbc b/Optional/sky-and-grading/data/DBFilesClient/MonkeyZoneGrade.dbc new file mode 100644 index 00000000..ff0badbc Binary files /dev/null and b/Optional/sky-and-grading/data/DBFilesClient/MonkeyZoneGrade.dbc differ diff --git a/Optional/sky-and-grading/data/LICENSE.md b/Optional/sky-and-grading/data/LICENSE.md new file mode 100644 index 00000000..54c1e507 --- /dev/null +++ b/Optional/sky-and-grading/data/LICENSE.md @@ -0,0 +1,13 @@ +# License + +The files in this directory are generated by `tools/build_grading_data.py` and +are original benilla project data. They are licensed under either of: + +- MIT (`../../../LICENSE-MIT`) +- Apache License 2.0 (`../../../LICENSE-APACHE`) + +at your option. + +No Blizzard asset or client-derived DBC is stored here. Sky models, textures, +and lighting DBC merges are produced only from the user's own installations at +build time and are written outside the repository. diff --git a/Optional/sky-and-grading/data/World/LUTs/DuskwoodCold.blp b/Optional/sky-and-grading/data/World/LUTs/DuskwoodCold.blp new file mode 100644 index 00000000..a8c6794e Binary files /dev/null and b/Optional/sky-and-grading/data/World/LUTs/DuskwoodCold.blp differ diff --git a/Optional/sky-and-grading/data/World/LUTs/ElwynnWarmGreen.blp b/Optional/sky-and-grading/data/World/LUTs/ElwynnWarmGreen.blp new file mode 100644 index 00000000..7497b93e Binary files /dev/null and b/Optional/sky-and-grading/data/World/LUTs/ElwynnWarmGreen.blp differ diff --git a/Optional/sky-and-grading/data/World/LUTs/Identity.blp b/Optional/sky-and-grading/data/World/LUTs/Identity.blp new file mode 100644 index 00000000..9692cd08 Binary files /dev/null and b/Optional/sky-and-grading/data/World/LUTs/Identity.blp differ diff --git a/Optional/sky-and-grading/data/World/LUTs/RedridgeWarmRed.blp b/Optional/sky-and-grading/data/World/LUTs/RedridgeWarmRed.blp new file mode 100644 index 00000000..69d15b8a Binary files /dev/null and b/Optional/sky-and-grading/data/World/LUTs/RedridgeWarmRed.blp differ diff --git a/Optional/sky-and-grading/data/World/LUTs/WestfallGolden.blp b/Optional/sky-and-grading/data/World/LUTs/WestfallGolden.blp new file mode 100644 index 00000000..297a4bb6 Binary files /dev/null and b/Optional/sky-and-grading/data/World/LUTs/WestfallGolden.blp differ diff --git a/Optional/sky-and-grading/tools/build_grading_data.py b/Optional/sky-and-grading/tools/build_grading_data.py new file mode 100644 index 00000000..596b7c06 --- /dev/null +++ b/Optional/sky-and-grading/tools/build_grading_data.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 +"""Build MONKEY post-lane BLP2 LUT strips and MonkeyZoneGrade.dbc. + +Outputs ordinary loose patch-tree files below this directory. Copy the resulting +DBFilesClient/ and World/ trees into an isolated Data directory for capture runs. +""" + +from pathlib import Path +import math +import struct + +ROOT = Path(__file__).resolve().parent.parent / "data" +LUT_DIR = ROOT / "World" / "LUTs" +DBC_DIR = ROOT / "DBFilesClient" + + +def clamp(value): + return max(0.0, min(1.0, value)) + + +def identity(c): + return c + + +def elwynn(c): + r, g, b = c + return (clamp(r * 1.025 + 0.010 * g), clamp(g * 1.035 + 0.012 * r), clamp(b * 0.975)) + + +def duskwood(c): + r, g, b = c + y = r * 0.299 + g * 0.587 + b * 0.114 + r, g, b = (y + (r - y) * 0.78, y + (g - y) * 0.78, y + (b - y) * 0.78) + return (clamp(r * 0.94), clamp(g * 0.97), clamp(b * 1.045 + 0.008)) + + +def westfall(c): + r, g, b = c + return (clamp(r * 1.045 + 0.008 * g), clamp(g * 1.018), clamp(b * 0.94)) + + +def redridge(c): + r, g, b = c + return (clamp(r * 1.055 + 0.008), clamp(g * 0.982), clamp(b * 0.955)) + + +def write_blp(path, transform): + width, height = 1024, 32 + header_bytes = 148 + palette_bytes = 256 * 4 + pixel_offset = header_bytes + palette_bytes + pixel_size = width * height * 4 + header = bytearray(header_bytes) + header[0:4] = b"BLP2" + struct.pack_into(" str: + digest = hashlib.sha256() + with path.open("rb") as stream: + while True: + chunk = stream.read(1024 * 1024) + if not chunk: + break + digest.update(chunk) + return digest.hexdigest() + + +def fetch(url: str, target: Path) -> Path: + if target.is_file() and target.stat().st_size: + return target + target.parent.mkdir(parents=True, exist_ok=True) + partial = target.with_suffix(target.suffix + ".part") + with urllib.request.urlopen(url, timeout=120) as response, partial.open("wb") as output: + shutil.copyfileobj(response, output, 1024 * 1024) + partial.replace(target) + return target + + +def casc_root(install: Path) -> Path: + install = Path(install).resolve() + if (install / "Data" / "config").is_dir(): + return install + if (install.parent / "Data" / "config").is_dir(): + return install.parent + raise FileNotFoundError("no shared CASC Data/config below %s or its parent" % install) + + +def select_build_key(install: Path, preferred: str | None = None) -> tuple[str, str]: + info = install / ".build.info" + if not info.is_file(): + raise FileNotFoundError(info) + with info.open("r", encoding="utf-8-sig", newline="") as stream: + reader = csv.reader(stream, delimiter="|") + raw_headers = next(reader) + headers = [header.split("!", 1)[0] for header in raw_headers] + rows = [dict(zip(headers, row)) for row in reader] + active = [row for row in rows if row.get("Active") == "1"] or rows + products = [preferred] if preferred else [] + products.extend(name for name in ("wow_classic_beta", "wow") if name != preferred) + for wanted in products: + for row in active: + if row.get("Product") == wanted: + key = row.get("Build Key", "").lower() + if len(key) == 32: + return key, wanted + raise RuntimeError("no active WoW Forever/classic-beta or retail build in .build.info") + + +def reset_owned_directory(path: Path, parent: Path, force: bool) -> None: + path = path.resolve() + parent = parent.resolve() + if path.parent != parent: + raise RuntimeError("refusing to reset unexpected path %s" % path) + if path.exists(): + if not force: + raise FileExistsError("build directory exists; pass --force: %s" % path) + shutil.rmtree(path) + path.mkdir(parents=True) + + +def is_within(path: Path, parent: Path) -> bool: + try: + path.relative_to(parent) + return True + except ValueError: + return False + + +def copy_own_grading(stage: Path) -> None: + for source in sorted(DATA.rglob("*")): + if not source.is_file() or source.name == "LICENSE.md": + continue + relative = source.relative_to(DATA) + target = stage / relative + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(source, target) + + +def archive_base(client_data: Path, letter: str) -> Path | None: + wanted = ("patch-%s.mpq" % letter).casefold() + matches = [ + path for path in client_data.iterdir() + if path.is_file() and path.name.casefold() == wanted + ] + if len(matches) > 1: + raise RuntimeError("multiple case-insensitive matches for %s" % wanted) + return matches[0] if matches else None + + +def stage_pairs(stage: Path) -> list[tuple[Path, str]]: + return [ + (path, str(path.relative_to(stage)).replace("/", "\\")) + for path in sorted(stage.rglob("*")) + if path.is_file() + ] + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--wow-forever", type=Path, help="WoW Forever or retail install root") + parser.add_argument("--client-data", type=Path, required=True, help="1.12.1 Data directory") + parser.add_argument("--out", type=Path, required=True, help="output directory") + parser.add_argument("--grading-only", action="store_true", help="pack only project-owned grading data") + parser.add_argument("--wf-colours", action="store_true", help="moderate locally extracted WoW Forever night colours") + parser.add_argument("--pywowlib", type=Path, help="pinned external pywowlib checkout") + parser.add_argument("--stormlib", type=Path, help="path to StormLib.dll") + parser.add_argument("--listfile", type=Path, help="existing community-listfile.csv") + parser.add_argument("--tact-keys", type=Path, help="existing public WoW TACT key list") + parser.add_argument("--patch-letter", default="Z", help="single patch character (default: Z)") + parser.add_argument("--force", action="store_true", help="replace an existing output archive") + args = parser.parse_args() + if len(args.patch_letter) != 1 or not args.patch_letter.isalpha(): + parser.error("--patch-letter must be one ASCII letter") + if not args.client_data.is_dir(): + parser.error("--client-data is not a directory") + if args.wf_colours and args.grading_only: + parser.error("--wf-colours cannot be combined with --grading-only") + if not args.grading_only and (not args.wow_forever or not args.pywowlib): + parser.error("sky builds require --wow-forever and --pywowlib") + return args + + +def main() -> int: + if sys.version_info < (3, 8): + raise RuntimeError("Python 3.8 or newer is required") + args = parse_args() + output = args.out.resolve() + client_data = args.client_data.resolve() + if is_within(output, client_data): + raise ValueError("--out must not be inside the 1.12.1 Data directory") + if args.wow_forever: + modern_input = casc_root(args.wow_forever) + if is_within(output, modern_input): + raise ValueError("--out must not be inside the modern WoW installation") + output.mkdir(parents=True, exist_ok=True) + archive = output / ("patch-%s.MPQ" % args.patch_letter.upper()) + if archive.exists() and not args.force: + raise FileExistsError("output archive exists; pass --force: %s" % archive) + stage, work = output / "stage", output / "work" + reset_owned_directory(stage, output, args.force) + reset_owned_directory(work, output, args.force) + copy_own_grading(stage) + + storm = Storm(find_stormlib(args.stormlib)) + report: dict[str, object] = {"grading_only": args.grading_only} + if not args.grading_only: + cache = output / "cache" + listfile_path = args.listfile or fetch(LISTFILE_URL, cache / "community-listfile.csv") + tact_keys = args.tact_keys or fetch(TACT_KEYS_URL, cache / "WoW.txt") + install = modern_input + requested_name = args.wow_forever.resolve().name.casefold() + preferred = { + "_classic_beta_": "wow_classic_beta", + "_retail_": "wow", + }.get(requested_name) + build_key, product = select_build_key(install, preferred) + reader = CascReader(str(install), build_key, str(listfile_path), str(tact_keys)) + reader.load_encoding() + reader.load_root() + modules = load_pywowlib(args.pywowlib) + source_root = work / "source" + extract_sources(reader, read_listfile(listfile_path), source_root, modules) + conversions = convert_all(source_root, stage, listfile_path, modules) + + winners, winner_names = winning_payloads(storm, args.client_data, REQUIRED) + wf_changed = 0 + if args.wf_colours: + light_data = WDC5(reader.open_fdid(1375580)) + winners["LightIntBand.dbc"], wf_changed = apply_wf_colours( + winners["LightIntBand.dbc"], light_data + ) + merged, dbc_report = build_merged(winners) + dbc_dir = stage / "DBFilesClient" + dbc_dir.mkdir(parents=True, exist_ok=True) + for name, payload in merged.items(): + (dbc_dir / name).write_bytes(payload) + check_bands(merged["LightIntBand.dbc"], merged["LightFloatBand.dbc"]) + report.update( + { + "modern_product": product, + "build_key": build_key, + "dbc_winners": winner_names, + "dbc_merge": dbc_report, + "wf_colour_rows_changed": wf_changed, + "conversions": conversions, + } + ) + + if archive.exists(): + archive.unlink() + base = archive_base(args.client_data, args.patch_letter) + storm.pack(archive, stage_pairs(stage), base_archive=base) + + files = { + str(path.relative_to(stage)).replace("/", "\\"): { + "bytes": path.stat().st_size, + "sha256": sha256(path), + } + for path in sorted(stage.rglob("*")) + if path.is_file() + } + report.update( + { + "archive": str(archive), + "archive_base": str(base) if base else None, + "archive_bytes": archive.stat().st_size, + "archive_sha256": sha256(archive), + "stage_files": files, + } + ) + (output / "build-report.json").write_text( + json.dumps(report, indent=2) + "\n", encoding="utf-8" + ) + print( + "built %s with %d staged files%s" + % (archive, len(files), " (WoW Forever colours)" if args.wf_colours else "") + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/Optional/sky-and-grading/tools/casc.py b/Optional/sky-and-grading/tools/casc.py new file mode 100644 index 00000000..c2f2dd47 --- /dev/null +++ b/Optional/sky-and-grading/tools/casc.py @@ -0,0 +1,413 @@ +"""Small, read-only local CASC/TACT reader for the installed WoW builds. + +The implementation deliberately indexes only the latest local v7 .idx file per +bucket and opens payloads on demand. It never writes to the CASC installation. +""" +from __future__ import annotations + +import hashlib +import pathlib +import struct +import zlib +from typing import Dict, Iterable, List, Optional, Tuple + + +class CascError(Exception): + pass + + +class EncryptedBlock(CascError): + def __init__(self, key_name: bytes): + super().__init__("encrypted BLTE block; missing TACT key %s" % key_name.hex()) + self.key_name = key_name + + +def _u24be(data: bytes) -> int: + return int.from_bytes(data, "big") + + +def _rotl32(x: int, n: int) -> int: + return ((x << n) | (x >> (32 - n))) & 0xFFFFFFFF + + +def _salsa20_block(key: bytes, nonce8: bytes, counter: int) -> bytes: + # Salsa20/20 with the 64-bit nonce/64-bit block-counter construction. + if len(key) == 16: + constants = b"expand 16-byte k" + key = key + key + elif len(key) == 32: + constants = b"expand 32-byte k" + else: + raise CascError("Salsa20 key must contain 16 or 32 bytes") + c = struct.unpack("<4I", constants) + k = struct.unpack("<8I", key) + n0, n1 = struct.unpack("<2I", nonce8) + state = [c[0], k[0], k[1], k[2], k[3], c[1], n0, n1, + counter & 0xFFFFFFFF, (counter >> 32) & 0xFFFFFFFF, + c[2], k[4], k[5], k[6], k[7], c[3]] + x = state[:] + for _ in range(10): + # Column round then row round (Salsa20 double-round). + for a, b, c_, d in ((0, 4, 8, 12), (5, 9, 13, 1), + (10, 14, 2, 6), (15, 3, 7, 11), + (0, 1, 2, 3), (5, 6, 7, 4), + (10, 11, 8, 9), (15, 12, 13, 14)): + x[b] ^= _rotl32((x[a] + x[d]) & 0xFFFFFFFF, 7) + x[c_] ^= _rotl32((x[b] + x[a]) & 0xFFFFFFFF, 9) + x[d] ^= _rotl32((x[c_] + x[b]) & 0xFFFFFFFF, 13) + x[a] ^= _rotl32((x[d] + x[c_]) & 0xFFFFFFFF, 18) + return struct.pack("<16I", *((x[i] + state[i]) & 0xFFFFFFFF for i in range(16))) + + +def salsa20_xor(data: bytes, key: bytes, nonce8: bytes) -> bytes: + out = bytearray(len(data)) + for pos in range(0, len(data), 64): + stream = _salsa20_block(key, nonce8, pos // 64) + part = data[pos:pos + 64] + for j, value in enumerate(part): + out[pos + j] = value ^ stream[j] + return bytes(out) + + +def load_tact_keys(path: pathlib.Path) -> Dict[bytes, bytes]: + keys: Dict[bytes, bytes] = {} + if not path.exists(): + return keys + for raw in path.read_text(encoding="utf-8", errors="replace").splitlines(): + line = raw.strip() + if not line or line.startswith("#"): + continue + bits = line.replace(";", " ").split() + if len(bits) < 2: + continue + try: + name, key = bytes.fromhex(bits[0]), bytes.fromhex(bits[1]) + except ValueError: + continue + if len(name) == 8 and len(key) in (16, 32): + keys[name] = key + return keys + + +class CascReader: + """Read a product from one local modern Battle.net CASC installation.""" + + def __init__(self, install: str, build_key: str, + listfile: Optional[str] = None, + tact_keys: Optional[str] = None): + self.install = pathlib.Path(install) + self.data_dir = self.install / "Data" / "data" + self.config_dir = self.install / "Data" / "config" + self.build_key = build_key.lower() + self.config = self._read_config(self.build_key) + self.keys = load_tact_keys(pathlib.Path(tact_keys)) if tact_keys else {} + self.by_ekey9: Dict[bytes, Tuple[int, int, int]] = {} + self._load_local_indices() + self.names_by_fdid: Dict[int, str] = {} + self.fdids_by_name: Dict[str, int] = {} + if listfile: + self._load_listfile(pathlib.Path(listfile)) + self.encoding_map: Dict[bytes, List[bytes]] = {} + self.root_map: Dict[int, List[bytes]] = {} + self.encoding_header = {} + self.root_header = {} + + def _config_path(self, key: str) -> pathlib.Path: + return self.config_dir / key[:2] / key[2:4] / key + + def _read_config(self, key: str) -> Dict[str, List[str]]: + result: Dict[str, List[str]] = {} + for raw in self._config_path(key).read_text(encoding="ascii").splitlines(): + line = raw.strip() + if not line or line.startswith("#") or "=" not in line: + continue + name, value = line.split("=", 1) + result[name.strip()] = value.strip().split() + return result + + def _latest_indices(self) -> Iterable[pathlib.Path]: + groups: Dict[str, pathlib.Path] = {} + for path in self.data_dir.glob("*.idx"): + bucket = path.name[:2].lower() + old = groups.get(bucket) + # Generation is encoded in the remaining filename hex. The newest + # index in each bucket is sufficient because local indices are full. + if old is None or int(path.stem[2:], 16) > int(old.stem[2:], 16): + groups[bucket] = path + if len(groups) != 16: + raise CascError("expected 16 local index buckets, found %d" % len(groups)) + return (groups["%02x" % i] for i in range(16)) + + def _load_local_indices(self) -> None: + for expected_bucket, path in enumerate(self._latest_indices()): + data = path.read_bytes() + if len(data) < 40: + raise CascError("short local index " + str(path)) + header_len, = struct.unpack_from(" len(data): + raise CascError("truncated index entries in " + str(path)) + while pos < end: + key9 = data[pos:pos + key_bytes] + packed = int.from_bytes(data[pos + key_bytes:pos + key_bytes + offset_bytes], "big") + stored_size = int.from_bytes( + data[pos + key_bytes + offset_bytes:pos + entry_size], "little") + archive = packed >> 30 + offset = packed & 0x3FFFFFFF + self.by_ekey9[key9] = (archive, offset, stored_size) + pos += entry_size + + def _load_listfile(self, path: pathlib.Path) -> None: + with path.open("r", encoding="utf-8", errors="replace") as stream: + for raw in stream: + try: + left, name = raw.rstrip("\r\n").split(";", 1) + fdid = int(left) + except (ValueError, TypeError): + continue + normalized = name.replace("\\", "/").lower() + self.names_by_fdid[fdid] = name + self.fdids_by_name[normalized] = fdid + + def _decode_blte_chunk(self, chunk: bytes, chunk_index: int) -> bytes: + if not chunk: + return b"" + mode, payload = chunk[0:1], chunk[1:] + if mode == b"N": + return payload + if mode == b"Z": + return zlib.decompress(payload) + if mode == b"E": + if len(payload) < 3: + raise CascError("short encrypted BLTE block") + nkey = payload[0] + key_name = payload[1:1 + nkey][::-1] + at = 1 + nkey + niv = payload[at] + iv = bytearray(payload[at + 1:at + 1 + niv]) + at += 1 + niv + algorithm = payload[at:at + 1] + ciphertext = payload[at + 1:] + key = self.keys.get(key_name) + if key is None: + raise EncryptedBlock(key_name) + if algorithm != b"S": + raise CascError("unknown BLTE encryption algorithm %r" % algorithm) + for i in range(4): + if i < len(iv): + iv[i] ^= (chunk_index >> (8 * i)) & 0xFF + nonce = bytes(iv[:8]).ljust(8, b"\0") + clear = salsa20_xor(ciphertext, key, nonce) + return self._decode_blte_chunk(clear, chunk_index) + raise CascError("unknown BLTE block mode %r" % mode) + + def decode_blte(self, data: bytes) -> bytes: + if data[:4] != b"BLTE": + raise CascError("payload is not BLTE") + header_size, = struct.unpack_from(">I", data, 4) + if header_size == 0: + return self._decode_blte_chunk(data[8:], 0) + if header_size < 12 or header_size > len(data): + raise CascError("invalid BLTE header size") + flags = data[8] + count = _u24be(data[9:12]) + if flags != 0x0F or header_size < 12 + count * 24: + raise CascError("invalid BLTE chunk table") + chunks = [] + pos = 12 + for _ in range(count): + compressed, decompressed = struct.unpack_from(">II", data, pos) + digest = data[pos + 8:pos + 24] + chunks.append((compressed, decompressed, digest)) + pos += 24 + pos = header_size + output = [] + for index, (compressed, decompressed, digest) in enumerate(chunks): + raw = data[pos:pos + compressed] + if len(raw) != compressed: + raise CascError("truncated BLTE chunk") + if hashlib.md5(raw).digest() != digest: + raise CascError("BLTE chunk checksum mismatch") + clear = self._decode_blte_chunk(raw, index) + if len(clear) != decompressed: + raise CascError("BLTE decompressed-size mismatch") + output.append(clear) + pos += compressed + return b"".join(output) + + def open_ekey(self, ekey: bytes) -> bytes: + if isinstance(ekey, str): + ekey = bytes.fromhex(ekey) + location = self.by_ekey9.get(ekey[:9]) + if location is None: + raise CascError("encoding key is not present locally: " + ekey.hex()) + archive, offset, stored_size = location + data_path = self.data_dir / ("data.%03d" % archive) + with data_path.open("rb") as stream: + stream.seek(offset) + record = stream.read(stored_size) + if len(record) != stored_size or len(record) < 30: + raise CascError("short local CASC record for " + ekey.hex()) + # Local data header: reversed 16-byte (zero-extended) EKey, payload size, + # flags/checksum fields; the encoded BLTE stream begins at byte 30. + if record[30:34] != b"BLTE": + raise CascError("local CASC record has no BLTE payload") + return self.decode_blte(record[30:]) + + def load_encoding(self) -> None: + pair = self.config.get("encoding") + if not pair or len(pair) < 2: + raise CascError("build config has no encoding CKey/EKey pair") + data = self.open_ekey(bytes.fromhex(pair[1])) + self._parse_encoding(data) + + def _parse_encoding(self, data: bytes) -> None: + if data[:2] != b"EN": + raise CascError("bad encoding-file signature") + # Header fields are big-endian on disk. + version, ckey_size, ekey_size = data[2], data[3], data[4] + cpage_kib, epage_kib = struct.unpack_from(">HH", data, 5) + cpages, epages = struct.unpack_from(">II", data, 9) + unknown = data[17] + espec_size, = struct.unpack_from(">I", data, 18) + self.encoding_header = dict(version=version, ckey_size=ckey_size, + ekey_size=ekey_size, cpage_kib=cpage_kib, + epage_kib=epage_kib, cpages=cpages, + epages=epages, unknown=unknown, + espec_size=espec_size, + decoded_size=len(data)) + # The ESpec string block follows the 22-byte header. Each CKey page-index + # entry is first-key+MD5; the fixed-size pages immediately follow it. + pos = 22 + espec_size + cpage_size = cpage_kib * 1024 + epage_size = epage_kib * 1024 + cindex_pos = pos + cdata_pos = cindex_pos + cpages * (ckey_size + 16) + mapping: Dict[bytes, List[bytes]] = {} + for page in range(cpages): + page_data = data[cdata_pos + page * cpage_size:cdata_pos + (page + 1) * cpage_size] + at = 0 + while at < len(page_data): + if at + 2 > len(page_data): + break + count = int.from_bytes(page_data[at:at + 2], "little") + at += 2 + if count == 0: + break + need = 4 + ckey_size + count * ekey_size + if at + need > len(page_data): + raise CascError("truncated encoding CKey page") + _decoded_size = int.from_bytes(page_data[at:at + 4], "big") + at += 4 + ckey = page_data[at:at + ckey_size] + at += ckey_size + ekeys = [] + for _ in range(count): + ekeys.append(page_data[at:at + ekey_size]) + at += ekey_size + mapping[ckey] = ekeys + self.encoding_map = mapping + + def open_ckey(self, ckey: bytes) -> bytes: + if isinstance(ckey, str): + ckey = bytes.fromhex(ckey) + if not self.encoding_map: + self.load_encoding() + ekeys = self.encoding_map.get(ckey) + if not ekeys: + raise CascError("content key absent from encoding file: " + ckey.hex()) + errors = [] + for ekey in ekeys: + try: + return self.open_ekey(ekey) + except CascError as exc: + errors.append(str(exc)) + raise CascError("no local encoding for CKey %s: %s" % (ckey.hex(), "; ".join(errors))) + + def load_root(self) -> None: + roots = self.config.get("root") + if not roots: + raise CascError("build config has no root CKey") + data = self.open_ckey(bytes.fromhex(roots[0])) + self._parse_root(data) + + def _parse_root(self, data: bytes) -> None: + if data[:4] != b"TSFM" or len(data) < 24: + raise CascError("unsupported root signature") + header_size, version, total_files, named_files, flags = \ + struct.unpack_from("<5I", data, 4) + if header_size != 24 or version != 2: + raise CascError("unsupported TSFM root version/header") + mapping: Dict[int, List[bytes]] = {} + pos = header_size + seen_total = 0 + seen_named = 0 + blocks = 0 + while pos < len(data): + if pos + 17 > len(data): + raise CascError("truncated TSFM block header") + count, content_flags, locale_flags = struct.unpack_from(" len(data): + raise CascError("truncated TSFM block data") + fdid = -1 + for index in range(count): + delta, = struct.unpack_from(" bytes: + if not self.root_map: + self.load_root() + ckeys = self.root_map.get(int(fdid)) + if not ckeys: + raise CascError("FileDataID absent from root: %d" % fdid) + errors = [] + for ckey in ckeys: + try: + return self.open_ckey(ckey) + except CascError as exc: + errors.append(str(exc)) + raise CascError("no readable root entry for FDID %d: %s" % (fdid, "; ".join(errors))) + + def open_name(self, path: str) -> bytes: + normalized = path.replace("\\", "/").lower() + fdid = self.fdids_by_name.get(normalized) + if fdid is None: + raise CascError("path absent from listfile: " + path) + return self.open_fdid(fdid) diff --git a/Optional/sky-and-grading/tools/check_bands.py b/Optional/sky-and-grading/tools/check_bands.py new file mode 100644 index 00000000..3d6859f2 --- /dev/null +++ b/Optional/sky-and-grading/tools/check_bands.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python3 +"""Check corrected Light band clone row IDs in loose files or an MPQ chain.""" + +import argparse +from pathlib import Path + +from dbc_build import check_bands +from mpq_tools import Storm, copies_for, find_stormlib + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("path", type=Path, help="stage directory or client Data directory") + parser.add_argument("--stormlib", type=Path) + parser.add_argument("--stage", action="store_true", help="path is a loose stage tree") + args = parser.parse_args() + if args.stage: + root = args.path / "DBFilesClient" + integer = (root / "LightIntBand.dbc").read_bytes() + floating = (root / "LightFloatBand.dbc").read_bytes() + else: + storm = Storm(find_stormlib(args.stormlib)) + payloads = [] + for name in ("LightIntBand.dbc", "LightFloatBand.dbc"): + copies = copies_for(storm, args.path, "DBFilesClient\\" + name) + if not copies: + raise FileNotFoundError(name) + payloads.append(copies[-1][1]) + integer, floating = payloads + check_bands(integer, floating) + print("PASS: clone rows use (P-1)*count + band + 1 and match params 23/21") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/Optional/sky-and-grading/tools/dbc_build.py b/Optional/sky-and-grading/tools/dbc_build.py new file mode 100644 index 00000000..bc2612d2 --- /dev/null +++ b/Optional/sky-and-grading/tools/dbc_build.py @@ -0,0 +1,200 @@ +"""Winner-safe lighting DBC merge for the optional sky patch.""" + +from __future__ import annotations + +import struct +from typing import Dict, List, Set, Tuple + + +REQUIRED = ( + "Light.dbc", + "LightParams.dbc", + "LightIntBand.dbc", + "LightFloatBand.dbc", + "LightSkybox.dbc", +) + + +def parse_dbc(data: bytes, label: str) -> Tuple[List[List[int]], bytes, int]: + if len(data) < 20: + raise ValueError("truncated DBC: %s" % label) + magic, count, fields, record_size, string_size = struct.unpack_from("<4s4I", data) + if magic != b"WDBC" or record_size != fields * 4: + raise ValueError("invalid WDBC schema: %s" % label) + expected = 20 + count * record_size + string_size + if len(data) != expected: + raise ValueError("invalid WDBC length: %s" % label) + records = [ + list(struct.unpack_from("<%dI" % fields, data, 20 + i * record_size)) + for i in range(count) + ] + return records, data[20 + count * record_size :], fields + + +def pack_dbc(records: List[List[int]], strings: bytes, fields: int) -> bytes: + if any(len(record) != fields for record in records): + raise ValueError("inconsistent DBC record width") + header = struct.pack("<4s4I", b"WDBC", len(records), fields, fields * 4, len(strings)) + body = b"".join(struct.pack("<%dI" % fields, *record) for record in records) + return header + body + strings + + +def indexed(records: List[List[int]], label: str) -> Dict[int, List[int]]: + result = {record[0]: record for record in records} + if len(result) != len(records): + raise ValueError("duplicate row ID in %s" % label) + return result + + +def band_row_id(param: int, band: int, per: int) -> int: + return (param - 1) * per + band + 1 + + +def append_string(block: bytearray, value: str) -> int: + encoded = value.encode("utf-8") + b"\0" + start = 0 + while True: + offset = block.find(encoded, start) + if offset < 0: + offset = len(block) + block.extend(encoded) + return offset + if offset == 0 or block[offset - 1] == 0: + return offset + start = offset + 1 + + +def build_merged(winners: Dict[str, bytes]) -> Tuple[Dict[str, bytes], Dict[str, object]]: + missing = [name for name in REQUIRED if name not in winners] + if missing: + raise ValueError("missing winning source DBCs: %s" % ", ".join(missing)) + + light, light_strings, light_fields = parse_dbc(winners["Light.dbc"], "Light") + params, params_strings, params_fields = parse_dbc( + winners["LightParams.dbc"], "LightParams" + ) + ints, int_strings, int_fields = parse_dbc( + winners["LightIntBand.dbc"], "LightIntBand" + ) + floats, float_strings, float_fields = parse_dbc( + winners["LightFloatBand.dbc"], "LightFloatBand" + ) + skies, sky_strings, sky_fields = parse_dbc( + winners["LightSkybox.dbc"], "LightSkybox" + ) + if (light_fields, params_fields, int_fields, float_fields, sky_fields) != ( + 12, 9, 34, 34, 2 + ): + raise ValueError("unexpected source lighting schema; use an unmodified 1.12 DBC chain") + if any(block != b"\0" for block in ( + light_strings, params_strings, int_strings, float_strings + )): + raise ValueError("numeric lighting DBC has a non-empty string block") + + light_by_id = indexed(light, "Light") + params_by_id = indexed(params, "LightParams") + int_by_id = indexed(ints, "LightIntBand") + float_by_id = indexed(floats, "LightFloatBand") + + targets = {20: 1139, 539: 1139, 6: 1140, 65: 1140, 67: 1140} + expected = {20: 23, 539: 23, 6: 21, 65: 21, 67: 21} + if light_by_id.get(14, [None] * 8)[7] != 16: + raise ValueError("Light row 14 is not on expected Swamp of Sorrows param 16") + for row_id, new_param in targets.items(): + if row_id not in light_by_id or light_by_id[row_id][7] != expected[row_id]: + raise ValueError("unexpected Light[%d].Clear" % row_id) + light_by_id[row_id][7] = new_param + + start_param = max( + max(params_by_id), + (max(int_by_id) + 17) // 18, + (max(float_by_id) + 5) // 6, + ) + 1 + if start_param != 1139: + raise ValueError( + "winner-derived clone start is %d, expected 1139; refusing an unsafe merge" + % start_param + ) + + for source_param, new_param, skybox_id in ((23, 1139, 8), (21, 1140, 9)): + clone = params_by_id[source_param][:] + clone[0] = new_param + clone[2] = skybox_id + params.append(clone) + for per, source, destination in ( + (18, int_by_id, ints), + (6, float_by_id, floats), + ): + for band in range(per): + source_id = band_row_id(source_param, band, per) + target_id = band_row_id(new_param, band, per) + if source_id not in source or target_id in source: + raise ValueError("unsafe band clone %d -> %d" % (source_id, target_id)) + row = source[source_id][:] + row[0] = target_id + destination.append(row) + + if params_by_id.get(269, [None, None, None])[2] != 0: + raise ValueError("unexpected LightParams[269].LightSkybox") + params_by_id[269][2] = 10 + + strings = bytearray(sky_strings) + extended = [[row[0], row[1], 0, 0] for row in skies] + specs = ( + (8, r"Environments\Stars\10can_volcanicsky01.mdx", 15, ""), + (9, r"Environments\Stars\bonewastesskybox.mdx", 4, ""), + ( + 10, + r"Environments\Stars\8drk_darkshoreelune_sky01.mdx", + 2, + r"Environments\Stars\hyjal_skybox.mdx", + ), + ) + for row_id, model, flags, celestial in specs: + extended.append( + [row_id, append_string(strings, model), flags, append_string(strings, celestial)] + ) + + light.sort(key=lambda row: row[0]) + params.sort(key=lambda row: row[0]) + ints.sort(key=lambda row: row[0]) + floats.sort(key=lambda row: row[0]) + extended.sort(key=lambda row: row[0]) + outputs = { + "Light.dbc": pack_dbc(light, light_strings, 12), + "LightParams.dbc": pack_dbc(params, params_strings, 9), + "LightIntBand.dbc": pack_dbc(ints, int_strings, 34), + "LightFloatBand.dbc": pack_dbc(floats, float_strings, 34), + "LightSkybox.dbc": pack_dbc(extended, bytes(strings), 4), + } + report = { + "source_rows": { + name: len(parse_dbc(winners[name], name)[0]) for name in REQUIRED + }, + "output_rows": { + name: len(parse_dbc(payload, name)[0]) for name, payload in outputs.items() + }, + "changed_light_rows": sorted(targets), + "light_14_clear": 16, + "clone_params": [1139, 1140], + } + return outputs, report + + +def check_bands(int_payload: bytes, float_payload: bytes) -> None: + for label, payload, per in ( + ("LightIntBand", int_payload, 18), + ("LightFloatBand", float_payload, 6), + ): + records, _, _ = parse_dbc(payload, label) + rows = indexed(records, label) + for source_param, target_param in ((23, 1139), (21, 1140)): + for band in range(per): + source_id = band_row_id(source_param, band, per) + target_id = band_row_id(target_param, band, per) + if target_id not in rows or rows[source_id][1:] != rows[target_id][1:]: + raise ValueError("%s clone mismatch at row %d" % (label, target_id)) + top = band_row_id(1140, per - 1, per) + stray = [row_id for row_id in rows if row_id > top] + if stray: + raise ValueError("%s has rows above param 1140 span" % label) diff --git a/Optional/sky-and-grading/tools/mpq_tools.py b/Optional/sky-and-grading/tools/mpq_tools.py new file mode 100644 index 00000000..667c8177 --- /dev/null +++ b/Optional/sky-and-grading/tools/mpq_tools.py @@ -0,0 +1,243 @@ +"""Small StormLib wrapper used by the optional-data builder. + +StormLib is loaded from the user's machine. This module never writes to a +client installation: candidates are copied or created below the requested +output directory and are verified after compaction. +""" + +from __future__ import annotations + +import ctypes +import os +from pathlib import Path +import re +import shutil +import stat +import tempfile +from typing import Dict, Iterable, List, Optional, Tuple + + +MPQ_OPEN_READ_ONLY = 0x00000100 +MPQ_FILE_COMPRESS = 0x00000200 +MPQ_FILE_REPLACEEXISTING = 0x80000000 +MPQ_COMPRESSION_ZLIB = 0x02 +MAX_FILE_COUNT = 8192 + + +def find_stormlib(explicit: Optional[Path]) -> Path: + candidates = [] + if explicit: + candidates.append(Path(explicit)) + env = os.environ.get("STORMLIB_PATH") + if env: + candidates.append(Path(env)) + candidates.extend( + [ + Path.cwd() / "StormLib.dll", + Path(__file__).resolve().parent / "StormLib.dll", + ] + ) + for path in candidates: + if path.is_file(): + return path.resolve() + raise FileNotFoundError( + "StormLib.dll was not found; pass --stormlib or set STORMLIB_PATH" + ) + + +class Storm: + def __init__(self, library: Path): + if os.name != "nt": + raise OSError("this StormLib ctypes wrapper currently supports Windows") + self.library = Path(library) + self.dll = ctypes.WinDLL(str(self.library), use_last_error=True) + handle = ctypes.c_void_p + signatures = { + "SFileOpenArchive": ( + ctypes.c_bool, + [ctypes.c_wchar_p, ctypes.c_uint32, ctypes.c_uint32, ctypes.POINTER(handle)], + ), + "SFileCreateArchive": ( + ctypes.c_bool, + [ctypes.c_wchar_p, ctypes.c_uint32, ctypes.c_uint32, ctypes.POINTER(handle)], + ), + "SFileSetMaxFileCount": (ctypes.c_bool, [handle, ctypes.c_uint32]), + "SFileAddFileEx": ( + ctypes.c_bool, + [handle, ctypes.c_wchar_p, ctypes.c_char_p, ctypes.c_uint32, + ctypes.c_uint32, ctypes.c_uint32], + ), + "SFileCompactArchive": ( + ctypes.c_bool, + [handle, ctypes.c_char_p, ctypes.c_bool], + ), + "SFileOpenFileEx": ( + ctypes.c_bool, + [handle, ctypes.c_char_p, ctypes.c_uint32, ctypes.POINTER(handle)], + ), + "SFileGetFileSize": ( + ctypes.c_uint32, + [handle, ctypes.POINTER(ctypes.c_uint32)], + ), + "SFileReadFile": ( + ctypes.c_bool, + [handle, ctypes.c_void_p, ctypes.c_uint32, + ctypes.POINTER(ctypes.c_uint32), ctypes.c_void_p], + ), + "SFileCloseFile": (ctypes.c_bool, [handle]), + "SFileCloseArchive": (ctypes.c_bool, [handle]), + } + for name, (result, arguments) in signatures.items(): + function = getattr(self.dll, name) + function.restype = result + function.argtypes = arguments + + @staticmethod + def fail(message: str) -> None: + error = ctypes.get_last_error() + raise OSError(error, "%s (Win32 error %d)" % (message, error)) + + def open(self, archive: Path, flags: int = 0) -> ctypes.c_void_p: + handle = ctypes.c_void_p() + if not self.dll.SFileOpenArchive(str(archive), 0, flags, ctypes.byref(handle)): + self.fail("SFileOpenArchive failed: %s" % archive) + return handle + + def create(self, archive: Path) -> ctypes.c_void_p: + handle = ctypes.c_void_p() + if not self.dll.SFileCreateArchive( + str(archive), 0, MAX_FILE_COUNT, ctypes.byref(handle) + ): + self.fail("SFileCreateArchive failed: %s" % archive) + return handle + + def close_archive(self, handle: ctypes.c_void_p) -> None: + if handle.value and not self.dll.SFileCloseArchive(handle): + self.fail("SFileCloseArchive failed") + + def read(self, archive: Path, member: str) -> Optional[bytes]: + archive_handle = self.open(archive, MPQ_OPEN_READ_ONLY) + try: + file_handle = ctypes.c_void_p() + encoded = member.encode("ascii") + if not self.dll.SFileOpenFileEx( + archive_handle, encoded, 0, ctypes.byref(file_handle) + ): + return None + try: + high = ctypes.c_uint32(0) + size = self.dll.SFileGetFileSize(file_handle, ctypes.byref(high)) + if size == 0xFFFFFFFF or high.value: + self.fail("SFileGetFileSize failed: %s" % member) + buffer = ctypes.create_string_buffer(size) + received = ctypes.c_uint32(0) + if not self.dll.SFileReadFile( + file_handle, buffer, size, ctypes.byref(received), None + ): + self.fail("SFileReadFile failed: %s" % member) + if received.value != size: + raise OSError("short MPQ read for %s" % member) + return buffer.raw[:size] + finally: + if file_handle.value and not self.dll.SFileCloseFile(file_handle): + self.fail("SFileCloseFile failed: %s" % member) + finally: + self.close_archive(archive_handle) + + def pack( + self, + archive: Path, + files: Iterable[Tuple[Path, str]], + base_archive: Optional[Path] = None, + ) -> None: + archive.parent.mkdir(parents=True, exist_ok=True) + if base_archive: + shutil.copy2(base_archive, archive) + os.chmod(archive, archive.stat().st_mode | stat.S_IWRITE) + handle = self.open(archive) + else: + handle = self.create(archive) + pairs = list(files) + listfile_temp = None + if not base_archive: + listfile_temp = tempfile.TemporaryDirectory(prefix="benilla-listfile-") + listfile_path = Path(listfile_temp.name) / "listfile.txt" + listfile_path.write_text( + "\r\n".join(member for _, member in pairs) + "\r\n(listfile)\r\n", + encoding="ascii", + ) + pairs.append((listfile_path, "(listfile)")) + try: + if not self.dll.SFileSetMaxFileCount(handle, MAX_FILE_COUNT): + self.fail("SFileSetMaxFileCount failed") + for disk_path, member in pairs: + if not self.dll.SFileAddFileEx( + handle, + str(disk_path), + member.encode("ascii"), + MPQ_FILE_COMPRESS | MPQ_FILE_REPLACEEXISTING, + MPQ_COMPRESSION_ZLIB, + MPQ_COMPRESSION_ZLIB, + ): + self.fail("SFileAddFileEx failed: %s" % member) + finally: + self.close_archive(handle) + + handle = self.open(archive) + try: + if not self.dll.SFileCompactArchive(handle, None, False): + self.fail("SFileCompactArchive failed") + finally: + self.close_archive(handle) + + for disk_path, member in pairs: + actual = self.read(archive, member) + if actual != disk_path.read_bytes(): + raise RuntimeError("packed member failed verification: %s" % member) + if listfile_temp is not None: + listfile_temp.cleanup() + + +def mount_key(path: Path) -> Tuple[int, int, str]: + name = path.name.casefold() + if name == "patch.mpq": + return (1, 0, name) + numeric = re.fullmatch(r"patch-(\d+)\.mpq", name) + if numeric: + return (2, int(numeric.group(1)), name) + letter = re.fullmatch(r"patch-(.)\.mpq", name) + if letter: + return (3, ord(letter.group(1)), name) + return (0, 0, name) + + +def archives_in_mount_order(data_dir: Path) -> List[Path]: + paths = [ + path for path in Path(data_dir).iterdir() + if path.is_file() and path.suffix.casefold() == ".mpq" + ] + return sorted(paths, key=mount_key) + + +def copies_for(storm: Storm, data_dir: Path, member: str) -> List[Tuple[Path, bytes]]: + result = [] + for archive in archives_in_mount_order(data_dir): + payload = storm.read(archive, member) + if payload is not None: + result.append((archive, payload)) + return result + + +def winning_payloads( + storm: Storm, data_dir: Path, names: Iterable[str] +) -> Tuple[Dict[str, bytes], Dict[str, str]]: + payloads: Dict[str, bytes] = {} + winners: Dict[str, str] = {} + for name in names: + copies = copies_for(storm, data_dir, "DBFilesClient\\" + name) + if not copies: + raise FileNotFoundError("no mounted copy of DBFilesClient\\%s" % name) + archive, payload = copies[-1] + payloads[name] = payload + winners[name] = archive.name + return payloads, winners diff --git a/Optional/sky-and-grading/tools/sky_convert.py b/Optional/sky-and-grading/tools/sky_convert.py new file mode 100644 index 00000000..a109c9ff --- /dev/null +++ b/Optional/sky-and-grading/tools/sky_convert.py @@ -0,0 +1,409 @@ +"""Extract four modern sky models from local CASC and convert them to M2 v256. + +pywowlib is an external dependency. The conversion and v256 compatibility +fixes here are original glue; no pywowlib source is redistributed. +""" + +from __future__ import annotations + +import importlib +import json +import os +from pathlib import Path +import shutil +import struct +import subprocess +import sys +from typing import Dict, Iterable, List, Tuple + + +PYWOWLIB_COMMIT = "55276dc5c2195da7fe136638a2a59716622f8c65" +MODELS = ( + ("blasted_lands", 130481, "bonewastesskybox.m2"), + ("mount_hyjal_main", 2322316, "8drk_darkshoreelune_sky01.m2"), + ("mount_hyjal_celestial", 317347, "hyjal_skybox.m2"), + ("burning_steppes", 4505901, "10can_volcanicsky01.m2"), +) + + +def load_pywowlib(checkout: Path) -> Dict[str, object]: + checkout = Path(checkout).resolve() + package = checkout if checkout.name.casefold() == "pywowlib" else checkout / "pywowlib" + if not (package / "m2_file.py").is_file(): + raise FileNotFoundError("--pywowlib must name a pywowlib checkout") + try: + head = subprocess.check_output( + ["git", "-C", str(package), "rev-parse", "HEAD"], text=True + ).strip() + except (OSError, subprocess.CalledProcessError) as error: + raise RuntimeError("the pywowlib checkout must retain its .git metadata") from error + if head != PYWOWLIB_COMMIT: + raise RuntimeError( + "pywowlib HEAD is %s; expected pinned commit %s" % (head, PYWOWLIB_COMMIT) + ) + sys.path.insert(0, str(package.parent)) + modules = { + "common": importlib.import_module("pywowlib.file_formats.wow_common_types"), + "types": importlib.import_module("pywowlib.io_utils.types"), + "m2_file": importlib.import_module("pywowlib.m2_file"), + "m2_format": importlib.import_module("pywowlib.file_formats.m2_format"), + "m2_chunks": importlib.import_module("pywowlib.file_formats.m2_chunks"), + "skin_format": importlib.import_module("pywowlib.file_formats.skin_format"), + } + return modules + + +def read_listfile(path: Path) -> Dict[int, str]: + result = {} + with Path(path).open("r", encoding="utf-8", errors="replace") as stream: + for line in stream: + raw_id, separator, name = line.rstrip("\r\n").partition(";") + if separator and raw_id.isdigit(): + result[int(raw_id)] = name + return result + + +def source_version(path: Path) -> int: + data = path.read_bytes()[:16] + if data[:4] == b"MD20": + return struct.unpack_from(" int: + versions = {256: 0, 263: 1, 264: 2, 272: 3, 273: 5, 274: 6} + if version not in versions: + raise ValueError("unsupported M2 version %d" % version) + return versions[version] + + +def extract_sources(reader: object, listfile: Dict[int, str], root: Path, modules: Dict[str, object]) -> None: + M2File = modules["m2_file"].M2File + manager = modules["common"].M2VersionsManager() + for _, file_id, output_name in MODELS: + source_dir = root / str(file_id) + source_dir.mkdir(parents=True, exist_ok=True) + model_path = source_dir / output_name + model_path.write_bytes(reader.open_fdid(file_id)) + version = source_version(model_path) + manager.set_m2_version(modules["common"].M2Versions.from_expansion_number(expansion_for(version))) + model = M2File(expansion_for(version), filepath=str(model_path)) + dependencies = model.find_model_dependencies() + ids = set(int(value) for value in dependencies.skins) + for index, texture in enumerate(model.root.textures): + texture_id = int(getattr(texture, "fdid", 0) or 0) + if not texture_id and model.txid and index < len(model.txid.texture_ids): + texture_id = int(model.txid.texture_ids[index]) + if texture.type == 0 and not texture.filename.value and texture_id: + ids.add(texture_id) + for dependency in sorted(ids): + game_path = listfile.get(dependency) + if not game_path: + raise KeyError("FileDataID %d is absent from the community listfile" % dependency) + local = source_dir / Path(game_path.replace("\\", "/")).name + local.write_bytes(reader.open_fdid(dependency)) + + +def walk_objects(root: object) -> Iterable[object]: + seen = set() + stack = [root] + while stack: + value = stack.pop() + if id(value) in seen: + continue + seen.add(id(value)) + yield value + if isinstance(value, (str, bytes, int, float, bool, type(None), tuple)): + continue + if isinstance(value, list): + stack.extend(value) + continue + values = getattr(value, "values", None) + if isinstance(values, list): + stack.extend(values) + attributes = getattr(value, "__dict__", None) + if attributes: + stack.extend(attributes.values()) + + +def collect_tracks(root: object, track_base: type) -> List[object]: + return [value for value in walk_objects(root) if isinstance(value, track_base)] + + +def assign_sequence_times(root: object) -> None: + timestamp = 0 + for sequence in root.sequences: + duration = int(getattr(sequence, "duration", 0) or 0) + sequence.start_timestamp = timestamp + sequence.end_timestamp = timestamp + duration + timestamp = sequence.end_timestamp + 1000 + + +def denormalize_tracks(root: object, modules: Dict[str, object]) -> None: + common, types, fmt = modules["common"], modules["types"], modules["m2_format"] + M2Array, M2Range = common.M2Array, fmt.M2Range + sequences = list(root.sequences) + + def array(value_type: type, values: List[object]) -> object: + result = M2Array(value_type) + result.values = list(values) + result.n_elements = len(result.values) + return result + + for track in collect_tracks(root, fmt.M2TrackBase): + timestamp_arrays = track.timestamps.values + if not timestamp_arrays or not isinstance(timestamp_arrays[0], M2Array): + continue + has_values = hasattr(track, "values") + value_arrays = track.values.values if has_values else [] + value_type = value_arrays[0].type if value_arrays else types.uint32 + flat_times: List[int] = [] + flat_values: List[object] = [] + ranges: List[Tuple[int, int]] = [] + if track.global_sequence >= 0: + flat_times = [int(value) for value in timestamp_arrays[0].values] + flat_values = list(value_arrays[0].values) if value_arrays else [] + else: + last_value = None + for index, sequence in enumerate(sequences): + times = timestamp_arrays[index].values if index < len(timestamp_arrays) else [] + values = value_arrays[index].values if index < len(value_arrays) else [] + offset = len(flat_times) + if times: + flat_times.extend(int(value) + sequence.start_timestamp for value in times) + if has_values: + flat_values.extend(values) + last_value = values[-1] + ranges.append((offset, offset + len(times) - 1)) + elif has_values and last_value is not None: + flat_times.append(sequence.start_timestamp) + flat_values.append(last_value) + ranges.append((offset, offset)) + else: + ranges.append((offset, max(0, offset - 1))) + ranges.append((0, 0)) + if has_values and flat_values and isinstance(flat_values[0], fmt.M2CompQuaternion): + converted = [] + for value in flat_values: + ww, xx, yy, zz = value.to_quaternion() + converted.append((xx, yy, zz, ww)) + flat_values = converted + value_type = types.quat + interpolation = M2Array(M2Range) + interpolation.values = [] + for minimum, maximum in ranges: + item = M2Range() + item.minimum, item.maximum = minimum, maximum + interpolation.values.append(item) + interpolation.n_elements = len(interpolation.values) + track.interpolation_ranges = interpolation + track.timestamps = array(types.uint32, flat_times) + if has_values: + track.values = array(value_type, flat_values) + + +def stage_sources(shader_id: int, count: int) -> Tuple[int, ...]: + shader_id &= 0xFFFF + if count <= 1: + return (-1,) if shader_id & 0x80 else ((1,) if shader_id & 0x4000 else (0,)) + first = -1 if shader_id & 0x80 else 0 + second = -1 if shader_id & 0x8 else (1 if shader_id & 0x4000 else 0) + return (first, second) + + +def find_or_append(table: List[int], values: Tuple[int, ...]) -> int: + for index in range(len(table) - len(values) + 1): + if tuple(table[index : index + len(values)]) == values: + return index + table.extend(values) + return len(table) - len(values) + + +def fix_texture_units(model: object) -> None: + root = model.root + coordinates = [0, 1] + weights = list(root.transparency_lookup_table.values) + transforms = list(root.texture_transforms_lookup_table.values) + textures = list(root.texture_lookup_table.values) + pair_cache = {} + for skin in model.skins: + for unit in skin.texture_units.values: + count = int(unit.texture_count) + if count not in (1, 2): + raise ValueError("unsupported textureCount %d" % count) + unit.texture_coord_combo_index = find_or_append( + coordinates, stage_sources(unit.shader_id, count) + ) + for stage in range(count): + tex = unit.texture_combo_index + stage + xform = unit.texture_transform_combo_index + stage + if tex >= len(textures) or textures[tex] >= len(root.textures.values): + raise ValueError("texture combo is out of range") + if xform >= len(transforms) or transforms[xform] >= len(root.texture_transforms.values): + raise ValueError("texture transform combo is out of range") + weight = unit.texture_weight_combo_index + if weight >= len(weights) or weights[weight] >= len(root.texture_weights.values): + raise ValueError("texture weight combo is out of range") + if count == 2: + track = weights[weight] + if track not in pair_cache: + pair_cache[track] = find_or_append(weights, (track, track)) + unit.texture_weight_combo_index = pair_cache[track] + root.tex_unit_lookup_table.values = coordinates + root.tex_unit_lookup_table.n_elements = len(coordinates) + root.transparency_lookup_table.values = weights + root.transparency_lookup_table.n_elements = len(weights) + + +def make_classic(model: object, modules: Dict[str, object]) -> None: + common, fmt, chunks = modules["common"], modules["m2_format"], modules["m2_chunks"] + source_root = model.root + assign_sequence_times(source_root) + denormalize_tracks(source_root, modules) + for sequence in source_root.sequences: + if not hasattr(sequence, "blend_time"): + sequence.blend_time = max( + int(getattr(sequence, "blend_time_in", 0)), + int(getattr(sequence, "blend_time_out", 0)), + ) + common.M2VersionsManager().set_m2_version(common.M2Versions.CLASSIC) + root = chunks.MD20() + for name in list(root.__dict__): + if name in source_root.__dict__ and name not in {"magic", "version", "m2_version", "_size"}: + setattr(root, name, getattr(source_root, name)) + model.root = root + model.version = common.M2Versions.CLASSIC + for value in walk_objects(root): + if isinstance(value, fmt.M2TrackBase) and not hasattr(value, "interpolation_ranges"): + value.interpolation_ranges = common.M2Array(fmt.M2Range) + if hasattr(value, "m2_version"): + value.m2_version = common.M2Versions.CLASSIC + if value.__class__.__name__ == "M2SkinProfile": + value._size = 44 + for skin in model.skins: + for value in walk_objects(skin): + if hasattr(value, "m2_version"): + value.m2_version = common.M2Versions.CLASSIC + if value.__class__.__name__ == "M2SkinProfile": + value._size = 44 + + +def write_classic(model: object, path: Path, modules: Dict[str, object]) -> None: + skin_class = modules["skin_format"].M2SkinProfile + bone_class = modules["m2_format"].M2CompBone + types = modules["types"] + original_skin_write = skin_class.write + original_skin_size = getattr(skin_class, "size", None) + original_bone_write = bone_class.write + + def write_embedded(skin: object, stream: object) -> object: + position = stream.tell() + stream.write(b"\0" * 44) + stream.seek(position) + skin.vertex_indices.write(stream) + skin.triangle_indices.write(stream) + skin.bone_indices.write(stream) + skin.submeshes.write(stream) + skin.texture_units.write(stream) + types.uint32.write(stream, skin.bone_count_max) + return skin + + def write_classic_bone(bone: object, stream: object) -> object: + types.int32.write(stream, bone.key_bone_id) + types.uint32.write(stream, bone.flags) + types.int16.write(stream, bone.parent_bone) + types.uint16.write(stream, bone.submesh_id) + bone.translation.write(stream) + bone.rotation.write(stream) + bone.scale.write(stream) + types.vec3D.write(stream, bone.pivot) + return bone + + skin_class.write = write_embedded + skin_class.size = staticmethod(lambda: 44) + bone_class.write = write_classic_bone + try: + model.root.skin_profiles.values = list(model.skins) + model.root.skin_profiles.n_elements = len(model.skins) + model.root.num_skin_profiles = len(model.skins) + with path.open("wb") as stream: + model.root.write(stream) + stream.seek(0, 2) + padding = (16 - stream.tell() % 16) % 16 + stream.write(b"\0" * padding) + finally: + skin_class.write = original_skin_write + if original_skin_size is None: + delattr(skin_class, "size") + else: + skin_class.size = original_skin_size + bone_class.write = original_bone_write + + +def convert_all( + source_root: Path, + output_root: Path, + listfile_path: Path, + modules: Dict[str, object], +) -> List[Dict[str, object]]: + listfile = read_listfile(listfile_path) + output_root.mkdir(parents=True, exist_ok=True) + result = [] + for key, file_id, output_name in MODELS: + source_dir = source_root / str(file_id) + source_path = source_dir / output_name + version = source_version(source_path) + common = modules["common"] + common.M2VersionsManager().set_m2_version( + common.M2Versions.from_expansion_number(expansion_for(version)) + ) + model = modules["m2_file"].M2File(expansion_for(version), filepath=str(source_path)) + dependencies = model.find_model_dependencies() + skins = [] + for dependency in dependencies.skins: + name = listfile.get(int(dependency)) + if not name: + raise KeyError("skin FileDataID %d missing from listfile" % dependency) + skins.append(str(source_dir / Path(name.replace("\\", "/")).name)) + model.read_additional_files(skins, {}) + textures = [] + for index, texture in enumerate(model.root.textures): + file_id_value = int(getattr(texture, "fdid", 0) or 0) + if not file_id_value and model.txid and index < len(model.txid.texture_ids): + file_id_value = int(model.txid.texture_ids[index]) + if texture.type == 0 and not texture.filename.value: + game_path = listfile.get(file_id_value) + if not game_path: + raise KeyError("texture FileDataID %d missing from listfile" % file_id_value) + texture.filename.value = game_path.replace("/", "\\") + local = source_dir / Path(game_path.replace("\\", "/")).name + target = output_root.joinpath(*game_path.replace("\\", "/").split("/")) + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(local, target) + textures.append(str(target.relative_to(output_root)).replace("/", "\\")) + model.root.global_flags &= 0x17 + if hasattr(model.root, "texture_combiner_combos"): + model.root.texture_combiner_combos.values = [] + model.root.texture_combiner_combos.n_elements = 0 + for material in model.root.materials: + if int(material.blending_mode) > 6: + material.blending_mode = 2 + fix_texture_units(model) + make_classic(model, modules) + target = output_root / "Environments" / "Stars" / output_name + target.parent.mkdir(parents=True, exist_ok=True) + write_classic(model, target, modules) + result.append( + { + "key": key, + "source_file_data_id": file_id, + "source_version": version, + "output": str(target), + "output_bytes": target.stat().st_size, + "skins_embedded": len(model.skins), + "textures": textures, + } + ) + return result diff --git a/Optional/sky-and-grading/tools/verify_client_chain.py b/Optional/sky-and-grading/tools/verify_client_chain.py new file mode 100644 index 00000000..53a6070c --- /dev/null +++ b/Optional/sky-and-grading/tools/verify_client_chain.py @@ -0,0 +1,111 @@ +#!/usr/bin/env python3 +"""Verify every staged payload is the mounted winner in a candidate Data chain.""" + +from __future__ import annotations + +import argparse +import hashlib +from pathlib import Path + +from dbc_build import check_bands, indexed, parse_dbc +from mpq_tools import Storm, copies_for, find_stormlib + + +def digest(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def semantic_checks(payloads: dict[str, bytes]) -> None: + light, _, fields = parse_dbc(payloads["DBFilesClient\\Light.dbc"], "Light") + if fields != 12: + raise ValueError("Light.dbc is not 12 fields") + light_rows = indexed(light, "Light") + expected = {20: 1139, 539: 1139, 6: 1140, 65: 1140, 67: 1140, 14: 16} + for row_id, clear in expected.items(): + if light_rows[row_id][7] != clear: + raise ValueError("Light[%d].Clear is not %d" % (row_id, clear)) + params, _, fields = parse_dbc( + payloads["DBFilesClient\\LightParams.dbc"], "LightParams" + ) + if fields != 9: + raise ValueError("LightParams.dbc is not 9 fields") + param_rows = indexed(params, "LightParams") + for row_id, skybox in ((1139, 8), (1140, 9), (269, 10)): + if param_rows[row_id][2] != skybox: + raise ValueError("LightParams[%d] skybox mismatch" % row_id) + if 1141 in param_rows: + raise ValueError("stray LightParams 1141") + check_bands( + payloads["DBFilesClient\\LightIntBand.dbc"], + payloads["DBFilesClient\\LightFloatBand.dbc"], + ) + skies, strings, fields = parse_dbc( + payloads["DBFilesClient\\LightSkybox.dbc"], "LightSkybox" + ) + if fields != 4 or not strings: + raise ValueError("LightSkybox.dbc is not the extended four-field layout") + sky_rows = indexed(skies, "LightSkybox") + for row_id in (8, 9, 10): + if row_id not in sky_rows: + raise ValueError("LightSkybox row %d is missing" % row_id) + grade, _, fields = parse_dbc( + payloads["DBFilesClient\\MonkeyZoneGrade.dbc"], "MonkeyZoneGrade" + ) + if fields != 5 or len(grade) != 4: + raise ValueError("MonkeyZoneGrade must have four five-field rows") + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("data_dir", type=Path) + parser.add_argument("--stage", type=Path, required=True) + parser.add_argument("--stormlib", type=Path) + parser.add_argument("--require-winner", default="patch-Z.mpq") + args = parser.parse_args() + storm = Storm(find_stormlib(args.stormlib)) + expected = { + str(path.relative_to(args.stage)).replace("/", "\\"): path.read_bytes() + for path in sorted(args.stage.rglob("*")) + if path.is_file() + } + mounted = {} + failures = [] + for member, wanted in expected.items(): + copies = copies_for(storm, args.data_dir, member) + if not copies: + failures.append("missing %s" % member) + continue + archive, actual = copies[-1] + mounted[member] = actual + if args.require_winner.casefold() != "any" and ( + archive.name.casefold() != args.require_winner.casefold() + ): + failures.append("%s wins from %s" % (member, archive.name)) + if actual != wanted: + failures.append( + "%s differs: %s != %s" % (member, digest(actual), digest(wanted)) + ) + required = { + "DBFilesClient\\Light.dbc", + "DBFilesClient\\LightParams.dbc", + "DBFilesClient\\LightIntBand.dbc", + "DBFilesClient\\LightFloatBand.dbc", + "DBFilesClient\\LightSkybox.dbc", + "DBFilesClient\\MonkeyZoneGrade.dbc", + } + if required <= set(mounted): + try: + semantic_checks(mounted) + except (KeyError, ValueError) as error: + failures.append("semantic check: %s" % error) + if failures: + print("FAIL") + for failure in failures: + print(" " + failure) + return 1 + print("PASS: %d staged payloads are mounted winners" % len(expected)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/Optional/sky-and-grading/tools/wdc.py b/Optional/sky-and-grading/tools/wdc.py new file mode 100644 index 00000000..05b67f0c --- /dev/null +++ b/Optional/sky-and-grading/tools/wdc.py @@ -0,0 +1,171 @@ +"""Minimal WDC5 reader for dense modern WoW DB2 tables.""" +from __future__ import annotations + +import copy +import struct + + +class WdcError(Exception): + pass + + +def _bits_le(data: bytes, bit_offset: int, bit_count: int) -> int: + if bit_count == 0: + return 0 + start = bit_offset // 8 + shift = bit_offset & 7 + count = (shift + bit_count + 7) // 8 + value = int.from_bytes(data[start:start + count], "little") >> shift + return value & ((1 << bit_count) - 1) + + +class WDC5: + HEADER_OFFSET = 0x88 + + def __init__(self, data: bytes): + self.data = data + if data[:4] != b"WDC5" or len(data) < self.HEADER_OFFSET + 68: + raise WdcError("not a supported WDC5 table") + o = self.HEADER_OFFSET + (self.record_count, self.field_count, self.record_size, + self.string_table_size, self.table_hash, self.layout_hash, + self.min_id, self.max_id, self.locale) = struct.unpack_from("<9I", data, o) + self.flags, self.id_index = struct.unpack_from("= self.record_count: + raise WdcError("WDC5 relationship record index is out of bounds") + self.relationship_by_index[record_index] = foreign_id + self.rows[self.row_ids[record_index]]["parent_id"] = foreign_id + for i in range(self.copy_table_count): + new_id, source_id = struct.unpack_from(" additional: + raise WdcError("palette index out of bounds") + return struct.unpack_from(" poff + additional: + raise WdcError("palette-array index out of bounds") + return struct.unpack_from("<%dI" % width, self.pallet, base) + raise WdcError("unsupported WDC5 storage type %d" % kind) + + def string(self, row, field: int) -> str: + relative = row["values"][field] + start = row["record_offset"] + relative + if start < self.strings_offset or start >= self.strings_offset + self.section_string_size: + return "" + end = self.data.find(b"\0", start, self.strings_offset + self.section_string_size) + if end < 0: + end = self.strings_offset + self.section_string_size + return self.data[start:end].decode("utf-8", errors="replace") + + @staticmethod + def as_float(value: int) -> float: + return struct.unpack(" Tuple[List[int], List[int]]: + count = row[1] + if count > 16: + raise ValueError("LightIntBand row has more than 16 keys") + return row[2 : 2 + count], row[18 : 18 + count] + + +def make_band(record_id: int, times: List[int], values: List[int]) -> List[int]: + if len(times) != len(values) or len(times) > 16: + raise ValueError("invalid band key arrays") + return [record_id, len(times)] + times + [0] * (16 - len(times)) + values + [0] * ( + 16 - len(values) + ) + + +def rgb_tuple(value: int) -> Tuple[float, float, float]: + return ( + ((value >> 16) & 255) / 255.0, + ((value >> 8) & 255) / 255.0, + (value & 255) / 255.0, + ) + + +def rgb_int(rgb: Iterable[float]) -> int: + channels = [max(0, min(255, round(value * 255.0))) for value in rgb] + return (channels[0] << 16) | (channels[1] << 8) | channels[2] + + +def sample(row: List[int], time: int) -> int: + times, values = unpack_band(row) + if not times: + return 0 + if len(times) == 1: + return values[0] & 0xFFFFFF + time %= DAY + if time < times[0] or time >= times[-1]: + left_t, right_t = times[-1], times[0] + DAY + left_v, right_v = values[-1], values[0] + query = time + DAY if time < times[0] else time + else: + right = bisect.bisect_right(times, time) + left_t, right_t = times[right - 1], times[right] + left_v, right_v = values[right - 1], values[right] + query = time + amount = (query - left_t) / (right_t - left_t) + return rgb_int( + a + (b - a) * amount for a, b in zip(rgb_tuple(left_v), rgb_tuple(right_v)) + ) + + +def srgb_to_linear(value: float) -> float: + return value / 12.92 if value <= 0.04045 else ((value + 0.055) / 1.055) ** 2.4 + + +def linear_to_srgb(value: float) -> float: + value = max(0.0, value) + return 12.92 * value if value <= 0.0031308 else 1.055 * value ** (1 / 2.4) - 0.055 + + +def rgb_to_oklab(rgb: Tuple[float, float, float]) -> Tuple[float, float, float]: + red, green, blue = (srgb_to_linear(value) for value in rgb) + ll = 0.4122214708 * red + 0.5363325363 * green + 0.0514459929 * blue + mm = 0.2119034982 * red + 0.6806995451 * green + 0.1073969566 * blue + ss = 0.0883024619 * red + 0.2817188376 * green + 0.6299787005 * blue + ll, mm, ss = ( + math.copysign(abs(ll) ** (1 / 3), ll), + math.copysign(abs(mm) ** (1 / 3), mm), + math.copysign(abs(ss) ** (1 / 3), ss), + ) + return ( + 0.2104542553 * ll + 0.7936177850 * mm - 0.0040720468 * ss, + 1.9779984951 * ll - 2.4285922050 * mm + 0.4505937099 * ss, + 0.0259040371 * ll + 0.7827717662 * mm - 0.8086757660 * ss, + ) + + +def oklab_to_rgb(lab: Tuple[float, float, float]) -> Tuple[float, float, float]: + light, aa, bb = lab + ll = light + 0.3963377774 * aa + 0.2158037573 * bb + mm = light - 0.1055613458 * aa - 0.0638541728 * bb + ss = light - 0.0894841775 * aa - 1.2914855480 * bb + ll, mm, ss = ll ** 3, mm ** 3, ss ** 3 + red = 4.0767416621 * ll - 3.3077115913 * mm + 0.2309699292 * ss + green = -1.2684380046 * ll + 2.6097574011 * mm - 0.3413193965 * ss + blue = -0.0041960863 * ll - 0.7034186147 * mm + 1.7076147010 * ss + return tuple( + max(0.0, min(1.0, linear_to_srgb(value))) for value in (red, green, blue) + ) + + +def chroma_hue(lab: Tuple[float, float, float]) -> Tuple[float, float]: + _, aa, bb = lab + return math.hypot(aa, bb), math.degrees(math.atan2(bb, aa)) % 360.0 + + +def clamp_duskwood( + base: Tuple[float, float, float], mixed: Tuple[float, float, float] +) -> Tuple[float, float, float]: + base_chroma, base_hue = chroma_hue(base) + mixed_chroma, mixed_hue = chroma_hue(mixed) + mixed_chroma = min(mixed_chroma, base_chroma + 0.015) + if mixed_chroma < 1e-8: + return (mixed[0], 0.0, 0.0) + if base_chroma >= 0.006: + shift = (mixed_hue - base_hue + 180.0) % 360.0 - 180.0 + hue = base_hue + max(-12.0, min(12.0, shift)) + else: + hue = 155.0 + radians = math.radians(hue) + return ( + mixed[0], + mixed_chroma * math.cos(radians), + mixed_chroma * math.sin(radians), + ) + + +def blend(base_value: int, wf_value: int, amount: float, duskwood: bool) -> int: + if base_value == wf_value or amount <= 0.0: + return base_value + base = rgb_to_oklab(rgb_tuple(base_value)) + modern = rgb_to_oklab(rgb_tuple(wf_value)) + mixed = tuple(a + (b - a) * amount for a, b in zip(base, modern)) + if duskwood: + mixed = clamp_duskwood(base, mixed) + return rgb_int(oklab_to_rgb(mixed)) + + +def strength(time: int, night: float, day: float) -> float: + day_weight = (1.0 - math.cos(2.0 * math.pi * (time % DAY) / DAY)) * 0.5 + return night + (day - night) * day_weight + + +def modern_field_for_band(band: int) -> int: + if band <= 7: + return 3 + band + if band == 8: + return 20 + return band + 2 + + +def apply_wf_colours(base_payload: bytes, light_data: object) -> Tuple[bytes, int]: + records, strings, fields = parse_dbc(base_payload, "LightIntBand") + if fields != 34 or strings != b"\0": + raise ValueError("unexpected LightIntBand layout") + by_id = {row[0]: row for row in records} + modern: Dict[int, List[object]] = {} + for row in light_data.rows.values(): + values = row["values"] + param = row.get("parent_id", values[1]) + if param in TARGET_PARAMS: + modern.setdefault(param, []).append(row) + missing = [param for param in TARGET_PARAMS if not modern.get(param)] + if missing: + raise ValueError("modern LightData is missing target params: %s" % missing) + + changed = 0 + for param in TARGET_PARAMS: + rows = sorted( + modern[param], key=lambda row: (row["values"][2] & 0xFFFF, row["id"]) + ) + if len(rows) > 16: + step = (len(rows) - 1) / 15.0 + rows = [rows[round(index * step)] for index in range(16)] + for band in BLENDED_BANDS: + record_id = (param - 1) * 18 + band + 1 + if record_id not in by_id: + raise ValueError("missing LightIntBand row %d" % record_id) + base_row = by_id[record_id] + modern_times = [row["values"][2] & 0xFFFF for row in rows] + field = modern_field_for_band(band) + modern_values = [row["values"][field] & 0xFFFFFF for row in rows] + modern_row = make_band(record_id, modern_times, modern_values) + base_times, _ = unpack_band(base_row) + times = sorted(set(base_times) | set(modern_times) | set(ANCHORS)) + if len(times) > 16: + raise ValueError("moderated row %d needs more than 16 keys" % record_id) + duskwood = param in DUSKWOOD_LOCAL + night, day = ((0.30, 0.15) if duskwood else (0.50, 0.25)) + values = [ + blend( + sample(base_row, time), + sample(modern_row, time), + strength(time, night, day), + duskwood, + ) + for time in times + ] + if any(sample(base_row, time) != value for time, value in zip(times, values)): + by_id[record_id][:] = make_band(record_id, times, values) + changed += 1 + return pack_dbc(records, strings, fields), changed diff --git a/README.md b/README.md index 129c6dab..6be255b0 100644 --- a/README.md +++ b/README.md @@ -126,6 +126,8 @@ at any IP or hostname, appending the auth port if yours is remapped (`WOW_HOST=play.example.com:5000`). Credentials go in at the login screen, or set `WOW_USER` / `WOW_PASS` to skip it. +Optional, locally built skybox and colour-grading data: [`Optional/sky-and-grading/`](Optional/sky-and-grading/README.md). + ## Contributing Issues and pull requests are open. [`docs/CONTRIBUTING.md`](docs/CONTRIBUTING.md) says what gets diff --git a/crates/benilla-app/assets/ui/OptionsFrame.xml b/crates/benilla-app/assets/ui/OptionsFrame.xml index 74dc3699..dc95de76 100644 --- a/crates/benilla-app/assets/ui/OptionsFrame.xml +++ b/crates/benilla-app/assets/ui/OptionsFrame.xml @@ -543,6 +543,7 @@ foliageWind = "Foliage Wind", windGrass = "Grass", windTrees = "Grass + Trees", + foliageWindStrength = "Foliage Wind Strength", -- MONKEY (fix-wind) -- MONKEY (ao): localised contact-shadow row. ambientOcclusion = "Ambient Occlusion", -- MONKEY (lampfog) @@ -707,6 +708,11 @@ "Grass bends in two travelling wind waves and parts around nearby characters. " .. "Grass + Trees also gives classified tree and bush foliage a gentle sway.\n\n" .. "Off makes no vertex displacement and restores the unchanged image. Tree shadows stay static.", + -- MONKEY (fix-wind) + FOLIAGE_WIND_STRENGTH = + "How far grass and foliage swing in the wind. 100% is the standard sway; lower " + .. "is calmer, higher is a blustery day. Storms swing everything harder on top. " + .. "Grass still parts around characters the same way. Needs Foliage Wind on.", -- MONKEY (ao): what the contact shadows touch and what they spare. AMBIENT_OCCLUSION = "Soft contact shadows where surfaces meet: under props and furniture, in the corners " @@ -782,6 +788,7 @@ foliageWind = "Ветер листвы", windGrass = "Трава", windTrees = "Трава + деревья", + foliageWindStrength = "Сила ветра листвы", -- MONKEY (fix-wind) -- MONKEY (ao): Russian contact-shadow row. ambientOcclusion = "Затенение окружения", -- MONKEY (lampfog) @@ -956,6 +963,11 @@ "Трава гнётся двумя бегущими волнами ветра и расступается вокруг персонажей. " .. "Трава + деревья также добавляет мягкое качание листвы деревьев и кустов.\n\n" .. "Выключено не смещает вершины и восстанавливает исходный вид. Тени деревьев остаются неподвижными.", + -- MONKEY (fix-wind) + FOLIAGE_WIND_STRENGTH = + "Насколько сильно трава и листва качаются на ветру. 100% — обычное качание; меньше — " + .. "спокойнее, больше — ветреный день. В грозу всё качается ещё сильнее. " + .. "Трава по-прежнему расступается вокруг персонажей. Работает, если включён ветер листвы.", -- MONKEY (ao): Russian contact-shadow explanation. AMBIENT_OCCLUSION = "Мягкие контактные тени там, где поверхности соприкасаются: под предметами и мебелью, " @@ -1049,6 +1061,8 @@ "RowRainSurfaces", -- MONKEY (wind): appended for conflict-free lane merges. "RowFoliageWind", + -- MONKEY (fix-wind): the sway strength, right under its tier. + "RowFoliageWindStrength", -- MONKEY (ao): appended so the other rows keep their indices. "RowAmbientOcclusion", -- MONKEY (lampfog) @@ -1574,6 +1588,9 @@ -- MONKEY (daylight: terrain torch casters): the ground only casts for outdoor fires. OptionsRow_SetEnabled(getglobal(base .. "RowTorchTerrainShadows"), GetCVar("exteriorShadows") ~= "0") + -- MONKEY (fix-wind): the strength only matters while some foliage sways. + OptionsRow_SetEnabled(getglobal(base .. "RowFoliageWindStrength"), + GetCVar("foliageWind") ~= "0") return end if key == "Controls" then @@ -3260,12 +3277,25 @@ }) - - + + + + OptionsRow_OnLoad(self, "foliageWindStrength", + BENILLA_ADVGFX.foliageWindStrength, + "BENILLA_TOOLTIP_FOLIAGE_WIND_STRENGTH") + OptionsSliderRow_Setup(self, 0.25, 3, 0.25) + + + + + + + + OptionsRow_OnLoad(self, "ambientOcclusion", BENILLA_ADVGFX.ambientOcclusion, "BENILLA_TOOLTIP_AMBIENT_OCCLUSION") diff --git a/crates/benilla-app/src/capture/scenarios.rs b/crates/benilla-app/src/capture/scenarios.rs index 8aa90ad1..7b5fdd20 100644 --- a/crates/benilla-app/src/capture/scenarios.rs +++ b/crates/benilla-app/src/capture/scenarios.rs @@ -1808,6 +1808,16 @@ pub(super) const ON_DEMAND: &[Scenario] = &[ minute: 0, ui: None, }, + // MONKEY (leftovers): the Westfall sea with the visible sun ~5° up on its fixed 45° bearing + // (minute 1200), framed so the sun sits above the open water: the sun-shaft sky mask test. + Scenario { + name: "leftovers-sea-sunset", + map: Some(MAP_AZEROTH), + eye: [-10500.0, 2112.0, 6.0], + look: [-10420.0, 2192.0, 8.0], + minute: 1200, + ui: None, + }, ]; /// MONKEY (p0 baseline): the Elwynn programme view, above the canopy east of Goldshire looking diff --git a/crates/benilla-app/src/cvars.rs b/crates/benilla-app/src/cvars.rs index 8e6286cb..ff90cf2e 100644 --- a/crates/benilla-app/src/cvars.rs +++ b/crates/benilla-app/src/cvars.rs @@ -964,6 +964,12 @@ pub(crate) const REGISTERED: &[Registered] = &[ "2", "benilla's own: foliage wind, 0 Off / 1 Grass / 2 Grass + trees", ), + // MONKEY (fix-wind): the sway strength slider under Foliage Wind; not on the preset ladder. + ours( + "foliageWindStrength", + "1", + "benilla's own: foliage wind strength, a gain on the sway, 0.25..3 (1 = the shipped tuning)", + ), // MONKEY (fog): the distance-fog model. 0 = the 1.12 linear fog (byte-identical), 1 = Modern // (exponential, sun/horizon colour, end-fog shift, fog end decoupled from farclip past 777). // Default 0; the Graphics preset's High sets 1. @@ -4420,6 +4426,7 @@ mod tests { app.init_resource::(); // MONKEY (wind) app.init_resource::(); + app.init_resource::(); // MONKEY (fix-wind) app.init_resource::(); // MONKEY (wet) app.add_observer(crate::monkey_gfx::on_cvar); }, diff --git a/crates/benilla-app/src/monkey_gfx.rs b/crates/benilla-app/src/monkey_gfx.rs index 8b40b233..261b48eb 100644 --- a/crates/benilla-app/src/monkey_gfx.rs +++ b/crates/benilla-app/src/monkey_gfx.rs @@ -6,12 +6,13 @@ //! |---|---|---|---|---| //! | `skyDither` | 0 Off / 1 On | 0 | 1 | `benilla_world::ffx_glow::SkyDither` | //! | `foliageWind` | 0 Off / 1 Grass / 2 Grass + trees | 2 | 2 | `benilla_world::wind::FoliageWind` | +//! | `foliageWindStrength` | 0.25..3 sway gain | 1 | not governed | `benilla_world::wind::FoliageWindStrength` | //! | `fogModel` | 0 Classic / 1 Modern | 0 | 1 | `benilla_world::lighting::FogModelSetting` | //! | `rainSurfaces` | 0 Off / 1 On | 1 | 1 | `benilla_world::weather::RainSurfaces` | use benilla_world::ffx_glow::SkyDither; use benilla_world::lighting::FogModelSetting; -use benilla_world::wind::FoliageWind; +use benilla_world::wind::{FoliageWind, FoliageWindStrength}; use bevy::prelude::*; use benilla_world::weather::RainSurfaces; // MONKEY (wet) @@ -20,6 +21,7 @@ pub(crate) fn on_cvar( ev: On, mut dither: ResMut, mut foliage_wind: ResMut, + mut wind_strength: ResMut, // MONKEY (fix-wind) // MONKEY (fog) mut fog_model: ResMut, mut rain: ResMut, // MONKEY (wet) @@ -41,6 +43,13 @@ pub(crate) fn on_cvar( foliage_wind.0 = want; } } + // MONKEY (fix-wind): the sway gain, live, clamped to the slider's 0.25..3. + "foliagewindstrength" => { + let want = FoliageWindStrength::from_cvar(ev.num()); + if *wind_strength != want { + *wind_strength = want; + } + } // MONKEY (fog): `WOW_FOGMODEL` still wins for the session (captures). "fogmodel" => fog_model.set_from_cvar(ev.num()), // MONKEY (wet): wet surfaces + rain ripples. @@ -74,6 +83,7 @@ impl Plugin for MonkeyGfxPlugin { .insert_resource(FoliageWind( foliage_wind_pin().unwrap_or(FoliageWind::REGISTERED), )) + .init_resource::() // MONKEY (fix-wind) .init_resource::() .init_resource::() // MONKEY (wet) .add_observer(on_cvar); diff --git a/crates/benilla-app/src/pipe_warm/menagerie.rs b/crates/benilla-app/src/pipe_warm/menagerie.rs index 8701048e..43b0195b 100644 --- a/crates/benilla-app/src/pipe_warm/menagerie.rs +++ b/crates/benilla-app/src/pipe_warm/menagerie.rs @@ -63,6 +63,11 @@ pub(super) struct WarmLanes<'w> { /// The UI quad store (2262): the minimap interior composite's tile material is built here /// through its own production builder, for the tile-quad warm rig below. ui_quads: ResMut<'w, Assets>, + /// MONKEY (leftovers): the two sun-shadow proxy lanes, `ShadowCasterMaterial` (solid) and + /// `CutoutShadowCasterMaterial` (alpha-tested leaves). + /// `Option`: an app without the shadow plugins (tests, the glue) has no store. + shadow_solid: Option>>, + shadow_cutout: Option>>, } /// One portrait/paperdoll booth camera + its layer ([`crate::portrait`], 0938): the booths run @@ -700,6 +705,52 @@ pub(super) fn spawn_menagerie( )); count += 1; + // MONKEY (leftovers): the sun-shadow proxy lanes. Production proxies live on the private + // layer 31, which the sun rig's directional light casts from and the world camera also draws + // (forward discards every fragment). The solid proxy usually compiles at entry anyway (the + // player's own caster), but the cutout one first appears with the first leaf card in reach, + // which can be long after the cover lifts. Each rig sits on layers 0 and 31: the world camera + // mints the forward pipeline, the shadow views (when a sun rig is live) the depth one. Meshes + // are the production attribute sets (`empty_shadow_mesh` / `empty_cutout_mesh`), non-empty. + let proxy_layers = bevy::camera::visibility::RenderLayers::from_layers(&[ + 0, + crate::shadow_core::PLAYER_SHADOW_LAYER, + ]); + let tri = [[0.0, 0.0, 0.0], [1.0, 0.0, 0.0], [1.0, 1.0, 0.0]]; + let mut solid_mesh = crate::shadow_core::empty_shadow_mesh(); + solid_mesh.insert_attribute(Mesh::ATTRIBUTE_POSITION, tri.to_vec()); + solid_mesh.insert_indices(Indices::U32(vec![0, 1, 2])); + let mut cutout_mesh = crate::shadow_core::empty_cutout_mesh(); + cutout_mesh.insert_attribute(Mesh::ATTRIBUTE_POSITION, tri.to_vec()); + cutout_mesh.insert_attribute(Mesh::ATTRIBUTE_UV_0, vec![[0.0f32, 0.0], [1.0, 0.0], [1.0, 1.0]]); + cutout_mesh.insert_indices(Indices::U32(vec![0, 1, 2])); + if let Some(store) = lanes.shadow_solid.as_mut() { + let solid = store.add(crate::shadow_core::ShadowCasterMaterial {}); + spawn_lane_rig( + commands, + cam, + Some(proxy_layers.clone()), + &meshes.add(solid_mesh), + None, + solid, + &mut count, + ); + } + if let Some(store) = lanes.shadow_cutout.as_mut() { + // A real stand-in leaf, so the `#[texture]` binding never lands on the retry path. + let leaf = lanes.images.add(Image::default()); + let cutout = store.add(crate::world_shadow::CutoutShadowCasterMaterial { leaf }); + spawn_lane_rig( + commands, + cam, + Some(proxy_layers), + &meshes.add(cutout_mesh), + None, + cutout, + &mut count, + ); + } + count } diff --git a/crates/benilla-app/src/post/sun_shafts.rs b/crates/benilla-app/src/post/sun_shafts.rs index 77887be3..fdf382f2 100644 --- a/crates/benilla-app/src/post/sun_shafts.rs +++ b/crates/benilla-app/src/post/sun_shafts.rs @@ -6,6 +6,8 @@ use super::bloom::BloomLabel; use crate::video::VideoConfig; use benilla_world::{lighting::WowLighting, view::WorldCamera, wmo_portal::CameraInteriorClaim}; +// MONKEY (leftovers): LightSkybox flag 0x8. +use benilla_world::skybox::CameraSkybox; use bevy::{ core_pipeline::{ core_3d::graph::{Core3d, Node3d}, @@ -36,6 +38,12 @@ struct ShaftView { sun: Vec4, // Gamma-space sun colour; the framebuffer is still gamma-space here. color: Vec4, + // MONKEY (leftovers): the view ray's world height is `dot(horizon.xyz, (ndc.x, ndc.y, 1))` + // over `length((ndc.x * ray.x, ndc.y * ray.y, 1))`; the mask counts a depth-0 pixel as sky + // only above `horizon.w` (sin elevation). Water writes no depth, so the sea read as sky. + horizon: Vec4, + // xy = tan half-FOV per axis (1/P00, 1/P11); zw unused. + ray: Vec4, } impl Plugin for SunShaftsPlugin { @@ -79,6 +87,7 @@ fn update_views( video: Res, lighting: Res, interior: Res, + sky: Option>, mut cameras: Query< ( Entity, @@ -92,6 +101,11 @@ fn update_views( >, ) { let to_sun = lighting.celestial_dir().normalize_or_zero(); + // MONKEY (leftovers): a `0x8` sky forces the shafts: its weight lifts the daylight ramp, so + // a low or just-set sun keeps full shafts (a sun far below the horizon still ends them). + let force = sky.map_or(0.0, |s| { + s.flag_weight(benilla_formats::SKYBOX_FORCE_SUN_SHAFTS) + }); for (entity, camera, transform, projection, mut camera3d, old) in &mut cameras { let disabled = !video.sun_shafts || !camera.is_active || interior.0.is_some() || to_sun == Vec3::ZERO; @@ -121,18 +135,22 @@ fn update_views( ) .max_element(); let edge_fade = 1.0 - smoothstep(0.0, 0.18, outside); - let daylight = smoothstep(-0.02, 0.16, to_sun.y); + let daylight = smoothstep(-0.02, 0.16, to_sun.y) + .max(force.clamp(0.0, 1.0) * smoothstep(-0.12, -0.02, to_sun.y)); let strength = edge_fade * daylight; if strength <= 0.001 { commands.entity(entity).remove::(); continue; } camera3d.depth_texture_usages.0 |= TextureUsages::TEXTURE_BINDING.bits(); + let (horizon, ray) = horizon_row(transform, projection); let next = ShaftView { sun: uv.extend(strength).extend(28.0), color: Vec3::from_array(lighting.diffuse) .lerp(Vec3::ONE, 0.18) .extend(0.0), + horizon, + ray, }; if old != Some(&next) { commands.entity(entity).insert(next); @@ -140,6 +158,22 @@ fn update_views( } } +/// MONKEY (leftovers): the view ray's world-up component as a linear form in NDC, plus the per-axis +/// ray scale to normalise it. A pixel is sky only above `-0.01` (≈ 0.6° below the horizon): a sun +/// just under the horizon still finds sky next to it, the sea below it never does. +fn horizon_row(transform: &GlobalTransform, projection: &Projection) -> (Vec4, Vec4) { + let Projection::Perspective(p) = projection else { + // No horizon test: `(0, 0, 1)` is always "up". + return (Vec4::new(0.0, 0.0, 1.0, -2.0), Vec4::new(0.0, 0.0, 0.0, 0.0)); + }; + let ty = (p.fov * 0.5).tan(); + let tx = ty * p.aspect_ratio; + let rot = transform.affine().matrix3; + // View dir = right·x·tx + up·y·ty − back; only its world Y matters. + let row = Vec3::new(rot.x_axis.y * tx, rot.y_axis.y * ty, -rot.z_axis.y); + (row.extend(-0.01), Vec4::new(tx, ty, 0.0, 0.0)) +} + fn smoothstep(a: f32, b: f32, value: f32) -> f32 { let t = ((value - a) / (b - a)).clamp(0.0, 1.0); t * t * (3.0 - 2.0 * t) @@ -197,13 +231,17 @@ fn init_pipeline( let mask_layouts = [false, true].map(|multisampled| { BindGroupLayoutDescriptor::new( "post_sun_shafts_mask_layout", - &BindGroupLayoutEntries::single( + // MONKEY (leftovers): + the view uniform, for the horizon row. + &BindGroupLayoutEntries::sequential( ShaderStages::FRAGMENT, - if multisampled { - texture_depth_2d_multisampled() - } else { - texture_depth_2d() - }, + ( + if multisampled { + texture_depth_2d_multisampled() + } else { + texture_depth_2d() + }, + uniform_buffer::(false), + ), ), ) }); @@ -302,7 +340,12 @@ fn prepare_masks( let Some(size) = camera.physical_viewport_size else { continue; }; - let rows = [shaft.sun.to_array(), shaft.color.to_array()]; + let rows = [ + shaft.sun.to_array(), + shaft.color.to_array(), + shaft.horizon.to_array(), + shaft.ray.to_array(), + ]; match uniform { Some(uniform) => queue.write_buffer(&uniform.0, 0, bytemuck::cast_slice(&rows)), None => { @@ -376,7 +419,7 @@ impl ViewNode for ShaftNode { let mask_bind = device.create_bind_group( "post_sun_shafts_mask", &cache.get_bind_group_layout(&settings.mask_layouts[multisampled]), - &BindGroupEntries::single(depth.view()), + &BindGroupEntries::sequential((depth.view(), uniform.0.as_entire_binding())), ); let out = target.post_process_write(); let bind = device.create_bind_group( @@ -444,4 +487,29 @@ mod tests { assert_eq!(smoothstep(0.0, 1.0, 2.0), 1.0); assert!((smoothstep(0.0, 1.0, 0.5) - 0.5).abs() < 1.0e-6); } + + /// MONKEY (leftovers): the shader's horizon form, evaluated on the CPU, is the view ray's + /// world-up sine: the screen centre of a level camera sits on the horizon, a pitched-down one + /// sees below it, and the top/bottom edges sit at ±half the vertical FOV. + #[test] + fn horizon_row_is_the_view_ray_elevation() { + let p = PerspectiveProjection { + fov: 1.0, + aspect_ratio: 16.0 / 9.0, + ..default() + }; + let proj = Projection::Perspective(p.clone()); + let up = |t: &Transform, ndc: Vec2| { + let (h, r) = horizon_row(&GlobalTransform::from(*t), &proj); + h.truncate().dot(ndc.extend(1.0)) / (ndc * r.truncate().truncate()).extend(1.0).length() + }; + let level = Transform::default().looking_to(Vec3::X, Vec3::Y); + assert!(up(&level, Vec2::ZERO).abs() < 1e-5); + assert!((up(&level, Vec2::new(0.0, 1.0)) - 0.5f32.sin()).abs() < 1e-4); + assert!((up(&level, Vec2::new(0.0, -1.0)) + 0.5f32.sin()).abs() < 1e-4); + let down = Transform::default().looking_to(Vec3::new(1.0, -0.3, 0.0), Vec3::Y); + let dir = Vec3::new(1.0, -0.3, 0.0).normalize(); + assert!((up(&down, Vec2::ZERO) - dir.y).abs() < 1e-5); + assert!(up(&down, Vec2::ZERO) < -0.01, "sea below the horizon is not sky"); + } } diff --git a/crates/benilla-app/src/post/sun_shafts.wgsl b/crates/benilla-app/src/post/sun_shafts.wgsl index f7dec61f..845a3a88 100644 --- a/crates/benilla-app/src/post/sun_shafts.wgsl +++ b/crates/benilla-app/src/post/sun_shafts.wgsl @@ -4,6 +4,14 @@ #import bevy_core_pipeline::fullscreen_vertex_shader::FullscreenVertexOutput +struct Shafts { + sun: vec4, + color: vec4, + // MONKEY (leftovers): world-up of the view ray in NDC (xyz) and the sky threshold (w). + horizon: vec4, + ray: vec4, +}; + #ifdef SHAFT_MASK #ifdef MULTISAMPLED @@ -11,6 +19,7 @@ #else @group(0) @binding(0) var depth: texture_depth_2d; #endif +@group(0) @binding(1) var shafts: Shafts; @fragment fn fs_mask(in: FullscreenVertexOutput) -> @location(0) vec4 { @@ -19,7 +28,12 @@ fn fs_mask(in: FullscreenVertexOutput) -> @location(0) vec4 { // MONKEY (fix-post): one sample, not a max over all of them: the march is a blur. // Only the cleared / sky-pinned depth (exactly 0 on infinite reverse-Z) is sky; a ramp on // `near / z` would move with the live `nearclip` cvar and pour shafts through far terrain. - let sky = select(0.0, 1.0, textureLoad(depth, p, 0) <= 1.0e-7); + // MONKEY (leftovers): water writes no depth, so depth 0 under the horizon is the sea (or the + // far-clipped ground), never sky. + let ndc = (vec2(p) + 0.5) / vec2(dims) * vec2(2.0, -2.0) + vec2(-1.0, 1.0); + let up = dot(shafts.horizon.xyz, vec3(ndc, 1.0)) + / length(vec3(ndc * shafts.ray.xy, 1.0)); + let sky = select(0.0, 1.0, textureLoad(depth, p, 0) <= 1.0e-7 && up > shafts.horizon.w); return vec4(sky, 0.0, 0.0, 1.0); } @@ -28,11 +42,6 @@ fn fs_mask(in: FullscreenVertexOutput) -> @location(0) vec4 { @group(0) @binding(0) var scene: texture_2d; @group(0) @binding(1) var scene_sampler: sampler; @group(0) @binding(2) var mask: texture_2d; - -struct Shafts { - sun: vec4, - color: vec4, -}; @group(0) @binding(3) var shafts: Shafts; @fragment diff --git a/crates/benilla-app/src/world_shadow.rs b/crates/benilla-app/src/world_shadow.rs index e04aa5a4..8ab4a256 100644 --- a/crates/benilla-app/src/world_shadow.rs +++ b/crates/benilla-app/src/world_shadow.rs @@ -84,13 +84,14 @@ fn refresh_streamed_shadows( /// `prepass_fragment_shader` makes the otherwise depth-only directional-shadow pass run a fragment — /// the exact pair Bevy's prepass specializer requires to alpha-test in the shadow map. #[derive(Asset, AsBindGroup, TypePath, Debug, Clone)] -struct CutoutShadowCasterMaterial { +// MONKEY (leftovers): crate-visible for the pipe_warm menagerie. +pub(crate) struct CutoutShadowCasterMaterial { /// The leaf sheet, bound at material binding 0/1. Reusing the SAME image handle the forward pass /// draws makes the sampler inherit that image's clamp/repeat address mode, so the shadow /// silhouette matches the visible foliage (a cutout card's out-of-range UVs must clamp). #[texture(0)] #[sampler(1)] - leaf: Handle, + pub(crate) leaf: Handle, } impl Material for CutoutShadowCasterMaterial { diff --git a/crates/benilla-assets/src/shaders/wind_hook.wgsl b/crates/benilla-assets/src/shaders/wind_hook.wgsl index 28e60f68..0b61c428 100644 --- a/crates/benilla-assets/src/shaders/wind_hook.wgsl +++ b/crates/benilla-assets/src/shaders/wind_hook.wgsl @@ -21,6 +21,14 @@ fn wind_base_dir(frame: MonkeyFrame) -> vec2 { return vec2(cos(frame.wind_a.z), sin(frame.wind_a.z)); } +// MONKEY (fix-wind): gusts rolling across a field: a 55 yd band travelling downwind (68 cycles +// per wrap, about 8 yd/s at the base speed) that swells the sway to 1.2x and lulls it to 0.55x. +fn wind_gust_roll(frame: MonkeyFrame, xz: vec2) -> f32 { + let band = sin(dot(wind_base_dir(frame), xz) * (WIND_TAU / 55.0) + - wind_travel_phase(frame.wind_b.x, 68.0)); + return mix(0.55, 1.2, 0.5 + 0.5 * band); +} + fn wind_cross_dir(dir: vec2) -> vec2 { // WarcraftXL's secondary wave is 35 degrees off the primary. return vec2( @@ -51,19 +59,20 @@ fn grass_offset( let base_secondary = wind_cross_dir(base_primary); let wave_primary = WIND_TAU / 18.0; let wave_secondary = WIND_TAU / 6.5; - // WarcraftXL's k * 0.33 * speed (primary) and k * 0.33 * 1.37 * speed (secondary), as whole - // cycles per wrap: 75 and 285. + // MONKEY (fix-wind): visible tuning. At the 9 yd/s base speed the primary wave runs a 2.5 s + // period (182 cycles per wrap) and the cross flutter 1.1 s (410); both speed up with gusts. let phase_primary = wave_primary * dot(base_primary, world.xz) - - wind_travel_phase(frame.wind_b.x, 75.0) + tuft_phase; + - wind_travel_phase(frame.wind_b.x, 182.0) + tuft_phase; let phase_secondary = wave_secondary * dot(base_secondary, world.xz) - - wind_travel_phase(frame.wind_b.x, 285.0) + tuft_phase * 1.7; + - wind_travel_phase(frame.wind_b.x, 410.0) + tuft_phase * 1.7; let gust = 1.0 + 0.5 * (frame.wind_a.w - 0.5) * 2.0; let variance = mix(0.7, 1.3, fract(sin(tuft_phase * 12.9898) * 43758.5453)); let distance_fade = 1.0 / (1.0 + distance(camera, world) * 0.015); var xz = ( - dir_primary * 0.060 * (sin(phase_primary) + 0.35) - + dir_secondary * 0.020 * sin(phase_secondary) - ) * gust * variance * weight * distance_fade * frame.wind_b.y * frame.wind_b.z; + dir_primary * 0.18 * (sin(phase_primary) + 0.35) + + dir_secondary * 0.06 * sin(phase_secondary) + ) * gust * variance * wind_gust_roll(frame, world.xz) * weight * distance_fade + * frame.wind_b.y * frame.wind_b.z; let count = min(u32(frame.misc.x), 8u); for (var i = 0u; i < 8u; i += 1u) { @@ -77,7 +86,8 @@ fn grass_offset( xz += radial * 0.50 * edge * cone * weight * frame.wind_b.z; } } - return vec3(xz.x, 0.0, xz.y); + // MONKEY (fix-wind): a bent tip dips, so a blade reads as bending rather than stretching. + return vec3(xz.x, -min(0.6 * dot(xz, xz), 0.25), xz.y); } // MONKEY (wind): our tree/bush design (WarcraftXL has no tree wind). Only leaf-card batches call @@ -95,26 +105,29 @@ fn tree_offset( // A one-yard bush still needs a crown; tree leaf cards generally begin above this range and // therefore reach full weight, while the first 0.15 yd remains a planted base. let crown = smoothstep(0.15, 1.5, h); - let weight = crown * crown; + // MONKEY (fix-wind): a taller crown swings further (lever arm): 0.43x at 1.5 yd, 1x at 6 yd, + // capped at 1.3x from 8.5 yd, so a bush nods while a canopy sways. + let weight = crown * crown * clamp(0.25 + h * 0.12, 0.25, 1.3); let dir = normalize(frame.wind_a.xy + vec2(1.0e-6, 0.0)); let cross = vec2(-dir.y, dir.x); let base = wind_base_dir(frame); let base_cross = vec2(-base.y, base.x); let phase = fract(sin(dot(anchor.xz, vec2(0.173, 0.317))) * 43758.5453) * WIND_TAU; - // 0.08 rad/yd sway, 0.61x side wave and a 2.3 rad/s-at-base-speed flutter, as whole cycles - // per wrap of travel: 52, 32 and 167. + // MONKEY (fix-wind): visible tuning. At the 9 yd/s base speed: a 4 s crown sway (114 cycles + // per wrap), a 5.5 s side wave (82) and a 0.8 s leaf flutter (569). let slow = sin(dot(base, world.xz) * (WIND_TAU / 34.0) - - wind_travel_phase(frame.wind_b.x, 52.0) + phase); + - wind_travel_phase(frame.wind_b.x, 114.0) + phase); let side = sin(dot(base_cross, world.xz) * (WIND_TAU / 21.0) - - wind_travel_phase(frame.wind_b.x, 32.0) + phase * 1.7); - let flutter = sin(wind_travel_phase(frame.wind_b.x, 167.0) + - wind_travel_phase(frame.wind_b.x, 82.0) + phase * 1.7); + let flutter = sin(wind_travel_phase(frame.wind_b.x, 569.0) + dot(world.xz, vec2(1.7, 2.1)) + phase * 2.0); let gust = 0.75 + frame.wind_a.w * 0.5; let fade = 1.0 / (1.0 + distance(camera, world) * 0.006); let xz = ( - dir * (0.045 + slow * 0.055) - + cross * side * 0.025 - + dir * flutter * 0.012 - ) * weight * gust * fade * frame.wind_b.y * frame.wind_b.w; - return vec3(xz.x, 0.0, xz.y); + dir * (0.08 + slow * 0.20) + + cross * side * 0.08 + + dir * flutter * 0.03 + ) * weight * gust * mix(0.75, 1.05, (wind_gust_roll(frame, anchor.xz) - 0.55) / 0.65) + * fade * frame.wind_b.y * frame.wind_b.w; + return vec3(xz.x, -0.15 * dot(xz, xz), xz.y); } diff --git a/crates/benilla-formats/src/lib.rs b/crates/benilla-formats/src/lib.rs index c47b7f3b..39c7d54d 100644 --- a/crates/benilla-formats/src/lib.rs +++ b/crates/benilla-formats/src/lib.rs @@ -135,7 +135,11 @@ pub use light::{Atmosphere, LightCatalog, Submersion, ZERO_KEY_COLOR, ZERO_KEY_S // MONKEY (fog): `LightFogBand.dbc`. pub use light::{FogBand, FogBandCatalog, FOG_BANDS_PER_PARAM}; // MONKEY (skybox): the skybox row and the zone walk's entries. -pub use light::{SkyboxDef, ZoneSkybox, SKYBOX_FOG_BLEND, SKYBOX_FULL_DAY, SKYBOX_KEEP_CELESTIAL}; +// MONKEY (leftovers): + the 0x8 / 0x10 flags. +pub use light::{ + SkyboxDef, ZoneSkybox, SKYBOX_FOG_BLEND, SKYBOX_FORCE_SUN_SHAFTS, SKYBOX_FULL_DAY, + SKYBOX_KEEP_CELESTIAL, SKYBOX_NO_SUN_FOG, +}; mod loading_screen; pub use loading_screen::{load_loading_screens, LoadingScreenCatalog}; mod liquid; diff --git a/crates/benilla-formats/src/light.rs b/crates/benilla-formats/src/light.rs index c9a016f5..b07c4be6 100644 --- a/crates/benilla-formats/src/light.rs +++ b/crates/benilla-formats/src/light.rs @@ -19,7 +19,11 @@ mod tables; pub use atmosphere::Atmosphere; pub use fog_band::{FogBand, FogBandCatalog, FOG_BANDS_PER_PARAM}; -pub use skybox::{SkyboxDef, ZoneSkybox, SKYBOX_FOG_BLEND, SKYBOX_FULL_DAY, SKYBOX_KEEP_CELESTIAL}; +// MONKEY (leftovers): + the 0x8 / 0x10 flags. +pub use skybox::{ + SkyboxDef, ZoneSkybox, SKYBOX_FOG_BLEND, SKYBOX_FORCE_SUN_SHAFTS, SKYBOX_FULL_DAY, + SKYBOX_KEEP_CELESTIAL, SKYBOX_NO_SUN_FOG, +}; use atmosphere::{ FB_CLOUD_DENSITY, FB_FOG_END, FB_FOG_START_MULT, IB_AMBIENT, IB_CLOUD_GBASE, IB_CLOUD_SLOPE, IB_CLOUD_SUN, IB_DIFFUSE, IB_FOG_COLOR, IB_OCEAN_DEEP, IB_OCEAN_SHALLOW, IB_RIVER_DEEP, diff --git a/crates/benilla-formats/src/light/skybox.rs b/crates/benilla-formats/src/light/skybox.rs index ab6c319a..cca12efd 100644 --- a/crates/benilla-formats/src/light/skybox.rs +++ b/crates/benilla-formats/src/light/skybox.rs @@ -26,6 +26,10 @@ pub const SKYBOX_FULL_DAY: u32 = 0x1; pub const SKYBOX_KEEP_CELESTIAL: u32 = 0x2; /// Flag `0x4`: an alpha cone in the final fog colour draws over the model's horizon. pub const SKYBOX_FOG_BLEND: u32 = 0x4; +/// MONKEY (leftovers): flag `0x8`: sun shafts run under this sky even where they would fade. +pub const SKYBOX_FORCE_SUN_SHAFTS: u32 = 0x8; +/// MONKEY (leftovers): flag `0x10`: the fog's sun-colour lobe is off under this sky. +pub const SKYBOX_NO_SUN_FOG: u32 = 0x10; /// One `LightSkybox.dbc` row. #[derive(Clone, Debug, PartialEq, Eq)] diff --git a/crates/benilla-world/src/lighting/daylight/census.rs b/crates/benilla-world/src/lighting/daylight/census.rs index 71101c25..63481c1b 100644 --- a/crates/benilla-world/src/lighting/daylight/census.rs +++ b/crates/benilla-world/src/lighting/daylight/census.rs @@ -306,10 +306,18 @@ fn lamp_terrain_occlusion_scan() { .find_map(|c| benilla_formats::terrain_height_at(c, [x, y, 1.0e4])) }; let mut rows = Vec::new(); + // MONKEY (leftovers): optional 6th field = light height above the doodad origin (default the + // lamppost's 3.5 yd; a campfire's flame sits near 1 yd). + let lift = p.get(5).and_then(|v| v.parse::().ok()).unwrap_or(3.5); + // 7th field = how far the ground must rise above the ray to count (default 5 cm; the receiver's + // normal offset is 0.15 yd, so a grazing 5 cm crest is below what a cube map can resolve). + let clear = p.get(6).and_then(|v| v.parse::().ok()).unwrap_or(0.05); for l in lamps { - let head = Vec3::new(l[0], l[1], l[2] + 3.5); + let head = Vec3::new(l[0], l[1], l[2] + lift); let (mut total, mut blocked) = (0u32, 0u32); let mut best = (0.0f32, 0.0f32); + // MONKEY (leftovers): blocked samples inside 15 yd, where a fire's pool is still visible. + let mut near = 0u32; for ring in 1..=10 { let d = ring as f32 * 2.5; for k in 0..36 { @@ -321,22 +329,23 @@ fn lamp_terrain_occlusion_scan() { let steps = (d / 0.5) as i32; let hit = (1..steps).any(|s| { let q = head.lerp(target, s as f32 / steps as f32); - h(q.x, q.y).is_some_and(|g| g > q.z + 0.05) + h(q.x, q.y).is_some_and(|g| g > q.z + clear) }); if hit { blocked += 1; best = (a.to_degrees(), d); + near += (d <= 15.0) as u32; } } } if total > 0 { - rows.push((blocked as f32 / total as f32, l, best)); + rows.push((blocked as f32 / total as f32, l, best, near, h(l[0], l[1]).unwrap_or(f32::NAN))); } } - rows.sort_by(|a, b| b.0.total_cmp(&a.0)); - for (f, l, (a, d)) in rows.iter().take(12) { + rows.sort_by(|a, b| b.3.cmp(&a.3).then(b.0.total_cmp(&a.0))); + for (f, l, (a, d), near, g) in rows.iter().take(12) { println!( - "lamp ({:8.1},{:7.1},{:6.1}) terrain-blocked {:5.1}% e.g. bearing {a:5.0} deg at {d:4.1} yd", + "lamp ({:8.1},{:7.1},{:6.1}) ground {g:6.2} terrain-blocked {:5.1}% ({near} within 15 yd) e.g. bearing {a:5.0} deg at {d:4.1} yd", l[0], l[1], l[2], f * 100.0 ); } diff --git a/crates/benilla-world/src/lighting/fog_model.rs b/crates/benilla-world/src/lighting/fog_model.rs index 22a0d21b..4ff8249d 100644 --- a/crates/benilla-world/src/lighting/fog_model.rs +++ b/crates/benilla-world/src/lighting/fog_model.rs @@ -204,6 +204,8 @@ pub(super) fn update_fog_model( eye_liquid: crate::liquid::EyeLiquid, viewer: Res, weather: Option>, + // MONKEY (leftovers): LightSkybox flag 0x10 (last frame's backdrop; the skybox resolves later). + sky: Option>, mut frame: ResMut, ) { let derived = derived_band(&lighting); @@ -233,6 +235,9 @@ pub(super) fn update_fog_model( }; let mut next = *frame; fill_frame(&mut next, setting.0, &lighting, &band); + // MONKEY (leftovers): a `0x10` skybox fades the sun-fog lobe out by its weight. + let no_sun = sky.map_or(0.0, |s| s.flag_weight(benilla_formats::SKYBOX_NO_SUN_FOG)); + next.sun_fog_strength *= 1.0 - no_sun.clamp(0.0, 1.0); if *frame != next { *frame = next; } diff --git a/crates/benilla-world/src/skybox.rs b/crates/benilla-world/src/skybox.rs index 9df4635a..3029dc01 100644 --- a/crates/benilla-world/src/skybox.rs +++ b/crates/benilla-world/src/skybox.rs @@ -92,6 +92,15 @@ impl CameraSkybox { }) } + /// MONKEY (leftovers): the heaviest main layer's weight among those carrying `flag`, 0 if none. + pub fn flag_weight(&self, flag: u32) -> f32 { + self.0 + .iter() + .filter(|l| !l.celestial && l.flags & flag != 0) + .map(|l| l.weight) + .fold(0.0, f32::max) + } + fn layer(&self, path: &str) -> Option<&SkyboxLayer> { self.0.iter().find(|l| l.path == path) } @@ -1084,4 +1093,23 @@ mod tests { }); assert_eq!(sky.primary().map(|l| l.path.as_str()), Some("b")); } + + /// MONKEY (leftovers): a flag's weight is its heaviest main carrier; celestial layers and + /// layers without the flag do not count. + #[test] + fn flag_weight_is_the_heaviest_main_carrier() { + let with = |path, w, flags| SkyboxLayer { + flags, + ..layer(path, w) + }; + let mut sky = CameraSkybox(vec![with("a", 0.3, 0x8), with("b", 0.7, 0x1), with("c", 0.5, 0x18)]); + sky.0.push(SkyboxLayer { + celestial: true, + ..with("d", 1.0, 0x18) + }); + assert_eq!(sky.flag_weight(benilla_formats::SKYBOX_FORCE_SUN_SHAFTS), 0.5); + assert_eq!(sky.flag_weight(benilla_formats::SKYBOX_NO_SUN_FOG), 0.5); + assert_eq!(sky.flag_weight(benilla_formats::SKYBOX_FOG_BLEND), 0.0); + assert_eq!(CameraSkybox::default().flag_weight(0x8), 0.0); + } } diff --git a/crates/benilla-world/src/wind/mod.rs b/crates/benilla-world/src/wind/mod.rs index 668e0b5f..1f88610c 100644 --- a/crates/benilla-world/src/wind/mod.rs +++ b/crates/benilla-world/src/wind/mod.rs @@ -91,6 +91,37 @@ impl FoliageWind { } } +/// MONKEY (fix-wind): the player's foliage sway strength (`foliageWindStrength`), a plain gain on +/// the sway amplitude: 0.25..3, 1 = the shipped tuning. Parting around units ignores it. +#[derive(Resource, Clone, Copy, Debug, PartialEq)] +pub struct FoliageWindStrength(pub f32); + +impl FoliageWindStrength { + pub const MIN: f32 = 0.25; + pub const MAX: f32 = 3.0; + /// The registered default. + pub const REGISTERED: f32 = 1.0; + + /// A CVar value into the slider's range (non-finite reads as the default). + pub fn from_cvar(v: f32) -> Self { + Self(if v.is_finite() { + v.clamp(Self::MIN, Self::MAX) + } else { + Self::REGISTERED + }) + } +} + +impl Default for FoliageWindStrength { + fn default() -> Self { + Self(Self::REGISTERED) + } +} + +/// MONKEY (fix-wind): a storm swings the foliage harder, up to +60 % at a full storm, on top of the +/// faster travel the storm's wind speed already gives. +pub const STORM_SWAY_GAIN: f32 = 0.6; + impl Default for FoliageWind { fn default() -> Self { // The resource exists before the CVar host. Its ordinary default therefore has to equal @@ -186,11 +217,24 @@ fn update_wind( time: Res