# JFA Outline Banding Investigation

## Date: 2026-02-25

## Problem
Outlines rendered via the JFA pipeline have a "marching ants" pattern of missing outline pixels. The artifact depends on **which side of the camera** the target is on (screen-space position), not the left/right side of the model itself. Camera-angle dependent.

## What Was Ruled Out

### 1. Silhouette RT is clean (CONFIRMED)
Added `DEBUG_SHOW_SILHOUETTE` comptime flag in `d3d9_hook.zig` that skips JFA and composites the raw silhouette RT directly to the backbuffer. The silhouette was solid with no banding - the cached draw replay produces correct geometry. **Stale VB hypothesis is NOT the cause.**

### 2. JFA sentinel value (1.0, 1.0) → (-1.0, -1.0) (NO IMPROVEMENT)
Changed the JFA init shader sentinel from `(1.0, 1.0)` to `(-1.0, -1.0)` so unflooded pixels can't act as false seeds near the right screen edge. This did NOT fix the marching ants pattern. The sentinel is currently set to `(-1.0, -1.0)` in the code (uncommitted).

### 3. JFA pass count changes pattern but doesn't fix it
- `[2, 1]` → vertical bands of missing outline
- `[8, 4, 2, 1]` → marching ants alternating pattern (current uncommitted state)

### 4. Mid-DIP silhouette drawing corrupts render state (ABANDONED)
Attempted drawing silhouettes directly in DIP hook. Corrupts WoW's GxDevice state. Reverted.

### 5. Game object occlusion (FIXED, committed as 661e156)
Batch reorder puts outline targets last so depth buffer has full scene geometry when stencil marks are written.

## Current Uncommitted Changes in d3d9_hook.zig
- JFA steps changed from `[2, 1]` to `[8, 4, 2, 1]`
- JFA sentinel changed from `(1.0, 1.0)` to `(-1.0, -1.0)`
- `DEBUG_SHOW_SILHOUETTE` comptime flag added (currently `false`), with `debug_sil_ps` shader
- Debug shader had a cmp operand inversion bug that was fixed (`c0.w, c0.x` not `c0.x, c0.w`)

## Remaining Investigation - JFA Pipeline Bug

Since the silhouette is clean, the bug is in the JFA shaders (Phase 2). Possible causes:

### A. JFA propagation shader distance comparison logic
The propagation shader uses `cmp` (ps_3_0) which tests `>= 0` vs `< 0`. When `new_dist² == best_dist²` (exactly equal), `cmp` picks the OLD seed (`>= 0` branch). This tie-breaking might cause systematic bias where seeds from certain directions are always preferred, creating directional artifacts. Worth testing: swap `cmp` operands to prefer new seed on ties, or add a small epsilon.

### B. dp2add precision on DXVK
`dp2add r2.z, r2, r2, c1.x` computes `r2.x*r2.x + r2.y*r2.y + 0.0`. DXVK translates this to Vulkan - there may be precision differences vs native D3D9 that affect distance comparisons, especially for pixels equidistant from multiple seeds.

### C. Neighbor sampling at texture edges
When `mad r4.xy, offset, step_uv, v0.xy` goes outside [0,1], CLAMP addressing returns the edge texel. This could feed stale/wrong seed UVs into the comparison. Clamping the sample coordinate to valid range before comparison could help.

### D. The JFA algorithm itself may not suit this use case
Consider alternative approaches:
- **Screen-space dilation** (iterative morphological expand of silhouette) - simpler, no distance field needed
- **Gaussian blur difference** - blur silhouette, subtract original, threshold
- **Sobel/edge detection** on the silhouette RT
- Docs in `docs/` describe these alternatives
- Reference articles: ameye.dev "5 ways to draw an outline", Ben Golus "Quest for Very Wide Outlines"

## Key Files
- `src/outline/d3d9_hook.zig` - D3D9 hooks, JFA pipeline, all shaders
- `src/outline/model_hook.zig` - batch reordering, rendering_outline flag
- `src/outline/tracker.zig` - per-frame model tracking
- `src/outline/types.zig` - D3D9 constants
- `reference/c_overlay/d3d9_hook.cpp` - C reference (uses 3-pass shell extrusion, not JFA)

## Build / Environment
- `zig build` from the repo root
- Zig 0.15, target x86-windows-msvc (32-bit DLL injected into WoW 3.3.5)
- Linux host (Arch, kernel 6.16.1), game via Wine/DXVK
- Git last commit: `661e156` (game object occlusion fix)