WeirdUtils/docs/outline-rendering-research.md

Outline Rendering Research: Techniques for WoW 1.12.1 D3D9 Hook

Context

DLL-injected hook into WoW 1.12.1 (D3D9, Shader Model 3.0, 32-bit). Drawing colored outlines around specific player/NPC models via IDirect3DDevice9 vtable hooks (DrawIndexedPrimitive, EndScene, Reset).

Requirements

Category	Walls occlude?	Other units occlude?	Notes
Alive targets	Yes	No	Outlines show over other units for combat visibility
Dead friendlies	No	No	Visible through everything for finding corpses

Current Approach

3-pass stencil with normal-extrusion vertex shader in the DIP hook:

• Pass 1: Mark body in stencil (no color write)
• Pass 2: Draw enlarged outline via VS that extrudes vertices along normals
• Pass 3: Restore state, draw normal model on top

Batch reordering (model_hook.zig) moves outline targets to render first in the M2 batch list so only terrain+WMO depth exists at outline time.

Current Approach Problems

• Outline thickness varies with mesh geometry (sharp edges get thinner outlines)
• Gaps at separate body parts (WoW characters have separate meshes for armor, capes, etc.)
• Concave silhouettes produce artifacts
• Requires batch reordering to get the right depth state

---

How Shipped Games Do It

League of Legends (Riot Games)

Uses depth-buffer + Sobel edge detection tightly integrated into the rendering pipeline:

• During the skinned mesh rendering pass, shaders write scaled depth to a secondary buffer via Multiple Render Targets (MRT).
• Outlines are produced by running a Sobel filter on that scaled depth buffer. The Sobel filter finds discontinuities in depth corresponding to silhouette edges.
• The detected edge is rendered back over the skinned mesh - done per-mesh individually, not as a single full-screen post-process.
• For GPUs that do not support MRT, there is a fallback using stencil buffers.
• Rendering order places outlines as a dedicated stage between skinned meshes and grass/water in a 13-stage pipeline.

This is notably clean: it piggybacks on the already-required mesh render pass (no extra geometry pass), and the Sobel filter on a per-object depth buffer gives crisp, uniform-width outlines without the variable-thickness problem of normal extrusion.

Sources:

• https://technology.riotgames.com/news/trip-down-lol-graphics-pipeline
• https://www.gamedeveloper.com/programming/a-layer-by-layer-breakdown-of-i-league-of-legends-i-rendering-process

Valve Source Engine (Left 4 Dead / DOTA 2 / TF2)

Uses the "L4D Glow Effect" - a stencil + render-to-texture + blur approach. Used across Left 4 Dead, TF2, CS:GO, and DOTA 2 (pre-Source 2):

1. Stencil pass: Draw the entity onto the Stencil Buffer. Creates a "cutout" mask of the entity's silhouette.
2. Color pass: Draw the entity with the desired glow color (flat/constant color) onto a separate Render Target ("GlowBuff1").
3. Blur + composite: Blur GlowBuff1 (using a second RT "GlowBuff2" for ping-pong blur passes), then render the blurred result to the screen while respecting the stencil buffer. The stencil test ensures only the blurred pixels that extend beyond the entity's silhouette are visible, producing a halo/outline effect.

The stencil cutout is the key innovation - it prevents the glow color from appearing inside the character, so you only see the outline fringe.

Sources:

• https://developer.valvesoftware.com/wiki/L4D_Glow_Effect
• https://developer.valvesoftware.com/wiki/L4D_Glow_Effect.cpp
• https://developer.valvesoftware.com/wiki/L4D_Glow_Effect_(2013_SDK)

World of Warcraft (Retail, Warlords of Draenor+)

Has an "Outline Mode" highlighting targeted/moused-over characters with color-coded outlines (green = friendly, red = hostile, yellow = neutral, blue = non-PVP):

• The EffectGlow system uses 4 render targets in a chain: the scene is box-blurred (FFXBox4 shader, 2x2), then Gaussian-blurred twice (FFXGauss4, 4-tap Gaussian), then composited with the original scene (FFXGlow shader).
• Intermediate blur targets are at 1/4 width x 1/4 height of the screen.
• The Gaussian blur uses asymmetric weights: vec2(0.125, 0.375) for center and adjacent samples.

Sources:

• https://wowdev.wiki/Rendering/ScreenEffects
• https://wowdev.wiki/Rendering

Unreal Engine (Fortnite etc.)

Uses Custom Depth + Custom Stencil post-process approach:

• Objects that need outlines write to the Custom Depth buffer (optional secondary depth buffer).
• A Custom Stencil byte per pixel distinguishes different objects/groups for different outline colors.
• A post-process material reads Custom Depth/Stencil buffers and applies edge detection or dilation.
• UE also ships a JFA package for wide outlines.

Sources:

• https://www.michalorzelek.com/blog/tutorial-creating-outline-effect-around-objects/
• https://dev.epicgames.com/community/learning/tutorials/zj3x/unreal-engine-fortnite-overlay-materials-for-outlines-and-other-fx

---

Candidate Techniques (All SM3.0 Compatible)

1. Screen-Space Dilation (Valve/L4D Style)

How it works:

Render each target's silhouette as flat color to an offscreen render target (depth-tested against terrain for alive, no depth for dead). Then run a pixel shader that samples an NxN neighborhood - if any sample is "on", the pixel is outline. Subtract the original mask to get just the ring. Composite over backbuffer.

// SM3.0 pixel shader - fixed-size box dilation
sampler2D SilhouetteTex;
float2 TexelSize; // (1.0/screenW, 1.0/screenH)

float4 DilatePS(float2 uv : TEXCOORD0) : COLOR0
{
    float hit = 0.0;
    for (int y = -3; y <= 3; y++) {
        for (int x = -3; x <= 3; x++) {
            float2 offset = float2(x, y) * TexelSize;
            hit = max(hit, tex2D(SilhouetteTex, uv + offset).r);
        }
    }
    float original = tex2D(SilhouetteTex, uv).r;
    float outline = hit - original;
    return float4(OutlineColor.rgb, outline * OutlineColor.a);
}

For circular (non-square) outlines, add a distance check:

if (length(float2(x, y)) <= OutlineRadius) {
    hit = max(hit, tex2D(SilhouetteTex, uv + offset).r);
}

Pros:

• Perfectly uniform outline width regardless of mesh geometry
• Simple shaders (a single dilation pass is ~49 tex samples for 7x7)
• Proven D3D9 technique (Valve shipped this)

Cons:

• Square corners at large radii (box kernel artifact)
• Cost grows as O(N^2) with outline width - impractical beyond ~8px
• Needs 2-3 render targets

Performance: 49 texture samples per pixel at 1024x768 = ~38M samples. On modern hardware: effectively free (<1ms). On 2004-era hardware: 2-4ms.

Occlusion: All 3 requirements satisfiable. The mask generation controls occlusion; the dilation is purely 2D.

2. Jump Flood Algorithm (JFA)

How it works:

The JFA (Rong & Tan, 2006) computes an approximate 2D distance transform on the GPU using O(log N) pixel shader passes. This is the foundation of high-quality screen-space outlines in modern games.

Step 1 - Seed initialization: Render unit silhouettes into a binary mask. An init shader reads this mask: "on" pixels output their own UV coordinates, "off" pixels get a sentinel value (e.g., (9999, 9999)). Output format: RG16F (two channels for x,y coordinates).

// Seed init PS
float4 SeedInitPS(float2 uv : TEXCOORD0) : COLOR0
{
    float silhouette = tex2D(UnitMask, uv).r;
    if (silhouette > 0.5)
        return float4(uv.x, uv.y, 0, 1);
    else
        return float4(9999, 9999, 0, 0);
}

Step 2 - JFA propagation (iterative): Execute ceil(log2(maxOutlineRadius)) passes. For pass k, step size = 2^(N-k-1) (starts large, halves each pass). Each pixel samples itself and 8 compass neighbors at the step offset (9 total samples in a 3x3 grid with large spacing). Keep the seed coordinate nearest to the current pixel. Ping-pong between two render targets.

// JFA propagation PS
sampler2D CurrentJFA;
float2 TexelSize;
float StepSize;

float4 JFAPassPS(float2 uv : TEXCOORD0) : COLOR0
{
    float2 bestSeed = float2(9999, 9999);
    float bestDist = 1e10;

    for (int dy = -1; dy <= 1; dy++) {
        for (int dx = -1; dx <= 1; dx++) {
            float2 sampleUV = uv + float2(dx, dy) * StepSize * TexelSize;
            float2 candidate = tex2D(CurrentJFA, sampleUV).rg;
            if (candidate.x < 9000.0) {
                float d = length(candidate - uv);
                if (d < bestDist) {
                    bestDist = d;
                    bestSeed = candidate;
                }
            }
        }
    }
    return float4(bestSeed, 0, 1);
}

Step 3 - Distance readout and outline generation: After all passes, each texel holds the UV of the nearest seed. Convert to pixel-space distance and threshold:

float4 OutlinePS(float2 uv : TEXCOORD0) : COLOR0
{
    float2 nearestSeed = tex2D(JFAResult, uv).rg;
    float dist = length((nearestSeed - uv) / TexelSize);
    float inSilhouette = tex2D(SilhouetteMask, uv).r;

    // Hard outline:
    float outline = (dist < OutlineWidth && inSilhouette < 0.5) ? 1.0 : 0.0;

    // Or anti-aliased:
    // float outline = smoothstep(OutlineWidth, OutlineWidth - 1.0, dist) * (1.0 - inSilhouette);

    return float4(OutlineColor.rgb, outline * OutlineColor.a);
}

D3D9/SM3.0 compatibility: Fully compatible. Each pass is a simple pixel shader with 9 texture samples and simple arithmetic. No gather, no integer bitops, no geometry/compute shaders required. Ping-pong between two textures is standard D3D9. The only requirement is that D3D9 does not allow reading and writing the same surface - alternate between two textures each pass.

Pros:

• Exact circular distance field - perfectly round outlines at any width
• Anti-aliasable (smoothstep on the distance)
• Cost is O(log2(N)) passes - a 32px outline costs only 5 passes
• Enables soft glow, pulsing, gradient effects for free (just change threshold function)
• Nothing requires anything beyond SM2.0

Cons:

• More render target switches than dilation (5-10 passes vs 1)
• Needs two RG16F render targets (ping-pong)
• Slightly more implementation complexity
• Cannot use bilinear filtering on JFA output (it stores coordinates, not distances)

Performance: 10 fullscreen passes at 9 samples each = 90M samples at 1024x768. On modern hardware: sub-millisecond. Can run at half resolution (512x384) to halve cost with minimal quality loss for outlines up to 5-6px.

Occlusion: Same as dilation - mask generation is independent of outline generation.

JFA quality: Approximation error bounded at sqrt(2)/2 pixels at jump step boundaries. For outlines up to ~20px, visually imperceptible. Results are smooth, rotationally symmetric, and anti-aliasable.

Sources:

• Rong & Tan, "Jump Flooding in GPU with Applications to Voronoi Diagram and Distance Transform," ACM I3D 2006
• https://www.comp.nus.edu.sg/~tants/jfa.html
• https://bgolus.medium.com/the-quest-for-very-wide-outlines-ba82ed442cd9
• https://gist.github.com/bgolus/a18c1a3fc9af2d73cc19169a809eb195
• https://itscai.us/blog/post/jfa/
• https://blog.demofox.org/2016/02/29/fast-voronoi-diagrams-and-distance-dield-textures-on-the-gpu-with-the-jump-flooding-algorithm/
• https://en.wikipedia.org/wiki/Jump_flooding_algorithm
• https://www.shadertoy.com/view/4syGWK
• https://mini.gmshaders.com/p/gm-shaders-mini-jfa
• RTSDF paper (extends JFA): https://arxiv.org/abs/2210.04449

3. Sobel Edge Detection on ID/Depth Buffer

How it works:

Render each target with a unique ID value into an R8 render target (depth-tested). Run a 3x3 Sobel filter - pixels where neighboring IDs differ are edges.

float4 SobelEdgePS(float2 uv : TEXCOORD0) : COLOR0
{
    float tl = tex2D(IDTex, uv + float2(-1,-1) * TexelSize).r;
    float tc = tex2D(IDTex, uv + float2( 0,-1) * TexelSize).r;
    float tr = tex2D(IDTex, uv + float2( 1,-1) * TexelSize).r;
    float ml = tex2D(IDTex, uv + float2(-1, 0) * TexelSize).r;
    float mr = tex2D(IDTex, uv + float2( 1, 0) * TexelSize).r;
    float bl = tex2D(IDTex, uv + float2(-1, 1) * TexelSize).r;
    float bc = tex2D(IDTex, uv + float2( 0, 1) * TexelSize).r;
    float br = tex2D(IDTex, uv + float2( 1, 1) * TexelSize).r;

    float gx = -tl - 2*ml - bl + tr + 2*mr + br;
    float gy = -tl - 2*tc - tr + bl + 2*bc + br;
    float edge = sqrt(gx*gx + gy*gy);

    float outline = step(0.001, edge);
    return float4(OutlineColor.rgb, outline);
}

Pros:

• Cheapest option (single pass, 9 samples)
• Crisp, accurate silhouette detection
• No variable-thickness artifacts

Cons:

• Produces only 1-2px outlines - can't thicken without adding dilation anyway
• Detects unit-to-unit boundaries too (unwanted internal edges between overlapping characters)
• Alone, not sufficient for controllable-width outlines

Performance: 8 texture samples per pixel. Roughly 0.1-0.2ms at 1024x768. Fastest of all approaches.

Occlusion: Req 1 yes, Req 3 yes, Req 2 partially (spurious edges at unit overlaps).

4. Gaussian Blur Difference (WoW Retail Style)

How it works:

Render silhouette to RT. Separable Gaussian blur (H pass + V pass). Subtract original from blurred → outline. Can downsample to 1/4 res for performance (like retail WoW does for bloom).

// Gaussian blur 5-tap (separable - run horizontal then vertical)
float weights[5] = {0.0625, 0.25, 0.375, 0.25, 0.0625};

float4 GaussianBlurPS(float2 uv : TEXCOORD0) : COLOR0
{
    float result = 0;
    for (int i = -2; i <= 2; i++) {
        float2 offset = float2(i, 0) * TexelSize; // horizontal pass
        result += tex2D(SilhouetteTex, uv + offset).r * weights[i+2];
    }
    return float4(result, result, result, 1);
}

// Outline extraction
float4 OutlineExtractPS(float2 uv : TEXCOORD0) : COLOR0
{
    float blurred = tex2D(BlurredTex, uv).r;
    float original = tex2D(OriginalTex, uv).r;
    float outline = saturate(blurred - original);
    return float4(OutlineColor.rgb, outline * OutlineColor.a);
}

Pros:

• Very cheap with separable blur: O(2N) samples total
• Soft, aesthetically pleasing glow
• Downsampling to 1/4 res makes a 4px kernel act like a 16px outline

Cons:

• Soft/gradient edges, not crisp - looks like a glow, not a hard outline
• Width control is imprecise (tied to blur sigma)
• Can't produce a hard-edged outline without thresholding (which re-introduces aliasing)

Performance: 2 fullscreen passes with 5 samples each = 10 samples total. Very cheap.

Occlusion: Same as dilation - mask generation is independent.

5. Normal Extrusion (Current Approach, Refined)

How it works (fix for variable thickness):

Extrude in clip space instead of object space. Multiply extrusion by pos.w to compensate for perspective division:

float4 pos = mul(WorldViewProj, float4(Position, 1.0));
float2 screenNormal = normalize(mul((float2x2)WorldViewProj, Normal.xy));
pos.xy += screenNormal * OutlineWidth * pos.w;
output.Position = pos;

This makes outline thickness uniform in screen pixels at any depth.

Pros:

• Minimal change to existing code
• No render targets needed
• Works entirely in the DIP hook

Cons:

• Still geometry-dependent at concave silhouettes and mesh part boundaries
• Gaps at separate body parts (WoW characters have separate meshes for armor, capes, etc.)
• Still requires batch reordering for occlusion

---

Comparison Matrix

	Width uniformity	Max width	Quality	Passes	Render targets	Complexity
Dilation	Uniform	~8px	Good (square corners)	1 fullscreen + 1/unit	2-3 RGBA8	Medium
JFA	Uniform, circular	Unlimited	Excellent, AA	log2(R) + 2	2 RG16F + 1 mask	Medium-High
Sobel	1px only	1-2px	Crisp but thin	1 fullscreen + 1/unit	1 R8	Low
Gaussian blur	Soft gradient	~16px (at 1/4 res)	Soft glow	3 fullscreen + 1/unit	3 RGBA8	Medium
Normal extrusion	Non-uniform	~4px	Variable	3/unit in DIP	0	Low

---

Architectural Insight: Two-Phase Separation

All screen-space techniques (1-4) share a two-phase architecture that differs fundamentally from the current normal-extrusion approach:

Phase A - Silhouette mask generation (in DIP hook, per-unit)

• For alive targets: render unit geometry with depth test ON against scene depth → writes to RT_Silhouette
• For dead targets: render with depth test OFF → writes to RT_Dead
• This is where the 3 occlusion requirements are enforced

Phase B - Outline generation + composite (in EndScene, once per frame)

• Dilate/JFA/blur the mask → extract outline ring → alpha-blend over backbuffer
• This is purely 2D, knows nothing about depth

This separation means you can swap the outline renderer (dilation vs JFA vs blur) without touching occlusion logic at all.

The Remaining Hard Problem: Requirement 2

Requirement 2 (other units don't occlude outlines) is the hardest to satisfy. It requires that when generating the silhouette mask for alive targets, the depth test uses a terrain-only depth buffer that excludes other characters. Two approaches:

A. Terrain-only depth buffer (cleanest): Create a secondary D24S8 surface. During DIP, when WoW draws terrain/WMO/game objects, also render a depth-only pass into this secondary buffer. When rendering outline silhouettes, bind this buffer instead of the scene depth.

B. Keep batch reordering (current approach): model_hook.zig already moves outline targets to render first in the M2 batch list. At that point, only terrain depth exists. Simpler but couples outline rendering to draw order.

---

Recommendation

JFA with batch reordering for Req 2.

Rationale:

• Batch reordering already solves Req 2 without needing a secondary depth buffer - outline targets render when only terrain depth exists
• JFA gives the best outline quality (uniform, circular, anti-aliased, any width) at O(log2(N)) cost
• The outline width of 2-3px only needs ~2 JFA passes - nearly free
• Glow/pulse effects come for free if desired
• Everything is SM2.0 compatible, let alone SM3.0
• The silhouette mask pass replaces the current pass 1+2 (stencil body + normal extrusion) with a simpler "render flat color to RT"
• The stencil bit for preventing later units from painting over outlines still works alongside this

Concrete Implementation Plan

Resources to create at device creation/reset:

RT_Silhouette:  RGBA8, screen size - mask for all outline targets
RT_JFA_A:       RG16F, screen size - JFA ping-pong buffer A
RT_JFA_B:       RG16F, screen size - JFA ping-pong buffer B

Hook intercept points:

EndScene (start of hook):
  - Clear RT_Silhouette

DrawIndexedPrimitive (when rendering_outline):
  - Save current RT and DS
  - Bind RT_Silhouette as render target (keep scene depth for alive, unbind for dead)
  - Render unit as flat color (category-colored)
  - Restore original RT and DS
  - Write stencil bits as before (for unit-over-outline prevention)
  - Render normal model (pass 3 equivalent)

EndScene (end of hook, before calling original):
  - Run JFA: init pass → flood passes → decode+composite as fullscreen quad
  - Alpha-blend outline over backbuffer

Critical D3D9 state management for outline passes:

Save before outline pass:
  GetRenderTarget(0, &savedRT0)
  GetDepthStencilSurface(&savedDS)
  GetVertexShader(&savedVS)
  GetPixelShader(&savedPS)
  GetRenderState(D3DRS_ZENABLE, ...)
  GetRenderState(D3DRS_ALPHABLENDENABLE, ...)
  GetRenderState(D3DRS_STENCILENABLE, ...)
  GetRenderState(D3DRS_COLORWRITEENABLE, ...)
  GetRenderState(D3DRS_ZWRITEENABLE, ...)
  GetRenderState(D3DRS_DEPTHBIAS, ...)
  Viewport, stream sources, vertex declaration, index buffer

Restore all after outline composite.

---

Additional References

• "Inking the Cube" (GPU Gems 1, Chapter 11, Everitt) - screen-space dilation
• "Advanced Techniques in Real-Time Rendering" (GDC 2011, de Carpentier) - screen-space outlines
• "Post-Processing Effects in Games" (GDC 2013, Wihlidal) - Sobel ID-buffer approach
• Unreal Engine 4 custom depth/stencil outline documentation
• https://ameye.dev/notes/rendering-outlines/ - "5 Ways to Draw an Outline"
• https://linework.ameye.dev/soft-outline/ - soft outline documentation
• https://www.codeproject.com/Articles/128527/Stencil-Buffer-Glows-Part-1
• https://www.codeproject.com/Articles/156323/Stencil-Buffer-Glows-Part-2
• https://www.tomlooman.com/unreal-engine-soft-outline/
• https://aras-p.info/texts/D3D9GPUHacks.html - D3D9 GPU hacks reference
• https://ameye.dev/notes/edge-detection-outlines/ - edge detection outlines
• https://www.videopoetics.com/tutorials/pixel-perfect-outline-shaders-unity/ - pixel-perfect outlines