Direct3D12 sample doesn't render properly in release builds

[cx20]

Summary

Overview

This issue is regarding the DirectX12 triangle sample located at:
crates/samples/windows/direct3d12/src/main.rs

Issue Description

The triangle sample shows different behavior between debug and release builds:

• Debug build: Triangle renders correctly
• Release build: Triangle doesn't render when the rendering logic relies on WM_PAINT messages

Reproduction Steps

1. Clone the repository
2. Build and run the DirectX12 triangle sample in debug mode

cargo run

• Triangle renders correctly

3. Build and run in release mode

cargo run --release

• The window appears but remains blank (white background) with no triangle visible

Root Cause

The current implementation relies on WM_PAINT messages for rendering:

fn sample_wndproc<S: DXSample>(sample: &mut S, message: u32, wparam: WPARAM) -> bool {
 match message {
     WM_PAINT => {
         sample.update();
         sample.render();
         true
     }
     ...
 }
}

This approach may not trigger frequently enough in release builds due to optimization, leading to inconsistent rendering behavior.

Suggested Fix

Implement continuous rendering in the main message loop instead of relying on WM_PAINT:

loop {
    let mut message = MSG::default();
    if unsafe { PeekMessageA(&mut message, None, 0, 0, PM_REMOVE) }.into() {
        unsafe {
            _ = TranslateMessage(&message);
            DispatchMessageA(&message);
        }
        if message.message == WM_QUIT {
            break;
        }
    } else {
        // Render when no messages are available
        sample.update();
        sample.render();
    }
}

Environment

Rust Version: rustc 1.85.0-nightly (dd84b7d5e 2024-12-27)
Windows Version: Windows 11 Version 24H2 (OS Build 26100.2605)
Graphics Card: NVIDIA GeForce RTX 2060
Driver Version: GeForce Game Ready Driver 555.97

Crate manifest

https://github.com/microsoft/windows-rs/blob/master/crates/samples/windows/direct3d12/Cargo.toml

Crate code

https://github.com/microsoft/windows-rs/blob/master/crates/samples/windows/direct3d12/src/main.rs

[kennykerr]

Good catch - that sample can definitely be simplified.

[damyanp]

The sample follows the same strategy as the C++ version (https://github.com/microsoft/DirectX-Graphics-Samples/blob/master/Samples/Desktop/D3D12HelloWorld/src/HelloTriangle/Win32Application.cpp#L104). I've not heard of any issues with that one. It'd be good to try and understand why there's a difference in behavior here before making any changes.

[kennykerr]

I tested and sure enough --release builds don't render properly - I haven't looked beyond that.

[riverar]

The WM_CREATE arm in the window message handler is being optimized away in release mode, which is surprising since the SetWindowLongPtrA call ultimately interacts with an extern function. Could this be a Rust compiler bug? Or are we confusing the compiler with our codegen or cfg attributes? (cc @ChrisDenton)

Note making the pointer read volatile or using the return value of SetWindowLongPtrA resolves the issue, but neither approach feels ideal in this case.

[ChrisDenton]

I'll look into it when I have a moment but from the description it sounds suspiciously like there's UB lurking somewhere in the unsafe code.

[ChrisDenton]

I looked at the code 🎉. Thanks to @riverar for narrowing it down, I think I identified the cause as being the transmute, which creates a pointer without provenance. Hopefully #3419 should fix this, if you'd like to test it.

[riverar]

I saw that too but it doesn't quite explain why the compiler suddenly optimizes the entire arm out. Something still seems amiss.

[ChrisDenton]

Dereferencing a pointer without provenance is UB. Any branch that reaches UB "can't happen" and so LLVM is within its right to optimize out unreachable branches.

[kennykerr]

It certainly fixes the problem. 😉

[riverar]

Printing the return value of the SetWindowLongPtrA call also works. 🙃

I'm OK with the change, since transmuting integers to pointers for non-zero sized reads is explicitly documented as undefined behavior, but it feels like there's something else lurking here.

[ChrisDenton]

I think the UB fully explains it. It's just that println, volatile, etc are very good ways to persuade LLVM not to optimize something out.