Skip to content

Renderer/log-hygiene: per-frame GPU->CPU buffer readback (GL_DYNAMIC_COPY VIDEO->HOST) logged unthrottled every frame across scenes #551

Closed

Description

@drsnuggles8

Summary

While sweeping 26 sandbox scenes over the MCP diagnostics server (Debug editor), olo_log_tail/OloEngine.log in almost every actively-rendering scene is flooded with the same OpenGL driver performance warning, emitted every frame:

[warning] OpenGL performance warning (source: API, id: 131186): Buffer performance warning:
Buffer object 90 (bound to NONE, usage hint is GL_DYNAMIC_COPY) is being copied/moved
from VIDEO memory to HOST memory.

At the editor's frame rate (655 fps observed on the sandbox), that is hundreds of identical lines per second. Two distinct problems:

  1. Log-spam / hygiene (primary, cheap): the GL debug-message callback logs this repeated performance-severity message unthrottled, ballooning OloEngine.log in Debug. This is the same failure mode that bit Renderer: FrameDataBuffer capacity caps (4096 bones / 16384 instances / 1024 materials) overflow ungracefully with unthrottled error spam #524 (126k identical lines stalled the main thread and timed out MCP marshals). The callback should dedup/rate-limit repeated messages (and/or drop GL_DEBUG_SEVERITY performance notifications below error).
  2. Per-frame GPU→CPU buffer readback (secondary): the warning means a GL_DYNAMIC_COPY buffer is migrated VIDEO→HOST every frame because the CPU reads it back (OpenGLStorageBuffer::GetData → glGetNamedBufferSubData, OpenGLStorageBuffer.cpp:51-54, or an editor picking PBO). A per-frame synchronous readback defeats GPU residency; even if small, it is worth eliminating (fence + latent readback, or persistent-mapped ring).

Evidence

  • Observed across particle_test (40+/window), FontRenderingTest, SponzaDeferred (22), WaterTest, FogTest, GodRayUnderwater, PrecipitationTest, SSGITest, etc. — it is renderer-wide, not scene-specific (same buffer id 90 each time).
  • Not a shader/asset problem: olo_shader_errors = 0 and olo_assets_problems = 0 in every swept scene.
  • GL_DYNAMIC_COPY is created in OpenGLStorageBuffer.cpp:91; the only CPU-readback path there is GetData()/glGetNamedBufferSubData (a full sync stall). Candidate producers: the GPU-cull survivor-count readback, the occlusion result readback, or the editor's per-frame entity-ID picking.

Reproduction

Run the Debug editor on any mesh scene (e.g. particle_test, SponzaDeferred) and tail OloEditor/OloEngine.log (or olo_log_tail { minLevel: "warn" }): the id-131186 buffer-migration warning repeats every frame.

Fix direction

  1. Throttle the GL debug callback for repeated / performance-severity messages (dedup by message id + a per-id rate limit). This is a small, self-contained fix and stops the Debug log flood immediately. (Flagged as potentially "fixable now".)
  2. Identify buffer 90's producer (glGetNamedBufferSubData caller / picking PBO) and convert the per-frame sync readback to a fenced/latent readback so the buffer stays GPU-resident.

Note: GL debug output is typically compiled out in Dist, so the log flood is Debug-only; the underlying readback exists in all configs.

Score

capability: 1
craft: 1
stability: 3
decay: 1
effort: 2
confidence: 0.8
learning: 2
fun: 2
kano: performance
blocked_by: []
blocks: []

Rated per issue-scoring · score = confidence × (capability + craft + stability + decay) / effort, derived by the picker.

Activity

  1. added
    performancePerformance / data-oriented optimization
    robustnessProduction hardening / shipping robustness
    on Jul 4, 2026
  2. coderabbitai commented on Jul 4, 2026

    @coderabbitai
    Contributor
    🔗 Related PRs

    #100 - Feature/rendercommand_sorting [closed]
    #465 - Fix/flaky fog visual evidence test [merged]
    #528 - fix(renderer): reset water-depth/planar-reflection texture publications per frame (#505) [merged]
    #529 - fix(renderer): lazily upload deferred font textures so headless-first-touch fonts render (#520) [merged]
    #532 - fix(renderer): resize GTAO scratch textures with the FSR1 scene band [merged]


    🧪 Issue enrichment is currently in open beta.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

  3. drsnuggles8 commented on Jul 5, 2026

    @drsnuggles8
    OwnerAuthor

    Fixed on branch `feature/gl-debug-log-throttle` (commit 2dca73c):

    1. GL debug-callback throttle (`OpenGLDebug.cpp`) - `GL_DEBUG_TYPE_PERFORMANCE` messages are now deduped per message id with exponential backoff (occurrences 1-10 logged individually, then every 10th, 100th, ...), with the running occurrence count included in the log line. Stops the per-frame flood regardless of the underlying cause.

    2. Root-cause fix (`PrecipitationSystem.cpp`/`.h`) - traced the driver's "buffer migrated VIDEO->HOST" warning (id 131186) to `PrecipitationSystem::Update` reading `GPUParticleSystem::GetAliveCount()` (a `glGetNamedBufferSubData` on a `GL_DYNAMIC_COPY` SSBO) every frame, purely to size a compute dispatch group count. `Precipitation_Feed.comp` already self-bounds via the GPU-resident alive count (`if (idx >= aliveCount) return;`), so the CPU no longer needs that value - the dispatch is now sized to fixed buffer capacity instead. The debug-only feed-count stat is still refreshed, just every 30 frames instead of every frame.

    Verified: `OloEngine`/`OloEditor` build clean, editor runs stably for several minutes with no crashes, and the throttle's occurrence-counting was confirmed live against a real GL performance message (id 131218) in `OloEngine.log`.

  4. drsnuggles8 commented on Jul 5, 2026

    @drsnuggles8
    OwnerAuthor

    PR: #560

  5. drsnuggles8 commented on Jul 9, 2026

    @drsnuggles8
    OwnerAuthor

    Follow-up: the other recurring GL debug-callback performance warning that the #560 throttle muted but did not fix — id 131218 "Vertex shader in program N is being recompiled based on GL state" — is now root-caused and fixed in #602.

    Cause was unrelated to the buffer readback tracked here: NVIDIA revalidates the currently bound program against the currently bound framebuffer during glClear, so each pass's leftover program got its vertex shader JIT-recompiled at the next pass's cross-FBO clear. Fixed by scoping every clear in Utils::GLClearProgramGuard (unbind + restore).

    #602 also upgrades the debug callback this issue's fix introduced throttling for: driver messages that reference a program id now resolve to the shader name ([program 31 = 'ShadowDepthSkinned'] via a link-time id→name registry + glObjectLabel), and id-131218 messages capture the native call stack of the GL call they fired under — which is what made the root-cause findable.

    🤖 Generated with Claude Code

  6. drsnuggles8 commented on Aug 21, 2026

    @drsnuggles8
    OwnerAuthor

    Superseded in practice by #874 (issue #721): the general non-stalling alternative this issue's item 2 asked for now exists.

    GPUReadbackStats gives any GPU-driven pass a counter it can atomicAdd into, copied each frame into a fenced device-to-host ring and read a few frames late — IsFenceSignaled is polled, never waited on, and a full ring skips the capture rather than blocking on the oldest slot. A pass that wants to report what it did no longer has to invent a per-frame glGetNamedBufferSubData off a GL_DYNAMIC_COPY buffer, which is what produced the 131186 VIDEO→HOST migration flood recorded here.

    To be precise about scope: this does not retro-fix any existing readback site. It removes the reason to add the next one. If a site named here still does a synchronous read, moving it onto the channel is a small follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingperformancePerformance / data-oriented optimizationrobustnessProduction hardening / shipping robustness

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions