[wip] Add initial design for Visualizers #3979
Conversation
matthewtrepte
requested review from
ClemensSchwarke,
Mayankm96,
Toni-SM,
hhansen-bdai,
jtigue-bdai,
kellyguo11,
masoudmoghani,
ooctipus,
ossamaAhmed,
pascal-roth,
peterd-NV and
xyao-nv
as code owners
github-actions
bot
added
documentation
Greptile OverviewGreptile SummaryThis PR introduces a visualizer abstraction layer to support multiple rendering backends (Newton OpenGL, Omniverse, Rerun) for Isaac Lab 3.0. The architecture includes a base Key Changes:
Issues Found:
Confidence Score: 2/5
Important Files ChangedFile Analysis
Sequence DiagramsequenceDiagram
participant Script as Training Script
participant SimCtx as SimulationContext
participant Provider as NewtonSceneDataProvider
participant NewtonMgr as NewtonManager
participant Visualizer as NewtonVisualizer
participant Viewer as NewtonViewerGL
Script->>SimCtx: initialize_visualizers()
SimCtx->>Provider: get_scene_data()
Provider->>NewtonMgr: _model, _state_0 (private access)
NewtonMgr-->>Provider: model, state
Provider-->>SimCtx: scene_data dict
SimCtx->>Visualizer: create_visualizer()
SimCtx->>Visualizer: initialize(scene_data)
Visualizer->>Viewer: NewtonViewerGL(cfg)
Visualizer->>Viewer: set_model(model)
Viewer-->>Visualizer: initialized
Visualizer-->>SimCtx: visualizer ready
loop Training Loop
Script->>SimCtx: step_visualizers(dt)
SimCtx->>Visualizer: is_running()?
Visualizer->>Viewer: is_running()
Viewer-->>Visualizer: true
Visualizer-->>SimCtx: true
alt Training Paused
SimCtx->>Visualizer: is_training_paused()?
Visualizer->>Viewer: is_training_paused()
Viewer-->>Visualizer: true
Visualizer-->>SimCtx: true
loop While Paused
SimCtx->>Visualizer: step(0.0, provider)
Visualizer->>Viewer: begin_frame/end_frame
Note over SimCtx,Viewer: BLOCKS simulation thread
end
end
alt Rendering Paused
SimCtx->>Visualizer: is_rendering_paused()?
Visualizer->>Viewer: is_rendering_paused()
Viewer-->>Visualizer: true
Visualizer-->>SimCtx: true
Note over SimCtx: continue (skip step)
end
SimCtx->>Visualizer: step(dt, provider)
Visualizer->>Provider: get_state()
Provider->>NewtonMgr: _state_0 (private access)
NewtonMgr-->>Provider: current state
Provider-->>Visualizer: state
Visualizer->>Viewer: begin_frame(time)
Visualizer->>Viewer: log_state(state)
Visualizer->>Viewer: end_frame()
end
Script->>SimCtx: close_visualizers()
SimCtx->>Visualizer: close()
Visualizer->>Viewer: cleanup
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| # Frame lifecycle is now properly handled by try-finally | ||
| pass # Silently ignore to avoid log spam - the viewer will recover |
There was a problem hiding this comment.
style: Silently suppressing exceptions can hide bugs
The pass statement after the try/finally block will hide any rendering errors. While the comment explains this is to avoid log spam, completely ignoring exceptions can make debugging difficult and hide serious issues.
| # Frame lifecycle is now properly handled by try-finally | |
| pass # Silently ignore to avoid log spam - the viewer will recover | |
| except Exception as e: | |
| # Log at debug level to avoid spam but still track issues | |
| import omni.log | |
| omni.log.debug(f"Rendering error in Newton visualizer: {e}") |
| while visualizer.is_training_paused() and visualizer.is_running(): | ||
| if isinstance(visualizer, NewtonVisualizer): | ||
| # Update state before rendering during pause | ||
| visualizer.update_state(NewtonManager._state_0) | ||
| visualizer.step(0.0) # Step with 0 dt during pause |
There was a problem hiding this comment.
logic: Potential infinite loop if visualizer hangs
The while loop blocks indefinitely if the visualizer is paused. If is_running() never returns False and the user never unpauses, this will hang the entire simulation thread. The inner step(0.0) call could also raise exceptions that aren't caught here.
Consider adding a timeout or making this non-blocking to prevent simulation hangs.
| if visualizer.is_rendering_paused(): | ||
| continue |
There was a problem hiding this comment.
logic: Rendering continues when paused
When rendering is paused via is_rendering_paused(), this continue skips the visualization step but the simulation still runs. However, the visualizer's ImGui state still needs to be updated every frame (as noted in the comment on line 347 of newton_visualizer.py). This could cause the UI to become unresponsive.
The Newton visualizer's step() should probably still be called with a flag to update UI but skip rendering.
| else: | ||
| # Skip unsupported visualizer types for now | ||
| omni.log.warn( | ||
| f"Visualizer type '{type(viz_cfg).__name__}' is not yet implemented. Skipping." | ||
| ) | ||
| continue | ||
|
|
There was a problem hiding this comment.
style: Accessing private class variable directly
The code directly accesses NewtonManager._model and NewtonManager._state_0 which are private class variables (indicated by the leading underscore). This creates tight coupling and could break if the NewtonManager implementation changes.
Consider adding public getter methods in NewtonManager like get_model() and get_state() for better encapsulation.
|
|
||
| # Initialize visualizers after simulation is set up (only on first reset) | ||
| if not soft and not self._visualizers: | ||
| self.initialize_visualizers() |
There was a problem hiding this comment.
style: Check visualizer recreation behavior after cleanup
The condition if not soft and not self._visualizers means visualizers are only initialized if the list is empty. If visualizers were previously created and then closed (making the list empty again), they won't be recreated on the next reset.
Verify this matches the intended behavior for simulation resets.
|
|
||
| from __future__ import annotations | ||
|
|
||
| from .ov_visualizer_cfg import RerunVisualizerCfg |
There was a problem hiding this comment.
syntax: incorrect import - importing RerunVisualizerCfg from ov_visualizer_cfg instead of rerun_visualizer_cfg
| from .ov_visualizer_cfg import RerunVisualizerCfg | |
| from .rerun_visualizer_cfg import RerunVisualizerCfg |
|
|
||
|
|
||
| class RerunVisualizer(Visualizer): | ||
| """Omniverse Visualizer implementation.""" |
There was a problem hiding this comment.
syntax: docstring copy-pasted from OmniverseVisualizer instead of describing Rerun visualizer
| """Omniverse Visualizer implementation.""" | |
| """Rerun Visualizer implementation.""" |
| # | ||
| # SPDX-License-Identifier: BSD-3-Clause | ||
|
|
||
| """Omniverse Visualizer implementation.""" |
There was a problem hiding this comment.
syntax: module docstring copy-pasted from Omniverse instead of describing Rerun
| """Omniverse Visualizer implementation.""" | |
| """Rerun Visualizer implementation.""" |
| class OmniverseVisualizer(Visualizer): | ||
| """Omniverse Visualizer implementation.""" | ||
| def __init__(self, cfg: OVVisualizerCfg): | ||
| super().__init__(cfg) | ||
| # stub for now No newline at end of file |
There was a problem hiding this comment.
logic: stub visualizer missing abstract method implementations - base class requires initialize(), step(), close(), and is_running() to be implemented
| class RerunVisualizer(Visualizer): | ||
| """Omniverse Visualizer implementation.""" | ||
| def __init__(self, cfg: RerunVisualizerCfg): | ||
| super().__init__(cfg) | ||
| # stub for now No newline at end of file |
There was a problem hiding this comment.
logic: stub visualizer missing abstract method implementations - base class requires initialize(), step(), close(), and is_running() to be implemented
matthewtrepte
changed the title
| while visualizer.is_training_paused() and visualizer.is_running(): | ||
| # Step with 0 dt during pause, pass scene provider for state updates | ||
| visualizer.step(0.0, self._scene_provider) |
There was a problem hiding this comment.
logic: This while loop will block the simulation thread indefinitely if visualizer is paused and never unpauses. If is_running() becomes false during pause, the visualizer is removed but simulation remains blocked. The inner step(0.0) call could also raise exceptions that aren't handled.
Add a timeout or make this non-blocking to prevent simulation hangs.
| def update_state(self, state) -> None: | ||
| """Update the simulation state for visualization. | ||
| This method should be called before step() to provide the latest simulation state. | ||
| Args: | ||
| state: The Newton State object to visualize. | ||
| """ | ||
| self._state = state |
There was a problem hiding this comment.
style: Unused method - update_state() is never called. The visualizer pulls state from scene_provider in step() (line 365). Consider removing if not needed.
| self.cfg = cfg | ||
| self._is_initialized = False | ||
| self._is_closed = False |
There was a problem hiding this comment.
style: _is_initialized and _is_closed flags must be manually set by subclasses - error-prone. Consider setting them in base class methods or document this requirement.
1 participant
Description
Initial design to support multiple visualizers with Isaac Lab 3.0
The visualizers
Type of change
Screenshots
Please attach before and after screenshots of the change if applicable.
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there