[BUG] Output error `Failed to process telemetry record: KeyError('sm_utilization')` in application logs when using `--gpu-telemetry pynvml`.

[chenwenxiaolive]

1. start vLLM
2. start aiperf with pynvml:

aiperf profile \
    --model "Qwen/Qwen3-32B" \
    --endpoint-type chat \
    --streaming \
    --url localhost:8000 \
    --input-file traces/conversation_trace_15mins.jsonl \
    --custom-dataset-type mooncake_trace \
    --gpu-telemetry pynvml \
    --artifact-dir "result/tp_qwen3_32b" \

Got errors:

[ajcasagrande]

@chenwenxiaolive What version is this? Latest main or latest from pypi?

[ajcasagrande]

I investigated this and confirmed it is specific to the release/0.8.0 line.

Root cause:

• In release/0.8.0, GPU telemetry time-series storage fixes its metric schema from the first telemetry scrape.
• If the first PyNVML scrape does not include sm_utilization, but a later scrape does, processing the later record raises:
  KeyError('sm_utilization')
• That matches the reported log symptom:
  Failed to process telemetry record: KeyError('sm_utilization')

Validation:

• I tested with a real NVIDIA GPU: NVIDIA RTX 6000 Ada Generation.
• I used the in-repo aiperf-mock-server and ran aiperf profile --gpu-telemetry pynvml.
• A normal PyNVML run on release/0.8.0 did not reproduce naturally on my machine because every PyNVML row included sm_utilization.
• To verify the suspected edge case, I injected a temporary PyNVML shim that makes the first process-utilization query fail, causing the first telemetry row to omit sm_utilization while later rows include it.
• With that shim, release/0.8.0 reproduced the issue exactly:
  • first telemetry row: no sm_utilization
  • later telemetry rows: sm_utilization present
  • logs repeatedly showed Failed to process telemetry record: KeyError('sm_utilization')

I then ran the same shimmed reproduction against:

• release/0.9.0
• origin/main at 83bdcd92fb21e136ade3a22435c1f087240e5c45

Both handled the missing-first / present-later sm_utilization case correctly:

• telemetry export contained one row missing sm_utilization
• later rows contained sm_utilization
• no KeyError
• GPU telemetry summary still rendered SM Utilization (%)

The fix appears to come from the dynamic telemetry schema change introduced in:

68df6ce559 feat(gpu-telemetry): add AMD ROCm telemetry collector via amdsmi (#908)

That change makes telemetry metric arrays dynamic and NaN-backed, so late-arriving metrics are allocated safely instead of indexing into a schema fixed by the first scrape.

Conclusion:

• release/0.9.0 and main are already fixed.
• release/0.8.0 still has the bug and would need a backport of the dynamic telemetry schema handling from src/aiperf/common/models/telemetry_models.py.
• There is also a separate PyNVML issue in release/0.8.0: final telemetry scrape logs 'PyNVMLTelemetryCollector' object has no attribute 'collect_and_process_metrics', which is also fixed in later branches.

Manual install from latest main:

# Install AIPerf from the current main commit tested above
uv pip install 'git+https://github.com/ai-dynamo/aiperf.git@83bdcd92fb21e136ade3a22435c1f087240e5c45'

# Optional: install the in-repo mock server from the same commit
uv pip install 'aiperf-mock-server @ git+https://github.com/ai-dynamo/aiperf.git@83bdcd92fb21e136ade3a22435c1f087240e5c45#subdirectory=tests/aiperf_mock_server'

# Confirm versions / entrypoints resolve
aiperf --help
aiperf-mock-server --help