Skip to content

Extending pt-active signal creation to multi-aiu/table traces - #96

Merged
lasch merged 2 commits into
mainfrom
multi-aiu-total-cycles
Apr 20, 2026
Merged

lasch merged 2 commits into
mainfrom
multi-aiu-total-cycles

Conversation

@lasch

@lasch lasch commented Apr 14, 2026

Copy link
Copy Markdown
Member

#93 introduced computation of average pt-utilization based on total ideal cycles. This PR extends this approach to track the necessary status across multi-device runs.

The state tracking was extracted into separate classes and the pt-counter event creation and sanity checking refactored into separate functions. Overlap detection functionality is preserved.

  • extended delayed zero-event creation for overlap detection because:
  • ... accumulation of observed durations in case of graph breaks (e.g. multi-device case with communication)
  • currently: accumulating actual kernel time as opposed to total time between first kernel start and last kernel end (self.accumulate_event_time = True hard-coded but easy to change if necessary)
  • use a 'default' pt-utilization of 101% to indicate problems if computation comes back with >100% (warnings issued as well, pt-statistics would be wrong either way)

lasch added 2 commits April 14, 2026 15:11
…graph-breaks

 * observed time accumulation per rank and per job/token
 * event/counter overlap treatment for multi-aiu

Signed-off-by: Lars Schneidenbach <schneidenbach@us.ibm.com>
Signed-off-by: Lars Schneidenbach <schneidenbach@us.ibm.com>

@yuhaohaoyu yuhaohaoyu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@lasch
lasch merged commit 0751ce4 into main Apr 20, 2026
2 checks passed
@lasch
lasch deleted the multi-aiu-total-cycles branch April 20, 2026 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants