Skip to content

chore: strip [default0] prefix from trainer logs#2180

Open
mikasenghaas wants to merge 1 commit intomainfrom
chore/strip-torchrun-log-prefix
Open

chore: strip [default0] prefix from trainer logs#2180
mikasenghaas wants to merge 1 commit intomainfrom
chore/strip-torchrun-log-prefix

Conversation

@mikasenghaas
Copy link
Copy Markdown
Member

@mikasenghaas mikasenghaas commented Apr 2, 2026

Summary

  • Strip the [default0]: prefix that torchrun's --tee flag prepends to every log line (note, that these can still be seen for in the per-rank logs)
  • Uses sed -u in the tail/tee pipelines to remove the prefix in real-time
  • Also sets --role=trainer for clearer per-rank log files in the torchrun log directory
  • Applied to both local launchers (rl.py, sft.py) and SLURM templates

Before:

[default0]:00:19:20    INFO Starting training loop (max_steps=100)
[default0]:00:20:21 SUCCESS Step 0 | Time: 60.98s | Loss: 0.0000 | ...

After:

00:19:20    INFO Starting training loop (max_steps=100)
00:20:21 SUCCESS Step 0 | Time: 60.98s | Loss: 0.0000 | ...

🤖 Generated with Claude Code


Note

Low Risk
Logging-only changes that adjust torchrun args and shell pipelines; main risk is portability/quoting issues when spawning shell=True or running on systems without compatible sed.

Overview
Trainer launch commands now pass torchrun --role=trainer so per-rank log files are labeled more clearly.

For both local (rl.py, sft.py) and multi-node SLURM templates, the displayed/tee’d trainer log stream is piped through sed -u to strip the leading [...] rank prefix (e.g., [default0]:) while keeping the underlying per-rank torchrun logs intact.

Written by Cursor Bugbot for commit 0332018. This will update automatically on new commits. Configure here.

torchrun's --tee flag prepends every log line with [<role><rank>]: (e.g.
[default0]:). This adds noise without value since we already filter to
specific ranks. Strip the prefix with sed -u in the tail/tee pipelines
and set --role=trainer for clearer per-rank log files.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@mikasenghaas mikasenghaas requested a review from samsja April 2, 2026 10:03
@mikasenghaas mikasenghaas marked this pull request as ready for review April 2, 2026 10:04
@mikasenghaas mikasenghaas changed the title Strip [default0] prefix from trainer logs chore: strip [default0] prefix from trainer logs Apr 2, 2026
Copy link
Copy Markdown
Member

@samsja samsja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nah we need this for when we have multi rank trainer writting to the same file. Also I am against doing too many magic beyond tail. The file is the file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants