Skip to content

fix: provide TensorBoard logging dir in finetuning script#295

Open
ElhamDevelopmentStudio wants to merge 1 commit into
QwenLM:mainfrom
ElhamDevelopmentStudio:fix/finetune-tensorboard-logging-dir
Open

fix: provide TensorBoard logging dir in finetuning script#295
ElhamDevelopmentStudio wants to merge 1 commit into
QwenLM:mainfrom
ElhamDevelopmentStudio:fix/finetune-tensorboard-logging-dir

Conversation

@ElhamDevelopmentStudio

Copy link
Copy Markdown

Summary

Fix the TensorBoard logging crash in finetuning/sft_12hz.py by providing Accelerate with a valid log directory.

Fixes #286

Problem

The fine-tuning script initializes Accelerate like this:

Accelerator(
    gradient_accumulation_steps=4,
    mixed_precision="bf16",
    log_with="tensorboard",
)

With accelerate==1.12.0, this raises:

ValueError: Logging with `tensorboard` requires a `logging_dir` to be passed in.

That causes sft_12hz.py to fail before training starts.

Changes

  • Add a new optional CLI argument: --logging_dir
  • Default logging_dir to <output_model_path>/logs when not provided
  • Create the log directory before constructing Accelerator
  • Pass the directory to Accelerator(..., project_dir=logging_dir)

Why This Approach

  • Keeps TensorBoard logging enabled instead of disabling it as a workaround
  • Preserves current script behavior for existing users
  • Gives users an explicit override when they want logs written elsewhere

Example

Existing command still works:

python sft_12hz.py \
  --init_model_path Qwen/Qwen3-TTS-12Hz-1.7B-Base \
  --output_model_path output \
  --train_jsonl train_with_codes.jsonl \
  --batch_size 32 \
  --lr 2e-6 \
  --num_epochs 10 \
  --speaker_name speaker_test

TensorBoard logs will now default to:

output/logs

Optional custom path:

python sft_12hz.py \
  --init_model_path Qwen/Qwen3-TTS-12Hz-1.7B-Base \
  --output_model_path output \
  --train_jsonl train_with_codes.jsonl \
  --logging_dir ./logs \
  --batch_size 32 \
  --lr 2e-6 \
  --num_epochs 10 \
  --speaker_name speaker_test

Validation

Validated in a clean temporary environment with accelerate==1.12.0 and tensorboard:

  • Reproduced the original ValueError without project_dir
  • Confirmed Accelerator(..., project_dir=...) succeeds
  • Confirmed patched sft_12hz.py gets past Accelerator initialization
  • Confirmed default and custom log directories are created as expected
  • Ran python -m py_compile finetuning/sft_12hz.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Finetune: Value Error: Logging with "tensorboard" requires a 'logging_dir' to be passed in

1 participant