Skip to content

--max-tokens silently dropped on audio path - VibeVoice long-form transcripts truncate at backend default #59

Description

@mzau

Observed

Transcribing a ~30 min audio file with VibeVoice-ASR-8bit produces a diarized transcript that ends mid-word inside a segment at ~20.5 min. Whisper transcribes the same file completely. Raising --max-tokens has no effect.

Root cause (two layers)

  1. mlxk: AudioRunner.transcribe() receives the CLI's --max-tokens (run.py:563), but _transcribe_single() never forwards it to generate_transcription() — it is silently dropped. The # Ignored (Whisper generates full transcription) assumption holds for Whisper (chunked decoding, per-chunk generation, no global budget) but not for single-pass models like VibeVoice.
  2. mlx-audio: VibeVoice generate() defaults to max_tokens=8192 while the same module accepts (and trims to) audio up to 59 minutes — the default budget covers only ~20 min of transcript. Internally inconsistent upstream, but out of mlxk's hands.

Constraint analysis

  • The model metadata declares no output-token budget: no generation_config.json; config.json carries only the decoder context window (decoder_config.max_position_embeddings = 131072).
  • mlx-audio exposes no queryable per-model maximumgenerate_transcription() is a **kwargs passthrough with signature filtering; the 8192 is a function-signature default, not a limit.

Fix design

A free, unclamped max_tokens is not robust: an oversized budget combined with a degenerate no-EOS loop runs for hours at 100 % GPU (pseudo-hang). Therefore:

  1. Default: keep the upstream signature default (8192) — mlxk does not invent its own number.
  2. User override: forward --max-tokens in _transcribe_single() (gen_kwargs["max_tokens"] = max_tokens), clamped to the metadata ceiling (context window from config.json). Input-aware clamping (context minus audio prefill) can only happen upstream, since the audio token count is unknown before encoding — mlxk clamps statically as a guardrail.
  3. mlx-audio's signature filter drops the kwarg for models whose generate() doesn't accept it; Whisper neutrality to be verified before landing.

Acceptance

  • mlxk run VibeVoice-ASR-8bit --audio <30min.mp3> --max-tokens 32768 → complete transcript.
  • Whisper path behavior unchanged (chunked decoding, --translate long-form mitigation intact).
  • --max-tokens values above the model's context window are clamped, not forwarded verbatim.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions