Skip to content

[bugfix] fix qwen3_tts #9721

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Jintao-Huang merged 2 commits into from
Jul 10, 2026
Merged

[bugfix] fix qwen3_tts #9721

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bfc02fa
fix qwen3_tts
Jintao-Huang Jul 10, 2026
b259304
fix
Jintao-Huang Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/source/Instruction/Supported-models-and-datasets.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -778,10 +778,10 @@
|[Qwen/Qwen2-Audio-7B](https://modelscope.cn/models/Qwen/Qwen2-Audio-7B)|qwen2_audio|qwen2_audio|hermes|transformers>=4.45,<4.49, librosa|&#x2718;|audio|[Qwen/Qwen2-Audio-7B](https://huggingface.co/Qwen/Qwen2-Audio-7B)|
|[Qwen/Qwen3-ASR-1.7B](https://modelscope.cn/models/Qwen/Qwen3-ASR-1.7B)|qwen3_asr|qwen3_asr|hermes|qwen-asr, transformers==4.57.6|&#x2714;|audio|[Qwen/Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B)|
|[Qwen/Qwen3-ASR-0.6B](https://modelscope.cn/models/Qwen/Qwen3-ASR-0.6B)|qwen3_asr|qwen3_asr|hermes|qwen-asr, transformers==4.57.6|&#x2714;|audio|[Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B)|
|[Qwen/Qwen3-TTS-12Hz-1.7B-Base](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-1.7B-Base)|qwen3_tts|qwen3_tts|hermes|qwen-tts|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base)|
|[Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice)|qwen3_tts|qwen3_tts|hermes|qwen-tts|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice)|
|[Qwen/Qwen3-TTS-12Hz-0.6B-Base](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-0.6B-Base)|qwen3_tts|qwen3_tts|hermes|qwen-tts|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-0.6B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base)|
|[Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice)|qwen3_tts|qwen3_tts|hermes|qwen-tts|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice)|
|[Qwen/Qwen3-TTS-12Hz-1.7B-Base](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-1.7B-Base)|qwen3_tts|qwen3_tts|hermes|qwen-tts, transformers<5|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base)|
|[Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice)|qwen3_tts|qwen3_tts|hermes|qwen-tts, transformers<5|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice)|
|[Qwen/Qwen3-TTS-12Hz-0.6B-Base](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-0.6B-Base)|qwen3_tts|qwen3_tts|hermes|qwen-tts, transformers<5|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-0.6B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base)|
|[Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice)|qwen3_tts|qwen3_tts|hermes|qwen-tts, transformers<5|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice)|
|[Qwen/Qwen3-VL-2B-Instruct](https://modelscope.cn/models/Qwen/Qwen3-VL-2B-Instruct)|qwen3_vl|qwen3_vl|hermes|transformers>=4.57, qwen_vl_utils>=0.0.14, decord|&#x2714;|vision, video|[Qwen/Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct)|
|[Qwen/Qwen3-VL-2B-Thinking](https://modelscope.cn/models/Qwen/Qwen3-VL-2B-Thinking)|qwen3_vl|qwen3_vl|hermes|transformers>=4.57, qwen_vl_utils>=0.0.14, decord|&#x2714;|vision, video|[Qwen/Qwen3-VL-2B-Thinking](https://huggingface.co/Qwen/Qwen3-VL-2B-Thinking)|
|[Qwen/Qwen3-VL-2B-Instruct-FP8](https://modelscope.cn/models/Qwen/Qwen3-VL-2B-Instruct-FP8)|qwen3_vl|qwen3_vl|hermes|transformers>=4.57, qwen_vl_utils>=0.0.14, decord|&#x2714;|vision, video|[Qwen/Qwen3-VL-2B-Instruct-FP8](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct-FP8)|
Expand Down
8 changes: 4 additions & 4 deletions docs/source_en/Instruction/Supported-models-and-datasets.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -779,10 +779,10 @@ The table below introduces the models integrated with ms-swift:
|[Qwen/Qwen2-Audio-7B](https://modelscope.cn/models/Qwen/Qwen2-Audio-7B)|qwen2_audio|qwen2_audio|hermes|transformers>=4.45,<4.49, librosa|&#x2718;|audio|[Qwen/Qwen2-Audio-7B](https://huggingface.co/Qwen/Qwen2-Audio-7B)|
|[Qwen/Qwen3-ASR-1.7B](https://modelscope.cn/models/Qwen/Qwen3-ASR-1.7B)|qwen3_asr|qwen3_asr|hermes|qwen-asr, transformers==4.57.6|&#x2714;|audio|[Qwen/Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B)|
|[Qwen/Qwen3-ASR-0.6B](https://modelscope.cn/models/Qwen/Qwen3-ASR-0.6B)|qwen3_asr|qwen3_asr|hermes|qwen-asr, transformers==4.57.6|&#x2714;|audio|[Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B)|
|[Qwen/Qwen3-TTS-12Hz-1.7B-Base](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-1.7B-Base)|qwen3_tts|qwen3_tts|hermes|qwen-tts|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base)|
|[Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice)|qwen3_tts|qwen3_tts|hermes|qwen-tts|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice)|
|[Qwen/Qwen3-TTS-12Hz-0.6B-Base](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-0.6B-Base)|qwen3_tts|qwen3_tts|hermes|qwen-tts|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-0.6B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base)|
|[Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice)|qwen3_tts|qwen3_tts|hermes|qwen-tts|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice)|
|[Qwen/Qwen3-TTS-12Hz-1.7B-Base](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-1.7B-Base)|qwen3_tts|qwen3_tts|hermes|qwen-tts, transformers<5|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base)|
|[Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice)|qwen3_tts|qwen3_tts|hermes|qwen-tts, transformers<5|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice)|
|[Qwen/Qwen3-TTS-12Hz-0.6B-Base](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-0.6B-Base)|qwen3_tts|qwen3_tts|hermes|qwen-tts, transformers<5|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-0.6B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base)|
|[Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice](https://modelscope.cn/models/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice)|qwen3_tts|qwen3_tts|hermes|qwen-tts, transformers<5|&#x2718;|audio, tts|[Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice)|
|[Qwen/Qwen3-VL-2B-Instruct](https://modelscope.cn/models/Qwen/Qwen3-VL-2B-Instruct)|qwen3_vl|qwen3_vl|hermes|transformers>=4.57, qwen_vl_utils>=0.0.14, decord|&#x2714;|vision, video|[Qwen/Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct)|
|[Qwen/Qwen3-VL-2B-Thinking](https://modelscope.cn/models/Qwen/Qwen3-VL-2B-Thinking)|qwen3_vl|qwen3_vl|hermes|transformers>=4.57, qwen_vl_utils>=0.0.14, decord|&#x2714;|vision, video|[Qwen/Qwen3-VL-2B-Thinking](https://huggingface.co/Qwen/Qwen3-VL-2B-Thinking)|
|[Qwen/Qwen3-VL-2B-Instruct-FP8](https://modelscope.cn/models/Qwen/Qwen3-VL-2B-Instruct-FP8)|qwen3_vl|qwen3_vl|hermes|transformers>=4.57, qwen_vl_utils>=0.0.14, decord|&#x2714;|vision, video|[Qwen/Qwen3-VL-2B-Instruct-FP8](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct-FP8)|
Expand Down
4 changes: 1 addition & 3 deletions swift/model/models/qwen.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1808,11 +1808,9 @@ def tts_forward(self,
codec_i_embedding = codec_i_embedding * codec_mask.unsqueeze(-1)
input_embeddings = input_embeddings + codec_i_embedding

# Forward through talker (shifted by 1 for autoregressive prediction)
outputs = self.talker(
inputs_embeds=input_embeddings[:, :-1, :],
attention_mask=attention_mask[:, :-1],
labels=codec_0_labels[:, 1:],
output_hidden_states=True,
)
Comment thread
Jintao-Huang marked this conversation as resolved.

Expand Down Expand Up @@ -1874,7 +1872,7 @@ def get_model(self, model_dir: str, config, processor, model_kwargs) -> PreTrain
Qwen3TTSLoader,
model_arch=ModelArch.qwen3_tts,
architectures=['Qwen3TTSForConditionalGeneration'],
requires=['qwen-tts'],
requires=['qwen-tts', 'transformers<5'],
tags=['audio', 'tts'],
))

Expand Down
12 changes: 10 additions & 2 deletions swift/template/templates/qwen.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1234,10 +1234,18 @@ def compute_sft_loss(self, model, inputs, num_items_in_batch=None, trainer=None)
inputs.pop('ref_mels')
inputs['speaker_embedding'] = speaker_embedding
outputs = model(**inputs)
talker_loss = outputs.loss
logits = outputs.logits
shift_labels = inputs['codec_0_labels'][:, 1:].contiguous()
talker_loss = F.cross_entropy(
logits.reshape(-1, logits.shape[-1]).float(),
shift_labels.reshape(-1).to(logits.device),
ignore_index=-100,
)
sub_talker_loss = getattr(outputs, 'sub_talker_loss', None)
if sub_talker_loss is not None:
outputs.loss = talker_loss + 0.3 * sub_talker_loss
outputs['loss'] = talker_loss + 0.3 * sub_talker_loss
else:
outputs['loss'] = talker_loss
return outputs
Comment thread
Jintao-Huang marked this conversation as resolved.

def save_callback(self, model, output_dir):
Expand Down
Loading