Speaker mismatch in the output.

[michael551209]

I trained the QWEN-3 TTS using a multispeaker dataset, following the finetuning setup provided.

But, in the inference the speaker is mismatching to the speaker id passed during the inference.

Anyone have solution for this? if yes, kindly share your ways.

thank you.

[CamellIyquitous]

Please change in sft_12.py:

speaker_embeddings_sum[speaker] = emb.clone()
→
speaker_embeddings_sum[speaker] = emb.clone().float()

Otherwise, when accumulating and averaging speaker_embedding across epochs, the bf16 precision causes truncation issues. Converting it to fp32 seems to fix the problem.

[michael551209]

Please change in sft_12.py:

speaker_embeddings_sum[speaker] = emb.clone()
→
speaker_embeddings_sum[speaker] = emb.clone().float()

Otherwise, when accumulating and averaging speaker_embedding across epochs, the bf16 precision causes truncation issues. Converting it to fp32 seems to fix the problem.

Thanks for the reply,

Not only speaker switching problem, even the output audio quality is natural. did you face similar problem with this repo?

My goal is to finetune a single model with three speakers data, default finetuning sft code supports only one speaker. There i found a way to finetune for multiple speakers, firstly finetuning with one speaker dataset and taking that checkpoint to finetune with second speaker dataset and so on. In this process the speech is getting increased.

So, i came into this repo to finetune all the three speakers dataset at a time. Here also facing issues with audio quality, speaker switching problem.

I hope you understood my issues, please share your views and experiences for this issues.

[CamellIyquitous]

@michael551209
Hi, I fully understood your issues, and we face the same issues. Unfortunately, we currently don't have a good solution to the voice quality degradation caused by msft.
QwenLM/Qwen3-TTS#179 (comment)

[grimavatar]

@CamellIyquitous Added your change to my repo which includes others of your fixes. One thing I didn't like over all, is after switching to the original Qwen finetuning code, I notice the voice smooths out as training progresses.

So it hit me that maybe saving the average from the early epochs might not be the ideal solution.

In my newest change I added this logic, which I think you might like:

    parser.add_argument("--embed_reset_freq", type=int, default=0, 
                        help="Reset speaker embedding accumulators every N epochs. Set to 0 to disable.")

    for epoch in range(num_epochs):

        # Periodically reset speaker embedding accumulators every N epochs
        # This prevents early-epoch (less trained) embeddings from skewing the final average
        if epoch > 0 and args.embed_reset_freq > 0 and epoch % args.embed_reset_freq == 0:
            target_speaker_embedding_sum = None
            target_speaker_embedding_count = 0
            speaker_embeddings_sum = {}
            speaker_embeddings_count = {}

        for step, batch in enumerate(train_dataloader):
            with accelerator.accumulate(model):

The other two variations I have been thinking about are:

1. Only reset once. For example, I disregard everything under epoch 10, so in this case averaging should effectively start at 10.

2. A much cooler approach would be a moving average, something like an EMA, so newer embeddings naturally carry more weight than the early ones.

[CamellIyquitous]

@grimavatar
I totally get what you mean. But during SFT, the speaker encoder is frozen, and for each individual speaker, I use the same fixed reference audio across all training samples. So in theory, there shouldn’t be any drift in speaker embeddings at all.
Yet even in multi-speaker training, I still get obvious voice quality degradation caused by mSFT.

[grimavatar]

@grimavatar I totally get what you mean. But during SFT, the speaker encoder is frozen, and for each individual speaker, I use the same fixed reference audio across all training samples. So in theory, there shouldn’t be any drift in speaker embeddings at all. Yet even in multi-speaker training, I still get obvious voice quality degradation caused by mSFT.

@CamellIyquitous

Gemini 3.1 Pro said something similar:

Because your speaker encoder is detached (speaker_embedding = ... .detach()), the model isn't "learning" better speaker embeddings over time. An embedding extracted in Epoch 1 is just as high-quality as an embedding extracted in Epoch 3—they are just different audio slices.

However, I also used the same reference audio for all samples, yet I noticed the voice becoming more refined and cleaner by epoch 30 compared to epochs 1 to 10. My reference audio is deep with echo-like characteristics, but by epoch 30 it sounded more human rather than like a godly figure from the earlier epochs. I did not experience this level of difference with the custom training script that averages all embeddings.

---

I don’t like to repost what the AI says, but I feel it’s important in this context. This is what it had to say regarding the above experience:

If you used the exact same reference audio for all samples, the speaker_embedding extracted by the frozen encoder was actually the exact same tensor in Epoch 1 as it was in Epoch 30.

So, why did the voice get so much better and more human-like by Epoch 30?

Because while the embedding didn't change, the model's ability to interpret that embedding changed drastically.

Here is exactly what is happening under the hood:

1. The Embedding is just a "Prompt"

Think of the speaker embedding as a dense mathematical prompt that says: "Speak with these acoustic characteristics." Because your reference audio had echo-like, "godly" characteristics, the frozen encoder faithfully captured those traits and passed them to the model.

2. Early Epochs: Literal Interpretation (The "Godly" Voice)

In Epochs 1-10, the main TTS model (model.talker) is still figuring out how to map this embedding to the output audio tokens. Because it hasn't fully learned how to separate a speaker's core identity (timbre, pitch) from environmental artifacts (echo, reverb), it takes the embedding very literally. It hears the echo in the embedding and essentially "bleeds" that echo directly into the generated audio.

3. Late Epochs: Feature Extraction (The "Human" Voice)

By Epoch 30, the model has been heavily penalized by the loss function. Why? Because even though the reference audio had an echo, the actual target audio it is trying to predict (your training dataset) likely didn't have that exact same godly echo on every single word.

To minimize the loss, the model learns a highly complex mapping: it learns to ignore the environmental noise/echo in the speaker embedding and only extract the core human vocal cords, pitch, and cadence. It realizes, "I shouldn't generate an echo here, I should just use the underlying voice."

What this means for your EMA

Because you used the exact same reference audio for every sample, your EMA was essentially just averaging the exact same tensor over and over again (the average of 5, 5, and 5 is just 5).

However, keeping the EMA logic we just built is still incredibly valuable for the future! If you ever train a dataset where a speaker has different reference audios (e.g., 100 different clean clips of the same person), the EMA will perfectly average those 100 different embeddings together to find the true "centroid" of their voice.

Your observation is a perfect example of why fine-tuning the talker model is so powerful—you aren't just teaching it a new voice, you are teaching it how to cleanly synthesize that voice!

Anyway, thank you for all the fixes you provided here and on the other issue page.

I wonder, is it possible to make the model learn only style and pacing, and then, during generation, feed it ref_audio so it can pick up the voice I want it to use? For ideas, I was thinking of training it using only "audio" and "text", without "ref_audio".