Feature request: Ensemble (multi-model combination) mode

[c469591]

Hello,
is it possible to add the functionality of combining multiple models,
similar to UVR's Ensemble Mode? And can we specify the way of combination,
like choosing Min Spec, Max Spec, Average in UVR? Thank you.

[hijaek]

I was looking for the same

[beveradb]

It's certainly possible!

I'm personally not keen on diving back into the UVR code again any time soon to figure out how those features are implemented, but PRs are very much welcome on this repo and I'd happily pair with anyone interested to help them get up to speed with it :)

Most of the core logic in this project was cherry picked from https://github.com/Anjok07/ultimatevocalremovergui/blob/master/separate.py

[c469591]

Hi,
I noticed that currently only the MD model is supported. Is it possible to add the VR model? The VR model for noise reduction is very useful. Thank you!

[beveradb]

Anyone is welcome to submit pull requests to this repo :)

[c469591]

Thank you.

[beveradb]

Hey folks, FYI I've been working on adding support for VR models this week, and I released audio-separator version 0.14 earlier today with initial support for VR models!

Please give it a try and see if it works for you!

I'm still working on documentation, tests and some packaging issues but the package on PyPI should "just work".

There's a new CLI parameter audio-separator --list_models which just prints all the models which are supported out of the box, and the interface has changed slightly (you now specify model filename with extension too).

I will inevitably be working on "ensemble mode" and model chaining functionality later this month, as I've been contracted to add support for stem splitting (which kinda goes hand in hand with that).

That said, it's already pretty easy to use audio-separator with multiple models in a row as the output filenames are consistent so you can easily script it to process a file with one model after another!

[c469591]

Hello, I am very diligent and excited that now we can also use the CLI version of the VR model. Thank you so much.
I was wondering if it would be possible to add a synthesis feature in the future, similar to UVR, which can merge multiple documents processed from different models. This could greatly enhance the sound quality of the extracted files.

[beveradb]

Yes, I plan to implement that - hopefully later this month! 😄

[c469591]

Thank you! I'd like to ask a question that's been asked many times before: does MDX now support the 23c model?

[beveradb]

Thank you! I'd like to ask a question that's been asked many times before: does MDX now support the 23c model?

I'm afraid not quite yet (that's still on my TODO list), but it's not far away now; I intend to implement that later this week or next!

[beveradb]

Hey @c469591, the latest version of audio-separator now supports MDX, VR and Demucs models.
I haven't yet finished implementation of the checkpoint models (MDX23C) but I plan to add that later this week.

I'm actually not very familiar with the ensemble mode in UVR; I'll try and dig into it and understand exactly what it's doing later this week too.
However, would you be able to explain what it does from your perspective, or provide any example audio files where it produces better results than a single model? Seeing great results from something motivates me to implement it!

Thank you!

[c469591]

Hello,
Based on my years of experience using UVR, the ensemble mode roughly works like this; it consists of several steps.
The first step is to run each selected model individually.
For example, if I have chosen the 23c from MDX and the 5_HP-Karaoke-UVR from VR, UVR will first run 23c to generate separate accompaniment and vocal files, then it will run 5_HP-Karaoke-UVR to produce another set of accompaniment and vocal files.
Next, all the accompaniment tracks are merged into one file using a method that I'm not aware of; similarly for the vocal files.
I speculate that it might use some strategy like audio phase cancellation to nullify identical sounds across multiple tracks while merging different sounds together—though this is just an unfounded guess.
In the end, after merging, you get an accompaniment with harmonies because 5_HP-Karaoke-UVR includes harmonies.
Additionally, since 23c processes richer instrumental details in its accompaniments, you end up with a result that's generally better than what you'd get from any single model alone.
Of course, if one of the models didn't completely eliminate vocals from its track those remnants would also be included in the final mixed-down accompaniment file.
That's my understanding of ensemble mode—I hope this helps you!