forked from RVC-Boss/GPT-SoVITS
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
a0fcbb5
commit ef91575
Showing
4 changed files
with
1,469 additions
and
288 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,263 +1,65 @@ | ||
| <div align="center"> | ||
| # GPT-SoVITS-Infer | ||
|
|
||
| <h1>GPT-SoVITS-WebUI</h1> | ||
| A Powerful Few-shot Voice Conversion and Text-to-Speech WebUI.<br><br> | ||
| This is the inference code of [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS) that can be developer-friendly. | ||
|
|
||
| [](https://github.com/RVC-Boss/GPT-SoVITS) | ||
| ## Prepare the environment | ||
|
|
||
| <img src="https://counter.seku.su/cmoe?name=gptsovits&theme=r34" /><br> | ||
|
|
||
| [](https://colab.research.google.com/github/RVC-Boss/GPT-SoVITS/blob/main/colab_webui.ipynb) | ||
| [](https://github.com/RVC-Boss/GPT-SoVITS/blob/main/LICENSE) | ||
| [](https://huggingface.co/lj1995/GPT-SoVITS/tree/main) | ||
| [](https://discord.gg/dnrgs5GHfG) | ||
|
|
||
|
|
||
| [**English**](./README.md) | [**中文简体**](./docs/cn/README.md) | [**日本語**](./docs/ja/README.md) | [**한국어**](./docs/ko/README.md) | ||
|
|
||
| </div> | ||
|
|
||
| --- | ||
|
|
||
| ## Features: | ||
|
|
||
| 1. **Zero-shot TTS:** Input a 5-second vocal sample and experience instant text-to-speech conversion. | ||
|
|
||
| 2. **Few-shot TTS:** Fine-tune the model with just 1 minute of training data for improved voice similarity and realism. | ||
|
|
||
| 3. **Cross-lingual Support:** Inference in languages different from the training dataset, currently supporting English, Japanese, and Chinese. | ||
|
|
||
| 4. **WebUI Tools:** Integrated tools include voice accompaniment separation, automatic training set segmentation, Chinese ASR, and text labeling, assisting beginners in creating training datasets and GPT/SoVITS models. | ||
|
|
||
| **Check out our [demo video](https://www.bilibili.com/video/BV12g4y1m7Uw) here!** | ||
|
|
||
| Unseen speakers few-shot fine-tuning demo: | ||
|
|
||
| https://github.com/RVC-Boss/GPT-SoVITS/assets/129054828/05bee1fa-bdd8-4d85-9350-80c060ab47fb | ||
|
|
||
| **User guide: [简体中文](https://www.yuque.com/baicaigongchang1145haoyuangong/ib3g1e) | [English](https://rentry.co/GPT-SoVITS-guide#/)** | ||
|
|
||
| ## Installation | ||
|
|
||
| For users in China region, you can [click here](https://www.codewithgpu.com/i/RVC-Boss/GPT-SoVITS/GPT-SoVITS-Official) to use AutoDL Cloud Docker to experience the full functionality online. | ||
|
|
||
| ### Tested Environments | ||
|
|
||
| - Python 3.9, PyTorch 2.0.1, CUDA 11 | ||
| - Python 3.10.13, PyTorch 2.1.2, CUDA 12.3 | ||
| - Python 3.9, PyTorch 2.2.2, macOS 14.4.1 (Apple silicon) | ||
| - Python 3.9, PyTorch 2.2.2, CPU devices | ||
|
|
||
| _Note: numba==0.56.4 requires py<3.11_ | ||
|
|
||
| ### Windows | ||
|
|
||
| If you are a Windows user (tested with win>=10), you can directly download the [pre-packaged distribution](https://huggingface.co/lj1995/GPT-SoVITS-windows-package/resolve/main/GPT-SoVITS-beta.7z?download=true) and double-click on _go-webui.bat_ to start GPT-SoVITS-WebUI. | ||
|
|
||
| Users in China region can download [the 0217 package](https://www.icloud.com.cn/iclouddrive/061bfkcVJcBfsMfLF5R2XKdTQ#GPT-SoVITS-beta0217) or [the 0306fix2 package](https://www.icloud.com.cn/iclouddrive/09aaTLf96aa92dbLe0fPNM5CQ#GPT-SoVITS-beta0306fix2) by clicking the links and then selecting "Download a copy." | ||
|
|
||
| _Note: The 0306fix2 version doubles the inference speed and fixes all issues with the no reference text mode._ | ||
|
|
||
| ### Linux | ||
|
|
||
| ```bash | ||
| conda create -n GPTSoVits python=3.9 | ||
| conda activate GPTSoVits | ||
| bash install.sh | ||
| ``` | ||
|
|
||
| ### macOS | ||
|
|
||
| **Note: The models trained with GPUs on Macs result in significantly lower quality compared to those trained on other devices, so we are temporarily using CPUs instead.** | ||
|
|
||
| 1. Install Xcode command-line tools by running `xcode-select --install` | ||
| 2. Install FFmpeg by running `brew install ffmpeg` or `conda install ffmpeg`. | ||
| 3. Install the program by running the following commands: | ||
|
|
||
| ```bash | ||
| conda create -n GPTSoVits python=3.9 | ||
| conda activate GPTSoVits | ||
|
|
||
| pip install -r requirements.txt | ||
| ``` | ||
|
|
||
| ### Install Manually | ||
|
|
||
| #### Install Dependences | ||
|
|
||
| ```bash | ||
| pip install -r requirements.txt | ||
| ``` | ||
|
|
||
| #### Install FFmpeg | ||
|
|
||
| ##### Conda Users | ||
|
|
||
| ```bash | ||
| conda install ffmpeg | ||
| ``` | ||
|
|
||
| ##### Ubuntu/Debian Users | ||
|
|
||
| ```bash | ||
| sudo apt install ffmpeg | ||
| sudo apt install libsox-dev | ||
| conda install -c conda-forge 'ffmpeg<7' | ||
| ``` | ||
|
|
||
| ##### Windows Users | ||
|
|
||
| Download and place [ffmpeg.exe](https://huggingface.co/lj1995/VoiceConversionWebUI/blob/main/ffmpeg.exe) and [ffprobe.exe](https://huggingface.co/lj1995/VoiceConversionWebUI/blob/main/ffprobe.exe) in the GPT-SoVITS root. | ||
|
|
||
| ### Using Docker | ||
|
|
||
| #### docker-compose.yaml configuration | ||
|
|
||
| 0. Regarding image tags: Due to rapid updates in the codebase and the slow process of packaging and testing images, please check [Docker Hub](https://hub.docker.com/r/breakstring/gpt-sovits) for the currently packaged latest images and select as per your situation, or alternatively, build locally using a Dockerfile according to your own needs. | ||
| 1. Environment Variables： | ||
|
|
||
| - is_half: Controls half-precision/double-precision. This is typically the cause if the content under the directories 4-cnhubert/5-wav32k is not generated correctly during the "SSL extracting" step. Adjust to True or False based on your actual situation. | ||
|
|
||
| 2. Volumes Configuration，The application's root directory inside the container is set to /workspace. The default docker-compose.yaml lists some practical examples for uploading/downloading content. | ||
| 3. shm_size： The default available memory for Docker Desktop on Windows is too small, which can cause abnormal operations. Adjust according to your own situation. | ||
| 4. Under the deploy section, GPU-related settings should be adjusted cautiously according to your system and actual circumstances. | ||
|
|
||
| #### Running with docker compose | ||
| As we all know, the dependencies of an AI project are always a mess. Here is how I prepare the environment for this project, by conda: | ||
|
|
||
| ``` | ||
| docker compose -f "docker-compose.yaml" up -d | ||
| conda install python=3.10 | ||
| conda install pytorch=2.1 torchvision torchaudio pytorch-lightning pytorch-cuda=12.1 -c pytorch -c nvidia | ||
| conda install ffmpeg=6.1.1 -c conda-forge | ||
| ``` | ||
|
|
||
| #### Running with docker command | ||
| You can also try to prepare the environment with cpu only options, which should work, but I have not tested it yet. | ||
|
|
||
| As above, modify the corresponding parameters based on your actual situation, then run the following command: | ||
| After the environment is ready, you can install the package by pip: | ||
|
|
||
| ``` | ||
| docker run --rm -it --gpus=all --env=is_half=False --volume=G:\GPT-SoVITS-DockerTest\output:/workspace/output --volume=G:\GPT-SoVITS-DockerTest\logs:/workspace/logs --volume=G:\GPT-SoVITS-DockerTest\SoVITS_weights:/workspace/SoVITS_weights --workdir=/workspace -p 9880:9880 -p 9871:9871 -p 9872:9872 -p 9873:9873 -p 9874:9874 --shm-size="16G" -d breakstring/gpt-sovits:xxxxx | ||
| pip install GPT-SoVITS | ||
| ``` | ||
|
|
||
| ## Pretrained Models | ||
|
|
||
| Download pretrained models from [GPT-SoVITS Models](https://huggingface.co/lj1995/GPT-SoVITS) and place them in `GPT_SoVITS/pretrained_models`. | ||
|
|
||
| For UVR5 (Vocals/Accompaniment Separation & Reverberation Removal, additionally), download models from [UVR5 Weights](https://huggingface.co/lj1995/VoiceConversionWebUI/tree/main/uvr5_weights) and place them in `tools/uvr5/uvr5_weights`. | ||
|
|
||
| Users in China region can download these two models by entering the links below and clicking "Download a copy" | ||
|
|
||
| - [GPT-SoVITS Models](https://www.icloud.com.cn/iclouddrive/056y_Xog_HXpALuVUjscIwTtg#GPT-SoVITS_Models) | ||
|
|
||
| - [UVR5 Weights](https://www.icloud.com.cn/iclouddrive/0bekRKDiJXboFhbfm3lM2fVbA#UVR5_Weights) | ||
|
|
||
| For Chinese ASR (additionally), download models from [Damo ASR Model](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/files), [Damo VAD Model](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/files), and [Damo Punc Model](https://modelscope.cn/models/damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch/files) and place them in `tools/asr/models`. | ||
|
|
||
| For English or Japanese ASR (additionally), download models from [Faster Whisper Large V3](https://huggingface.co/Systran/faster-whisper-large-v3) and place them in `tools/asr/models`. Also, [other models](https://huggingface.co/Systran) may have the similar effect with smaller disk footprint. | ||
|
|
||
| Users in China region can download this model by entering the links below | ||
| I do not add the packages related to torch to the dependencies of GPT-SoVITS-Infer. Check if the environment is ready if things go wrong. | ||
|
|
||
| - [Faster Whisper Large V3](https://www.icloud.com/iclouddrive/0c4pQxFs7oWyVU1iMTq2DbmLA#faster-whisper-large-v3) (clicking "Download a copy") | ||
| ## Usage Example | ||
|
|
||
| - [Faster Whisper Large V3](https://hf-mirror.com/Systran/faster-whisper-large-v3) (HuggingFace mirror site) | ||
|
|
||
| ## Dataset Format | ||
|
|
||
| The TTS annotation .list file format: | ||
| Download the pretrained model from [HuggingFace](https://huggingface.co/lj1995/GPT-SoVITS) to `pretrained_models` folder (or any place you like): | ||
|
|
||
| ``` | ||
| vocal_path|speaker_name|language|text | ||
| git clone https://huggingface.co/lj1995/GPT-SoVITS pretrained_models | ||
| ``` | ||
|
|
||
| Language dictionary: | ||
|
|
||
| - 'zh': Chinese | ||
| - 'ja': Japanese | ||
| - 'en': English | ||
|
|
||
| Example: | ||
| Create a `playground` folder (or any place you like) to save prompt audio and outputs: | ||
|
|
||
| ``` | ||
| D:\GPT-SoVITS\xxx/xxx.wav|xxx|en|I like playing Genshin. | ||
| mkdir playground | ||
| wget -P playground https://huggingface.co/datasets/BeautyyuYanli/sample_files/resolve/main/%E4%BD%A0%E5%A5%BD%20ChatGPT%EF%BC%8C%E8%AF%B7%E9%97%AE%E4%BD%A0%E7%9F%A5%E9%81%93%E4%B8%BA%E4%BB%80%E4%B9%88%E9%B2%81%E8%BF%85%E6%9A%B4%E6%89%93%E5%91%A8%E6%A0%91%E4%BA%BA%E5%90%97%EF%BC%9F.wav | ||
| ``` | ||
|
|
||
| ## Todo List | ||
|
|
||
| - [ ] **High Priority:** | ||
|
|
||
| - [x] Localization in Japanese and English. | ||
| - [x] User guide. | ||
| - [x] Japanese and English dataset fine tune training. | ||
|
|
||
| - [ ] **Features:** | ||
| - [ ] Zero-shot voice conversion (5s) / few-shot voice conversion (1min). | ||
| - [ ] TTS speaking speed control. | ||
| - [ ] Enhanced TTS emotion control. | ||
| - [ ] Experiment with changing SoVITS token inputs to probability distribution of vocabs. | ||
| - [ ] Improve English and Japanese text frontend. | ||
| - [ ] Develop tiny and larger-sized TTS models. | ||
| - [x] Colab scripts. | ||
| - [ ] Try expand training dataset (2k hours -> 10k hours). | ||
| - [ ] better sovits base model (enhanced audio quality) | ||
| - [ ] model mix | ||
|
|
||
| ## (Optional) If you need, here will provide the command line operation mode | ||
| Use the command line to open the WebUI for UVR5 | ||
| ``` | ||
| python tools/uvr5/webui.py "<infer_device>" <is_half> <webui_port_uvr5> | ||
| ``` | ||
| If you can't open a browser, follow the format below for UVR processing,This is using mdxnet for audio processing | ||
| ``` | ||
| python mdxnet.py --model --input_root --output_vocal --output_ins --agg_level --format --device --is_half_precision | ||
| ``` | ||
| This is how the audio segmentation of the dataset is done using the command line | ||
| ``` | ||
| python audio_slicer.py \ | ||
| --input_path "<path_to_original_audio_file_or_directory>" \ | ||
| --output_root "<directory_where_subdivided_audio_clips_will_be_saved>" \ | ||
| --threshold <volume_threshold> \ | ||
| --min_length <minimum_duration_of_each_subclip> \ | ||
| --min_interval <shortest_time_gap_between_adjacent_subclips> | ||
| --hop_size <step_size_for_computing_volume_curve> | ||
| ``` | ||
| This is how dataset ASR processing is done using the command line(Only Chinese) | ||
| ``` | ||
| python tools/asr/funasr_asr.py -i <input> -o <output> | ||
| ``` | ||
| ASR processing is performed through Faster_Whisper(ASR marking except Chinese) | ||
|
|
||
| (No progress bars, GPU performance may cause time delays) | ||
| ``` | ||
| python ./tools/asr/fasterwhisper_asr.py -i <input> -o <output> -l <language> | ||
| ``` | ||
| A custom list save path is enabled | ||
| The prompt audio should be about 3-10 seconds. | ||
|
|
||
| ## Credits | ||
| Generate the output: | ||
|
|
||
| Special thanks to the following projects and contributors: | ||
| ```python | ||
| from scipy.io import wavfile | ||
|
|
||
| ### Theoretical | ||
| - [ar-vits](https://github.com/innnky/ar-vits) | ||
| - [SoundStorm](https://github.com/yangdongchao/SoundStorm/tree/master/soundstorm/s1/AR) | ||
| - [vits](https://github.com/jaywalnut310/vits) | ||
| - [TransferTTS](https://github.com/hcy71o/TransferTTS/blob/master/models.py#L556) | ||
| - [contentvec](https://github.com/auspicious3000/contentvec/) | ||
| - [hifi-gan](https://github.com/jik876/hifi-gan) | ||
| - [fish-speech](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/generate.py#L41) | ||
| ### Pretrained Models | ||
| - [Chinese Speech Pretrain](https://github.com/TencentGameMate/chinese_speech_pretrain) | ||
| - [Chinese-Roberta-WWM-Ext-Large](https://huggingface.co/hfl/chinese-roberta-wwm-ext-large) | ||
| ### Text Frontend for Inference | ||
| - [paddlespeech zh_normalization](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/paddlespeech/t2s/frontend/zh_normalization) | ||
| - [LangSegment](https://github.com/juntaosun/LangSegment) | ||
| ### WebUI Tools | ||
| - [ultimatevocalremovergui](https://github.com/Anjok07/ultimatevocalremovergui) | ||
| - [audio-slicer](https://github.com/openvpi/audio-slicer) | ||
| - [SubFix](https://github.com/cronrpc/SubFix) | ||
| - [FFmpeg](https://github.com/FFmpeg/FFmpeg) | ||
| - [gradio](https://github.com/gradio-app/gradio) | ||
| - [faster-whisper](https://github.com/SYSTRAN/faster-whisper) | ||
| - [FunASR](https://github.com/alibaba-damo-academy/FunASR) | ||
|
|
||
| ## Thanks to all contributors for their efforts | ||
| inference = GPTSoVITSInference( | ||
| bert_path="pretrained_models/chinese-roberta-wwm-ext-large", | ||
| cnhubert_base_path="pretrained_models/chinese-hubert-base", | ||
| ) | ||
| inference.load_sovits("pretrained_models/s2G488k.pth") | ||
| inference.load_gpt( | ||
| "pretrained_models/s1bert25hz-2kh-longer-epoch=68e-step=50232.ckpt" | ||
| ) | ||
| prompt_text = "你好 ChatGPT，请问你知道为什么鲁迅暴打周树人吗？" | ||
| inference.set_prompt_audio( | ||
| prompt_audio_path=f"playground/{prompt_text}.wav", | ||
| prompt_text=prompt_text, | ||
| ) | ||
|
|
||
| <a href="https://github.com/RVC-Boss/GPT-SoVITS/graphs/contributors" target="_blank"> | ||
| <img src="https://contrib.rocks/image?repo=RVC-Boss/GPT-SoVITS" /> | ||
| </a> | ||
| sample_rate, data = inference.get_tts_wav( | ||
| text="鲁迅为什么暴打周树人？？？这是一个问题\n\n自古以来，文人相轻，鲁迅和周树人也不例外。鲁迅和周树人是中国现代文学史上的两位伟大作家，他们的文学成就都是不可磨灭的。但是，鲁迅和周树人之间的关系并不和谐，两人之间曾经发生过一次激烈的冲突，甚至还打了起来。那么，鲁迅为什么会暴打周树人呢？这是一个问题。 ", | ||
| ) | ||
| wavfile.write(f"playground/output.wav", sample_rate, data) | ||
| ``` |
Oops, something went wrong.