Skip to content

Latest commit

 

History

History
122 lines (104 loc) · 6.25 KB

README.md

File metadata and controls

122 lines (104 loc) · 6.25 KB

Phonon

Text-to-speech at 24 kHz with Gradium's Phonon 31M English, built on Kyutai's Pocket TTS and the Mimi codec. The weights, tokenizer and voices are Apache-2.0.

# From the repository root, in a new Python 3.11+ environment. Install
# ExecuTorch from this checkout; it also builds CUDA if nvcc is on PATH,
# Vulkan if glslc is on PATH, and MLX if `xcrun --find metal` succeeds.
./install_executorch.sh
python -m pip install safetensors tokenizers huggingface_hub

# Choose xnnpack (CPU), cuda (NVIDIA GPU), mlx (Apple Silicon), or vulkan (GPU).
# For webgpu (the browser), see WebGPU below.
BACKEND=xnnpack
# Export: FP32 for every backend. Add --dtype bf16 for BF16 on CUDA or MLX.
python -m executorch.examples.models.phonon.export_phonon \
  --backend "$BACKEND" \
  --output-dir "phonon_exports/$BACKEND"

# Build the matching runner (Vulkan requires the Vulkan SDK and glslc).
# Remove cmake-out before building for another backend.
make "phonon-${BACKEND/xnnpack/cpu}"

# Synthesize
DATA_PATH=""
if [ "$BACKEND" = cuda ]; then
  DATA_PATH="phonon_exports/cuda/aoti_cuda_blob.ptd"
fi
cmake-out/examples/models/phonon/phonon_runner \
  --model_path="phonon_exports/$BACKEND/phonon.pte" \
  --data_path="$DATA_PATH" \
  --text="Hello from ExecuTorch." \
  --output=hello.wav
# Prints "Wrote hello.wav: 2.16 s of audio, 27 frames, ..." for an FP32 export.

Export downloads gradium/phonon-31m-en at a pinned revision (or reads --checkpoint-dir) and writes phonon.pte, phonon.json (the inputs and outputs of its four methods), the tokenizer and the default voice. --voice_path takes another voice from gradium/phonon-voices; --temperature (0.3 by default, 0 for no noise) and --seed control sampling. run_phonon.py is the same runner in Python and generates the same audio from the same program (python -m executorch.examples.models.phonon.run_phonon --model phonon_exports/$BACKEND/phonon.pte --text "Hello from ExecuTorch." --output hello_py.wav, plus --data-path for CUDA).

--stream writes 16-bit mono PCM to stdout as each 80 ms frame is decoded, instead of the WAV file, so playback starts after the first frame:

cmake-out/examples/models/phonon/phonon_runner --model_path=... --text="..." --stream |
  aplay -q -f S16_LE -r 24000 -c 1     # macOS: play -q -t raw -r 24000 -e signed -b 16 -c 1 -

XNNPACK delegates the linear and convolution ops; the flow LM's attention runs as the ExecuTorch custom op llama::sdpa_with_kv_cache.

For Vulkan, MLX and CUDA, the export rewrites the Mimi decoder's transposed convolutions as convolutions, which those backends run faster. XNNPACK's lowering does this itself for the dense ones; the XNNPACK export still swaps the depthwise upsampler for a multiply. For CUDA, the Mimi decoder also keeps its streaming state on the GPU between frames: its mimi_decode takes (latent, reset) instead of the latent and the state (see phonon.json).

--quantize 4bit or --quantize 8bit quantizes the linear layers of the flow LM's transformer and flow net with the scheme of the backend; the codec stays in the export dtype. Without --quantize, the export is full precision.

Backend --quantize 4bit --quantize 8bit Size of phonon.pte (FP32: about 155 MiB)
XNNPACK 8-bit dynamic activations, 4-bit weights 8-bit dynamic activations, 8-bit weights 64, 75 MiB
CUDA 4-bit weights, BF16 8-bit weights 64, 62 MiB with BF16
Vulkan, WebGPU 4-bit weights not available 66 MiB
MLX asymmetric 4-bit weights 8-bit weights 68, 76 MiB; 41, 51 MiB with BF16

--backend cuda --quantize 4bit exports with --dtype bf16, which the int4 kernel needs. A level without a kernel on the backend fails before the download and names the options that work. 8-bit and the asymmetric 4-bit of CUDA and MLX keep the timing of FP32. The symmetric 4-bit of XNNPACK, Vulkan and WebGPU keeps the words but can hold the silence at the end of a sentence for up to two seconds.

Vulkan and MLX run each method in one partition. Vulkan needs glslc, for example from the Vulkan SDK (on Linux, extract the tarball and source setup-env.sh). MLX needs the full Xcode with the Metal toolchain, selected with xcode-select before ./install_executorch.sh (see the MLX backend); the first run compiles the Metal kernels.

CUDA export requires a CUDA toolkit; with its nvcc on PATH, ./install_executorch.sh installs CUDA-enabled PyTorch and the CUDA backend. Each method runs in one CUDA partition, with float32 inputs and outputs and FP32 (or BF16) model computation. Export produces phonon.pte and aoti_cuda_blob.ptd; pass the latter through --data_path as shown above. If the runner cannot load a method on a system with an older libstdc++, put the exporting Python environment's libraries first: export LD_LIBRARY_PATH="$(python -c 'import sys; print(sys.prefix)')/lib:$LD_LIBRARY_PATH".

BF16 halves the weights, but small differences grow through the autoregressive loop into a different take of the same text; FP32 matches the official runtime sample for sample.

WebGPU runs Phonon in the browser on the ExecuTorch Wasm runtime; try it in the Text to Speech demo of the ExecuTorch WebGPU Playground. Export with --backend webgpu --webgpu-compat-ops --output-dir phonon_exports/webgpu. The WebGPU backend does not delegate aten.elu or 1-D transposed convolution yet; --webgpu-compat-ops replaces them (and native_layer_norm and aten.silu) with equivalent ops, so every method but the text embedding runs on the GPU. Without it, those ops run on the CPU.

Phonon reads English only; spell out numbers and symbols. The tests (python -m pytest examples/models/phonon/tests -q) compare the exported programs of a small random model with a reference implementation of the official runtime (see NOTICE). To check your own export, synthesize "The quick brown fox jumps over the lazy dog." with --temperature=0: every FP32 backend gives 37 frames (2.96 s), as the official runtime does.