AI Name-Pronunciation Fairness Benchmark

Measuring how accurately text-to-speech (TTS) systems pronounce personal names from African languages (Yoruba, Igbo, Hausa) compared to common English names — to surface and quantify bias in deployed speech systems.

Status: v1.0.0 — first archival release. 80 names (20 per language). The pipeline synthesizes with a chosen TTS engine (gTTS, espeak-ng, or Meta's neural MMS), scores pronunciation via Whisper back-transcription, and aggregates into per-language stats, charts, significance tests, and bootstrap confidence intervals (forest plots + an ASR-scorer sensitivity analysis). See CHANGELOG.md for what each version added.

📄 Paper-style writeup → (abstract · method · results · limitations · ethics) · master results table → · detailed findings →: gTTS shows a large, significant bias against African names; "neural" MMS is not fairer (its metric just saturates until a carrier test unmasks it); and MMS ships no Igbo voice at all.

Why this matters

Screen readers, voice assistants, and automated systems mispronouncing a person's name is a real, repeated harm — it disproportionately affects people with names outside the English/Western mainstream. A reproducible benchmark turns "this feels biased" into measurable evidence.

Project layout

name-pronunciation-benchmark/
  data/names.csv          studied dataset — 80 names (20/language); results use this
  data/names_extended.csv larger set — 120 names (30/language) for bigger runs
  data/SOURCES.md         dataset provenance, sourcing + validation notes
  data/DATASHEET.md       Datasheets for Datasets (Gebru et al.) record
  ttsbench/
    dataset.py          load names from CSV  -> list[Name]
    engines.py          TTSEngine interface + GTTSEngine (add more here)
    synth.py            run names through an engine, write audio + manifest.csv
  run.py                CLI entry point
  sensitivity.py        robustness check: re-score a run across Whisper sizes
  forest_plot.py        forest plot: per-run gap + 95% CI (docs/forest_plot.png)
  per_language_forest.py  per-language gap vs English for one run, with CIs
  cross_engine.py       per-name CER agreement between two engines (Spearman)
  outputs*/             per-run scores.csv + manifest.csv (tracked); audio ignored
  requirements.txt

Quick start

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python run.py                              # gTTS, English voice
python run.py --engine espeak --out outputs_espeak   # espeak-ng (needs the binary)
python run.py --engine mms --carrier --out outputs_mms_carrier   # carrier sentence

Outputs land in outputs/audio/* with outputs/manifest.csv describing every file. Use a separate --out dir per engine to keep runs side by side.

One-command pipeline — a Makefile runs synthesize → score → analyze for each engine so the whole study reproduces from scratch:

make setup PYTHON=.venv/bin/python          # install deps
make all   PYTHON=.venv/bin/python          # gTTS + carrier runs + error tables
make test  PYTHON=.venv/bin/python          # unit tests
# individual: make gtts | gtts-carrier | espeak | mms | mms-carrier | errors | clean

Carrier sentences (--carrier): speak each name inside a sentence ("My name is {name}.") instead of in isolation. Neural engines (e.g. MMS) are trained on connected speech and mangle lone words, which saturates the metric and hides real differences; a carrier gives them natural speech to synthesize, making cross-engine comparison fair. Scoring auto-detects the carrier (recorded in the manifest) and isolates the name region before measuring error.

Engines

id	library / binary	key?	offline	reproducible	notes
gtts	gTTS (pip)	no	no	weak	real Google system; unofficial endpoint
espeak	espeak-ng (binary)	no	yes	strong	deterministic; can emit IPA phonemes
mms	transformers (pip)	no	yes	strong	Meta MMS-TTS neural model; per-language voices (yor/hau; no ibo)
elevenlabs	ElevenLabs REST	yes	no	versioned	commercial neural TTS (a real product voice); set ELEVENLABS_API_KEY

The mms engine pulls PyTorch (heavy). Install the espeak-ng binary separately (it is not pip-installable):

brew install espeak-ng          # macOS (needs Homebrew)
# sudo apt-get install espeak-ng   # Debian/Ubuntu

Design notes (the "why")

• Engine abstraction. TTSEngine is an abstract base class; GTTSEngine is the first implementation. Swapping in an offline/open engine or a commercial one later is a single new subclass — the pipeline is unchanged.
• English voice on every name (for now). v0.1 uses one English voice for all names. That is the core bias probe: how does an English-centric TTS — the kind embedded in most assistants — handle African names? Native-language voices are a separate axis to add later.
• Everything is logged. The manifest records engine + version + language + UTC timestamp + file path per name, so runs are auditable and comparable.

⚠️ Reproducibility considerations

• gTTS is not stable over time. It calls an unofficial Google Translate endpoint whose output can change silently. Treat generated audio as a primary artifact to archive, not something you can always regenerate identically. Always record the date and library version (the manifest does).
• The scored tables are committed; the audio is not. Each run's scores.csv and manifest.csv are small text and are tracked, so every statistic and figure regenerates from the repo with just pip install -r requirements-dev.txt — no TTS/ASR stack. tests/test_reproducibility.py recomputes the headline numbers from those tables and asserts the documented findings, so they can't silently drift. Only the large regenerable outputs*/audio/ is git-ignored; for a paper, archive a fixed audio snapshot (zipped release, Git LFS, or Zenodo / OSF) and cite it.
• Pinned versions. requirements.txt and requirements-dev.txt pin exact versions so the pipeline reproduces bit-for-bit; loosen to >= only if you just want to try the tools.
• Prefer open, versioned models (Coqui/Piper/espeak-ng) or documented cloud APIs (Google Cloud TTS, Amazon Polly, Azure) for results you intend to publish — they can be pinned to a model/voice version.

⚠️ Ethical considerations

• Names are people. Use common given names or documented name lists, and cite their source. Do not target or profile private individuals. Provenance: the names are common given names compiled from general knowledge (not a cited corpus) and are not yet validated by native speakers — see data/SOURCES.md and the full data/DATASHEET.md (Gebru et al.) for the provenance, sourcing, and validation notes. Before any publication, replace/verify them with documented sources and native-speaker review of spellings and diacritics.
• Ground truth needs native speakers. Scoring "correct" pronunciation requires a reference (IPA and/or native-speaker recordings). Source this with the informed consent and credit of native speakers; treat it as human- subjects data and check whether your institution's IRB/ethics board applies.
• Frame findings to reduce harm. The goal is to expose and fix bias, not to caricature languages or communities. Involve native speakers in interpretation.
• Respect terms of service. Heavy automated use of unofficial endpoints (gTTS) may violate provider ToS; use official APIs for anything at scale.
• Metric bias. However you eventually score pronunciation (ASR back-transcription, phoneme distance vs IPA, human MOS ratings), the metric has its own biases — document and justify the choice.

Scoring

pip install faster-whisper                 # one-time (heavier dependency)
python score.py --run outputs --model base # transcribe + score a synthesis run

Each clip is transcribed with Whisper and compared to the intended name via Character Error Rate (CER); results are written to <run>/scores.csv and summarised per language.

Phoneme scoring (ASR-independent) — experimental

pip install allosaurus av
python phoneme_score.py --run outputs --names data/names.csv   # isolated-name run

The CER metric depends on an English word recogniser. This layer instead reads phones directly from the audio with a universal phoneme recogniser (allosaurus) — no word-level ASR — and measures a Phoneme Error Rate (PER) against a reference in the ipa column (ttsbench/phonemes.py).

Two honest limitations found while building it (this is infrastructure, not yet a headline result):

1. No reference for the target languages. espeak-ng has no Yoruba/Igbo/Hausa voices, so references can't be auto-generated. English references are provided (espeak-en G2P); African-name references require native-speaker validation before PER is meaningful — the tool consumes them the moment they exist.
2. The universal recogniser is noisy on short single-word clips (e.g. it may reduce James to a couple of phones), so PER is currently dominated by recogniser error, not pronunciation quality. Longer, carrier-phrase audio may help — a promising direction, not a solved metric.

So the value here is a reproducible, ASR-independent phoneme pipeline ready for native-speaker references, plus a documented, honest account of why phoneme-level bias measurement is genuinely hard.

Analysis

pip install matplotlib
python analyze.py --scores outputs/scores.csv --out outputs/analysis \
                  --title "gTTS (English voice)"

Aggregates a scores.csv into per-language statistics (summary.csv) and a bar chart of mean CER by language (cer_by_language.png).

Preliminary findings (v0.5, gTTS, English voice, n=80 — 20 per language)

language	mean CER	exact-match rate
english	0.01	95%
igbo	0.32	15%
yoruba	0.37	10%
hausa	0.42	25%

English vs African CER gap: +0.36 (stable across the n=16 and n=80 runs). English names transcribe almost perfectly; African names show 30–40× higher error (e.g. Oluwaseun → "Alois Seyoon", Folake → "for locker").

Why the bigger sample mattered: at n=16, Hausa looked better (0.14) than Igbo/Yoruba — but that small sample was dominated by globally-common pan-Islamic names (Aisha, Ibrahim). At n=80 the effect washes out and Hausa is 0.42. A textbook case of small-sample bias, and the reason for expanding the dataset.

Statistical significance

CER is bounded and skewed, so we use the non-parametric Mann-Whitney U test (with Cliff's delta as a distribution-free effect size). African-name CER is significantly higher than English-name CER:

• Overall (African n=60 vs English n=20): U=1094, p ≈ 9.2×10⁻⁹, Cliff's δ = +0.82 (large).
• Per language vs English (Holm-corrected): Yoruba (δ=+0.89), Igbo (δ=+0.84), and Hausa (δ=+0.74) are each significantly worse — all p_holm ≤ 3.6×10⁻⁶, all large effects.

We also report 95% bootstrap confidence intervals (2,000 seeded resamples) for the mean-CER gap and Cliff's delta, so each effect carries its uncertainty. Across every run the gap CI stays clear of zero except for the two saturated isolated-word baselines — the interval estimates independently confirm which gaps are real (see the gap 95% CI column in RESULTS.md).

The forest plot below shows this across every run at a glance — each engine × condition as a gap estimate with its 95% CI. Blue intervals clear zero (a real gap); the two grey ones that straddle zero are exactly the metric-saturated isolated-word baselines (python forest_plot.py):

Reproduce with python analyze.py (writes outputs/analysis/stats.txt). Caveat: this establishes the gap for this pipeline, not a universal claim across all TTS systems. The gap is, however, robust to the ASR scorer: re-scoring the same audio with Whisper tiny/base/small keeps it large and significant at every size (δ = +0.75 to +0.84, all p < 10⁻⁷) — a better recognizer narrows the gap but does not close it, as expected if the bias lives in the TTS. See docs/sensitivity_whisper.md (python sensitivity.py).

Metric caveat. ASR back-transcription mixes (1) TTS pronunciation quality and (2) how ASR-friendly the audio is. A clean English baseline (gTTS: 0.00) isolates the name-origin effect; a robotic voice (espeak-ng) inflates error for all names. Compare within an engine; treat the gap as more robust than absolute CER. Reference-based scoring (IPA / native-speaker) is the rigorous follow-up.

Citing & archiving

If you use this benchmark or its findings, please cite it. A CITATION.cff is included, so GitHub shows a "Cite this repository" button (top-right of the repo) with ready-made APA/BibTeX.

@software{amhanesi_name_pronunciation_benchmark,
  author  = {Amhanesi, Peace},
  title   = {AI Name-Pronunciation Fairness Benchmark},
  year    = {2026},
  url     = {https://github.com/amhanesipeace/name-pronunciation-benchmark},
  license = {MIT}
}

Archiving for a DOI (Zenodo). Code is versioned here; the generated audio is git-ignored (large/binary). To mint a permanent, citable DOI from a tagged release:

1. Sign in at zenodo.org with GitHub and, under Settings → GitHub, flip this repository On. (Do this before step 2 — Zenodo only archives releases created after the switch is on.)
2. Publish a GitHub release for the tag (v1.0.0): gh release create v1.0.0 --title "v1.0.0" --notes-file CHANGELOG.md.
3. Zenodo auto-archives the release and issues a DOI; add its badge here and the doi: field to CITATION.cff.

For a fully reproducible snapshot, attach one run's audio/, manifest.csv and scores.csv to the Zenodo record (the repo ships only code + results). Licensed under MIT.

Roadmap

1. ✅ v0.1 — synthesis + manifest (gTTS)
2. ✅ Second engine: espeak-ng (offline, deterministic, IPA-capable)
3. ✅ Scoring v1: ASR back-transcription (Whisper) + CER, summarised by language
4. ✅ Analysis: per-language stats + bar chart (analyze.py)
5. ✅ Expand the dataset to 80 names (20 per language, balanced)
6. ✅ Statistical testing: Mann-Whitney U + Cliff's delta, Holm-corrected
7. Add reference pronunciations (IPA / native-speaker audio) + cite name sources
8. Add more engines (Coqui neural; cloud APIs for real deployed systems)