Add Advanced Evaluation Metrics and Model Investigation Tools

docs/EVALUATION_METRICS_UPGRADE.md

@@ -0,0 +1,193 @@
+# Evaluation Metrics Upgrade
+
+## Overview
+
+The unified evaluator (`src/evaluation/metrics.py`) has been upgraded with additional metrics for comprehensive speech-to-text evaluation.
+
+## New Metrics
+
+### 1. Diarization Error Rate (DER)
+
+**Purpose**: Measures accuracy of speaker diarization (who spoke when).
+
+**Formula**: `DER = (Missed Speech + False Alarm + Speaker Confusion) / Total Reference Duration`
+
+**Usage**:
+```python
+from src.evaluation.metrics import STTEvaluator
+
+evaluator = STTEvaluator()
+der_score = evaluator.calculate_der(
+    reference_segments=[
+        {'start': 0.0, 'end': 5.0, 'speaker': 'A'},
+        {'start': 5.0, 'end': 10.0, 'speaker': 'B'}
+    ],
+    hypothesis_segments=[
+        {'start': 0.1, 'end': 5.1, 'speaker': 'A'},
+        {'start': 5.1, 'end': 10.1, 'speaker': 'B'}
+    ],
+    tolerance=0.25  # 250ms tolerance collar
+)
+```
+
+**Note**: Requires speaker segment information (start, end, speaker ID) in RTTM-like format.
+
+### 2. Verb Error Rate
+
+**Purpose**: Measures accuracy of verb transcription specifically, as verbs are critical for meaning.
+
+**Calculation**: Compares verbs extracted from reference vs hypothesis using NLTK POS tagging.
+
+**Usage**:
+```python
+evaluator = STTEvaluator()
+verb_rate = evaluator.calculate_verb_error_rate(
+    reference="The patient was diagnosed with pneumonia.",
+    hypothesis="The patient was diagnose with pneumonia."
+)
+```
+
+**Returns**: Error rate (0-1), where 0 is perfect and 1 is complete failure.
+
+### 3. Domain Error Rate
+
+**Purpose**: Measures accuracy within specific domains (medical, legal, technical, business).
+
+**Calculation**: Groups transcripts by detected domain and calculates domain-specific WER.
+
+**Usage**:
+```python
+evaluator = STTEvaluator()
+domain_rates = evaluator.calculate_domain_error_rate(
+    references=["Patient shows symptoms of fever.", "The court ruled in favor."],
+    hypotheses=["Patient show symptom of fever.", "The court rule in favor."]
+)
+# Returns: {'medical': 0.33, 'legal': 0.25}
+```
+
+**Customization**: Domain keywords can be customized via `evaluator.domain_keywords`.
+
+## Complete Evaluation Example
+
+```python
+from src.evaluation.metrics import STTEvaluator
+
+evaluator = STTEvaluator()
+
+results = evaluator.evaluate_batch(
+    references=["Reference transcript 1", "Reference transcript 2"],
+    hypotheses=["Hypothesis transcript 1", "Hypothesis transcript 2"],
+    include_verb_rate=True,
+    include_domain_rate=True,
+    include_der=False,  # Requires segments
+    reference_segments=None,
+    hypothesis_segments=None
+)
+
+print(f"WER: {results['wer']:.4f}")
+print(f"CER: {results['cer']:.4f}")
+print(f"Verb Error Rate: {results['verb_error_rate']:.4f}")
+print(f"Domain Error Rates: {results['domain_error_rates']}")
+```
+
+## Dependencies
+
+New dependencies added to `requirements.txt`:
+- `nltk>=3.8.0` - For POS tagging (verb extraction)
+- `pyannote.metrics>=4.0.0` - For advanced DER calculation (optional)
+
+## Model Investigation Scripts
+
+### AReal (RealtimeSTT) Investigation
+
+**Script**: `experiments/investigate_areal.py`
+
+**Purpose**: Evaluate AReal/RealtimeSTT model performance on available data.
+
+**Usage**:
+```bash
+python experiments/investigate_areal.py
+```
+
+**Output**: 
+- Latency metrics
+- WER, CER, Verb Error Rate, Domain Error Rate
+- Results saved to `experiments/evaluation_outputs/areal_evaluation_results.json`
+
+### Miles/Moonshine Investigation
+
+**Script**: `experiments/investigate_miles.py`
+
+**Purpose**: Evaluate Miles/Moonshine model performance on available data.
+
+**Usage**:
+```bash
+python experiments/investigate_miles.py
+```
+
+**Output**:
+- Latency metrics
+- WER, CER, Verb Error Rate, Domain Error Rate
+- Results saved to `experiments/evaluation_outputs/miles_moonshine_evaluation_results.json`
+
+## Oracle Teacher Script
+
+**Script**: `experiments/oracle_teacher.py`
+
+**Purpose**: Generate synthetic "gold" transcripts using GPT-4o/Llama 3 API for cases without ground truth.
+
+**Usage**:
+```bash
+# Using OpenAI GPT-4o
+export OPENAI_API_KEY='your-key'
+python experiments/oracle_teacher.py \
+    --audio-dir data \
+    --output experiments/evaluation_outputs/oracle_gold_transcripts.json \
+    --api-type openai \
+    --model gpt-4o \
+    --limit 10
+
+# Using Ollama (Llama 3)
+python experiments/oracle_teacher.py \
+    --audio-dir data \
+    --output experiments/evaluation_outputs/oracle_gold_transcripts.json \
+    --api-type llama \
+    --model llama3 \
+    --limit 10
+```
+
+**How it works**:
+1. Gets baseline transcript from Whisper
+2. Refines transcript using LLM (GPT-4o/Llama 3)
+3. LLM fixes errors, adds punctuation, corrects grammar
+4. Outputs high-quality "gold" transcript
+
+**Output Format**:
+```json
+[
+  {
+    "audio_file": "data/test.wav",
+    "baseline_transcript": "the patient was diagnose with pneumonia",
+    "gold_transcript": "The patient was diagnosed with pneumonia.",
+    "refinement_time": 2.5,
+    "model": "gpt-4o",
+    "api_type": "openai"
+  }
+]
+```
+
+## Benefits
+
+1. **Comprehensive Evaluation**: Multiple metrics provide different perspectives on model performance
+2. **Domain-Specific Analysis**: Domain Error Rate helps identify domain-specific weaknesses
+3. **Linguistic Accuracy**: Verb Error Rate focuses on critical grammatical elements
+4. **Speaker Analysis**: DER enables multi-speaker evaluation
+5. **Gold Transcript Generation**: Oracle Teacher creates high-quality references for evaluation
+
+## Future Enhancements
+
+- [ ] Add semantic similarity metrics (BERTScore, BLEU)
+- [ ] Add confidence score analysis
+- [ ] Add temporal alignment metrics
+- [ ] Support for more domain keywords
+- [ ] Batch processing optimization for Oracle Teacher

docs/EVALUATION_VERIFICATION_SUMMARY.md

@@ -0,0 +1,150 @@
+# Evaluation Numbers Verification Summary
+
+**Date**: December 2024  
+**Status**: Verification Complete
+
+---
+
+## Purpose and Context
+
+This document tracks the **verification process** for quantitative metrics reported in our research paper (`report.md`). The verification process ensures scientific accuracy by:
+
+1. **Cross-referencing reported values** against actual evaluation outputs (JSON files from `experiments/evaluation_outputs/`)
+2. **Identifying discrepancies** between initial estimates and measured values
+3. **Distinguishing verified metrics** (from actual evaluation runs) from **estimated metrics** (theoretical/expected based on component analysis)
+4. **Documenting limitations** (e.g., lack of ground truth data) that prevent full verification
+5. **Providing transparency** about which numbers in the report are measured vs. estimated
+
+**Why This Matters**: In academic/research contexts, distinguishing between verified measurements and theoretical estimates is essential for reproducibility and credibility. This document serves as an audit trail for the evaluation numbers in our report, ensuring readers can trust baseline metrics while understanding where improvements are estimated rather than measured.
+
+---
+
+## ✅ VERIFIED NUMBERS (From Actual Evaluation Files)
+
+### Baseline Model Performance
+- **WER**: 10.0% (0.1000) - ✅ Verified from `evaluation_summary.json`
+- **CER**: 2.27% (0.0227) - ✅ Verified from `evaluation_summary.json`
+- **Model Parameters**: 72,593,920 (72.6M) - ✅ Verified
+- **Mean Latency**: 5.29 seconds - ✅ Verified from `benchmark_report.json`
+- **Throughput**: 2.65 samples/second - ✅ Verified from `benchmark_report.json`
+- **Device**: CPU - ✅ Verified
+
+### Evaluation Dataset
+- **Total Samples Evaluated**: 2 samples (from evaluation_summary.json)
+- **Note**: Small sample size limits statistical power
+
+---
+
+## ⚠️ NUMBERS UPDATED IN REPORT
+
+### Fixed Discrepancies:
+1. **Latency**: Updated from 0.72s → **5.29s** (actual measured value)
+2. **Throughput**: Updated from 2.97 → **2.65 samples/s** (actual measured value)
+3. **Baseline WER in comparison table**: Updated from 25-30% → **10.0%** (actual measured value)
+
+---
+
+## 📊 NUMBERS THAT REQUIRE GROUND TRUTH DATA
+
+The following numbers in the report require actual ground truth reference transcripts to verify:
+
+### Full System Performance
+- Full system WER (currently estimated as 8.0-9.0%)
+- Full system CER (currently estimated as 1.8-2.0%)
+- Error detection precision/recall
+- Correction success rates
+
+### Ablation Study Results
+- Component-specific WER contributions
+- Configuration-specific performance metrics
+
+### Statistical Analysis
+- Paired t-test p-values
+- Cohen's d effect sizes
+- Confidence intervals
+
+### Why These Need Ground Truth:
+- Current evaluation uses baseline transcription as reference (WER = 0%)
+- Need actual human-verified transcripts to measure real improvements
+- Error detection and correction metrics require known errors
+
+---
+
+## 🔧 HOW TO VERIFY REMAINING NUMBERS
+
+### Option 1: Use Existing Ground Truth Dataset
+```python
+# If you have a dataset with ground truth transcripts:
+from src.integration import UnifiedSTTSystem
+
+system = UnifiedSTTSystem()
+results = system.evaluate_batch(
+    audio_files=["audio1.wav", "audio2.wav"],
+    reference_transcripts=["ground truth 1", "ground truth 2"]
+)
+```
+
+### Option 2: Create Synthetic Test Cases
+```python
+# Create test cases with known errors:
+# 1. Transcribe audio with baseline
+# 2. Introduce known errors
+# 3. Use as reference
+# 4. Measure correction effectiveness
+```
+
+### Option 3: Use Public STT Datasets
+- LibriSpeech
+- Common Voice
+- TIMIT
+- Any dataset with ground truth transcripts
+
+---
+
+## 📝 REPORT STATUS
+
+### ✅ Verified Sections:
+- Baseline model performance (WER, CER, latency, throughput)
+- Model parameters and configuration
+- Evaluation framework description
+
+### ⚠️ Estimated/Theoretical Sections:
+- Full system performance improvements
+- Ablation study results
+- Statistical significance values
+- Component contributions
+- Error detection metrics
+
+### 📌 Notes Added to Report:
+- Italicized notes indicating verified vs estimated numbers
+- Disclaimers about dataset limitations
+- Framework capabilities vs actual measured results
+
+---
+
+## 🎯 RECOMMENDATIONS
+
+1. **For Full Verification**: Obtain ground truth transcripts for test audio files
+2. **For Report**: Current report accurately reflects verified baseline metrics
+3. **For Future Work**: Run comprehensive evaluation with ground truth when available
+4. **For Presentation**: Clearly distinguish between verified metrics and theoretical estimates
+
+---
+
+## 📊 ACTUAL MEASURED VALUES SUMMARY
+
+| Metric | Verified Value | Source |
+|--------|---------------|--------|
+| Baseline WER | 10.0% | evaluation_summary.json |
+| Baseline CER | 2.27% | evaluation_summary.json |
+| Mean Latency | 5.29s | benchmark_report.json |
+| Throughput | 2.65 samples/s | benchmark_report.json |
+| Model Params | 72.6M | evaluation_summary.json |
+| Device | CPU | evaluation_summary.json |
+| Samples Evaluated | 2 | evaluation_summary.json |
+
+---
+
+**Conclusion**: Baseline metrics are verified and accurate. Full system improvements require ground truth data for proper evaluation.
+
+