Consolidate eval scripts into one

[shivangi221b]

Summary

• Refactored evaluation into a single module and script:

  • src/evaluation/metrics.py now exposes EvaluationModule (streaming + batch) alongside STTEvaluator.
  • New unified script experiments/run_evaluation.py replaces:
    • run_comprehensive_evaluations.py
    • kavya_evaluation_framework.py
    • run_benchmark.py
    • verify_evaluation_numbers.py.

• Added LLM-as-gold evaluation mode:

  • --gold-from-llm on run_evaluation.py uses LlamaLLMCorrector (Ollama) to generate gold references from baseline transcripts.
  • Supports --eval-set, --audio-dir + --refs, and --audio-dir + --gold-from-llm.
  • Output JSON includes gold_source to indicate how references were obtained.

• Improved robustness and compatibility:

  • src/agent/ollama_llm.py updated to handle new typed responses (GenerateResponse / ChatResponse) from the ollama Python client.
  • run_evaluation.py no longer crashes when WER/CER are None; logs warnings and prints N/A in the text report.
  • Relaxed bitsandbytes requirement (>=0.40.0) to avoid install failures on macOS ARM; other core deps (torch, transformers, etc.) now install cleanly.

• Consolidated and updated documentation & references:

  • Merged docs/EVAL_SCRIPTS_RUNDOWN.md and docs/EVALUATION_VERIFICATION_SUMMARY.md into a single docs/EVALUATION_SUMMARY.md (how to run, options, verification, metrics reference).
  • Updated README.md, COMPREHENSIVE_DEMO.py, docs/SETUP_INSTRUCTIONS.md, and docs/QUICK_REFERENCE.md to point to run_evaluation.py and the new evaluation flow.

Testing

• Ran experiments/run_evaluation.py with:
  • --eval-set (JSON-based eval set).
  • --audio-dir ... --gold-from-llm with Ollama running (llama3.2:3b).
• Verified:
  • evaluation_report.json and evaluation_report.txt are generated with correct WER/CER and gold_source.
  • No errors when there are zero valid predictions (WER/CER = null / N/A).

[Copilot]

Pull request overview

This PR consolidates four separate evaluation scripts into a single unified script (experiments/run_evaluation.py) and adds support for using LLM (Ollama) as a gold-standard reference generator. The changes improve the evaluation workflow by providing a consistent interface for running evaluations on both baseline (Whisper) and fine-tuned (Wav2Vec2) models.

Changes:

• Unified evaluation: New run_evaluation.py script replaces run_comprehensive_evaluations.py, kavya_evaluation_framework.py, run_benchmark.py, and verify_evaluation_numbers.py
• LLM-as-gold mode: Added --gold-from-llm flag to use Ollama for generating reference transcripts from baseline outputs
• Enhanced compatibility: Updated ollama_llm.py to handle both dict and typed responses from different ollama package versions
• New evaluation module: Added EvaluationModule class for unified streaming and batch evaluation
• Documentation consolidation: Merged separate evaluation docs into docs/EVALUATION_SUMMARY.md
• Dependency relaxation: Changed bitsandbytes requirement from >=0.43.0 to >=0.40.0 for macOS compatibility

Reviewed changes

Copilot reviewed 17 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
experiments/run_evaluation.py	New unified evaluation script supporting eval sets, audio+refs, and LLM-as-gold modes
src/evaluation/metrics.py	Added EvaluationModule class for unified streaming and batch evaluation
src/agent/ollama_llm.py	Updated to handle both dict and typed responses from ollama package
tests/test_metrics.py	Added comprehensive tests for new EvaluationModule class
requirements.txt	Relaxed bitsandbytes version requirement for macOS ARM compatibility
docs/EVALUATION_SUMMARY.md	Consolidated evaluation documentation with usage examples
docs/SETUP_INSTRUCTIONS.md	Updated to reference new unified evaluation script
docs/QUICK_REFERENCE.md	Updated quick reference with new evaluation command
README.md	Updated evaluation workflow and examples
COMPREHENSIVE_DEMO.py	Updated demo to reference new evaluation script
scripts/setup_gcp_gpu.sh	Updated example command to use new evaluation script
.gitignore	Added .venv to ignore list
experiments/run_comprehensive_evaluations.py	Deleted (consolidated into run_evaluation.py)
experiments/kavya_evaluation_framework.py	Deleted (consolidated into run_evaluation.py)
experiments/run_benchmark.py	Deleted (consolidated into run_evaluation.py)
experiments/verify_evaluation_numbers.py	Deleted (consolidated into run_evaluation.py)
docs/EVALUATION_VERIFICATION_SUMMARY.md	Deleted (merged into EVALUATION_SUMMARY.md)
experiments/evaluation_outputs/evaluation_report.json	Added example evaluation output
experiments/evaluation_outputs/evaluation_report.txt	Added example evaluation summary

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.