Add unified data gathering system with plugin architecture - #25
Add unified data gathering system with plugin architecture#25GAInTheHouse wants to merge 26 commits into
Conversation
…EFERENCE at root, no extra docs, no agent_evaluator) Co-authored-by: Cursor <cursoragent@cursor.com>
- ✅ Created Dockerfile.training (GPU) and Dockerfile.training.cpu (CPU) - ✅ Verified all dependencies: LoRA/PEFT, Wav2Vec2, PyTorch, audio processing - ✅ Added Docker Compose configuration for training - ✅ Created verification and build scripts - ✅ Comprehensive documentation (quickstart, troubleshooting, completion report) - ✅ CPU container built and verified successfully (3.05GB) - ✅ GPU container ready for GCP deployment All Week 1 tasks for Kavya completed. Training environment is reproducible and ready for GCP deployment. Co-authored-by: Cursor <cursoragent@cursor.com>
…Docker docs - Move run_comprehensive_evaluations.py to experiments/ - Move verify_evaluation_numbers.py to experiments/ - Move EVALUATION_VERIFICATION_SUMMARY.md to docs/ and expand explanation - Delete redundant WEEK1_DOCKER_BUILD_SUMMARY.md (consolidated into docs/WEEK1_TRAINING_DOCKER.md) - Delete redundant WEEK1_TRAINING_DOCKER_QUICKSTART.md (consolidated into docs/WEEK1_TRAINING_DOCKER.md) - Add Quick Start section to docs/WEEK1_TRAINING_DOCKER.md - Expand verification summary purpose and context explanation Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Pull request overview
This pull request adds comprehensive data processing scripts and training infrastructure for the Adaptive Self-Learning Agentic AI System. The changes introduce scripts to download and process speech datasets from multiple sources (OpenSLR, Hugging Face), create audio augmentations, and set up Dockerized training environments with LoRA and Wav2Vec2 support.
Changes:
- Data download and processing pipeline supporting LibriSpeech, MUSAN, RIRS, Common Voice, and VoxPopuli datasets
- Audio augmentation utilities for adding noise, reverb, dropouts, and clipping to create robust training data
- Dockerized training environment with GPU and CPU variants, including all necessary dependencies for LoRA-based fine-tuning
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/openslr_download.py | Downloads OpenSLR datasets with resumable downloads and checksum verification |
| scripts/hf_download.py | Downloads and processes Hugging Face datasets, materializing audio to WAV format |
| scripts/make_librispeech_manifest.py | Generates CSV manifests for LibriSpeech datasets |
| scripts/bootstrap_data.py | Unified script orchestrating all data download, manifest generation, and audio derivation tasks |
| scripts/augment_audio.py | Audio augmentation using MUSAN noise and RIRS reverb with configurable SNR levels |
| scripts/verify_training_docker.sh | Verification script for Docker training environment dependencies |
| scripts/fix_docker_credentials.sh | Helper script for Docker credential issues |
| scripts/fix_and_build_training_docker.sh | Automated Docker build script with credential fixing |
| Dockerfile.training | GPU-enabled training container with CUDA 11.8 support |
| Dockerfile.training.cpu | CPU-only training container for local testing |
| docker-compose.training.yml | Docker Compose configuration for training environment |
| .dockerignore.training | Excludes unnecessary files from training Docker context |
| requirements.txt | Reorganized dependencies into logical sections |
| experiments/verify_evaluation_numbers.py | Script to verify reported evaluation metrics |
| experiments/run_comprehensive_evaluations.py | Comprehensive evaluation runner for all system components |
| experiments/evaluation_results/evaluation_report_20251214_225445.txt | Example evaluation report output |
| docs/WEEK1_TRAINING_DOCKER.md | Documentation for Week 1 training Docker deliverable |
| docs/EVALUATION_VERIFICATION_SUMMARY.md | Summary of evaluation metric verification process |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 6 changed files in this pull request and generated 11 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 9 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…luating what we had in main
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 35 out of 37 changed files in this pull request and generated 7 comments.
Comments suppressed due to low confidence (1)
scripts/deprecated/download_datasets.py:3
- The
??emoji at the start of the deprecation notice docstring is unprofessional for production code. It should be replaced with plain text such asDEPRECATEDorWARNING: DEPRECATED.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 26 changed files in this pull request and generated 14 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
GAInTheHouse
changed the title
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 26 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # region agent log | ||
| import json | ||
| log_data = {"hypothesisId": "D", "runId": "debug1", "location": "openslr_plugin.py:169", "message": "HEAD request failed", "data": {"url": url, "error_type": type(head_exc).__name__, "error_msg": str(head_exc)}, "timestamp": int(__import__('time').time() * 1000)} | ||
| try: | ||
| with open(_debug_log_path('download'), 'a') as f: | ||
| f.write(json.dumps(log_data) + '\n') | ||
| except: pass | ||
| # endregion | ||
|
|
||
| self.logger.warning("HEAD request failed for %s", url) |
There was a problem hiding this comment.
Similar to huggingface_plugin.py, this file contains debug logging infrastructure (_LOG_DIR, _SESSION_TS, _debug_log_path()) and an # region agent log block with "hypothesisId": "D" and "runId": "debug1" fields that are clearly leftover development artifacts. The except: pass clause at line 189 silently suppresses all exceptions from the debug file write. These should be removed.
| # region agent log | |
| import json | |
| log_data = {"hypothesisId": "D", "runId": "debug1", "location": "openslr_plugin.py:169", "message": "HEAD request failed", "data": {"url": url, "error_type": type(head_exc).__name__, "error_msg": str(head_exc)}, "timestamp": int(__import__('time').time() * 1000)} | |
| try: | |
| with open(_debug_log_path('download'), 'a') as f: | |
| f.write(json.dumps(log_data) + '\n') | |
| except: pass | |
| # endregion | |
| self.logger.warning("HEAD request failed for %s", url) | |
| self.logger.warning( | |
| "HEAD request failed for %s: %s", | |
| url, | |
| head_exc, | |
| ) |
|
|
||
| try: | ||
| # Import augmentation script | ||
| sys.path.insert(0, str(Path(__file__).parent.parent)) | ||
| from augment_audio import main as augment_main | ||
|
|
||
| # Note: This is a simplified integration | ||
| # Full augmentation would require calling augment_audio with proper args | ||
| LOGGER.info("Augmentation integration: Call augment_audio.py separately") | ||
| LOGGER.info(" Example: python scripts/augment_audio.py --manifest-dir %s", args.manifest_dir) | ||
|
|
||
| except ImportError as exc: | ||
| LOGGER.warning("Could not import augment_audio: %s", exc) | ||
|
|
There was a problem hiding this comment.
The --augment and --derive-variants flags are advertised in the CLI help as functional features ("Generate augmented variants after download", "Generate low-quality and corrupted variants"). However, when either flag is passed, the code only imports augment_main (which it never calls) and then just logs a message telling the user to call augment_audio.py separately. The augment_main variable is imported but never used. Users invoking --augment will expect augmentation to happen but nothing occurs, which is misleading.
| try: | |
| # Import augmentation script | |
| sys.path.insert(0, str(Path(__file__).parent.parent)) | |
| from augment_audio import main as augment_main | |
| # Note: This is a simplified integration | |
| # Full augmentation would require calling augment_audio with proper args | |
| LOGGER.info("Augmentation integration: Call augment_audio.py separately") | |
| LOGGER.info(" Example: python scripts/augment_audio.py --manifest-dir %s", args.manifest_dir) | |
| except ImportError as exc: | |
| LOGGER.warning("Could not import augment_audio: %s", exc) | |
| try: | |
| # Import augmentation script | |
| sys.path.insert(0, str(Path(__file__).parent.parent)) | |
| from augment_audio import main as augment_main | |
| # Invoke augmentation. The augment_audio.main function is expected | |
| # to handle its own argument parsing from sys.argv. | |
| LOGGER.info("Starting augmentation via augment_audio.main()") | |
| try: | |
| augment_main() | |
| except Exception as aug_exc: | |
| LOGGER.error("Augmentation failed: %s", aug_exc) | |
| LOGGER.error( | |
| "You may run augmentation manually, e.g.: " | |
| "python scripts/augment_audio.py --manifest-dir %s", | |
| args.manifest_dir, | |
| ) | |
| except ImportError as exc: | |
| LOGGER.warning("Could not import augment_audio: %s", exc) | |
| LOGGER.warning( | |
| "You can run augmentation manually, e.g.: " | |
| "python scripts/augment_audio.py --manifest-dir %s", | |
| args.manifest_dir, | |
| ) |
| dataset_type = self._infer_dataset_type(dataset_name, data_dir) | ||
|
|
||
| if dataset_type == "librispeech": | ||
| return librispeech.generate(data_dir, manifest_dir, force) | ||
| elif dataset_type == "musan": | ||
| return musan.generate(data_dir, manifest_dir, force) | ||
| elif dataset_type == "rirs": | ||
| return rirs.generate(data_dir, manifest_dir, force) | ||
| elif dataset_type == "st_aeds": | ||
| return st_aeds.generate(data_dir, manifest_dir, force) | ||
| else: | ||
| self.logger.warning( | ||
| "No manifest generator for dataset type: %s", dataset_type | ||
| ) | ||
| return [] |
There was a problem hiding this comment.
The tedlium3 dataset is registered in dataset_registry.yaml and will be downloaded when using --sources all or --sources openslr, but _infer_dataset_type correctly returns "tedlium" for it — a type for which there is no manifest generator (the generate_manifest method only handles "librispeech", "musan", "rirs", and "st_aeds"). The result is a silent warning with no manifest output. Either a tedlium manifest generator should be added, or the tedlium3 dataset should be noted as download-only in the registry/documentation.
| dest = (out_dir / member.name).resolve() | ||
| if not str(dest).startswith(str(out_dir_resolved)): | ||
| raise RuntimeError(f"Unsafe archive member: {member.name}") | ||
| tf.extract(member, out_dir) | ||
| extracted_count += 1 | ||
| elif archive_name.endswith(".zip"): | ||
| with zipfile.ZipFile(archive_path, mode="r") as zf: | ||
| for info in zf.infolist(): | ||
| dest = (out_dir / info.filename).resolve() | ||
| if not str(dest).startswith(str(out_dir_resolved)): | ||
| raise RuntimeError(f"Unsafe archive member: {info.filename}") | ||
| zf.extract(info, out_dir) |
There was a problem hiding this comment.
The path-traversal check on lines 350 and 358 uses str(dest).startswith(str(out_dir_resolved)), which has a well-known false-positive bypass: if out_dir_resolved is /tmp/data, a malicious archive entry with a resolved path of /tmp/data-evil/file would pass the check because "/tmp/data-evil/file".startswith("/tmp/data") is True. The check should use dest.is_relative_to(out_dir_resolved) (Python 3.9+) or compare dest.parts against out_dir_resolved.parts, or at minimum compare against str(out_dir_resolved) + os.sep.
| # region agent log | ||
| import json | ||
| from pathlib import Path as LogPath | ||
| log_data = {"hypothesisId": "A", "runId": "debug1", "location": "huggingface_plugin.py:92", "message": "Attempting HF load", "data": {"dataset_name": dataset_name, "config": dataset_config, "splits": splits}, "timestamp": int(__import__('time').time() * 1000)} | ||
| try: | ||
| with open(_debug_log_path('download'), 'a') as f: | ||
| f.write(json.dumps(log_data) + '\n') | ||
| except: pass | ||
| # endregion | ||
|
|
||
| dataset = load_dataset( | ||
| dataset_name, | ||
| name=dataset_config, | ||
| cache_dir=str(output_dir.parent / ".hf_cache"), | ||
| ) | ||
|
|
||
| # region agent log | ||
| log_data2 = {"hypothesisId": "A,B", "runId": "debug1", "location": "huggingface_plugin.py:110", "message": "HF load success", "data": {"dataset_name": dataset_name, "available_splits": list(dataset.keys()) if hasattr(dataset, 'keys') else []}, "timestamp": int(__import__('time').time() * 1000)} | ||
| try: | ||
| with open(_debug_log_path('download'), 'a') as f: | ||
| f.write(json.dumps(log_data2) + '\n') | ||
| except: pass | ||
| # endregion | ||
|
|
||
| except Exception as exc: | ||
| # region agent log | ||
| log_data3 = {"hypothesisId": "A,B,C", "runId": "debug1", "location": "huggingface_plugin.py:121", "message": "HF load failed", "data": {"dataset_name": dataset_name, "error_type": type(exc).__name__, "error_msg": str(exc)}, "timestamp": int(__import__('time').time() * 1000)} | ||
| try: | ||
| with open(_debug_log_path('download'), 'a') as f: | ||
| f.write(json.dumps(log_data3) + '\n') | ||
| except: pass | ||
| # endregion | ||
|
|
There was a problem hiding this comment.
The huggingface_plugin.py and openslr_plugin.py files contain extensive debug logging code that appears to be leftover development/agent artifacts. This includes:
_LOG_DIR,_SESSION_TS, and_debug_log_path()infrastructure in both files- Numerous
# region agent log/# endregionblocks containing dictionaries with"hypothesisId","runId": "debug1"fields - Repeated
import json,import os,import timeinside loops (inside these debug blocks) - Bare
except: passclauses silencing all exceptions on the file writes
These blocks should be removed before merging into production. They pollute the logs directory with debug JSON files on every run and use hardcoded "runId": "debug1" strings suggesting they were injected by a development agent. The bare except: pass clauses also suppress any I/O errors silently.
| # region agent log | |
| import json | |
| from pathlib import Path as LogPath | |
| log_data = {"hypothesisId": "A", "runId": "debug1", "location": "huggingface_plugin.py:92", "message": "Attempting HF load", "data": {"dataset_name": dataset_name, "config": dataset_config, "splits": splits}, "timestamp": int(__import__('time').time() * 1000)} | |
| try: | |
| with open(_debug_log_path('download'), 'a') as f: | |
| f.write(json.dumps(log_data) + '\n') | |
| except: pass | |
| # endregion | |
| dataset = load_dataset( | |
| dataset_name, | |
| name=dataset_config, | |
| cache_dir=str(output_dir.parent / ".hf_cache"), | |
| ) | |
| # region agent log | |
| log_data2 = {"hypothesisId": "A,B", "runId": "debug1", "location": "huggingface_plugin.py:110", "message": "HF load success", "data": {"dataset_name": dataset_name, "available_splits": list(dataset.keys()) if hasattr(dataset, 'keys') else []}, "timestamp": int(__import__('time').time() * 1000)} | |
| try: | |
| with open(_debug_log_path('download'), 'a') as f: | |
| f.write(json.dumps(log_data2) + '\n') | |
| except: pass | |
| # endregion | |
| except Exception as exc: | |
| # region agent log | |
| log_data3 = {"hypothesisId": "A,B,C", "runId": "debug1", "location": "huggingface_plugin.py:121", "message": "HF load failed", "data": {"dataset_name": dataset_name, "error_type": type(exc).__name__, "error_msg": str(exc)}, "timestamp": int(__import__('time').time() * 1000)} | |
| try: | |
| with open(_debug_log_path('download'), 'a') as f: | |
| f.write(json.dumps(log_data3) + '\n') | |
| except: pass | |
| # endregion | |
| dataset = load_dataset( | |
| dataset_name, | |
| name=dataset_config, | |
| cache_dir=str(output_dir.parent / ".hf_cache"), | |
| ) | |
| except Exception as exc: |
Introduces a modular data gathering system that replaces fragmented download scripts with a single, plugin-based pipeline.
Summary
scripts/gather_data.pyfor all dataset downloadsscripts/data_gatherer/dataset_registry.yaml--sources all(122GB, special setup); use--datasets voxpopuliwhen neededKey changes
scripts/data_gatherer/with plugins, manifest generators, and dataset registryscripts/augment_audio.pyfor MUSAN/RIRS augmentationscripts/data_gather.mdas the main data gathering guidedownload_datasets.pyandpreprocess_data.py