Skip to content

Add Modular Testing Suite and Test Harness for HealthLLM.jl - #54

Open
harshitchand998-ai wants to merge 7 commits into
JuliaHealth:mainfrom
harshitchand998-ai:test-harness
Open

harshitchand998-ai wants to merge 7 commits into
JuliaHealth:mainfrom
harshitchand998-ai:test-harness

Conversation

@harshitchand998-ai

Copy link
Copy Markdown

This PR implements Issue #32 by introducing a modular testing suite and test harness for HealthLLM.jl.

What changed

  • Added a modular test harness with reusable executors and validators.
  • Added exact, normalized text, content, structured, and execution-based validation.
  • Added mock LLM and RAG testing support for deterministic offline tests.
  • Added golden reference fixtures and synthetic test corpus.
  • Added unit tests for utilities, schemas, database functionality, and the harness itself.
  • Added query-generation, LLM, golden regression, FunSQL execution, and end-to-end RAG tests.
  • Updated test/runtests.jl to integrate the new test suites.
  • Added documentation describing the testing harness and testing workflow.
  • Extended query functionality required by the new test coverage.

Testing
The new test suite is designed to run primarily without external LLM APIs or PostgreSQL dependencies, while allowing live integrations to remain opt-in.

Closes #32

@codecov

codecov Bot commented Sep 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.78%. Comparing base (677f98e) to head (982aeb5).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #54   +/-   ##
=======================================
  Coverage   60.78%   60.78%           
=======================================
  Files           4        4           
  Lines         102      102           
=======================================
  Hits           62       62           
  Misses         40       40           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@harshitchand998-ai

Copy link
Copy Markdown
Author

@ParamThakkar123 can you pls review it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Pre-GSoC] Create Testing Suite Harness for HealthLLM.jl

1 participant