Feature/fhir api gateway (#128)

* Refactor sandbox to separate module

* Fix tests

* Gateway module WIP

* Update poetry.lock

* Add fhirclient, bump python min to 3.9, update poetry

* Update module structure

* Use consistent pydantic models to silence serialization warning

* Fix bugs in configs

* Fix code snippet in docs

* Migrating service module to gateway/services

* Implement cdshooks and notereader services plus clean up on base and events

* Remove async from cdshooks

* Update relatesTo access

* Add fields to CdsRequest

* Update sandbox usage and tests

* Update tests for gateway module

* Remove scrap

* Add HealthChainAPI class and FhirRouter placeholder

* Update poetry.lock

* Update CI python version

* Fix typo

* Pass test

* Fix namespace conflict

* Fix patching issue in tests for python 3.10

* Fix pydantic to <2.11

* Tidy up structure

* Added event dispatch and unified everything to gateways

* Update dependencies

* Added dependency injection and protocol with tests

* Deprecate Service module

* Fix tests

* Chaotic WIP

* Added connection pool management

* Add validation of response

* Add handler validation and read only method WIP

* poetry.lock

* Only run CI on non-draft PRs

* Simpler FHIR client implementation

* Update poetry - remove fhirclient

* Update dependencies

* Added oauth2.0 flow and dynamic jwt token management

* Added tests for client and auth

* Update connection pool impl and fhir methods

* Update tests for client and pool

* Refactor cdshooks and notereader as services instead of gateways

* Add validation for client secret path in auth and add tests

* Add FHIR error handling to methods

* Refactor FHIRGateway to /core and clean up base class + separate connection and error handling

* Remove result validation and use strict type[Resource] inputs

* Move connection string methods to auth config and removed unused methods

* Remove debug log

* Refactor event dispatch

* Clean up tests

* Minor refactor

* Add crud operations to fhirgateway

* Add metadata and status endpoints in fhirgateway

* Clean up protocols and event dispatcher

* Refactor event emission

* Update tests

* Refactor HealthChainAPI lifespan and route management

* Refactor EventDispatcher and tidy some tests

* Fix pass kwargs to client

* Potential fix for code scanning alert no. 9: Incomplete URL substring sanitization

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Test fix for python3.9

* 3.9 Event loop fix

* Update .gitignore

* Use asyncio in test

* Use asyncio instead of anyio

* Remove unused dependencies

* Add gateway reference to docs

* Update README.md

* Update docs

* Remove security and monitoring modules from gateway (moved to separate branch)

* Clean up sandbox docs

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Author: jenniferjiangkells

.github/workflows/ci.yml

@@ -8,9 +8,11 @@ on:
     branches: [ "main" ]
   pull_request:
     branches: [ "main" ]
+    types: [opened, synchronize, reopened, ready_for_review]
 
 jobs:
   test:
+    if: github.event.pull_request.draft == false
     strategy:
       matrix:
         python-version: ["3.9", "3.10", "3.11"]

.gitignore

@@ -165,3 +165,5 @@ scrap/
 .vscode/
 .ruff_cache/
 .python-version
+.cursor/
+scripts/

README.md

@@ -10,32 +10,115 @@
 
 </div>
 
-Build simple, portable, and scalable AI and NLP applications in a healthcare context 💫 🏥.
+Connect your AI models to any healthcare system with a few lines of Python 💫 🏥.
+
+Integrating AI with electronic health records (EHRs) is complex, manual, and time-consuming. Let's try to change that.
 
-Integrating electronic health record systems (EHRs) data is complex, and so is designing reliable, reactive algorithms involving unstructured healthcare data. Let's try to change that.
 
 ```bash
 pip install healthchain
 ```
 First time here? Check out our [Docs](https://dotimplement.github.io/HealthChain/) page!
 
-Came here from NHS RPySOC 2024 ✨?
-[CDS sandbox walkthrough](https://dotimplement.github.io/HealthChain/cookbook/cds_sandbox/)
-[Slides](https://speakerdeck.com/jenniferjiangkells/building-healthcare-context-aware-applications-with-healthchain)
 
 ## Features
-- [x] 🔥 Build FHIR-native pipelines or use [pre-built ones](https://dotimplement.github.io/HealthChain/reference/pipeline/pipeline/#prebuilt) for your healthcare NLP and ML tasks
-- [x] 🔌 Connect pipelines to any EHR system with built-in [CDA and FHIR Connectors](https://dotimplement.github.io/HealthChain/reference/pipeline/connectors/connectors/)
-- [x] 🔄 Convert between FHIR, CDA, and HL7v2 with the [InteropEngine](https://dotimplement.github.io/HealthChain/reference/interop/interop/)
-- [x] 🧪 Test your pipelines in full healthcare-context aware [sandbox](https://dotimplement.github.io/HealthChain/reference/sandbox/sandbox/) environments
-- [x] 🗃️ Generate [synthetic healthcare data](https://dotimplement.github.io/HealthChain/reference/utilities/data_generator/) for testing and development
-- [x] 🚀 Deploy sandbox servers locally with [FastAPI](https://fastapi.tiangolo.com/)
+- [x] 🔌 **Gateway**: Connect to multiple EHR systems with [unified API](https://dotimplement.github.io/HealthChain/reference/gateway/gateway/) supporting FHIR, CDS Hooks, and SOAP/CDA protocols
+- [x] 🔥 **Pipelines**: Build FHIR-native ML workflows or use [pre-built ones](https://dotimplement.github.io/HealthChain/reference/pipeline/pipeline/#prebuilt) for your healthcare NLP and AI tasks
+- [x] 🔄 **InteropEngine**: Convert between FHIR, CDA, and HL7v2 with a [template-based engine](https://dotimplement.github.io/HealthChain/reference/interop/interop/)
+- [x] 🔒 Type-safe healthcare data with full type hints and Pydantic validation for [FHIR resources](https://dotimplement.github.io/HealthChain/reference/utilities/fhir_helpers/)
+- [x] ⚡ Event-driven architecture with real-time event handling and [audit trails](https://dotimplement.github.io/HealthChain/reference/gateway/events/) built-in
+- [x] 🚀 Deploy production-ready applications with [HealthChainAPI](https://dotimplement.github.io/HealthChain/reference/gateway/api/) and FastAPI integration
+- [x] 🧪 Generate [synthetic healthcare data](https://dotimplement.github.io/HealthChain/reference/utilities/data_generator/) and [sandbox testing](https://dotimplement.github.io/HealthChain/reference/sandbox/sandbox/) utilities
 
 ## Why use HealthChain?
--  **EHR integrations are manual and time-consuming** - HealthChain abstracts away complexities so you can focus on AI development, not EHR configurations.
--  **It's difficult to track and evaluate multiple integration instances** - HealthChain provides a framework to test the real-world resilience of your whole system, not just your models.
--  [**Most healthcare data is unstructured**](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6372467/) - HealthChain is optimized for real-time AI and NLP applications that deal with realistic healthcare data.
-- **Built by health tech developers, for health tech developers** - HealthChain is tech stack agnostic, modular, and easily extensible.
+-  **EHR integrations are manual and time-consuming** - **HealthChainAPI** abstracts away complexities so you can focus on AI development, not learning FHIR APIs, CDS Hooks, and authentication schemes.
+-  **Healthcare data is fragmented and complex** - **InteropEngine** handles the conversion between FHIR, CDA, and HL7v2 so you don't have to become an expert in healthcare data standards.
+-  [**Most healthcare data is unstructured**](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6372467/) - HealthChain **Pipelines** are optimized for real-time AI and NLP applications that deal with realistic healthcare data.
+- **Built by health tech developers, for health tech developers** - HealthChain is tech stack agnostic, modular, and easily extensible with built-in compliance and audit features.
+
+## HealthChainAPI
+
+The HealthChainAPI provides a secure, asynchronous integration layer that coordinates multiple healthcare systems in a single application.
+
+### Multi-Protocol Support
+
+Connect to multiple healthcare data sources and protocols:
+
+```python
+from healthchain.gateway import (
+    HealthChainAPI, FHIRGateway,
+    CDSHooksService, NoteReaderService
+)
+
+# Create your healthcare application
+app = HealthChainAPI(
+    title="My Healthcare AI App",
+    description="AI-powered patient care platform"
+)
+
+# FHIR for patient data from multiple EHRs
+fhir = FHIRGateway()
+fhir.add_source("epic", "fhir://fhir.epic.com/r4?client_id=...")
+fhir.add_source("medplum", "fhir://api.medplum.com/fhir/R4/?client_id=...")
+
+# CDS Hooks for real-time clinical decision support
+cds = CDSHooksService()
+
+@cds.hook("patient-view", id="allergy-alerts")
+def check_allergies(request):
+    # Your AI logic here
+    return {"cards": [...]}
+
+# SOAP for clinical document processing
+notes = NoteReaderService()
+
+@notes.method("ProcessDocument")
+def process_note(request):
+    # Your NLP pipeline here
+    return processed_document
+
+# Register everything
+app.register_gateway(fhir)
+app.register_service(cds)
+app.register_service(notes)
+
+# Your API now handles:
+# /fhir/* - Patient data, observations, etc.
+# /cds/* - Real-time clinical alerts
+# /soap/* - Clinical document processing
+```
+
+### FHIR Operations with AI Enhancement
+
+```python
+from healthchain.gateway import FHIRGateway
+from fhir.resources.patient import Patient
+
+gateway = FHIRGateway()
+gateway.add_source("epic", "fhir://fhir.epic.com/r4?...")
+
+# Add AI transformations to FHIR data
+@gateway.transform(Patient)
+async def enhance_patient(id: str, source: str = None) -> Patient:
+    async with gateway.modify(Patient, id, source) as patient:
+        # Get lab results and process with AI
+        lab_results = await gateway.search(
+            Observation,
+            {"patient": id, "category": "laboratory"},
+            source
+        )
+        insights = nlp_pipeline.process(patient, lab_results)
+
+        # Add AI summary to patient record
+        patient.extension = patient.extension or []
+        patient.extension.append({
+            "url": "http://healthchain.org/fhir/summary",
+            "valueString": insights.summary
+        })
+        return patient
+
+# Automatically available at: GET /fhir/transform/Patient/123?source=epic
+```
 
 ## Pipeline
 Pipelines provide a flexible way to build and manage processing pipelines for NLP and ML tasks that can easily integrate with complex healthcare systems.
@@ -139,116 +222,40 @@ cda_data = engine.from_fhir(fhir_resources, dest_format=FormatType.CDA)
 
 ## Sandbox
 
-Sandboxes provide a staging environment for testing and validating your pipeline in a realistic healthcare context.
-
-### Clinical Decision Support (CDS)
-[CDS Hooks](https://cds-hooks.org/) is an [HL7](https://cds-hooks.hl7.org) published specification for clinical decision support.
-
-**When is this used?** CDS hooks are triggered at certain events during a clinician's workflow in an electronic health record (EHR), e.g. when a patient record is opened, when an order is elected.
-
-**What information is sent**: the context of the event and [FHIR](https://hl7.org/fhir/) resources that are requested by your service, for example, the patient ID and information on the encounter and conditions they are being seen for.
-
-**What information is returned**: “cards” displaying text, actionable suggestions, or links to launch a [SMART](https://smarthealthit.org/) app from within the workflow.
-
+Test your AI applications in realistic healthcare contexts with [CDS Hooks](https://cds-hooks.org/) sandbox environments.
 
 ```python
 import healthchain as hc
-
-from healthchain.pipeline import SummarizationPipeline
 from healthchain.sandbox.use_cases import ClinicalDecisionSupport
-from healthchain.models import Card, Prefetch, CDSRequest
-from healthchain.data_generator import CdsDataGenerator
-from typing import List
 
 @hc.sandbox
 class MyCDS(ClinicalDecisionSupport):
-    def __init__(self) -> None:
-        self.pipeline = SummarizationPipeline.from_model_id(
-            "facebook/bart-large-cnn", source="huggingface"
-        )
-        self.data_generator = CdsDataGenerator()
+    def __init__(self):
+        self.pipeline = SummarizationPipeline.from_model_id("facebook/bart-large-cnn")
 
-    # Sets up an instance of a mock EHR client of the specified workflow
     @hc.ehr(workflow="encounter-discharge")
-    def ehr_database_client(self) -> Prefetch:
+    def ehr_database_client(self):
         return self.data_generator.generate_prefetch()
 
-    # Define your application logic here
-    @hc.api
-    def my_service(self, data: CDSRequest) -> CDSRequest:
-        result = self.pipeline(data)
-        return result
-```
-
-### Clinical Documentation
-
-The `ClinicalDocumentation` use case implements a real-time Clinical Documentation Improvement (CDI) service. It helps convert free-text medical documentation into coded information that can be used for billing, quality reporting, and clinical decision support.
-
-**When is this used?** Triggered when a clinician opts in to a CDI functionality (e.g. Epic NoteReader) and signs or pends a note after writing it.
-
-**What information is sent**: A [CDA (Clinical Document Architecture)](https://www.hl7.org.uk/standards/hl7-standards/cda-clinical-document-architecture/) document which contains continuity of care data and free-text data, e.g. a patient's problem list and the progress note that the clinician has entered in the EHR.
-
-```python
-import healthchain as hc
-
-from healthchain.pipeline import MedicalCodingPipeline
-from healthchain.sandbox.use_cases import ClinicalDocumentation
-from healthchain.models import CdaRequest, CdaResponse
-from fhir.resources.documentreference import DocumentReference
-
-@hc.sandbox
-class NotereaderSandbox(ClinicalDocumentation):
-    def __init__(self):
-        self.pipeline = MedicalCodingPipeline.from_model_id(
-            "en_core_sci_md", source="spacy"
-        )
-
-    # Load an existing CDA file
-    @hc.ehr(workflow="sign-note-inpatient")
-    def load_data_in_client(self) -> DocumentReference:
-        with open("/path/to/cda/data.xml", "r") as file:
-            xml_string = file.read()
-
-        cda_document_reference = create_document_reference(
-            data=xml_string,
-            content_type="text/xml",
-            description="Original CDA Document loaded from my sandbox",
-        )
-        return cda_document_reference
-
-    @hc.api
-    def my_service(self, data: CdaRequest) -> CdaResponse:
-        annotated_ccd = self.pipeline(data)
-        return annotated_ccd
-```
-### Running a sandbox
-
-Ensure you run the following commands in your `mycds.py` file:
-
-```python
 cds = MyCDS()
 cds.start_sandbox()
-```
-This will populate your EHR client with the data generation method you have defined, send requests to your server for processing, and save the data in the `./output` directory.
 
-Then run:
-```bash
-healthchain run mycds.py
+# Run with: healthchain run mycds.py
 ```
-By default, the server runs at `http://127.0.0.1:8000`, and you can interact with the exposed endpoints at `/docs`.
 
 ## Road Map
-- [x] 🔄 Transform and validate healthcare HL7v2, CDA to FHIR with template-based interop engine
-- [ ] 🏥 Runtime connection health and EHR integration management - connect to FHIR APIs and legacy systems
+- [ ] 🔒 Built-in HIPAA compliance validation and PHI detection
 - [ ] 📊 Track configurations, data provenance, and monitor model performance with MLFlow integration
 - [ ] 🚀 Compliance monitoring, auditing at deployment as a sidecar service
-- [ ] 🔒 Built-in HIPAA compliance validation and PHI detection
-- [ ] 🧠 Multi-modal pipelines that that have built-in NLP to utilize unstructured data
+- [ ] 🔄 HL7v2 parsing and FHIR profile conversion support
+- [ ] 🧠 Multi-modal pipelines
+
 
 ## Contribute
 We are always eager to hear feedback and suggestions, especially if you are a developer or researcher working with healthcare systems!
 - 💡 Let's chat! [Discord](https://discord.gg/UQC6uAepUz)
 - 🛠️ [Contribution Guidelines](CONTRIBUTING.md)
 
-## Acknowledgement
-This repository makes use of [fhir.resources](https://github.com/nazrulworld/fhir.resources), and [CDS Hooks](https://cds-hooks.org/) developed by [HL7](https://www.hl7.org/) and [Boston Children’s Hospital](https://www.childrenshospital.org/).
+
+## Acknowledgements 🤗
+This project builds on [fhir.resources](https://github.com/nazrulworld/fhir.resources) and [CDS Hooks](https://cds-hooks.org/) standards developed by [HL7](https://www.hl7.org/) and [Boston Children's Hospital](https://www.childrenshospital.org/).

docs/assets/images/openapi_docs.png

@@ -1,6 +1,6 @@
-# Welcome to HealthChain
+# Welcome to HealthChain 💫 🏥
 
-HealthChain 💫🏥 is an open-source Python framework designed to streamline the development, testing, and validation of AI, Natural Language Processing, and Machine Learning applications in a healthcare context.
+HealthChain is an open-source Python framework for building real-time AI applications in a healthcare context.
 
 [ :fontawesome-brands-discord: Join our Discord](https://discord.gg/UQC6uAepUz){ .md-button .md-button--primary }
     
@@ -19,19 +19,19 @@ HealthChain 💫🏥 is an open-source Python framework designed to streamline t
 
     [:octicons-arrow-right-24: Pipeline](reference/pipeline/pipeline.md)
 
--   :octicons-beaker-24:{ .lg .middle } __Test in a sandbox__
+-   :material-connection:{ .lg .middle } __Connect to multiple data sources__
 
     ---
 
-    Test your models in a full health-context aware environment from day 1
+    Connect to multiple healthcare data sources and protocols with **HealthChainAPI**.
 
-    [:octicons-arrow-right-24: Sandbox](reference/sandbox/sandbox.md)
+    [:octicons-arrow-right-24: Gateway](reference/gateway/gateway.md)
 
 -   :material-database:{ .lg .middle } __Interoperability__
 
     ---
 
-    Configuration-driven InteropEngine to convert between FHIR, CDA, and HL7v2
+    Configuration-driven **InteropEngine** to convert between FHIR, CDA, and HL7v2
 
     [:octicons-arrow-right-24: Interoperability](reference/interop/interop.md)
 
@@ -49,16 +49,17 @@ HealthChain 💫🏥 is an open-source Python framework designed to streamline t
 
 ## Why HealthChain?
 
-You've probably heard every *AI will revolutionize healthcare* pitch by now, but if you're one of the people who think: wait, can we go beyond just vibe-checking and *actually* build products that are reliable, reactive, and easy to scale in complex healthcare systems? Then HealthChain is probably for you.
+Healthcare AI development has a **missing middleware layer**. Traditional enterprise integration engines move data around, EHR platforms serve end users, but there's nothing in between for developers building AI applications that need to talk to multiple healthcare systems. Few solutions are open-source, and even fewer are built in modern Python where most ML/AI libraries thrive.
 
-Specifically, HealthChain addresses two challenges:
+HealthChain fills that gap with:
 
-1. **Scaling Electronic Health Record system (EHRs) integrations of real-time AI, NLP, and ML applications is a manual and time-consuming process.**
+- **🔥 FHIR-native ML pipelines** - Pre-built NLP/ML pipelines optimized for structured / unstructured healthcare data, or build your own with familiar Python libraries such as 🤗 Hugging Face, 🤖 LangChain, and 📚 spaCy
+- **🔒 Type-safe healthcare data** - Full type hints and Pydantic validation for FHIR resources with automatic data validation and error handling
+- **🔌 Multi-protocol connectivity** - Handle FHIR, CDS Hooks, and SOAP/CDA in the same codebase with OAuth2 authentication and connection pooling
+- **⚡ Event-driven architecture** - Real-time event handling with audit trails and workflow automation built-in
+- **🔄 Built-in interoperability** - Convert between FHIR, CDA, and HL7v2 using a template-based engine
+- **🚀 Production-ready deployment** - FastAPI integration for scalable, real-time applications
 
-2. **Testing and evaluating unstructured data in complex, outcome focused systems is a challenging and labour-intensive task.**
-
-We believe more efficient end-to-end pipeline and integration testing at an early stage in development will give you back time to focus on what actually matters: developing safer, more effective and more explainable models that scale to real-world *adoption*. Building products for healthcare in a process that is *human*-centric.
-
-HealthChain is made by a (very) small team with experience in software engineering, machine learning, and healthcare NLP. We understand that good data science is about more than just building models, and that good engineering is about more than just building systems. This rings especially true in healthcare, where people, processes, and technology all play a role in making an impact.
+HealthChain is made by a small team with experience in software engineering, machine learning, and healthcare NLP. We understand that good data science is about more than just building models, and that good engineering is about more than just building systems. This rings especially true in healthcare, where people, processes, and technology all play a role in making an impact.
 
 For inquiries and collaborations, please get [in touch](mailto:[email protected])!