Fix HIE integration: per-instance pipelines, OpenCR matching, module patches, and facility identification#3
Merged
ibacher merged 52 commits intoApr 9, 2026
Conversation
mherman22
force-pushed
the
fix/pipeline-shr-integration
branch
from
6c963b8 to
49020e9
Compare
mherman22
changed the title
… config - Fix pipeline sink auth: use OpenHIM client `shr-pipeline` (with correct sha512 password hash) instead of API user `root@openhim.org`, which does not work on OpenHIM's HTTP transaction port (5001) - Update SHR image to `main` tag which includes the fix for double-serialized JSON responses in the bundle POST handler (see DIGI-UW/shared-health-record#112) - Add nginx reverse proxy config for the pipeline control panel, accessible at https://pipeline.<domain> instead of requiring port 8095 in the firewall - Add pipeline service to reverse-proxy network so nginx can reach it - Add HAPI FHIR post-deploy script to set `enforce_referential_integrity_on_write=false` and `auto_create_placeholder_reference_targets=true` — required because the pipeline pushes resources by type (not in dependency order) and the instant tooling does not apply dotted env vars from docker-compose.yml - Update README with Step 10b and Known Gotchas entry for the HAPI FHIR override
- Rewrite README.md as a single professional document with clear architecture diagram, data flow table, quick start guide, component reference, troubleshooting section, and security notes - Remove SETUP-GUIDE.md (content consolidated into README) - Add openhim_public network to all iSantePlus instances in docker-compose.yml so the mpi-client module can reach openhim-core:5001 to push patient identities to OpenCR
…ting The pipeline starts faster than iSantePlus (~10s vs ~10min). Without this fix, the first run hits a non-FHIR response, creates a corrupt DWH baseline, and all subsequent incremental runs find no new data. The wrapper script (wait-and-start.sh): 1. Polls the FHIR source /metadata endpoint every 10s (up to 10min) 2. Clears any stale DWH so the first run is always a full sync 3. Hands off to the original entrypoint This eliminates the need to manually trigger a full run after deployment.
…nfig Security: - Restrict SHR nginx proxy to GET/HEAD only (writes must go through OpenHIM) - Change OpenHIM /xds-order-notification channel from public to private auth - Remove DELETE method from OpenCR channel - Remove allow_multiple_delete from HAPI FHIR overrides - Remove exposed pipeline management ports (8095-8098) Reliability: - Add deploy section with restart_policy and memory limits to all 4 pipelines - Fix wait-and-start.sh to exit with error (not proceed) when FHIR source times out, so Docker Swarm restart policy handles retries - Fix wait-and-start.sh to read DWH prefix from env var override (was always clearing pipeline_DWH regardless of instance, breaking pipelines 2-4) - Stagger pipeline schedules (offsets 0/1/2/3 min) to avoid thundering herd - Reduce HikariCP pool size from 250 to 80 Per-instance iSantePlus configuration: - Add post-start.sh that runs after OpenMRS boots to set per-instance global properties (mpi-client.pid.local, xds-sender endpoints, password) - Inject post-start.sh into startup.sh via Dockerfile - Add ISANTEPLUS_INSTANCE env var per service in docker-compose.yml - This permanently fixes the OpenCR patient overwrite bug where all instances shared the same identifier URI
Without this, the SHR nginx proxy (shr.<domain>) gets a 502 because nginx cannot reach hapi-fhir_hapi-fhir on the reverse-proxy_public network. Previously this was applied via docker service update but was lost on every redeploy.
The previous Dockerfile injected `post-start.sh &` before `wait ${!}`.
Since ${!} captures the PID of the last background process, it waited
for post-start.sh (which exits after configuring properties) instead
of Tomcat. This caused all 4 iSantePlus containers to exit every ~7 min.
Fix: capture Tomcat's PID immediately after launching it (TOMCAT_PID=$!)
and wait on that specific PID instead of ${!}.
Also updates README with comprehensive redeployment procedures,
expected errors table, and post-deploy verification checklist.
Documents the 6-step process: MySQL database, OpenHIM client, docker-compose service, nginx proxy, data pipeline, and .env config. Explains why each step is needed (OpenCR patient identity matching) and includes the password hash generation command.
Root cause: all iSantePlus databases shared the same idgen sequence (next_sequence_value = 810022), so all instances generated identical patient IDs (e.g., 1000NG for the first patient). OpenCR matched patients by iSantePlus ID and merged them into one record. Fix: offset each database's idgen sequence by 1,000,000 per instance (openmrs=1M, openmrs2=2M, openmrs3=3M, openmrs4=4M) so IDs never collide across instances.
mherman22
force-pushed
the
fix/pipeline-shr-integration
branch
from
7b746a1 to
4c53178
Compare
mherman22
changed the title
mherman22
force-pushed
the
fix/pipeline-shr-integration
branch
from
90707b9 to
4c53178
Compare
Root cause: all iSantePlus databases shared the same idgen sequence, generating identical patient IDs (e.g., 1000NG) across all instances. OpenCR matched on these IDs and merged patients from different facilities. Fix: use the native OpenMRS idgen LocationBasedPrefixProvider to generate facility-specific patient IDs (e.g., HUEH1000NG, LAPAIX1000NG). Changes to SQL dump (isanteplus-db.sql): - idgen_seq_id_gen.prefix set to 'provider:LocationBasedPrefixProvider' - Added 'Facility ID Prefix' location attribute type (ID 12) - Added 'idgen.prefixLocationAttributeType' global property Changes to MySQL Dockerfile: - Now copies 20-configure-per-instance.sh into the image (was missing) Changes to 20-configure-per-instance.sh: - Simplified to only fix xds-sender endpoints (idgen now handled in dump) Changes to post-start.sh: - Sets the 'Facility ID Prefix' location attribute on the default location using the FACILITY_NAME env var (e.g., HUEH) - Simplified to remove multi-instance config that's no longer needed Changes to docker-compose.yml: - Added FACILITY_NAME=HUEH env var to isanteplus service References: - https://github.com/openmrs/openmrs-module-idgen/blob/master/api/src/main/java/org/openmrs/module/idgen/prefixprovider/LocationBasedPrefixProvider.java
- Container doesn't have python3 — use grep/sed for JSON parsing - Location API rejects v=ref parameter — removed it - Now correctly finds location UUID and sets Facility ID Prefix attribute
Replaced stale ISANTEPLUS_INSTANCE with FACILITY_NAME which drives the idgen LocationBasedPrefixProvider for facility-specific patient IDs: - isanteplus: HUEH - isanteplus2: LAPAIX - isanteplus3: OFATMA - isanteplus4: FSC
The LocationBasedPrefixProvider approach doesn't work with idgen 4.4.1 and prefixes with characters outside the Luhn Mod-30 set (no B,I,O,Q,S,Z) caused "Select a preferred identifier" errors on patient creation. New approach: each facility DB gets a unique idgen sequence offset (100k, 200k, 300k, ...) and a unique mpi-client.pid.local FHIR system URI so OpenCR distinguishes patient sources without needing ID prefixes. Changes: - Remove FACILITY_NAME env var from docker-compose and post-start.sh - Remove LocationBasedPrefixProvider, Facility ID Prefix attribute type, and idgen.prefixLocationAttributeType from SQL dump - Add sequence offset + unique mpi-client.pid.local per instance in 20-configure-per-instance.sh - Rewrite README with comprehensive troubleshooting guide covering SSL certs, iSantePlus boot issues, idgen config, OpenCR deduplication, and common operational procedures
…ve DirectRunner workaround Changed from: - DirectRunner (single-threaded workaround for credential serialization) - PostgreSQL JDBC sink To: - FlinkRunner (default, parallel processing) - HTTP FHIR-Search SOURCE with proper credentials - Source credentials: admin/Admin123 to iSantePlus FHIR API - Sink: Keep PostgreSQL JDBC to avoid sink-side credential issues Using volume mounts in docker-compose to ensure config changes are reflected.
Using HTTP sink for now (not PostgreSQL JDBC) to test if source credentials work. The instant CLI is providing these sink values, so align our config file with the deployed reality. Focus: Test if SOURCE HTTP credentials (admin/Admin123 to iSantePlus) work when using FlinkRunner, since openelis-openmrs-hie reports no serialization issues.
mherman22
changed the title
mherman22
force-pushed
the
fix/pipeline-shr-integration
branch
2 times, most recently
from
5acef10 to
5b83683
Compare
…nal image - Fix sink username: shr-pipeline (not root@openhim.org) per openhim-import.json - Add Location to resourceList to prevent referential integrity errors - Add hapi-postgres-config mount to docker-compose configs - Use original upstream fhir-data-pipes image (FlinkRunner works correctly) - Clean up stale comments about serialization workarounds
mherman22
force-pushed
the
fix/pipeline-shr-integration
branch
from
5b83683 to
deeca1f
Compare
Adds fhir-aggregator-mediator as an Instant OpenHIE package that fans out FHIR search requests to all iSantePlus instances, merges and deduplicates results, and presents a single endpoint to the pipeline. Package contents: - docker-compose.yml: Docker Swarm service with config mounts - swarm.sh: Instant CLI lifecycle (init/up/down/destroy) - package-metadata.json: Package descriptor with dependencies - config/config.json: Source definitions, pagination, performance - config/mediator.json: OpenHIM channel and mediator registration - build-image.sh: Builds Docker image from upstream repo Pipeline config change: - fhirServerUrl points to fhir-aggregator_fhir-aggregator:3000/fhir - No credentials needed (aggregator handles auth to each source) Mediator source: https://github.com/mherman22/fhir-aggregator-mediator
mherman22
force-pushed
the
fix/pipeline-shr-integration
branch
from
b39bdc2 to
9fda378
Compare
- New nginx config: pipeline.* routes to pipeline controller web UI - Add reverse-proxy_public network to pipeline docker-compose - Pipeline control panel accessible via https://pipeline.<domain>/
mherman22
force-pushed
the
fix/pipeline-shr-integration
branch
from
9fda378 to
0325026
Compare
All 4 instances competing for CPU/memory during simultaneous boot caused classloader failures and OOM. Now instances start one at a time, each waiting for HTTP 200 on /metadata before the next starts. - Scale all to 0 after stack deploy - Start each instance sequentially with health check (15 min timeout) - Auto-restart on HTTP 500 (classloader error recovery) - Instance list configurable via INSTANCES array
- Add deduplication: Patient by identifier (OpenMRS PID), default by resourceId - Add performance.requestTimeoutMs (120s Express-level timeout) - Set rejectUnauthorized: false (internal Docker network, no TLS between services)
The previous matching config had only 2 rules with strict exact-match
thresholds (jaro-winkler 1.0), which caused missed matches for:
- Patients with minor name typos ("Jean" vs "Jeen")
- Patients with swapped given/family names ("Jean Baptiste" vs "Baptiste Jean")
- Patients with birthdate entry errors (1985-03-15 vs 1985-03-51)
- Patients missing identifiers (no biometric, no national code)
- Patients with missing birthdates
New matching rules (evaluated in order, first match wins):
1. Biometric exact match — fingerprint (strongest identifier)
2. Code National exact match — national ID alone
3. Code National + full demographics — national ID confirmed by name/gender/DOB
4. iSantePlus ID + full demographics — facility ID confirmed by demographics
5. Exact demographics — name + gender + DOB (no identifier needed)
6. Swapped name detection — given↔family + gender + DOB
7. Fuzzy demographics — jaro-winkler 0.8 for names + DateDamerau 0.8 for DOB
8. Exact names + fuzzy birthdate — catches DOB typos with exact name
9. Code National + partial demographics — for patients with missing DOB
10. Phone + demographics — last resort for patients without identifiers
All rules use Haiti-specific identifier systems:
- http://isanteplus.org/openmrs/fhir2/6-biometrics-national-reference-code
- http://isanteplus.org/openmrs/fhir2/5-code-national
- http://isanteplus.org/openmrs/fhir2/3-isanteplus-id
Rules 5-10 use split thresholds:
- potentialMatchThreshold: 5 (flags for manual review)
- autoMatchThreshold: 6 (auto-links only when ALL fields match)
This prevents false auto-merges while still surfacing potential matches
for human review in the OpenCR dashboard.
4 tasks
…o fix/pipeline-shr-integration
8 tasks
Replaces xds-sender-2.5.0-SNAPSHOT.omod with 2.5.9 built from IsantePlus/openmrs-module-xds-sender fix/pull-notifications-ssl branch. Includes fix for SSLPeerUnverifiedException in PullNotificationsTask: when xdssender.exportCcd.ignoreCerts=true, the OkHttpClient now bypasses hostname verification — matching the existing behavior for CCD export. See: IsantePlus/openmrs-module-xds-sender#110 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Built from IsantePlus/openmrs-module-mpi-client fix/proxy-privileges branch (based on v1.1.4 tag). Fixes APIAuthenticationException: "Privileges required: Get Identifier Types" in PatientSyncWorker and PatientUpdateWorker by adding Context.addProxyPrivilege() for required privileges. See: IsantePlus/openmrs-module-mpi-client#58 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…fiers - Add isanteplus, shr-pipeline, openshr to OpenCR clients config so the CRUX UI can resolve Submitting System and Point of Service from the x-openhim-clientid header - Add Code PC (9-code-pc) and Biometrics National Reference Code FHIR system URI mappings to fhir_patient_identifier_system table so iSantePlus sends them with proper system URIs - Add codepc system entry to OpenCR config for UI label display
- Change exportIdentitiferType key from "Patient ID" to "iSantePlus ID" to match the actual OpenMRS identifier type name in the DB - Add iSante ID mapping to exportIdentitiferType - Remove ECID from autoXref (generated by CR, not a local identifier)
The SHR mediator queries OpenCR via OpenHIM for golden record resolution. The CR channel requires authentication. Add clientRegistryUsername/clientRegistryPassword using the isanteplus client which has the emr role allowed on the CR channel.
Replace the generic "isanteplus" OpenHIM client with per-facility clients (hueh, lapaix, ofatma, etc.) so OpenCR can identify which facility submitted each patient via the x-openhim-clientid header. - Add FACILITY_ID/FACILITY_NAME env vars to each iSantePlus service - Update post-start.sh to set xdssender.oshr.username to FACILITY_ID - Create per-facility OpenHIM clients with emr role - Add all facilities to OpenCR clients config for CRUX UI display - Update SHR mediator to use openshr client for CR queries
The decision rules used snake_case espath values (code_national, isanteplus_id) but Elasticsearch indexes these fields in camelCase (codeNational). This caused matching rules to never find any candidates, resulting in separate golden records for patients with the same Code National across facilities.
mherman22
force-pushed
the
fix/pipeline-shr-integration
branch
from
d72fe68 to
724a1d7
Compare
mherman22
changed the title
Replace the single pipeline (via FHIR aggregator) with one pipeline per iSantePlus instance, each pointing directly at its FHIR endpoint: - pipeline-isanteplus1 → http://isanteplus:8080/openmrs/ws/fhir2/R4 - pipeline-isanteplus2 → http://isanteplus2:8080/openmrs/ws/fhir2/R4 Both sink to the SHR via OpenHIM. Reduced heap from 2g to 1g per pipeline since each handles a single instance. Remove fhir-aggregator-mediator package entirely — not needed when each site has its own pipeline.
Pipeline-1 runs at :00, Pipeline-2 at :30. Prevents 409 Conflict errors from duplicate Practitioner/Location UUIDs across cloned iSantePlus databases when both pipelines write to the SHR simultaneously.
mherman22
changed the title
The pipelines are internal services that don't need external access. Remove the nginx proxy config, reverse-proxy network, and pipeline subdomain. Add terminal-based pipeline management instructions to the README including status checks, manual triggers, log inspection, and how to add pipelines for new instances.
Document the correct docker stack deploy order with dependency notes. Later packages depend on earlier ones (e.g., pipelines must be last because they need iSantePlus, OpenHIM, and HAPI FHIR running).
Document both deployment approaches with dependency ordering: - Instant CLI (recommended): handles env var injection, requires build-image.sh after local changes - Manual docker stack deploy: uses local files directly, useful for development and debugging
The mpi-client uses sendingApplication as the Basic auth username and authtoken as the password when talking to OpenCR via OpenHIM. These were set to the generic 'isanteplus' which no longer exists as an OpenHIM client (replaced with per-facility clients). Update post-start.sh to set both mpi-client.msg.sendingApplication and mpi-client.security.authtoken to FACILITY_ID.
The per-facility client hashes were generated with sha512(salt + password) but OpenHIM uses sha512(password + salt). This caused 401 Unauthorized on all mpi-client and xds-sender requests from iSantePlus to OpenHIM. Also fix importer password back to instant101 for environments with persisted MongoDB data.
Author
|
Hey @ibacher here is the test SHR instance with some data https://shr.sedishtest.live/fhir/Patient, i wanted you to have a look at especially the For the instances i am deploying these changes, see https://uwdigi.atlassian.net/wiki/spaces/HAIT/pages/931299355/Roaming+Care+Overview Also feel free to review this and perhaps merge it so that i can start on a clean PR for the followup commit on your comments here. |
mherman22
force-pushed
the
fix/pipeline-shr-integration
branch
2 times, most recently
from
cd35f11 to
41856d9
Compare
Code PC (patient_identifier_type_id=9) was missing from the fhir_patient_identifier_system table, causing it to appear in FHIR/OpenCR without a system URI or label. Map it to http://isanteplus.org/openmrs/fhir2/9-code-pc. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These files are tracked by Git LFS — the prior commit incorrectly modified the LFS pointer instead of the actual content. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Built from mherman22/openmrs-module-mpi-client fix/proxy-privileges branch rebased on main, which includes: - Remove getPatient() trigger from MPI sync advice (DIGI-UW#56) - Skip MPI sync when CR endpoint is not configured (DIGI-UW#57) - Add proxy privileges to PatientSyncWorker and PatientUpdateWorker (DIGI-UW#58) Fixes: importing a patient from the MPI no longer creates duplicate local patients from golden record seealso links. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Includes: - Implement FhirMpiClientServiceImpl.importPatient() with duplicate check - Add deduplication guard to PatientUpdateWorker to prevent ghost exports Fixes: IsantePlus/openmrs-module-mpi-client#59 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ibacher
approved these changes
Apr 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes the end-to-end data flow across the Sedish Haiti HIE — from iSantePlus EMR instances through OpenHIM, the SHR mediator, HAPI FHIR, and the OpenCR Client Registry. The goal is to enable the roaming care workflow: a patient registered at one facility can be identified and their full record retrieved at any other facility via the golden record in the Client Registry.
FHIR Data Pipeline — per-instance architecture
Each iSantePlus instance now gets its own dedicated pipeline instead of routing through a shared FHIR aggregator mediator.
Files changed:
Decision: Pipelines are staggered (pipeline 1 at :00, pipeline 2 at :30) to avoid 409 Conflict errors from shared Practitioner/Location UUIDs across cloned databases.
Sink credentials: Changed from `root@openhim.org` to `shr-pipeline` with `instant101`.
Resource order: `Practitioner,Location,Patient,Encounter,Observation,...` — referenced resources must sync before resources that reference them to satisfy HAPI FHIR's referential integrity enforcement.
iSantePlus EMR
Files changed:
Decision: The base image's `startup.sh` uses `wait${!}` which waits for the last background process (post-start.sh) instead of Tomcat. When post-start.sh exits, the container dies. The Dockerfile patches this to capture `TOMCAT_PID=$ !` and use `wait $TOMCAT_PID`.
OpenHIM Interoperability Layer
Files changed:
Decision: Per-facility clients enable OpenCR to identify the Point of Service (which facility submitted the patient) via the `x-openhim-clientid` header. OpenCR stamps this as a `meta.tag` on the Patient resource and displays it in the CRUX UI. The password hash order (`password + salt`, not `salt + password`) matches OpenHIM's implementation — getting this wrong causes silent 401s.
Shared Health Record (SHR) Mediator
Files changed:
Client Registry (OpenCR)
Files changed:
Decision: The `espath` mismatch was subtle — the decision rules used snake_case (`code_national`) but OpenCR's ES indexing uses camelCase (`codeNational`). This caused deterministic matching rules to silently fail. The fix was a one-character change per field but had a large impact on matching accuracy.
HAPI FHIR Datastore
Files changed:
Database (MySQL)
Files changed:
Reverse Proxy (Nginx)
Files changed:
README
Complete rewrite covering architecture, deployment (both Instant CLI and manual docker stack), pipeline management from terminal, adding new instances, SSL certificates, troubleshooting, and OpenHIM password management.
Test results