diff --git a/docs/fhir/fhir-api.md b/docs/fhir/fhir-api.md index bcc9094b8..4963191f0 100644 --- a/docs/fhir/fhir-api.md +++ b/docs/fhir/fhir-api.md @@ -68,38 +68,120 @@ in the QBP message. #### Including additional resources -By default the Bundle contains only the requested resource type (e.g., `Immunization` -records) and any warnings. Use `_include` to add related resources. +**You get only what you ask for.** By default the Bundle contains the requested resource type +(e.g. `Immunization` records) and any warnings, and nothing else. The subject `Patient`, the +administering `Practitioner` and `Location`, the `Organization` behind +`ImmunizationRecommendation.authority`, and the evaluated-history `Immunization` on a recommendation +query are all **absent** unless you ask for them with `_include` or `_revinclude`. + +A reference to a resource you did not ask for is still delivered in full — the `reference` value the +conversion produced, plus `identifier` and `display` where it has them. This service serves no read +endpoint for a reference target, so resolve it either by asking for the target with `_include`, or +against your own data using the `identifier` and `display`. + +**Use case: I want immunization records, the patient, and the administering provider** + +``` +GET /fhir/{destinationId}/Immunization?family=Smith&given=John&birthdate=2000-01-01 + &_include=Immunization:patient + &_include=Immunization:performer +``` + +Each `_include` is required. Without them you get the `Immunization` records alone. **Use case: I want everything — immunization records and all referenced resources** ``` GET /fhir/{destinationId}/Immunization?family=Smith&given=John&birthdate=2000-01-01 &_include=*:* - &_revinclude=Provenance:target + &_include=Resource:source:* ``` -`_include=*:*` follows all references from all returned resources. Adding -`_revinclude=Provenance:target` also includes a `Provenance` record for each result -identifying the IIS as the data source. +`_include=*:*` follows all references from all returned resources, transitively. It reaches only +resources a returned entry points **at**; add `_include=Resource:source:*` for the conversion-created +resources that nothing references (see +[Conversion-created resources](rsp-to-fhir.md#conversion-created-resources)). -**Use case: I want immunization records and the patient only** +**Use case: I want the forecast plus the evaluated history it was computed from** ``` -GET /fhir/{destinationId}/Immunization?family=Smith&given=John&birthdate=2000-01-01 - &_include=Immunization:patient +GET /fhir/{destinationId}/ImmunizationRecommendation?... + &_include=ImmunizationRecommendation:patient + &_revinclude=Immunization:patient + &_include=Immunization:authority ``` -**Use case: I want immunization records and the administering provider or organization** +The evaluated-history `Immunization` carry `protocolApplied.doseNumber`, +`protocolApplied.seriesDoses`, `protocolApplied.authority` and `programEligibility`, which the +`/Immunization` query cannot return because the Z32 response behind it does not carry the source OBX +segments. All three parameters matter: + +- `_include=ImmunizationRecommendation:patient` is **not optional**. The `Immunization` reference the + `Patient`, not the recommendation, so a `_revinclude` has nothing to resolve from until the + `Patient` is in the Bundle. +- `_include=Immunization:authority` is what brings in the schedule `Organization` that + `protocolApplied.authority` points at — it is registered on the `Immunization`, not on the + `ImmunizationRecommendation`. + +Filter on `search.mode = match` for the forecast alone. + +#### Migrating from the previous behavior + +Earlier builds returned referenced resources, and the evaluated history on a recommendation query, +without being asked. To get that Bundle back: ``` -GET /fhir/{destinationId}/Immunization?family=Smith&given=John&birthdate=2000-01-01 - &_include=Immunization:patient - &_include=Immunization:performer +GET /fhir/{destinationId}/ImmunizationRecommendation?...&_include=*:*&_revinclude=Immunization ``` +That returns the same entries, with one difference: a reference the conversion built without a +resource behind it — a `PractitionerRole` pointing at a `Practitioner`, for example — now keeps its +literal `reference` value instead of being reduced to `identifier` and `display`. No `_include` +recovers such a target, because there is no resource to return. + +`_include=*:*` alone is not enough on a recommendation query: the evaluated history is reachable only +in reverse, which is why the `_revinclude` is there. + > In the response Bundle, directly matched resources have `search.mode = match`; > included resources have `search.mode = include`; warnings have `search.mode = outcome`. +> Select `search.mode = match` to get just the resources you queried for. + +#### Search parameter names + +An `_include` or `_revinclude` naming a search path the conversion never registered matches nothing +and is not an error, so a typo fails silently. The names below are the ones useful for immunization +queries. They are not the whole set — the conversion registers a name per reference it builds, and +also registers `subject`, `practitioner`, `organization`, `encounter`, `device`, `focus`, `target`, +`information` and others on the resources that carry those references. + +| Parameter | Reaches | +|---|---| +| `_include=ImmunizationRecommendation:patient` | the subject `Patient` | +| `_include=ImmunizationRecommendation:authority` | the `Organization` the forecast cites | +| `_include=Immunization:patient` | the subject `Patient` | +| `_include=Immunization:authority` | the `Organization` behind `protocolApplied.authority` | +| `_include=Immunization:location` | the administering `Location` | +| `_include=Immunization:performer` | the administering `Practitioner` / `PractitionerRole` | +| `_include=Immunization:manufacturer` | the vaccine manufacturer `Organization` | +| `_revinclude=Observation` | the dose and forecast `Observation` (needs an anchor, see below) | +| `_revinclude=Observation:part-of` | only the `Observation` linked to a dose (needs an anchor) | +| `_revinclude=Immunization` | the evaluated-history doses (needs an anchor) | +| `_include=Resource:source:` | conversion-created resources of that type, or `*` for all | + +`_include` follows references forward; `_revinclude` finds resources pointing **at** something already +in the Bundle, so it needs a forward `_include` first unless the requested type is itself the target. +Any resource already in the Bundle can serve as that anchor, including one retained by +`_include=Resource:source:...` — a white-listed resource is walked like any other. + +One known gap: a **type-qualified** `_revinclude` matches on the resource type carried by the +reference, and the conversion gives `Provenance` an id with no type on it, so `_revinclude=Provenance` +matches nothing. The wildcard form and the white-list both reach it: + +``` +_include=Resource:source:DocumentReference&_revinclude=*:* returns the Provenance +_include=Resource:source:Provenance returns the Provenance +_revinclude=Provenance returns nothing +``` #### Response diff --git a/docs/fhir/rsp-to-fhir.md b/docs/fhir/rsp-to-fhir.md index 617263fa8..64f621221 100644 --- a/docs/fhir/rsp-to-fhir.md +++ b/docs/fhir/rsp-to-fhir.md @@ -20,10 +20,133 @@ conversion, `FhirController` performs two post-processing steps: [Resource ID Design](index.md#resource-id-design)). 2. **`filter`** — reduces the bundle to the requested resource type (plus any resources requested via `_include` / `_revinclude`), sets `Bundle.type = searchset`, and marks - each entry with `search.mode`. + each entry with `search.mode`. See [Searchset entries](#searchset-entries) below. The segment parsers that produce each resource type are listed below. +### Searchset entries + +Every entry in a returned bundle carries a `search.mode`; anything that would not get one is +removed. + +| `search.mode` | What it means | +|---|---| +| `match` | A resource of the type the query asked for. Clients select these to get the hits. | +| `include` | A resource the client asked for with `_include` / `_revinclude`, or white-listed with `_include=Resource:source:...`. | +| `outcome` | An `OperationOutcome` reporting a conversion warning or error. | + +Three rules govern what a bundle holds: + +- **You get only what you asked for.** The bundle holds the requested type, `OperationOutcome` + entries, and whatever an `_include` / `_revinclude` reached. Nothing else. A resource is **not** + returned because a returned entry references it, and not because the RSP message happened to carry + it. The shape of a response is predictable from the query alone. +- **Only the requested type is `match`.** A `GET /ImmunizationRecommendation` never labels an + `Immunization` as `match`, and vice versa. Joined resources are `include`, per the R4 definition + of a resource "added to the results because of a join". +- **A reference may point outside the bundle.** A reference is delivered exactly as the conversion + produced it — the `reference` value, plus `identifier` and `display` where it has them. Nothing is + stripped or rewritten. This service serves no read endpoint for a reference target, so a client + resolves it by asking for the target with `_include` or against its own data. A few references + arrive with none of the three populated; that is how the conversion produced them, and the + searchset does not synthesise content for them. + +#### `_revinclude` resolves only from a resource already in the bundle + +The conversion records a reverse reference on the resource being *pointed at*, so a `_revinclude` is +resolved by walking the resources already retained. Reaching a resource that points at another +non-requested resource therefore takes a forward `_include` first: + +``` +# returns no Observation - they reference the Patient, which is not in the bundle +GET /fhir/{destinationId}/ImmunizationRecommendation?...&_revinclude=Observation + +# returns them - the Patient is now in the bundle to anchor the reverse hit +GET /fhir/{destinationId}/ImmunizationRecommendation?... + &_include=ImmunizationRecommendation:patient&_revinclude=Observation +``` + +Any retained resource the sought resource references will serve as the anchor, not only the one whose +search name the parameter names — the conversion accumulates every reverse search name onto one +canonical reference per resource. So qualifying a `_revinclude` narrows *which resources* come back, +not which path the lookup travels. A search name the conversion never registered matches nothing and +is not an error. + +#### Conversion-created resources + +Resources the conversion synthesises as a side effect (`Practitioner`, `Location`, +`Organization`, `RelatedPerson`, `DocumentReference`, `Provenance`) are **not** returned unless a +forward `_include` reaches them as the target of a reference, or the client white-lists them: + +``` +_include=Resource:source:* all conversion-created resources +_include=Resource:source:Organization just the named type +``` + +A white-listed resource is walked like any other returned resource, so a `_revinclude` can reach +further resources through it. + +One known gap: `_revinclude=Provenance` returns nothing, because the conversion gives `Provenance` a +bare id with no resource type on it, and a type-qualified `_revinclude` matches on that type. The +wildcard form works, and so does the white-list: + +``` +_include=Resource:source:DocumentReference&_revinclude=*:* returns the Provenance +_include=Resource:source:Provenance returns the Provenance +_revinclude=Provenance returns nothing +``` + +#### Getting the Z42 evaluated history + +A Z42 response carries the patient's evaluated doses alongside the forecast. Those `Immunization` +resources are not returned by a plain recommendation query, and they are not redundant with a +`/Immunization` query: that path sends Z34 and receives Z32, which does not carry `30973-2` (dose +number), `59782-3` (doses in series), `59779-9` (schedule used) or `64994-7` (funding eligibility), so +`protocolApplied.doseNumber`, `protocolApplied.seriesDoses`, `protocolApplied.authority` and +`programEligibility` are reachable **only** through the recommendation query. Ask for them: + +``` +GET /fhir/{destinationId}/ImmunizationRecommendation?... + &_include=ImmunizationRecommendation:patient + &_revinclude=Immunization:patient + &_include=Immunization:authority +``` + +| Entry | `search.mode` | Why | +|---|---|---| +| `ImmunizationRecommendation` | `match` | the requested type | +| `OperationOutcome` | `outcome` | conversion warnings, always returned | +| `Patient` | `include` | `_include=ImmunizationRecommendation:patient` | +| `Immunization` (one per dose) | `include` | `_revinclude=Immunization:patient`, anchored on the `Patient` | +| `Organization` | `include` | `_include=Immunization:authority` | + +The `_include` on `patient` is not optional — drop it and the `_revinclude` has nothing to resolve +from. `_include=Immunization:authority` is what makes `protocolApplied.authority` resolve inside the +bundle; the schedule `Organization` is registered on the `Immunization`, not on the +`ImmunizationRecommendation`. Select `search.mode = "match"` for the forecast alone. + +`Observation` resources are never returned unless the client asks for them with `_revinclude` **and** +retains a resource they reference, on any query. There is little reason to ask on a recommendation +query: the forecast `Observation` resources carry only what the recommendation itself already carries +(`vaccineCode`, `forecastStatus`, `dateCriterion`, `doseNumber`, `seriesDoses`, `forecastReason`, +`authority`), and one IIS response can hold dozens of them — 92 in one real capture. + +``` +GET /fhir/{destinationId}/ImmunizationRecommendation?... + &_include=ImmunizationRecommendation:patient + &_revinclude=Observation +``` + +That returns every `Observation` in the response: the forecast ones, plus the dose-level ones +belonging to the evaluated history. Both arrive as `include`, never as `match`. The forward `_include` +is what anchors the reverse lookup — the `Observation` reference the `Patient`, so without it the +`_revinclude` returns nothing. + +Dose-level `Observation` resources link to their `Immunization` via `partOf`, so +`_revinclude=Observation:part-of` narrows the request to those alone. Forecast `Observation` +resources have no such link — R4 does not permit `Observation.partOf` to reference an +`ImmunizationRecommendation` — so they are emitted unlinked and the narrowed form excludes them. + --- ## MSH → MessageHeader / Organization / Endpoint / Bundle diff --git a/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/.openspec.yaml b/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/.openspec.yaml new file mode 100644 index 000000000..5081c9876 --- /dev/null +++ b/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-08-12 diff --git a/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/design.md b/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/design.md new file mode 100644 index 000000000..dd4c29f76 --- /dev/null +++ b/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/design.md @@ -0,0 +1,209 @@ +## Context + +See `proposal.md` — Why, and `specs/fhir-searchset-filtering/spec.md` for the behaviour contract. + +All of the affected code is the searchset filter in +`src/main/java/gov/cdc/izgateway/xform/endpoints/fhir/FhirController.java`. It runs on the response +side of a FHIR query, entirely downstream of the transformation pipeline: + +```mermaid +sequenceDiagram + participant C as FHIR client + participant F as FhirController + participant H as HubController + participant P as Camel route -> PipelineRunnerService + participant D as Downstream Hub / IIS + + C->>F: GET /fhir/{dest}/ImmunizationRecommendation?...&_revinclude=... + F->>F: normalizeSubjectToPatient, build QBP_Q11 + F->>H: submitSoapRequest(SubmitSingleMessageRequest) + H->>P: request pipes (REQUEST direction) + P->>D: RSP query + D-->>P: RSP^K11 (Z42) + P-->>H: response pipes (RESPONSE direction, reversed order) + H-->>F: SubmitSingleMessageResponse + F->>F: convertResponseToFHIR (v2tofhir) -> Bundle + F->>F: adjustIdentifiers + rect rgb(240,240,240) + F->>F: filter <-- THE ONLY CODE THIS CHANGE TOUCHES + Note over F: preFilter -> markIncludedResources -> cleanupBundleOfUnmarkedResources + end + F-->>C: Bundle (searchset) +``` + +Two consequences of that position, both of which keep this change small: + +- **The pipeline is upstream and unaffected.** Pipe ordering, `DataFlowDirection.RESPONSE` reversal, + and the `x-loopback: true` short-circuit all happen inside `submitSoapRequest`, before + `convertResponseToFHIR` is reached. `filter` sees only the converted bundle. Loopback is a SOAP-path + header and is not reachable from the FHIR endpoints, so it needs no handling here. +- **No crypto, no observability, no persistence.** Nothing in `filter` touches BCFIPS providers, + keystores, or SSL reload, so the FIPS constraints do not bear on it. `filter` and its helpers carry + no `@CaptureXformAdvice`, so no AspectJ pointcut changes and no javaagent implications. The + configuration model is untouched, so neither the file nor the DynamoDB repository backend is + involved and there is no `SPRING_DATABASE=migrate` implication. + +Three facts about the current implementation shape the approach: + +1. **The reference graph already exists.** `v2tofhir` stamps each resource with a `References` and a + `Reverses` `Set` in `userData`, and each `Reference` carries its target under the + `Resource` user-data key. `markIncludedResources` already walks it. No traversal or FHIRPath code + is needed to answer "what does this retained entry point at". +2. **Removal happens in two places, at two different times.** `preFilter` physically removes + conversion-created resources from the bundle (`removeInfrastructureCreatedResources`, the + `it.remove()` at `:1120`) *before* anything knows which resources will be retained. + `cleanupBundleOfUnmarkedResources` removes everything still unmarked *after*. The subject `Patient` + dies in the second; the schedule `Organization` dies in the first. +3. **`markIncludedResources` grows its worklist as it iterates** (`for (int i = 0; i < resources.size(); i++)`), + so anything added to `resources` is itself traversed. Reachability closure is already transitive. + +## Goals / Non-Goals + +**Goals:** + +- Fix both defects at the filter, so every FHIR query endpoint is covered by one change rather than + the Z42 path only. +- Keep the change additive from a client's point of view: entries may be added and a `search.mode` + label corrected, but no resource content is altered or removed relative to today. +- Reduce the number of places a resource can be removed from the searchset, so the reference + integrity guarantee is checkable in one place. + +**Non-Goals:** + +- No new class, no new configuration property, no new dependency. +- No change to `toIncludeList` / `normalizeInclude` `_include` parsing, or to `includeMatches` + resolution semantics. The search names the conversion registers are `v2tofhir`'s concern; a + parameter that matches nothing today still matches nothing. +- No pagination or bundle-size limit. Out of scope even though this change can grow bundles. + +## Decisions + +### 1. Fix at the filter, not on the Z42 path + +Both defects are properties of the searchset filter, and the filter is shared by `/Immunization`, +`/ImmunizationRecommendation`, `/Patient`, and `Patient/$match`. The field report reached them +through Z42, but `Immunization.patient` (also 1..1) dangles identically on the `/Immunization` path. + +*Alternative considered:* special-case `ImmunizationRecommendation` to retain its `patient` and +`authority`. Rejected — smaller in the ticket, larger in total, because it leaves the identical +defect on every sibling endpoint and adds a resource-type branch to code that is currently +type-agnostic. + +### 2. `SearchEntryMode.INCLUDE` in `checkReferences` + +One token, at `FhirController.java:1182`. `checkReferences` is reached only from +`markIncludedResources`, and only for a resource that satisfied an `_include` or `_revinclude` — +which is the definition of `include` in R4. There is no path through it that should produce a +`match`; resources of the requested type are labelled by `preFilter` before `checkReferences` runs, +and `preFilter` wins because it labels the bundle entry directly while `checkReferences` labels +resource `userData` that `cleanupBundleOfUnmarkedResources` only applies when the entry's mode is +still null (`:1159`). + +### 3. Close the reference graph by **retaining** the target, not by stripping the reference + +To satisfy "A returned searchset contains no dangling references" there are exactly two mechanisms: +add the missing resource to the bundle, or remove the `reference` element that points at it. Chosen: +retain the resource, with `search.mode = "include"`. + +*Rationale:* + +- **Additive rather than destructive.** Retaining only adds entries. Stripping would delete a + `Reference.reference` element that `v2tofhir` populated — a silent content change for any client + reading that field today. +- **R4 sanctions it explicitly:** "the server has the prerogative to return additional search results + if it believes them to be relevant." A resource that a returned resource points at is relevant by + construction. +- **It is what the field validation asked for**, having looked at real Nevada and Alaska responses. + +*Alternative considered — strip `reference`, keep `identifier` + `display`:* this is a legal R4 +logical reference, it satisfies the 1..1 cardinality on `patient` (the element is still present), it +grows no bundle, and it preserves the existing "the enriched reference is enough for production use" +stance at `:1118` exactly. It was rejected on the additive-vs-destructive point above, but it remains +the right tool for a target that genuinely cannot be retained — see Decision 5 and the first risk. + +### 4. Unify removal into `cleanupBundleOfUnmarkedResources` + +Because `preFilter` removes conversion-created resources before retention is known (Context fact 2), +"keep it if something references it" is unanswerable at that point. Rather than teach `preFilter` to +look ahead, invert it: stop removing in `preFilter`, and let the single existing sweep in +`cleanupBundleOfUnmarkedResources` remove whatever is still unmarked once marking is complete. + +`removeInfrastructureCreatedResources` reduces to its white-list half — mark +`Resource:source:` hits `INCLUDE`, and otherwise do nothing. The `Provenance` carve-out at +`:1113` is then dead code and comes out: it exists only to skip an `it.remove()` that no longer +happens, and the `_revinclude=Provenance` it protects is handled by ordinary include marking either +way. Net effect is fewer lines and one removal site instead of two, which also keeps +`preFilter`'s cyclomatic complexity down rather than up. + +*Alternative considered:* a pre-pass that computes the retained set, then let `preFilter` consult it. +Rejected — that is a second traversal of the same graph `markIncludedResources` already walks. + +### 5. Reachability closure is unconditional, and runs where include marking already runs + +In `markIncludedResources`, after the two `checkReferences` calls, walk the same `References` set +once more and retain every target that is not already retained, marking it `INCLUDE`. The existing +growing-worklist loop makes this transitive for free (Context fact 3), which is required: if a +retained `Patient` itself references a `managingOrganization`, that reference must resolve too. + +Guard for a null target — `ref.getUserData("Resource")` is null for any reference the conversion did +not build through `ParserUtils.toReference`. Such a target cannot be retained, so it is the one case +where Decision 3's rejected alternative applies; see the first risk. + +Only *forward* references are closed. `Reverses` is not walked unconditionally, and must not be: the +92 forecast `Observation` resources reach the recommendation through `Reverses`, not `References`, so +they stay out of a plain `GET /ImmunizationRecommendation` — which the spec requires and which is the +whole point of the item-3 non-goal in the proposal. + +### 6. Checkstyle + +`ai-checkstyle.xml` fails the build at `validate`. The relevant limits here are cyclomatic complexity +and `MultipleStringLiterals` (max 3–4). Decision 4 removes branches from `preFilter` / +`removeInfrastructureCreatedResources`, and Decision 5 adds a short loop to `markIncludedResources`, +so complexity should net out flat or lower. The `"Resource"` and `"References"` user-data keys are +already repeated string literals in this file; the new code SHALL reuse extracted constants rather +than add another occurrence of either. + +## Risks / Trade-offs + +- **A reference whose target was never in the bundle still dangles.** Decision 5 can only retain a + resource the bundle actually holds. → Verification must assert the guarantee over real captures + rather than assume it. If a capture exposes such a reference, apply Decision 3's alternative + narrowly — clear `Reference.reference` on that reference, leaving `identifier` and `display` — as a + final sweep in `cleanupBundleOfUnmarkedResources`. Scoped to targets that cannot be retained, this + stays small and does not reopen Decision 3. +- **`/Immunization` bundles grow, by an amount nobody has measured.** A Z32 with N administered doses + can now retain the performer `Practitioner` and `Location` each dose references, where today those + are dropped and only the enriched reference survives. → Measure against the `ehex-testing` captures + before merging, both endpoints, entry counts before and after. The bound is the reference graph of + the matched resources, not the message: the Observations are excluded by Decision 5, so the Nevada + Z42 case grows by two entries, not eighty. If `/Immunization` growth proves unacceptable, the + containment is to restrict unconditional closure to targets without a `Parser.SOURCE` marker and + apply the reference-stripping fallback to the rest — a change to one predicate. +- **This partially reverses the deliberate stance at `:1118`** that an enriched reference is enough + for production use. → It is narrowed rather than abandoned: a conversion-created resource is now + retained only when a *retained* entry actually points at it, and is still dropped otherwise. The + `Resource:source:` white-list keeps its meaning for the unreferenced case. +- **The `search.mode` label change is client-visible.** A client selecting entries by + `mode == "match"` will now see fewer entries from an `_include` / `_revinclude` query. → That is the + intended correction and the reason the proposal marks it BREAKING; the previous label was a spec + violation. Call it out in the release notes. +- **The behaviour has no test coverage on either side of the boundary.** No test in `src/test` + references `_include` or `_revinclude`, none has a Z42 fixture, and none asserts on `Immunization` + or `ImmunizationRecommendation` bundle content. → The fixture is a prerequisite of the change, not + a follow-up; `tasks.md` sequences it before the code edits. + +## Migration Plan + +No data migration, no configuration change, no coordinated deploy — the change is confined to +response assembly in one service. Ship in the normal `feature -> develop -> Release-* -> main` flow. +Rollback is redeploying the previous image; nothing persists across the change, so a rollback simply +restores the previous labelling. + +Two sequencing notes: + +- Independent of the `v2tofhir` `2.4.0` -> `2.5.0` bump, which the user is handling separately. The + fixes are correct against both versions. Adding the Z42 test fixture, however, is most useful + against `2.5.0`, since only there does a Z42 produce the history/forecast split the fixture should + assert on. +- Release notes must carry the `match` -> `include` labelling change for `_include` / `_revinclude` + consumers. diff --git a/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/proposal.md b/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/proposal.md new file mode 100644 index 000000000..8d2cbb99a --- /dev/null +++ b/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/proposal.md @@ -0,0 +1,95 @@ +## Why + +Validating real Z42 (`RSP^K11`, evaluated history + forecast) responses from live IIS test systems +(Nevada, Alaska) against a running Transformation Service for the eHealthExchange project surfaced +two FHIR R4 conformance defects in the searchset filter that post-processes every FHIR query +response (`FhirController.filter`): + +1. Every `_include` / `_revinclude` hit is labelled `search.mode = "match"`, so a client cannot + distinguish the resources it asked for from the resources joined in to support them. +2. The filter deletes resources that surviving entries still reference, so the delivered bundle + ships mandatory (1..1) references — `ImmunizationRecommendation.patient`, `Immunization.patient` — + that resolve to nothing in the bundle and to no endpoint this service serves. + +Both are pre-existing, both are independent of the concurrent `v2tofhir` change +(`fix-z42-history-forecast-split`), and both are now highly visible on the Z42 path: after that +change the recommendation is often the *only* `match` in the bundle, so its dangling `patient` and +`authority` references are all the client has. + +## What Changes + +- **`_include` / `_revinclude` results are labelled `include`, not `match`.** `checkReferences` + stamps `SearchEntryMode.MATCH` on every join target (`FhirController.java:1182`); it SHALL stamp + `SearchEntryMode.INCLUDE`. R4 defines `include` as exactly "added to the results because of a + join", and clients are expected to filter on `mode = 'match'` to get the hits. **BREAKING** for + any client that relies on includes being labelled `match` — that reliance is a spec violation, and + the only path that ever produced `INCLUDE` today is the hard-coded `_include=Resource:source:` + pseudo-parameter (`:1107`). +- **No entry in a returned searchset carries a reference to a resource absent from that searchset.** + Resources still referenced by a surviving entry SHALL be retained with `search.mode = "include"`. + R4 sanctions this without the client asking: "the server has the prerogative to return additional + search results if it believes them to be relevant." + - Concretely this restores the `Patient` (referenced by `Immunization.patient` and + `ImmunizationRecommendation.patient`, both 1..1 required) and the schedule `Organization` + referenced by `ImmunizationRecommendation.authority`. + - The fix is at the filter, not at the Z42 path, because two *different* removal paths produce the + dangling reference and both affect `/Immunization` as well as `/ImmunizationRecommendation`: + `Patient` survives `preFilter` unmarked and is dropped by `cleanupBundleOfUnmarkedResources` + (`:1166`); the schedule `Organization` carries `Parser.SOURCE` and is dropped earlier by + `removeInfrastructureCreatedResources` (`:1120`). + +### Non-goals + +- **Forecast `Observation` resources stay filtered out of a query that does not ask for them.** + `v2tofhir` emits one `Observation` per OBX (92 for the Nevada message); all but two duplicate + data already carried on the `recommendation` component. Recovering the two that do not + (`30982-3` Reason Code, `59779-9` Schedule Used) is being handled in `v2tofhir` by mapping them + to `recommendation.forecastReason` and `ImmunizationRecommendation.authority`. Explicitly + considered and rejected — no change here. + + A caller who passes `_revinclude=Observation` still gets them, as they always could: the filter + walks the reverse direction only on an explicit parameter, so the default query returns none. + That is the R4-correct division and is not a goal of this change either way. +- `.gitignore` for `ehex-testing/` and the `v2tofhir` `2.4.0` → `2.5.0` version bump are handled + manually outside this change. + +## Capabilities + +### New Capabilities + +- `fhir-searchset-filtering`: How the service post-processes a converted response bundle into a + FHIR R4 searchset — which entries are retained, and what `Bundle.entry.search.mode` each entry + carries. Covers `match` / `include` / `outcome` classification, `_include` / `_revinclude` + resolution against the search names `v2tofhir` registers on `Reference.userData`, the + `_include=Resource:source:` pseudo-parameter for infrastructure-created resources, and + bundle-internal reference integrity. Applies to every FHIR query endpoint + (`/Immunization`, `/ImmunizationRecommendation`, `/Patient`, `Patient/$match`), so it does not + belong under `fhir-immunization-query`. + +### Modified Capabilities + +None. `fhir-immunization-query` covers request-side `subject`→`patient` aliasing and is untouched. + +## Impact + +- **Code**: `src/main/java/gov/cdc/izgateway/xform/endpoints/fhir/FhirController.java` only — + `checkReferences`, `preFilter` / `cleanupBundleOfUnmarkedResources` / + `removeInfrastructureCreatedResources`. No new class. +- **Inbound path**: FHIR REST only. The SOAP/HL7 v2 inbound path (`IISHubService`, `IISService`) is + untouched. +- **Outbound path**: none. The outbound QBP_Q11 / Z44 / Z34 query and the `izghub` / `iis` producer + contracts are unchanged — this change only reshapes the response bundle returned to the FHIR + caller. +- **Config model**: unchanged. No `Organization` / `Pipeline` / `Solution` / `Operation` / + `Precondition` change, so no `dependency` on either repository backend and no + `SPRING_DATABASE=migrate` implication. Existing organization transformation configurations are + unaffected. +- **Downstream Hub/IIS consumers**: unaffected — they see no FHIR. +- **FHIR clients**: bundles gain up to two entries per response (`Patient`, schedule `Organization`), + both labelled `include`; `_include` / `_revinclude` entries change label from `match` to + `include`. `docs/CONFIGURATION_REFERENCE.md` needs no change (no new property). +- **Tests**: no existing test in `src/test` references `_include`, `_revinclude`, or has a Z42 + fixture, and no test asserts on `Immunization` / `ImmunizationRecommendation` bundle content — so + this behaviour is entirely uncovered today. `FhirControllerTests` should stay green; new coverage + is needed. +- **Dependencies**: none added. diff --git a/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/specs/fhir-searchset-filtering/spec.md b/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/specs/fhir-searchset-filtering/spec.md new file mode 100644 index 000000000..0aa6d5033 --- /dev/null +++ b/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/specs/fhir-searchset-filtering/spec.md @@ -0,0 +1,262 @@ +## Purpose + +This capability defines how the IZ Gateway Transformation Service turns a converted HL7 v2 response +into a FHIR R4 searchset for a FHIR query caller: which entries the response bundle retains, what +`Bundle.entry.search.mode` each retained entry carries, how `_include` and `_revinclude` are +resolved, and the integrity guarantee that no retained entry references a resource the bundle does +not contain. It applies to every FHIR query endpoint the service exposes and is independent of the +transformation pipeline, which does not participate in searchset assembly. + +## ADDED Requirements + +### Requirement: FHIR query responses are returned as a searchset + +The service SHALL return the response to a FHIR query as a `Bundle` of type `searchset`, regardless +of the bundle type produced by the HL7 v2 to FHIR conversion. Every retained entry SHALL carry a +populated `Bundle.entry.search.mode`. The requested resource type SHALL be determined from the +request path, and for the `Patient/$match` operation the requested type SHALL be `Patient`. + +This requirement governs the FHIR REST inbound path only. The SOAP/HL7 v2 inbound path and the +outbound query sent to the downstream Hub or IIS SHALL be unchanged, as SHALL the transformation +pipeline (organizations, pipelines, solutions, operations, and preconditions) — no organization +configuration affects searchset assembly. + +#### Scenario: response bundle is typed as a searchset +- **GIVEN** a query to a FHIR query endpoint that produces a response bundle +- **WHEN** the service returns the response to the caller +- **THEN** `Bundle.type` SHALL be `searchset` + +#### Scenario: every retained entry carries a search mode +- **GIVEN** a returned searchset +- **WHEN** the caller inspects any entry in the bundle +- **THEN** that entry SHALL have a populated `Bundle.entry.search.mode` + +#### Scenario: the $match operation resolves to the Patient type +- **GIVEN** a request to `POST /fhir/{destination}/Patient/$match` +- **WHEN** the response searchset is assembled +- **THEN** the requested resource type SHALL be `Patient`, and `Patient` resources SHALL be + classified as described in "Resources of the requested type are labelled `match`" + +### Requirement: Resources of the requested type are labelled `match` + +The service SHALL set `search.mode = "match"` on, and SHALL retain, every entry whose resource type +equals the resource type requested by the query. Resources of any other type SHALL NOT be labelled +`match`. + +#### Scenario: requested type is labelled match +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` +- **WHEN** the converted bundle contains one `ImmunizationRecommendation` +- **THEN** that entry SHALL be retained with `search.mode = "match"` + +#### Scenario: a resource of another type is not labelled match +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` +- **WHEN** the converted bundle also contains `Immunization` resources produced from the same + HL7 v2 response +- **THEN** those `Immunization` entries SHALL NOT be labelled `match` +- **AND** they SHALL be retained only if another requirement in this capability requires it + +### Requirement: `OperationOutcome` entries are labelled `outcome` + +The service SHALL retain every `OperationOutcome` in the converted bundle and SHALL set +`search.mode = "outcome"` on it, so conversion warnings and errors always reach the caller. + +#### Scenario: conversion warnings survive filtering +- **GIVEN** a converted bundle containing one or more `OperationOutcome` resources +- **WHEN** the searchset is assembled +- **THEN** each `OperationOutcome` entry SHALL be retained with `search.mode = "outcome"` + +### Requirement: `_include` and `_revinclude` results are labelled `include` + +The service SHALL set `search.mode = "include"` on every resource retained because it satisfied an +`_include` or `_revinclude` parameter. It SHALL NOT label such a resource `match`. In FHIR R4, +`include` is the defined value for an entry "added to the results because of a join", and clients +filter on `mode = "match"` to obtain the hits; labelling joined resources `match` makes the hits +indistinguishable from their supporting resources. + +An `_include` or `_revinclude` parameter SHALL be resolved against the search-parameter names the +HL7 v2 to FHIR conversion registered for each reference. A parameter naming a resource type or +search name that is not registered SHALL match nothing and SHALL NOT be an error. Wildcard (`*`) +resource types and search names SHALL match any value. + +This is a **BREAKING** change to the labelling a client observes: a client that previously read +`_include` and `_revinclude` results as `match` SHALL now read them as `include`. + +#### Scenario: a reverse-included resource is labelled include +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` with + `_revinclude=Observation` +- **WHEN** the converted bundle contains one `ImmunizationRecommendation` and supporting + `Observation` resources that reference it +- **THEN** the `ImmunizationRecommendation` entry SHALL have `search.mode = "match"` +- **AND** every retained `Observation` entry SHALL have `search.mode = "include"` +- **AND** the caller SHALL be able to obtain exactly the requested resources by selecting entries + with `search.mode = "match"` + +#### Scenario: a forward-included resource is labelled include +- **GIVEN** a request to a FHIR query endpoint with an `_include` parameter that matches a reference + on a resource of the requested type +- **WHEN** the searchset is assembled +- **THEN** the referenced resource SHALL be retained with `search.mode = "include"` + +#### Scenario: an unmatched include parameter is not an error +- **GIVEN** a request with an `_include` or `_revinclude` parameter naming a search name that the + conversion does not register for any reference +- **WHEN** the searchset is assembled +- **THEN** the request SHALL succeed +- **AND** no additional entry SHALL be retained on account of that parameter + +### Requirement: A returned searchset contains no dangling references + +No entry retained in a returned searchset SHALL carry a `Reference.reference` element pointing at a +resource that the searchset does not contain. When a resource would otherwise be removed but is +still referenced by a retained entry, the service SHALL retain it with `search.mode = "include"`. +Where the referenced resource cannot be retained because the searchset never held it, the service +SHALL instead clear the `reference` element, preserving `Reference.identifier` and +`Reference.display` so the target remains readable as a logical reference. A reference reduced this +way still satisfies a 1..1 cardinality, because the element itself remains present. + +FHIR R4 permits this without the client asking for it: "the server has the prerogative to return +additional search results if it believes them to be relevant." The defect this closes is that +mandatory 1..1 references — `Immunization.patient`, `ImmunizationRecommendation.patient` — and the +populated `ImmunizationRecommendation.authority` resolved to nothing in the delivered bundle and to +no endpoint this service serves, so a client performing local reference resolution rather than a +follow-up fetch could not resolve them. + +This requirement SHALL take precedence over "Conversion-created resources are retained only when +white-listed": a conversion-created resource that a retained entry references SHALL be retained. + +#### Scenario: the subject Patient is retained for an immunization query +- **GIVEN** a request to `GET /fhir/{destination}/Immunization` with no `_include` parameter +- **WHEN** the retained `Immunization` entries carry a `patient` reference to a `Patient` in the + converted bundle +- **THEN** that `Patient` SHALL be retained with `search.mode = "include"` +- **AND** every `Immunization.patient` reference in the searchset SHALL resolve to it + +#### Scenario: the subject Patient is retained for a recommendation query +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` with no `_include` + parameter +- **WHEN** the retained `ImmunizationRecommendation` carries a `patient` reference to a `Patient` in + the converted bundle +- **THEN** that `Patient` SHALL be retained with `search.mode = "include"` + +#### Scenario: the schedule Organization behind authority is retained +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` with no `_include` + parameter +- **WHEN** the retained `ImmunizationRecommendation` carries an `authority` reference to an + `Organization` created by the conversion from the immunization schedule used +- **THEN** that `Organization` SHALL be retained with `search.mode = "include"`, notwithstanding + that it is a conversion-created resource + +#### Scenario: no reference in a returned searchset dangles +- **GIVEN** any returned searchset +- **WHEN** every `Reference.reference` element on every retained entry is resolved against the + entries of that same searchset +- **THEN** every such reference SHALL resolve to an entry present in the searchset + +#### Scenario: an unretainable target is reduced to a logical reference +- **GIVEN** a retained entry carrying a reference to a resource the converted bundle never held — + for example a reference the HL7 v2 to FHIR conversion did not register in its reference + bookkeeping, so no target resource is available to retain +- **WHEN** the searchset is assembled +- **THEN** that reference SHALL have no `reference` element +- **AND** its `identifier` and `display` SHALL be preserved +- **AND** the searchset SHALL still satisfy "no reference in a returned searchset dangles" + +#### Scenario: retaining a referenced resource does not promote it to a match +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` +- **WHEN** a `Patient` and an `Organization` are retained solely to satisfy reference integrity +- **THEN** neither SHALL be labelled `match` +- **AND** selecting entries with `search.mode = "match"` SHALL yield only the + `ImmunizationRecommendation` resources + +### Requirement: A recommendation query returns the evaluated history it was sent with + +On a query for `ImmunizationRecommendation`, the service SHALL retain the `Immunization` resources +converted from the same response, with `search.mode = "include"`. It SHALL NOT label them `match`, +so a client can still isolate the forecast it asked for by selecting `search.mode = "match"`. + +A Z42 response ("Return Evaluated History and Forecast") carries the patient's evaluated history +alongside the forecast, split by RXA-5: `998^No Vaccine Administered` becomes a `recommendation` +component, any other CVX code becomes an `Immunization`. Those `Immunization` resources carry +evaluation data reachable through no other call — `protocolApplied.doseNumber` and `seriesDoses` +(OBX `30973-2` / `59782-3`), `protocolApplied.authority` (OBX `59779-9`), and `programEligibility` +(OBX `64994-7`). The `/Immunization` path does not compensate, because it sends Z34 and receives +Z32, which carries none of those OBX codes. + +This applies only to the recommendation query path. On an `/Immunization` query the same resources +are the matches, and their labelling SHALL be unchanged. + +#### Scenario: evaluated history accompanies the forecast +- **GIVEN** a Z42 response containing administered doses and forecasts for one patient +- **WHEN** the client issues `GET /fhir/{destination}/ImmunizationRecommendation` with no + `_include` parameter +- **THEN** one `Immunization` per administered dose SHALL be retained with + `search.mode = "include"` +- **AND** the single `ImmunizationRecommendation` SHALL be the only entry labelled `match` + +#### Scenario: included history carries the evaluation data +- **GIVEN** a returned recommendation searchset containing evaluated history +- **WHEN** the client reads an included `Immunization` +- **THEN** the `protocolApplied.doseNumber`, `protocolApplied.seriesDoses`, + `protocolApplied.authority` and `programEligibility` values present in the source response SHALL + be populated +- **AND** `protocolApplied.authority` SHALL resolve to an `Organization` in the same searchset + +#### Scenario: each included dose keeps its own identifier +- **GIVEN** a Z42 response with more than one administered dose +- **WHEN** the searchset is assembled +- **THEN** each included `Immunization` SHALL carry the filler order number of its own ORC-3 +- **AND** no two entries in the searchset SHALL collide on resource type and id + +#### Scenario: an immunization query is unaffected +- **GIVEN** a client issuing `GET /fhir/{destination}/Immunization` +- **WHEN** the searchset is assembled +- **THEN** the `Immunization` resources SHALL be labelled `match`, not `include` + +### Requirement: Conversion-created resources are retained only when white-listed + +Resources that the HL7 v2 to FHIR conversion synthesises as a side effect of datatype and message +parsing — rather than from a dedicated segment the caller queried for — SHALL be removed from the +searchset unless the caller white-lists them, because the enriched reference they are the target of +already carries an identifier and display text sufficient for production use. + +The caller SHALL be able to white-list them with the `_include=Resource:source:` parameter, +where `` is a resource type or `*` for all such resources; a white-listed resource SHALL be +retained with `search.mode = "include"`. Resources referenced by a retained entry SHALL be retained +regardless of this requirement, per "A returned searchset contains no dangling references". + +#### Scenario: conversion-created resources are removed by default +- **GIVEN** a query whose converted bundle contains conversion-created `Practitioner` and `Location` + resources that no retained entry references +- **WHEN** the caller supplies no `_include=Resource:source:...` parameter +- **THEN** those entries SHALL NOT appear in the returned searchset + +#### Scenario: a caller white-lists conversion-created resources by type +- **GIVEN** the same query +- **WHEN** the caller supplies `_include=Resource:source:Practitioner` +- **THEN** the conversion-created `Practitioner` entries SHALL be retained with + `search.mode = "include"` + +#### Scenario: a caller white-lists all conversion-created resources +- **GIVEN** the same query +- **WHEN** the caller supplies `_include=Resource:source:*` +- **THEN** every conversion-created resource SHALL be retained with `search.mode = "include"` + +### Requirement: Unclassified entries are removed + +The service SHALL remove from the returned searchset every entry that no requirement in this +capability retains. A caller SHALL therefore never receive an entry whose `search.mode` is absent. + +#### Scenario: an unreferenced, unrequested resource is removed +- **GIVEN** a converted bundle containing a resource that is not of the requested type, is not an + `OperationOutcome`, satisfies no `_include` or `_revinclude` parameter, and is referenced by no + retained entry +- **WHEN** the searchset is assembled +- **THEN** that entry SHALL NOT appear in the returned searchset + +#### Scenario: forecast observations are removed from a plain recommendation query +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` with no `_include` or + `_revinclude` parameter +- **WHEN** the converted bundle contains `Observation` resources carrying the forecast detail +- **THEN** those `Observation` entries SHALL NOT appear in the returned searchset +- **AND** the returned `ImmunizationRecommendation` SHALL remain self-contained — it SHALL NOT + reference any of the removed `Observation` resources diff --git a/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/tasks.md b/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/tasks.md new file mode 100644 index 000000000..a20e1cb77 --- /dev/null +++ b/openspec/changes/archive/2026-08-13-fix-fhir-searchset-include-mode/tasks.md @@ -0,0 +1,206 @@ +## 1. Test harness and fixtures (prerequisite) + +Per `design.md` — Risks, this behaviour has zero coverage today, so the fixtures land before the +code edits and are expected to fail until section 3 and 4 are done. + +Tests go in the existing `src/test/java/gov/cdc/izgateway/xform/endpoints/fhir/FhirControllerTests.java`, +which is plain JUnit 5 + Mockito and constructs `FhirController` directly through the +`controller(hubReturning(...))` helper. **Deliberate deviation from the project default:** these tests +do NOT get `@SpringBootTest`. The searchset filter needs no Spring context, only two of the repo's +test classes carry the annotation, and adding a context here would slow the suite for no coverage. +All tests still run under Maven surefire with its existing env/keystore setup. + +- [x] 1.1 Extend the `fhirRequest(uri, accept)` helper (`FhirControllerTests.java:470`) to accept + query parameters, stubbing `getParameterValues("_include")` and `getParameterValues("_revinclude")` + alongside the existing `getParameterMap()`. Keep the two-argument form working for existing + callers. +- [x] 1.2 Add an `RSP_Z42_MESSAGE` fixture: an `RSP^K11` with `QAK`/`QPD` of + `Z42^Request Evaluated History and Forecast^CDCPHINVS`, `MSH-21` carrying the Z42 profile, one + `PID`, at least two administered `ORC`/`RXA` groups (`RXA-5` a real CVX), at least one forecast + group (`RXA-5 == 998`), and forecast `OBX` segments including `59779-9` Immunization Schedule + Used (`VXC16^ACIP^CDCPHINVS`) so `ImmunizationRecommendation.authority` is populated. +- [x] 1.3 Add an `RSP_Z32_MESSAGE` fixture: an `RSP^K11` Z32 response with one `PID` and at least two + administered `ORC`/`RXA` groups, so the `/Immunization` path is covered independently of Z42. +- [x] 1.4 Add a reusable assertion helper `assertNoDanglingReferences(Bundle)` that walks every + `Reference` on every entry and asserts each populated `Reference.reference` resolves to an entry + in the same bundle. This is the check that would have caught the withdrawn + `supportingPatientInformation` link; it is reused by every test below. +- [x] 1.5 Confirm which `v2tofhir` version the suite compiles against (`pom.xml:110-111`). The + history/forecast split assertions in 3.3 and 4.4 require `2.5.0`; if the tree is still on + `2.4.0`, write them but mark them `@Disabled` with a reference to this change, and note it in + the PR so the version bump un-disables them rather than silently skipping. + +## 2. Verify current (broken) behaviour is captured + +- [x] 2.1 Run `mvn test -Dtest=FhirControllerTests` and record which of the new tests fail. Confirm + the failures are exactly the two defects — `include` labelled `match`, and a dangling + `patient` / `authority` — and not a fixture error. A fixture that produces an empty or + unconverted bundle proves nothing. + +## 3. Fix 1 — label `_include` / `_revinclude` results as `include` + +- [x] 3.1 In `checkReferences` (`FhirController.java:1182`), change the `setUserData` value from + `SearchEntryMode.MATCH` to `SearchEntryMode.INCLUDE`. +- [x] 3.2 Add a test: `GET /fhir/dev/ImmunizationRecommendation` with `_revinclude=Observation` + against `RSP_Z42_MESSAGE` returns the recommendation with `search.mode == MATCH` and every + retained `Observation` with `search.mode == INCLUDE`. +- [x] 3.3 Add a test: selecting entries with `search.mode == MATCH` from that same response yields + only `ImmunizationRecommendation` resources — no `Observation`, no `Immunization`. (This is the + assertion that depends on the v2tofhir history/forecast split; see 1.5.) +- [x] 3.4 Add a test: an `_include` naming a search name the conversion does not register succeeds + with no additional entries and no error. + +## 4. Fix 2 — no dangling references in a returned searchset + +Implements `design.md` decisions 3, 4, and 5. Do 4.1 and 4.2 in one commit — 4.1 alone would leak +conversion-created resources into every response. + +- [x] 4.1 Reduce `removeInfrastructureCreatedResources` to its white-list half: keep the + `matchesSource` branch that marks `Resource:source:` hits `INCLUDE`, drop the + `it.remove()` at `:1120`, and drop the now-dead `Provenance` carve-out at `:1113`. Remove the + `Iterator` parameter, which is no longer used. Removal now happens only in + `cleanupBundleOfUnmarkedResources`. +- [x] 4.2 In `markIncludedResources`, after the two existing `checkReferences` calls, walk the + resource's `References` set once more and, for each reference whose target is not already in + `resources`, add it and mark it `SearchEntryMode.INCLUDE`. Guard against a null target + (`ref.getUserData("Resource")` is null for any reference not built through + `ParserUtils.toReference`). Do NOT walk `Reverses` — that is what keeps the forecast + `Observation` resources out of a plain query. +- [x] 4.3 Add a test on `RSP_Z42_MESSAGE`, plain `GET /fhir/dev/ImmunizationRecommendation` with no + `_include`: the `Patient` and the schedule `Organization` are present with + `search.mode == INCLUDE`, neither is labelled `MATCH`, and `assertNoDanglingReferences` passes. +- [x] 4.4 Add a test on the same response: no `Observation` entry is present, and the returned + `ImmunizationRecommendation` references none of the removed `Observation` resources. +- [x] 4.5 Add a test on `RSP_Z32_MESSAGE`, plain `GET /fhir/dev/Immunization`: every + `Immunization.patient` resolves to a `Patient` entry in the bundle, and + `assertNoDanglingReferences` passes. +- [x] 4.6 Add a test that the `_include=Resource:source:*` and `_include=Resource:source:` + white-list still retains unreferenced conversion-created resources with + `search.mode == INCLUDE`, and that without it they are still absent. This pins the behaviour + 4.1 refactors around. +- [x] 4.7 Add a test that `OperationOutcome` entries survive with `search.mode == OUTCOME`, and that + the bundle type is `SEARCHSET`. Cheap regression guard on the parts of `preFilter` this change + moves code around in. + +## 5. Measure the bundle-growth risk + +`design.md` — Risks flags `/Immunization` growth as real and unmeasured. Resolve it before the PR is +reviewed, not after. + +> **Measured against the real Nevada and Alaska captures** (`~/Downloads/ehex-testing`, driven +> through the mocked hub; the captures are deliberately NOT committed — they carry live vendor IIS +> patient demographics). +> +> *Superseded by section 7 for the recommendation path:* the "after" counts below predate the +> evaluated-history include, which takes NV from 5 to 13 and AK from 5 to 11. The `/Immunization` +> row is still current. +> +> | Capture / query | Entries before | Entries after | Dangling refs before | after | +> |---|---|---|---|---| +> | NV `/Immunization` (2 doses) | 4 | 11 | 6 | **0** | +> | NV `/ImmunizationRecommendation` (92 OBX) | 3 | 5 | 2 | **0** | +> | AK `/ImmunizationRecommendation` (88 OBX, 13 RXA) | 3 | 5 | 2 | **0** | +> +> The Nevada dangling reference reproduced byte-for-byte from the field notes: +> `ImmunizationRecommendation -> Patient/TlYwMDAwfDM5NzM1NjU`. +> +> Both recommendation queries return **exactly one** `ImmunizationRecommendation` (NV: 16 +> components, AK: 10), `authority` = ACIP resolves in-bundle, and **0** of the 92/88 OBX +> Observations leak in. `/Immunization` returns exactly the 2 administered doses. 5.2 satisfied +> on every clause. +> +> `/Immunization` growth is the open trade-off: +7 entries for 2 doses (`Patient`, 4 `Location`, +> 2 `Organization`), i.e. roughly `3N + 3` added for N doses — about 4x on a 13-dose record. Every +> added resource is genuinely referenced by a match, but they are exactly the DatatypeConverter +> resources the original code deliberately dropped as "the enriched reference is enough". See 5.3. +> +> One design assumption did not survive contact: closure over v2tofhir's `References` user-data +> alone was **not** sufficient. `PractitionerRole -> Practitioner` is a reference v2tofhir does not +> register through `ParserUtils.toReference`, so it stayed dangling. The design's pre-authorised +> fallback (`clearUnresolvableReferences`) was implemented as a final sweep in `filter`, reducing +> any still-unresolvable reference to a logical reference with `identifier` + `display`. Without +> it the no-dangling-references guarantee does not hold. + +- [x] 5.1 Against the `ehex-testing` captures (Nevada, Alaska), record entry counts before and after + the change for `GET /Immunization` and `GET /ImmunizationRecommendation`, and put the numbers + in the PR description. Expectation from the design: Z42 grows by ~2 entries; `/Immunization` + grows by the distinct performer `Practitioner` and `Location` set. +- [x] 5.2 Run the end-to-end check from the field notes against those captures: `GET /Immunization` + returns the administered doses, `GET /ImmunizationRecommendation` returns exactly one + recommendation with one component per forecast, and no reference in either bundle points at a + resource absent from that bundle. +- [x] 5.3 **Decided: not applied.** With the real numbers in (see above), the growth was judged + acceptable — the added `Location` / `Organization` resources are genuinely referenced by a + match, and returning them as resources rather than an inline `display` string is a gain for + the client. The containment below remains the documented lever if a jurisdiction with very + long immunization histories reports a payload-size problem. + The containment, if ever needed: restrict the 4.2 closure to targets without a + `Parser.SOURCE` marker and clear `Reference.reference` (keeping `identifier` and `display`) + on the rest — a change to one predicate, since `clearUnresolvableReferences` already exists. + +## 6. Docs, build gates, and review + +- [x] 6.1 Document the searchset contract for API consumers: `match` vs `include` vs `outcome`, the + `_include=Resource:source:` white-list, and the guarantee that references resolve within + the bundle. `docs/CONFIGURATION_REFERENCE.md` needs no change (no new property) — put this + where the FHIR endpoints are described, or add a short section to `docs/QUICK_START.md`. +- [x] 6.2 Update the Newman/Postman collection in `testing/scripts/` if any request there asserts on + `search.mode`, since `_include` / `_revinclude` results now report `include`. Check before + editing — the collection may not exercise these parameters at all. +- [x] 6.3 Call the `match` -> `include` label change out as **BREAKING** in the PR description and + release notes. +- [x] 6.4 Run `mvn clean package` and confirm Checkstyle passes at the `validate` phase — watch + cyclomatic complexity on `markIncludedResources` (4.2 adds a loop) and `MultipleStringLiterals` + on the `"Resource"` / `"References"` user-data keys, which are already repeated in this file; + reuse or extract constants rather than adding another literal. +- [x] 6.5 Confirm the OWASP dependency-check gate still passes under CVSS 7. No dependency is added + or changed by this work, so this is a no-op check unless the separate `v2tofhir` bump lands in + the same branch. +- [x] 6.6 No security review needed: nothing here touches mTLS, JWT, `Roles`, `AccessControlValve`, + or any BCFIPS crypto path. Confirm this still holds at review time — the change must not alter + which resources a caller is authorised to see, only which are labelled and retained. Note that + fix 2 does return resources (`Patient`, `Organization`) that were previously filtered out; + confirm they come from the same IIS response the caller already receives and introduce no new + data disclosure. + +## 7. Include the Z42 evaluated history on a recommendation query + +Added after the sections above were complete. A Z42 carries evaluated history that the `/Immunization` +path cannot reach, because that path sends Z34 and receives Z32, which lacks OBX `30973-2`, +`59782-3` and `59779-9`. + +- [x] 7.1 In `preFilter`, mark `Immunization` entries `SearchEntryMode.INCLUDE` when the requested + type is `ImmunizationRecommendation`. Keyed on `requested`, not on a plumbed-through + `queryType` — the two are derived from the same URI, so no new parameter is needed. +- [x] 7.2 Extract the repeated "set mode, add to `resources`" into `markEntry`, and drop the + `revIncludes` parameter `preFilter` no longer uses (dead since 4.1). +- [x] 7.3 Extend `RSP_Z42_MESSAGE` so the two administered doses carry the Z42-only evaluation OBX + segments (`30956-7`, `30973-2`, `59782-3`, `59779-9`, `64994-7`). +- [x] 7.4 Tests: history returned as `include` and never `match`; `doseNumber` / `seriesDoses` / + `authority` / `programEligibility` populated with `authority` resolving in-bundle; per-dose + ORC-3 identifiers distinct with no `Type/id` collision; `/Immunization` still labels its doses + `match`. +- [x] 7.5 Retarget `forecastObservationsCannotBeRevincluded` to + `observationsArriveOnlyWhenRevincludedAndNeverAsMatch`. Two rounds here. First, including the + history made history Observations reachable through their `partof` link to the now-included + `Immunization` resources. Second, the current `v2tofhir` build also lets an explicit + `_revinclude=Observation` reach the *forecast* Observations (15 retained on the fixture: 10 + history, 5 forecast), so the "nothing can revinclude a forecast Observation" claim no longer + holds and the `partOf` assertion failed. That is correct behaviour, not a regression: the + filter still never walks `Reverses` on its own (design decision 5), so the property worth + pinning is the opt-in one — zero Observations by default, `include` and never `match` when + the caller asks. The default-query assertion lives in + `plainRecommendationQueryIsSelfContainedWithoutObservations`, unchanged. + +> **Verified against the real captures.** No dangling references on any of the three. +> +> | Capture / query | Entries | Composition | +> |---|---|---| +> | NV Z42 `/ImmunizationRecommendation` | 13 | 1 IR `match`; `include`: 2 Immunization, 1 Patient, 3 Organization, 4 Location; 2 outcome | +> | AK Z42 `/ImmunizationRecommendation` | 11 | 1 IR `match`; `include`: 3 Immunization, 1 Patient, 2 Organization, 2 Location; 2 outcome | +> | NV Z32 `/Immunization` | 11 | unchanged from section 5 — 2 Immunization `match` | +> +> NV doses carry the expected `NV0000|41348935` and `NV0000|41348937`, `doseNumber` = 1, and +> `programEligibility`. Both doses' `protocolApplied.authority` and the recommendation's +> `authority` resolve to the *same* ACIP `Organization` (`VXC16`) — v2tofhir dedupes it across all +> 18 `59779-9` occurrences. diff --git a/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/.openspec.yaml b/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/.openspec.yaml new file mode 100644 index 000000000..149631464 --- /dev/null +++ b/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-08-17 diff --git a/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/design.md b/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/design.md new file mode 100644 index 000000000..5f0efe264 --- /dev/null +++ b/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/design.md @@ -0,0 +1,288 @@ +## Context + +See `proposal.md` — Why. This change is confined to the searchset assembly step inside +`FhirController.filter(Bundle, HttpServletRequest)`. That step runs after the response has already +come back from the downstream Hub or IIS, been converted by `MessageParser`, and been carried back +through the transformation pipeline. Nothing upstream of it is involved. + +Two facts shape the approach: + +**The target behavior already shipped.** `develop` returns exactly the strict contract this change +restores: `preFilter` marks the requested type `match` and `OperationOutcome` `outcome`, everything +else falls through to `cleanupBundleOfUnmarkedResources`, and an unresolved reference keeps its +literal value because no code touches it. The `IGDD-3285` branch added three things on top — +`retainReferencedResources`, `clearUnresolvableReferences`, and a Z42 `Immunization` branch in +`preFilter` — and those three are what this change removes. So this is a subtraction, not a design +problem, and the risk profile is that of reverting to code that has been in `develop` all along. + +**One branch change is worth keeping, and one is worth keeping for a reason `develop` did not +have.** The `include`-not-`match` labelling of join hits (`checkReferences`) is a genuine fix and +stays. Separately, the branch restructured removal: `develop` removed conversion-created resources +eagerly inside `preFilter` with `Iterator.remove()`, whereas the branch marks nothing and lets the +single `cleanupBundleOfUnmarkedResources` pass do all removal. That restructuring stays too — see +Decisions. + +## Goals / Non-Goals + +**Goals:** + +- Delete the three additions so the returned entry set matches `develop` exactly. +- Keep the branch's `include` labelling and its single-cleanup-pass structure. +- Leave every `Reference` untouched, including mandatory 1..1 references whose target is omitted. +- Keep `filter` and its helpers within Checkstyle's complexity and length limits, which subtraction + makes easier, and clean up the imports the deletions strand. + +**Non-Goals:** + +- No configuration switch to select strict or lenient assembly. See Decisions. +- No convenience so a bare `_revinclude=Immunization` reaches the Z42 evaluated history through the + `Patient` without the `Patient` being retained. The caller sends both parameters. +- No folding of `_include=Resource:source:*` into a bare `_include=*:*`. The two parameters keep + their present, separate meanings. +- No read endpoint for a reference target. A literal reference in a returned searchset stays + unfetchable from this service, and the spec says so. +- No change to `MessageParser`, to `v2tofhir` reference bookkeeping (`References`, `Reverses`, + `SEARCH_NAMES`, `REVERSE_NAMES`, `Parser.SOURCE`), or to the CapabilityStatement. + +## Decisions + +### Subtract the three additions rather than gate them behind a property + +A property such as `xform.fhir.strict-searchset` would let the eHealth Exchange pilot keep the +current behavior while new callers get the strict contract. + +Rejected. A FHIR response shape is part of the API contract, and a deployment-time switch means two +contracts that no caller can discover — the CapabilityStatement does not advertise it, and a caller +cannot tell from a response which mode produced it. It also doubles the assembly paths under test +forever. The pilot's migration is adding `_include` parameters to queries it already sends, which is +smaller than the cost of carrying the switch. + +Alternative considered and rejected: keep the auto-retain but drop the retained resources' own +`match`-adjacent visibility some other way. There is no such way — the entry is either in the bundle +or not. + +### Deliver a reference exactly as the conversion produced it, empty or not + +Removing the stripping does not make every reference readable. Of the five references the pre-change +code shipped without a `reference` element on the Z32 fixture, only two were stripped by +`clearUnresolvableReferences`; the other three carry no `reference`, no `identifier`, and no `display` +as `v2tofhir` produces them, and they stay that way after the deletion. + +This change does not attempt to populate them. Doing so would mean synthesising content the +conversion did not produce, which is the conversion's concern, not the searchset's. The delta spec +says so explicitly, so a later reader does not mistake an empty reference for a regression this +change introduced. + +### Delete `clearUnresolvableReferences` rather than keep it as a narrow fallback + +The branch's fallback — strip `reference`, keep `identifier` and `display` — could be kept for the +narrow case it was written for: a reference built directly with `new Reference(...)` rather than +through `ParserUtils.toReference`, which v2tofhir never registered, so no target resource exists to +retain. + +Rejected. Once nothing is auto-retained, that case is no longer distinguishable from the ordinary +one: after this change, *most* references point outside the searchset, so a rule that strips +unresolvable references would strip nearly all of them. And stripping discards information — the +conversion produced that reference value, and a caller correlating a response against its own data +may want it. `develop` shipped the literal value in both cases and no defect was raised against +that. Deleting the method and its `toRelativeReference` and `forEachReference` helpers is therefore +the smaller contract, not just the smaller diff. + +### Keep the branch's single-cleanup-pass removal, not `develop`'s eager removal + +`develop`'s `removeInfrastructureCreatedResources` called `it.remove()` on a conversion-created +resource during `preFilter`, before include marking had run. That forced a special case: a +`MessageParser`-sourced `Provenance` was spared from removal when `_revinclude` named `Provenance`. +The branch's `whitelistInfrastructureCreatedResources` removes nothing; it only marks, and all +removal happens once in `cleanupBundleOfUnmarkedResources` after include marking. Keep the branch's +version: one removal point is easier to reason about than removal split across two passes. + +The carve-out is dropped because it was dead code: `develop` only skipped the `it.remove()` without +setting a `search.mode`, so `cleanupBundleOfUnmarkedResources` deleted the `Provenance` on the next +pass regardless. `_revinclude=Provenance` returned nothing on `develop` and returns nothing now — +measured on the Z32 fixture at matched v2tofhir versions — so removing the carve-out changes no +observable behavior. + +A white-listed resource **is** traversed. `whitelistInfrastructureCreatedResources` adds it to the +`resources` list, so `markIncludedResources` visits it and walks its reverse references like any +other retained resource. The reason `_revinclude=Provenance` fails is unrelated to white-listing — +see the known limitation below. + +### Known limitation: a type-qualified `_revinclude` cannot match `Provenance` + +`_revinclude=Provenance:target` is the standard FHIR way to ask for the provenance of search results, +and it returns nothing here. This is a pre-existing defect, unchanged by this work, and it is not in +`FhirController`. + +Measured on the Z32 fixture, with the `DocumentReference` that the `Provenance` resources reference +white-listed so it is retained and traversed: + +``` +_include=Resource:source:DocumentReference & _revinclude=*:* -> Provenance x23 retained +_include=Resource:source:DocumentReference & _revinclude=Provenance -> none retained +``` + +Same retained resources, same traversal, different only in the type check. `includeMatches` compares +the parameter's type against `ref.getReferenceElement().getResourceType()`, and for `Provenance` that +is `null`, so `"Provenance".equals(null)` fails. `*` short-circuits the check, which is why the +wildcard form works. + +The null comes from v2tofhir. `Parser.addResource` assigns every resource a type-qualified id with +`new IdType(resource.fhirType(), id)`, but the auto-created `Provenance` is added straight to the +bundle with `p.setId(getIdGenerator().get())`, bypassing that line. Probing the converted bundle, +`Provenance` is the only one of fourteen resource types with a bare id; the other thirteen are all +type-qualified, which is why `_revinclude=Immunization` and `_revinclude=Observation` do work. + +Two candidate fixes, neither in scope here: + +1. **v2tofhir** — give the auto-created `Provenance` a type-qualified id like every other resource. + Fixes it for every consumer, and removes the inconsistency at its source. +2. **xform, defensively** — have `includeMatches` compare against the target resource's `fhirType()`, + which it already holds via `ref.getUserData(RESOURCE_KEY)`, instead of parsing the type out of the + reference string. Robust regardless of id shape, and works against older v2tofhir. + +Both are behavior additions rather than part of restoring the strict contract, so they belong in their +own change. The delta spec therefore states the general rule and records this as a limitation rather +than specifying the current behavior as intended. + +### A reverse include resolves only from a retained resource + +This is not a decision so much as a consequence that had to be discovered and then written down. +`ParserUtils.toReference(target, source, names)` records the forward reference on the source and a +reference **to the source** in the target's `Reverses` set. A `_revinclude` is therefore resolved by +traversing the resource being pointed at — so it fires only when that resource is already in the +retained set. + +Once the auto-retain is gone, this changes observable behavior on the recommendation path, because +the evaluated-history `Immunization` and the forecast `Observation` both reference the `Patient` +rather than the `ImmunizationRecommendation`. Measured on the Z42 fixture: `_revinclude=Observation` +alone returns 0 `Observation`; with `_include=*:*` added, which retains the `Patient`, it returns 15. +The same applies to `_revinclude=Immunization`, which is why the documented recovery recipe pairs it +with a forward `_include`. + +One further consequence: the schedule `Organization` behind `protocolApplied.authority` is registered +on the `Immunization` (`OBXParser` line 478), not on the `ImmunizationRecommendation`, so +`_include=Immunization:authority` is what retains it. `_include=ImmunizationRecommendation:authority` +retains the separate `Organization` the forecast itself points at (`OBXParser` line 631). A caller +wanting the evaluation data resolvable inside the searchset needs the former. + +### Retain `IMMUNIZATION_RECOMMENDATION`, `RESOURCE_KEY`, and the `Immunization` import; drop the rest + +Deleting the Z42 branch removes the only use of `IMMUNIZATION_RECOMMENDATION` inside `filter`, but +the constant is still used by `addSearchableResource` in the CapabilityStatement, and the +`Immunization` import is still used by the `subject`-to-`patient` parameter aliasing and by +identifier validation. Both stay. `RESOURCE_KEY` loses its use in `retainReferencedResources` but +keeps the one in `checkReferences`. + +`Property` and `java.util.function.Consumer` are used only by `forEachReference`, and +`java.util.HashSet` only by `clearUnresolvableReferences`. All three imports must go with the +deletions or Checkstyle's `UnusedImports` module fails the build at the `validate` phase. + +One constant is added rather than retained. `matchesSource` compared the `_include=Resource:source:…` +parameter's type against the literal `"Resource"`, which SonarQube flagged as duplicating +`RESOURCE_KEY`. They spell the same by coincidence and must not be merged: `RESOURCE_KEY` is an +internal v2tofhir user-data key that v2tofhir may rename, while the literal in `matchesSource` is a +token in the caller's URL, documented in `docs/fhir/rsp-to-fhir.md`. Sharing one constant would let a +rename of either silently change the other with no compile error, so the URL token gets its own +`SOURCE_INCLUDE_TYPE`. + +### No pipeline, transport, or persistence involvement + +Searchset assembly reads only the converted `Bundle` and the inbound `HttpServletRequest` query +parameters. It consults no repository, so neither the file nor the DynamoDB backend is touched and +`SPRING_DATABASE=migrate` has nothing to migrate. It reads no organization configuration, so no +`Organization`, `Pipeline`, `Solution`, `Operation`, or `Precondition` changes and no existing +configuration is invalidated. It performs no crypto, so the BouncyCastle FIPS providers and BCFKS +keystores are not implicated. It is not annotated `@CaptureXformAdvice` and is not a +`SolutionOperation`, so the AspectJ advice and the `aspectjweaver` / `spring-instrument` javaagents +see no change — `PipelineAdvice` records the transformations that ran, which is upstream of +assembly. + +### Flow + +Assembly sits at the end of the existing response path and is unchanged in position: + +``` +Client FhirController XformRouter/Camel Downstream (izghub|iis) + | | | | + | GET /fhir/{dest}/Immunization?_include=... | | + |----------------------->| | | + | | build HL7 v2 Z34/Z44 | | + | |---------------------->| | + | | | REQUEST-direction pipes | + | | |------------------------->| + | | | RSP (Z32/Z42) | + | | |<-------------------------| + | | | RESPONSE-direction pipes | + | | | (reverse pipe order) | + | |<----------------------| | + | | MessageParser.convert -> Bundle (type=message) | + | | | + | | filter(bundle, req): | + | | type = searchset | + | | preFilter -> match / outcome | + | | markIncludedResources -> include (per _include)| + | | cleanupBundleOfUnmarkedResources -> remove rest| + | | (references left exactly as converted) | + |<-----------------------| | + | searchset Bundle | | +``` + +With `x-loopback: true` the downstream call is short-circuited and the transformed request is +returned directly, so `filter` is not reached — loopback tests exercise the pipeline, not assembly. +The RESPONSE-direction reverse pipe ordering happens strictly before conversion, so pipe order has +no bearing on which entries survive assembly. + +### Error handling + +Assembly stays non-throwing. A missing or malformed `_include` value already resolves through +`normalizeInclude` to wildcards rather than raising, and an `_include` naming a type or search name +the conversion never registered simply matches nothing — the delta spec keeps that behavior, and the +existing "an unmatched include parameter is not an error" scenario in the main spec still governs it. +The deletions remove code paths, so they introduce no new failure mode; conversion warnings continue +to reach the caller as `OperationOutcome` entries with `search.mode = "outcome"`, which no part of +this change touches. + +## Risks / Trade-offs + +- **The eHealth Exchange pilot is already exposed to the strict behavior** → The branch has not + merged to `develop`, but CI deploys it to the dev ECS cluster, and the Newman case `TS_TC_07e` + passes there. That case asserts `search.mode = "include"` on an `_include` hit, which pre-change + code labelled `match`, so `dev.xform.izgateway.org` is demonstrably serving this branch. The + notification is therefore overdue rather than pre-emptive: give the pilot contact the parameter list + from the proposal's What Changes section, and confirm they read `Patient` from an `_include` they + send rather than from an unrequested entry. +- **Z42 evaluation data becomes hard to discover** → It is reachable through no other call, and the + three-parameter combination that reaches it in full is not something a caller would guess: + `_include=ImmunizationRecommendation:patient` to anchor the reverse lookup, + `_revinclude=Immunization:patient` for the doses, and `_include=Immunization:authority` so + `protocolApplied.authority` resolves inside the searchset — that `Organization` is registered on the + `Immunization`, not on the recommendation, as recorded above. Mitigation is documentation, not code: + `docs/fhir/rsp-to-fhir.md` carries a worked example showing all three parameters, the resulting + entry modes, and which OBX codes the included `Immunization` carry. +- **A caller doing local reference resolution inside the bundle now fails to resolve + `Immunization.patient`** → This is the defect the branch set out to fix, and this change + reinstates it deliberately: the reference is delivered with its `identifier` and `display`, and a + caller that needs the target in the bundle sends `_include`. Recorded in the delta spec's + **Migration** note for the removed requirement so the decision is not relitigated as a bug. +- **Deleting methods strands imports and fails the build late** → `UnusedImports` fires at the + `validate` phase, before compilation, so a stranded `Property`, `Consumer`, or `HashSet` import + fails fast rather than reaching CI. Task list carries an explicit Checkstyle verification step. +- **Test assertions written for the branch behavior invert rather than disappear** → The + `FhirControllerTests` cases added on the branch assert auto-retained `Patient` / `Organization`, + Z42 history, and stripped references. Each has a strict counterpart — the resource absent by + default, present when requested, and the reference intact — so the fixtures are reused and only + the assertions change. Deleting them outright would lose coverage of the new contract. + +## Migration Plan + +1. Land the deletions and the test inversion on `IGDD-3285` before the branch merges to `develop`, + so `develop` never carries the auto-retain behavior. +2. Update `docs/fhir/fhir-api.md` and `docs/fhir/rsp-to-fhir.md` in the same commit range as the + code, since both currently document the auto-retain and the reference stripping as the contract. +3. Notify the eHealth Exchange pilot with the `_include` / `_revinclude` parameter list. This is + overdue rather than pending: CI has already deployed the branch to the dev ECS cluster, so the + pilot is seeing the strict behavior on `dev.xform.izgateway.org` now. Do it before the merge to + `develop` promotes it further. +4. Rollback: revert the change commits. There is no data migration, no persisted state, and no + configuration to undo, so rollback is a redeploy of the previous image. diff --git a/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/proposal.md b/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/proposal.md new file mode 100644 index 000000000..19324acb0 --- /dev/null +++ b/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/proposal.md @@ -0,0 +1,102 @@ +## Why + +A FHIR query should return the resource type the caller asked for and nothing else. Anything +beyond that is the caller's decision, expressed with `_include` and `_revinclude`. The +`fhir-searchset-filtering` capability, as it stands on the `IGDD-3285` branch, breaks that rule: a +plain `GET /fhir/{destination}/Immunization` also returns a `Patient`, and a plain +`GET /fhir/{destination}/ImmunizationRecommendation` also returns a `Patient`, an `Organization`, +and every `Immunization` in the Z42 evaluated history — none of them requested. The branch reached +that behavior while closing a real defect (mandatory 1..1 references such as `Immunization.patient` +resolved to nothing in the delivered bundle), but the cure changed the contract the caller sees: +they can no longer predict the shape of a response from the query they sent. + +This change restores the strict contract before the branch merges. Reference resolution goes back +to what `develop` does today — an unresolved reference keeps its literal value and the caller +resolves it, or does not, on their own terms. + +## What Changes + +- **BREAKING** A FHIR query response contains only entries of the requested resource type + (`search.mode = "match"`) and `OperationOutcome` entries (`search.mode = "outcome"`). Every other + resource in the converted bundle is removed unless the caller asked for it. +- **BREAKING** Referenced resources are no longer retained just because a returned entry points at + them. `Patient` no longer accompanies an `/Immunization` query, and `Patient` and the schedule + `Organization` no longer accompany an `/ImmunizationRecommendation` query. Callers that want them + send `_include=Immunization:patient`, `_include=ImmunizationRecommendation:patient`, + `_include=ImmunizationRecommendation:authority`, or a wildcard such as `_include=*:*`. +- **BREAKING** The Z42 evaluated-history `Immunization` resources are no longer returned on an + `/ImmunizationRecommendation` query. A caller who wants the evaluated history and its OBX-derived + evaluation data asks for it — `_include=ImmunizationRecommendation:patient` together with + `_revinclude=Immunization:patient`, adding `_include=Immunization:authority` so + `protocolApplied.authority` resolves inside the searchset. `_include=*:*&_revinclude=Immunization` + reproduces the pre-change payload exactly. +- A reference whose target is not in the returned bundle keeps its `reference` element unchanged. + The service no longer strips the element down to `identifier` and `display`. This matches + `develop` and matches how a FHIR server ordinarily answers a search: the literal reference is the + caller's to resolve. +- Unchanged from the branch, and deliberately kept: an `_include` or `_revinclude` hit is labelled + `search.mode = "include"`, not `match`, so a caller can still isolate the hits their query asked + for by selecting `mode = "match"`. +- Unchanged from the branch: conversion-created resources (those carrying `Parser.SOURCE`) stay + opt-in through `_include=Resource:source:` and `_include=Resource:source:*`, or through a + forward `_include` that reaches them as the target of a reference. The branch's deferral of all + removal to a single cleanup pass is kept, and `develop`'s `MessageParser`/`Provenance` carve-out is + dropped as dead code — it set no `search.mode`, so cleanup deleted the resource regardless. + Dropping it changes no observable behavior: a type-qualified `_revinclude=Provenance` returned + nothing on `develop` and returns nothing now, for a reason unrelated to this change and recorded as + a known limitation in design.md. + +## Capabilities + +### New Capabilities + +None. + +### Modified Capabilities + +- `fhir-searchset-filtering`: Removes the requirement "A returned searchset contains no dangling + references" and the requirement "A recommendation query returns the evaluated history it was + sent with". + + Adds three requirements. "A searchset contains only what the caller asked for" states the strict + default. "A reference to a resource outside the searchset is left intact" replaces the removed + reference handling, so an unresolved reference keeps the value the conversion produced rather than + being reduced to a logical reference. "A reverse include resolves only from a retained resource" + writes down a rule the code has always had but no spec stated: a `_revinclude` is resolved by + traversing resources already in the searchset, so reaching a resource that points at another + non-requested resource takes a forward `_include` first. That rule was invisible while the + auto-retain always put the subject `Patient` in the bundle, and became observable once it was + removed. + + Modifies two requirements. The conversion-created white-list drops its dependency on the removed + dangling-reference rule. The removal of unclassified entries now governs strictly more entries, + because being referenced no longer spares an entry. The requirements covering searchset typing, + `match` labelling, `outcome` labelling and `include` labelling are unchanged. + +## Impact + +- **Inbound paths**: FHIR REST only. The SOAP/HL7 v2 inbound path (`IISHubService`, `IISService`) + is untouched. +- **Outbound paths**: none. The query sent downstream to `izghub` or `iis` is unchanged, as is the + HL7 v2 request built for it. No downstream Hub or IIS consumer sees any difference. +- **Config model**: unchanged. No `Organization`, `Pipeline`, `Solution`, `Operation`, or + `Precondition` change, and searchset assembly consults no organization configuration. Existing + organization transformation configurations are unaffected. +- **Repository backends**: unchanged. Nothing is persisted by this change, so neither the file nor + the DynamoDB backend is touched and `SPRING_DATABASE=migrate` has no migration implication. No + entry in `docs/CONFIGURATION_REFERENCE.md` or `docs/APPLICATION_CONFIGURATION_STORAGE.md` + changes; no runtime property is added. +- **Code**: `FhirController.java` — remove `retainReferencedResources` and its call, remove + `clearUnresolvableReferences` with its `toRelativeReference` and `forEachReference` helpers, and + remove the Z42 `Immunization` branch from `preFilter`. +- **Tests**: `FhirControllerTests.java` — the assertions added on this branch for auto-retained + `Patient`/`Organization`, for the Z42 evaluated history, and for reference stripping are replaced + by assertions that those resources are absent by default and present when asked for, and that an + unresolved reference keeps its literal value. +- **Backward compatibility**: this is a breaking change for a FHIR caller that relies on receiving + unrequested resources. The eHealth Exchange pilot is the known consumer; it must add the + `_include` / `_revinclude` parameters for anything beyond the requested type. Callers already on + `develop` behavior see no change other than `include` in place of `match` on join hits. +- **Docs**: `docs/fhir/fhir-api.md` and `docs/fhir/rsp-to-fhir.md` describe the auto-retain + behavior and the reference stripping, and both need rewriting around the strict contract. The + CapabilityStatement declares no `searchInclude` values, so it needs no change. diff --git a/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/specs/fhir-searchset-filtering/spec.md b/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/specs/fhir-searchset-filtering/spec.md new file mode 100644 index 000000000..5c17d0e9a --- /dev/null +++ b/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/specs/fhir-searchset-filtering/spec.md @@ -0,0 +1,341 @@ +## ADDED Requirements + +### Requirement: A searchset contains only what the caller asked for + +The service SHALL return, in the response to a FHIR query, only entries the caller asked for: +resources of the requested type, `OperationOutcome` resources, and resources retained because they +satisfied an `_include` or `_revinclude` parameter the caller supplied. It SHALL NOT retain a +resource on any other ground. In particular, the service SHALL NOT retain a resource merely because +a retained entry references it, and SHALL NOT retain a resource merely because the HL7 v2 response +happened to carry it. + +The shape of a response SHALL therefore be predictable from the query alone: the same query against +the same HL7 v2 response SHALL yield the same set of resource types regardless of the destination it +was routed to, because searchset assembly reads only the converted bundle and the query parameters +and consults no organization, pipeline or solution. + +This requirement is **BREAKING** for a caller relying on unrequested resources arriving. It governs +the FHIR REST inbound path only. The SOAP/HL7 v2 inbound message contract, the outbound query sent +to the downstream Hub or IIS, and the transformation pipeline (organizations, pipelines, solutions, +operations, and preconditions) SHALL be unchanged, and existing organization transformation +configurations SHALL continue to work untouched. + +#### Scenario: a plain immunization query returns immunizations only +- **GIVEN** a request to `GET /fhir/{destination}/Immunization` with no `_include` or `_revinclude` + parameter +- **WHEN** the converted bundle contains `Immunization` resources and the `Patient` they reference +- **THEN** the returned searchset SHALL contain the `Immunization` entries with + `search.mode = "match"` +- **AND** it SHALL NOT contain the `Patient` + +#### Scenario: a plain recommendation query returns the forecast only +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` with no `_include` or + `_revinclude` parameter +- **WHEN** the converted bundle contains an `ImmunizationRecommendation`, the `Patient`, the + schedule `Organization` behind `authority`, `Immunization` resources from the evaluated history, + and forecast `Observation` resources +- **THEN** the returned searchset SHALL contain only the `ImmunizationRecommendation` entries, with + `search.mode = "match"` + +#### Scenario: the caller asks for the subject patient +- **GIVEN** the same immunization query +- **WHEN** the caller supplies `_include=Immunization:patient` +- **THEN** the `Patient` SHALL be retained with `search.mode = "include"` + +#### Scenario: the caller asks for everything the returned resources reference +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` +- **WHEN** the caller supplies `_include=*:*` +- **THEN** every resource reachable by a reference from a retained entry SHALL be retained with + `search.mode = "include"` + +#### Scenario: the caller asks for the evaluated history and its evaluation data +- **GIVEN** a Z42 response containing administered doses and forecasts for one patient +- **WHEN** the caller issues `GET /fhir/{destination}/ImmunizationRecommendation` with + `_include=ImmunizationRecommendation:patient`, `_revinclude=Immunization:patient`, and + `_include=Immunization:authority` +- **THEN** the `Patient` and one `Immunization` per administered dose SHALL be retained with + `search.mode = "include"` +- **AND** the `Organization` behind `protocolApplied.authority` SHALL be retained, because the + authority is registered on the `Immunization` and not on the `ImmunizationRecommendation` +- **AND** the `ImmunizationRecommendation` SHALL be the only entry labelled `match` + +#### Scenario: the routing destination does not change the returned types +- **GIVEN** the same FHIR query issued against two different destinations +- **WHEN** both produce the same HL7 v2 response +- **THEN** the two returned searchsets SHALL contain the same resource types with the same + `search.mode` on each + +### Requirement: A reverse include resolves only from a retained resource + +The service SHALL resolve a `_revinclude` parameter against the reverse references of resources +already retained in the searchset. A resource that references a retained entry SHALL therefore be +reached by `_revinclude` only when the entry it references is itself retained, and a caller SHALL be +able to retain that intermediate entry with `_include`. + +This follows from how the HL7 v2 to FHIR conversion bookkeeps references: a reverse reference is +recorded on the resource being pointed at, so it is discoverable only by traversing that resource. +The consequence is observable and callers depend on it — the evaluated-history `Immunization` and the +forecast `Observation` both reference the `Patient` rather than the `ImmunizationRecommendation`, so +neither is reachable on a recommendation query until the `Patient` is retained. + +Any retained resource that the sought resource references SHALL serve as the anchor; the anchor need +not be the one whose search name the parameter names. The conversion keeps a single canonical +`Reference` per resource and accumulates every reverse search name onto it, so a qualified +`_revinclude` matches when the named search name is among the accumulated names, regardless of which +retained resource the reverse reference was found on. A search name the conversion never registered +SHALL match nothing. + +A `_revinclude` naming a resource type SHALL match on the resource type carried by the reverse +reference. A `_revinclude=*` SHALL match any type. Where the conversion produces a resource whose id +carries no resource type, a type-qualified `_revinclude` cannot identify it and matches nothing while +the wildcard form still reaches it. `Provenance` is the only resource type the HL7 v2 to FHIR +conversion currently gives such an id, so `_revinclude=Provenance` matches nothing while +`_revinclude=*` and `_include=Resource:source:Provenance` both reach it. That is a defect in the +conversion rather than intended behavior, and this capability does not require it to stay that way. + +#### Scenario: a reverse include finds nothing when the referenced entry is absent +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` with + `_revinclude=Observation` and no `_include` +- **WHEN** the forecast `Observation` resources reference the `Patient`, which is not retained +- **THEN** no `Observation` entry SHALL be retained +- **AND** the request SHALL succeed + +#### Scenario: retaining the intermediate entry makes the reverse include resolve +- **GIVEN** the same request with `_include=ImmunizationRecommendation:patient` added +- **WHEN** the searchset is assembled +- **THEN** the `Patient` SHALL be retained with `search.mode = "include"` +- **AND** every `Observation` referencing it SHALL be retained with `search.mode = "include"` + +#### Scenario: another retained resource serves as the anchor +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` with + `_include=ImmunizationRecommendation:authority` and `_revinclude=Immunization:patient`, and no + `_include` naming `patient` +- **WHEN** the evaluated-history `Immunization` reference both the `Patient`, which is not retained, + and the schedule `Organization`, which is +- **THEN** those `Immunization` entries SHALL be retained with `search.mode = "include"` +- **AND** the `Patient` SHALL NOT be retained + +#### Scenario: an unregistered reverse search name matches nothing +- **GIVEN** the same request with `_revinclude=Immunization:nosuchsearchname` +- **WHEN** the searchset is assembled +- **THEN** no `Immunization` entry SHALL be retained +- **AND** the request SHALL succeed + +#### Scenario: the evaluated history is reached through the subject patient +- **GIVEN** a Z42 response and a request to `GET /fhir/{destination}/ImmunizationRecommendation` +- **WHEN** the caller supplies `_include=ImmunizationRecommendation:patient` and + `_revinclude=Immunization:patient` +- **THEN** one `Immunization` per administered dose SHALL be retained with `search.mode = "include"` + +### Requirement: A reference to a resource outside the searchset is left intact + +The service SHALL leave every `Reference` on a retained entry as the HL7 v2 to FHIR conversion +produced it. Where the referenced resource is not present in the returned searchset, the service +SHALL NOT remove, rewrite, or reduce the `reference` element, and SHALL NOT alter the reference's +`identifier` or `display`. Resolving such a reference is the caller's decision: the literal value, +the identifier, and the display text are all delivered, and the caller may resolve the reference +against its own source of truth, request the target with `_include`, or ignore it. + +This restores the reference handling callers saw before this capability existed, and it applies to +mandatory 1..1 references — `Immunization.patient`, `ImmunizationRecommendation.patient` — as much +as to optional ones. + +The service makes no guarantee that every delivered reference is readable. Where the HL7 v2 to FHIR +conversion produces a `Reference` carrying no `reference`, no `identifier`, and no `display`, the +service SHALL deliver it as produced. Populating such a reference is the conversion's concern, not +the searchset's. + +#### Scenario: a mandatory reference keeps its literal value +- **GIVEN** a request to `GET /fhir/{destination}/Immunization` with no `_include` parameter +- **WHEN** the returned `Immunization` entries carry a `patient` reference to a `Patient` that the + searchset does not contain +- **THEN** each `patient` reference SHALL retain the `reference` value the conversion produced +- **AND** its `identifier` and `display` SHALL be unchanged + +#### Scenario: an optional reference to an omitted resource is left alone +- **GIVEN** a returned recommendation searchset +- **WHEN** the retained `ImmunizationRecommendation` carries an `authority` reference to an + `Organization` the searchset does not contain +- **THEN** that reference SHALL be delivered unchanged + +#### Scenario: a reference the conversion left empty is delivered as produced +- **GIVEN** a converted bundle in which a retained entry holds a `Reference` with no `reference`, no + `identifier`, and no `display` +- **WHEN** the searchset is assembled +- **THEN** that reference SHALL be delivered unchanged +- **AND** the searchset SHALL NOT be rejected on that account + +#### Scenario: asking for the target changes nothing about the reference +- **GIVEN** the same immunization query +- **WHEN** the caller supplies `_include=Immunization:patient` so the `Patient` is retained +- **THEN** the `patient` reference SHALL carry the same value it carries when the `Patient` is + omitted +- **AND** it SHALL resolve to the retained `Patient` + +## MODIFIED Requirements + +### Requirement: `_include` and `_revinclude` results are labelled `include` + +The service SHALL set `search.mode = "include"` on every resource retained because it satisfied an +`_include` or `_revinclude` parameter. It SHALL NOT label such a resource `match`. In FHIR R4, +`include` is the defined value for an entry "added to the results because of a join", and clients +filter on `mode = "match"` to obtain the hits; labelling joined resources `match` makes the hits +indistinguishable from their supporting resources. + +An `_include` or `_revinclude` parameter SHALL be resolved against the search-parameter names the +HL7 v2 to FHIR conversion registered for each reference. A parameter naming a resource type or +search name that is not registered SHALL match nothing and SHALL NOT be an error. Wildcard (`*`) +resource types and search names SHALL match any value. + +This is a **BREAKING** change to the labelling a client observes: a client that previously read +`_include` and `_revinclude` results as `match` SHALL now read them as `include`. +Reverse resolution depends on the referencing resource reaching a retained entry, per "A reverse +include resolves only from a retained resource". On the immunization path the `Observation` reference +the retained `Immunization` directly, so `_revinclude=Observation` resolves with no forward +`_include`. On the recommendation path they reference the `Patient` rather than the +`ImmunizationRecommendation`, so the same parameter alone retains nothing. + + +#### Scenario: a reverse-included resource is labelled include +- **GIVEN** a request to `GET /fhir/{destination}/Immunization` with `_revinclude=Observation` +- **WHEN** the converted bundle contains `Immunization` resources and dose-level `Observation` + resources that reference them through `partOf` +- **THEN** the `Immunization` entries SHALL have `search.mode = "match"` +- **AND** every retained `Observation` entry SHALL have `search.mode = "include"` +- **AND** the caller SHALL be able to obtain exactly the requested resources by selecting entries + with `search.mode = "match"` + +#### Scenario: a forward-included resource is labelled include +- **GIVEN** a request to a FHIR query endpoint with an `_include` parameter that matches a reference + on a resource of the requested type +- **WHEN** the searchset is assembled +- **THEN** the referenced resource SHALL be retained with `search.mode = "include"` + +#### Scenario: an unmatched include parameter is not an error +- **GIVEN** a request with an `_include` or `_revinclude` parameter naming a search name that the + conversion does not register for any reference +- **WHEN** the searchset is assembled +- **THEN** the request SHALL succeed +- **AND** no additional entry SHALL be retained on account of that parameter + +### Requirement: Conversion-created resources are retained only when white-listed + +Resources that the HL7 v2 to FHIR conversion synthesises as a side effect of datatype and message +parsing — rather than from a dedicated segment the caller queried for — SHALL be removed from the +searchset unless the caller white-lists them, because the enriched reference they are the target of +already carries an identifier and display text sufficient for production use. + +The caller SHALL be able to white-list them with the `_include=Resource:source:` parameter, +where `` is a resource type or `*` for all such resources; a white-listed resource SHALL be +retained with `search.mode = "include"`. Such a resource SHALL also be retained when a forward +`_include` reaches it as the target of a reference on a retained entry, on the same terms as any +other resource — so `_include=Immunization:location` retains the conversion-created `Location` +without the `Resource:source` form. + +A white-listed resource SHALL participate in `_include` and `_revinclude` traversal on the same terms +as any other retained resource. Being retained by the white-list rather than by a caller's join does +not exempt it from being traversed, so a `_revinclude` MAY reach a further resource through it. + +#### Scenario: conversion-created resources are removed by default +- **GIVEN** a query whose converted bundle contains conversion-created `Practitioner` and `Location` + resources that no retained entry references +- **WHEN** the caller supplies no `_include=Resource:source:...` parameter +- **THEN** those entries SHALL NOT appear in the returned searchset + +#### Scenario: a caller white-lists conversion-created resources by type +- **GIVEN** the same query +- **WHEN** the caller supplies `_include=Resource:source:Practitioner` +- **THEN** the conversion-created `Practitioner` entries SHALL be retained with + `search.mode = "include"` + +#### Scenario: a caller white-lists all conversion-created resources +- **GIVEN** the same query +- **WHEN** the caller supplies `_include=Resource:source:*` +- **THEN** every conversion-created resource SHALL be retained with `search.mode = "include"` + +#### Scenario: a forward include reaches a conversion-created resource +- **GIVEN** a query whose converted bundle contains a conversion-created `Location` that a retained + `Immunization` references +- **WHEN** the caller supplies `_include=Immunization:location` and no `Resource:source` parameter +- **THEN** that `Location` SHALL be retained with `search.mode = "include"` + +#### Scenario: a white-listed resource is traversed like any other +- **GIVEN** a query whose converted bundle contains conversion-created `Provenance` resources, which + reference a conversion-created `DocumentReference` rather than a resource of the requested type +- **WHEN** the caller supplies `_include=Resource:source:DocumentReference` and `_revinclude=*:*` +- **THEN** the `DocumentReference` SHALL be retained by the white-list +- **AND** the `Provenance` resources referencing it SHALL be retained with `search.mode = "include"`, + reached by traversing the white-listed `DocumentReference` + +#### Scenario: a reverse include reaching nothing is not an error +- **GIVEN** the same query +- **WHEN** a `_revinclude` names a resource type that no reverse reference on any retained resource + resolves to +- **THEN** no additional entry SHALL be retained +- **AND** the request SHALL succeed + +### Requirement: Unclassified entries are removed + +The service SHALL remove from the returned searchset every entry that no requirement in this +capability retains, whether or not a retained entry references it. A caller SHALL therefore never +receive an entry whose `search.mode` is absent. + +#### Scenario: an unreferenced, unrequested resource is removed +- **GIVEN** a converted bundle containing a resource that is not of the requested type, is not an + `OperationOutcome`, satisfies no `_include` or `_revinclude` parameter, and is referenced by no + retained entry +- **WHEN** the searchset is assembled +- **THEN** that entry SHALL NOT appear in the returned searchset + +#### Scenario: being referenced does not save an entry from removal +- **GIVEN** a converted bundle containing a resource that satisfies no `_include` or `_revinclude` + parameter but is referenced by a retained entry +- **WHEN** the searchset is assembled +- **THEN** that entry SHALL NOT appear in the returned searchset +- **AND** the reference to it SHALL be delivered unchanged, per "A reference to a resource outside + the searchset is left intact" + +#### Scenario: forecast observations are removed from a plain recommendation query +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` with no `_include` or + `_revinclude` parameter +- **WHEN** the converted bundle contains `Observation` resources carrying the forecast detail +- **THEN** those `Observation` entries SHALL NOT appear in the returned searchset + +## REMOVED Requirements + +### Requirement: A returned searchset contains no dangling references + +**Reason**: The guarantee was bought by returning resources the caller did not ask for — the +subject `Patient` on every query, and the schedule `Organization` on a recommendation query — which +made the shape of a response unpredictable from the query. Its fallback behavior, reducing an +unretainable reference to `identifier` and `display`, also silently discarded the literal reference +value the conversion produced. Both are replaced by "A searchset contains only what the caller asked +for" and "A reference to a resource outside the searchset is left intact": the searchset may now +contain a reference to a resource it does not hold, exactly as it did before this capability +existed. + +**Migration**: A caller that relied on the referenced resource arriving unasked SHALL request it — +`_include=Immunization:patient`, `_include=ImmunizationRecommendation:patient`, +`_include=ImmunizationRecommendation:authority`, or `_include=*:*` for all of them. A caller that +resolves references locally within the bundle SHALL either request the targets with `_include` or +resolve the reference's `identifier` and `display` against its own source of truth; the service +serves no read endpoint for a reference target, so a literal reference value in a returned searchset +is not fetchable from this service. + +### Requirement: A recommendation query returns the evaluated history it was sent with + +**Reason**: The evaluated history is not what an `ImmunizationRecommendation` query asked for. +Returning it unasked contradicts "A searchset contains only what the caller asked for", and the +`include` labelling it used already told the caller these entries were not hits — so the caller can +ask for them by the ordinary means instead. + +**Migration**: A caller that wants the Z42 evaluated history and the evaluation data it carries — +`protocolApplied.doseNumber` and `seriesDoses` (OBX `30973-2` / `59782-3`), +`protocolApplied.authority` (OBX `59779-9`), and `programEligibility` (OBX `64994-7`) — SHALL send +`_include=ImmunizationRecommendation:patient&_revinclude=Immunization:patient`, adding +`_include=Immunization:authority` to resolve `protocolApplied.authority` within the searchset, or +`_include=*:*&_revinclude=Immunization` to obtain everything the pre-change service returned. The +`Immunization` resources are reached through the `Patient` they reference, so retaining the `Patient` +with `_include` is required for the `_revinclude` to reach them, per "A reverse include resolves only +from a retained resource". This data remains reachable through no other call: the `/Immunization` +path sends Z34 and receives Z32, which carries none of those OBX codes. diff --git a/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/tasks.md b/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/tasks.md new file mode 100644 index 000000000..f2641e6be --- /dev/null +++ b/openspec/changes/archive/2026-08-17-fhir-searchset-strict-includes/tasks.md @@ -0,0 +1,267 @@ +## 0. Capture the pre-change baseline + +This group runs first, while the auto-retain code is still in place. Task 4.2 asserts against what it +records, so it cannot be taken after the deletions in groups 1 and 2. + +- [x] 0.1 Run the Z42 test message through `GET .../ImmunizationRecommendation` with no query + parameters and record, per entry, the resource type and `search.mode`. Keep the record in the + test class as the expected value for the round-trip test, not as a checked-in JSON fixture, so + it stays readable next to the assertion that uses it. + + Recorded, 7 entries: `OperationOutcome` x2 `outcome`, `ImmunizationRecommendation` x1 `match`, + `Patient` x1 `include`, `Immunization` x2 `include`, `Organization` x1 `include`. No + `Location`, manufacturer `Organization`, or performer `Practitioner` — the Z42 fixture's + `ORC`/`RXA` segments carry no RXA-10 performer and no RXA-11 facility, so the conversion never + creates them. +- [x] 0.2 Do the same for the Z32 test message through `GET .../Immunization` with no query + parameters, and additionally record which references have no `reference` element — the + `PractitionerRole` to `Practitioner` case that task 4.5 inverts. + + Recorded, 10 entries: `OperationOutcome` x2 `outcome`, `Immunization` x2 `match`, `Patient` x1 + `include`, `PractitionerRole` x2 `include`, `Practitioner` x1 `include`, `Location` x2 + `include`. Five references arrive with no `reference` element — four on the two + `PractitionerRole` resources and one on a `Location`. + + Corrected after the deletions landed: only **two** of those five were cleared by + `clearUnresolvableReferences`, the two carrying `display = "Carl Clinician"`. The other three + carry no `reference`, no `identifier` and no `display` as v2tofhir produces them, and they stay + empty afterwards. So this change restores two references, not five, and does not introduce the + three empty ones. Task 4.5 asserts the corrected numbers, and the delta spec records that the + searchset does not synthesise content for a reference the conversion left empty. + +## 1. Remove the unrequested-retain behavior + +- [x] 1.1 In `FhirController.java`, delete the `retainReferencedResources(resources, refs)` call from + `markIncludedResources` and delete the `retainReferencedResources` method with its Javadoc. +- [x] 1.2 Delete the `r instanceof Immunization && IMMUNIZATION_RECOMMENDATION.equals(requested)` + branch from `preFilter`, so the method marks only the requested type and `OperationOutcome` + before falling through to `whitelistInfrastructureCreatedResources`. +- [x] 1.3 Rewrite the `preFilter` Javadoc: drop the Z42 evaluated-history rationale paragraph and + state the strict rule instead, noting that the evaluated history and its OBX-derived data are + now reached with `_include=ImmunizationRecommendation:patient` plus + `_revinclude=Immunization:patient`. +- [x] 1.4 Confirm `IMMUNIZATION_RECOMMENDATION` is still referenced by `addSearchableResource` and + that the `org.hl7.fhir.r4.model.Immunization` import is still used by the `subject`-to-`patient` + aliasing and identifier validation, so neither is removed. + +## 2. Remove the reference stripping + +- [x] 2.1 Delete the `clearUnresolvableReferences(bundle)` call from `filter`, so `filter` ends at + `cleanupBundleOfUnmarkedResources`. +- [x] 2.2 Delete `clearUnresolvableReferences`, `toRelativeReference`, and `forEachReference` with + their Javadoc. +- [x] 2.3 Remove the now-unused imports `java.util.HashSet`, `java.util.function.Consumer`, and + `org.hl7.fhir.r4.model.Property`. Confirm `RESOURCE_KEY` is still used by `checkReferences` + and that `java.util.Set` is still used by the `References` / `Reverses` handling. +- [x] 2.4 Confirm `checkReferences` still sets `SearchEntryMode.INCLUDE` on an `_include` / + `_revinclude` hit — this labelling is deliberately kept, not reverted. +- [x] 2.5 Confirm `whitelistInfrastructureCreatedResources` still removes nothing and that all + removal remains in the single `cleanupBundleOfUnmarkedResources` pass. + + Verified. A white-listed resource **is** added to `resources` and **is** traversed, so it can + anchor a `_revinclude` — `_include=Resource:source:DocumentReference&_revinclude=*:*` retains 23 + `Provenance` on the Z32 fixture. The reason `_revinclude=Provenance` returns nothing is unrelated to white-listing: + v2tofhir gives `Provenance` a bare id, so the type check in `includeMatches` compares + `"Provenance"` against `null`. `develop`'s carve-out was dead code, since it set no + `search.mode` and cleanup deleted the resource anyway. Root cause and candidate fixes recorded + in design.md as a known limitation, out of scope here. + +## 3. Invert the branch's unit tests + +- [x] 3.1 In `FhirControllerTests.java`, delete the `assertNoDanglingReferences` and + `collectReferences` helpers and the `org.hl7.fhir.r4.model.Base` import they need. + + Corrected in delivery: `assertNoDanglingReferences` was deleted, but `collectReferences` and + the `Base` import were kept and repurposed under the `referencesOf` helper — the new + no-stripping tests (3.6, 4.5) need to walk a resource's reference tree. +- [x] 3.2 Replace `recommendationQueryRetainsPatientAndAuthorityOrganization` with a test asserting + that a plain `GET .../ImmunizationRecommendation` returns neither the `Patient` nor the + schedule `Organization`, and a second test asserting both arrive with + `search.mode = "include"` when `_include=ImmunizationRecommendation:patient` and + `_include=ImmunizationRecommendation:authority` are supplied. +- [x] 3.3 Replace `recommendationQueryIncludesTheEvaluatedHistory` with a test asserting no + `Immunization` entry appears on a plain recommendation query. +- [x] 3.4 Rework `includedHistoryCarriesTheZ42OnlyEvaluationData` and + `includedHistoryHasStableDistinctIdentifiers` to send the history parameters, then keep their + existing assertions on `protocolApplied.doseNumber`, `seriesDoses`, `protocolApplied.authority`, + `programEligibility`, and per-dose ORC-3 identifiers. + + Delivered as the shared `HISTORY_PARAMS` constant, which needed a third parameter beyond the + `_include=ImmunizationRecommendation:patient&_revinclude=Immunization:patient` originally + written here: `_include=Immunization:authority`, without which the `protocolApplied.authority` + assertion fails because that `Organization` is registered on the `Immunization` rather than on + the recommendation. +- [x] 3.5 Replace `immunizationQueryResolvesItsPatientReference` with a test asserting that on a + plain `GET .../Immunization` the `Patient` is absent and every `Immunization.patient` keeps the + literal `reference` value the conversion produced, with `identifier` and `display` unchanged. +- [x] 3.6 Delete `unretainableTargetIsReducedToALogicalReference` and replace it with a test + asserting no `Reference` in a returned searchset has had its `reference` element cleared. Task + 4.5 covers the specific `PractitionerRole` case this test used, so do not duplicate it here. +- [x] 3.7 Narrow `plainRecommendationQueryIsSelfContainedWithoutObservations` to the `Observation` + absence assertion, dropping the self-containment claim about references. +- [x] 3.8 Run the tests kept unchanged and confirm they still pass: + `revincludedObservationsAreLabelledIncludeNotMatch`, `selectingMatchYieldsOnlyTheRequestedType`, + `observationsArriveOnlyWhenRevincludedAndNeverAsMatch`, `unmatchedIncludeParameterIsNotAnError`, + `immunizationQueryStillReturnsHistoryAsMatch`, + `conversionCreatedResourcesStillNeedWhitelistingWhenUnreferenced`, + `outcomesSurviveAndBundleIsASearchset`, `namedTypeWhitelistRetainsOnlyThatType`, + `partOfRevincludeNarrowsToTheHistoryObservations`, `matchOperationLabelsThePatientAsMatch`. + +## 4. Add unit tests for the new requirements + +- [x] 4.1 Add a test for "a plain immunization query returns immunizations only": on the Z32 fixture + the searchset holds exactly 4 entries — `Immunization` x2 as `match` and `OperationOutcome` x2 + as `outcome` — down from the 10 recorded in task 0.2. Assert `Patient`, `PractitionerRole`, + `Practitioner`, and `Location` are all absent. +- [x] 4.2 Add a round-trip test proving `_include=*:*&_revinclude=Immunization` reproduces the task + 0.1 baseline exactly: 7 entries, `ImmunizationRecommendation` x1 as `match`, + `OperationOutcome` x2 as `outcome`, and `Patient` x1, `Immunization` x2, `Organization` x1 as + `include`. Compare the full type-and-mode multiset against the recorded baseline rather than + spot-checking types, so a regression in either direction fails. This is the parameter pair + task 5.4 documents as the way to recover the old payload, and nothing currently tests + `_include=*:*` at all despite `docs/fhir/fhir-api.md` advertising it. +- [x] 4.3 Add a test that `_include=*:*&_revinclude=Immunization` still returns no `Observation` + entry, matching the pre-change payload — `retainReferencedResources` never walked the reverse + direction, and naming only `Immunization` in the `_revinclude` preserves that. +- [x] 4.4 Add a test that `_include=*:*` alone, with no `_revinclude`, returns no `Immunization` on a + recommendation query, since the evaluated history is reachable only in reverse. This pins why + the second parameter is not optional. Note that the task 0.1 baseline run cannot pre-verify + this: with the auto-retain still in place, `_include=*:*` alone returns the same 7 entries, + because `preFilter` supplies the `Immunization` regardless of any parameter. The assertion is + only meaningful after the deletions in group 1. +- [x] 4.5 Add a test for the cleared references on the Z32 immunization path — the case + `unretainableTargetIsReducedToALogicalReference` covered. Under `_include=*:*` the + `PractitionerRole` and `Location` resources are retained, and the two references that carried a + `display` now keep the literal value the conversion produced. + + Correcting task 0.2: of the five references recorded without a `reference` element, only two + were stripped by `clearUnresolvableReferences`. The other three carry no `reference`, no + `identifier`, and no `display` as v2tofhir produces them, and they stay empty after the + deletion. Do not assert that no empty `Reference` exists — assert instead that the count of + references lacking a literal value dropped from five to three, so the test pins what this + change actually did. +- [x] 4.6 Add a test that a `patient` reference carries the same value whether or not + `_include=Immunization:patient` retained the target, and resolves to the retained `Patient` + when it did. +- [x] 4.7 Add a test pinning that `_revinclude` does not reach a white-listed resource: + `_revinclude=Provenance` alone retains no `Provenance`, and adding + `_include=Resource:source:DocumentReference` retains the `DocumentReference` but still no + `Provenance`. Assert both requests succeed. This replaces the assumption corrected in task 2.5. +- [x] 4.7a Add a test for the general rule that a reverse include resolves only from a retained + resource: on the Z42 recommendation path `_revinclude=Observation` alone retains no + `Observation`, and adding `_include=ImmunizationRecommendation:patient` makes the same + `_revinclude` retain them as `include`. +- [x] 4.8 Add a Z42 fixture variant carrying RXA-10 performer and RXA-11 facility on the two + administered doses, so the conversion produces `Location` and performer resources the existing + `RSP_Z42_MESSAGE` does not. Task 0.1 established the current fixture yields no `Location` at + all, so the four-hop chain cannot be tested without this. Add it as a separate constant and + leave `RSP_Z42_MESSAGE` untouched, so no existing entry-count assertion moves. +- [x] 4.9 Using that fixture, add a test for the four-type query a caller actually sends — + `_include=ImmunizationRecommendation:patient&_include=ImmunizationRecommendation:authority&_revinclude=Immunization&_include=Immunization:location` + — asserting `Patient`, `Organization`, `Immunization`, and `Location` all arrive as `include` + and no `Observation` does. This exercises the four-hop chain through the growing `resources` + list (recommendation to `Patient` and `Organization` forward, `Patient` to `Immunization` + reverse, `Immunization` to `Location` forward) in a single pass. Add a second assertion running + the same four parameters in reversed URL order and comparing the two searchsets, confirming + parameter order does not matter. +- [x] 4.10 Add tests for what anchors a reverse include. Delivered as two: + `revincludeWithNoRetainedAnchorFindsNothing`, which drops **every** forward `_include` from the + task 4.9 query and gets no `Immunization`, and `anyRetainedReferencedResourceAnchorsTheReverseInclude`, + which keeps only `_include=ImmunizationRecommendation:authority` and still gets the doses. + + Originally written as "dropping `_include=ImmunizationRecommendation:patient` returns no + `Immunization`". That is false: the 4.9 query also carries + `_include=ImmunizationRecommendation:authority`, and the retained `Organization` anchors the + reverse hit in the `Patient`'s place, because the doses reference it through + `protocolApplied.authority`. Split into the two tests above once measured. +- [x] 4.11 Add a test that `Organization` and `Location` arrive under an ordinary `_include` without + any `_include=Resource:source:...` parameter, even though both are conversion-created + (`Parser.SOURCE`). This is the behavior the modified white-list requirement pins. +- [x] 4.12 Add a test that the routing destination does not affect the returned resource types: issue + the same query against two destinations and assert the two searchsets hold the same resource + types and search modes. Originally written as "two organizations whose pipelines apply different + transformations"; narrowed because searchset assembly takes no organization, pipeline or solution + input, so a pipeline fixture would be testing the absence of a wire that never existed. +- [x] 4.13 Remove the temporary `zzzTemporaryBaselineDump` test and its `dumpBaseline` helper, added + only to capture the group 0 baseline. +- [x] 4.14 Run the full class via Maven so the surefire env vars and `target/` BCFKS keystores are in + place: `mvn test -Dtest=FhirControllerTests`. Note that the Mockito inline mock maker needs JVM + self-attach, so this cannot run inside a restricted sandbox. + +## 5. Update documentation + +- [x] 5.1 Rewrite the searchset section of `docs/fhir/fhir-api.md`: state that a query returns the + requested type plus `OperationOutcome` only, correct the claim that a referenced resource is + returned with the records that reference it, correct the claim that `_include=Immunization:patient` + and `_include=Immunization:performer` "add nothing", and remove the passage describing a + reference reduced to `identifier` and `display`. +- [x] 5.2 Rewrite the filtering and search-mode sections of `docs/fhir/rsp-to-fhir.md`: redefine + `include` as an `_include` / `_revinclude` hit or a `Resource:source` white-list hit only, and + drop "a resource a match references". +- [x] 5.3 Add a worked Z42 example to `docs/fhir/rsp-to-fhir.md` showing + `_include=ImmunizationRecommendation:patient&_revinclude=Immunization:patient&_include=Immunization:authority`, + the entry modes it produces, and which OBX codes (`30973-2`, `59782-3`, `59779-9`, `64994-7`) + the included `Immunization` carry — the discoverability mitigation from design.md. State that + the `_include` on `patient` is not optional: the `Immunization` reference the `Patient`, not the + recommendation, so the `_revinclude` finds nothing without the `Patient` retained. State that + `_include=Immunization:authority` is what resolves `protocolApplied.authority` inside the + searchset, because that `Organization` is registered on the `Immunization` and not on the + `ImmunizationRecommendation`. +- [x] 5.3a Document the general rule in `docs/fhir/rsp-to-fhir.md`: a `_revinclude` resolves only + from a resource already in the searchset, so reaching a resource that references another + non-requested resource takes a forward `_include` first. Note that any retained resource can + serve as that anchor, including one retained by the `Resource:source` white-list — a white-listed + resource is traversed like any other — and that a type-qualified `_revinclude` matches nothing + for a resource whose id carries no resource type, which today is only `Provenance`. +- [x] 5.4 Add a migration note to `docs/fhir/fhir-api.md` for callers upgrading from the pre-change + behavior: `_include=*:*&_revinclude=Immunization` on an `ImmunizationRecommendation` query + returns the same entry set the service returned before, with the one difference that a + reference built outside the conversion's bookkeeping — `PractitionerRole` to `Practitioner` — + now keeps its literal value rather than being reduced to `identifier` and `display`. Note that + `_include=*:*` alone does not return the evaluated history, because it is reachable only in + reverse. +- [x] 5.5 Document the search-parameter names a caller can actually use, since an unregistered name + silently matches nothing: `ImmunizationRecommendation:patient`, + `ImmunizationRecommendation:authority`, `Immunization:patient`, `Immunization:authority`, + `Immunization:location`, `Immunization:performer`, `Immunization:manufacturer`, + `Observation:partof` / `part-of`, and `Resource:source:`. Note which are forward-only. +- [x] 5.6 Document that a reference in a returned searchset may point outside it and that this + service serves no read endpoint for the target, so the caller resolves it with `_include` or + against its own data using `identifier` and `display`. +- [x] 5.7 Confirm no change is needed in `docs/CONFIGURATION_REFERENCE.md`, + `docs/APPLICATION_CONFIGURATION_STORAGE.md`, or `docs/QUICK_START.md` — this change adds no + runtime property, touches no configuration entity, and alters no `curl` example there. + +## 6. Update the integration test collection + +- [x] 6.1 Confirm the existing `TS_TC_07*` / `TS_TC_08*` FHIR cases in + `testing/scripts/TS_Integration_Test.postman_collection.json` still pass — they assert only + `Bundle` type, at least one entry, and `Immunization` presence with vaccine code `208`, none of + which this change alters. +- [x] 6.2 Add a case asserting that a plain FHIR `Immunization` query returns no `Patient` entry. +- [x] 6.3 Add a case sending `_include=Immunization:patient` and asserting the `Patient` is present + with `entry.search.mode == "include"`. +- [x] 6.4 Deferred, not delivered. A case sending `_include=*:*&_revinclude=Immunization` against + `ImmunizationRecommendation` was added and then removed: IZ Gateway Hub does not currently + respond with Z42, so no dev fixture produces a forecast and the case failed with no + `ImmunizationRecommendation` in the bundle. The recovery recipe stays covered by the unit test + `recoveryParametersReproduceThePreChangePayload`, which runs against the Z42 fixture directly. + Re-add the integration case once the Hub returns Z42 (separate work, tracked by the team). +- [x] 6.5 Mirror the new cases into the JWT Okta folder, matching how the existing FHIR cases are + duplicated there. Two delivered (`TS_TC_07d`, `TS_TC_07e`); the third is deferred per task 6.4. + +## 7. Build gates and review + +- [x] 7.1 Run `mvn clean package -DskipDependencyCheck=true` and confirm Checkstyle passes at the + `validate` phase, with particular attention to `UnusedImports` after the deletions in tasks 1 + and 2. +- [x] 7.2 Confirm no `@JsonSubTypes` registration is needed: this change adds no `Operation` or + `Precondition` subclass, so `Operation.java` and `Precondition.java` are untouched. +- [x] 7.3 Confirm no security review is required: no `AccessControlValve`, `Roles`, `XformPrincipal`, + mTLS, JWT, BouncyCastle FIPS, or BCFKS keystore code is touched, and no authorization decision + depends on searchset assembly. +- [x] 7.4 Run `mvn verify` and confirm the OWASP dependency-check result is unchanged and stays under + CVSS 7 — this change adds and removes no dependency. +- [x] 7.5 Delete the untracked scratch file + `src/test/java/gov/cdc/izgateway/xform/endpoints/fhir/AkZ42Scratch.java` before the branch + merges. diff --git a/openspec/specs/fhir-searchset-filtering/spec.md b/openspec/specs/fhir-searchset-filtering/spec.md new file mode 100644 index 000000000..ab07d242b --- /dev/null +++ b/openspec/specs/fhir-searchset-filtering/spec.md @@ -0,0 +1,366 @@ +# fhir-searchset-filtering Specification + +## Purpose + +This capability defines how the IZ Gateway Transformation Service turns a converted HL7 v2 response +into a FHIR R4 searchset for a FHIR query caller: which entries the response bundle retains, what +`Bundle.entry.search.mode` each retained entry carries, and how `_include` and `_revinclude` are +resolved. It applies to every FHIR query endpoint the service exposes and is independent of the +transformation pipeline, which does not participate in searchset assembly. + +## Requirements + +### Requirement: FHIR query responses are returned as a searchset + +The service SHALL return the response to a FHIR query as a `Bundle` of type `searchset`, regardless +of the bundle type produced by the HL7 v2 to FHIR conversion. Every retained entry SHALL carry a +populated `Bundle.entry.search.mode`. The requested resource type SHALL be determined from the +request path, and for the `Patient/$match` operation the requested type SHALL be `Patient`. + +This requirement governs the FHIR REST inbound path only. The SOAP/HL7 v2 inbound path and the +outbound query sent to the downstream Hub or IIS SHALL be unchanged, as SHALL the transformation +pipeline (organizations, pipelines, solutions, operations, and preconditions) — no organization +configuration affects searchset assembly. + +#### Scenario: response bundle is typed as a searchset +- **GIVEN** a query to a FHIR query endpoint that produces a response bundle +- **WHEN** the service returns the response to the caller +- **THEN** `Bundle.type` SHALL be `searchset` + +#### Scenario: every retained entry carries a search mode +- **GIVEN** a returned searchset +- **WHEN** the caller inspects any entry in the bundle +- **THEN** that entry SHALL have a populated `Bundle.entry.search.mode` + +#### Scenario: the $match operation resolves to the Patient type +- **GIVEN** a request to `POST /fhir/{destination}/Patient/$match` +- **WHEN** the response searchset is assembled +- **THEN** the requested resource type SHALL be `Patient`, and `Patient` resources SHALL be + classified as described in "Resources of the requested type are labelled `match`" + +### Requirement: Resources of the requested type are labelled `match` + +The service SHALL set `search.mode = "match"` on, and SHALL retain, every entry whose resource type +equals the resource type requested by the query. Resources of any other type SHALL NOT be labelled +`match`. + +#### Scenario: requested type is labelled match +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` +- **WHEN** the converted bundle contains one `ImmunizationRecommendation` +- **THEN** that entry SHALL be retained with `search.mode = "match"` + +#### Scenario: a resource of another type is not labelled match +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` +- **WHEN** the converted bundle also contains `Immunization` resources produced from the same + HL7 v2 response +- **THEN** those `Immunization` entries SHALL NOT be labelled `match` +- **AND** they SHALL be retained only if another requirement in this capability requires it + +### Requirement: `OperationOutcome` entries are labelled `outcome` + +The service SHALL retain every `OperationOutcome` in the converted bundle and SHALL set +`search.mode = "outcome"` on it, so conversion warnings and errors always reach the caller. + +#### Scenario: conversion warnings survive filtering +- **GIVEN** a converted bundle containing one or more `OperationOutcome` resources +- **WHEN** the searchset is assembled +- **THEN** each `OperationOutcome` entry SHALL be retained with `search.mode = "outcome"` + +### Requirement: A searchset contains only what the caller asked for + +The service SHALL return, in the response to a FHIR query, only entries the caller asked for: +resources of the requested type, `OperationOutcome` resources, and resources retained because they +satisfied an `_include` or `_revinclude` parameter the caller supplied. It SHALL NOT retain a +resource on any other ground. In particular, the service SHALL NOT retain a resource merely because +a retained entry references it, and SHALL NOT retain a resource merely because the HL7 v2 response +happened to carry it. + +The shape of a response SHALL therefore be predictable from the query alone: the same query against +the same HL7 v2 response SHALL yield the same set of resource types regardless of the destination it +was routed to, because searchset assembly reads only the converted bundle and the query parameters +and consults no organization, pipeline or solution. + +This requirement is **BREAKING** for a caller relying on unrequested resources arriving. It governs +the FHIR REST inbound path only. The SOAP/HL7 v2 inbound message contract, the outbound query sent +to the downstream Hub or IIS, and the transformation pipeline (organizations, pipelines, solutions, +operations, and preconditions) SHALL be unchanged, and existing organization transformation +configurations SHALL continue to work untouched. + +#### Scenario: a plain immunization query returns immunizations only +- **GIVEN** a request to `GET /fhir/{destination}/Immunization` with no `_include` or `_revinclude` + parameter +- **WHEN** the converted bundle contains `Immunization` resources and the `Patient` they reference +- **THEN** the returned searchset SHALL contain the `Immunization` entries with + `search.mode = "match"` +- **AND** it SHALL NOT contain the `Patient` + +#### Scenario: a plain recommendation query returns the forecast only +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` with no `_include` or + `_revinclude` parameter +- **WHEN** the converted bundle contains an `ImmunizationRecommendation`, the `Patient`, the + schedule `Organization` behind `authority`, `Immunization` resources from the evaluated history, + and forecast `Observation` resources +- **THEN** the returned searchset SHALL contain only the `ImmunizationRecommendation` entries, with + `search.mode = "match"` + +#### Scenario: the caller asks for the subject patient +- **GIVEN** the same immunization query +- **WHEN** the caller supplies `_include=Immunization:patient` +- **THEN** the `Patient` SHALL be retained with `search.mode = "include"` + +#### Scenario: the caller asks for everything the returned resources reference +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` +- **WHEN** the caller supplies `_include=*:*` +- **THEN** every resource reachable by a reference from a retained entry SHALL be retained with + `search.mode = "include"` + +#### Scenario: the caller asks for the evaluated history and its evaluation data +- **GIVEN** a Z42 response containing administered doses and forecasts for one patient +- **WHEN** the caller issues `GET /fhir/{destination}/ImmunizationRecommendation` with + `_include=ImmunizationRecommendation:patient`, `_revinclude=Immunization:patient`, and + `_include=Immunization:authority` +- **THEN** the `Patient` and one `Immunization` per administered dose SHALL be retained with + `search.mode = "include"` +- **AND** the `Organization` behind `protocolApplied.authority` SHALL be retained, because the + authority is registered on the `Immunization` and not on the `ImmunizationRecommendation` +- **AND** the `ImmunizationRecommendation` SHALL be the only entry labelled `match` + +#### Scenario: the routing destination does not change the returned types +- **GIVEN** the same FHIR query issued against two different destinations +- **WHEN** both produce the same HL7 v2 response +- **THEN** the two returned searchsets SHALL contain the same resource types with the same + `search.mode` on each + +### Requirement: `_include` and `_revinclude` results are labelled `include` + +The service SHALL set `search.mode = "include"` on every resource retained because it satisfied an +`_include` or `_revinclude` parameter. It SHALL NOT label such a resource `match`. In FHIR R4, +`include` is the defined value for an entry "added to the results because of a join", and clients +filter on `mode = "match"` to obtain the hits; labelling joined resources `match` makes the hits +indistinguishable from their supporting resources. + +An `_include` or `_revinclude` parameter SHALL be resolved against the search-parameter names the +HL7 v2 to FHIR conversion registered for each reference. A parameter naming a resource type or +search name that is not registered SHALL match nothing and SHALL NOT be an error. Wildcard (`*`) +resource types and search names SHALL match any value. + +This is a **BREAKING** change to the labelling a client observes: a client that previously read +`_include` and `_revinclude` results as `match` SHALL now read them as `include`. +Reverse resolution depends on the referencing resource reaching a retained entry, per "A reverse +include resolves only from a retained resource". On the immunization path the `Observation` reference +the retained `Immunization` directly, so `_revinclude=Observation` resolves with no forward +`_include`. On the recommendation path they reference the `Patient` rather than the +`ImmunizationRecommendation`, so the same parameter alone retains nothing. + + +#### Scenario: a reverse-included resource is labelled include +- **GIVEN** a request to `GET /fhir/{destination}/Immunization` with `_revinclude=Observation` +- **WHEN** the converted bundle contains `Immunization` resources and dose-level `Observation` + resources that reference them through `partOf` +- **THEN** the `Immunization` entries SHALL have `search.mode = "match"` +- **AND** every retained `Observation` entry SHALL have `search.mode = "include"` +- **AND** the caller SHALL be able to obtain exactly the requested resources by selecting entries + with `search.mode = "match"` + +#### Scenario: a forward-included resource is labelled include +- **GIVEN** a request to a FHIR query endpoint with an `_include` parameter that matches a reference + on a resource of the requested type +- **WHEN** the searchset is assembled +- **THEN** the referenced resource SHALL be retained with `search.mode = "include"` + +#### Scenario: an unmatched include parameter is not an error +- **GIVEN** a request with an `_include` or `_revinclude` parameter naming a search name that the + conversion does not register for any reference +- **WHEN** the searchset is assembled +- **THEN** the request SHALL succeed +- **AND** no additional entry SHALL be retained on account of that parameter + +### Requirement: A reverse include resolves only from a retained resource + +The service SHALL resolve a `_revinclude` parameter against the reverse references of resources +already retained in the searchset. A resource that references a retained entry SHALL therefore be +reached by `_revinclude` only when the entry it references is itself retained, and a caller SHALL be +able to retain that intermediate entry with `_include`. + +This follows from how the HL7 v2 to FHIR conversion bookkeeps references: a reverse reference is +recorded on the resource being pointed at, so it is discoverable only by traversing that resource. +The consequence is observable and callers depend on it — the evaluated-history `Immunization` and the +forecast `Observation` both reference the `Patient` rather than the `ImmunizationRecommendation`, so +neither is reachable on a recommendation query until the `Patient` is retained. + +Any retained resource that the sought resource references SHALL serve as the anchor; the anchor need +not be the one whose search name the parameter names. The conversion keeps a single canonical +`Reference` per resource and accumulates every reverse search name onto it, so a qualified +`_revinclude` matches when the named search name is among the accumulated names, regardless of which +retained resource the reverse reference was found on. A search name the conversion never registered +SHALL match nothing. + +A `_revinclude` naming a resource type SHALL match on the resource type carried by the reverse +reference. A `_revinclude=*` SHALL match any type. Where the conversion produces a resource whose id +carries no resource type, a type-qualified `_revinclude` cannot identify it and matches nothing while +the wildcard form still reaches it. `Provenance` is the only resource type the HL7 v2 to FHIR +conversion currently gives such an id, so `_revinclude=Provenance` matches nothing while +`_revinclude=*` and `_include=Resource:source:Provenance` both reach it. That is a defect in the +conversion rather than intended behavior, and this capability does not require it to stay that way. + +#### Scenario: a reverse include finds nothing when the referenced entry is absent +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` with + `_revinclude=Observation` and no `_include` +- **WHEN** the forecast `Observation` resources reference the `Patient`, which is not retained +- **THEN** no `Observation` entry SHALL be retained +- **AND** the request SHALL succeed + +#### Scenario: retaining the intermediate entry makes the reverse include resolve +- **GIVEN** the same request with `_include=ImmunizationRecommendation:patient` added +- **WHEN** the searchset is assembled +- **THEN** the `Patient` SHALL be retained with `search.mode = "include"` +- **AND** every `Observation` referencing it SHALL be retained with `search.mode = "include"` + +#### Scenario: another retained resource serves as the anchor +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` with + `_include=ImmunizationRecommendation:authority` and `_revinclude=Immunization:patient`, and no + `_include` naming `patient` +- **WHEN** the evaluated-history `Immunization` reference both the `Patient`, which is not retained, + and the schedule `Organization`, which is +- **THEN** those `Immunization` entries SHALL be retained with `search.mode = "include"` +- **AND** the `Patient` SHALL NOT be retained + +#### Scenario: an unregistered reverse search name matches nothing +- **GIVEN** the same request with `_revinclude=Immunization:nosuchsearchname` +- **WHEN** the searchset is assembled +- **THEN** no `Immunization` entry SHALL be retained +- **AND** the request SHALL succeed + +#### Scenario: the evaluated history is reached through the subject patient +- **GIVEN** a Z42 response and a request to `GET /fhir/{destination}/ImmunizationRecommendation` +- **WHEN** the caller supplies `_include=ImmunizationRecommendation:patient` and + `_revinclude=Immunization:patient` +- **THEN** one `Immunization` per administered dose SHALL be retained with `search.mode = "include"` + +### Requirement: A reference to a resource outside the searchset is left intact + +The service SHALL leave every `Reference` on a retained entry as the HL7 v2 to FHIR conversion +produced it. Where the referenced resource is not present in the returned searchset, the service +SHALL NOT remove, rewrite, or reduce the `reference` element, and SHALL NOT alter the reference's +`identifier` or `display`. Resolving such a reference is the caller's decision: the literal value, +the identifier, and the display text are all delivered, and the caller may resolve the reference +against its own source of truth, request the target with `_include`, or ignore it. + +This restores the reference handling callers saw before this capability existed, and it applies to +mandatory 1..1 references — `Immunization.patient`, `ImmunizationRecommendation.patient` — as much +as to optional ones. + +The service makes no guarantee that every delivered reference is readable. Where the HL7 v2 to FHIR +conversion produces a `Reference` carrying no `reference`, no `identifier`, and no `display`, the +service SHALL deliver it as produced. Populating such a reference is the conversion's concern, not +the searchset's. + +#### Scenario: a mandatory reference keeps its literal value +- **GIVEN** a request to `GET /fhir/{destination}/Immunization` with no `_include` parameter +- **WHEN** the returned `Immunization` entries carry a `patient` reference to a `Patient` that the + searchset does not contain +- **THEN** each `patient` reference SHALL retain the `reference` value the conversion produced +- **AND** its `identifier` and `display` SHALL be unchanged + +#### Scenario: an optional reference to an omitted resource is left alone +- **GIVEN** a returned recommendation searchset +- **WHEN** the retained `ImmunizationRecommendation` carries an `authority` reference to an + `Organization` the searchset does not contain +- **THEN** that reference SHALL be delivered unchanged + +#### Scenario: a reference the conversion left empty is delivered as produced +- **GIVEN** a converted bundle in which a retained entry holds a `Reference` with no `reference`, no + `identifier`, and no `display` +- **WHEN** the searchset is assembled +- **THEN** that reference SHALL be delivered unchanged +- **AND** the searchset SHALL NOT be rejected on that account + +#### Scenario: asking for the target changes nothing about the reference +- **GIVEN** the same immunization query +- **WHEN** the caller supplies `_include=Immunization:patient` so the `Patient` is retained +- **THEN** the `patient` reference SHALL carry the same value it carries when the `Patient` is + omitted +- **AND** it SHALL resolve to the retained `Patient` + +### Requirement: Conversion-created resources are retained only when white-listed + +Resources that the HL7 v2 to FHIR conversion synthesises as a side effect of datatype and message +parsing — rather than from a dedicated segment the caller queried for — SHALL be removed from the +searchset unless the caller white-lists them, because the enriched reference they are the target of +already carries an identifier and display text sufficient for production use. + +The caller SHALL be able to white-list them with the `_include=Resource:source:` parameter, +where `` is a resource type or `*` for all such resources; a white-listed resource SHALL be +retained with `search.mode = "include"`. Such a resource SHALL also be retained when a forward +`_include` reaches it as the target of a reference on a retained entry, on the same terms as any +other resource — so `_include=Immunization:location` retains the conversion-created `Location` +without the `Resource:source` form. + +A white-listed resource SHALL participate in `_include` and `_revinclude` traversal on the same terms +as any other retained resource. Being retained by the white-list rather than by a caller's join does +not exempt it from being traversed, so a `_revinclude` MAY reach a further resource through it. + +#### Scenario: conversion-created resources are removed by default +- **GIVEN** a query whose converted bundle contains conversion-created `Practitioner` and `Location` + resources that no retained entry references +- **WHEN** the caller supplies no `_include=Resource:source:...` parameter +- **THEN** those entries SHALL NOT appear in the returned searchset + +#### Scenario: a caller white-lists conversion-created resources by type +- **GIVEN** the same query +- **WHEN** the caller supplies `_include=Resource:source:Practitioner` +- **THEN** the conversion-created `Practitioner` entries SHALL be retained with + `search.mode = "include"` + +#### Scenario: a caller white-lists all conversion-created resources +- **GIVEN** the same query +- **WHEN** the caller supplies `_include=Resource:source:*` +- **THEN** every conversion-created resource SHALL be retained with `search.mode = "include"` + +#### Scenario: a forward include reaches a conversion-created resource +- **GIVEN** a query whose converted bundle contains a conversion-created `Location` that a retained + `Immunization` references +- **WHEN** the caller supplies `_include=Immunization:location` and no `Resource:source` parameter +- **THEN** that `Location` SHALL be retained with `search.mode = "include"` + +#### Scenario: a white-listed resource is traversed like any other +- **GIVEN** a query whose converted bundle contains conversion-created `Provenance` resources, which + reference a conversion-created `DocumentReference` rather than a resource of the requested type +- **WHEN** the caller supplies `_include=Resource:source:DocumentReference` and `_revinclude=*:*` +- **THEN** the `DocumentReference` SHALL be retained by the white-list +- **AND** the `Provenance` resources referencing it SHALL be retained with `search.mode = "include"`, + reached by traversing the white-listed `DocumentReference` + +#### Scenario: a reverse include reaching nothing is not an error +- **GIVEN** the same query +- **WHEN** a `_revinclude` names a resource type that no reverse reference on any retained resource + resolves to +- **THEN** no additional entry SHALL be retained +- **AND** the request SHALL succeed + +### Requirement: Unclassified entries are removed + +The service SHALL remove from the returned searchset every entry that no requirement in this +capability retains, whether or not a retained entry references it. A caller SHALL therefore never +receive an entry whose `search.mode` is absent. + +#### Scenario: an unreferenced, unrequested resource is removed +- **GIVEN** a converted bundle containing a resource that is not of the requested type, is not an + `OperationOutcome`, satisfies no `_include` or `_revinclude` parameter, and is referenced by no + retained entry +- **WHEN** the searchset is assembled +- **THEN** that entry SHALL NOT appear in the returned searchset + +#### Scenario: being referenced does not save an entry from removal +- **GIVEN** a converted bundle containing a resource that satisfies no `_include` or `_revinclude` + parameter but is referenced by a retained entry +- **WHEN** the searchset is assembled +- **THEN** that entry SHALL NOT appear in the returned searchset +- **AND** the reference to it SHALL be delivered unchanged, per "A reference to a resource outside + the searchset is left intact" + +#### Scenario: forecast observations are removed from a plain recommendation query +- **GIVEN** a request to `GET /fhir/{destination}/ImmunizationRecommendation` with no `_include` or + `_revinclude` parameter +- **WHEN** the converted bundle contains `Observation` resources carrying the forecast detail +- **THEN** those `Observation` entries SHALL NOT appear in the returned searchset diff --git a/pom.xml b/pom.xml index 3326ef5e5..41df31a7c 100644 --- a/pom.xml +++ b/pom.xml @@ -108,7 +108,7 @@ gov.cdc.izgw v2tofhir - 2.4.0 + 2.5.2-SNAPSHOT org.xerial diff --git a/src/main/java/gov/cdc/izgateway/xform/endpoints/fhir/FhirController.java b/src/main/java/gov/cdc/izgateway/xform/endpoints/fhir/FhirController.java index 7907db7d3..5d4da9dcc 100644 --- a/src/main/java/gov/cdc/izgateway/xform/endpoints/fhir/FhirController.java +++ b/src/main/java/gov/cdc/izgateway/xform/endpoints/fhir/FhirController.java @@ -152,6 +152,20 @@ public class FhirController { private static final String IMMUNIZATION_RECOMMENDATION = "ImmunizationRecommendation"; /** Name of the Patient {@code $match} operation advertised in the CapabilityStatement. */ private static final String MATCH_OPERATION = "match"; + + /** User-data key under which v2tofhir stores a Reference's target resource. */ + private static final String RESOURCE_KEY = "Resource"; + /** + * The resource-type token in the {@code _include=Resource:source:} white-list parameter, + * used as a pseudo-type meaning "any resource the conversion created". + * + *

This spells the same as {@link #RESOURCE_KEY} by coincidence and must not be merged with + * it. This one is part of the URL syntax callers type, documented in + * {@code docs/fhir/rsp-to-fhir.md}; {@code RESOURCE_KEY} is an internal v2tofhir user-data key + * that v2tofhir is free to rename. Sharing one constant would let a rename of either silently + * change the other.

+ */ + private static final String SOURCE_INCLUDE_TYPE = "Resource"; /** Canonical OperationDefinition URL for the Patient {@code $match} operation. */ private static final String PATIENT_MATCH_DEFINITION = "http://hl7.org/fhir/OperationDefinition/Patient-match"; @@ -1051,7 +1065,7 @@ private Bundle filter(Bundle bundle, HttpServletRequest req) { } List resources = new ArrayList<>(); - preFilter(bundle, includes, revIncludes, requested, resources); + preFilter(bundle, includes, requested, resources); markIncludedResources(includes, revIncludes, resources); @@ -1059,71 +1073,74 @@ private Bundle filter(Bundle bundle, HttpServletRequest req) { return bundle; } - private void preFilter(Bundle bundle, List includes, List revIncludes, String requested, - List resources) { - Iterator it = bundle.getEntry().iterator(); - while (it.hasNext()) { - BundleEntryComponent entry = it.next(); + /** + * Classify each entry: the requested type is a {@code match}, an {@link OperationOutcome} is an + * {@code outcome}, and anything else is left for later marking or removal. + * + *

Nothing is retained here that the caller did not ask for. A resource of another type + * survives only if an {@code _include} or {@code _revinclude} parameter reaches it, so the shape + * of a response is predictable from the query alone.

+ * + *

That includes the evaluated history on a recommendation query. A Z42 response ("Return + * Evaluated History and Forecast") carries the patient's evaluated history alongside the + * forecast, and v2tofhir splits it by RXA-5: {@code 998^No Vaccine Administered} becomes a + * {@code recommendation} component, any real CVX code becomes an Immunization. Those + * Immunizations carry evaluation data reachable through no other call — + * {@code protocolApplied.doseNumber} / {@code seriesDoses} (OBX 30973-2 / 59782-3), + * {@code protocolApplied.authority} (OBX 59779-9) and {@code programEligibility} (OBX 64994-7), + * none of which the Z32 behind {@code /Immunization} returns — but wanting them is the caller's + * call to make. They are reached with + * {@code _include=ImmunizationRecommendation:patient&_revinclude=Immunization:patient}: the + * Immunizations reference the Patient rather than the recommendation, so the reverse hit is + * found on a retained Patient.

+ */ + private void preFilter(Bundle bundle, List includes, String requested, List resources) { + for (BundleEntryComponent entry : bundle.getEntry()) { Resource r = entry.getResource(); if (r != null && r.fhirType().equals(requested)) { - entry.getSearch().setMode(SearchEntryMode.MATCH); - if (!resources.contains(r)) { - resources.add(r); - } + markEntry(entry, resources, SearchEntryMode.MATCH); } else if (r instanceof OperationOutcome) { - entry.getSearch().setMode(SearchEntryMode.OUTCOME); - if (!resources.contains(r)) { - resources.add(r); - } + markEntry(entry, resources, SearchEntryMode.OUTCOME); } else { - removeInfrastructureCreatedResources(resources, includes, revIncludes, it, r); + whitelistInfrastructureCreatedResources(resources, includes, r); } } } - - private void removeInfrastructureCreatedResources(List resources, List includes, List revIncludes, - Iterator it, Resource r) { - if (r != null && r.getUserData(Parser.SOURCE) != null) { - // Some DatatypeConverter and MessageParser created resources have limited utility. - // What we should we do with those depends on what resources the - // user asks to include. These infrastructure crafted resources can be white-listed - // using the include or revinclude parameters. - - // Users can white-list these resources with the following _include parameters: - // All: - // _include=Resource:source:* - // DatatypeConverter created Organization/Practitioner/RelatedPerson/Location - // _include=Resource:source:Organization - // MessageParser created DocumentReference/Provenance - // _include=Resource:source:DocumentReference - String source = r.getUserData(Parser.SOURCE).toString(); - if ( - // ANY Source requested - // DatatypeConverter created resources including Organization, Practitioner, RelatedPerson, and Location - // MessageParser created resources including DocumentReference and Provenance - matchesSource(includes, r.fhirType()) - ) { - // Explicitly mark these as included resources. - r.setUserData(SearchEntryMode.class.getName(), SearchEntryMode.INCLUDE); - resources.add(r); - return; - } - - // If users reverse include provenance, don't delete the MessageParser crafted Provenance resources. - if (source.equals(MessageParser.class.getName()) - && revIncludes.stream().anyMatch(rinc -> "Provenance".equals(rinc.getParamType()))) { - // Let normal include handling mark Provenance reverse includes. - return; - } - // Remove classes crafted by the infrastructure as being generally not useful because - // the enriched reference (name and identifier) is enough for production use. - it.remove(); + + private void markEntry(BundleEntryComponent entry, List resources, SearchEntryMode mode) { + entry.getSearch().setMode(mode); + if (!resources.contains(entry.getResource())) { + resources.add(entry.getResource()); + } + } + + /** + * Mark the infrastructure-created resources the caller explicitly white-listed. + * + *

Some DatatypeConverter and MessageParser created resources have limited utility, because + * the enriched reference (name and identifier) is generally enough for production use. They are + * therefore not retained by default. A caller can white-list them with:

+ * 
+     * _include=Resource:source:*                  all of them
+     * _include=Resource:source:Organization       DatatypeConverter created Organization/Practitioner/RelatedPerson/Location
+     * _include=Resource:source:DocumentReference  MessageParser created DocumentReference/Provenance
+     *
+ * + *

Nothing is removed here. Anything left unmarked once include marking has finished is + * removed by {@link #cleanupBundleOfUnmarkedResources}, which is the single removal point — + * that is what lets {@link #markIncludedResources} rescue a resource that a retained entry + * still references, whether or not the caller white-listed it.

+ */ + private void whitelistInfrastructureCreatedResources(List resources, List includes, Resource r) { + if (r != null && r.getUserData(Parser.SOURCE) != null && matchesSource(includes, r.fhirType())) { + r.setUserData(SearchEntryMode.class.getName(), SearchEntryMode.INCLUDE); + resources.add(r); } } private boolean matchesSource(List includes, String target) { for (Include include: includes) { - if ("Resource".equals(include.getParamType()) + if (SOURCE_INCLUDE_TYPE.equals(include.getParamType()) && Parser.SOURCE.equals(include.getParamName()) && Arrays.asList("*", target, null).contains(include.getParamTargetType()) ) { @@ -1175,11 +1192,14 @@ private void checkReferences(List includes, List resources, R // For each forward include, e.g., _include=Immunization:patient:Patient for (Include include: includes) { if (includeMatches(include, r, ref, reverse)) { - Resource target = (Resource) ref.getUserData("Resource"); + Resource target = (Resource) ref.getUserData(RESOURCE_KEY); if (!resources.contains(target)) { resources.add(target); } - target.setUserData(SearchEntryMode.class.getName(), SearchEntryMode.MATCH); + // An _include/_revinclude hit is a join, which R4 labels "include". + // Labelling it "match" makes the hits indistinguishable from the + // resources joined in to support them. + target.setUserData(SearchEntryMode.class.getName(), SearchEntryMode.INCLUDE); } } } diff --git a/src/test/java/gov/cdc/izgateway/xform/endpoints/fhir/FhirControllerTests.java b/src/test/java/gov/cdc/izgateway/xform/endpoints/fhir/FhirControllerTests.java index bd5f067a0..4bed19096 100644 --- a/src/test/java/gov/cdc/izgateway/xform/endpoints/fhir/FhirControllerTests.java +++ b/src/test/java/gov/cdc/izgateway/xform/endpoints/fhir/FhirControllerTests.java @@ -9,24 +9,39 @@ import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; import static org.mockito.Mockito.doReturn; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; +import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; +import java.util.HashSet; +import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import java.util.Set; +import java.util.stream.Stream; +import org.apache.commons.lang3.StringUtils; +import org.hl7.fhir.r4.model.Base; import org.hl7.fhir.r4.model.Bundle; +import org.hl7.fhir.r4.model.Bundle.BundleEntryComponent; import org.hl7.fhir.r4.model.Bundle.BundleType; +import org.hl7.fhir.r4.model.Bundle.SearchEntryMode; +import org.hl7.fhir.r4.model.Property; +import org.hl7.fhir.r4.model.Reference; import org.hl7.fhir.r4.model.CapabilityStatement; import org.hl7.fhir.r4.model.CodeableConcept; import org.hl7.fhir.r4.model.Coding; import org.hl7.fhir.r4.model.DateType; import org.hl7.fhir.r4.model.Enumerations; import org.hl7.fhir.r4.model.HumanName; +import org.hl7.fhir.r4.model.Immunization; +import org.hl7.fhir.r4.model.Immunization.ImmunizationProtocolAppliedComponent; +import org.hl7.fhir.r4.model.ImmunizationRecommendation; +import org.hl7.fhir.r4.model.Observation; import org.hl7.fhir.r4.model.OperationOutcome; import org.hl7.fhir.r4.model.OperationOutcome.IssueSeverity; import org.hl7.fhir.r4.model.OperationOutcome.IssueType; @@ -219,11 +234,759 @@ void isPatientReferenceClassifiesReferenceTypes() { "PID|1||0000001^^^TEST^MR||CuyahogaAIRA^MarnyAIRA^^^^^L||19600507|F" ); + /** + * A Z32 response (evaluated history) for an immunization history query: one patient, + * two administered doses, each with an administering performer and facility so the + * conversion produces referenced Practitioner/Organization resources. + */ + private static final String RSP_Z32_MESSAGE = String.join("\r", + "MSH|^~\\&|TESTIIS|TESTIIS|TESTAPP|TESTORG|20240101120000||RSP^K11^RSP_K11|X235|P|2.5.1|||||||||Z32^CDCPHINVS", + "MSA|AA|1234", + "QAK|Q1|OK|Z34^Request Immunization History^CDCPHINVS", + "QPD|Z34^Request Immunization History^CDCPHINVS|Q1|0000001^^^TEST^MR", + "PID|1||0000001^^^TEST^MR||CuyahogaAIRA^MarnyAIRA^^^^^L||19600507|F", + "ORC|RE||IZ-1^NDA|||||||^Nurse^Nancy^^^^^^NDA^L||^Clinician^Carl^^^^^^NDA^L", + "RXA|0|1|20200101||208^COVID-19 mRNA vaccine^CVX|0.3|mL^milliliters^UCUM|||" + + "^Nurse^Nancy^^^^^^NDA^L|^^^TESTFAC^^^^^Test Facility", + "OBX|1|CE|64994-7^Vaccine funding program eligibility category^LN|1|" + + "V02^VFC eligible Medicaid/Medicaid managed care^HL70064||||||F", + "ORC|RE||IZ-2^NDA|||||||^Nurse^Nancy^^^^^^NDA^L||^Clinician^Carl^^^^^^NDA^L", + "RXA|0|1|20200201||208^COVID-19 mRNA vaccine^CVX|0.3|mL^milliliters^UCUM|||" + + "^Nurse^Nancy^^^^^^NDA^L|^^^TESTFAC^^^^^Test Facility", + "OBX|1|CE|64994-7^Vaccine funding program eligibility category^LN|1|" + + "V02^VFC eligible Medicaid/Medicaid managed care^HL70064||||||F" + ); + + /** + * A Z42 response (evaluated history + forecast): one patient, two administered doses carrying + * the evaluation OBX segments that only a Z42 returns (30973-2 dose number, 59782-3 doses in + * series, 59779-9 schedule used, 64994-7 funding eligibility), and one forecast group + * (RXA-5 == 998) carrying the forecast OBX segments. + */ + private static final String RSP_Z42_MESSAGE = String.join("\r", + "MSH|^~\\&|TESTIIS|TESTIIS|TESTAPP|TESTORG|20240101120000||RSP^K11^RSP_K11|X236|P|2.5.1|||||||||Z42^CDCPHINVS", + "MSA|AA|1234", + "QAK|Q1|OK|Z44^Request Evaluated History and Forecast^CDCPHINVS", + "QPD|Z44^Request Evaluated History and Forecast^CDCPHINVS|Q1|0000001^^^TEST^MR", + "PID|1||0000001^^^TEST^MR||CuyahogaAIRA^MarnyAIRA^^^^^L||19600507|F", + "ORC|RE||IZ-1^NDA", + "RXA|0|1|20200101||208^COVID-19 mRNA vaccine^CVX|0.3|mL^milliliters^UCUM", + "OBX|1|CE|30956-7^Vaccine Type^LN|1|208^COVID-19 mRNA vaccine^CVX||||||F", + "OBX|2|NM|30973-2^Dose Number in Series^LN|1|1|NA^Not Applicable^HL70353|||||F", + "OBX|3|NM|59782-3^Number of doses in primary series^LN|1|2|||||F", + "OBX|4|CE|59779-9^Immunization Schedule Used^LN|1|VXC16^ACIP^CDCPHINVS||||||F", + "OBX|5|CE|64994-7^Vaccine funding program eligibility category^LN|2|" + + "V02^VFC eligible Medicaid/Medicaid managed care^HL70064||||||F|||||VXC40^Vaccine Level^CDCPHINVS", + "ORC|RE||IZ-2^NDA", + "RXA|0|1|20200201||208^COVID-19 mRNA vaccine^CVX|0.3|mL^milliliters^UCUM", + "OBX|1|CE|30956-7^Vaccine Type^LN|1|208^COVID-19 mRNA vaccine^CVX||||||F", + "OBX|2|NM|30973-2^Dose Number in Series^LN|1|2|NA^Not Applicable^HL70353|||||F", + "OBX|3|NM|59782-3^Number of doses in primary series^LN|1|2|||||F", + "OBX|4|CE|59779-9^Immunization Schedule Used^LN|1|VXC16^ACIP^CDCPHINVS||||||F", + "OBX|5|CE|64994-7^Vaccine funding program eligibility category^LN|2|" + + "V02^VFC eligible Medicaid/Medicaid managed care^HL70064||||||F|||||VXC40^Vaccine Level^CDCPHINVS", + "ORC|RE||9999^NDA", + "RXA|0|1|20240101||998^No vaccine administered^CVX|999", + "OBX|1|CE|30956-7^Vaccine type^LN|1|208^COVID-19 mRNA vaccine^CVX||||||F", + "OBX|2|CE|59783-1^Status in immunization series^LN|1|LA13425-1^Complete^LN||||||F", + "OBX|3|TS|30981-5^Earliest date to give^LN|1|20240301||||||F", + "OBX|4|CE|30982-3^Reason applied by forecast logic^LN|1|" + + "LA12836-0^Reason applied by forecast logic^LN||||||F", + "OBX|5|CE|59779-9^Immunization Schedule Used^LN|1|VXC16^ACIP^CDCPHINVS||||||F" + ); + + /** + * The Z42 fixture with an administering performer (RXA-10) and facility (RXA-11) on each + * administered dose, so the conversion produces the Location and performer resources the plain + * {@link #RSP_Z42_MESSAGE} does not. Kept separate so no existing entry-count assertion moves. + */ + private static final String RSP_Z42_WITH_FACILITY_MESSAGE = RSP_Z42_MESSAGE.replace( + "RXA|0|1|20200101||208^COVID-19 mRNA vaccine^CVX|0.3|mL^milliliters^UCUM", + "RXA|0|1|20200101||208^COVID-19 mRNA vaccine^CVX|0.3|mL^milliliters^UCUM|||" + + "^Nurse^Nancy^^^^^^NDA^L|^^^TESTFAC^^^^^Test Facility") + .replace( + "RXA|0|1|20200201||208^COVID-19 mRNA vaccine^CVX|0.3|mL^milliliters^UCUM", + "RXA|0|1|20200201||208^COVID-19 mRNA vaccine^CVX|0.3|mL^milliliters^UCUM|||" + + "^Nurse^Nancy^^^^^^NDA^L|^^^TESTFAC^^^^^Test Facility"); + @AfterEach void clearRequestContext() { RequestContext.clear(); } + // --- searchset filtering ------------------------------------------------------------- + // + // See openspec/changes/fix-fhir-searchset-include-mode. The filter classifies every entry + // as match / include / outcome and drops the rest; these tests pin that contract, which + // had no coverage at all before this change. + + /** Every Reference held anywhere in a resource, found by walking its element tree. */ + private static List referencesOf(Base base) { + List found = new ArrayList<>(); + collectReferences(base, found, new HashSet<>()); + return found; + } + + private static void collectReferences(Base base, List found, Set seen) { + if (base == null || !seen.add(base)) { + return; + } + if (base instanceof Reference ref) { + found.add(ref); + return; // a Reference's own children (identifier, display) hold no further references + } + for (Property property : base.children()) { + for (Base child : property.getValues()) { + collectReferences(child, found, seen); + } + } + } + + private static final String RECOMMENDATION_URI = "/fhir/dev/ImmunizationRecommendation"; + + private static final String IMMUNIZATION_URI = "/fhir/dev/Immunization"; + + /** + * How a caller reaches the Z42 evaluated history. The Immunizations reference the Patient, not + * the recommendation, so the reverse hit is only found once the Patient is retained - which is + * why the forward _include is not optional. Immunization:authority is what retains the schedule + * Organization: protocolApplied.authority is registered on the Immunization, not on the + * ImmunizationRecommendation. + */ + private static final Map HISTORY_PARAMS = queryParams( + "_include", "ImmunizationRecommendation:patient", + "_revinclude", "Immunization:patient", + "_include", "Immunization:authority"); + + /** Everything the pre-change code returned unasked, requested explicitly. */ + private static final Map RECOVERY_PARAMS = queryParams( + "_include", "*:*", + "_revinclude", "Immunization"); + + /** The type-and-mode multiset of a searchset, as {@code Type/MODE -> count}. */ + private static Map typeModeCounts(Bundle bundle) { + Map counts = new java.util.TreeMap<>(); + for (BundleEntryComponent e : bundle.getEntry()) { + counts.merge(e.getResource().fhirType() + "/" + e.getSearch().getMode(), 1, Integer::sum); + } + return counts; + } + + private static Bundle query(String hl7, String uri, Map params) throws Exception { + initRequestContext(); + Bundle b = controller(hubReturning(hl7)).iisQuery("dev", fhirRequest(uri, null, params)).getBody(); + assertNotNull(b); + return b; + } + + // Observations only ever arrive because the caller asked for them. The default recommendation + // query returns none (see plainRecommendationQueryReturnsNoObservations), because + // the filter never walks the Reverses direction on its own. When the caller does ask with + // _revinclude=Observation, the forecast Observations arrive too - reaching them is the caller's + // choice, and they are never labelled match. + + @Test + void revincludedObservationsAreLabelledIncludeNotMatch() throws Exception { + Bundle b = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, + queryParams("_revinclude", "Observation")); + + List observations = resourcesOfType(b, "Observation"); + assertFalse(observations.isEmpty(), "the _revinclude should have retained the dose Observations"); + assertTrue(entriesWithMode(b, SearchEntryMode.INCLUDE).containsAll(observations), + "every revincluded Observation should be search.mode=include, not match"); + } + + @Test + void selectingMatchYieldsOnlyTheRequestedType() throws Exception { + Bundle b = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, + queryParams("_revinclude", "Observation")); + + List matches = entriesWithMode(b, SearchEntryMode.MATCH); + assertFalse(matches.isEmpty(), "the doses should be matches"); + assertTrue(matches.stream().allMatch(r -> "Immunization".equals(r.fhirType())), + () -> "only the requested type should be a match, got: " + + matches.stream().map(Resource::fhirType).distinct().toList()); + } + + @Test + void observationsArriveOnlyWhenRevincludedAndNeverAsMatch() throws Exception { + // The forecast detail is opt-in: absent by default, retained as include when asked for. + // The forward _include is required - the Observations reference the Patient, so the reverse + // hit is only found once the Patient is in the searchset. + Bundle b = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, queryParams( + "_include", "ImmunizationRecommendation:patient", + "_revinclude", "Observation")); + + List observations = resourcesOfType(b, "Observation"); + assertFalse(observations.isEmpty(), "the _revinclude should have retained Observations"); + assertTrue(entriesWithMode(b, SearchEntryMode.INCLUDE).containsAll(observations), + "a revincluded Observation is a join, so search.mode=include"); + assertTrue(entriesWithMode(b, SearchEntryMode.MATCH).stream() + .allMatch(r -> "ImmunizationRecommendation".equals(r.fhirType())), + "only the forecast the client asked for should be a match"); + } + + @Test + void unmatchedIncludeParameterIsNotAnError() throws Exception { + Bundle baseline = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, queryParams()); + Bundle b = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, + queryParams("_include", "ImmunizationRecommendation:nosuchsearchname")); + + assertEquals(baseline.getEntry().size(), b.getEntry().size(), + "an include naming an unregistered search name should retain nothing extra"); + } + + @Test + void plainRecommendationQueryReturnsNeitherPatientNorOrganization() throws Exception { + // Both are referenced by the returned forecast, and neither is returned for that reason: + // being referenced is not a reason to retain. + Bundle b = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, queryParams()); + + assertTrue(resourcesOfType(b, "Patient").isEmpty(), + "the subject Patient is not what the caller asked for"); + assertTrue(resourcesOfType(b, "Organization").isEmpty(), + "the schedule Organization behind authority is not what the caller asked for"); + assertTrue(b.getEntry().stream() + .allMatch(e -> "ImmunizationRecommendation".equals(e.getResource().fhirType()) + || e.getResource() instanceof OperationOutcome), + () -> "only the requested type and outcomes should survive, got: " + typeModeCounts(b)); + + // The references to both omitted resources are delivered exactly as the conversion produced + // them. This is the case the removed no-dangling-references rule used to paper over. + ImmunizationRecommendation forecast = + (ImmunizationRecommendation) resourcesOfType(b, "ImmunizationRecommendation").get(0); + Reference patient = forecast.getPatient(); + assertTrue(patient.hasReference(), "the mandatory patient reference keeps its literal value"); + assertTrue(patient.getReference().startsWith("Patient/"), + () -> "unexpected patient reference: " + patient.getReference()); + Reference authority = forecast.getAuthority(); + assertTrue(authority.hasReference(), + "the optional authority reference keeps its literal value even with the target omitted"); + assertTrue(authority.getReference().startsWith("Organization/"), + () -> "unexpected authority reference: " + authority.getReference()); + } + + @Test + void patientAndOrganizationArriveWhenIncluded() throws Exception { + Bundle b = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, queryParams( + "_include", "ImmunizationRecommendation:patient", + "_include", "ImmunizationRecommendation:authority")); + + List included = entriesWithMode(b, SearchEntryMode.INCLUDE); + assertFalse(resourcesOfType(b, "Patient").isEmpty(), "the Patient was asked for"); + assertFalse(resourcesOfType(b, "Organization").isEmpty(), "the Organization was asked for"); + assertTrue(included.containsAll(resourcesOfType(b, "Patient")), + "an _include hit is a join, so search.mode=include"); + assertTrue(included.containsAll(resourcesOfType(b, "Organization")), + "an _include hit is a join, so search.mode=include"); + assertTrue(entriesWithMode(b, SearchEntryMode.MATCH).stream() + .allMatch(r -> "ImmunizationRecommendation".equals(r.fhirType())), + "an included resource must not be promoted to a match"); + } + + @Test + void plainRecommendationQueryOmitsTheEvaluatedHistory() throws Exception { + // The evaluated history is not what an ImmunizationRecommendation query asked for. It is + // still reachable - see includedHistoryCarriesTheZ42OnlyEvaluationData for how. + Bundle b = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, queryParams()); + + assertTrue(resourcesOfType(b, "Immunization").isEmpty(), + "the Z42 evaluated history should not arrive unasked"); + } + + @Test + void evaluatedHistoryArrivesWhenRevincludedThroughThePatient() throws Exception { + Bundle b = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, HISTORY_PARAMS); + + List doses = resourcesOfType(b, "Immunization"); + assertEquals(2, doses.size(), "both administered doses should arrive"); + assertTrue(entriesWithMode(b, SearchEntryMode.INCLUDE).containsAll(doses), + "a revincluded dose is a join, so search.mode=include"); + assertTrue(entriesWithMode(b, SearchEntryMode.MATCH).stream() + .allMatch(r -> "ImmunizationRecommendation".equals(r.fhirType())), + "only the forecast the client asked for should be a match"); + } + + @Test + void includedHistoryCarriesTheZ42OnlyEvaluationData() throws Exception { + Bundle b = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, HISTORY_PARAMS); + + Set orgIds = resourcesOfType(b, "Organization").stream() + .map(r -> r.getIdElement().getIdPart()).collect(java.util.stream.Collectors.toSet()); + for (Resource r : resourcesOfType(b, "Immunization")) { + Immunization imm = (Immunization) r; + assertFalse(imm.getProtocolApplied().isEmpty(), "protocolApplied should be populated"); + ImmunizationProtocolAppliedComponent protocol = imm.getProtocolAppliedFirstRep(); + assertTrue(protocol.hasDoseNumberPositiveIntType(), "doseNumber from OBX 30973-2"); + assertTrue(protocol.hasSeriesDosesPositiveIntType(), "seriesDoses from OBX 59782-3"); + assertTrue(orgIds.contains(StringUtils.substringAfterLast( + protocol.getAuthority().getReference(), "/")), + "protocolApplied.authority should resolve to an Organization in the bundle"); + assertFalse(imm.getProgramEligibility().isEmpty(), "programEligibility from OBX 64994-7"); + } + } + + @Test + void includedHistoryHasStableDistinctIdentifiers() throws Exception { + Bundle b = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, HISTORY_PARAMS); + + List fillerOrderNumbers = resourcesOfType(b, "Immunization").stream() + .map(r -> ((Immunization) r).getIdentifierFirstRep().getValue()) + .sorted() + .toList(); + assertEquals(List.of("IZ-1", "IZ-2"), fillerOrderNumbers, + "each dose should carry its own ORC-3 filler order number"); + + Set ids = b.getEntry().stream() + .map(e -> e.getResource().fhirType() + "/" + e.getResource().getIdElement().getIdPart()) + .collect(java.util.stream.Collectors.toSet()); + assertEquals(b.getEntry().size(), ids.size(), "no two entries may collide on Type/id"); + } + + @Test + void immunizationQueryStillReturnsHistoryAsMatch() throws Exception { + // The include is scoped to the recommendation path; /Immunization is unchanged. + Bundle b = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, queryParams()); + + assertTrue(entriesWithMode(b, SearchEntryMode.MATCH) + .containsAll(resourcesOfType(b, "Immunization")), + "on /Immunization the doses are the matches, not includes"); + } + + @Test + void plainRecommendationQueryReturnsNoObservations() throws Exception { + Bundle b = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, queryParams()); + + assertTrue(resourcesOfType(b, "Observation").isEmpty(), + "forecast Observations should stay out of a plain recommendation query"); + } + + @Test + void immunizationPatientReferenceKeepsItsLiteralValueWhenTheTargetIsOmitted() throws Exception { + // Immunization.patient is 1..1 and its target is not returned. The reference is delivered + // exactly as the conversion produced it - resolving it is the caller's decision. + Bundle b = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, queryParams()); + + assertFalse(resourcesOfType(b, "Immunization").isEmpty(), "the doses should be matches"); + assertTrue(resourcesOfType(b, "Patient").isEmpty(), "the Patient was not asked for"); + for (Resource r : resourcesOfType(b, "Immunization")) { + Reference patient = ((Immunization) r).getPatient(); + assertTrue(patient.hasReference(), + "the patient reference must keep the value the conversion produced"); + assertTrue(patient.getReference().startsWith("Patient/"), + () -> "unexpected reference value: " + patient.getReference()); + } + } + + @Test + void conversionCreatedResourcesStillNeedWhitelistingWhenUnreferenced() throws Exception { + // Provenance/DocumentReference are MessageParser artifacts that nothing in the + // searchset references, so they are the ones the white-list still governs. + Bundle without = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, queryParams()); + assertTrue(resourcesOfType(without, "Provenance").isEmpty(), + "unreferenced conversion-created resources should be absent by default"); + + Bundle with = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, + queryParams("_include", "Resource:source:*")); + assertTrue(with.getEntry().size() > without.getEntry().size(), + "Resource:source:* should retain the conversion-created resources"); + assertTrue(entriesWithMode(with, SearchEntryMode.INCLUDE) + .containsAll(resourcesOfType(with, "Provenance")), + "white-listed resources should be search.mode=include"); + } + + @Test + void noReferenceWithReadableContentIsStrippedOfItsValue() throws Exception { + // The pre-change code shipped five references with no reference element on this fixture. + // Two were stripped by clearUnresolvableReferences and are restored here; the other three + // carry no reference, identifier or display as v2tofhir produces them, and stay empty. + Bundle b = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, queryParams("_include", "*:*")); + + List withoutValue = referencesWithoutLiteralValue(b); + assertEquals(3, withoutValue.size(), + () -> "only the references v2tofhir leaves empty should lack a value, got: " + + describe(b)); + assertTrue(withoutValue.stream().noneMatch(ref -> ref.hasIdentifier() || ref.hasDisplay()), + () -> "a reference carrying readable content must keep its value too, got: " + + describe(b)); + } + + /** Every reference in a bundle whose {@code reference} element is absent. */ + private static List referencesWithoutLiteralValue(Bundle bundle) { + return bundle.getEntry().stream() + .flatMap(e -> referencesOf(e.getResource()).stream()) + .filter(ref -> !ref.hasReference()) + .toList(); + } + + private static List describe(Bundle bundle) { + return bundle.getEntry().stream() + .flatMap(e -> referencesOf(e.getResource()).stream() + .filter(ref -> !ref.hasReference()) + .map(ref -> e.getResource().fhirType() + "[identifier=" + + ref.getIdentifier().getValue() + ",display=" + ref.getDisplay() + "]")) + .toList(); + } + + @Test + void outcomesSurviveAndBundleIsASearchset() throws Exception { + Bundle b = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, queryParams()); + + assertEquals(BundleType.SEARCHSET, b.getType()); + assertFalse(entriesWithMode(b, SearchEntryMode.OUTCOME).isEmpty(), + "conversion OperationOutcomes should survive with mode=outcome"); + assertTrue(b.getEntry().stream().allMatch(e -> e.getSearch().getMode() != null), + "every retained entry must carry a search mode"); + } + + @Test + void namedTypeWhitelistRetainsOnlyThatType() throws Exception { + // Naming one type is enough - the caller does not have to ask for Resource:source:*. + // It is not necessarily *narrower* than the wildcard on this fixture: the MessageParser + // Provenance targets the other conversion-created resources, so the no-dangling-references + // rule pulls them in behind it either way. + Bundle baseline = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, queryParams()); + Bundle named = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, + queryParams("_include", "Resource:source:Provenance")); + + assertTrue(resourcesOfType(baseline, "Provenance").isEmpty(), + "without the white-list the Provenance stays out"); + assertFalse(resourcesOfType(named, "Provenance").isEmpty(), + "naming the type should white-list it"); + assertTrue(entriesWithMode(named, SearchEntryMode.INCLUDE) + .containsAll(resourcesOfType(named, "Provenance")), + "a white-listed resource is search.mode=include"); + } + + @Test + void partOfRevincludeNarrowsToTheHistoryObservations() throws Exception { + // Documented in docs/fhir/rsp-to-fhir.md: the unqualified _revinclude=Observation returns + // the forecast Observations too, and Observation:part-of is how a caller excludes them. + Bundle all = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, queryParams( + "_include", "*:*", "_revinclude", "Observation")); + Bundle narrowed = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, queryParams( + "_include", "*:*", "_revinclude", "Observation:part-of")); + + List narrowedObs = resourcesOfType(narrowed, "Observation"); + assertFalse(narrowedObs.isEmpty(), "the dose Observations link via partOf and should arrive"); + assertTrue(resourcesOfType(all, "Observation").size() > narrowedObs.size(), + "the unqualified form should also return the unlinked forecast Observations"); + assertTrue(narrowedObs.stream().noneMatch(r -> ((Observation) r).getPartOf().isEmpty()), + "Observation:part-of should retain only Observations carrying a partOf link"); + } + + @Test + void matchOperationLabelsThePatientAsMatch() throws Exception { + // $match has no resource type in the path; the filter resolves the requested type to + // Patient, so the matched Patient is the match rather than an unlabelled entry. + initRequestContext(); + Bundle b = (Bundle) controller(hubReturning(RSP_MESSAGE)) + .iisPatientMatch("dev", matchParameters(), fhirRequest(MATCH_URI, null)).getBody(); + + assertNotNull(b); + List patients = resourcesOfType(b, "Patient"); + assertFalse(patients.isEmpty(), "the matched Patient should be returned"); + assertTrue(entriesWithMode(b, SearchEntryMode.MATCH).containsAll(patients), + "on $match the Patient is the match"); + assertTrue(b.getEntry().stream().allMatch(e -> e.getSearch().getMode() != null), + "every retained entry must carry a search mode"); + } + + // --- strict searchset contract ------------------------------------------------------- + // + // See openspec/changes/fhir-searchset-strict-includes. A query returns the requested type and + // OperationOutcome only; everything else is opt-in via _include / _revinclude. The baselines + // asserted below were recorded from the pre-change code (tasks 0.1 and 0.2). + + @Test + void plainImmunizationQueryReturnsImmunizationsOnly() throws Exception { + Bundle b = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, queryParams()); + + assertEquals(Map.of("Immunization/MATCH", 2, "OperationOutcome/OUTCOME", 2), + typeModeCounts(b), + "the pre-change code returned 10 entries here; only the doses were asked for"); + for (String absent : List.of("Patient", "PractitionerRole", "Practitioner", "Location")) { + assertTrue(resourcesOfType(b, absent).isEmpty(), + () -> absent + " was not asked for and must not arrive"); + } + } + + @Test + void recoveryParametersReproduceThePreChangePayload() throws Exception { + // _include=*:* & _revinclude=Immunization is what docs/fhir/fhir-api.md names as the way to + // get back what the service used to return unasked. This pins that it actually does: the + // expected multiset is the recorded pre-change baseline for a plain recommendation query. + Bundle b = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, RECOVERY_PARAMS); + + assertEquals(Map.of( + "ImmunizationRecommendation/MATCH", 1, + "OperationOutcome/OUTCOME", 2, + "Patient/INCLUDE", 1, + "Immunization/INCLUDE", 2, + "Organization/INCLUDE", 1), + typeModeCounts(b), + "the recovery parameters must reproduce the pre-change searchset exactly"); + } + + @Test + void recoveryParametersStillReturnNoObservations() throws Exception { + // The pre-change code never walked the reverse direction on its own, so the forecast + // Observations stayed out. Naming only Immunization in the _revinclude preserves that. + Bundle b = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, RECOVERY_PARAMS); + + assertTrue(resourcesOfType(b, "Observation").isEmpty(), + "_revinclude=Immunization must not drag in the forecast Observations"); + } + + @Test + void forwardWildcardAloneDoesNotReachTheEvaluatedHistory() throws Exception { + // The Immunizations are reachable only in reverse, so the _revinclude is not optional. + Bundle b = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, queryParams("_include", "*:*")); + + assertTrue(resourcesOfType(b, "Immunization").isEmpty(), + "_include=*:* follows forward references only"); + assertFalse(resourcesOfType(b, "Patient").isEmpty(), + "the forward wildcard should still reach the Patient"); + } + + @Test + void theTwoStrippedPractitionerReferencesAreRestored() throws Exception { + // The two references the pre-change code stripped carried display="Carl Clinician" - the + // PractitionerRole -> Practitioner link built outside v2tofhir bookkeeping. They now ship + // with the value the conversion produced. + Bundle b = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, queryParams("_include", "*:*")); + + List roles = resourcesOfType(b, "PractitionerRole"); + assertFalse(roles.isEmpty(), "the wildcard include should reach the PractitionerRole"); + List namedRefs = roles.stream() + .flatMap(r -> referencesOf(r).stream()) + .filter(Reference::hasDisplay) + .toList(); + assertFalse(namedRefs.isEmpty(), "the named practitioner references should be present"); + assertTrue(namedRefs.stream().allMatch(Reference::hasReference), + "a reference the conversion gave a value must keep it"); + } + + @Test + void patientReferenceValueIsTheSameWhetherOrNotTheTargetIsIncluded() throws Exception { + Bundle without = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, queryParams()); + Bundle with = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, + queryParams("_include", "Immunization:patient")); + + String refWithout = ((Immunization) resourcesOfType(without, "Immunization").get(0)) + .getPatient().getReference(); + String refWith = ((Immunization) resourcesOfType(with, "Immunization").get(0)) + .getPatient().getReference(); + + assertEquals(refWithout, refWith, + "asking for the target must not change the reference value"); + List patients = resourcesOfType(with, "Patient"); + assertEquals(1, patients.size(), "the _include should have retained the Patient"); + assertEquals("Patient/" + patients.get(0).getIdElement().getIdPart(), refWith, + "the reference should resolve to the retained Patient"); + } + + @Test + void aWhitelistedResourceIsTraversedLikeAnyOther() throws Exception { + // whitelistInfrastructureCreatedResources adds the resource to the retained list, so + // markIncludedResources visits it and walks its reverse references. The Provenance point at + // the DocumentReference, so white-listing the DocumentReference lets a _revinclude reach them. + Bundle b = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, queryParams( + "_include", "Resource:source:DocumentReference", + "_revinclude", "*:*")); + + assertFalse(resourcesOfType(b, "DocumentReference").isEmpty(), + "the white-list should retain the DocumentReference"); + List provenances = resourcesOfType(b, "Provenance"); + assertFalse(provenances.isEmpty(), + "the Provenance should be reached by traversing the white-listed DocumentReference"); + assertTrue(entriesWithMode(b, SearchEntryMode.INCLUDE).containsAll(provenances), + "a resource reached by a join is search.mode=include"); + } + + @Test + void typeQualifiedRevincludeCannotMatchProvenance() throws Exception { + // Known limitation, recorded in design.md and not introduced here: v2tofhir gives Provenance + // a bare id with no resource type, and includeMatches compares a type-qualified _revinclude + // against that type, so "Provenance".equals(null) fails. The wildcard form short-circuits the + // check - see aWhitelistedResourceIsTraversedLikeAnyOther. Change this test when that is fixed. + Bundle b = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, queryParams( + "_include", "Resource:source:DocumentReference", + "_revinclude", "Provenance")); + + assertFalse(resourcesOfType(b, "DocumentReference").isEmpty(), + "the white-list still retains the DocumentReference"); + assertTrue(resourcesOfType(b, "Provenance").isEmpty(), + "the type-qualified form cannot match a bare-id resource"); + } + + @Test + void reverseIncludeResolvesOnlyFromARetainedResource() throws Exception { + Bundle without = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, + queryParams("_revinclude", "Observation")); + assertTrue(resourcesOfType(without, "Observation").isEmpty(), + "the Observations reference the Patient, which is not retained"); + + Bundle with = query(RSP_Z42_MESSAGE, RECOMMENDATION_URI, queryParams( + "_include", "ImmunizationRecommendation:patient", + "_revinclude", "Observation")); + List observations = resourcesOfType(with, "Observation"); + assertFalse(observations.isEmpty(), + "retaining the Patient should make the same _revinclude resolve"); + assertTrue(entriesWithMode(with, SearchEntryMode.INCLUDE).containsAll(observations), + "a revincluded Observation is a join, so search.mode=include"); + } + + @Test + void conversionCreatedResourcesArriveUnderAnOrdinaryInclude() throws Exception { + // Location and the performer resources carry Parser.SOURCE, and an ordinary _include + // retains them anyway - the Resource:source white-list is not required. + Bundle b = query(RSP_Z32_MESSAGE, IMMUNIZATION_URI, + queryParams("_include", "Immunization:location")); + + List locations = resourcesOfType(b, "Location"); + assertFalse(locations.isEmpty(), + "Immunization:location should retain the conversion-created Location"); + assertTrue(entriesWithMode(b, SearchEntryMode.INCLUDE).containsAll(locations), + "an _include hit is a join, so search.mode=include"); + } + + /** The four parameters a caller sends to get Patient, Organization, Immunization and Location. */ + private static final String[] FOUR_TYPE_PARAMS = { + "_include", "ImmunizationRecommendation:patient", + "_include", "ImmunizationRecommendation:authority", + "_revinclude", "Immunization", + "_include", "Immunization:location" + }; + + @Test + void fourTypeQueryWalksTheWholeChainInOnePass() throws Exception { + // recommendation -> Patient + Organization forward, Patient -> Immunization reverse, + // Immunization -> Location forward. The resources list grows as it is iterated, so all + // four hops resolve in a single pass. + Bundle b = query(RSP_Z42_WITH_FACILITY_MESSAGE, RECOMMENDATION_URI, + queryParams(FOUR_TYPE_PARAMS)); + + List included = entriesWithMode(b, SearchEntryMode.INCLUDE); + for (String type : List.of("Patient", "Organization", "Immunization", "Location")) { + List found = resourcesOfType(b, type); + assertFalse(found.isEmpty(), () -> type + " was asked for and should arrive"); + assertTrue(included.containsAll(found), () -> type + " should be search.mode=include"); + } + assertTrue(resourcesOfType(b, "Observation").isEmpty(), + "no _revinclude named Observation, so none should arrive"); + assertTrue(entriesWithMode(b, SearchEntryMode.MATCH).stream() + .allMatch(r -> "ImmunizationRecommendation".equals(r.fhirType())), + "only the requested type should be a match"); + } + + @Test + void parameterOrderDoesNotChangeTheSearchset() throws Exception { + String[] reversed = new String[FOUR_TYPE_PARAMS.length]; + for (int i = 0; i < FOUR_TYPE_PARAMS.length; i += 2) { + reversed[FOUR_TYPE_PARAMS.length - 2 - i] = FOUR_TYPE_PARAMS[i]; + reversed[FOUR_TYPE_PARAMS.length - 1 - i] = FOUR_TYPE_PARAMS[i + 1]; + } + + Bundle forward = query(RSP_Z42_WITH_FACILITY_MESSAGE, RECOMMENDATION_URI, + queryParams(FOUR_TYPE_PARAMS)); + Bundle backward = query(RSP_Z42_WITH_FACILITY_MESSAGE, RECOMMENDATION_URI, + queryParams(reversed)); + + assertEquals(typeModeCounts(forward), typeModeCounts(backward), + "every include is applied to every resource as it is reached, so order cannot matter"); + } + + @Test + void revincludeWithNoRetainedAnchorFindsNothing() throws Exception { + // With no forward _include at all, nothing the Immunizations reference is retained, so + // there is no resource to resolve the reverse hit from. + Bundle b = query(RSP_Z42_WITH_FACILITY_MESSAGE, RECOMMENDATION_URI, queryParams( + "_revinclude", "Immunization", + "_include", "Immunization:location")); + + assertTrue(resourcesOfType(b, "Immunization").isEmpty(), + "no retained resource anchors the reverse hit"); + assertTrue(resourcesOfType(b, "Location").isEmpty(), + "and the Location hangs off the Immunization, so it is lost with it"); + } + + @Test + void anyRetainedReferencedResourceAnchorsTheReverseInclude() throws Exception { + // The Patient is the anchor a caller reaches for, but it is not the only one: the doses also + // reference the schedule Organization through protocolApplied.authority, so retaining that + // Organization resolves an unqualified _revinclude=Immunization just as well. + Bundle b = query(RSP_Z42_WITH_FACILITY_MESSAGE, RECOMMENDATION_URI, queryParams( + "_include", "ImmunizationRecommendation:authority", + "_revinclude", "Immunization")); + + assertTrue(resourcesOfType(b, "Patient").isEmpty(), "the Patient was not asked for"); + assertEquals(2, resourcesOfType(b, "Immunization").size(), + "the Organization anchors the reverse hit in the Patient's place"); + } + + @Test + void qualifyingARevincludeDoesNotPinTheTraversalPath() throws Exception { + // ParserUtils.createReference caches one canonical Reference per resource and + // addSearchNames accumulates onto it, so the Immunization's reverse names are the union of + // every path that points at it - "patient" and "authority" both. The same Reference instance + // sits in the Patient's and the Organization's Reverses sets, so naming a search path does + // not restrict which retained resource the reverse hit may resolve from. + Bundle viaOrganization = query(RSP_Z42_WITH_FACILITY_MESSAGE, RECOMMENDATION_URI, queryParams( + "_include", "ImmunizationRecommendation:authority", + "_revinclude", "Immunization:patient")); + + assertTrue(resourcesOfType(viaOrganization, "Patient").isEmpty(), + "the Patient was not asked for"); + assertEquals(2, resourcesOfType(viaOrganization, "Immunization").size(), + "Immunization:patient still resolves, anchored on the retained Organization"); + + // An unregistered name is the case that does restrict: it matches nothing. + Bundle unregistered = query(RSP_Z42_WITH_FACILITY_MESSAGE, RECOMMENDATION_URI, queryParams( + "_include", "ImmunizationRecommendation:authority", + "_revinclude", "Immunization:nosuchsearchname")); + + assertTrue(resourcesOfType(unregistered, "Immunization").isEmpty(), + "a search name the conversion never registered matches nothing"); + } + + @Test + void theRoutingKeyDoesNotChangeTheReturnedTypes() throws Exception { + // Searchset assembly reads only the converted bundle and the query parameters. It consults no + // organization, pipeline or solution, so the destination that selected them cannot affect what + // comes back. Asserted on the routing key rather than on two pipeline configurations, because + // there is no config input to assembly for a pipeline fixture to vary. + initRequestContext(); + Bundle first = controller(hubReturning(RSP_Z42_MESSAGE)) + .iisQuery("dev", fhirRequest(RECOMMENDATION_URI, null, HISTORY_PARAMS)).getBody(); + initRequestContext(); + Bundle second = controller(hubReturning(RSP_Z42_MESSAGE)) + .iisQuery("other", fhirRequest("/fhir/other/ImmunizationRecommendation", null, + HISTORY_PARAMS)).getBody(); + + assertNotNull(first); + assertNotNull(second); + assertEquals(typeModeCounts(first), typeModeCounts(second), + "the same query against the same response must yield the same types and modes"); + } + + /** The resources in a bundle carrying the given search mode. */ + private static List entriesWithMode(Bundle bundle, SearchEntryMode mode) { + return bundle.getEntry().stream() + .filter(e -> e.getResource() != null && e.getSearch() != null + && mode.equals(e.getSearch().getMode())) + .map(BundleEntryComponent::getResource) + .toList(); + } + + + private static List resourcesOfType(Bundle bundle, String fhirType) { + return bundle.getEntry().stream() + .map(BundleEntryComponent::getResource) + .filter(r -> r != null && fhirType.equals(r.fhirType())) + .toList(); + } + @Test void matchHonorsFhirJsonAccept() throws Exception { initRequestContext(); @@ -468,13 +1231,39 @@ private static HubController hubReturning(String hl7Message) throws Exception { } private static HttpServletRequest fhirRequest(String uri, String accept) { + return fhirRequest(uri, accept, Collections.emptyMap()); + } + + /** + * A mock request carrying query parameters, so searchset tests can drive + * {@code _include} / {@code _revinclude} and the patient identifier the query needs. + * Absent parameters read back as null, matching a real request. + */ + private static HttpServletRequest fhirRequest(String uri, String accept, Map params) { HttpServletRequest req = mock(HttpServletRequest.class); - when(req.getParameterMap()).thenReturn(Collections.emptyMap()); + when(req.getParameterMap()).thenReturn(params); when(req.getRequestURI()).thenReturn(uri); when(req.getHeader(HttpHeaders.ACCEPT)).thenReturn(accept); + when(req.getParameterValues(anyString())) + .thenAnswer(i -> params.get(i.getArgument(0, String.class))); + when(req.getParameter(anyString())).thenAnswer(i -> { + String[] values = params.get(i.getArgument(0, String.class)); + return values == null || values.length == 0 ? null : values[0]; + }); return req; } + /** Query parameters selecting the fixture patient, plus any extras under test. */ + private static Map queryParams(String... extras) { + Map params = new LinkedHashMap<>(); + params.put(IzQuery.PATIENT_LIST, new String[] {"TEST|0000001"}); + for (int i = 0; i < extras.length; i += 2) { + params.merge(extras[i], new String[] {extras[i + 1]}, + (a, b) -> Stream.concat(Arrays.stream(a), Arrays.stream(b)).toArray(String[]::new)); + } + return params; + } + private static Parameters matchParameters() { Patient patient = new Patient(); patient.addName(new HumanName().setFamily("CuyahogaAIRA").addGiven("MarnyAIRA")); diff --git a/testing/scripts/TS_Integration_Test.postman_collection.json b/testing/scripts/TS_Integration_Test.postman_collection.json index c08930d93..41775ebb5 100644 --- a/testing/scripts/TS_Integration_Test.postman_collection.json +++ b/testing/scripts/TS_Integration_Test.postman_collection.json @@ -1795,6 +1795,148 @@ } }, "response": [] + }, + { + "name": "TS_TC_07d FHIR Query Returns No Patient By Default", + "event": [ + { + "listen": "test", + "script": { + "type": "text/javascript", + "exec": [ + "pm.test(\"Status code is 200\", function () {", + " pm.response.to.have.status(200);", + "});", + "let response = pm.response.json();", + "pm.test(\"Verify Immunization is present\", function() {", + " pm.expect(response.entry.find(e => e.resource && e.resource.resourceType === \"Immunization\")).to.not.be.undefined;", + "});", + "pm.test(\"Verify no Patient is returned without _include\", function() {", + " pm.expect(response.entry.find(e => e.resource && e.resource.resourceType === \"Patient\")).to.be.undefined;", + "});", + "pm.test(\"Every entry carries a search mode\", function() {", + " response.entry.forEach(e => pm.expect(e.search && e.search.mode).to.not.be.undefined);", + "});" + ] + } + } + ], + "request": { + "auth": { + "type": "noauth" + }, + "method": "GET", + "header": [ + { + "key": "", + "value": "", + "type": "text", + "disabled": true + } + ], + "url": { + "raw": "{{protocol}}://{{host}}:{{port}}/fhir/dev/Immunization?patient.given={{fhirPatientGiven}}&patient.family={{fhirPatientFamily}}&patient.birthdate={{fhirPatientDOB}}", + "protocol": "{{protocol}}", + "host": [ + "{{host}}" + ], + "port": "{{port}}", + "path": [ + "fhir", + "dev", + "Immunization" + ], + "query": [ + { + "key": "patient.given", + "value": "{{fhirPatientGiven}}" + }, + { + "key": "patient.family", + "value": "{{fhirPatientFamily}}" + }, + { + "key": "patient.birthdate", + "value": "{{fhirPatientDOB}}" + } + ] + } + }, + "response": [] + }, + { + "name": "TS_TC_07e FHIR Query With _include Returns Patient", + "event": [ + { + "listen": "test", + "script": { + "type": "text/javascript", + "exec": [ + "pm.test(\"Status code is 200\", function () {", + " pm.response.to.have.status(200);", + "});", + "let response = pm.response.json();", + "let patient = response.entry.find(e => e.resource && e.resource.resourceType === \"Patient\");", + "pm.test(\"Verify Patient is present when included\", function() {", + " pm.expect(patient).to.not.be.undefined;", + "});", + "pm.test(\"Verify the included Patient is search.mode=include\", function() {", + " pm.expect(patient.search.mode).to.equal(\"include\");", + "});", + "pm.test(\"Verify only Immunization is a match\", function() {", + " response.entry.filter(e => e.search.mode === \"match\")", + " .forEach(e => pm.expect(e.resource.resourceType).to.equal(\"Immunization\"));", + "});" + ] + } + } + ], + "request": { + "auth": { + "type": "noauth" + }, + "method": "GET", + "header": [ + { + "key": "", + "value": "", + "type": "text", + "disabled": true + } + ], + "url": { + "raw": "{{protocol}}://{{host}}:{{port}}/fhir/dev/Immunization?patient.given={{fhirPatientGiven}}&patient.family={{fhirPatientFamily}}&patient.birthdate={{fhirPatientDOB}}&_include=Immunization:patient", + "protocol": "{{protocol}}", + "host": [ + "{{host}}" + ], + "port": "{{port}}", + "path": [ + "fhir", + "dev", + "Immunization" + ], + "query": [ + { + "key": "patient.given", + "value": "{{fhirPatientGiven}}" + }, + { + "key": "patient.family", + "value": "{{fhirPatientFamily}}" + }, + { + "key": "patient.birthdate", + "value": "{{fhirPatientDOB}}" + }, + { + "key": "_include", + "value": "Immunization:patient" + } + ] + } + }, + "response": [] } ], "auth": { @@ -16351,6 +16493,140 @@ } }, "response": [] + }, + { + "name": "TS_TC_07d FHIR Query Returns No Patient By Default with JWT", + "event": [ + { + "listen": "test", + "script": { + "type": "text/javascript", + "exec": [ + "pm.test(\"Status code is 200\", function () {", + " pm.response.to.have.status(200);", + "});", + "let response = pm.response.json();", + "pm.test(\"Verify Immunization is present\", function() {", + " pm.expect(response.entry.find(e => e.resource && e.resource.resourceType === \"Immunization\")).to.not.be.undefined;", + "});", + "pm.test(\"Verify no Patient is returned without _include\", function() {", + " pm.expect(response.entry.find(e => e.resource && e.resource.resourceType === \"Patient\")).to.be.undefined;", + "});", + "pm.test(\"Every entry carries a search mode\", function() {", + " response.entry.forEach(e => pm.expect(e.search && e.search.mode).to.not.be.undefined);", + "});" + ] + } + } + ], + "request": { + "method": "GET", + "header": [ + { + "key": "x-xform-organization", + "value": "0d15449b-fb08-4013-8985-20c148b353fe", + "type": "text" + } + ], + "url": { + "raw": "{{no_cert_protocol}}://{{no_cert_host}}:{{no_cert_port}}/fhir/dev/Immunization?patient.given={{fhirPatientGiven}}&patient.family={{fhirPatientFamily}}&patient.birthdate={{fhirPatientDOB}}", + "protocol": "{{no_cert_protocol}}", + "host": [ + "{{no_cert_host}}" + ], + "port": "{{no_cert_port}}", + "path": [ + "fhir", + "dev", + "Immunization" + ], + "query": [ + { + "key": "patient.given", + "value": "{{fhirPatientGiven}}" + }, + { + "key": "patient.family", + "value": "{{fhirPatientFamily}}" + }, + { + "key": "patient.birthdate", + "value": "{{fhirPatientDOB}}" + } + ] + } + }, + "response": [] + }, + { + "name": "TS_TC_07e FHIR Query With _include Returns Patient with JWT", + "event": [ + { + "listen": "test", + "script": { + "type": "text/javascript", + "exec": [ + "pm.test(\"Status code is 200\", function () {", + " pm.response.to.have.status(200);", + "});", + "let response = pm.response.json();", + "let patient = response.entry.find(e => e.resource && e.resource.resourceType === \"Patient\");", + "pm.test(\"Verify Patient is present when included\", function() {", + " pm.expect(patient).to.not.be.undefined;", + "});", + "pm.test(\"Verify the included Patient is search.mode=include\", function() {", + " pm.expect(patient.search.mode).to.equal(\"include\");", + "});", + "pm.test(\"Verify only Immunization is a match\", function() {", + " response.entry.filter(e => e.search.mode === \"match\")", + " .forEach(e => pm.expect(e.resource.resourceType).to.equal(\"Immunization\"));", + "});" + ] + } + } + ], + "request": { + "method": "GET", + "header": [ + { + "key": "x-xform-organization", + "value": "0d15449b-fb08-4013-8985-20c148b353fe", + "type": "text" + } + ], + "url": { + "raw": "{{no_cert_protocol}}://{{no_cert_host}}:{{no_cert_port}}/fhir/dev/Immunization?patient.given={{fhirPatientGiven}}&patient.family={{fhirPatientFamily}}&patient.birthdate={{fhirPatientDOB}}&_include=Immunization:patient", + "protocol": "{{no_cert_protocol}}", + "host": [ + "{{no_cert_host}}" + ], + "port": "{{no_cert_port}}", + "path": [ + "fhir", + "dev", + "Immunization" + ], + "query": [ + { + "key": "patient.given", + "value": "{{fhirPatientGiven}}" + }, + { + "key": "patient.family", + "value": "{{fhirPatientFamily}}" + }, + { + "key": "patient.birthdate", + "value": "{{fhirPatientDOB}}" + }, + { + "key": "_include", + "value": "Immunization:patient" + } + ] + } + }, + "response": [] } ] },