Skip to content

Commit 936d5ab

Browse files
authored
feat(dr-egeria): implement Link/Detach Schema Attribute Definition (ISSUE-48, Option 2) (#288)
Per user direction: "implement Option 2 for SchemaAttributeDefinition - and we'll replace it once the new endpoint is available." New Dr.Egeria command family: Link/Detach Schema Attribute Definition (Data Designer family, commands_data_designer.json), a new "Schema Attribute" attribute (Reference Name style, existing_element: SchemaAttribute, alongside the existing "Data Field" attribute), and a new LinkSchemaAttributeDefinitionProcessor (md_processing/v2/ data_designer.py): - Link/Attach/Add: calls MetadataExpert._async_create_related_elements with a NewRelatedElementsRequestBody (typeName: SchemaAttributeDefinition), captures the returned relationship GUID into parsed_output["guid"], displays it in the result markdown. - Detach/Unlink/Remove: looks the relationship up by its element pair via _async_find_relationships_between_elements rather than requiring a separate GUID attribute -- safe because SchemaAttributeDefinition's relationshipCategory is UNI_LINK with AT_MOST_ONE cardinality on both ends (confirmed live via ValidMetadataManager.get_all_relationship_defs()), so there's no multi-instance ambiguity. Deletes via _async_delete_related_elements(relationship_guid, ...). - supports_target_element_lookup() overridden to False (relationship-only processor -- see ISSUE-68's AsyncBaseCommandProcessor upsert-rewrite-bug fix for why this matters for any relationship processor). No new pyegeria SDK surface needed -- _async_create_related_elements/ _async_find_relationships_between_elements/_async_delete_related_elements all already existed on MetadataExpert (Option 2's whole premise). Verified live, full round trip against a real DataField/SchemaAttribute pair: Link via --process created the relationship and displayed its GUID; a direct query confirmed it existed; Detach via --process removed it; the same query afterward returned "No elements found". New tests: test_link_schema_attribute_definition.py (5 tests, fake client, no live server needed). Full pytest tests/micro-tests/ green (one pre-existing, unrelated failure in test_sync_members.py confirmed present on upstream main before this change too -- from PR #287, not touched here). PYEGERIA_ISSUES.md: ISSUE-48 moved to Fixed/Resolved with full writeup, explicitly flagged as temporary -- migrate to a bespoke DataDesigner wrapper (Option 1) once Egeria ships a real SchemaAttributeDefinition endpoint; re-check the OpenAPI-spec verification in this issue's own history periodically. Also fixed while resolving a stash-pop merge conflict: no functional change, just interleaved cleanly with PR #287's unrelated "Detach Data Value Specification from Element" addition that landed on upstream/main concurrently. Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
1 parent 668f4d8 commit 936d5ab

6 files changed

Lines changed: 378 additions & 82 deletions

File tree

‎PYEGERIA_ISSUES.md‎

Lines changed: 59 additions & 82 deletions
Original file line numberDiff line numberDiff line change
@@ -768,88 +768,6 @@ on an Egeria Server capability that doesn't exist yet — but the pyegeria/
768768
Dr.Egeria-side work each will need once that capability ships is written
769769
into the entry now, so it isn't rediscovered from scratch later.
770770

771-
### ISSUE-48: No OMVS wrapper exists for the `SchemaAttributeDefinition` relationship (physical `SchemaAttribute` ↔ logical `DataField`)
772-
773-
**Status:** deferred 2026-08-15 — user's decision: wait until Egeria ships
774-
the actual REST endpoint for this relationship before building a pyegeria
775-
wrapper or Dr.Egeria command against it. Both options considered below
776-
remain valid candidates once that lands; no implementation work done here.
777-
778-
**Re-checked 2026-08-19, still deferred, still valid — but Option 2 now
779-
confirmed to actually work.** Fetched the live server's full `/v3/api-docs`
780-
OpenAPI spec (1.77 MB, successfully retrieved) and grepped both the URL
781-
paths and the schema component names for `SchemaAttributeDefinition`:
782-
**zero hits in either.** No bespoke REST endpoint has shipped — the wait
783-
condition from 2026-08-15 hasn't been met, so the deferral stands as-is.
784-
Separately, live-tested Option 2 (the generic
785-
`MetadataExpert`/`NewRelatedElementsRequestBody` path,
786-
`typeName: "SchemaAttributeDefinition"`) end-to-end against a real
787-
`DataField`/`SchemaAttribute` pair — **it works today**: created the
788-
relationship (`GUIDResponse` with a real guid), then cleaned it up via
789-
`_async_delete_related_elements`. Previously this was listed as a
790-
considered-but-untested option; now confirmed viable, so if the team
791-
wants to unblock a Dr.Egeria command without waiting further, Option 2 is
792-
ready to use as-is — no new pyegeria SDK surface required, same
793-
mechanism `CurationLinkProcessor` already uses for `ResourceList`/
794-
`MoreInformation`.
795-
796-
**Original status:** open — SDK gap to validate with the team, not yet a confirmed
797-
bug. Raised 2026-08-07 while triaging a request to add three "missing"
798-
Dr.Egeria Data Designer commands (SemanticAssignment, DataClassAssignment,
799-
SchemaTypeImplementation). Two of the three turned out not to be gaps at
800-
all once checked against the real type system and existing compact specs:
801-
802-
- **SemanticAssignment** (schema/field → glossary term) already exists —
803-
`Link Semantic Assignment` / `Unlink Semantic Assignment` in the
804-
**Curation** family (`commands_curation_compact.json`, `OM_TYPE:
805-
SemanticAssignment`, wired to `ClassificationExplorer.
806-
_async_setup_semantic_assignment` in `CurationLinkProcessor`). Its
807-
`Target Element` is a generic GUID, so it already works against a data
808-
field or schema attribute today.
809-
- **DataClassAssignment** on a physical schema attribute is also already
810-
covered — and the type name itself is stale. The old `DataClassAssignment`
811-
relationship was replaced in this Egeria version by
812-
`DataValueAssignmentRelationship` (0540 —
813-
`OpenMetadataTypesArchive1_2.java`), whose end 1 is `Referenceable` (any
814-
element, including a schema attribute). Data Designer's existing `Assign
815-
Data Value Specification` command (`OM_TYPE: DataValueAssignment`)
816-
already covers this generically via its `Element Id` attribute.
817-
818-
**The third one is a real gap, at the SDK level, not just the Dr.Egeria
819-
command-spec level.** "SchemaTypeImplementation" isn't a real type name in
820-
this Egeria version at all. The type that actually connects a physical
821-
`SchemaAttribute` to a logical `DataField` is `SchemaAttributeDefinition`
822-
(0580, `OpenMetadataTypesArchive5_3.java` —
823-
`getSchemaAttributeDefinitionRelationship()`), ends `derivedFromDataField`
824-
(on `DataField`, AT_MOST_ONE) / `equivalentSchemaAttribute` (on
825-
`SchemaAttribute`, AT_MOST_ONE). Confirmed via grep across `pyegeria/omvs/`
826-
and `pyegeria/http clients/*.http`: **no bespoke wrapper method exists**
827-
for setting up or clearing this relationship — no `_async_link_*`/
828-
`_async_setup_*` method, no `.http` worked example, in `data_designer.py`,
829-
`data_engineer.py`, or `classification_explorer.py`.
830-
831-
**Options considered, not yet decided:**
832-
1. Add a bespoke wrapper (e.g.
833-
`DataDesigner._async_link_schema_attribute_definition`/
834-
`_async_detach_schema_attribute_definition`) mirroring
835-
`_async_setup_semantic_assignment`'s shape, then wire a Dr.Egeria
836-
`Link/Unlink Schema Attribute Definition` command to it. Most
837-
consistent with how other bespoke-family relationships are handled, but
838-
requires new SDK surface first.
839-
2. Skip the bespoke wrapper and drive it through the existing generic
840-
`MetadataExpert._async_create_related_elements`/
841-
`_async_delete_related_elements` (typeName-based) — the same mechanism
842-
`CurationLinkProcessor` already uses for other Tier-2 gaps with no
843-
dedicated method (`ResourceList`, `MoreInformation`). No new SDK method
844-
needed, but less discoverable/typed than a bespoke wrapper.
845-
846-
No command has been added to the compact JSON for this yet — deferred
847-
pending the team's input on which approach (or whether the relationship
848-
is even meant to be user-authorable via Dr.Egeria, vs. only ever
849-
system-derived from a physical→logical mapping tool).
850-
851-
---
852-
853771
---
854772

855773
# Quick reference: which OMVS client class for which purpose
@@ -881,6 +799,65 @@ system-derived from a physical→logical mapping tool).
881799

882800
## Fixed / Resolved
883801

802+
### ISSUE-48: No OMVS wrapper exists for the `SchemaAttributeDefinition` relationship (physical `SchemaAttribute` ↔ logical `DataField`)
803+
804+
**Status:** implemented 2026-08-19 via Option 2 (the generic
805+
`MetadataExpert` relationship mechanism) — deferred 2026-08-15, re-checked
806+
2026-08-19 (still no bespoke Egeria endpoint, confirmed against the full
807+
`/v3/api-docs` spec; Option 2 confirmed viable live), then built the same
808+
day per the user's go-ahead.
809+
810+
**What shipped:** a new Dr.Egeria command family, `Link/Detach Schema
811+
Attribute Definition` (`Data Designer` family,
812+
`commands_data_designer.json`, new `Schema Attribute` attribute alongside
813+
the existing `Data Field`), backed by a new
814+
`LinkSchemaAttributeDefinitionProcessor`
815+
(`md_processing/v2/data_designer.py`):
816+
- **Link/Attach/Add:** calls
817+
`MetadataExpert._async_create_related_elements` with a
818+
`NewRelatedElementsRequestBody` (`typeName: SchemaAttributeDefinition`),
819+
captures the returned relationship GUID into
820+
`parsed_output["guid"]`, and displays it in the result markdown.
821+
- **Detach/Unlink/Remove:** looks the relationship up by its element pair
822+
via `_async_find_relationships_between_elements`
823+
(`end1EntityGUIDs`/`end2EntityGUIDs`) rather than requiring a separate
824+
GUID attribute — safe because `SchemaAttributeDefinition`'s
825+
`relationshipCategory` is `UNI_LINK` with `AT_MOST_ONE` cardinality on
826+
both ends (confirmed live via
827+
`ValidMetadataManager.get_all_relationship_defs()`), so there's no
828+
multi-instance ambiguity to resolve. Then deletes via
829+
`_async_delete_related_elements(relationship_guid, ...)`.
830+
- `supports_target_element_lookup()` overridden to `False` (relationship-
831+
only processor — see ISSUE-68's `AsyncBaseCommandProcessor` upsert-
832+
rewrite-bug fix for why this matters; avoids the same latent Update-verb
833+
hazard flagged there for any future relationship processor).
834+
835+
**No new pyegeria SDK surface required** — `_async_create_related_elements`/
836+
`_async_find_relationships_between_elements`/`_async_delete_related_elements`
837+
all already existed on `MetadataExpert`.
838+
839+
**Verified live**, full round trip against a real `DataField`/
840+
`SchemaAttribute` pair: `Link Schema Attribute Definition` via
841+
`--process` created the relationship and displayed its GUID; a direct
842+
`_async_find_relationships_between_elements` query confirmed it existed;
843+
`Detach Schema Attribute Definition` via `--process` removed it; the same
844+
query afterward returned "No elements found", confirming clean deletion.
845+
846+
**Tests:** `test_link_schema_attribute_definition.py` (5 tests, fake
847+
client, no live server needed).
848+
849+
**Explicitly temporary, per the user's own framing** ("implement Option 2
850+
... and we'll replace it once the new endpoint is available") — re-check
851+
this issue's own "still deferred" 2026-08-19 OpenAPI-spec check
852+
periodically; once Egeria ships a bespoke `SchemaAttributeDefinition`
853+
endpoint, migrate `LinkSchemaAttributeDefinitionProcessor` to a dedicated
854+
`DataDesigner._async_link_schema_attribute_definition`/
855+
`_async_detach_schema_attribute_definition` wrapper (Option 1) instead —
856+
same compact-spec command, same attributes, just a different
857+
implementation underneath.
858+
859+
---
860+
884861
### ISSUE-55: Desired enhancement — true "exclude type" / NOT semantics for `findMetadataElements`
885862

886863
**Status:** fixed 2026-08-18 — Egeria shipped it. Raised 2026-08-05

‎md_processing/data/compact_commands/commands_data_designer.json‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3264,6 +3264,25 @@
32643264
"max_cardinality": 1,
32653265
"level": "Domain",
32663266
"Journal Entry": ""
3267+
},
3268+
"Schema Attribute": {
3269+
"variable_name": "schema_attribute",
3270+
"inUpdate": true,
3271+
"attr_labels": "",
3272+
"examples": "",
3273+
"default_value": "",
3274+
"valid_values": [],
3275+
"existing_element": "SchemaAttribute",
3276+
"description": "A physical schema attribute (e.g. a TabularColumn). Preferably a qualified name.",
3277+
"generated": false,
3278+
"style": "Reference Name",
3279+
"user_specified": true,
3280+
"unique": false,
3281+
"input_required": true,
3282+
"min_cardinality": 1,
3283+
"max_cardinality": 1,
3284+
"level": "Domain",
3285+
"Journal Entry": ""
32673286
}
32683287
},
32693288
"bundles": {
@@ -3880,6 +3899,28 @@
38803899
"Element Id"
38813900
],
38823901
"Journal Entry": ""
3902+
},
3903+
"Link Schema Attribute Definition": {
3904+
"display_name": "schema_attribute_definition",
3905+
"qn_prefix": "",
3906+
"alternate_names": [],
3907+
"family": "Data Designer",
3908+
"description": "Link a logical Data Field to the physical Schema Attribute that implements it (e.g. a TabularColumn), or the reverse lookup. Implemented via the generic MetadataExpert relationship mechanism (typeName: SchemaAttributeDefinition) -- no bespoke Egeria REST endpoint exists for this relationship yet (PYEGERIA_ISSUES.md ISSUE-48); will be migrated to a dedicated wrapper once Egeria ships one.",
3909+
"verb": "Link",
3910+
"upsert": false,
3911+
"attach": true,
3912+
"level": "Basic",
3913+
"find_method": "",
3914+
"find_constraints": "",
3915+
"extra_find": "",
3916+
"extra_constraints": "",
3917+
"OM_TYPE": "SchemaAttributeDefinition",
3918+
"bundle": "Link Command Base",
3919+
"custom_attributes": [
3920+
"Data Field",
3921+
"Schema Attribute"
3922+
],
3923+
"Journal Entry": ""
38833924
}
38843925
}
38853926
}

‎md_processing/dr_egeria.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -315,6 +315,7 @@ def reg(base, cls):
315315
DataValueSpecificationProcessor, DataClassProcessor, DataStructureProcessor, DataFieldProcessor, DataGrainProcessor,
316316
LinkDataFieldProcessor, LinkFieldToStructureProcessor, LinkDataValueDefinitionProcessor, LinkDataValueCompositionProcessor,
317317
LinkDataClassCompositionProcessor, LinkCertificationTypeToStructureProcessor,
318+
LinkSchemaAttributeDefinitionProcessor,
318319
AssignDataValueSpecificationProcessor
319320
)
320321

@@ -332,6 +333,7 @@ def reg(base, cls):
332333
reg("Link Data Value Composition", LinkDataValueCompositionProcessor)
333334
reg("Link Data Class Composition", LinkDataClassCompositionProcessor)
334335
reg("Link Certification Type to Data Structure", LinkCertificationTypeToStructureProcessor)
336+
reg("Link Schema Attribute Definition", LinkSchemaAttributeDefinitionProcessor)
335337
# "Assign Data Value Specification" (verb Attach/Link/Add) and "Detach
336338
# Data Value Specification from Element" (verb Detach/Unlink/Remove) both
337339
# route to the same AssignDataValueSpecificationProcessor, which branches

‎md_processing/v2/data_designer.py‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -922,6 +922,110 @@ async def fetch_element(self, guid: str) -> Optional[Dict[str, Any]]:
922922
return None
923923

924924

925+
class LinkSchemaAttributeDefinitionProcessor(AsyncBaseCommandProcessor):
926+
"""
927+
Processor for Link/Detach Schema Attribute Definition commands.
928+
929+
PYEGERIA_ISSUES.md ISSUE-48: no bespoke Egeria REST endpoint exists yet
930+
for the SchemaAttributeDefinition relationship (DataField <->
931+
SchemaAttribute) -- confirmed live via the server's full /v3/api-docs
932+
OpenAPI spec, zero hits for this type in either the URL paths or the
933+
schema components. This processor uses Option 2 from that issue: the
934+
generic MetadataExpert relationship mechanism
935+
(_async_create_related_elements/_async_delete_related_elements,
936+
typeName-based) rather than a dedicated wrapper method. Replace this
937+
with a bespoke DataDesigner._async_link_schema_attribute_definition/
938+
_async_detach_schema_attribute_definition (Option 1) once Egeria ships
939+
a real endpoint -- re-check ISSUE-48 before assuming it hasn't.
940+
941+
SchemaAttributeDefinition's relationshipCategory is UNI_LINK (at most
942+
one instance between a given pair -- confirmed live via
943+
ValidMetadataManager.get_all_relationship_defs(), both ends are
944+
AT_MOST_ONE cardinality), so Detach can safely look the relationship
945+
up by its element pair via _async_find_relationships_between_elements
946+
rather than requiring a separate GUID attribute -- there's no
947+
multi-instance ambiguity to resolve.
948+
"""
949+
950+
def get_command_spec(self) -> Dict[str, Any]:
951+
return get_command_spec(f"{self.command.verb} Schema Attribute Definition")
952+
953+
def supports_target_element_lookup(self) -> bool:
954+
# Relationship-only processor -- see GovernanceLinkProcessor's
955+
# identical override (md_processing/v2/governance.py) for why this
956+
# matters: without it, AsyncBaseCommandProcessor.execute()'s
957+
# Create<->Update upsert-transition logic can silently rewrite the
958+
# verb (ISSUE-68 follow-up).
959+
return False
960+
961+
async def fetch_as_is(self) -> Optional[Dict[str, Any]]:
962+
return None
963+
964+
async def _find_relationship_guid(self, data_field_guid: str, schema_attribute_guid: str) -> Optional[str]:
965+
result = await self.client.metadata_expert._async_find_relationships_between_elements({
966+
"class": "FindRelationshipRequestBody",
967+
"relationshipTypeName": "SchemaAttributeDefinition",
968+
"end1EntityGUIDs": [data_field_guid],
969+
"end2EntityGUIDs": [schema_attribute_guid],
970+
})
971+
# _async_find_relationships_between_elements returns the raw
972+
# {"relationships": [...], "mermaidGraph": ...} envelope (not a bare
973+
# list) whenever a match is found -- see
974+
# metadata_expert.process_related_element_list()'s final `return
975+
# elements` (the outer envelope dict, not the unwrapped list it
976+
# just validated non-empty).
977+
if not isinstance(result, dict):
978+
return None
979+
relationships = result.get("relationships") or []
980+
for rel in relationships:
981+
if isinstance(rel, dict) and rel.get("relationshipGUID"):
982+
return rel["relationshipGUID"]
983+
return None
984+
985+
async def apply_changes(self) -> str:
986+
verb = self.command.verb
987+
object_type = getattr(self, 'canonical_object_type', self.command.object_type)
988+
attributes = self.parsed_output["attributes"]
989+
990+
data_field_guid = attributes.get('Data Field', {}).get('guid')
991+
schema_attribute_guid = attributes.get('Schema Attribute', {}).get('guid')
992+
if not (data_field_guid and schema_attribute_guid):
993+
missing = []
994+
if not data_field_guid: missing.append("'Data Field'")
995+
if not schema_attribute_guid: missing.append("'Schema Attribute'")
996+
raise ValueError(f"Cannot {verb.lower()} Schema Attribute Definition: resolution failed for {', '.join(missing)}")
997+
998+
if verb in ["Link", "Attach", "Add"]:
999+
body = {
1000+
"class": "NewRelatedElementsRequestBody",
1001+
"typeName": "SchemaAttributeDefinition",
1002+
"metadataElement1GUID": data_field_guid,
1003+
"metadataElement2GUID": schema_attribute_guid,
1004+
"properties": {"class": "SchemaAttributeDefinitionProperties"},
1005+
}
1006+
new_rel_guid = await self.client.metadata_expert._async_create_related_elements(body)
1007+
logger.success(f"Linked Data Field {data_field_guid} to Schema Attribute {schema_attribute_guid}")
1008+
if new_rel_guid:
1009+
self.parsed_output["guid"] = new_rel_guid
1010+
return f"\n\n## {verb} {object_type}\n\nLinked {data_field_guid} to {schema_attribute_guid}. Relationship GUID: {new_rel_guid}"
1011+
return f"\n\n## {verb} {object_type}\n\nLinked {data_field_guid} to {schema_attribute_guid}"
1012+
1013+
elif verb in ["Detach", "Unlink", "Remove"]:
1014+
relationship_guid = await self._find_relationship_guid(data_field_guid, schema_attribute_guid)
1015+
if not relationship_guid:
1016+
raise ValueError(
1017+
f"Cannot {verb.lower()} Schema Attribute Definition: no existing relationship found "
1018+
f"between {data_field_guid} and {schema_attribute_guid}."
1019+
)
1020+
await self.client.metadata_expert._async_delete_related_elements(
1021+
relationship_guid, {"class": "DeleteRelationshipRequestBody"}
1022+
)
1023+
logger.success(f"Detached Schema Attribute Definition {relationship_guid}")
1024+
return f"\n\n## {verb} {object_type}\n\nDetached relationship {relationship_guid} between {data_field_guid} and {schema_attribute_guid}"
1025+
1026+
return self.command.raw_block
1027+
1028+
9251029
class AssignDataValueSpecificationProcessor(AsyncBaseCommandProcessor):
9261030
"""
9271031
Processor for Assign/Attach Data Value Specification to Element commands,
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
___
2+
3+
## Link Schema Attribute Definition
4+
> Link a logical Data Field to the physical Schema Attribute that implements it (e.g. a TabularColumn), or the reverse lookup. Implemented via the generic MetadataExpert relationship mechanism (typeName: SchemaAttributeDefinition) -- no bespoke Egeria REST endpoint exists for this relationship yet (PYEGERIA_ISSUES.md ISSUE-48); will be migrated to a dedicated wrapper once Egeria ships one.
5+
6+
### Data Field
7+
> **Input Required**: True
8+
9+
> **Attribute Type**: Reference Name
10+
11+
> **Description**: A data field name. Preferably a qualified name.
12+
13+
14+
### Schema Attribute
15+
> **Input Required**: True
16+
17+
> **Attribute Type**: Reference Name
18+
19+
> **Description**: A physical schema attribute (e.g. a TabularColumn). Preferably a qualified name.
20+
21+
22+
### Label
23+
> **Input Required**: False
24+
25+
> **Attribute Type**: Simple
26+
27+
> **Description**: A label used to identify or categorise a relationship link.
28+
29+
> **Alternative Labels**: Wire Label
30+
31+
32+
### Journal Entry
33+
> **Input Required**: False
34+
35+
> **Attribute Type**: Simple
36+
37+
> **Description**: A text entry into a journal.
38+
39+
40+
### Description
41+
> **Input Required**: False
42+
43+
> **Attribute Type**: Simple
44+
45+
> **Description**: A description.
46+
47+
48+
___

0 commit comments

Comments
 (0)