Feature ETP-3354: Dynamic DAS mapping layer

.planning/PROJECT.md

@@ -0,0 +1,99 @@
+# PROJECT.md
+
+## Project Name
+Dynamic DAS Mapping Layer (Option 5)
+
+## Vision
+Eliminate the code generation + compilation cycle for DAS entity mappings by making DTOs, converters, controllers, and field mappings fully dynamic at runtime. JPA entities remain generated (they change rarely), but everything above them becomes metadata-driven, reading `etrx_*` configuration tables directly.
+
+## Problem Statement
+Currently, any change to entity projections, field mappings, or connector configurations requires:
+1. Modifying `etrx_*` configuration tables in the database
+2. Running `generate.entities` Gradle task (FreeMarker templates -> Java source)
+3. Compiling the generated code
+4. Restarting the DAS service
+
+This creates a slow feedback loop, makes it impossible to add/modify mappings at runtime, and forces a full redeployment for configuration-only changes. The generated code (DTOs, converters, controllers, retrievers) follows predictable patterns that can be interpreted at runtime.
+
+## Solution: Pragmatic Hybrid Approach (Option 5)
+
+**Keep generated (change rarely):**
+- JPA Entity classes (Hibernate mappings to database tables)
+- JPA Repositories (Spring Data interfaces)
+- Base entity model (`modules_gen/com.etendorx.entities/src/main/entities/`)
+
+**Make dynamic at runtime:**
+- DTOs (Read/Write) -> `Map<String, Object>` driven by `etrx_projection_entity` + `etrx_entity_field`
+- Converters (Entity <-> DTO) -> Generic converter reading field metadata from DB
+- REST Controllers -> Single generic controller dispatching based on projection/entity name
+- JsonPath converters -> Dynamic field extraction using metadata-driven jsonpath expressions
+- Field mappings -> Already partially dynamic via `OBCONFieldMapping`, extend to all mappings
+- External ID resolution -> Already dynamic, no changes needed
+
+## Key Technical Decisions
+
+1. **Generic DTO representation**: Use `Map<String, Object>` instead of typed DTO classes
+2. **Metadata-driven conversion**: Read `etrx_entity_field` at runtime to know which entity properties map to which DTO fields
+3. **Dynamic REST endpoints**: Register/unregister REST routes based on `etrx_projection` table contents
+4. **Caching layer**: Cache metadata reads (projection definitions, field mappings) with invalidation on config change
+5. **Backwards compatibility**: Existing generated code continues to work; dynamic layer is additive, not replacement initially
+
+## Architecture
+
+```
+HTTP Request
+    |
+    v
+[Generic REST Controller]  -- reads projection metadata from cache/DB
+    |
+    v
+[Dynamic DTO Converter]    -- converts Entity <-> Map<String,Object> using field metadata
+    |
+    v
+[JPA Repository]           -- (still generated, stays as-is)
+    |
+    v
+[JPA Entity]               -- (still generated, stays as-is)
+    |
+    v
+[Database]
+```
+
+## Tech Stack Context
+- Java 17, Spring Boot 3.1.4, Spring Cloud 2022.0.4
+- Spring Data JPA + Hibernate
+- PostgreSQL (primary), Oracle (secondary)
+- Kafka for async processing
+- JWT authentication via Edge gateway
+- Existing codebase: ~200+ source files across 15+ modules
+
+## Constraints
+- Must not break existing generated mapping flow (coexistence during migration)
+- Must maintain JWT authentication and authorization model
+- Must maintain ExternalId resolution for connector integrations
+- Must support existing connector field mapping patterns (OBCONFieldMapping)
+- Performance: Dynamic resolution must not significantly degrade REST API response times
+- Must work with existing Spring Data REST projections until fully migrated
+
+## Success Criteria
+- New projections/entities can be exposed via REST without code generation or restart
+- Field mapping changes take effect without recompilation
+- Existing generated code continues to function during migration period
+- API response format remains compatible with current consumers
+- Metadata caching ensures performance parity with generated code
+
+## Milestone 1: Dynamic DAS Core
+Build the foundation: generic controller, dynamic DTO conversion, metadata caching, and integration with existing JPA layer.
+
+## Team Context
+- Etendo RX development team
+- Existing familiarity with Spring Boot, JPA, code generation pipeline
+- Codebase already mapped in `.planning/codebase/`
+
+## Codebase State
+- Brownfield: Large existing codebase with established patterns
+- Generated code in `modules_gen/`
+- Core libraries in `libs/`
+- Service modules in `modules_core/`
+- Integration modules in `modules/`
+- Full codebase analysis available in `.planning/codebase/*.md`

.planning/REQUIREMENTS.md

@@ -0,0 +1,82 @@
+# REQUIREMENTS.md
+
+## Milestone 1: Dynamic DAS Core
+
+### Functional Requirements
+
+**FR-1: Dynamic Entity Metadata Loading**
+- Load `etrx_projection`, `etrx_projection_entity`, `etrx_entity_field` at runtime
+- Build in-memory metadata model from these tables
+- Support all field mapping types: DM (Direct Mapping), JM (Java Mapping), CV (Constant Value), JP (JsonPath)
+- Resolve entity relationships (ad_table_id references) dynamically
+
+**FR-2: Generic DTO Conversion**
+- Convert JPA Entity -> `Map<String, Object>` (Read DTO) using field metadata
+- Convert `Map<String, Object>` (Write DTO) -> JPA Entity using field metadata
+- Support nested entity references (resolve related entities by ID)
+- Support `identifiesUnivocally` fields for entity lookup
+- Handle null values, type coercion, and date formatting
+- Support `MappingUtils.handleBaseObject()` equivalent for null safety
+
+**FR-3: Generic REST Controller**
+- Single controller that handles all dynamically-registered projections
+- URL pattern: `/{mappingPrefix}/{externalName}` matching existing convention
+- Support CRUD operations: GET (list with pagination), GET by ID, POST (create), PUT (update)
+- Support `json_path` query parameter for nested JSON extraction
+- Support batch POST (array of entities)
+- Maintain Swagger/OpenAPI documentation
+
+**FR-4: Dynamic Repository Layer**
+- Generic repository that wraps existing JPA repositories
+- Lookup JPA repository by entity class name at runtime
+- Support pagination via Spring Data `Pageable`
+- Integrate with `RestCallTransactionHandler` for transaction management
+- Support upsert logic (check existence before create)
+
+**FR-5: External ID Integration**
+- Maintain existing `ExternalIdService` integration
+- Call `add()` and `flush()` during save operations
+- Support `convertExternalToInternalId()` for incoming references
+- Work with `etrx_instance_connector` and `etrx_instance_externalid` tables
+
+**FR-6: Audit Trail Integration**
+- Set audit fields (createdBy, updatedBy, creationDate, updated) via `AuditServiceInterceptor`
+- Apply to both new and updated entities
+
+**FR-7: Validation**
+- Validate incoming data against `ismandatory` field metadata
+- Integrate with Jakarta Validator for entity-level constraints
+- Return meaningful error messages on validation failure
+
+**FR-8: Metadata Caching**
+- Cache projection/entity/field metadata in memory
+- Support cache invalidation (initially manual, later event-driven)
+- Minimize database queries for metadata lookups
+
+### Non-Functional Requirements
+
+**NFR-1: Performance**
+- Dynamic endpoint response time within 20% of generated equivalent
+- Metadata cache hit ratio > 95% under normal operation
+- No additional database round-trips for metadata on cached requests
+
+**NFR-2: Backwards Compatibility**
+- Existing generated controllers continue to work unchanged
+- No modification to JPA entities or repositories
+- Coexistence: both generated and dynamic endpoints can serve simultaneously
+- Same JSON response format as generated DTOs
+
+**NFR-3: Security**
+- Respect existing JWT authentication via Edge gateway
+- No new authentication/authorization surface
+- Validate all input to prevent injection
+
+**NFR-4: Observability**
+- Log dynamic endpoint registration at startup
+- Log metadata cache misses
+- Error logging consistent with existing patterns (Log4j2)
+
+**NFR-5: Testability**
+- Unit tests for generic converter with mock metadata
+- Integration tests for dynamic endpoint CRUD operations
+- Test coexistence with generated endpoints

.planning/ROADMAP.md

@@ -0,0 +1,160 @@
+# ROADMAP.md
+
+## Milestone 1: Dynamic DAS Core
+
+### Phase 1: Dynamic Metadata Service
+**Goal:** Load and cache etrx_* projection/entity/field metadata at runtime, providing a query API for other components.
+
+**Requirements covered:** FR-1, FR-8
+
+**Plans:** 3 plans
+
+Plans:
+- [x] 01-01-PLAN.md -- Models, dependencies, and DynamicMetadataService interface
+- [x] 01-02-PLAN.md -- Cache config and DynamicMetadataServiceImpl implementation
+- [x] 01-03-PLAN.md -- Unit tests for DynamicMetadataService
+
+**Deliverables:**
+- `DynamicMetadataService` that reads `etrx_projection`, `etrx_projection_entity`, `etrx_entity_field` from DB
+- In-memory cache with `ProjectionMetadata`, `EntityFieldMetadata` models
+- API: `getProjection(name)`, `getProjectionEntity(projectionName, entityName)`, `getFields(projectionEntityId)`
+- Support all field mapping types (DM, JM, CV, JP)
+- Cache invalidation method (manual trigger)
+- Unit tests with mock repositories
+
+**Success criteria:**
+- Can load all existing projections from DB at startup
+- Field metadata correctly represents all mapping types
+- Cache serves repeated lookups without DB queries
+- Tests cover: loading, caching, cache miss, invalid projection name
+
+---
+
+### Phase 2: Generic DTO Converter
+**Goal:** Convert between JPA entities and `Map<String, Object>` using runtime metadata from Phase 1.
+
+**Requirements covered:** FR-2, FR-6
+
+**Plans:** 3 plans
+
+Plans:
+- [x] 02-01-PLAN.md -- Foundation: strategy interface, PropertyAccessor, ConversionContext, simple strategies (DM, CV, CM)
+- [x] 02-02-PLAN.md -- Complex strategies (EM, JM, JP) and DynamicDTOConverter orchestrator
+- [x] 02-03-PLAN.md -- Unit tests for converter and strategies
+
+**Deliverables:**
+- `DynamicDTOConverter` implementing bidirectional conversion
+- Entity -> Map (read): iterate fields from metadata, extract values via reflection/property access
+- Map -> Entity (write): iterate fields from metadata, set values on entity via reflection/property access
+- Type coercion handlers (String, Number, Date, Boolean, reference entities)
+- Null safety matching `MappingUtils.handleBaseObject()` behavior
+- Support for related entity resolution (lookup by ID for foreign keys)
+- Audit field integration via `AuditServiceInterceptor`
+- Unit tests with real JPA entities and mock metadata
+
+**Success criteria:**
+- Can convert any JPA entity to Map using its projection metadata
+- Can populate any JPA entity from Map with correct types
+- Related entities resolved by ID from database
+- Null values handled consistently with generated converters
+- Tests cover: simple fields, references, nulls, type coercion, dates
+
+---
+
+### Phase 3: Generic Repository Layer
+**Goal:** Dynamic repository using EntityManager directly for CRUD + pagination + batch, with exact transaction orchestration matching generated repos.
+
+**Requirements covered:** FR-4, FR-5, FR-7
+
+**Plans:** 2 plans
+
+Plans:
+- [x] 03-01-PLAN.md -- EntityClassResolver, DynamicRepositoryException, and DynamicRepository with full CRUD/batch/pagination
+- [x] 03-02-PLAN.md -- Unit tests for EntityClassResolver and DynamicRepository
+
+**Deliverables:**
+- `EntityClassResolver` resolving entity classes via Hibernate metamodel at startup
+- `DynamicRepository` with findById, findAll (pagination + filtering), save (upsert), update, saveBatch
+- Transaction management via `RestCallTransactionHandler` (manual begin/commit for writes)
+- External ID integration (`ExternalIdService.add()`, `flush()` called twice per save)
+- Jakarta Validator integration with "id" property skip
+- CriteriaBuilder-based dynamic field filtering on DIRECT_MAPPING fields
+- Unit tests with mocked dependencies verifying exact order of operations
+
+**Success criteria:**
+- CRUD operations work end-to-end with dynamic conversion
+- Transaction boundaries match generated repository behavior exactly
+- External IDs registered after merge, flushed twice per save
+- Validation errors returned for missing mandatory fields (skipping "id")
+- Tests cover: create, read, update, list, upsert, validation failure, batch
+
+---
+
+### Phase 4: Generic REST Controller & Endpoint Registration
+**Goal:** Single REST controller that dynamically serves all projections, with endpoint registration matching existing URL patterns.
+
+**Requirements covered:** FR-3, NFR-1, NFR-2, NFR-3, NFR-4
+
+**Plans:** 3 plans
+
+Plans:
+- [x] 04-01-PLAN.md -- ExternalIdTranslationService and DynamicEndpointRegistry supporting services
+- [x] 04-02-PLAN.md -- DynamicRestController with GET/POST/PUT endpoints, json_path, batch support
+- [x] 04-03-PLAN.md -- Unit tests for controller, translation service, and endpoint registry
+
+**Deliverables:**
+- `DynamicRestController` handling `/{projectionName}/{entityExternalName}/**` routes
+- GET `/` - list with pagination (delegates to DynamicRepository.findAll)
+- GET `/{id}` - get by ID (delegates to DynamicRepository.findById)
+- POST `/` - create entity/entities, support `json_path` parameter
+- PUT `/{id}` - update entity
+- JSON response format compatible with existing generated DTOs
+- Request routing: resolve projection + entity from URL path
+- Batch POST support (array of entities)
+- Integration with existing JWT authentication (no changes to Edge)
+- Logging of dynamic endpoint access
+- OpenAPI documentation via SpringDoc annotations
+- Unit tests for all components
+
+**Success criteria:**
+- Dynamic endpoints accessible at same URL patterns as generated ones
+- Response JSON structure identical to generated DTO output
+- JWT authentication works without changes
+- Pagination, sorting work correctly
+- Batch POST creates multiple entities
+- Tests cover: GET list, GET by ID, POST single, POST batch, PUT, 404, validation errors
+
+---
+
+### Phase 5: Coexistence & Migration Support
+**Goal:** Ensure dynamic and generated endpoints coexist, provide toggle mechanism, and validate equivalence.
+
+**Requirements covered:** NFR-2, NFR-5
+
+**Deliverables:**
+- Configuration property to enable/disable dynamic endpoints per projection
+- Fallback: if dynamic endpoint fails, log and optionally delegate to generated
+- Comparison tool/test: call both dynamic and generated endpoints, diff responses
+- Documentation: how to migrate a projection from generated to dynamic
+- Performance benchmark: dynamic vs generated endpoint response times
+- Integration test suite validating feature parity
+
+**Success criteria:**
+- Both dynamic and generated endpoints serve simultaneously without conflicts
+- Configuration toggles work correctly
+- Response comparison shows identical output for same inputs
+- Performance within 20% of generated endpoints
+- Migration path documented and tested for at least one projection
+
+---
+
+## Phase Dependencies
+```
+Phase 1 (Metadata)
+    -> Phase 2 (Converter)
+        -> Phase 3 (Repository)
+            -> Phase 4 (Controller)
+                -> Phase 5 (Coexistence)
+```
+
+All phases are sequential - each builds on the previous.