feat(elasticsearch): support for composable index templates

[sgomezvillamor]

Fix(elasticsearch): Ingest both legacy and composable index templates

Problem:
The Elasticsearch/OpenSearch source connector previously only ingested legacy index templates (using the get_template() API). Modern OpenSearch (v2.17) and Elasticsearch (7.8+) also support composable index templates, which are accessed via a different API (get_index_template()) and have a distinct data structure. This led to incomplete index template ingestion, where only legacy templates were discovered, or none if only composable templates existed.

Solution:
This PR updates the Elasticsearch source to support both types of index templates:

1. Dual API Calls: The get_workunits_internal method now attempts to fetch both legacy and composable index templates.
2. Smart Parsing: The _extract_mcps method has been enhanced to correctly parse the metadata (mappings, settings, aliases) from both legacy templates (root-level fields) and composable templates (fields nested under template).

Impact:
This bug fix ensures that all index templates (both legacy and composable) are now correctly ingested from OpenSearch and Elasticsearch clusters, providing comprehensive metadata coverage. The changes are backward compatible with older clusters that only use legacy templates.

Related Issue:
CUS-6603

---

Slack Thread

 

[cursor]

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
_{Learn more about Cursor Agents}

[codecov]

Codecov Report

❌ Patch coverage is 3.33333% with 58 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...ion/src/datahub/ingestion/source/elastic_search.py	3.33%	58 Missing ⚠️

📢 Thoughts on this report? Let us know!