feat(knowledge): add Azure AI Search backend

[harmke]

Overview

Adds Azure AI Search as a Knowledge Layer backend for AI-Q. This gives deployments a managed alternative to the existing LlamaIndex/ChromaDB and Foundational RAG backends while preserving the shared Knowledge API, NAT function registration, document summaries, and citation formatting.

The adapter generates embeddings client-side through the existing NVIDIA embedding integration and stores document chunks in one Azure AI Search index per logical collection.

This PR includes:

• azure_ai_search retriever and ingestor adapters with vector, hybrid, and opt-in semantic-ranked retrieval.
• API-key authentication through AZURE_SEARCH_API_KEY or Azure identity authentication through DefaultAzureCredential.
• Managed-identity setup guidance for the required Search Service Contributor and Search Index Data Contributor roles.
• Shared embedding defaults through AIQ_EMBED_BASE_URL and AIQ_EMBED_MODEL, with Azure index dimensions configured through AIQ_EMBED_DIM.
• Collision-safe physical index names and ownership/schema markers so AI-Q only lists or mutates indexes it owns.
• PDF, DOCX, TXT, and Markdown ingestion through LlamaIndex readers, including chunk metadata, page-aware citations, and optional summaries.
• Canonical UUID file IDs, asynchronous ingestion status, same-name replacement handling, batch-size enforcement, upload rollback, delete retries, eventual-consistency-aware validation, pagination, and TTL cleanup.
• Azure-specific installation, configuration, and operating documentation.
• Focused unit tests and adapter compliance coverage.

This PR does not provision Azure infrastructure. The Azure AI Search service, required permissions, and embedding endpoint must already exist.

Validation

All commands were run from the repository root unless noted.

$ uv run pytest tests/knowledge_layer_tests/test_azure_ai_search.py -q
..................................                                       [100%]
34 passed in 12.96s

$ uv run ruff check sources/knowledge_layer/src/register.py sources/knowledge_layer/src/azure_ai_search tests/knowledge_layer_tests/test_azure_ai_search.py tests/knowledge_layer_tests/run_adapter_compliance.py
All checks passed!

$ uv run ruff format --check sources/knowledge_layer/src/register.py sources/knowledge_layer/src/azure_ai_search tests/knowledge_layer_tests/test_azure_ai_search.py tests/knowledge_layer_tests/run_adapter_compliance.py
5 files already formatted

Adapter registration and factory compliance:

$ uv run python tests/knowledge_layer_tests/run_adapter_compliance.py --backend azure_ai_search --quick --config '{"endpoint":"https://example.search.windows.net","start_ttl_cleanup":false}'
Summary: 4/4 passed, 0 failed
All tests PASSED - Adapter is compliant!

Documentation build:

$ cd docs
$ uv run make html
build succeeded, 3 warnings.

The three Sphinx warnings are unresolved cross-reference warnings in unrelated existing pages (architecture/agents/sandbox.md and integration/agent-skills.md).

Live Azure service validation was also completed against a Basic-tier Azure AI Search service using NVIDIA-hosted nvidia/nv-embed-v1 embeddings (4096 dimensions). The full adapter compliance flow passed 14/14 under admin-key authentication, covering collection creation, TXT ingestion, asynchronous status polling, file listing, hybrid retrieval with semantic ranking, chunk normalization, file deletion, and collection cleanup. DefaultAzureCredential also selected a system-assigned managed identity in Azure Container Apps, and index creation/deletion, document upload/deletion, and querying all succeeded with the documented Search Service Contributor and Search Index Data Contributor roles.

• I ran the relevant local checks or explained why they are not applicable.
• I added or updated tests for behavior changes.
• I updated documentation for user-facing or contributor-facing changes.
• I confirmed this PR does not include secrets, credentials, or internal-only data.
• I certify this contribution under the Developer Certificate of Origin (DCO) and signed my commits with git commit -s or an equivalent sign-off.

Where should reviewers start?

Start with sources/knowledge_layer/src/azure_ai_search/adapter.py for index ownership, schema validation, ingestion, replacement, and retrieval behavior. Then review sources/knowledge_layer/src/register.py for NAT configuration and authentication defaults, followed by tests/knowledge_layer_tests/test_azure_ai_search.py for the expected lifecycle and failure handling.

The main design decision is that AI-Q creates and manages only namespaced indexes carrying its ownership marker; unmanaged or incompatible indexes are ignored rather than modified.

Related Issues

• None.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Caution

Review failed

An error occurred during the review process. Please try again later.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Caution

Failed to replace (edit) comment. This is likely due to insufficient permissions or the comment being deleted.

Error details

{}