Add cuvs-bench-elastic: HTTP backend for Elasticsearch GPU vector search

[afourniernv]

Introduce cuvs-bench-elastic as an optional plugin for cuvs-bench that provides an Elasticsearch backend. The backend communicates with Elasticsearch via HTTP and supports HNSW indexing with optional GPU acceleration when using the Elasticsearch GPU image (cuVS-accelerated vector search).

• Add cuvs_bench_elastic package with backend and config loader entry points
• Extend cuvs_bench registry and search spaces for pluggable backends
• Add elastic and integration optional dependencies to cuvs-bench
• Add modularization tests and integration test scaffolding (disabled until CI has ES GPU image, cuVS libs, and GPU runner)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cjnolet]

Just want to confirm- are there any other protocols to interact with Elasticsearch or is HTTP/Rest pretty much it? Mostly asking to make sure it's the best (fastest) way to comunicate so we aren't adding additional overheads in the mix.

[afourniernv]

Confirmed. For Elasticsearch the supported client path here is the official HTTP/REST interface; the backend does not add another protocol layer on top of that. We re-ran the backend live against the Brev deployment after the recent fixes. Leaving this open since it is more an interface tradeoff than a code bug.

[cjnolet]

We should decouple the index type from the backend type. User should be able to specify the backend and the algorithm they want to run in that backend (or maybe the two are coupled, but the backend should not assume the algorithm).

So for example, what if the index type is not supported? What happens when a user wants to test diskbbq against hnsw/cagra? We should decouple these from the beginning, ideally with separate functions for each (for e.g. _parse_cagra_params(), _parse_hnsw_params())

[cjnolet]

It's also important that we throw a proper (descriptive) error when the backend doesn't support an index type. For example, if someone passes in "SCaNN" to the Elasticsearch backend, they should get a message to the effect of "Received params for SCaNN index type, but Elasticsearch backend does not support this index type. Please check configuration."

[afourniernv]

Partially addressed. The backend now validates supported algorithm names, index types, and similarities up front with descriptive errors instead of failing later in Elasticsearch. It still models the benchmark family as elastic_* plus build-param type, so this is better than before but not a fully generic backend/index abstraction yet.

[cjnolet]

It's great to see improved test coverage!

[cjnolet]

@afourniernv this is a really big PR so I'm reviewing in stages. Thank you for bearing with me. I finally had some time to look through the implementation for comprehensively. Mostly minor, but important things.

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

Release Notes

• New Features

  • Added Elasticsearch GPU backend support enabling index creation and kNN search operations with HNSW algorithms.
  • Implemented plugin discovery system for dynamic backend registration and extensibility.

• Tests

  • Added comprehensive test suite for backend modularization, optional dependencies, and plugin discovery mechanisms.

Walkthrough

Adds a complete Elasticsearch GPU HNSW backend to cuvs-bench with pluggable entry-point discovery, centralized recall computation, configuration loading, search-space parameters, and comprehensive tests. Also modernizes OpenSearch array initialization and extends the registry to support optional backend dependencies with installation hints.

Changes

Elasticsearch GPU Backend with Plugin Discovery

Layer / File(s)	Summary
Plugin Discovery & Registry Infrastructure python/cuvs_bench/cuvs_bench/backends/registry.py, python/cuvs_bench/pyproject.toml	Adds entry-point-based plugin loading for optional backends and config loaders, with helper functions to generate pip install hints and rewrite ImportErrors when optional dependencies are missing. Wires cuvs_bench.backends.elastic and cuvs_bench.config_loaders.elastic entry points to the elasticsearch:register target; registers new elastic optional dependency group.
Centralized Recall Computation python/cuvs_bench/cuvs_bench/orchestrator/orchestrator.py	Introduces _should_compute_recall() helper to centralize recall-computation gating logic based on result.success, presence of neighbors, and an authoritative-recall metadata flag. Unifies recall logic in both sweep and tune modes.
Elasticsearch Backend: Index Build & Search python/cuvs_bench/cuvs_bench/backends/elasticsearch.py	Implements ElasticBackend with lazy Elasticsearch client creation, preflight network checks, index creation/deletion/reuse logic, chunked bulk vector ingestion, multi-parameter kNN search with per-parameter recall computation, and latency percentile aggregation.
Elasticsearch Configuration Loader python/cuvs_bench/cuvs_bench/backends/elasticsearch.py	Implements ElasticConfigLoader to discover algorithm groups from YAML configs, resolve allowed algorithms/groups, and expand parameter combinations into BenchmarkConfig objects with backend connection settings. Provides register() and convenience APIs run_build, run_search, run_benchmark.
Elasticsearch Parameter Definitions python/cuvs_bench/cuvs_bench/backends/search_spaces.py, python/cuvs_bench/cuvs_bench/config/algos/elastic.yaml	Defines four Elasticsearch HNSW variants in search spaces (elastic_hnsw, elastic_int8_hnsw, elastic_int4_hnsw, elastic_bbq_hnsw) and complete benchmark configuration with base, quantized, and test parameter groups.
OpenSearch Array Initialization Fix python/cuvs_bench/cuvs_bench/backends/opensearch.py	Replaces np.zeros() with np.empty() for zero-sized neighbor/distance arrays in failure and dry-run paths.
Test Infrastructure & Configuration python/cuvs_bench/cuvs_bench/tests/conftest.py	Adds pytest configuration to conditionally register Elasticsearch backend support at test startup.
Comprehensive Modularization & Integration Tests python/cuvs_bench/cuvs_bench/tests/test_modularization.py	Smoke tests for backend registration and error messaging, mocked plugin loader validation, conditional Elasticsearch integration tests (build/search, network preflight, index reuse, lazy-loading, unsupported-type rejection), and utility function validation.
Build System Configuration dependencies.yaml	Adds py_elastic_cuvs_bench build target and bench_elastic dependency group to wire the Elasticsearch backend into the build system.

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• rapidsai/cuvs#2012: Modifies recall computation logic in orchestrator.py with different gating criteria for when to recompute recall from backend results.

Suggested reviewers

• cjnolet
• bdice
• jnke2016

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately and specifically describes the main change: adding an Elasticsearch HTTP backend for cuvs-bench with GPU vector search support.
Description check	✅ Passed	The description clearly relates to the changeset, explaining the Elasticsearch backend addition, entry points, registry extensions, optional dependencies, and testing scaffolding.
Docstring Coverage	✅ Passed	Docstring coverage is 87.50% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@dependencies.yaml`:
- Around line 596-600: The pyproject extra for "elastic" is drifting because
dependencies.yaml defines bench_elastic -> cuvs-bench-elastic>=26.4.0 while
python/cuvs_bench/pyproject.toml currently lists elastic =
["elasticsearch>=8.0"]; update the source-of-truth to keep them in sync by
changing dependencies.yaml's bench_elastic entry to include the same package(s)
as the pyproject extra (or update python/cuvs_bench/pyproject.toml to match
dependencies.yaml) so that project.optional-dependencies.elastic (via
py_elastic_cuvs_bench / bench_elastic) consistently maps to the intended package
list and version constraint.

In `@python/cuvs_bench/cuvs_bench/backends/elasticsearch.py`:
- Around line 256-269: The current checks return a BuildResult immediately when
dataset.training_vectors (and similarly dataset.query_vectors) are zero-length,
but never attempt to load file-backed data from dataset.base_file /
dataset.query_file; update the logic in the Elasticsearch backend where
dataset.training_vectors is checked (and the analogous block for query_vectors)
to: if vectors.size == 0 and dataset.base_file (or dataset.query_file) is set,
attempt to load the vectors from that file (e.g., via the project’s dataset
loading utility or numpy.load) and re-check size; only return the BuildResult
error (preserving algorithm=self.algo and other fields) if no file is present or
loading still yields an empty array. Ensure you reference
dataset.training_vectors, dataset.base_file, dataset.query_vectors,
dataset.query_file, and BuildResult when making the change so the file-backed
path is executed before failing.

In `@python/cuvs_bench/cuvs_bench/backends/registry.py`:
- Around line 415-440: The function _try_load_plugin currently returns after
loading the first matching entry point, which causes it to short-circuit and
skip checking the other registry group (_BACKENDS_GROUP vs
_CONFIG_LOADERS_GROUP) for the same plugin name; change the control flow so that
after a successful ep.load()() you do not return from _try_load_plugin but
instead continue scanning remaining groups/entry-points (i.e., remove/relocate
the unconditional return and only exit once all groups/entry-points have been
iterated or explicitly determine both backend and config-loader are loaded),
ensuring get_config_loader() can find same-named plugins that split
registrations across groups.

In `@python/cuvs_bench/cuvs_bench/tests/conftest.py`:
- Around line 19-23: pytest_configure currently only attempts "from
cuvs_bench_elastic import register" and swallows ImportError, which prevents the
in-tree backend's register() from being used; update pytest_configure in
conftest.py to try the external import first, and if that raises ImportError,
import the in-tree backend's register (e.g. from
cuvs_bench.backends.elasticsearch import register) as a fallback, and ensure any
remaining ImportError is not silently suppressed (log or re-raise) so
registration failures are visible; reference the pytest_configure function and
the register() symbol when making the change.

In `@python/cuvs_bench/cuvs_bench/tests/test_modularization.py`:
- Around line 35-1002: The tests use class-based pytest suites
(TestModularizationSmoke, TestPluginLoaderMocked, TestElasticWithExtraInstalled,
TestElasticHelpers) which diverges from the project's preferred standalone test
function style; refactor each test method (e.g., test_cpp_gbench_available,
test_import_error_with_elasticsearch_message_raises_helpful_error,
test_elastic_config_loader_tune_mode_returns_single_config,
test_distance_to_similarity, etc.) into top-level test_* functions, move any
per-class fixtures like _ensure_elastic_registered into module-level fixtures or
helper functions, convert helper methods such as _make_mock_register into plain
module helpers, and ensure cleanup/patching logic (registry.unregister,
unregister_config_loader, patch context managers) is preserved in the new
functions so behavior remains identical.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0a62f800-07f3-46e8-9700-760e774b2c2a

📥 Commits

Reviewing files that changed from the base of the PR and between 6672103 and 680c2ab.

📒 Files selected for processing (10)

• dependencies.yaml
• python/cuvs_bench/cuvs_bench/backends/elasticsearch.py
• python/cuvs_bench/cuvs_bench/backends/opensearch.py
• python/cuvs_bench/cuvs_bench/backends/registry.py
• python/cuvs_bench/cuvs_bench/backends/search_spaces.py
• python/cuvs_bench/cuvs_bench/config/algos/elastic.yaml
• python/cuvs_bench/cuvs_bench/orchestrator/orchestrator.py
• python/cuvs_bench/cuvs_bench/tests/conftest.py
• python/cuvs_bench/cuvs_bench/tests/test_modularization.py
• python/cuvs_bench/pyproject.toml

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== pyproject elastic extra ==="
sed -n '41,56p' python/cuvs_bench/pyproject.toml

echo
echo "=== dependencies.yaml elastic source-of-truth ==="
rg -n "py_elastic_cuvs_bench|bench_elastic|cuvs-bench-elastic|elasticsearch>=8.0" dependencies.yaml -A4 -B3

Repository: rapidsai/cuvs

Length of output: 1500

---

Fix elastic extra source-of-truth drift between dependencies.yaml and generated pyproject.toml

dependencies.yaml maps project.optional-dependencies.elastic (via py_elastic_cuvs_bench → bench_elastic) to cuvs-bench-elastic>=26.4.0, but python/cuvs_bench/pyproject.toml currently declares elastic = ["elasticsearch>=8.0"]—regenerating from dependencies.yaml will overwrite the committed pyproject extra and create install-hint/metadata drift.

Suggested fix

   bench_elastic:
     common:
       - output_types: [conda, pyproject, requirements]
         packages:
-          - cuvs-bench-elastic>=26.4.0
+          - elasticsearch>=8.0

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@dependencies.yaml` around lines 596 - 600, The pyproject extra for "elastic"
is drifting because dependencies.yaml defines bench_elastic ->
cuvs-bench-elastic>=26.4.0 while python/cuvs_bench/pyproject.toml currently
lists elastic = ["elasticsearch>=8.0"]; update the source-of-truth to keep them
in sync by changing dependencies.yaml's bench_elastic entry to include the same
package(s) as the pyproject extra (or update python/cuvs_bench/pyproject.toml to
match dependencies.yaml) so that project.optional-dependencies.elastic (via
py_elastic_cuvs_bench / bench_elastic) consistently maps to the intended package
list and version constraint.

Source: Coding guidelines

[coderabbitai]

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

HIGH: advertised elastic_* algorithms are not all selectable through the loader.

Line 110-114 exposes four supported algorithms, but Line 699-700 only accepts exact YAML name matches; with elastic.yaml naming only elastic_hnsw, requests like algorithms="elastic_int8_hnsw" are rejected.
Impact: tune/sweep paths cannot run several public algorithm names that are already exposed in search spaces and validation.

Have you considered normalizing elastic_int8_hnsw / elastic_int4_hnsw / elastic_bbq_hnsw to the elastic_hnsw config with an injected build.type, or adding dedicated YAML algorithm entries for each name?

As per coding guidelines, “Prevent API breaking changes to public Python interfaces, including removing or renaming public methods/attributes without proper deprecation (minimum one release cycle).”

Also applies to: 699-700, 719-726

Source: Coding guidelines

[coderabbitai]

⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

CRITICAL: file-backed dataset path is declared but never executed.

Line 256 and Line 441 fail immediately on empty in-memory arrays, but the code never loads from dataset.base_file / dataset.query_file even though the error text says it does.
Impact: with BenchmarkOrchestrator._create_dataset (python/cuvs_bench/cuvs_bench/orchestrator/orchestrator.py Line 633-634), normal backend runs fail before index/search work starts.

Proposed fix

-        vectors = dataset.training_vectors
-        if vectors.size == 0:
+        vectors = dataset.training_vectors
+        if vectors.size == 0 and dataset.base_file:
+            vectors = load_vectors(dataset.base_file)
+        if vectors.size == 0:
             return BuildResult(
                 index_path="",
                 build_time_seconds=0.0,
                 index_size_bytes=0,
@@
-        query_vectors = dataset.query_vectors
-        if query_vectors.size == 0:
+        query_vectors = dataset.query_vectors
+        if query_vectors.size == 0 and dataset.query_file:
+            query_vectors = load_vectors(dataset.query_file)
+        if query_vectors.size == 0:
             return SearchResult(
                 neighbors=np.empty((0, k), dtype=np.int64),
                 distances=np.empty((0, k), dtype=np.float32),

As per coding guidelines, “Ensure missing validation does not cause crashes on invalid input through proper size/type checks” and “Handle edge cases in memory resource code including zero-size arrays and alignment requirements.”

Also applies to: 441-455

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/cuvs_bench/cuvs_bench/backends/elasticsearch.py` around lines 256 -
269, The current checks return a BuildResult immediately when
dataset.training_vectors (and similarly dataset.query_vectors) are zero-length,
but never attempt to load file-backed data from dataset.base_file /
dataset.query_file; update the logic in the Elasticsearch backend where
dataset.training_vectors is checked (and the analogous block for query_vectors)
to: if vectors.size == 0 and dataset.base_file (or dataset.query_file) is set,
attempt to load the vectors from that file (e.g., via the project’s dataset
loading utility or numpy.load) and re-check size; only return the BuildResult
error (preserving algorithm=self.algo and other fields) if no file is present or
loading still yields an empty array. Ensure you reference
dataset.training_vectors, dataset.base_file, dataset.query_vectors,
dataset.query_file, and BuildResult when making the change so the file-backed
path is executed before failing.

Source: Coding guidelines

[coderabbitai]

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

HIGH: _try_load_plugin() can short-circuit config-loader discovery.

At Line 440, the function returns after the first successful load in _BACKENDS_GROUP, so get_config_loader() (Line 533) may never check _CONFIG_LOADERS_GROUP for same-named plugins that split registrations.

Suggested fix

-def _try_load_plugin(name: str) -> None:
+def _try_load_plugin(
+    name: str,
+    groups: tuple[str, ...] = (_BACKENDS_GROUP, _CONFIG_LOADERS_GROUP),
+) -> None:
@@
-    for group in (_BACKENDS_GROUP, _CONFIG_LOADERS_GROUP):
+    for group in groups:
@@
-            return  # Plugin loaded successfully
+            return  # Plugin loaded successfully
@@
 def get_config_loader(name: str) -> Type:
@@
     if name not in _CONFIG_LOADER_REGISTRY:
-        _try_load_plugin(name)
+        _try_load_plugin(
+            name,
+            groups=(_CONFIG_LOADERS_GROUP, _BACKENDS_GROUP),
+        )

As per coding guidelines, integration errors are HIGH-priority review targets.

Also applies to: 532-534

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/cuvs_bench/cuvs_bench/backends/registry.py` around lines 415 - 440,
The function _try_load_plugin currently returns after loading the first matching
entry point, which causes it to short-circuit and skip checking the other
registry group (_BACKENDS_GROUP vs _CONFIG_LOADERS_GROUP) for the same plugin
name; change the control flow so that after a successful ep.load()() you do not
return from _try_load_plugin but instead continue scanning remaining
groups/entry-points (i.e., remove/relocate the unconditional return and only
exit once all groups/entry-points have been iterated or explicitly determine
both backend and config-loader are loaded), ensuring get_config_loader() can
find same-named plugins that split registrations across groups.

Source: Coding guidelines

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== conftest registration block ==="
sed -n '8,24p' python/cuvs_bench/cuvs_bench/tests/conftest.py

echo
echo "=== in-tree elastic register symbol ==="
rg -n "def register\\(" python/cuvs_bench/cuvs_bench/backends/elasticsearch.py -C2 || true

echo
echo "=== legacy cuvs_bench_elastic module paths in repo ==="
fd -HI "cuvs_bench_elastic" python || true

Repository: rapidsai/cuvs

Length of output: 809

---

conftest elastic plugin registration ignores the in-tree backend and silently drops failures

python/cuvs_bench/cuvs_bench/tests/conftest.py’s pytest_configure only tries from cuvs_bench_elastic import register and swallows ImportError; the in-tree entry point python/cuvs_bench/cuvs_bench/backends/elasticsearch.py defines register(), and the legacy cuvs_bench_elastic module doesn’t exist in this repo—so source-tree runs may not register the elastic backend as the docstring claims.

Suggested fix

-    try:
-        from cuvs_bench_elastic import register
-        register()
-    except ImportError:
-        pass
+    try:
+        from cuvs_bench.backends.elasticsearch import register
+    except ImportError:
+        try:
+            from cuvs_bench_elastic import register  # legacy fallback
+        except ImportError:
+            return
+    register()

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/cuvs_bench/cuvs_bench/tests/conftest.py` around lines 19 - 23,
pytest_configure currently only attempts "from cuvs_bench_elastic import
register" and swallows ImportError, which prevents the in-tree backend's
register() from being used; update pytest_configure in conftest.py to try the
external import first, and if that raises ImportError, import the in-tree
backend's register (e.g. from cuvs_bench.backends.elasticsearch import register)
as a fallback, and ensure any remaining ImportError is not silently suppressed
(log or re-raise) so registration failures are visible; reference the
pytest_configure function and the register() symbol when making the change.

Source: Coding guidelines

[coderabbitai]

⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

HIGH: Test suite structure diverges from cuVS test contract (class-based tests instead of standalone functions).

Have you considered converting these class-based pytest suites into standalone test_* functions? This pattern can block consistency with cuVS test conventions and future test maintenance in this module.

As per coding guidelines, “Using test classes instead of standalone functions (cuVS prefers test_foo_bar() functions over class TestFoo)” should be treated as a HIGH test-quality issue.

🧰 Tools

🪛 Ruff (0.15.15)

[warning] 422-422: Unpacked variable dataset_config is never used

Prefix it with an underscore or any other dummy variable pattern

(RUF059)

---

[warning] 443-443: Unpacked variable dataset_config is never used

Prefix it with an underscore or any other dummy variable pattern

(RUF059)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/cuvs_bench/cuvs_bench/tests/test_modularization.py` around lines 35 -
1002, The tests use class-based pytest suites (TestModularizationSmoke,
TestPluginLoaderMocked, TestElasticWithExtraInstalled, TestElasticHelpers) which
diverges from the project's preferred standalone test function style; refactor
each test method (e.g., test_cpp_gbench_available,
test_import_error_with_elasticsearch_message_raises_helpful_error,
test_elastic_config_loader_tune_mode_returns_single_config,
test_distance_to_similarity, etc.) into top-level test_* functions, move any
per-class fixtures like _ensure_elastic_registered into module-level fixtures or
helper functions, convert helper methods such as _make_mock_register into plain
module helpers, and ensure cleanup/patching logic (registry.unregister,
unregister_config_loader, patch context managers) is preserved in the new
functions so behavior remains identical.

Source: Coding guidelines

[jnke2016]

One small cleanup: _load_fbin and _load_ibin are now thin wrappers around load_vectors and aren't called anywhere, so they can be removed.

[jnke2016]

Vector loading looks good now and matches the centralized approach. build() reads dataset.training_vectors and search() reads dataset.query_vectors and dataset.groundtruth_neighbors, so the framework handles loading transparently and the backend no longer parses files itself. This also picks up the shared format support (uint64 headers, .f16bin/.u8bin/.i8bin, subset_size) along

[jnke2016]

ElasticConfigLoader.load() can be removed. It only forwards to super().load(), so it adds nothing, the base ConfigLoader.load() already runs all the shared steps and calls into the backend-specific hooks _discover_algo_groups() and _build_benchmark_configs(), which is where the elastic logic lives. OpenSearch relies on this directly: it defines no load(), inherits the base one, and implements only those two hooks. Dropping it keeps elastic consistent with OpenSearch

[jnke2016]

Parameter expansion looks good. _discover_algo_groups() just returns the algo groups and _build_benchmark_configs() receives the already-expanded build_combos / search_combos from the base loader. The framework handles the expansion, same as OpenSearch.

[jnke2016]

Is the reason for connecting lazily through _get_client() inside build()/search(), rather than in initialize(), to avoid reaching a live server on a dry_run? The orchestrator always calls initialize(), even for a dry run, while build()/search() return early on dry_run before touching the client, so the lazy path seems to keep dry runs from needing a server.

[jnke2016]

This loop is redundant in the normal flow. build_params already copies idx.build_param (line 223), and the loader spreads that same dict into backend_config via **build_param (line 818), so every _BUILD_PARAM_KEY in self.config is already in build_params. The remaining entries in self.config are connection keys (host, port, …) that the loop's _BUILD_PARAM_KEYS filter skips. So it never adds anything and can be dropped, leaving idx.build_param as the single source.

The only case where the loop does something is the unusual one where indexes is empty (idx is None, so build_params starts as {}) — but that doesn't happen in the orchestrator path, since each config carries exactly one IndexConfig.

[jnke2016]

The module-level run_build / run_search / run_benchmark helpers duplicate the entry point already provided by BenchmarkOrchestrator and the CLI, they just call orchestrator.run_benchmark(...) as below:

run_build(...)

# Python API

BenchmarkOrchestrator(backend_type="elastic").run_benchmark(
    build=True, search=False,
    dataset=..., dataset_path=..., algorithms=..., host=..., port=..., force=force,
)

# CLI

python -m cuvs_bench.run --build \
    --dataset <name> --dataset-path <path> --algorithms <group> \
    --backend-config <elastic-config>

run_search(...)

# Python API

BenchmarkOrchestrator(backend_type="elastic").run_benchmark(
    build=False, search=True,
    dataset=..., dataset_path=..., algorithms=..., host=..., port=...,
)

# CLI

python -m cuvs_bench.run --search \
    --dataset <name> --dataset-path <path> --algorithms <group> \
    --backend-config <elastic-config>

run_benchmark(...)

# Python API

BenchmarkOrchestrator(backend_type="elastic").run_benchmark(
    build=build, search=search,
    dataset=..., dataset_path=..., algorithms=..., host=..., port=..., force=force,
)

# CLI

python -m cuvs_bench.run --build --search \
    --dataset <name> --dataset-path <path> --algorithms <group> \
    --backend-config <elastic-config>