From 7a2ca750eb47d1d3bb7747665f20a0f328deaceb Mon Sep 17 00:00:00 2001 From: James Bourbeau Date: Fri, 10 Apr 2026 14:18:40 -0500 Subject: [PATCH 1/9] Add initial opensearch deployment Signed-off-by: James Bourbeau --- deploy/README.md | 220 +++++++++++++++++ deploy/bench/Dockerfile | 38 +++ deploy/bench/run.py | 213 +++++++++++++++++ deploy/docker-compose.yml | 155 ++++++++++++ deploy/opensearch/Dockerfile | 16 ++ deploy/opensearch/opensearch.yml | 21 ++ deploy/remote-index-build/Dockerfile | 8 + deploy/remote-index-build/run.py | 337 +++++++++++++++++++++++++++ 8 files changed, 1008 insertions(+) create mode 100644 deploy/README.md create mode 100644 deploy/bench/Dockerfile create mode 100644 deploy/bench/run.py create mode 100644 deploy/docker-compose.yml create mode 100644 deploy/opensearch/Dockerfile create mode 100644 deploy/opensearch/opensearch.yml create mode 100644 deploy/remote-index-build/Dockerfile create mode 100644 deploy/remote-index-build/run.py diff --git a/deploy/README.md b/deploy/README.md new file mode 100644 index 0000000000..f83bd82e48 --- /dev/null +++ b/deploy/README.md @@ -0,0 +1,220 @@ +# OpenSearch GPU Remote Index Build Benchmark + +Docker Compose benchmark of [OpenSearch's GPU-accelerated remote index build](https://docs.opensearch.org/latest/vector-search/remote-index-build/) feature using [cuvs-bench](https://github.com/jrbourbeau/cuvs/tree/main/python/cuvs_bench). Spins up all required services, builds a kNN index on a GPU, and runs a cuvs-bench search benchmark sweep. + +## How it works + +OpenSearch's kNN plugin can offload Faiss HNSW index construction to a dedicated GPU service. Rather than building the index in-process on the OpenSearch node, the workflow is: + +``` +OpenSearch flushes a segment + → uploads raw vectors + doc-IDs to S3 (MinIO in this demo) + → POSTs /_build to the remote-index-builder service + → service downloads vectors from MinIO + → builds Faiss HNSW index on GPU + → uploads finished index back to MinIO + → OpenSearch downloads the GPU-built index and merges it into the shard +``` + +## Services + +| Service | Image | Purpose | +|---|---|---| +| `minio` | `minio/minio` | S3-compatible object store — staging area for vectors and built indexes | +| `minio-init` | `minio/mc` | One-shot: creates the `opensearch-vectors` bucket | +| `opensearch` | custom build of `opensearchproject/opensearch` | OpenSearch node with kNN plugin and `repository-s3` plugin | +| `remote-index-builder` | `opensearchproject/remote-vector-index-builder:api-latest` | FastAPI service that builds Faiss indexes on the GPU | +| `bench` | custom Python | Registers repo + cluster settings, runs cuvs-bench build/search benchmark | + +## Requirements + +- **NVIDIA GPU** with CUDA support +- **NVIDIA Container Toolkit** — [installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) +- **Docker Compose v2** +- **ANN benchmark dataset** in binary format (`.fbin`) — see [Dataset format](#dataset-format) + +## Usage + +Set the host kernel parameter required by OpenSearch (once per reboot): + +```bash +sudo sysctl -w vm.max_map_count=262144 +``` + +Set required environment variables: + +```bash +export DATASET_PATH=/path/to/ann-benchmark-datasets # directory containing dataset files +``` + +Optionally configure the benchmark: + +```bash +export DATASET=sift-128-euclidean # default +export BENCH_GROUPS=test # test | base | large (default: test) +export K=10 # number of neighbors (default: 10) +``` + +Download the dataset (one-time setup, skipped automatically if already present): + +```bash +docker compose build bench +docker compose run --rm --no-deps bench python -m cuvs_bench.get_dataset \ + --dataset ${DATASET:-sift-128-euclidean} \ + --dataset-path /data/datasets +``` + +Start all services: + +```bash +docker compose up --build +``` + +The `bench` container logs its progress through index build, GPU verification, and search. When complete you'll see a results table: + +``` +════════════════════════════════════════════════════════════ + Benchmark complete! +════════════════════════════════════════════════════════════ + + OpenSearch : http://opensearch:9200 + MinIO console : http://localhost:9001 (minioadmin / minioadmin) +``` + +To tear everything down: + +```bash +docker compose down -v +``` + +## What the bench script does + +1. Registers MinIO as an OpenSearch S3 snapshot repository +2. Applies cluster settings to enable remote index build and point OpenSearch at the builder service +3. Runs `cuvs-bench` build phase (handled entirely by the OpenSearch backend): + - Creates the kNN index and bulk-ingests dataset vectors + - Force-merges the index to trigger the GPU build + - Polls MinIO every 5 s for `.faiss` files confirming GPU build completion + - Records total build time (ingestion + GPU build) in the result +4. Runs `cuvs-bench` search phase and prints a recall/QPS/latency table + +## Dataset format + +cuvs-bench reads binary vector files with a simple header: + +``` +[4 bytes: n_rows as uint32] +[4 bytes: n_cols as uint32] +[n_rows × n_cols × itemsize bytes: vector data] +``` + +Supported extensions: `.fbin` (float32), `.f16bin` (float16), `.u8bin` (uint8), `.i8bin` (int8). + +`DATASET_PATH` should be a directory where each dataset lives in its own subdirectory named after the dataset, e.g.: + +``` +$DATASET_PATH/ + sift-128-euclidean/ + base.fbin + query.fbin + groundtruth.neighbors.ibin +``` + +## Key configuration + +**Cluster settings** (applied by `bench/run.py`): + +```json +{ + "persistent": { + "knn.remote_index_build.enabled": true, + "knn.remote_index_build.repository": "vector-repo", + "knn.remote_index_build.service.endpoint": "http://remote-index-builder:1025" + } +} +``` + +**Parameter groups** (`BENCH_GROUPS`): + +| Group | Build params | Search params | Use case | +|---|---|---|---| +| `test` | 1 combo (m=16, ef_construction=100) | ef_search: 50, 100 | Quick smoke test | +| `base` | 6 combos (m × ef_construction sweep) | ef_search: 50–512 | Standard benchmark | +| `large` | 2 combos (larger m, ef_construction) | ef_search: 100–1024 | High-recall benchmark | + +## GPU build verification + +The cuvs-bench OpenSearch backend polls MinIO every 5 seconds for `.faiss` files under `s3://opensearch-vectors/knn-indexes/`. The remote-index-builder is the only component that writes `.faiss` files back to the bucket, so their presence is definitive proof the GPU build completed. + +The build raises a `TimeoutError` (causing the `bench` container to exit with code 1) if the expected number of `.faiss` files does not appear within 600 seconds. + +## Running without a GPU + +This is intentionally not supported. The `bench` container will exit 1 if `.faiss` files do not appear in MinIO within 600 s. If you want to experiment without hardware, remove the `deploy.resources.reservations` block from `remote-index-builder` in `docker-compose.yml` and be aware the benchmark will fail at the verification step. + +## Running tests + +The cuvs-bench OpenSearch backend has three tiers of tests. All run inside the `bench` container so no local Python environment is needed. + +**Build the bench image first** (or after any code changes): + +```bash +docker compose build --no-cache bench +``` + +### Unit tests (no server required) + +```bash +docker compose run --rm --no-deps bench \ + pytest /opt/cuvs/python/cuvs_bench/cuvs_bench/tests/test_opensearch.py -v +``` + +### Integration tests (live OpenSearch node) + +Requires a running OpenSearch node. Start it with: + +```bash +docker compose up -d --wait opensearch +``` + +Then run: + +```bash +docker compose run --rm --no-deps \ + -e OPENSEARCH_URL=http://opensearch:9200 \ + bench \ + pytest /opt/cuvs/python/cuvs_bench/cuvs_bench/tests/test_opensearch.py -v -m integration +``` + +### Remote index build integration tests (full GPU stack) + +Requires the full stack (OpenSearch, MinIO, and the remote index builder). Start all services with: + +```bash +docker compose up -d --wait minio minio-init opensearch remote-index-builder +``` + +Then run: + +```bash +docker compose run --rm --no-deps \ + -e OPENSEARCH_URL=http://opensearch:9200 \ + -e BUILDER_URL=http://remote-index-builder:1025 \ + -e S3_ENDPOINT=http://minio:9000 \ + -e S3_BUCKET=opensearch-vectors \ + -e S3_ACCESS_KEY=minioadmin \ + -e S3_SECRET_KEY=minioadmin \ + bench \ + pytest /opt/cuvs/python/cuvs_bench/cuvs_bench/tests/test_opensearch.py -v -m integration +``` + +This runs all integration tests including `TestOpenSearchRemoteIndexBuildIntegration`, which verifies the full GPU build flow end-to-end. + +## Ports + +| Port | Service | +|---|---| +| `9200` | OpenSearch REST API | +| `9000` | MinIO S3 API | +| `9001` | MinIO web console | +| `1025` | Remote index builder API | diff --git a/deploy/bench/Dockerfile b/deploy/bench/Dockerfile new file mode 100644 index 0000000000..db3403b0e8 --- /dev/null +++ b/deploy/bench/Dockerfile @@ -0,0 +1,38 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +FROM python:3.11-slim +WORKDIR /app + +RUN apt-get update && apt-get install -y --no-install-recommends git && rm -rf /var/lib/apt/lists/* + +# Sparse-clone just the cuvs_bench Python package. +# Installing via pip is not possible without CUDA (rapids-build-backend requires +# nvcc, and cuvs_bench declares a hard dep on the `cuvs` CUDA package). +# The opensearch backend is pure Python and needs neither, so we add the +# package directly to PYTHONPATH and install only the actual runtime deps. +RUN git clone --depth=1 --filter=blob:none --sparse --branch cuvs-bench-opensearch https://github.com/jrbourbeau/cuvs.git /opt/cuvs \ + && cd /opt/cuvs \ + && git sparse-checkout set python/cuvs_bench + +ENV PYTHONPATH=/opt/cuvs/python/cuvs_bench + +# Runtime dependencies from cuvs_bench/pyproject.toml (excluding `cuvs` itself) +# plus extras needed by the opensearch backend and this benchmark script. +RUN pip install --no-cache-dir \ + boto3 \ + pytest \ + botocore \ + click \ + h5py \ + matplotlib \ + "numpy<2" \ + "opensearch-py>=2.4.0,<3.0.0" \ + pandas \ + pyyaml \ + requests \ + scikit-learn \ + scipy + +COPY run.py . +CMD ["python", "-u", "run.py"] diff --git a/deploy/bench/run.py b/deploy/bench/run.py new file mode 100644 index 0000000000..645988d94c --- /dev/null +++ b/deploy/bench/run.py @@ -0,0 +1,213 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +""" +OpenSearch GPU Remote Index Build Benchmark +=========================================== +Steps: + 1. Register MinIO S3 snapshot repository with OpenSearch + 2. Configure cluster settings for GPU remote index build + 3. Build kNN index via cuvs-bench: + a. Bulk-ingest dataset vectors + b. Trigger force-merge to kick off the GPU build + c. Poll MinIO for .faiss files confirming GPU build completion + 4. Run cuvs-bench search benchmarks and print results +""" + +import os +import sys + +import requests +from cuvs_bench.orchestrator import BenchmarkOrchestrator +from cuvs_bench.backends.base import BuildResult, SearchResult + + +OPENSEARCH_URL = os.environ.get("OPENSEARCH_URL", "http://opensearch:9200") +OPENSEARCH_HOST = os.environ.get("OPENSEARCH_HOST", "opensearch") +OPENSEARCH_PORT = int(os.environ.get("OPENSEARCH_PORT", "9200")) +BUILDER_URL = os.environ.get("BUILDER_URL", "http://remote-index-builder:1025") +MINIO_URL = os.environ.get("MINIO_URL", "http://minio:9000") + +DATASET = os.environ.get("DATASET", "sift-128-euclidean") +DATASET_PATH = os.environ.get("DATASET_PATH", "/data/datasets") +BENCH_GROUPS = os.environ.get("BENCH_GROUPS", "test") +K = int(os.environ.get("K", "10")) + +BUCKET = "opensearch-vectors" +REPO_NAME = "vector-repo" + +session = requests.Session() +session.headers.update({"Content-Type": "application/json"}) + + +# ── helpers ─────────────────────────────────────────────────────────────────── + + +def banner(msg: str) -> None: + print(f"\n{'─' * 60}\n {msg}\n{'─' * 60}") + + +# ── OpenSearch setup ────────────────────────────────────────────────────────── + + +def register_repository() -> None: + banner(f"Registering S3 repository '{REPO_NAME}' (backed by MinIO)") + r = session.put( + f"{OPENSEARCH_URL}/_snapshot/{REPO_NAME}", + json={ + "type": "s3", + "settings": { + # endpoint / protocol / path_style_access are client-level settings + # configured in opensearch/opensearch.yml, not per-repository settings. + "bucket": BUCKET, + "base_path": "knn-indexes", + }, + }, + ) + r.raise_for_status() + print(f" {r.json()}") + + +def configure_cluster() -> None: + banner("Enabling GPU remote index build (cluster settings)") + r = session.put( + f"{OPENSEARCH_URL}/_cluster/settings", + json={ + "persistent": { + "knn.remote_index_build.enabled": True, + "knn.remote_index_build.repository": REPO_NAME, + "knn.remote_index_build.service.endpoint": BUILDER_URL, + } + }, + ) + r.raise_for_status() + print(f" {r.json()}") + + +# ── results ─────────────────────────────────────────────────────────────────── + + +def _print_result_row( + params: dict, recall: float, qps: float, latency_ms: float +) -> None: + params_str = ", ".join(f"{k}={v}" for k, v in params.items()) + print( + f" {params_str:<40} {recall:<12.4f} {qps:>8.1f} {latency_ms:>12.2f}" + ) + + +def print_results(results: list) -> None: + banner("Benchmark Results") + search_results = [r for r in results if isinstance(r, SearchResult)] + if not search_results: + print(" No search results returned.") + return + + header = f" {'params':<40} {'recall@' + str(K):<12} {'QPS':>8} {'latency (ms)':>12}" + print(header) + print(" " + "─" * (len(header) - 2)) + for r in search_results: + per_param = (r.metadata or {}).get("per_search_param_results") + if per_param: + for entry in per_param: + _print_result_row( + entry["search_params"], + entry["recall"], + entry["queries_per_second"], + entry["search_time_ms"], + ) + else: + _print_result_row( + r.search_params[0] if r.search_params else {}, + r.recall, + r.queries_per_second, + r.search_time_ms, + ) + + +# ── entrypoint ──────────────────────────────────────────────────────────────── + + +def main() -> None: + print("\n" + "═" * 60) + print(" OpenSearch GPU Remote Index Build Benchmark") + print("═" * 60) + print(f" OpenSearch : {OPENSEARCH_URL}") + print(f" GPU builder: {BUILDER_URL}") + print(f" Dataset : {DATASET} (path: {DATASET_PATH})") + print(f" Groups : {BENCH_GROUPS} k={K}") + + register_repository() + configure_cluster() + + orchestrator = BenchmarkOrchestrator(backend_type="opensearch") + + # Shared kwargs for both build and search phases + bench_kwargs = dict( + dataset=DATASET, + dataset_path=DATASET_PATH, + algorithms="opensearch_faiss_hnsw", + groups=BENCH_GROUPS, + host=OPENSEARCH_HOST, + port=OPENSEARCH_PORT, + use_ssl=False, + verify_certs=False, + remote_index_build=True, + # S3/MinIO config for GPU build verification (used by the backend) + remote_build_s3_endpoint=MINIO_URL, + remote_build_s3_bucket=BUCKET, + remote_build_s3_prefix="knn-indexes/", + remote_build_s3_access_key="minioadmin", + remote_build_s3_secret_key="minioadmin", + ) + + # ── Build phase ─────────────────────────────────────────────────────────── + # The backend handles the full GPU build flow: ingest vectors → force merge + # → poll MinIO for .faiss files confirming GPU build completion. + banner("Building index (GPU remote build via cuvs-bench)") + build_results = orchestrator.run_benchmark( + build=True, + search=False, + force=True, + bulk_batch_size=500, + **bench_kwargs, + ) + + index_names = [ + r.index_path + for r in build_results + if isinstance(r, BuildResult) and r.success and r.index_path + ] + if not index_names: + print(" ERROR: no indexes were successfully built") + sys.exit(1) + build_times = { + r.index_path: r.build_time_seconds + for r in build_results + if isinstance(r, BuildResult) and r.success and r.index_path + } + for name, t in build_times.items(): + print(f" {name}: built in {t:.1f}s") + + # ── Search phase ────────────────────────────────────────────────────────── + banner("Running search benchmarks (via cuvs-bench)") + search_results = orchestrator.run_benchmark( + build=False, + search=True, + count=K, + **bench_kwargs, + ) + + print_results(search_results) + + print("\n" + "═" * 60) + print(" Benchmark complete!") + print("═" * 60) + print(f"\n OpenSearch : {OPENSEARCH_URL}") + print(" MinIO console : http://localhost:9001 (minioadmin / minioadmin)") + print() + + +if __name__ == "__main__": + main() diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml new file mode 100644 index 0000000000..9f10f0aaca --- /dev/null +++ b/deploy/docker-compose.yml @@ -0,0 +1,155 @@ +# OpenSearch GPU Remote Index Build — Docker Compose Demo +# +# Architecture: +# minio S3-compatible object store used to exchange vector data +# minio-init One-shot container that creates the MinIO bucket +# opensearch OpenSearch node with kNN plugin (requires 2.17+) +# remote-index-builder GPU-accelerated Faiss index builder (FastAPI service) +# bench Registers repo + cluster settings, runs cuvs-bench build/search +# +# Requirements: +# - NVIDIA GPU with CUDA support +# - NVIDIA Container Toolkit https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html +# - Docker Compose v2 +# - vm.max_map_count >= 262144 on the host: +# sudo sysctl -w vm.max_map_count=262144 +# +# Usage: +# docker compose up --build +# +# Data flow: +# OpenSearch flushes a segment → uploads vectors + doc-IDs to MinIO +# OpenSearch POSTs /_build to the remote-index-builder with the S3 paths +# remote-index-builder downloads from MinIO, builds index on GPU, uploads result +# OpenSearch downloads the finished index from MinIO and merges it into the shard + +services: + + # ── Object Store ──────────────────────────────────────────────────────────── + minio: + image: minio/minio:latest + environment: + MINIO_ROOT_USER: minioadmin + MINIO_ROOT_PASSWORD: minioadmin + command: server /data --console-address ":9001" + ports: + - "9000:9000" # S3 API + - "9001:9001" # MinIO web console + volumes: + - minio-data:/data + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] + interval: 5s + timeout: 5s + retries: 12 + start_period: 10s + + minio-init: + image: minio/mc:latest + depends_on: + minio: + condition: service_healthy + entrypoint: > + /bin/sh -c " + mc alias set local http://minio:9000 minioadmin minioadmin && + mc mb --ignore-existing local/opensearch-vectors && + echo 'Bucket opensearch-vectors ready' + " + restart: "no" + + # ── OpenSearch ─────────────────────────────────────────────────────────────── + opensearch: + build: + context: ./opensearch + environment: + - OPENSEARCH_JAVA_OPTS=-Xms4g -Xmx4g + # Bypass the Docker entrypoint script — all config lives in opensearch.yml. + # Keystore credentials are baked into the image (see opensearch/Dockerfile). + command: ["/usr/share/opensearch/bin/opensearch"] + ulimits: + nofile: + soft: 65536 + hard: 65536 + volumes: + - opensearch-data:/usr/share/opensearch/data + # Mounts S3 client config (endpoint, protocol, path_style_access). + # These are client-level settings and cannot be set in the snapshot API. + - ./opensearch/opensearch.yml:/usr/share/opensearch/config/opensearch.yml:ro + ports: + - "9200:9200" + depends_on: + minio-init: + condition: service_completed_successfully + healthcheck: + # wait_for_status=yellow blocks until the cluster is at least yellow + test: ["CMD-SHELL", "curl -sf 'http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s'"] + interval: 15s + timeout: 10s + retries: 20 + start_period: 30s + + # ── GPU Index Builder ──────────────────────────────────────────────────────── + remote-index-builder: + image: opensearchproject/remote-vector-index-builder:api-latest + environment: + - AWS_ACCESS_KEY_ID=minioadmin + - AWS_SECRET_ACCESS_KEY=minioadmin + - AWS_DEFAULT_REGION=us-east-1 + # Route S3 calls to MinIO instead of AWS (requires boto3 >= 1.28) + - AWS_ENDPOINT_URL=http://minio:9000 + ports: + - "1025:1025" + depends_on: + minio-init: + condition: service_completed_successfully + healthcheck: + test: ["CMD-SHELL", "python3 -c 'import socket; socket.create_connection((\"localhost\", 1025), 2).close()'"] + interval: 5s + timeout: 5s + retries: 24 + start_period: 10s + restart: on-failure + # GPU device reservation + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: 1 + capabilities: [gpu] + + # ── Benchmark ──────────────────────────────────────────────────────────────── + # Required environment variables (set in shell or a .env file): + # DATASET_PATH Absolute path to the directory containing dataset files + # e.g. export DATASET_PATH=/data/ann-benchmarks + # Optional: + # DATASET Dataset name (default: sift-128-euclidean) + # BENCH_GROUPS Parameter sweep group: test | base | large (default: test) + # K Number of neighbors to search for (default: 10) + bench: + build: + context: ./bench + depends_on: + opensearch: + condition: service_healthy + remote-index-builder: + condition: service_healthy + minio: + condition: service_healthy + environment: + - OPENSEARCH_URL=http://opensearch:9200 + - OPENSEARCH_HOST=opensearch + - OPENSEARCH_PORT=9200 + - BUILDER_URL=http://remote-index-builder:1025 + - MINIO_URL=http://minio:9000 + - DATASET=${DATASET:-sift-128-euclidean} + - DATASET_PATH=/data/datasets + - BENCH_GROUPS=${BENCH_GROUPS:-test} + - K=${K:-10} + volumes: + - ${DATASET_PATH:?DATASET_PATH must be set to the directory containing dataset files}:/data/datasets + restart: "no" + +volumes: + minio-data: + opensearch-data: diff --git a/deploy/opensearch/Dockerfile b/deploy/opensearch/Dockerfile new file mode 100644 index 0000000000..af5c9b5629 --- /dev/null +++ b/deploy/opensearch/Dockerfile @@ -0,0 +1,16 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +FROM opensearchproject/opensearch:latest + +# The repository-s3 plugin is not bundled in the default OpenSearch image but +# is required for the remote vector index build feature: OpenSearch uses it to +# upload raw vectors and download GPU-built indexes via the MinIO bucket. +RUN /usr/share/opensearch/bin/opensearch-plugin install --batch repository-s3 + +# Pre-populate the keystore with MinIO credentials. The keystore lives in +# /usr/share/opensearch/config (not the data volume) so this survives restarts. +# The repository-s3 plugin reads credentials exclusively from the keystore. +RUN /usr/share/opensearch/bin/opensearch-keystore create && \ + echo "minioadmin" | /usr/share/opensearch/bin/opensearch-keystore add --stdin s3.client.default.access_key && \ + echo "minioadmin" | /usr/share/opensearch/bin/opensearch-keystore add --stdin s3.client.default.secret_key diff --git a/deploy/opensearch/opensearch.yml b/deploy/opensearch/opensearch.yml new file mode 100644 index 0000000000..ebb17cced3 --- /dev/null +++ b/deploy/opensearch/opensearch.yml @@ -0,0 +1,21 @@ +# Bind to all interfaces so other containers can reach OpenSearch +network.host: 0.0.0.0 + +# Single-node cluster — suppresses the production bootstrap checks that +# require seed_hosts / initial_cluster_manager_nodes to be configured. +# Must be in opensearch.yml (not an env var) because we exec the opensearch +# binary directly rather than going through the Docker entrypoint script. +discovery.type: single-node + +# Disable the security plugin — no SSL certs needed for this demo +plugins.security.disabled: true + +# S3 client used by the repository-s3 plugin. +# Credentials are injected into the keystore at container startup +# (see the `command` block in docker-compose.yml). +s3.client.default.endpoint: "minio:9000" +s3.client.default.protocol: "http" +s3.client.default.path_style_access: "true" +# The async S3 client requires a non-empty region even for non-AWS endpoints. +# "us-east-1" is a placeholder — MinIO ignores the region value entirely. +s3.client.default.region: "us-east-1" diff --git a/deploy/remote-index-build/Dockerfile b/deploy/remote-index-build/Dockerfile new file mode 100644 index 0000000000..e67d495429 --- /dev/null +++ b/deploy/remote-index-build/Dockerfile @@ -0,0 +1,8 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +FROM python:3.11-slim +WORKDIR /app +RUN pip install --no-cache-dir requests boto3 numpy +COPY run.py . +CMD ["python", "-u", "run.py"] diff --git a/deploy/remote-index-build/run.py b/deploy/remote-index-build/run.py new file mode 100644 index 0000000000..63ba918ff2 --- /dev/null +++ b/deploy/remote-index-build/run.py @@ -0,0 +1,337 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +""" +OpenSearch GPU Remote Index Build — End-to-End Demo +==================================================== +Steps: + 1. Register a MinIO-backed S3 snapshot repository with OpenSearch + 2. Configure cluster settings to enable GPU-based remote index building + 3. Create a kNN index (Faiss HNSW / L2) with remote build enabled + 4. Ingest 100,000 random 256-dimensional float vectors via the bulk API (8 parallel workers) + 5. Flush + force-merge to consolidate segments and trigger the GPU build + 6. Poll MinIO for a .faiss file — hard-fail if the GPU build never completes + 7. Execute a kNN search and print the top-10 nearest neighbors +""" + +import json +import os +import sys +import time +from concurrent.futures import ThreadPoolExecutor, as_completed + +import boto3 +from botocore.config import Config +import numpy as np +import requests + +OPENSEARCH_URL = os.environ.get("OPENSEARCH_URL", "http://opensearch:9200") +BUILDER_URL = os.environ.get("BUILDER_URL", "http://remote-index-builder:1025") +MINIO_URL = os.environ.get("MINIO_URL", "http://minio:9000") + +INDEX_NAME = "gpu-demo" +DIMENSION = 256 # matches common embedding model output sizes +NUM_DOCS = 200_000 +BUCKET = "opensearch-vectors" +REPO_NAME = "vector-repo" + +session = requests.Session() +session.headers.update({"Content-Type": "application/json"}) + + +def banner(msg: str) -> None: + print(f"\n{'─' * 60}") + print(f" {msg}") + print(f"{'─' * 60}") + + +# ── configuration ───────────────────────────────────────────────────────────── + + +def register_repository() -> None: + banner(f"Registering S3 repository '{REPO_NAME}' (backed by MinIO)") + r = session.put( + f"{OPENSEARCH_URL}/_snapshot/{REPO_NAME}", + json={ + "type": "s3", + "settings": { + # endpoint / protocol / path_style_access are client-level settings + # configured in opensearch/opensearch.yml, not per-repository settings. + "bucket": BUCKET, + "base_path": "knn-indexes", + }, + }, + ) + r.raise_for_status() + print(f" {r.json()}") + + +def configure_cluster() -> None: + banner("Enabling remote GPU index build (cluster settings)") + r = session.put( + f"{OPENSEARCH_URL}/_cluster/settings", + json={ + "persistent": { + "knn.remote_index_build.enabled": True, + "knn.remote_index_build.repository": REPO_NAME, + "knn.remote_index_build.service.endpoint": BUILDER_URL, + } + }, + ) + r.raise_for_status() + print(f" {r.json()}") + + +# ── index ───────────────────────────────────────────────────────────────────── + + +def create_index() -> None: + banner(f"Creating kNN index '{INDEX_NAME}'") + + resp = session.delete(f"{OPENSEARCH_URL}/{INDEX_NAME}") + if resp.status_code == 200: + print(" Deleted existing index") + + r = session.put( + f"{OPENSEARCH_URL}/{INDEX_NAME}", + json={ + "settings": { + "index.knn": True, + "index.knn.remote_index_build.enabled": True, + # Trigger GPU build for segments >= 1 KB (covers any real dataset) + "index.knn.remote_index_build.size.min": "1kb", + "number_of_shards": 1, + "number_of_replicas": 0, + }, + "mappings": { + "properties": { + "vector": { + "type": "knn_vector", + "dimension": DIMENSION, + "method": { + "name": "hnsw", + "engine": "faiss", + "space_type": "l2", + "parameters": {"m": 32, "ef_construction": 512}, + }, + }, + "doc_id": {"type": "integer"}, + "label": {"type": "keyword"}, + } + }, + }, + ) + r.raise_for_status() + print(f" {r.json()}") + + +# ── ingest ──────────────────────────────────────────────────────────────────── + + +def ingest_vectors() -> None: + batch_size = 500 + banner( + f"Ingesting {NUM_DOCS:,} random {DIMENSION}-dim vectors (bulk API, 8 workers)" + ) + + def send_batch(start: int) -> int: + end = min(start + batch_size, NUM_DOCS) + vecs = np.random.randn(end - start, DIMENSION).astype(np.float32) + lines = [] + for i, vec in enumerate(vecs, start): + lines.append( + json.dumps({"index": {"_index": INDEX_NAME, "_id": str(i)}}) + ) + lines.append( + json.dumps( + { + "vector": vec.tolist(), + "doc_id": i, + "label": f"item-{i:04d}", + } + ) + ) + payload = ("\n".join(lines) + "\n").encode("utf-8") + r = session.post( + f"{OPENSEARCH_URL}/_bulk", + data=payload, + headers={"Content-Type": "application/x-ndjson"}, + ) + r.raise_for_status() + body = r.json() + if body.get("errors"): + failed = [ + item["index"]["error"] + for item in body["items"] + if "error" in item.get("index", {}) + ] + print( + f" Warning: {len(failed)} error(s) in batch {start}–{end}: {failed[0]}" + ) + return (end - start) - len(failed) + return end - start + + ingested = 0 + starts = list(range(0, NUM_DOCS, batch_size)) + with ThreadPoolExecutor(max_workers=8) as executor: + futures = {executor.submit(send_batch, s): s for s in starts} + for future in as_completed(futures): + ingested += future.result() + if ingested % 10_000 == 0 or ingested >= NUM_DOCS: + print(f" Ingested {ingested:,}/{NUM_DOCS:,}") + + session.post(f"{OPENSEARCH_URL}/{INDEX_NAME}/_flush") + r = session.get(f"{OPENSEARCH_URL}/{INDEX_NAME}/_count") + print(f" Document count after flush: {r.json()['count']:,}") + + +# ── GPU build ───────────────────────────────────────────────────────────────── + + +def trigger_gpu_build() -> None: + banner("Triggering GPU index build via force merge") + print( + " OpenSearch will upload vectors to MinIO, then call the GPU builder." + ) + print( + " force_merge max_num_segments=1 consolidates all segments into one." + ) + r = session.post( + f"{OPENSEARCH_URL}/{INDEX_NAME}/_forcemerge?max_num_segments=1", + timeout=300, + ) + print(f" Force merge HTTP {r.status_code}") + + +def verify_gpu_build(timeout: int = 600) -> None: + """Confirm the GPU builder uploaded a .faiss index file to MinIO. + + The remote-index-builder is the *only* component that writes .faiss files + back to the S3 bucket, so their presence is definitive proof that the GPU + build completed. The kNN stats API does not expose remote build counters + in OpenSearch 3.x, so we poll MinIO directly via boto3 instead. + + Exits with code 1 if no .faiss file appears within `timeout` seconds. + """ + banner("Verifying GPU index build (polling MinIO for .faiss files)") + print(f" Bucket : {BUCKET}/knn-indexes/") + print(f" Timeout : {timeout}s (poll interval: 5s)\n") + + s3 = boto3.client( + "s3", + endpoint_url=MINIO_URL, + aws_access_key_id="minioadmin", + aws_secret_access_key="minioadmin", + region_name="us-east-1", + config=Config(signature_version="s3v4"), + ) + + deadline = time.time() + timeout + while time.time() < deadline: + try: + resp = s3.list_objects_v2(Bucket=BUCKET, Prefix="knn-indexes/") + faiss_files = [ + obj["Key"] + for obj in resp.get("Contents", []) + if obj["Key"].endswith(".faiss") + ] + if faiss_files: + print( + f" PASS: GPU build confirmed — {len(faiss_files)} .faiss file(s) in MinIO:" + ) + for f in faiss_files: + print(f" s3://{BUCKET}/{f}") + return + + remaining = int(deadline - time.time()) + all_keys = [obj["Key"] for obj in resp.get("Contents", [])] + print( + f" Waiting for .faiss file... objects={all_keys} ({remaining}s left)" + ) + except Exception as e: + print(f" MinIO check error: {e}") + time.sleep(5) + + print( + f"\n FAIL: no GPU-built .faiss index appeared in MinIO after {timeout}s" + ) + print("\n Possible causes:") + print( + " 1. remote-index-builder is unreachable from the OpenSearch container." + ) + print( + f" Verify the container is running and BUILDER_URL={BUILDER_URL} is correct." + ) + print( + " 2. Segment size never exceeded index.knn.remote_index_build.size.min." + ) + print(" Try increasing NUM_DOCS or lowering the size.min threshold.") + print( + " 3. No GPU is available inside the remote-index-builder container." + ) + print(" Check: docker compose logs remote-index-builder") + print(" Ensure the NVIDIA Container Toolkit is installed on the host.") + sys.exit(1) + + +# ── search ──────────────────────────────────────────────────────────────────── + + +def search_vectors() -> None: + banner("kNN test search (top-5 nearest neighbors)") + query_vec = np.random.randn(DIMENSION).astype(np.float32).tolist() + + r = session.post( + f"{OPENSEARCH_URL}/{INDEX_NAME}/_search", + json={ + "size": 10, + "query": {"knn": {"vector": {"vector": query_vec, "k": 10}}}, + "_source": ["doc_id", "label"], + }, + ) + r.raise_for_status() + hits = r.json()["hits"]["hits"] + total = r.json()["hits"]["total"]["value"] + + print(f" Index contains {total} documents") + print(f" Top {len(hits)} results:") + for rank, hit in enumerate(hits, 1): + src = hit["_source"] + print( + f" #{rank:>2} id={hit['_id']:>6} score={hit['_score']:.6f} label={src['label']}" + ) + + +# ── entrypoint ──────────────────────────────────────────────────────────────── + + +def main() -> None: + print("\n" + "═" * 60) + print(" OpenSearch GPU Remote Index Build — End-to-End Demo") + print("═" * 60) + print(f" OpenSearch : {OPENSEARCH_URL}") + print(f" GPU builder: {BUILDER_URL}") + print( + f" Vectors : {NUM_DOCS} × dim={DIMENSION} engine=faiss method=hnsw space=l2" + ) + + register_repository() + configure_cluster() + create_index() + ingest_vectors() + trigger_gpu_build() + verify_gpu_build() + search_vectors() + + print("\n" + "═" * 60) + print(" Demo complete!") + print("═" * 60) + print(f"\n OpenSearch is still running at {OPENSEARCH_URL}") + print(" MinIO console : http://localhost:9001 (minioadmin / minioadmin)") + print(f" GPU builder : {BUILDER_URL}") + print() + + +if __name__ == "__main__": + main() From d8bf286bd664a6b4fe565c7d21c046a372fa5f9a Mon Sep 17 00:00:00 2001 From: James Bourbeau Date: Tue, 28 Apr 2026 17:56:40 -0500 Subject: [PATCH 2/9] Update Signed-off-by: James Bourbeau --- deploy/DEPLOYMENT.md | 191 ++++++++++++++++++++++++ deploy/README.md | 118 +++++++-------- deploy/bench/Dockerfile | 7 +- deploy/bench/entrypoint.sh | 48 ++++++ deploy/bench/run.py | 212 ++++++++++++++++++--------- deploy/docker-compose.yml | 114 ++++++-------- deploy/opensearch/Dockerfile | 18 +-- deploy/opensearch/entrypoint.sh | 31 ++++ deploy/opensearch/opensearch.yml | 14 +- deploy/remote-index-build/Dockerfile | 3 - deploy/remote-index-build/run.py | 162 +++++++------------- 11 files changed, 576 insertions(+), 342 deletions(-) create mode 100644 deploy/DEPLOYMENT.md create mode 100644 deploy/bench/entrypoint.sh create mode 100644 deploy/opensearch/entrypoint.sh diff --git a/deploy/DEPLOYMENT.md b/deploy/DEPLOYMENT.md new file mode 100644 index 0000000000..181132e309 --- /dev/null +++ b/deploy/DEPLOYMENT.md @@ -0,0 +1,191 @@ +# OpenSearch GPU Remote Index Build — Deployment Guide + +This guide walks through running OpenSearch with GPU-accelerated vector index construction using the [remote index build service](https://docs.opensearch.org/latest/vector-search/remote-index-build/). When enabled, OpenSearch offloads Faiss HNSW index building to a dedicated GPU service rather than building indexes in-process on the data nodes. + +## How it works + +The GPU build is triggered automatically during normal ingest — no changes to your indexing workflow are required beyond the one-time cluster and index configuration described below. + +```mermaid +sequenceDiagram + participant Client + participant OpenSearch + participant S3 + participant Builder as Remote Index Builder (GPU) + + Client->>OpenSearch: Bulk ingest vectors + note over OpenSearch: segment flush + OpenSearch->>S3: Upload raw vectors + doc IDs + OpenSearch->>Builder: POST /_build (S3 paths) + Builder->>S3: Download vectors + note over Builder: Build Faiss HNSW on GPU + Builder->>S3: Upload .faiss index + OpenSearch->>S3: Download .faiss index + note over OpenSearch: Merge into shard + Client->>OpenSearch: kNN search query + OpenSearch->>Client: Top-k results +``` + +## Services + +| Service | Image | Purpose | +|---|---|---| +| `opensearch` | custom build of `opensearchproject/opensearch:3.6.0` | OpenSearch node with kNN plugin and `repository-s3` plugin | +| `remote-index-builder` | `opensearchproject/remote-vector-index-builder:api-latest` | GPU-accelerated Faiss HNSW index builder | + +The custom OpenSearch image adds the `repository-s3` plugin (required for S3-backed vector staging) and populates the S3 keystore from environment variables at startup so credentials are never baked into image layers. + +## Requirements + +- **Docker Compose v2** +- **NVIDIA GPU** with CUDA support +- **NVIDIA Container Toolkit** — [installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) +- **AWS S3 bucket** for staging vectors during the build + +## Setup + +Set the host kernel parameter required by OpenSearch (once per reboot): + +```bash +sudo sysctl -w vm.max_map_count=262144 +``` + +Set required environment variables: + +```bash +export S3_BUCKET= +export AWS_ACCESS_KEY_ID= +export AWS_SECRET_ACCESS_KEY= +``` + +Optionally configure the region and session token for temporary credentials: + +```bash +export AWS_DEFAULT_REGION=us-east-1 # default: us-east-1 +export AWS_SESSION_TOKEN= # required for temporary (STS) credentials +``` + +Start OpenSearch and the GPU builder: + +```bash +docker compose --profile gpu up --build -d --wait opensearch remote-index-builder +``` + +## Connecting OpenSearch to the GPU builder + +Before any index can use GPU builds, you need to register your S3 bucket as a snapshot repository and apply the cluster settings that point OpenSearch at the builder service. Run these once against a live cluster. + +**Register S3 repository:** + +```bash +curl -X PUT http://localhost:9200/_snapshot/ \ + -H "Content-Type: application/json" \ + -d '{ + "type": "s3", + "settings": { + "bucket": "", + "base_path": "knn-indexes", + "region": "us-east-1" + } + }' +``` + +**Apply cluster settings:** + +```bash +curl -X PUT http://localhost:9200/_cluster/settings \ + -H "Content-Type: application/json" \ + -d '{ + "persistent": { + "knn.remote_index_build.enabled": true, + "knn.remote_index_build.repository": "", + "knn.remote_index_build.service.endpoint": "http://remote-index-builder:1025" + } + }' +``` + +> **Note:** `remote-index-builder` resolves inside the Docker network. If OpenSearch and the builder are not on the same Docker network, replace this with a reachable hostname or IP. + +## Creating an index with GPU builds enabled + +Add `"index.knn.remote_index_build.enabled": true` to your index settings alongside the standard kNN configuration: + +```bash +curl -X PUT http://localhost:9200/my-vectors \ + -H "Content-Type: application/json" \ + -d '{ + "settings": { + "index.knn": true, + "index.knn.remote_index_build.enabled": true, + "number_of_shards": 1, + "number_of_replicas": 1 + }, + "mappings": { + "properties": { + "vector": { + "type": "knn_vector", + "dimension": 256, + "method": { + "name": "hnsw", + "engine": "faiss", + "space_type": "l2", + "parameters": { + "m": 32, + "ef_construction": 512 + } + } + } + } + } + }' +``` + +GPU builds are only available with the `faiss` engine. The `lucene` engine always builds locally. + +## Verifying the GPU build + +The `remote-index-build/` directory contains an end-to-end demo script that ingests 200,000 random vectors, triggers a force-merge, and confirms the GPU build completed by polling S3 for the resulting `.faiss` file. + +Run it inside a temporary container on the same Docker network: + +```bash +docker compose run --rm \ + -e OPENSEARCH_URL=http://opensearch:9200 \ + -e BUILDER_URL=http://remote-index-builder:1025 \ + -e S3_BUCKET=${S3_BUCKET} \ + -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \ + -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \ + -e AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} \ + -e AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-us-east-1} \ + -v $(pwd)/remote-index-build:/app/remote-index-build \ + --no-deps bench \ + python remote-index-build/run.py +``` + +Or run it directly if you have Python and the dependencies installed locally (`boto3`, `numpy`, `requests`), pointing `OPENSEARCH_URL` at `http://localhost:9200`. + +A successful run prints a `.faiss` file path in S3 and returns top-10 nearest-neighbor results. + +## Tearing down + +```bash +docker compose --profile gpu down -v +``` + +The `-v` flag removes the OpenSearch data volume. Omit it to preserve indexed data across restarts. + +## Production considerations + +This setup is a working demonstration, not a production-hardened deployment. Key differences to address before running in production: + +- **Security plugin**: `opensearch.yml` has `plugins.security.disabled: true`. Re-enable it and configure TLS and authentication for any non-local deployment. +- **Single-node cluster**: `discovery.type: single-node` bypasses multi-node bootstrap checks. Replace with a properly configured multi-node cluster for production. +- **Replicas**: The demo uses `number_of_replicas: 0`. Set this to at least `1` for production workloads. +- **S3 permissions**: The IAM credentials need `s3:GetObject`, `s3:PutObject`, `s3:ListBucket`, and `s3:DeleteObject` on the staging bucket. + +## Ports + +| Port | Service | +|---|---| +| `9200` | OpenSearch REST API | +| `1025` | Remote index builder API | diff --git a/deploy/README.md b/deploy/README.md index f83bd82e48..d5678d55a1 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,6 +1,6 @@ -# OpenSearch GPU Remote Index Build Benchmark +# OpenSearch kNN Benchmark -Docker Compose benchmark of [OpenSearch's GPU-accelerated remote index build](https://docs.opensearch.org/latest/vector-search/remote-index-build/) feature using [cuvs-bench](https://github.com/jrbourbeau/cuvs/tree/main/python/cuvs_bench). Spins up all required services, builds a kNN index on a GPU, and runs a cuvs-bench search benchmark sweep. +Docker Compose benchmark comparing CPU and GPU kNN index builds in OpenSearch using [cuvs-bench](https://github.com/jrbourbeau/cuvs/tree/main/python/cuvs_bench). Supports both local CPU builds and [GPU-accelerated remote index builds](https://docs.opensearch.org/latest/vector-search/remote-index-build/) via the `REMOTE_INDEX_BUILD` environment variable. ## How it works @@ -8,11 +8,11 @@ OpenSearch's kNN plugin can offload Faiss HNSW index construction to a dedicated ``` OpenSearch flushes a segment - → uploads raw vectors + doc-IDs to S3 (MinIO in this demo) + → uploads raw vectors + doc-IDs to S3 → POSTs /_build to the remote-index-builder service - → service downloads vectors from MinIO + → service downloads vectors from S3 → builds Faiss HNSW index on GPU - → uploads finished index back to MinIO + → uploads finished index back to S3 → OpenSearch downloads the GPU-built index and merges it into the shard ``` @@ -20,18 +20,18 @@ OpenSearch flushes a segment | Service | Image | Purpose | |---|---|---| -| `minio` | `minio/minio` | S3-compatible object store — staging area for vectors and built indexes | -| `minio-init` | `minio/mc` | One-shot: creates the `opensearch-vectors` bucket | | `opensearch` | custom build of `opensearchproject/opensearch` | OpenSearch node with kNN plugin and `repository-s3` plugin | | `remote-index-builder` | `opensearchproject/remote-vector-index-builder:api-latest` | FastAPI service that builds Faiss indexes on the GPU | -| `bench` | custom Python | Registers repo + cluster settings, runs cuvs-bench build/search benchmark | +| `bench` | custom Python | Downloads dataset, registers repo + cluster settings, runs cuvs-bench build/search benchmark, exports results, generates plots | ## Requirements -- **NVIDIA GPU** with CUDA support -- **NVIDIA Container Toolkit** — [installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) - **Docker Compose v2** - **ANN benchmark dataset** in binary format (`.fbin`) — see [Dataset format](#dataset-format) +- **GPU mode only** (`--profile gpu`, `REMOTE_INDEX_BUILD=true`): + - NVIDIA GPU with CUDA support + - NVIDIA Container Toolkit — [installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) + - AWS S3 bucket (or S3-compatible store) for staging vectors and built indexes ## Usage @@ -47,39 +47,35 @@ Set required environment variables: export DATASET_PATH=/path/to/ann-benchmark-datasets # directory containing dataset files ``` -Optionally configure the benchmark: +GPU mode also requires S3 credentials: ```bash -export DATASET=sift-128-euclidean # default -export BENCH_GROUPS=test # test | base | large (default: test) -export K=10 # number of neighbors (default: 10) +export S3_BUCKET=my-opensearch-vectors # S3 bucket name +export AWS_ACCESS_KEY_ID= +export AWS_SECRET_ACCESS_KEY= ``` -Download the dataset (one-time setup, skipped automatically if already present): +Optionally configure the benchmark: ```bash -docker compose build bench -docker compose run --rm --no-deps bench python -m cuvs_bench.get_dataset \ - --dataset ${DATASET:-sift-128-euclidean} \ - --dataset-path /data/datasets +export AWS_SESSION_TOKEN= # required when using temporary (STS) credentials +export AWS_DEFAULT_REGION=us-east-1 # AWS region for the S3 bucket (default: us-east-1) +export DATASET=sift-128-euclidean # default +export BENCH_GROUPS=test # test | base (default: test) +export K=10 # number of neighbors (default: 10) ``` Start all services: ```bash +# CPU build (no GPU required) docker compose up --build -``` - -The `bench` container logs its progress through index build, GPU verification, and search. When complete you'll see a results table: +# GPU build +docker compose --profile gpu up --build ``` -════════════════════════════════════════════════════════════ - Benchmark complete! -════════════════════════════════════════════════════════════ - OpenSearch : http://opensearch:9200 - MinIO console : http://localhost:9001 (minioadmin / minioadmin) -``` +The `bench` container logs its progress through each phase. When complete you'll see a results table followed by the paths to the generated plot PNGs under `$DATASET_PATH`. To tear everything down: @@ -87,16 +83,19 @@ To tear everything down: docker compose down -v ``` -## What the bench script does +## What the bench container does -1. Registers MinIO as an OpenSearch S3 snapshot repository -2. Applies cluster settings to enable remote index build and point OpenSearch at the builder service -3. Runs `cuvs-bench` build phase (handled entirely by the OpenSearch backend): +1. Downloads the dataset (skipped if already present in `$DATASET_PATH`) +2. **GPU mode only**: Registers the S3 bucket as an OpenSearch snapshot repository +3. **GPU mode only**: Applies cluster settings to enable remote index build and point OpenSearch at the builder service +4. Runs `cuvs-bench` build phase (handled entirely by the OpenSearch backend): - Creates the kNN index and bulk-ingests dataset vectors - - Force-merges the index to trigger the GPU build - - Polls MinIO every 5 s for `.faiss` files confirming GPU build completion - - Records total build time (ingestion + GPU build) in the result -4. Runs `cuvs-bench` search phase and prints a recall/QPS/latency table + - **GPU mode**: Waits for all ingestion-time GPU builds to complete, then force-merges to one segment to trigger the final GPU build and polls the kNN stats API every 5 s until the build is confirmed complete + - **CPU mode**: Force-merges the index to one segment + - Records total build time in the result +5. Runs `cuvs-bench` search phase and prints a recall/QPS/latency table +6. Exports benchmark JSON results to CSV (`cuvs_bench.run --data-export`) +7. Generates recall vs. latency/throughput plots as PNGs in `$DATASET_PATH` (`cuvs_bench.plot`) ## Dataset format @@ -139,18 +138,27 @@ $DATASET_PATH/ | Group | Build params | Search params | Use case | |---|---|---|---| | `test` | 1 combo (m=16, ef_construction=100) | ef_search: 50, 100 | Quick smoke test | -| `base` | 6 combos (m × ef_construction sweep) | ef_search: 50–512 | Standard benchmark | -| `large` | 2 combos (larger m, ef_construction) | ef_search: 100–1024 | High-recall benchmark | +| `base` | 16 combos (m=[32,64,96,128] × ef_construction=[64,128,256,512]) | ef_search: 10–800 | Standard benchmark | ## GPU build verification -The cuvs-bench OpenSearch backend polls MinIO every 5 seconds for `.faiss` files under `s3://opensearch-vectors/knn-indexes/`. The remote-index-builder is the only component that writes `.faiss` files back to the bucket, so their presence is definitive proof the GPU build completed. +The cuvs-bench OpenSearch backend polls the kNN stats API every 5 seconds, waiting for `index_build_success_count` to increment by the expected number of new builds and for all in-flight flush and merge operations to reach zero. -The build raises a `TimeoutError` (causing the `bench` container to exit with code 1) if the expected number of `.faiss` files does not appear within 600 seconds. +The build raises a `TimeoutError` (causing the `bench` container to exit with code 1) if the expected number of successful builds is not confirmed within 600 seconds. -## Running without a GPU +## CPU vs GPU comparison -This is intentionally not supported. The `bench` container will exit 1 if `.faiss` files do not appear in MinIO within 600 s. If you want to experiment without hardware, remove the `deploy.resources.reservations` block from `remote-index-builder` in `docker-compose.yml` and be aware the benchmark will fail at the verification step. +To compare CPU and GPU builds on the same dataset, run the benchmark twice — once in each mode — clearing the OpenSearch volume between runs so the index is rebuilt from scratch each time: + +```bash +# GPU build (starts the remote-index-builder via --profile gpu) +docker compose --profile gpu up --build +docker compose --profile gpu down -v + +# CPU build (no GPU or S3 required) +docker compose up --build +docker compose down -v +``` ## Running tests @@ -171,15 +179,10 @@ docker compose run --rm --no-deps bench \ ### Integration tests (live OpenSearch node) -Requires a running OpenSearch node. Start it with: +Requires a running OpenSearch node. S3 credentials are not required for these tests. ```bash docker compose up -d --wait opensearch -``` - -Then run: - -```bash docker compose run --rm --no-deps \ -e OPENSEARCH_URL=http://opensearch:9200 \ bench \ @@ -188,22 +191,17 @@ docker compose run --rm --no-deps \ ### Remote index build integration tests (full GPU stack) -Requires the full stack (OpenSearch, MinIO, and the remote index builder). Start all services with: - -```bash -docker compose up -d --wait minio minio-init opensearch remote-index-builder -``` - -Then run: +Requires the full stack (OpenSearch and the remote index builder) and S3 credentials. ```bash +docker compose --profile gpu up -d --wait opensearch remote-index-builder docker compose run --rm --no-deps \ -e OPENSEARCH_URL=http://opensearch:9200 \ -e BUILDER_URL=http://remote-index-builder:1025 \ - -e S3_ENDPOINT=http://minio:9000 \ - -e S3_BUCKET=opensearch-vectors \ - -e S3_ACCESS_KEY=minioadmin \ - -e S3_SECRET_KEY=minioadmin \ + -e S3_BUCKET=${S3_BUCKET} \ + -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \ + -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \ + -e AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} \ bench \ pytest /opt/cuvs/python/cuvs_bench/cuvs_bench/tests/test_opensearch.py -v -m integration ``` @@ -215,6 +213,4 @@ This runs all integration tests including `TestOpenSearchRemoteIndexBuildIntegra | Port | Service | |---|---| | `9200` | OpenSearch REST API | -| `9000` | MinIO S3 API | -| `9001` | MinIO web console | | `1025` | Remote index builder API | diff --git a/deploy/bench/Dockerfile b/deploy/bench/Dockerfile index db3403b0e8..b4eb2af023 100644 --- a/deploy/bench/Dockerfile +++ b/deploy/bench/Dockerfile @@ -1,6 +1,3 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - FROM python:3.11-slim WORKDIR /app @@ -34,5 +31,5 @@ RUN pip install --no-cache-dir \ scikit-learn \ scipy -COPY run.py . -CMD ["python", "-u", "run.py"] +COPY --chmod=755 run.py entrypoint.sh . +CMD ["/app/entrypoint.sh"] diff --git a/deploy/bench/entrypoint.sh b/deploy/bench/entrypoint.sh new file mode 100644 index 0000000000..4f48da9c4c --- /dev/null +++ b/deploy/bench/entrypoint.sh @@ -0,0 +1,48 @@ +#!/bin/bash +set -e + +DATASET="${DATASET:-sift-128-euclidean}" +BENCH_GROUPS="${BENCH_GROUPS:-test}" +K="${K:-10}" +# Auto-detect GPU mode: remote-index-builder only appears in Docker DNS when +# started via --profile gpu. DNS entries are registered at network setup time +# (before containers run), so this check is reliable by the time entrypoint +# executes (OpenSearch healthy check alone takes 30+ seconds). +if getent hosts remote-index-builder > /dev/null 2>&1; then + echo "remote-index-builder detected — waiting for it to be ready..." + until python3 -c 'import socket; socket.create_connection(("remote-index-builder", 1025), 2).close()' 2>/dev/null; do + sleep 5 + done + echo "remote-index-builder is ready." + export REMOTE_INDEX_BUILD=true +else + echo "remote-index-builder not available — using CPU build mode." + export REMOTE_INDEX_BUILD=false +fi + +# Step 1: Download dataset (skipped automatically if already present) +python -m cuvs_bench.get_dataset \ + --dataset "$DATASET" \ + --dataset-path /data/datasets + +# Step 2: Run benchmark (build + search + writes result JSON files) +python -u run.py + +# Step 3: Export JSON → CSV (required by cuvs_bench.plot) +python -m cuvs_bench.run --data-export \ + --dataset "$DATASET" \ + --dataset-path /data/datasets \ + --algorithms opensearch_faiss_hnsw \ + --groups "$BENCH_GROUPS" \ + --count "$K" \ + --batch-size 10000 \ + --search-mode latency + +# Step 4: Plot — PNGs written to /data/datasets (mounted from host $DATASET_PATH) +python -m cuvs_bench.plot \ + --dataset "$DATASET" \ + --dataset-path /data/datasets \ + --algorithms opensearch_faiss_hnsw \ + --groups "$BENCH_GROUPS" \ + --count "$K" \ + --output-filepath /data/datasets diff --git a/deploy/bench/run.py b/deploy/bench/run.py index 645988d94c..84b894438a 100644 --- a/deploy/bench/run.py +++ b/deploy/bench/run.py @@ -1,20 +1,20 @@ #!/usr/bin/env python3 -# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - """ OpenSearch GPU Remote Index Build Benchmark =========================================== Steps: - 1. Register MinIO S3 snapshot repository with OpenSearch + 1. Register S3 snapshot repository with OpenSearch 2. Configure cluster settings for GPU remote index build 3. Build kNN index via cuvs-bench: a. Bulk-ingest dataset vectors b. Trigger force-merge to kick off the GPU build - c. Poll MinIO for .faiss files confirming GPU build completion + c. Poll S3 for .faiss files confirming GPU build completion 4. Run cuvs-bench search benchmarks and print results + 5. Write gbench-compatible JSON result files so cuvs_bench.run --data-export + and cuvs_bench.plot can be used for CSV export and plotting """ +import json import os import sys @@ -23,18 +23,21 @@ from cuvs_bench.backends.base import BuildResult, SearchResult -OPENSEARCH_URL = os.environ.get("OPENSEARCH_URL", "http://opensearch:9200") +OPENSEARCH_URL = os.environ.get("OPENSEARCH_URL", "http://opensearch:9200") OPENSEARCH_HOST = os.environ.get("OPENSEARCH_HOST", "opensearch") OPENSEARCH_PORT = int(os.environ.get("OPENSEARCH_PORT", "9200")) -BUILDER_URL = os.environ.get("BUILDER_URL", "http://remote-index-builder:1025") -MINIO_URL = os.environ.get("MINIO_URL", "http://minio:9000") +BUILDER_URL = os.environ.get("BUILDER_URL", "http://remote-index-builder:1025") + +REMOTE_INDEX_BUILD = os.environ.get("REMOTE_INDEX_BUILD", "false").lower() == "true" -DATASET = os.environ.get("DATASET", "sift-128-euclidean") -DATASET_PATH = os.environ.get("DATASET_PATH", "/data/datasets") -BENCH_GROUPS = os.environ.get("BENCH_GROUPS", "test") -K = int(os.environ.get("K", "10")) +S3_BUCKET = os.environ.get("S3_BUCKET", "") +S3_REGION = os.environ.get("AWS_DEFAULT_REGION", "us-east-1") + +DATASET = os.environ.get("DATASET", "sift-128-euclidean") +DATASET_PATH = os.environ.get("DATASET_PATH", "/data/datasets") +BENCH_GROUPS = os.environ.get("BENCH_GROUPS", "test") +K = int(os.environ.get("K", "10")) -BUCKET = "opensearch-vectors" REPO_NAME = "vector-repo" session = requests.Session() @@ -43,25 +46,22 @@ # ── helpers ─────────────────────────────────────────────────────────────────── - def banner(msg: str) -> None: - print(f"\n{'─' * 60}\n {msg}\n{'─' * 60}") + print(f"\n{'─'*60}\n {msg}\n{'─'*60}") # ── OpenSearch setup ────────────────────────────────────────────────────────── - def register_repository() -> None: - banner(f"Registering S3 repository '{REPO_NAME}' (backed by MinIO)") + banner(f"Registering S3 repository '{REPO_NAME}'") r = session.put( f"{OPENSEARCH_URL}/_snapshot/{REPO_NAME}", json={ "type": "s3", "settings": { - # endpoint / protocol / path_style_access are client-level settings - # configured in opensearch/opensearch.yml, not per-repository settings. - "bucket": BUCKET, + "bucket": S3_BUCKET, "base_path": "knn-indexes", + "region": S3_REGION, }, }, ) @@ -70,31 +70,103 @@ def register_repository() -> None: def configure_cluster() -> None: - banner("Enabling GPU remote index build (cluster settings)") + if REMOTE_INDEX_BUILD: + banner("Enabling GPU remote index build (cluster settings)") + settings = { + "knn.remote_index_build.enabled": True, + "knn.remote_index_build.repository": REPO_NAME, + "knn.remote_index_build.service.endpoint": BUILDER_URL, + } + else: + banner("Disabling GPU remote index build (cluster settings)") + settings = { + "knn.remote_index_build.enabled": False, + } r = session.put( f"{OPENSEARCH_URL}/_cluster/settings", - json={ - "persistent": { - "knn.remote_index_build.enabled": True, - "knn.remote_index_build.repository": REPO_NAME, - "knn.remote_index_build.service.endpoint": BUILDER_URL, - } - }, + json={"persistent": settings}, ) r.raise_for_status() print(f" {r.json()}") -# ── results ─────────────────────────────────────────────────────────────────── - +# ── result files ───────────────────────────────────────────────────────────── -def _print_result_row( - params: dict, recall: float, qps: float, latency_ms: float +def write_result_files( + build_results: list, + search_results: list, + dataset: str, + dataset_path: str, + algo: str, + groups: str, + k: int, + batch_size: int = 10000, ) -> None: + """Write gbench-compatible JSON result files. + + Creates files under //result/{build,search}/ in the + same format the C++ backend produces, so ``cuvs_bench.run --data-export`` + and ``cuvs_bench.plot`` work without modification. + """ + build_dir = os.path.join(dataset_path, dataset, "result", "build") + search_dir = os.path.join(dataset_path, dataset, "result", "search") + os.makedirs(build_dir, exist_ok=True) + os.makedirs(search_dir, exist_ok=True) + + # Build JSON – one record per successfully built index. + # data_export.py assumes the build CSV has columns: + # [algo_name, index_name, time, threads, cpu_time, ...] + # so "threads" and "cpu_time" must be present in the JSON (they are not in + # skip_build_cols and therefore get included as columns 3 and 4). + build_benchmarks = [ + { + "name": r.index_path, + "real_time": r.build_time_seconds, + "time_unit": "s", + "threads": 1, + "cpu_time": r.build_time_seconds, + } + for r in build_results + if isinstance(r, BuildResult) and r.success and r.index_path + ] + + # Search JSON – zip build + search results to recover the index name, then + # expand per-search-param entries so each (index, ef_search) is one record. + build_list = [r for r in build_results if isinstance(r, BuildResult)] + search_list = [r for r in search_results if isinstance(r, SearchResult)] + search_benchmarks = [] + for build_r, search_r in zip(build_list, search_list): + if not search_r.success or not build_r.index_path: + continue + for entry in (search_r.metadata or {}).get("per_search_param_results", []): + search_benchmarks.append({ + "name": build_r.index_path, + "real_time": entry["search_time_ms"], + "time_unit": "ms", + "Recall": entry["recall"], + "items_per_second": entry["queries_per_second"], + # Latency field expected by data_export in seconds + "Latency": entry["search_time_ms"] / 1000.0, + }) + + build_file = os.path.join(build_dir, f"{algo},{groups}.json") + search_file = os.path.join(search_dir, f"{algo},{groups},k{k},bs{batch_size}.json") + + with open(build_file, "w") as fh: + json.dump({"benchmarks": build_benchmarks}, fh, indent=2) + with open(search_file, "w") as fh: + json.dump({"benchmarks": search_benchmarks}, fh, indent=2) + + print(f"\n Result files written:") + print(f" {build_file}") + print(f" {search_file}") + + +# ── results ─────────────────────────────────────────────────────────────────── + +def _print_result_row(params: dict, recall: float, qps: float, latency_ms: float) -> None: params_str = ", ".join(f"{k}={v}" for k, v in params.items()) - print( - f" {params_str:<40} {recall:<12.4f} {qps:>8.1f} {latency_ms:>12.2f}" - ) + print(f" {params_str:<40} {recall:<12.4f} {qps:>8.1f} {latency_ms:>12.2f}") def print_results(results: list) -> None: @@ -104,41 +176,38 @@ def print_results(results: list) -> None: print(" No search results returned.") return - header = f" {'params':<40} {'recall@' + str(K):<12} {'QPS':>8} {'latency (ms)':>12}" + header = f" {'params':<40} {'recall@'+str(K):<12} {'QPS':>8} {'latency (ms)':>12}" print(header) print(" " + "─" * (len(header) - 2)) for r in search_results: per_param = (r.metadata or {}).get("per_search_param_results") if per_param: for entry in per_param: - _print_result_row( - entry["search_params"], - entry["recall"], - entry["queries_per_second"], - entry["search_time_ms"], - ) + _print_result_row(entry["search_params"], entry["recall"], entry["queries_per_second"], entry["search_time_ms"]) else: - _print_result_row( - r.search_params[0] if r.search_params else {}, - r.recall, - r.queries_per_second, - r.search_time_ms, - ) + _print_result_row(r.search_params[0] if r.search_params else {}, r.recall, r.queries_per_second, r.search_time_ms) # ── entrypoint ──────────────────────────────────────────────────────────────── - def main() -> None: + if REMOTE_INDEX_BUILD and not S3_BUCKET: + print("ERROR: S3_BUCKET must be set when REMOTE_INDEX_BUILD=true") + sys.exit(1) + print("\n" + "═" * 60) - print(" OpenSearch GPU Remote Index Build Benchmark") + print(" OpenSearch kNN Benchmark") print("═" * 60) - print(f" OpenSearch : {OPENSEARCH_URL}") - print(f" GPU builder: {BUILDER_URL}") - print(f" Dataset : {DATASET} (path: {DATASET_PATH})") - print(f" Groups : {BENCH_GROUPS} k={K}") - - register_repository() + print(f" OpenSearch : {OPENSEARCH_URL}") + print(f" Remote index build : {REMOTE_INDEX_BUILD}") + if REMOTE_INDEX_BUILD: + print(f" GPU builder : {BUILDER_URL}") + print(f" S3 bucket : s3://{S3_BUCKET}/knn-indexes/ (region: {S3_REGION})") + print(f" Dataset : {DATASET} (path: {DATASET_PATH})") + print(f" Groups : {BENCH_GROUPS} k={K}") + + if REMOTE_INDEX_BUILD: + register_repository() configure_cluster() orchestrator = BenchmarkOrchestrator(backend_type="opensearch") @@ -153,24 +222,16 @@ def main() -> None: port=OPENSEARCH_PORT, use_ssl=False, verify_certs=False, - remote_index_build=True, - # S3/MinIO config for GPU build verification (used by the backend) - remote_build_s3_endpoint=MINIO_URL, - remote_build_s3_bucket=BUCKET, - remote_build_s3_prefix="knn-indexes/", - remote_build_s3_access_key="minioadmin", - remote_build_s3_secret_key="minioadmin", + remote_index_build=REMOTE_INDEX_BUILD, ) - # ── Build phase ─────────────────────────────────────────────────────────── - # The backend handles the full GPU build flow: ingest vectors → force merge - # → poll MinIO for .faiss files confirming GPU build completion. - banner("Building index (GPU remote build via cuvs-bench)") + mode = "GPU remote build" if REMOTE_INDEX_BUILD else "CPU" + banner(f"Building index ({mode} via cuvs-bench)") build_results = orchestrator.run_benchmark( build=True, search=False, force=True, - bulk_batch_size=500, + bulk_batch_size=10_000, **bench_kwargs, ) @@ -201,11 +262,22 @@ def main() -> None: print_results(search_results) + write_result_files( + build_results=build_results, + search_results=search_results, + dataset=DATASET, + dataset_path=DATASET_PATH, + algo=bench_kwargs["algorithms"], + groups=BENCH_GROUPS, + k=K, + ) + print("\n" + "═" * 60) print(" Benchmark complete!") print("═" * 60) - print(f"\n OpenSearch : {OPENSEARCH_URL}") - print(" MinIO console : http://localhost:9001 (minioadmin / minioadmin)") + print(f"\n OpenSearch : {OPENSEARCH_URL}") + if REMOTE_INDEX_BUILD: + print(f" GPU builder : {BUILDER_URL}") print() diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml index 9f10f0aaca..3aea398496 100644 --- a/deploy/docker-compose.yml +++ b/deploy/docker-compose.yml @@ -1,85 +1,68 @@ # OpenSearch GPU Remote Index Build — Docker Compose Demo # # Architecture: -# minio S3-compatible object store used to exchange vector data -# minio-init One-shot container that creates the MinIO bucket # opensearch OpenSearch node with kNN plugin (requires 2.17+) # remote-index-builder GPU-accelerated Faiss index builder (FastAPI service) # bench Registers repo + cluster settings, runs cuvs-bench build/search # # Requirements: -# - NVIDIA GPU with CUDA support -# - NVIDIA Container Toolkit https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html # - Docker Compose v2 # - vm.max_map_count >= 262144 on the host: # sudo sysctl -w vm.max_map_count=262144 # +# GPU mode only (--profile gpu): +# - NVIDIA GPU with CUDA support +# - NVIDIA Container Toolkit https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html +# - An S3 bucket and AWS credentials (set via environment variables below) +# +# Required environment variables (set in shell or a .env file): +# DATASET_PATH Absolute path to the directory containing dataset files +# +# GPU mode only: +# S3_BUCKET S3 bucket name for staging vectors and built indexes +# AWS_ACCESS_KEY_ID AWS access key ID +# AWS_SECRET_ACCESS_KEY AWS secret access key +# +# Optional environment variables: +# AWS_SESSION_TOKEN STS session token (required for temporary credentials) +# AWS_DEFAULT_REGION AWS region for the S3 bucket (default: us-east-1) +# REMOTE_INDEX_BUILD Override GPU/CPU mode detection (true/false); normally auto-detected +# DATASET Dataset name (default: sift-128-euclidean) +# BENCH_GROUPS Parameter sweep group: test | base | large (default: test) +# K Number of neighbors to search for (default: 10) +# # Usage: -# docker compose up --build +# CPU: docker compose up --build +# GPU: docker compose --profile gpu up --build # # Data flow: -# OpenSearch flushes a segment → uploads vectors + doc-IDs to MinIO +# OpenSearch flushes a segment → uploads vectors + doc-IDs to S3 # OpenSearch POSTs /_build to the remote-index-builder with the S3 paths -# remote-index-builder downloads from MinIO, builds index on GPU, uploads result -# OpenSearch downloads the finished index from MinIO and merges it into the shard +# remote-index-builder downloads from S3, builds index on GPU, uploads result +# OpenSearch downloads the finished index from S3 and merges it into the shard services: - # ── Object Store ──────────────────────────────────────────────────────────── - minio: - image: minio/minio:latest - environment: - MINIO_ROOT_USER: minioadmin - MINIO_ROOT_PASSWORD: minioadmin - command: server /data --console-address ":9001" - ports: - - "9000:9000" # S3 API - - "9001:9001" # MinIO web console - volumes: - - minio-data:/data - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] - interval: 5s - timeout: 5s - retries: 12 - start_period: 10s - - minio-init: - image: minio/mc:latest - depends_on: - minio: - condition: service_healthy - entrypoint: > - /bin/sh -c " - mc alias set local http://minio:9000 minioadmin minioadmin && - mc mb --ignore-existing local/opensearch-vectors && - echo 'Bucket opensearch-vectors ready' - " - restart: "no" - # ── OpenSearch ─────────────────────────────────────────────────────────────── opensearch: build: context: ./opensearch environment: - - OPENSEARCH_JAVA_OPTS=-Xms4g -Xmx4g - # Bypass the Docker entrypoint script — all config lives in opensearch.yml. - # Keystore credentials are baked into the image (see opensearch/Dockerfile). - command: ["/usr/share/opensearch/bin/opensearch"] + - OPENSEARCH_JAVA_OPTS=-Xms16g -Xmx16g + # S3 credentials — entrypoint.sh writes these into the keystore at startup. + # If unset, OpenSearch starts without S3 configured (remote index build unavailable). + - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-} + - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-} + - AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN:-} ulimits: nofile: soft: 65536 hard: 65536 volumes: - opensearch-data:/usr/share/opensearch/data - # Mounts S3 client config (endpoint, protocol, path_style_access). - # These are client-level settings and cannot be set in the snapshot API. - ./opensearch/opensearch.yml:/usr/share/opensearch/config/opensearch.yml:ro ports: - "9200:9200" - depends_on: - minio-init: - condition: service_completed_successfully healthcheck: # wait_for_status=yellow blocks until the cluster is at least yellow test: ["CMD-SHELL", "curl -sf 'http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s'"] @@ -90,18 +73,15 @@ services: # ── GPU Index Builder ──────────────────────────────────────────────────────── remote-index-builder: + profiles: [gpu] image: opensearchproject/remote-vector-index-builder:api-latest environment: - - AWS_ACCESS_KEY_ID=minioadmin - - AWS_SECRET_ACCESS_KEY=minioadmin - - AWS_DEFAULT_REGION=us-east-1 - # Route S3 calls to MinIO instead of AWS (requires boto3 >= 1.28) - - AWS_ENDPOINT_URL=http://minio:9000 + - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-} + - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-} + - AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN:-} + - AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-us-east-1} ports: - "1025:1025" - depends_on: - minio-init: - condition: service_completed_successfully healthcheck: test: ["CMD-SHELL", "python3 -c 'import socket; socket.create_connection((\"localhost\", 1025), 2).close()'"] interval: 5s @@ -119,37 +99,29 @@ services: capabilities: [gpu] # ── Benchmark ──────────────────────────────────────────────────────────────── - # Required environment variables (set in shell or a .env file): - # DATASET_PATH Absolute path to the directory containing dataset files - # e.g. export DATASET_PATH=/data/ann-benchmarks - # Optional: - # DATASET Dataset name (default: sift-128-euclidean) - # BENCH_GROUPS Parameter sweep group: test | base | large (default: test) - # K Number of neighbors to search for (default: 10) bench: build: context: ./bench depends_on: opensearch: condition: service_healthy - remote-index-builder: - condition: service_healthy - minio: - condition: service_healthy environment: - OPENSEARCH_URL=http://opensearch:9200 - OPENSEARCH_HOST=opensearch - OPENSEARCH_PORT=9200 - BUILDER_URL=http://remote-index-builder:1025 - - MINIO_URL=http://minio:9000 + - S3_BUCKET=${S3_BUCKET:-} + - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-} + - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-} + - AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN:-} + - AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-us-east-1} - DATASET=${DATASET:-sift-128-euclidean} - DATASET_PATH=/data/datasets - BENCH_GROUPS=${BENCH_GROUPS:-test} - K=${K:-10} volumes: - - ${DATASET_PATH:?DATASET_PATH must be set to the directory containing dataset files}:/data/datasets + - ${DATASET_PATH:-/tmp/datasets}:/data/datasets restart: "no" volumes: - minio-data: opensearch-data: diff --git a/deploy/opensearch/Dockerfile b/deploy/opensearch/Dockerfile index af5c9b5629..84192b53ce 100644 --- a/deploy/opensearch/Dockerfile +++ b/deploy/opensearch/Dockerfile @@ -1,16 +1,12 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -FROM opensearchproject/opensearch:latest +FROM opensearchproject/opensearch:3.6.0 # The repository-s3 plugin is not bundled in the default OpenSearch image but # is required for the remote vector index build feature: OpenSearch uses it to -# upload raw vectors and download GPU-built indexes via the MinIO bucket. +# upload raw vectors and download GPU-built indexes via S3. RUN /usr/share/opensearch/bin/opensearch-plugin install --batch repository-s3 -# Pre-populate the keystore with MinIO credentials. The keystore lives in -# /usr/share/opensearch/config (not the data volume) so this survives restarts. -# The repository-s3 plugin reads credentials exclusively from the keystore. -RUN /usr/share/opensearch/bin/opensearch-keystore create && \ - echo "minioadmin" | /usr/share/opensearch/bin/opensearch-keystore add --stdin s3.client.default.access_key && \ - echo "minioadmin" | /usr/share/opensearch/bin/opensearch-keystore add --stdin s3.client.default.secret_key +# entrypoint.sh populates the keystore from S3_ACCESS_KEY / S3_SECRET_KEY +# environment variables at container startup, then execs opensearch. +COPY --chmod=755 entrypoint.sh /entrypoint.sh + +ENTRYPOINT ["/entrypoint.sh"] diff --git a/deploy/opensearch/entrypoint.sh b/deploy/opensearch/entrypoint.sh new file mode 100644 index 0000000000..eff9812fdb --- /dev/null +++ b/deploy/opensearch/entrypoint.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# +# Populate the OpenSearch keystore with S3 credentials from environment variables, +# then start OpenSearch. Doing this at runtime (not image build time) avoids +# baking credentials into image layers. +# +# Required environment variables: +# AWS_ACCESS_KEY_ID AWS access key ID +# AWS_SECRET_ACCESS_KEY AWS secret access key +# +# Optional environment variables: +# AWS_SESSION_TOKEN STS session token (required for temporary credentials) +# +set -e + +# The repository-s3 plugin reads credentials exclusively from the keystore. +# If credentials are not set, skip keystore setup — S3 and remote index build +# will be unavailable, but OpenSearch itself will start normally. +if [ -n "${AWS_ACCESS_KEY_ID}" ] && [ -n "${AWS_SECRET_ACCESS_KEY}" ]; then + rm -f /usr/share/opensearch/config/opensearch.keystore + /usr/share/opensearch/bin/opensearch-keystore create + printf '%s' "${AWS_ACCESS_KEY_ID}" | /usr/share/opensearch/bin/opensearch-keystore add --stdin s3.client.default.access_key + printf '%s' "${AWS_SECRET_ACCESS_KEY}" | /usr/share/opensearch/bin/opensearch-keystore add --stdin s3.client.default.secret_key + if [ -n "${AWS_SESSION_TOKEN}" ]; then + printf '%s' "${AWS_SESSION_TOKEN}" | /usr/share/opensearch/bin/opensearch-keystore add --stdin s3.client.default.session_token + fi +else + echo "Warning: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY not set — S3 repository and remote index build will not be available" >&2 +fi + +exec /usr/share/opensearch/bin/opensearch diff --git a/deploy/opensearch/opensearch.yml b/deploy/opensearch/opensearch.yml index ebb17cced3..96a68a6bc0 100644 --- a/deploy/opensearch/opensearch.yml +++ b/deploy/opensearch/opensearch.yml @@ -3,19 +3,9 @@ network.host: 0.0.0.0 # Single-node cluster — suppresses the production bootstrap checks that # require seed_hosts / initial_cluster_manager_nodes to be configured. -# Must be in opensearch.yml (not an env var) because we exec the opensearch -# binary directly rather than going through the Docker entrypoint script. +# Must be in opensearch.yml (not an env var) because our entrypoint.sh +# execs the opensearch binary directly. discovery.type: single-node # Disable the security plugin — no SSL certs needed for this demo plugins.security.disabled: true - -# S3 client used by the repository-s3 plugin. -# Credentials are injected into the keystore at container startup -# (see the `command` block in docker-compose.yml). -s3.client.default.endpoint: "minio:9000" -s3.client.default.protocol: "http" -s3.client.default.path_style_access: "true" -# The async S3 client requires a non-empty region even for non-AWS endpoints. -# "us-east-1" is a placeholder — MinIO ignores the region value entirely. -s3.client.default.region: "us-east-1" diff --git a/deploy/remote-index-build/Dockerfile b/deploy/remote-index-build/Dockerfile index e67d495429..d8b2485a7e 100644 --- a/deploy/remote-index-build/Dockerfile +++ b/deploy/remote-index-build/Dockerfile @@ -1,6 +1,3 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - FROM python:3.11-slim WORKDIR /app RUN pip install --no-cache-dir requests boto3 numpy diff --git a/deploy/remote-index-build/run.py b/deploy/remote-index-build/run.py index 63ba918ff2..a9f6621689 100644 --- a/deploy/remote-index-build/run.py +++ b/deploy/remote-index-build/run.py @@ -1,17 +1,14 @@ #!/usr/bin/env python3 -# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - """ OpenSearch GPU Remote Index Build — End-to-End Demo ==================================================== Steps: - 1. Register a MinIO-backed S3 snapshot repository with OpenSearch + 1. Register an S3 snapshot repository with OpenSearch 2. Configure cluster settings to enable GPU-based remote index building 3. Create a kNN index (Faiss HNSW / L2) with remote build enabled 4. Ingest 100,000 random 256-dimensional float vectors via the bulk API (8 parallel workers) 5. Flush + force-merge to consolidate segments and trigger the GPU build - 6. Poll MinIO for a .faiss file — hard-fail if the GPU build never completes + 6. Poll S3 for a .faiss file — hard-fail if the GPU build never completes 7. Execute a kNN search and print the top-10 nearest neighbors """ @@ -22,44 +19,43 @@ from concurrent.futures import ThreadPoolExecutor, as_completed import boto3 -from botocore.config import Config import numpy as np import requests OPENSEARCH_URL = os.environ.get("OPENSEARCH_URL", "http://opensearch:9200") -BUILDER_URL = os.environ.get("BUILDER_URL", "http://remote-index-builder:1025") -MINIO_URL = os.environ.get("MINIO_URL", "http://minio:9000") +BUILDER_URL = os.environ.get("BUILDER_URL", "http://remote-index-builder:1025") + +S3_BUCKET = os.environ["S3_BUCKET"] +S3_REGION = os.environ.get("AWS_DEFAULT_REGION", "us-east-1") +# boto3 reads AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY automatically INDEX_NAME = "gpu-demo" -DIMENSION = 256 # matches common embedding model output sizes -NUM_DOCS = 200_000 -BUCKET = "opensearch-vectors" -REPO_NAME = "vector-repo" +DIMENSION = 256 # matches common embedding model output sizes +NUM_DOCS = 200_000 +REPO_NAME = "vector-repo" session = requests.Session() session.headers.update({"Content-Type": "application/json"}) def banner(msg: str) -> None: - print(f"\n{'─' * 60}") + print(f"\n{'─'*60}") print(f" {msg}") - print(f"{'─' * 60}") + print(f"{'─'*60}") # ── configuration ───────────────────────────────────────────────────────────── - def register_repository() -> None: - banner(f"Registering S3 repository '{REPO_NAME}' (backed by MinIO)") + banner(f"Registering S3 repository '{REPO_NAME}'") r = session.put( f"{OPENSEARCH_URL}/_snapshot/{REPO_NAME}", json={ "type": "s3", "settings": { - # endpoint / protocol / path_style_access are client-level settings - # configured in opensearch/opensearch.yml, not per-repository settings. - "bucket": BUCKET, + "bucket": S3_BUCKET, "base_path": "knn-indexes", + "region": S3_REGION, }, }, ) @@ -73,8 +69,8 @@ def configure_cluster() -> None: f"{OPENSEARCH_URL}/_cluster/settings", json={ "persistent": { - "knn.remote_index_build.enabled": True, - "knn.remote_index_build.repository": REPO_NAME, + "knn.remote_index_build.enabled": True, + "knn.remote_index_build.repository": REPO_NAME, "knn.remote_index_build.service.endpoint": BUILDER_URL, } }, @@ -85,7 +81,6 @@ def configure_cluster() -> None: # ── index ───────────────────────────────────────────────────────────────────── - def create_index() -> None: banner(f"Creating kNN index '{INDEX_NAME}'") @@ -97,27 +92,25 @@ def create_index() -> None: f"{OPENSEARCH_URL}/{INDEX_NAME}", json={ "settings": { - "index.knn": True, - "index.knn.remote_index_build.enabled": True, - # Trigger GPU build for segments >= 1 KB (covers any real dataset) - "index.knn.remote_index_build.size.min": "1kb", - "number_of_shards": 1, + "index.knn": True, + "index.knn.remote_index_build.enabled": True, + "number_of_shards": 1, "number_of_replicas": 0, }, "mappings": { "properties": { "vector": { - "type": "knn_vector", + "type": "knn_vector", "dimension": DIMENSION, "method": { - "name": "hnsw", - "engine": "faiss", + "name": "hnsw", + "engine": "faiss", "space_type": "l2", "parameters": {"m": 32, "ef_construction": 512}, }, }, "doc_id": {"type": "integer"}, - "label": {"type": "keyword"}, + "label": {"type": "keyword"}, } }, }, @@ -128,30 +121,17 @@ def create_index() -> None: # ── ingest ──────────────────────────────────────────────────────────────────── - def ingest_vectors() -> None: batch_size = 500 - banner( - f"Ingesting {NUM_DOCS:,} random {DIMENSION}-dim vectors (bulk API, 8 workers)" - ) + banner(f"Ingesting {NUM_DOCS:,} random {DIMENSION}-dim vectors (bulk API, 8 workers)") def send_batch(start: int) -> int: - end = min(start + batch_size, NUM_DOCS) + end = min(start + batch_size, NUM_DOCS) vecs = np.random.randn(end - start, DIMENSION).astype(np.float32) lines = [] for i, vec in enumerate(vecs, start): - lines.append( - json.dumps({"index": {"_index": INDEX_NAME, "_id": str(i)}}) - ) - lines.append( - json.dumps( - { - "vector": vec.tolist(), - "doc_id": i, - "label": f"item-{i:04d}", - } - ) - ) + lines.append(json.dumps({"index": {"_index": INDEX_NAME, "_id": str(i)}})) + lines.append(json.dumps({"vector": vec.tolist(), "doc_id": i, "label": f"item-{i:04d}"})) payload = ("\n".join(lines) + "\n").encode("utf-8") r = session.post( f"{OPENSEARCH_URL}/_bulk", @@ -161,14 +141,8 @@ def send_batch(start: int) -> int: r.raise_for_status() body = r.json() if body.get("errors"): - failed = [ - item["index"]["error"] - for item in body["items"] - if "error" in item.get("index", {}) - ] - print( - f" Warning: {len(failed)} error(s) in batch {start}–{end}: {failed[0]}" - ) + failed = [item["index"]["error"] for item in body["items"] if "error" in item.get("index", {})] + print(f" Warning: {len(failed)} error(s) in batch {start}–{end}: {failed[0]}") return (end - start) - len(failed) return end - start @@ -188,15 +162,10 @@ def send_batch(start: int) -> int: # ── GPU build ───────────────────────────────────────────────────────────────── - def trigger_gpu_build() -> None: banner("Triggering GPU index build via force merge") - print( - " OpenSearch will upload vectors to MinIO, then call the GPU builder." - ) - print( - " force_merge max_num_segments=1 consolidates all segments into one." - ) + print(" OpenSearch will upload vectors to S3, then call the GPU builder.") + print(" force_merge max_num_segments=1 consolidates all segments into one.") r = session.post( f"{OPENSEARCH_URL}/{INDEX_NAME}/_forcemerge?max_num_segments=1", timeout=300, @@ -205,71 +174,52 @@ def trigger_gpu_build() -> None: def verify_gpu_build(timeout: int = 600) -> None: - """Confirm the GPU builder uploaded a .faiss index file to MinIO. + """Confirm the GPU builder uploaded a .faiss index file to S3. The remote-index-builder is the *only* component that writes .faiss files back to the S3 bucket, so their presence is definitive proof that the GPU build completed. The kNN stats API does not expose remote build counters - in OpenSearch 3.x, so we poll MinIO directly via boto3 instead. + in OpenSearch 3.x, so we poll S3 directly via boto3 instead. Exits with code 1 if no .faiss file appears within `timeout` seconds. """ - banner("Verifying GPU index build (polling MinIO for .faiss files)") - print(f" Bucket : {BUCKET}/knn-indexes/") + banner("Verifying GPU index build (polling S3 for .faiss files)") + print(f" Bucket : s3://{S3_BUCKET}/knn-indexes/") print(f" Timeout : {timeout}s (poll interval: 5s)\n") - s3 = boto3.client( - "s3", - endpoint_url=MINIO_URL, - aws_access_key_id="minioadmin", - aws_secret_access_key="minioadmin", - region_name="us-east-1", - config=Config(signature_version="s3v4"), - ) + # boto3 picks up AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION + # from the environment automatically. + s3 = boto3.client("s3", region_name=S3_REGION) deadline = time.time() + timeout while time.time() < deadline: try: - resp = s3.list_objects_v2(Bucket=BUCKET, Prefix="knn-indexes/") + resp = s3.list_objects_v2(Bucket=S3_BUCKET, Prefix="knn-indexes/") faiss_files = [ obj["Key"] for obj in resp.get("Contents", []) if obj["Key"].endswith(".faiss") ] if faiss_files: - print( - f" PASS: GPU build confirmed — {len(faiss_files)} .faiss file(s) in MinIO:" - ) + print(f" PASS: GPU build confirmed — {len(faiss_files)} .faiss file(s) in S3:") for f in faiss_files: - print(f" s3://{BUCKET}/{f}") + print(f" s3://{S3_BUCKET}/{f}") return remaining = int(deadline - time.time()) all_keys = [obj["Key"] for obj in resp.get("Contents", [])] - print( - f" Waiting for .faiss file... objects={all_keys} ({remaining}s left)" - ) + print(f" Waiting for .faiss file... objects={all_keys} ({remaining}s left)") except Exception as e: - print(f" MinIO check error: {e}") + print(f" S3 check error: {e}") time.sleep(5) - print( - f"\n FAIL: no GPU-built .faiss index appeared in MinIO after {timeout}s" - ) + print(f"\n FAIL: no GPU-built .faiss index appeared in S3 after {timeout}s") print("\n Possible causes:") - print( - " 1. remote-index-builder is unreachable from the OpenSearch container." - ) - print( - f" Verify the container is running and BUILDER_URL={BUILDER_URL} is correct." - ) - print( - " 2. Segment size never exceeded index.knn.remote_index_build.size.min." - ) + print(" 1. remote-index-builder is unreachable from the OpenSearch container.") + print(f" Verify the container is running and BUILDER_URL={BUILDER_URL} is correct.") + print(" 2. Segment size never exceeded index.knn.remote_index_build.size.min.") print(" Try increasing NUM_DOCS or lowering the size.min threshold.") - print( - " 3. No GPU is available inside the remote-index-builder container." - ) + print(" 3. No GPU is available inside the remote-index-builder container.") print(" Check: docker compose logs remote-index-builder") print(" Ensure the NVIDIA Container Toolkit is installed on the host.") sys.exit(1) @@ -277,7 +227,6 @@ def verify_gpu_build(timeout: int = 600) -> None: # ── search ──────────────────────────────────────────────────────────────────── - def search_vectors() -> None: banner("kNN test search (top-5 nearest neighbors)") query_vec = np.random.randn(DIMENSION).astype(np.float32).tolist() @@ -291,30 +240,26 @@ def search_vectors() -> None: }, ) r.raise_for_status() - hits = r.json()["hits"]["hits"] + hits = r.json()["hits"]["hits"] total = r.json()["hits"]["total"]["value"] print(f" Index contains {total} documents") print(f" Top {len(hits)} results:") for rank, hit in enumerate(hits, 1): src = hit["_source"] - print( - f" #{rank:>2} id={hit['_id']:>6} score={hit['_score']:.6f} label={src['label']}" - ) + print(f" #{rank:>2} id={hit['_id']:>6} score={hit['_score']:.6f} label={src['label']}") # ── entrypoint ──────────────────────────────────────────────────────────────── - def main() -> None: print("\n" + "═" * 60) print(" OpenSearch GPU Remote Index Build — End-to-End Demo") print("═" * 60) print(f" OpenSearch : {OPENSEARCH_URL}") print(f" GPU builder: {BUILDER_URL}") - print( - f" Vectors : {NUM_DOCS} × dim={DIMENSION} engine=faiss method=hnsw space=l2" - ) + print(f" S3 bucket : s3://{S3_BUCKET}/knn-indexes/ (region: {S3_REGION})") + print(f" Vectors : {NUM_DOCS} × dim={DIMENSION} engine=faiss method=hnsw space=l2") register_repository() configure_cluster() @@ -328,7 +273,6 @@ def main() -> None: print(" Demo complete!") print("═" * 60) print(f"\n OpenSearch is still running at {OPENSEARCH_URL}") - print(" MinIO console : http://localhost:9001 (minioadmin / minioadmin)") print(f" GPU builder : {BUILDER_URL}") print() From d2627c95e2002736a498ee651469322c3bd49b77 Mon Sep 17 00:00:00 2001 From: James Bourbeau Date: Fri, 15 May 2026 16:48:53 -0500 Subject: [PATCH 3/9] Updates Signed-off-by: James Bourbeau --- deploy/DEPLOYMENT.md | 7 ++- deploy/README.md | 34 ++++++----- deploy/bench/Dockerfile | 4 +- deploy/bench/entrypoint.sh | 46 ++++++++++---- deploy/bench/run.py | 101 ++++++++++++++++++++++++------- deploy/docker-compose.yml | 54 +++++++++-------- deploy/opensearch/Dockerfile | 4 +- deploy/remote-index-build/run.py | 100 +++++++++++++++++++++--------- 8 files changed, 243 insertions(+), 107 deletions(-) diff --git a/deploy/DEPLOYMENT.md b/deploy/DEPLOYMENT.md index 181132e309..1a87f3098a 100644 --- a/deploy/DEPLOYMENT.md +++ b/deploy/DEPLOYMENT.md @@ -117,6 +117,7 @@ curl -X PUT http://localhost:9200/my-vectors \ "settings": { "index.knn": true, "index.knn.remote_index_build.enabled": true, + "index.knn.remote_index_build.size.min": "1kb", "number_of_shards": 1, "number_of_replicas": 1 }, @@ -140,11 +141,11 @@ curl -X PUT http://localhost:9200/my-vectors \ }' ``` -GPU builds are only available with the `faiss` engine. The `lucene` engine always builds locally. +GPU builds are only available with the `faiss` engine. The `lucene` engine always builds locally. The low `size.min` value above is useful for demos because it forces small flushed segments onto the remote path; use OpenSearch's default or a larger production threshold for real workloads. ## Verifying the GPU build -The `remote-index-build/` directory contains an end-to-end demo script that ingests 200,000 random vectors, triggers a force-merge, and confirms the GPU build completed by polling S3 for the resulting `.faiss` file. +The `remote-index-build/` directory contains an end-to-end demo script that ingests 200,000 random vectors, flushes the index to trigger remote builds, and confirms the GPU build completed by polling S3 for the resulting `.faiss` file. Run it inside a temporary container on the same Docker network: @@ -157,6 +158,8 @@ docker compose run --rm \ -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \ -e AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} \ -e AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-us-east-1} \ + -e REMOTE_BUILD_SIZE_MIN=${REMOTE_BUILD_SIZE_MIN:-} \ + -e REMOTE_BUILD_TIMEOUT=${REMOTE_BUILD_TIMEOUT:-1800} \ -v $(pwd)/remote-index-build:/app/remote-index-build \ --no-deps bench \ python remote-index-build/run.py diff --git a/deploy/README.md b/deploy/README.md index d5678d55a1..3ff8b8cab3 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,6 +1,6 @@ # OpenSearch kNN Benchmark -Docker Compose benchmark comparing CPU and GPU kNN index builds in OpenSearch using [cuvs-bench](https://github.com/jrbourbeau/cuvs/tree/main/python/cuvs_bench). Supports both local CPU builds and [GPU-accelerated remote index builds](https://docs.opensearch.org/latest/vector-search/remote-index-build/) via the `REMOTE_INDEX_BUILD` environment variable. +Docker Compose benchmark comparing CPU and GPU kNN index builds in OpenSearch using `cuvs-bench`. Supports both local CPU builds and [GPU-accelerated remote index builds](https://docs.opensearch.org/latest/vector-search/remote-index-build/) via the `REMOTE_INDEX_BUILD` environment variable. ## How it works @@ -31,7 +31,7 @@ OpenSearch flushes a segment - **GPU mode only** (`--profile gpu`, `REMOTE_INDEX_BUILD=true`): - NVIDIA GPU with CUDA support - NVIDIA Container Toolkit — [installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) - - AWS S3 bucket (or S3-compatible store) for staging vectors and built indexes + - AWS S3 bucket for staging vectors and built indexes ## Usage @@ -50,7 +50,7 @@ export DATASET_PATH=/path/to/ann-benchmark-datasets # directory containing dat GPU mode also requires S3 credentials: ```bash -export S3_BUCKET=my-opensearch-vectors # S3 bucket name +export S3_BUCKET=opepsearch-s3-bucket # S3 bucket name export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= ``` @@ -63,6 +63,8 @@ export AWS_DEFAULT_REGION=us-east-1 # AWS region for the S3 bucket (defau export DATASET=sift-128-euclidean # default export BENCH_GROUPS=test # test | base (default: test) export K=10 # number of neighbors (default: 10) +export BATCH_SIZE=10000 # query/bulk batch size (default: 10000) +export REMOTE_BUILD_TIMEOUT=1800 # seconds to wait for remote builds (default: 1800) ``` Start all services: @@ -90,8 +92,8 @@ docker compose down -v 3. **GPU mode only**: Applies cluster settings to enable remote index build and point OpenSearch at the builder service 4. Runs `cuvs-bench` build phase (handled entirely by the OpenSearch backend): - Creates the kNN index and bulk-ingests dataset vectors - - **GPU mode**: Waits for all ingestion-time GPU builds to complete, then force-merges to one segment to trigger the final GPU build and polls the kNN stats API every 5 s until the build is confirmed complete - - **CPU mode**: Force-merges the index to one segment + - **GPU mode**: Flushes segments, waits for all submitted remote GPU builds to complete, and polls the kNN stats API every 5 s until the build is confirmed complete + - **CPU mode**: Flushes and refreshes the local OpenSearch index - Records total build time in the result 5. Runs `cuvs-bench` search phase and prints a recall/QPS/latency table 6. Exports benchmark JSON results to CSV (`cuvs_bench.run --data-export`) @@ -138,13 +140,13 @@ $DATASET_PATH/ | Group | Build params | Search params | Use case | |---|---|---|---| | `test` | 1 combo (m=16, ef_construction=100) | ef_search: 50, 100 | Quick smoke test | -| `base` | 16 combos (m=[32,64,96,128] × ef_construction=[64,128,256,512]) | ef_search: 10–800 | Standard benchmark | +| `base` | 9 combos (m=[32,64,96] × ef_construction=[64,128,256]) | ef_search: 10–800 | Standard benchmark | ## GPU build verification -The cuvs-bench OpenSearch backend polls the kNN stats API every 5 seconds, waiting for `index_build_success_count` to increment by the expected number of new builds and for all in-flight flush and merge operations to reach zero. +The cuvs-bench OpenSearch backend snapshots remote-build stats before ingest, then polls the kNN stats API every 5 seconds until `index_build_success_count` catches up with `build_request_success_count` and all in-flight flush and merge operations reach zero. -The build raises a `TimeoutError` (causing the `bench` container to exit with code 1) if the expected number of successful builds is not confirmed within 600 seconds. +The build raises a `TimeoutError` (causing the `bench` container to exit with code 1) if the expected successful builds are not confirmed within `REMOTE_BUILD_TIMEOUT` seconds. If no remote build is observed shortly after ingest, the backend raises an error that suggests lowering `REMOTE_BUILD_SIZE_MIN`; leave it unset to use OpenSearch's default threshold, or set it explicitly to override that value. ## CPU vs GPU comparison @@ -177,9 +179,9 @@ docker compose run --rm --no-deps bench \ pytest /opt/cuvs/python/cuvs_bench/cuvs_bench/tests/test_opensearch.py -v ``` -### Integration tests (live OpenSearch node) +### Integration tests (live OpenSearch node only) -Requires a running OpenSearch node. S3 credentials are not required for these tests. +Requires a running OpenSearch node. S3 credentials and the GPU profile are not required for these tests. ```bash docker compose up -d --wait opensearch @@ -191,22 +193,24 @@ docker compose run --rm --no-deps \ ### Remote index build integration tests (full GPU stack) -Requires the full stack (OpenSearch and the remote index builder) and S3 credentials. +Requires the full stack (OpenSearch and the remote index builder), S3 credentials, and a GPU-capable host. Export the AWS credentials and region before starting OpenSearch; its S3 keystore is populated at container startup. Use `--profile gpu` when starting the services so Docker Compose includes `remote-index-builder`. The pytest command itself does not need the profile flag because it runs against the already-started services. ```bash +export AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-us-east-1} docker compose --profile gpu up -d --wait opensearch remote-index-builder docker compose run --rm --no-deps \ -e OPENSEARCH_URL=http://opensearch:9200 \ -e BUILDER_URL=http://remote-index-builder:1025 \ -e S3_BUCKET=${S3_BUCKET} \ - -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \ - -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \ - -e AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} \ + -e AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION} \ + -e S3_ACCESS_KEY=${AWS_ACCESS_KEY_ID} \ + -e S3_SECRET_KEY=${AWS_SECRET_ACCESS_KEY} \ + -e S3_SESSION_TOKEN=${AWS_SESSION_TOKEN} \ bench \ pytest /opt/cuvs/python/cuvs_bench/cuvs_bench/tests/test_opensearch.py -v -m integration ``` -This runs all integration tests including `TestOpenSearchRemoteIndexBuildIntegration`, which verifies the full GPU build flow end-to-end. +This lets the pytest `integration` marker decide which tests run. With only OpenSearch running, remote-build tests skip because the GPU builder and S3 environment are unavailable. With the GPU stack running, the same marker includes the remote-build coverage. ## Ports diff --git a/deploy/bench/Dockerfile b/deploy/bench/Dockerfile index b4eb2af023..429534d9e2 100644 --- a/deploy/bench/Dockerfile +++ b/deploy/bench/Dockerfile @@ -1,7 +1,9 @@ FROM python:3.11-slim WORKDIR /app -RUN apt-get update && apt-get install -y --no-install-recommends git && rm -rf /var/lib/apt/lists/* +RUN apt-get update \ + && apt-get install -y --no-install-recommends git \ + && rm -rf /var/lib/apt/lists/* # Sparse-clone just the cuvs_bench Python package. # Installing via pip is not possible without CUDA (rapids-build-backend requires diff --git a/deploy/bench/entrypoint.sh b/deploy/bench/entrypoint.sh index 4f48da9c4c..c7ed3cb585 100644 --- a/deploy/bench/entrypoint.sh +++ b/deploy/bench/entrypoint.sh @@ -4,20 +4,44 @@ set -e DATASET="${DATASET:-sift-128-euclidean}" BENCH_GROUPS="${BENCH_GROUPS:-test}" K="${K:-10}" -# Auto-detect GPU mode: remote-index-builder only appears in Docker DNS when -# started via --profile gpu. DNS entries are registered at network setup time -# (before containers run), so this check is reliable by the time entrypoint -# executes (OpenSearch healthy check alone takes 30+ seconds). -if getent hosts remote-index-builder > /dev/null 2>&1; then +BATCH_SIZE="${BATCH_SIZE:-10000}" +ALGORITHM="opensearch_faiss_hnsw" + +wait_for_builder() { echo "remote-index-builder detected — waiting for it to be ready..." until python3 -c 'import socket; socket.create_connection(("remote-index-builder", 1025), 2).close()' 2>/dev/null; do sleep 5 done echo "remote-index-builder is ready." - export REMOTE_INDEX_BUILD=true +} + +if [ -n "${REMOTE_INDEX_BUILD:-}" ]; then + case "${REMOTE_INDEX_BUILD,,}" in + true|1|yes) + wait_for_builder + export REMOTE_INDEX_BUILD=true + ;; + false|0|no) + echo "REMOTE_INDEX_BUILD=false — using CPU build mode." + export REMOTE_INDEX_BUILD=false + ;; + *) + echo "ERROR: REMOTE_INDEX_BUILD must be true or false when set (got '${REMOTE_INDEX_BUILD}')" >&2 + exit 1 + ;; + esac else - echo "remote-index-builder not available — using CPU build mode." - export REMOTE_INDEX_BUILD=false + # Auto-detect GPU mode: remote-index-builder only appears in Docker DNS when + # started via --profile gpu. DNS entries are registered at network setup time + # (before containers run), so this check is reliable by the time entrypoint + # executes (OpenSearch healthy check alone takes 30+ seconds). + if getent hosts remote-index-builder > /dev/null 2>&1; then + wait_for_builder + export REMOTE_INDEX_BUILD=true + else + echo "remote-index-builder not available — using CPU build mode." + export REMOTE_INDEX_BUILD=false + fi fi # Step 1: Download dataset (skipped automatically if already present) @@ -32,17 +56,17 @@ python -u run.py python -m cuvs_bench.run --data-export \ --dataset "$DATASET" \ --dataset-path /data/datasets \ - --algorithms opensearch_faiss_hnsw \ + --algorithms "$ALGORITHM" \ --groups "$BENCH_GROUPS" \ --count "$K" \ - --batch-size 10000 \ + --batch-size "$BATCH_SIZE" \ --search-mode latency # Step 4: Plot — PNGs written to /data/datasets (mounted from host $DATASET_PATH) python -m cuvs_bench.plot \ --dataset "$DATASET" \ --dataset-path /data/datasets \ - --algorithms opensearch_faiss_hnsw \ + --algorithms "$ALGORITHM" \ --groups "$BENCH_GROUPS" \ --count "$K" \ --output-filepath /data/datasets diff --git a/deploy/bench/run.py b/deploy/bench/run.py index 84b894438a..a66b852a83 100644 --- a/deploy/bench/run.py +++ b/deploy/bench/run.py @@ -7,8 +7,8 @@ 2. Configure cluster settings for GPU remote index build 3. Build kNN index via cuvs-bench: a. Bulk-ingest dataset vectors - b. Trigger force-merge to kick off the GPU build - c. Poll S3 for .faiss files confirming GPU build completion + b. Flush segments to kick off remote GPU builds when enabled + c. Poll kNN stats until every submitted remote build completes 4. Run cuvs-bench search benchmarks and print results 5. Write gbench-compatible JSON result files so cuvs_bench.run --data-export and cuvs_bench.plot can be used for CSV export and plotting @@ -29,6 +29,8 @@ BUILDER_URL = os.environ.get("BUILDER_URL", "http://remote-index-builder:1025") REMOTE_INDEX_BUILD = os.environ.get("REMOTE_INDEX_BUILD", "false").lower() == "true" +REMOTE_BUILD_SIZE_MIN = os.environ.get("REMOTE_BUILD_SIZE_MIN", "").strip() +REMOTE_BUILD_TIMEOUT = int(os.environ.get("REMOTE_BUILD_TIMEOUT", "1800")) S3_BUCKET = os.environ.get("S3_BUCKET", "") S3_REGION = os.environ.get("AWS_DEFAULT_REGION", "us-east-1") @@ -37,7 +39,9 @@ DATASET_PATH = os.environ.get("DATASET_PATH", "/data/datasets") BENCH_GROUPS = os.environ.get("BENCH_GROUPS", "test") K = int(os.environ.get("K", "10")) +BATCH_SIZE = int(os.environ.get("BATCH_SIZE", "10000")) +ALGORITHM = "opensearch_faiss_hnsw" REPO_NAME = "vector-repo" session = requests.Session() @@ -50,6 +54,16 @@ def banner(msg: str) -> None: print(f"\n{'─'*60}\n {msg}\n{'─'*60}") +def _recall_for_entry( + result: SearchResult, entry_index: int, entry_count: int +) -> float | None: + # cuVS computes recall in the orchestrator from SearchResult.neighbors. The + # OpenSearch backend returns neighbors for the final search-parameter run. + if entry_index == entry_count - 1: + return float(result.recall) + return None + + # ── OpenSearch setup ────────────────────────────────────────────────────────── def register_repository() -> None: @@ -135,22 +149,33 @@ def write_result_files( build_list = [r for r in build_results if isinstance(r, BuildResult)] search_list = [r for r in search_results if isinstance(r, SearchResult)] search_benchmarks = [] + skipped_without_recall = 0 for build_r, search_r in zip(build_list, search_list): if not search_r.success or not build_r.index_path: continue - for entry in (search_r.metadata or {}).get("per_search_param_results", []): - search_benchmarks.append({ - "name": build_r.index_path, - "real_time": entry["search_time_ms"], - "time_unit": "ms", - "Recall": entry["recall"], - "items_per_second": entry["queries_per_second"], - # Latency field expected by data_export in seconds - "Latency": entry["search_time_ms"] / 1000.0, - }) + per_param = (search_r.metadata or {}).get("per_search_param_results", []) + for entry_index, entry in enumerate(per_param): + recall = _recall_for_entry(search_r, entry_index, len(per_param)) + if recall is None: + skipped_without_recall += 1 + continue + latency_ms = float(entry["search_time_ms"]) + search_benchmarks.append( + { + "name": build_r.index_path, + "real_time": latency_ms, + "time_unit": "ms", + "Recall": recall, + "items_per_second": float(entry["queries_per_second"]), + # Latency field expected by data_export in seconds + "Latency": latency_ms / 1000.0, + } + ) build_file = os.path.join(build_dir, f"{algo},{groups}.json") - search_file = os.path.join(search_dir, f"{algo},{groups},k{k},bs{batch_size}.json") + search_file = os.path.join( + search_dir, f"{algo},{groups},k{k},bs{batch_size}.json" + ) with open(build_file, "w") as fh: json.dump({"benchmarks": build_benchmarks}, fh, indent=2) @@ -160,13 +185,23 @@ def write_result_files( print(f"\n Result files written:") print(f" {build_file}") print(f" {search_file}") + if skipped_without_recall: + print( + " skipped " + f"{skipped_without_recall} search rows without per-parameter recall" + ) # ── results ─────────────────────────────────────────────────────────────────── -def _print_result_row(params: dict, recall: float, qps: float, latency_ms: float) -> None: +def _print_result_row( + params: dict, recall: float | None, qps: float | None, latency_ms: float | None +) -> None: params_str = ", ".join(f"{k}={v}" for k, v in params.items()) - print(f" {params_str:<40} {recall:<12.4f} {qps:>8.1f} {latency_ms:>12.2f}") + recall_str = "n/a" if recall is None else f"{recall:.4f}" + qps_str = "n/a" if qps is None else f"{qps:.1f}" + latency_str = "n/a" if latency_ms is None else f"{latency_ms:.2f}" + print(f" {params_str:<40} {recall_str:<12} {qps_str:>8} {latency_str:>12}") def print_results(results: list) -> None: @@ -179,13 +214,25 @@ def print_results(results: list) -> None: header = f" {'params':<40} {'recall@'+str(K):<12} {'QPS':>8} {'latency (ms)':>12}" print(header) print(" " + "─" * (len(header) - 2)) + missing_recall_rows = 0 for r in search_results: - per_param = (r.metadata or {}).get("per_search_param_results") - if per_param: - for entry in per_param: - _print_result_row(entry["search_params"], entry["recall"], entry["queries_per_second"], entry["search_time_ms"]) - else: - _print_result_row(r.search_params[0] if r.search_params else {}, r.recall, r.queries_per_second, r.search_time_ms) + per_param = (r.metadata or {}).get("per_search_param_results", []) + entry_count = len(per_param) + for entry_index, entry in enumerate(per_param): + recall = _recall_for_entry(r, entry_index, entry_count) + if recall is None: + missing_recall_rows += 1 + _print_result_row( + entry["search_params"], + recall, + float(entry["queries_per_second"]), + float(entry["search_time_ms"]), + ) + if missing_recall_rows: + print( + "\n Note: this cuVS version does not report recall for every " + "OpenSearch search parameter row." + ) # ── entrypoint ──────────────────────────────────────────────────────────────── @@ -203,7 +250,10 @@ def main() -> None: if REMOTE_INDEX_BUILD: print(f" GPU builder : {BUILDER_URL}") print(f" S3 bucket : s3://{S3_BUCKET}/knn-indexes/ (region: {S3_REGION})") + print(f" Build size minimum : {REMOTE_BUILD_SIZE_MIN or 'OpenSearch default'}") + print(f" Build timeout : {REMOTE_BUILD_TIMEOUT}s") print(f" Dataset : {DATASET} (path: {DATASET_PATH})") + print(f" Algorithm : {ALGORITHM}") print(f" Groups : {BENCH_GROUPS} k={K}") if REMOTE_INDEX_BUILD: @@ -216,7 +266,7 @@ def main() -> None: bench_kwargs = dict( dataset=DATASET, dataset_path=DATASET_PATH, - algorithms="opensearch_faiss_hnsw", + algorithms=ALGORITHM, groups=BENCH_GROUPS, host=OPENSEARCH_HOST, port=OPENSEARCH_PORT, @@ -224,6 +274,10 @@ def main() -> None: verify_certs=False, remote_index_build=REMOTE_INDEX_BUILD, ) + if REMOTE_INDEX_BUILD: + bench_kwargs["remote_build_timeout"] = REMOTE_BUILD_TIMEOUT + if REMOTE_BUILD_SIZE_MIN: + bench_kwargs["remote_build_size_min"] = REMOTE_BUILD_SIZE_MIN # ── Build phase ─────────────────────────────────────────────────────────── mode = "GPU remote build" if REMOTE_INDEX_BUILD else "CPU" banner(f"Building index ({mode} via cuvs-bench)") @@ -231,7 +285,7 @@ def main() -> None: build=True, search=False, force=True, - bulk_batch_size=10_000, + bulk_batch_size=BATCH_SIZE, **bench_kwargs, ) @@ -270,6 +324,7 @@ def main() -> None: algo=bench_kwargs["algorithms"], groups=BENCH_GROUPS, k=K, + batch_size=BATCH_SIZE, ) print("\n" + "═" * 60) diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml index 3aea398496..e0da2e1adc 100644 --- a/deploy/docker-compose.yml +++ b/deploy/docker-compose.yml @@ -27,9 +27,12 @@ # AWS_SESSION_TOKEN STS session token (required for temporary credentials) # AWS_DEFAULT_REGION AWS region for the S3 bucket (default: us-east-1) # REMOTE_INDEX_BUILD Override GPU/CPU mode detection (true/false); normally auto-detected +# REMOTE_BUILD_SIZE_MIN Optional minimum segment size override for remote builds +# REMOTE_BUILD_TIMEOUT Remote build wait timeout in seconds (default: 1800) # DATASET Dataset name (default: sift-128-euclidean) -# BENCH_GROUPS Parameter sweep group: test | base | large (default: test) +# BENCH_GROUPS Parameter sweep group: test | base (default: test) # K Number of neighbors to search for (default: 10) +# BATCH_SIZE cuvs-bench query/bulk batch size (default: 10000) # # Usage: # CPU: docker compose up --build @@ -41,6 +44,12 @@ # remote-index-builder downloads from S3, builds index on GPU, uploads result # OpenSearch downloads the finished index from S3 and merges it into the shard +x-aws-env: &aws-env + AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID:-} + AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY:-} + AWS_SESSION_TOKEN: ${AWS_SESSION_TOKEN:-} + AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION:-us-east-1} + services: # ── OpenSearch ─────────────────────────────────────────────────────────────── @@ -48,12 +57,8 @@ services: build: context: ./opensearch environment: - - OPENSEARCH_JAVA_OPTS=-Xms16g -Xmx16g - # S3 credentials — entrypoint.sh writes these into the keystore at startup. - # If unset, OpenSearch starts without S3 configured (remote index build unavailable). - - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-} - - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-} - - AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN:-} + <<: *aws-env + OPENSEARCH_JAVA_OPTS: -Xms16g -Xmx16g ulimits: nofile: soft: 65536 @@ -76,10 +81,7 @@ services: profiles: [gpu] image: opensearchproject/remote-vector-index-builder:api-latest environment: - - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-} - - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-} - - AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN:-} - - AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-us-east-1} + <<: *aws-env ports: - "1025:1025" healthcheck: @@ -106,19 +108,23 @@ services: opensearch: condition: service_healthy environment: - - OPENSEARCH_URL=http://opensearch:9200 - - OPENSEARCH_HOST=opensearch - - OPENSEARCH_PORT=9200 - - BUILDER_URL=http://remote-index-builder:1025 - - S3_BUCKET=${S3_BUCKET:-} - - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-} - - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-} - - AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN:-} - - AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-us-east-1} - - DATASET=${DATASET:-sift-128-euclidean} - - DATASET_PATH=/data/datasets - - BENCH_GROUPS=${BENCH_GROUPS:-test} - - K=${K:-10} + <<: *aws-env + OPENSEARCH_URL: http://opensearch:9200 + OPENSEARCH_HOST: opensearch + OPENSEARCH_PORT: "9200" + BUILDER_URL: http://remote-index-builder:1025 + REMOTE_INDEX_BUILD: ${REMOTE_INDEX_BUILD:-} + REMOTE_BUILD_SIZE_MIN: ${REMOTE_BUILD_SIZE_MIN:-} + REMOTE_BUILD_TIMEOUT: ${REMOTE_BUILD_TIMEOUT:-1800} + S3_BUCKET: ${S3_BUCKET:-} + S3_ACCESS_KEY: ${AWS_ACCESS_KEY_ID:-} + S3_SECRET_KEY: ${AWS_SECRET_ACCESS_KEY:-} + S3_SESSION_TOKEN: ${AWS_SESSION_TOKEN:-} + DATASET: ${DATASET:-sift-128-euclidean} + DATASET_PATH: /data/datasets + BENCH_GROUPS: ${BENCH_GROUPS:-test} + K: ${K:-10} + BATCH_SIZE: ${BATCH_SIZE:-10000} volumes: - ${DATASET_PATH:-/tmp/datasets}:/data/datasets restart: "no" diff --git a/deploy/opensearch/Dockerfile b/deploy/opensearch/Dockerfile index 84192b53ce..b90fe9a073 100644 --- a/deploy/opensearch/Dockerfile +++ b/deploy/opensearch/Dockerfile @@ -5,8 +5,8 @@ FROM opensearchproject/opensearch:3.6.0 # upload raw vectors and download GPU-built indexes via S3. RUN /usr/share/opensearch/bin/opensearch-plugin install --batch repository-s3 -# entrypoint.sh populates the keystore from S3_ACCESS_KEY / S3_SECRET_KEY -# environment variables at container startup, then execs opensearch. +# entrypoint.sh populates the keystore from AWS credential environment +# variables at container startup, then execs opensearch. COPY --chmod=755 entrypoint.sh /entrypoint.sh ENTRYPOINT ["/entrypoint.sh"] diff --git a/deploy/remote-index-build/run.py b/deploy/remote-index-build/run.py index a9f6621689..6e3af60d88 100644 --- a/deploy/remote-index-build/run.py +++ b/deploy/remote-index-build/run.py @@ -6,8 +6,8 @@ 1. Register an S3 snapshot repository with OpenSearch 2. Configure cluster settings to enable GPU-based remote index building 3. Create a kNN index (Faiss HNSW / L2) with remote build enabled - 4. Ingest 100,000 random 256-dimensional float vectors via the bulk API (8 parallel workers) - 5. Flush + force-merge to consolidate segments and trigger the GPU build + 4. Ingest 200,000 random 256-dimensional float vectors via the bulk API (8 parallel workers) + 5. Flush segments to trigger the GPU build 6. Poll S3 for a .faiss file — hard-fail if the GPU build never completes 7. Execute a kNN search and print the top-10 nearest neighbors """ @@ -33,6 +33,8 @@ DIMENSION = 256 # matches common embedding model output sizes NUM_DOCS = 200_000 REPO_NAME = "vector-repo" +REMOTE_BUILD_SIZE_MIN = os.environ.get("REMOTE_BUILD_SIZE_MIN", "").strip() +REMOTE_BUILD_TIMEOUT = int(os.environ.get("REMOTE_BUILD_TIMEOUT", "1800")) session = requests.Session() session.headers.update({"Content-Type": "application/json"}) @@ -88,15 +90,21 @@ def create_index() -> None: if resp.status_code == 200: print(" Deleted existing index") + index_settings = { + "index.knn": True, + "index.knn.remote_index_build.enabled": True, + "number_of_shards": 1, + "number_of_replicas": 0, + } + if REMOTE_BUILD_SIZE_MIN: + index_settings["index.knn.remote_index_build.size.min"] = ( + REMOTE_BUILD_SIZE_MIN + ) + r = session.put( f"{OPENSEARCH_URL}/{INDEX_NAME}", json={ - "settings": { - "index.knn": True, - "index.knn.remote_index_build.enabled": True, - "number_of_shards": 1, - "number_of_replicas": 0, - }, + "settings": index_settings, "mappings": { "properties": { "vector": { @@ -123,15 +131,28 @@ def create_index() -> None: def ingest_vectors() -> None: batch_size = 500 - banner(f"Ingesting {NUM_DOCS:,} random {DIMENSION}-dim vectors (bulk API, 8 workers)") + banner( + f"Ingesting {NUM_DOCS:,} random {DIMENSION}-dim vectors " + "(bulk API, 8 workers)" + ) def send_batch(start: int) -> int: end = min(start + batch_size, NUM_DOCS) vecs = np.random.randn(end - start, DIMENSION).astype(np.float32) lines = [] for i, vec in enumerate(vecs, start): - lines.append(json.dumps({"index": {"_index": INDEX_NAME, "_id": str(i)}})) - lines.append(json.dumps({"vector": vec.tolist(), "doc_id": i, "label": f"item-{i:04d}"})) + lines.append( + json.dumps({"index": {"_index": INDEX_NAME, "_id": str(i)}}) + ) + lines.append( + json.dumps( + { + "vector": vec.tolist(), + "doc_id": i, + "label": f"item-{i:04d}", + } + ) + ) payload = ("\n".join(lines) + "\n").encode("utf-8") r = session.post( f"{OPENSEARCH_URL}/_bulk", @@ -141,8 +162,15 @@ def send_batch(start: int) -> int: r.raise_for_status() body = r.json() if body.get("errors"): - failed = [item["index"]["error"] for item in body["items"] if "error" in item.get("index", {})] - print(f" Warning: {len(failed)} error(s) in batch {start}–{end}: {failed[0]}") + failed = [ + item["index"]["error"] + for item in body["items"] + if "error" in item.get("index", {}) + ] + print( + f" Warning: {len(failed)} error(s) in batch " + f"{start}–{end}: {failed[0]}" + ) return (end - start) - len(failed) return end - start @@ -155,25 +183,25 @@ def send_batch(start: int) -> int: if ingested % 10_000 == 0 or ingested >= NUM_DOCS: print(f" Ingested {ingested:,}/{NUM_DOCS:,}") - session.post(f"{OPENSEARCH_URL}/{INDEX_NAME}/_flush") + session.post(f"{OPENSEARCH_URL}/{INDEX_NAME}/_refresh") r = session.get(f"{OPENSEARCH_URL}/{INDEX_NAME}/_count") - print(f" Document count after flush: {r.json()['count']:,}") + print(f" Document count after ingest: {r.json()['count']:,}") # ── GPU build ───────────────────────────────────────────────────────────────── def trigger_gpu_build() -> None: - banner("Triggering GPU index build via force merge") - print(" OpenSearch will upload vectors to S3, then call the GPU builder.") - print(" force_merge max_num_segments=1 consolidates all segments into one.") - r = session.post( - f"{OPENSEARCH_URL}/{INDEX_NAME}/_forcemerge?max_num_segments=1", - timeout=300, + banner("Triggering GPU index build via flush") + print( + " OpenSearch will upload eligible flushed segments to S3, " + "then call the GPU builder." ) - print(f" Force merge HTTP {r.status_code}") + r = session.post(f"{OPENSEARCH_URL}/{INDEX_NAME}/_flush", timeout=300) + r.raise_for_status() + print(f" Flush complete: {r.json()}") -def verify_gpu_build(timeout: int = 600) -> None: +def verify_gpu_build(timeout: int = REMOTE_BUILD_TIMEOUT) -> None: """Confirm the GPU builder uploaded a .faiss index file to S3. The remote-index-builder is the *only* component that writes .faiss files @@ -201,14 +229,20 @@ def verify_gpu_build(timeout: int = 600) -> None: if obj["Key"].endswith(".faiss") ] if faiss_files: - print(f" PASS: GPU build confirmed — {len(faiss_files)} .faiss file(s) in S3:") + print( + " PASS: GPU build confirmed — " + f"{len(faiss_files)} .faiss file(s) in S3:" + ) for f in faiss_files: print(f" s3://{S3_BUCKET}/{f}") return remaining = int(deadline - time.time()) all_keys = [obj["Key"] for obj in resp.get("Contents", [])] - print(f" Waiting for .faiss file... objects={all_keys} ({remaining}s left)") + print( + f" Waiting for .faiss file... objects={all_keys} " + f"({remaining}s left)" + ) except Exception as e: print(f" S3 check error: {e}") time.sleep(5) @@ -216,7 +250,10 @@ def verify_gpu_build(timeout: int = 600) -> None: print(f"\n FAIL: no GPU-built .faiss index appeared in S3 after {timeout}s") print("\n Possible causes:") print(" 1. remote-index-builder is unreachable from the OpenSearch container.") - print(f" Verify the container is running and BUILDER_URL={BUILDER_URL} is correct.") + print( + " Verify the container is running and " + f"BUILDER_URL={BUILDER_URL} is correct." + ) print(" 2. Segment size never exceeded index.knn.remote_index_build.size.min.") print(" Try increasing NUM_DOCS or lowering the size.min threshold.") print(" 3. No GPU is available inside the remote-index-builder container.") @@ -228,7 +265,7 @@ def verify_gpu_build(timeout: int = 600) -> None: # ── search ──────────────────────────────────────────────────────────────────── def search_vectors() -> None: - banner("kNN test search (top-5 nearest neighbors)") + banner("kNN test search (top-10 nearest neighbors)") query_vec = np.random.randn(DIMENSION).astype(np.float32).tolist() r = session.post( @@ -259,14 +296,19 @@ def main() -> None: print(f" OpenSearch : {OPENSEARCH_URL}") print(f" GPU builder: {BUILDER_URL}") print(f" S3 bucket : s3://{S3_BUCKET}/knn-indexes/ (region: {S3_REGION})") - print(f" Vectors : {NUM_DOCS} × dim={DIMENSION} engine=faiss method=hnsw space=l2") + print( + f" Vectors : {NUM_DOCS} × dim={DIMENSION} " + "engine=faiss method=hnsw space=l2" + ) + print(f" Build size minimum: {REMOTE_BUILD_SIZE_MIN or 'OpenSearch default'}") + print(f" Build timeout: {REMOTE_BUILD_TIMEOUT}s") register_repository() configure_cluster() create_index() ingest_vectors() trigger_gpu_build() - verify_gpu_build() + verify_gpu_build(timeout=REMOTE_BUILD_TIMEOUT) search_vectors() print("\n" + "═" * 60) From 80b8251b7aecc7d2ceefa0b389884c89969955dd Mon Sep 17 00:00:00 2001 From: James Bourbeau Date: Mon, 18 May 2026 10:45:53 -0500 Subject: [PATCH 4/9] Update Signed-off-by: James Bourbeau --- deploy/DEPLOYMENT.md | 20 ++++++++++++-------- deploy/README.md | 17 +++++++++++++---- deploy/docker-compose.yml | 18 ++++++++---------- deploy/opensearch/Dockerfile | 2 +- deploy/opensearch/entrypoint.sh | 18 +++++++++--------- deploy/remote-index-build/run.py | 7 ++++--- 6 files changed, 47 insertions(+), 35 deletions(-) diff --git a/deploy/DEPLOYMENT.md b/deploy/DEPLOYMENT.md index 1a87f3098a..28b184d206 100644 --- a/deploy/DEPLOYMENT.md +++ b/deploy/DEPLOYMENT.md @@ -33,7 +33,7 @@ sequenceDiagram | `opensearch` | custom build of `opensearchproject/opensearch:3.6.0` | OpenSearch node with kNN plugin and `repository-s3` plugin | | `remote-index-builder` | `opensearchproject/remote-vector-index-builder:api-latest` | GPU-accelerated Faiss HNSW index builder | -The custom OpenSearch image adds the `repository-s3` plugin (required for S3-backed vector staging) and populates the S3 keystore from environment variables at startup so credentials are never baked into image layers. +The custom OpenSearch image adds the `repository-s3` plugin (required for S3-backed vector staging). When static AWS keys are provided, the image populates the S3 keystore at startup so credentials are never baked into image layers. Without static keys, OpenSearch can fall back to the AWS default credential provider chain, such as an EC2 instance role. ## Requirements @@ -50,19 +50,24 @@ Set the host kernel parameter required by OpenSearch (once per reboot): sudo sysctl -w vm.max_map_count=262144 ``` -Set required environment variables: +Set the required bucket name: ```bash export S3_BUCKET= +``` + +If you are using static credentials instead of a default AWS credential provider, also export: + +```bash export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= +export AWS_SESSION_TOKEN= # required for temporary (STS) credentials ``` -Optionally configure the region and session token for temporary credentials: +Optionally configure the region: ```bash export AWS_DEFAULT_REGION=us-east-1 # default: us-east-1 -export AWS_SESSION_TOKEN= # required for temporary (STS) credentials ``` Start OpenSearch and the GPU builder: @@ -154,9 +159,6 @@ docker compose run --rm \ -e OPENSEARCH_URL=http://opensearch:9200 \ -e BUILDER_URL=http://remote-index-builder:1025 \ -e S3_BUCKET=${S3_BUCKET} \ - -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \ - -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \ - -e AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} \ -e AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-us-east-1} \ -e REMOTE_BUILD_SIZE_MIN=${REMOTE_BUILD_SIZE_MIN:-} \ -e REMOTE_BUILD_TIMEOUT=${REMOTE_BUILD_TIMEOUT:-1800} \ @@ -165,6 +167,8 @@ docker compose run --rm \ python remote-index-build/run.py ``` +Static AWS credential environment variables are passed through by the `bench` service when they are exported on the host. + Or run it directly if you have Python and the dependencies installed locally (`boto3`, `numpy`, `requests`), pointing `OPENSEARCH_URL` at `http://localhost:9200`. A successful run prints a `.faiss` file path in S3 and returns top-10 nearest-neighbor results. @@ -184,7 +188,7 @@ This setup is a working demonstration, not a production-hardened deployment. Key - **Security plugin**: `opensearch.yml` has `plugins.security.disabled: true`. Re-enable it and configure TLS and authentication for any non-local deployment. - **Single-node cluster**: `discovery.type: single-node` bypasses multi-node bootstrap checks. Replace with a properly configured multi-node cluster for production. - **Replicas**: The demo uses `number_of_replicas: 0`. Set this to at least `1` for production workloads. -- **S3 permissions**: The IAM credentials need `s3:GetObject`, `s3:PutObject`, `s3:ListBucket`, and `s3:DeleteObject` on the staging bucket. +- **S3 permissions**: The IAM principal used by OpenSearch and the builder needs `s3:GetObject`, `s3:PutObject`, `s3:ListBucket`, and `s3:DeleteObject` on the staging bucket. ## Ports diff --git a/deploy/README.md b/deploy/README.md index 3ff8b8cab3..a6181a0cf2 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -47,18 +47,25 @@ Set required environment variables: export DATASET_PATH=/path/to/ann-benchmark-datasets # directory containing dataset files ``` -GPU mode also requires S3 credentials: +GPU mode also requires an S3 bucket. Static AWS keys are supported, but optional +when the containers can use another AWS default credential provider such as an +EC2 instance role: + +```bash +export S3_BUCKET=opensearch-s3-bucket # S3 bucket name +``` + +If you are using static credentials instead of a default provider, also export: ```bash -export S3_BUCKET=opepsearch-s3-bucket # S3 bucket name export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= +export AWS_SESSION_TOKEN= # required when using temporary (STS) credentials ``` Optionally configure the benchmark: ```bash -export AWS_SESSION_TOKEN= # required when using temporary (STS) credentials export AWS_DEFAULT_REGION=us-east-1 # AWS region for the S3 bucket (default: us-east-1) export DATASET=sift-128-euclidean # default export BENCH_GROUPS=test # test | base (default: test) @@ -193,7 +200,9 @@ docker compose run --rm --no-deps \ ### Remote index build integration tests (full GPU stack) -Requires the full stack (OpenSearch and the remote index builder), S3 credentials, and a GPU-capable host. Export the AWS credentials and region before starting OpenSearch; its S3 keystore is populated at container startup. Use `--profile gpu` when starting the services so Docker Compose includes `remote-index-builder`. The pytest command itself does not need the profile flag because it runs against the already-started services. +Requires the full stack (OpenSearch and the remote index builder), S3 access, and a GPU-capable host. Export the S3 bucket and region before starting OpenSearch. If you are using static AWS keys, export them before startup so OpenSearch can populate its S3 keystore; otherwise the containers can use the AWS default credential provider chain. Use `--profile gpu` when starting the services so Docker Compose includes `remote-index-builder`. The pytest command itself does not need the profile flag because it runs against the already-started services. + +When using static AWS keys, map them into the test fixture's `S3_*` variable names: ```bash export AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-us-east-1} diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml index e0da2e1adc..4680107006 100644 --- a/deploy/docker-compose.yml +++ b/deploy/docker-compose.yml @@ -13,18 +13,19 @@ # GPU mode only (--profile gpu): # - NVIDIA GPU with CUDA support # - NVIDIA Container Toolkit https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html -# - An S3 bucket and AWS credentials (set via environment variables below) +# - An S3 bucket and credentials from AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY +# or another AWS default credential provider, such as an EC2 instance role # # Required environment variables (set in shell or a .env file): # DATASET_PATH Absolute path to the directory containing dataset files # # GPU mode only: # S3_BUCKET S3 bucket name for staging vectors and built indexes -# AWS_ACCESS_KEY_ID AWS access key ID -# AWS_SECRET_ACCESS_KEY AWS secret access key # # Optional environment variables: -# AWS_SESSION_TOKEN STS session token (required for temporary credentials) +# AWS_ACCESS_KEY_ID AWS access key ID (optional if using default credentials) +# AWS_SECRET_ACCESS_KEY AWS secret access key (optional if using default credentials) +# AWS_SESSION_TOKEN STS session token (required for temporary static credentials) # AWS_DEFAULT_REGION AWS region for the S3 bucket (default: us-east-1) # REMOTE_INDEX_BUILD Override GPU/CPU mode detection (true/false); normally auto-detected # REMOTE_BUILD_SIZE_MIN Optional minimum segment size override for remote builds @@ -45,9 +46,9 @@ # OpenSearch downloads the finished index from S3 and merges it into the shard x-aws-env: &aws-env - AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID:-} - AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY:-} - AWS_SESSION_TOKEN: ${AWS_SESSION_TOKEN:-} + AWS_ACCESS_KEY_ID: + AWS_SECRET_ACCESS_KEY: + AWS_SESSION_TOKEN: AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION:-us-east-1} services: @@ -117,9 +118,6 @@ services: REMOTE_BUILD_SIZE_MIN: ${REMOTE_BUILD_SIZE_MIN:-} REMOTE_BUILD_TIMEOUT: ${REMOTE_BUILD_TIMEOUT:-1800} S3_BUCKET: ${S3_BUCKET:-} - S3_ACCESS_KEY: ${AWS_ACCESS_KEY_ID:-} - S3_SECRET_KEY: ${AWS_SECRET_ACCESS_KEY:-} - S3_SESSION_TOKEN: ${AWS_SESSION_TOKEN:-} DATASET: ${DATASET:-sift-128-euclidean} DATASET_PATH: /data/datasets BENCH_GROUPS: ${BENCH_GROUPS:-test} diff --git a/deploy/opensearch/Dockerfile b/deploy/opensearch/Dockerfile index b90fe9a073..06be41a26c 100644 --- a/deploy/opensearch/Dockerfile +++ b/deploy/opensearch/Dockerfile @@ -6,7 +6,7 @@ FROM opensearchproject/opensearch:3.6.0 RUN /usr/share/opensearch/bin/opensearch-plugin install --batch repository-s3 # entrypoint.sh populates the keystore from AWS credential environment -# variables at container startup, then execs opensearch. +# variables when provided, then execs opensearch. COPY --chmod=755 entrypoint.sh /entrypoint.sh ENTRYPOINT ["/entrypoint.sh"] diff --git a/deploy/opensearch/entrypoint.sh b/deploy/opensearch/entrypoint.sh index eff9812fdb..4ea84342a8 100644 --- a/deploy/opensearch/entrypoint.sh +++ b/deploy/opensearch/entrypoint.sh @@ -1,21 +1,18 @@ #!/bin/bash # -# Populate the OpenSearch keystore with S3 credentials from environment variables, +# If static S3 credentials are provided, write them to the OpenSearch keystore, # then start OpenSearch. Doing this at runtime (not image build time) avoids -# baking credentials into image layers. +# baking credentials into image layers. If static credentials are not provided, +# repository-s3 can fall back to the AWS default credential provider chain, such +# as an EC2 instance role. # -# Required environment variables: +# Static credential environment variables: # AWS_ACCESS_KEY_ID AWS access key ID # AWS_SECRET_ACCESS_KEY AWS secret access key -# -# Optional environment variables: # AWS_SESSION_TOKEN STS session token (required for temporary credentials) # set -e -# The repository-s3 plugin reads credentials exclusively from the keystore. -# If credentials are not set, skip keystore setup — S3 and remote index build -# will be unavailable, but OpenSearch itself will start normally. if [ -n "${AWS_ACCESS_KEY_ID}" ] && [ -n "${AWS_SECRET_ACCESS_KEY}" ]; then rm -f /usr/share/opensearch/config/opensearch.keystore /usr/share/opensearch/bin/opensearch-keystore create @@ -24,8 +21,11 @@ if [ -n "${AWS_ACCESS_KEY_ID}" ] && [ -n "${AWS_SECRET_ACCESS_KEY}" ]; then if [ -n "${AWS_SESSION_TOKEN}" ]; then printf '%s' "${AWS_SESSION_TOKEN}" | /usr/share/opensearch/bin/opensearch-keystore add --stdin s3.client.default.session_token fi +elif [ -n "${AWS_ACCESS_KEY_ID}" ] || [ -n "${AWS_SECRET_ACCESS_KEY}" ]; then + echo "ERROR: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be set together" >&2 + exit 1 else - echo "Warning: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY not set — S3 repository and remote index build will not be available" >&2 + echo "AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY not set; using the AWS default credential provider chain for S3" >&2 fi exec /usr/share/opensearch/bin/opensearch diff --git a/deploy/remote-index-build/run.py b/deploy/remote-index-build/run.py index 6e3af60d88..de6ab477a0 100644 --- a/deploy/remote-index-build/run.py +++ b/deploy/remote-index-build/run.py @@ -27,7 +27,8 @@ S3_BUCKET = os.environ["S3_BUCKET"] S3_REGION = os.environ.get("AWS_DEFAULT_REGION", "us-east-1") -# boto3 reads AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY automatically +# boto3 uses AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY when set, otherwise it +# falls through to the default credential provider chain. INDEX_NAME = "gpu-demo" DIMENSION = 256 # matches common embedding model output sizes @@ -215,8 +216,8 @@ def verify_gpu_build(timeout: int = REMOTE_BUILD_TIMEOUT) -> None: print(f" Bucket : s3://{S3_BUCKET}/knn-indexes/") print(f" Timeout : {timeout}s (poll interval: 5s)\n") - # boto3 picks up AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION - # from the environment automatically. + # boto3 uses static AWS env vars when set, otherwise it falls through to + # the default credential provider chain. s3 = boto3.client("s3", region_name=S3_REGION) deadline = time.time() + timeout From 4d21712b70110e5a18bd24404bcffc0f60714748 Mon Sep 17 00:00:00 2001 From: James Bourbeau Date: Mon, 18 May 2026 13:07:57 -0500 Subject: [PATCH 5/9] Update Signed-off-by: James Bourbeau --- deploy/DEPLOYMENT.md | 6 +++--- deploy/README.md | 8 ++++---- deploy/bench/run.py | 16 +++++++++++----- deploy/docker-compose.yml | 4 ++-- deploy/remote-index-build/run.py | 13 +++++++++++-- 5 files changed, 31 insertions(+), 16 deletions(-) diff --git a/deploy/DEPLOYMENT.md b/deploy/DEPLOYMENT.md index 28b184d206..a04bc259d7 100644 --- a/deploy/DEPLOYMENT.md +++ b/deploy/DEPLOYMENT.md @@ -67,7 +67,7 @@ export AWS_SESSION_TOKEN= # required for temporary (STS) creden Optionally configure the region: ```bash -export AWS_DEFAULT_REGION=us-east-1 # default: us-east-1 +export AWS_DEFAULT_REGION=us-west-2 # default: us-west-2 ``` Start OpenSearch and the GPU builder: @@ -90,7 +90,7 @@ curl -X PUT http://localhost:9200/_snapshot/ \ "settings": { "bucket": "", "base_path": "knn-indexes", - "region": "us-east-1" + "region": "us-west-2" } }' ``` @@ -159,7 +159,7 @@ docker compose run --rm \ -e OPENSEARCH_URL=http://opensearch:9200 \ -e BUILDER_URL=http://remote-index-builder:1025 \ -e S3_BUCKET=${S3_BUCKET} \ - -e AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-us-east-1} \ + -e AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-us-west-2} \ -e REMOTE_BUILD_SIZE_MIN=${REMOTE_BUILD_SIZE_MIN:-} \ -e REMOTE_BUILD_TIMEOUT=${REMOTE_BUILD_TIMEOUT:-1800} \ -v $(pwd)/remote-index-build:/app/remote-index-build \ diff --git a/deploy/README.md b/deploy/README.md index a6181a0cf2..0d60367e65 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -44,7 +44,7 @@ sudo sysctl -w vm.max_map_count=262144 Set required environment variables: ```bash -export DATASET_PATH=/path/to/ann-benchmark-datasets # directory containing dataset files +export DATASET_PATH="$(pwd)/ann-benchmark-datasets" # directory containing dataset files ``` GPU mode also requires an S3 bucket. Static AWS keys are supported, but optional @@ -52,7 +52,7 @@ when the containers can use another AWS default credential provider such as an EC2 instance role: ```bash -export S3_BUCKET=opensearch-s3-bucket # S3 bucket name +export S3_BUCKET=opensearch-cuvs-bench # S3 bucket name ``` If you are using static credentials instead of a default provider, also export: @@ -66,7 +66,7 @@ export AWS_SESSION_TOKEN= # required when using temporary (STS) Optionally configure the benchmark: ```bash -export AWS_DEFAULT_REGION=us-east-1 # AWS region for the S3 bucket (default: us-east-1) +export AWS_DEFAULT_REGION=us-west-2 # AWS region for the S3 bucket (default: us-west-2) export DATASET=sift-128-euclidean # default export BENCH_GROUPS=test # test | base (default: test) export K=10 # number of neighbors (default: 10) @@ -205,7 +205,7 @@ Requires the full stack (OpenSearch and the remote index builder), S3 access, an When using static AWS keys, map them into the test fixture's `S3_*` variable names: ```bash -export AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-us-east-1} +export AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-us-west-2} docker compose --profile gpu up -d --wait opensearch remote-index-builder docker compose run --rm --no-deps \ -e OPENSEARCH_URL=http://opensearch:9200 \ diff --git a/deploy/bench/run.py b/deploy/bench/run.py index a66b852a83..25424bf033 100644 --- a/deploy/bench/run.py +++ b/deploy/bench/run.py @@ -32,8 +32,8 @@ REMOTE_BUILD_SIZE_MIN = os.environ.get("REMOTE_BUILD_SIZE_MIN", "").strip() REMOTE_BUILD_TIMEOUT = int(os.environ.get("REMOTE_BUILD_TIMEOUT", "1800")) -S3_BUCKET = os.environ.get("S3_BUCKET", "") -S3_REGION = os.environ.get("AWS_DEFAULT_REGION", "us-east-1") +S3_BUCKET = os.environ.get("S3_BUCKET", "").strip() +S3_REGION = os.environ.get("AWS_DEFAULT_REGION", "us-west-2") DATASET = os.environ.get("DATASET", "sift-128-euclidean") DATASET_PATH = os.environ.get("DATASET_PATH", "/data/datasets") @@ -222,6 +222,7 @@ def print_results(results: list) -> None: recall = _recall_for_entry(r, entry_index, entry_count) if recall is None: missing_recall_rows += 1 + continue _print_result_row( entry["search_params"], recall, @@ -230,8 +231,8 @@ def print_results(results: list) -> None: ) if missing_recall_rows: print( - "\n Note: this cuVS version does not report recall for every " - "OpenSearch search parameter row." + "\n Omitted " + f"{missing_recall_rows} timing-only search rows without recall." ) @@ -239,7 +240,12 @@ def print_results(results: list) -> None: def main() -> None: if REMOTE_INDEX_BUILD and not S3_BUCKET: - print("ERROR: S3_BUCKET must be set when REMOTE_INDEX_BUILD=true") + print( + "ERROR: S3_BUCKET is not set. Remote index build requires an S3 " + "bucket for vector and index staging. Set it before starting the " + "stack, for example: export S3_BUCKET=", + file=sys.stderr, + ) sys.exit(1) print("\n" + "═" * 60) diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml index 4680107006..f089d2e894 100644 --- a/deploy/docker-compose.yml +++ b/deploy/docker-compose.yml @@ -26,7 +26,7 @@ # AWS_ACCESS_KEY_ID AWS access key ID (optional if using default credentials) # AWS_SECRET_ACCESS_KEY AWS secret access key (optional if using default credentials) # AWS_SESSION_TOKEN STS session token (required for temporary static credentials) -# AWS_DEFAULT_REGION AWS region for the S3 bucket (default: us-east-1) +# AWS_DEFAULT_REGION AWS region for the S3 bucket (default: us-west-2) # REMOTE_INDEX_BUILD Override GPU/CPU mode detection (true/false); normally auto-detected # REMOTE_BUILD_SIZE_MIN Optional minimum segment size override for remote builds # REMOTE_BUILD_TIMEOUT Remote build wait timeout in seconds (default: 1800) @@ -49,7 +49,7 @@ x-aws-env: &aws-env AWS_ACCESS_KEY_ID: AWS_SECRET_ACCESS_KEY: AWS_SESSION_TOKEN: - AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION:-us-east-1} + AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION:-us-west-2} services: diff --git a/deploy/remote-index-build/run.py b/deploy/remote-index-build/run.py index de6ab477a0..a7548ded04 100644 --- a/deploy/remote-index-build/run.py +++ b/deploy/remote-index-build/run.py @@ -25,8 +25,8 @@ OPENSEARCH_URL = os.environ.get("OPENSEARCH_URL", "http://opensearch:9200") BUILDER_URL = os.environ.get("BUILDER_URL", "http://remote-index-builder:1025") -S3_BUCKET = os.environ["S3_BUCKET"] -S3_REGION = os.environ.get("AWS_DEFAULT_REGION", "us-east-1") +S3_BUCKET = os.environ.get("S3_BUCKET", "").strip() +S3_REGION = os.environ.get("AWS_DEFAULT_REGION", "us-west-2") # boto3 uses AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY when set, otherwise it # falls through to the default credential provider chain. @@ -291,6 +291,15 @@ def search_vectors() -> None: # ── entrypoint ──────────────────────────────────────────────────────────────── def main() -> None: + if not S3_BUCKET: + print( + "ERROR: S3_BUCKET is not set. The remote index build demo requires " + "an S3 bucket for vector and index staging. Set it before running, " + "for example: export S3_BUCKET=", + file=sys.stderr, + ) + sys.exit(1) + print("\n" + "═" * 60) print(" OpenSearch GPU Remote Index Build — End-to-End Demo") print("═" * 60) From b5ecb4b8b4a150d3a1beb294108ce3b43da38c92 Mon Sep 17 00:00:00 2001 From: James Bourbeau Date: Wed, 20 May 2026 10:36:48 -0500 Subject: [PATCH 6/9] Update Signed-off-by: James Bourbeau --- deploy/README.md | 8 +++-- deploy/bench/Dockerfile | 2 +- deploy/bench/entrypoint.sh | 8 +++-- deploy/bench/run.py | 62 ++++++++++++++++++++++++++++++-------- deploy/docker-compose.yml | 6 ++-- 5 files changed, 66 insertions(+), 20 deletions(-) diff --git a/deploy/README.md b/deploy/README.md index 0d60367e65..64f644de4b 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -70,7 +70,8 @@ export AWS_DEFAULT_REGION=us-west-2 # AWS region for the S3 bucket (defau export DATASET=sift-128-euclidean # default export BENCH_GROUPS=test # test | base (default: test) export K=10 # number of neighbors (default: 10) -export BATCH_SIZE=10000 # query/bulk batch size (default: 10000) +export BATCH_SIZE= # optional query batch size override +export BUILD_BATCH_SIZE= # optional bulk ingest batch size override export REMOTE_BUILD_TIMEOUT=1800 # seconds to wait for remote builds (default: 1800) ``` @@ -101,6 +102,7 @@ docker compose down -v - Creates the kNN index and bulk-ingests dataset vectors - **GPU mode**: Flushes segments, waits for all submitted remote GPU builds to complete, and polls the kNN stats API every 5 s until the build is confirmed complete - **CPU mode**: Flushes and refreshes the local OpenSearch index + - Uses the backend's automatic OpenSearch bulk-ingest batch sizing by default; set `BUILD_BATCH_SIZE` to override it - Records total build time in the result 5. Runs `cuvs-bench` search phase and prints a recall/QPS/latency table 6. Exports benchmark JSON results to CSV (`cuvs_bench.run --data-export`) @@ -146,8 +148,8 @@ $DATASET_PATH/ | Group | Build params | Search params | Use case | |---|---|---|---| -| `test` | 1 combo (m=16, ef_construction=100) | ef_search: 50, 100 | Quick smoke test | -| `base` | 9 combos (m=[32,64,96] × ef_construction=[64,128,256]) | ef_search: 10–800 | Standard benchmark | +| `test` | 1 combo (m=16) | ef_search: 10, 20 | Quick smoke test | +| `base` | 4 combos (m=[16,32,48,64]) | ef_search: 10, 20, 40, 60, 80, 120, 200, 400, 600, 800 | Standard benchmark | ## GPU build verification diff --git a/deploy/bench/Dockerfile b/deploy/bench/Dockerfile index 429534d9e2..c709e5af20 100644 --- a/deploy/bench/Dockerfile +++ b/deploy/bench/Dockerfile @@ -26,7 +26,7 @@ RUN pip install --no-cache-dir \ h5py \ matplotlib \ "numpy<2" \ - "opensearch-py>=2.4.0,<3.0.0" \ + "opensearch-py>=2.4.0" \ pandas \ pyyaml \ requests \ diff --git a/deploy/bench/entrypoint.sh b/deploy/bench/entrypoint.sh index c7ed3cb585..ad7ccda117 100644 --- a/deploy/bench/entrypoint.sh +++ b/deploy/bench/entrypoint.sh @@ -4,8 +4,12 @@ set -e DATASET="${DATASET:-sift-128-euclidean}" BENCH_GROUPS="${BENCH_GROUPS:-test}" K="${K:-10}" -BATCH_SIZE="${BATCH_SIZE:-10000}" ALGORITHM="opensearch_faiss_hnsw" +BATCH_SIZE="${BATCH_SIZE:-}" +DATA_EXPORT_BATCH_ARGS=() +if [ -n "$BATCH_SIZE" ]; then + DATA_EXPORT_BATCH_ARGS=(--batch-size "$BATCH_SIZE") +fi wait_for_builder() { echo "remote-index-builder detected — waiting for it to be ready..." @@ -59,7 +63,7 @@ python -m cuvs_bench.run --data-export \ --algorithms "$ALGORITHM" \ --groups "$BENCH_GROUPS" \ --count "$K" \ - --batch-size "$BATCH_SIZE" \ + "${DATA_EXPORT_BATCH_ARGS[@]}" \ --search-mode latency # Step 4: Plot — PNGs written to /data/datasets (mounted from host $DATASET_PATH) diff --git a/deploy/bench/run.py b/deploy/bench/run.py index 25424bf033..cc4bc15c16 100644 --- a/deploy/bench/run.py +++ b/deploy/bench/run.py @@ -39,7 +39,10 @@ DATASET_PATH = os.environ.get("DATASET_PATH", "/data/datasets") BENCH_GROUPS = os.environ.get("BENCH_GROUPS", "test") K = int(os.environ.get("K", "10")) -BATCH_SIZE = int(os.environ.get("BATCH_SIZE", "10000")) +BATCH_SIZE = os.environ.get("BATCH_SIZE", "").strip() +BATCH_SIZE = int(BATCH_SIZE) if BATCH_SIZE else None +BUILD_BATCH_SIZE = os.environ.get("BUILD_BATCH_SIZE", "").strip() +BUILD_BATCH_SIZE = int(BUILD_BATCH_SIZE) if BUILD_BATCH_SIZE else None ALGORITHM = "opensearch_faiss_hnsw" REPO_NAME = "vector-repo" @@ -64,6 +67,18 @@ def _recall_for_entry( return None +def _get_search_batch_size(search_results: list) -> int | None: + if BATCH_SIZE is not None: + return BATCH_SIZE + for result in search_results: + if not isinstance(result, SearchResult): + continue + batch_size = (result.metadata or {}).get("batch_size") + if batch_size is not None: + return int(batch_size) + return None + + # ── OpenSearch setup ────────────────────────────────────────────────────────── def register_repository() -> None: @@ -114,7 +129,7 @@ def write_result_files( algo: str, groups: str, k: int, - batch_size: int = 10000, + batch_size: int, ) -> None: """Write gbench-compatible JSON result files. @@ -261,6 +276,14 @@ def main() -> None: print(f" Dataset : {DATASET} (path: {DATASET_PATH})") print(f" Algorithm : {ALGORITHM}") print(f" Groups : {BENCH_GROUPS} k={K}") + print( + " Search batch size : " + f"{BATCH_SIZE if BATCH_SIZE is not None else 'backend default'}" + ) + print( + " Build batch size : " + f"{BUILD_BATCH_SIZE if BUILD_BATCH_SIZE is not None else 'backend auto'}" + ) if REMOTE_INDEX_BUILD: register_repository() @@ -268,8 +291,8 @@ def main() -> None: orchestrator = BenchmarkOrchestrator(backend_type="opensearch") - # Shared kwargs for both build and search phases - bench_kwargs = dict( + # Shared kwargs for both build and search phases. + common_kwargs = dict( dataset=DATASET, dataset_path=DATASET_PATH, algorithms=ALGORITHM, @@ -278,12 +301,20 @@ def main() -> None: port=OPENSEARCH_PORT, use_ssl=False, verify_certs=False, + ) + + build_kwargs = dict( + common_kwargs, remote_index_build=REMOTE_INDEX_BUILD, ) + if BUILD_BATCH_SIZE is not None: + build_kwargs["build_batch_size"] = BUILD_BATCH_SIZE if REMOTE_INDEX_BUILD: - bench_kwargs["remote_build_timeout"] = REMOTE_BUILD_TIMEOUT + build_kwargs["remote_build_timeout"] = REMOTE_BUILD_TIMEOUT if REMOTE_BUILD_SIZE_MIN: - bench_kwargs["remote_build_size_min"] = REMOTE_BUILD_SIZE_MIN + build_kwargs["remote_build_size_min"] = REMOTE_BUILD_SIZE_MIN + + # ── Build phase ─────────────────────────────────────────────────────────── mode = "GPU remote build" if REMOTE_INDEX_BUILD else "CPU" banner(f"Building index ({mode} via cuvs-bench)") @@ -291,8 +322,7 @@ def main() -> None: build=True, search=False, force=True, - bulk_batch_size=BATCH_SIZE, - **bench_kwargs, + **build_kwargs, ) index_names = [ @@ -313,24 +343,32 @@ def main() -> None: # ── Search phase ────────────────────────────────────────────────────────── banner("Running search benchmarks (via cuvs-bench)") - search_results = orchestrator.run_benchmark( + search_run_kwargs = dict( build=False, search=True, count=K, - **bench_kwargs, + **common_kwargs, ) + if BATCH_SIZE is not None: + search_run_kwargs["batch_size"] = BATCH_SIZE + search_results = orchestrator.run_benchmark(**search_run_kwargs) print_results(search_results) + batch_size = _get_search_batch_size(search_results) + if batch_size is None: + print(" ERROR: no successful search result reported a batch size") + sys.exit(1) + write_result_files( build_results=build_results, search_results=search_results, dataset=DATASET, dataset_path=DATASET_PATH, - algo=bench_kwargs["algorithms"], + algo=common_kwargs["algorithms"], groups=BENCH_GROUPS, k=K, - batch_size=BATCH_SIZE, + batch_size=batch_size, ) print("\n" + "═" * 60) diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml index f089d2e894..bbba494ace 100644 --- a/deploy/docker-compose.yml +++ b/deploy/docker-compose.yml @@ -33,7 +33,8 @@ # DATASET Dataset name (default: sift-128-euclidean) # BENCH_GROUPS Parameter sweep group: test | base (default: test) # K Number of neighbors to search for (default: 10) -# BATCH_SIZE cuvs-bench query/bulk batch size (default: 10000) +# BATCH_SIZE Optional cuvs-bench query batch size override +# BUILD_BATCH_SIZE Optional OpenSearch bulk ingest batch size override # # Usage: # CPU: docker compose up --build @@ -122,7 +123,8 @@ services: DATASET_PATH: /data/datasets BENCH_GROUPS: ${BENCH_GROUPS:-test} K: ${K:-10} - BATCH_SIZE: ${BATCH_SIZE:-10000} + BATCH_SIZE: ${BATCH_SIZE:-} + BUILD_BATCH_SIZE: ${BUILD_BATCH_SIZE:-} volumes: - ${DATASET_PATH:-/tmp/datasets}:/data/datasets restart: "no" From b1573ab1374dbbaf5dc4ce72197df5589e4f8f34 Mon Sep 17 00:00:00 2001 From: James Bourbeau Date: Wed, 20 May 2026 11:29:48 -0500 Subject: [PATCH 7/9] Update Signed-off-by: James Bourbeau --- deploy/README.md | 2 +- deploy/bench/entrypoint.sh | 9 +++------ deploy/bench/run.py | 7 +++---- 3 files changed, 7 insertions(+), 11 deletions(-) diff --git a/deploy/README.md b/deploy/README.md index 64f644de4b..337b2866d3 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -105,7 +105,7 @@ docker compose down -v - Uses the backend's automatic OpenSearch bulk-ingest batch sizing by default; set `BUILD_BATCH_SIZE` to override it - Records total build time in the result 5. Runs `cuvs-bench` search phase and prints a recall/QPS/latency table -6. Exports benchmark JSON results to CSV (`cuvs_bench.run --data-export`) +6. Exports benchmark JSON results to CSV 7. Generates recall vs. latency/throughput plots as PNGs in `$DATASET_PATH` (`cuvs_bench.plot`) ## Dataset format diff --git a/deploy/bench/entrypoint.sh b/deploy/bench/entrypoint.sh index ad7ccda117..64a65a41b5 100644 --- a/deploy/bench/entrypoint.sh +++ b/deploy/bench/entrypoint.sh @@ -5,11 +5,6 @@ DATASET="${DATASET:-sift-128-euclidean}" BENCH_GROUPS="${BENCH_GROUPS:-test}" K="${K:-10}" ALGORITHM="opensearch_faiss_hnsw" -BATCH_SIZE="${BATCH_SIZE:-}" -DATA_EXPORT_BATCH_ARGS=() -if [ -n "$BATCH_SIZE" ]; then - DATA_EXPORT_BATCH_ARGS=(--batch-size "$BATCH_SIZE") -fi wait_for_builder() { echo "remote-index-builder detected — waiting for it to be ready..." @@ -57,13 +52,15 @@ python -m cuvs_bench.get_dataset \ python -u run.py # Step 3: Export JSON → CSV (required by cuvs_bench.plot) +# --batch-size is ignored when --data-export is set, but Click prompts for it +# before entering main(), so pass a dummy value to keep the container non-interactive. python -m cuvs_bench.run --data-export \ --dataset "$DATASET" \ --dataset-path /data/datasets \ --algorithms "$ALGORITHM" \ --groups "$BENCH_GROUPS" \ --count "$K" \ - "${DATA_EXPORT_BATCH_ARGS[@]}" \ + --batch-size 1 \ --search-mode latency # Step 4: Plot — PNGs written to /data/datasets (mounted from host $DATASET_PATH) diff --git a/deploy/bench/run.py b/deploy/bench/run.py index cc4bc15c16..9a29b832a4 100644 --- a/deploy/bench/run.py +++ b/deploy/bench/run.py @@ -10,8 +10,7 @@ b. Flush segments to kick off remote GPU builds when enabled c. Poll kNN stats until every submitted remote build completes 4. Run cuvs-bench search benchmarks and print results - 5. Write gbench-compatible JSON result files so cuvs_bench.run --data-export - and cuvs_bench.plot can be used for CSV export and plotting + 5. Write gbench-compatible JSON result files for CSV export and plotting """ import json @@ -134,8 +133,8 @@ def write_result_files( """Write gbench-compatible JSON result files. Creates files under //result/{build,search}/ in the - same format the C++ backend produces, so ``cuvs_bench.run --data-export`` - and ``cuvs_bench.plot`` work without modification. + same format the C++ backend produces, so the cuvs-bench CSV exporters and + ``cuvs_bench.plot`` work without modification. """ build_dir = os.path.join(dataset_path, dataset, "result", "build") search_dir = os.path.join(dataset_path, dataset, "result", "search") From 26e8640c7eec6145f8692578b8181c9fd23fe3e0 Mon Sep 17 00:00:00 2001 From: James Bourbeau Date: Wed, 20 May 2026 12:45:02 -0500 Subject: [PATCH 8/9] Multi-node Signed-off-by: James Bourbeau --- deploy/bench/entrypoint.sh | 7 +- deploy/bench/run.py | 10 +- deploy/docker-compose.multinode.yml | 147 ++++++++++++++++++++++++++++ deploy/docker-compose.yml | 4 + deploy/opensearch/Dockerfile | 1 + 5 files changed, 162 insertions(+), 7 deletions(-) create mode 100644 deploy/docker-compose.multinode.yml diff --git a/deploy/bench/entrypoint.sh b/deploy/bench/entrypoint.sh index 64a65a41b5..78f050f9e4 100644 --- a/deploy/bench/entrypoint.sh +++ b/deploy/bench/entrypoint.sh @@ -7,11 +7,12 @@ K="${K:-10}" ALGORITHM="opensearch_faiss_hnsw" wait_for_builder() { - echo "remote-index-builder detected — waiting for it to be ready..." - until python3 -c 'import socket; socket.create_connection(("remote-index-builder", 1025), 2).close()' 2>/dev/null; do + builder_url="${BUILDER_URL:-http://remote-index-builder:1025}" + echo "Remote index build enabled — waiting for builder at ${builder_url}..." + until BUILDER_URL="${builder_url}" python3 -c 'import os, socket; from urllib.parse import urlparse; url = urlparse(os.environ["BUILDER_URL"]); socket.create_connection((url.hostname, url.port or 1025), 2).close()' 2>/dev/null; do sleep 5 done - echo "remote-index-builder is ready." + echo "Remote index builder is ready." } if [ -n "${REMOTE_INDEX_BUILD:-}" ]; then diff --git a/deploy/bench/run.py b/deploy/bench/run.py index 9a29b832a4..55bdc1777f 100644 --- a/deploy/bench/run.py +++ b/deploy/bench/run.py @@ -32,6 +32,7 @@ REMOTE_BUILD_TIMEOUT = int(os.environ.get("REMOTE_BUILD_TIMEOUT", "1800")) S3_BUCKET = os.environ.get("S3_BUCKET", "").strip() +S3_PREFIX = os.environ.get("S3_PREFIX", "knn-indexes").strip() or "knn-indexes" S3_REGION = os.environ.get("AWS_DEFAULT_REGION", "us-west-2") DATASET = os.environ.get("DATASET", "sift-128-euclidean") @@ -44,7 +45,8 @@ BUILD_BATCH_SIZE = int(BUILD_BATCH_SIZE) if BUILD_BATCH_SIZE else None ALGORITHM = "opensearch_faiss_hnsw" -REPO_NAME = "vector-repo" +REPO_NAME = os.environ.get("REMOTE_VECTOR_REPOSITORY", "vector-repo").strip() +REPO_NAME = REPO_NAME or "vector-repo" session = requests.Session() session.headers.update({"Content-Type": "application/json"}) @@ -88,7 +90,7 @@ def register_repository() -> None: "type": "s3", "settings": { "bucket": S3_BUCKET, - "base_path": "knn-indexes", + "base_path": S3_PREFIX, "region": S3_REGION, }, }, @@ -269,7 +271,8 @@ def main() -> None: print(f" Remote index build : {REMOTE_INDEX_BUILD}") if REMOTE_INDEX_BUILD: print(f" GPU builder : {BUILDER_URL}") - print(f" S3 bucket : s3://{S3_BUCKET}/knn-indexes/ (region: {S3_REGION})") + print(f" S3 bucket : s3://{S3_BUCKET}/{S3_PREFIX}/ (region: {S3_REGION})") + print(f" Repository : {REPO_NAME}") print(f" Build size minimum : {REMOTE_BUILD_SIZE_MIN or 'OpenSearch default'}") print(f" Build timeout : {REMOTE_BUILD_TIMEOUT}s") print(f" Dataset : {DATASET} (path: {DATASET_PATH})") @@ -313,7 +316,6 @@ def main() -> None: if REMOTE_BUILD_SIZE_MIN: build_kwargs["remote_build_size_min"] = REMOTE_BUILD_SIZE_MIN - # ── Build phase ─────────────────────────────────────────────────────────── mode = "GPU remote build" if REMOTE_INDEX_BUILD else "CPU" banner(f"Building index ({mode} via cuvs-bench)") diff --git a/deploy/docker-compose.multinode.yml b/deploy/docker-compose.multinode.yml new file mode 100644 index 0000000000..c88f95d9e5 --- /dev/null +++ b/deploy/docker-compose.multinode.yml @@ -0,0 +1,147 @@ +# Multi-node split of the benchmark stack. +# +# Copy this file to all three EC2 instances from the deploy-opensearch-tmp +# branch of https://github.com/jrbourbeau/cuvs. The default OpenSearch and +# bench images build from that checkout, so keep the opensearch/ directory on +# the OpenSearch node and the bench/ directory on the client node unless you +# set OPENSEARCH_IMAGE and BENCH_IMAGE to prebuilt images. The OpenSearch image +# contains opensearch.yml, so a prebuilt image does not need the opensearch/ +# directory at runtime. +# +# OpenSearch node: +# docker compose -f docker-compose.multinode.yml --profile opensearch up -d +# +# GPU remote index builder node: +# docker compose -f docker-compose.multinode.yml --profile builder up -d +# +# Client/benchmark node: +# docker compose -f docker-compose.multinode.yml --profile client build bench +# docker compose -f docker-compose.multinode.yml --profile client run --rm bench +# +# Docker Compose networks are host-local. Across EC2 instances, use private +# EC2 DNS names or private IPv4 addresses in OPENSEARCH_URL, +# OPENSEARCH_HOST, and REMOTE_INDEX_BUILDER_URL. + +name: opensearch-cuvs-bench-multinode + +x-aws-env: &aws-env + AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID:-} + AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY:-} + AWS_SESSION_TOKEN: ${AWS_SESSION_TOKEN:-} + AWS_REGION: ${AWS_REGION:-us-west-2} + AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION:-us-west-2} + S3_BUCKET: ${S3_BUCKET:-} + S3_PREFIX: ${S3_PREFIX:-knn-indexes} + +x-bench-env: &bench-env + <<: *aws-env + OPENSEARCH_URL: ${OPENSEARCH_URL:-http://opensearch:9200} + OPENSEARCH_HOST: ${OPENSEARCH_HOST:-opensearch} + OPENSEARCH_PORT: ${OPENSEARCH_PORT:-9200} + BUILDER_URL: ${REMOTE_INDEX_BUILDER_URL:-http://remote-index-builder:1025} + REMOTE_INDEX_BUILDER_URL: ${REMOTE_INDEX_BUILDER_URL:-http://remote-index-builder:1025} + REMOTE_INDEX_BUILD: ${REMOTE_INDEX_BUILD:-true} + REMOTE_BUILD_SIZE_MIN: ${REMOTE_BUILD_SIZE_MIN:-} + REMOTE_BUILD_TIMEOUT: ${REMOTE_BUILD_TIMEOUT:-1800} + REMOTE_VECTOR_REPOSITORY: ${REMOTE_VECTOR_REPOSITORY:-vector-repo} + DATASET: ${DATASET:-sift-128-euclidean} + DATASET_PATH: /data/datasets + BENCH_GROUPS: ${BENCH_GROUPS:-test} + K: ${K:-10} + BATCH_SIZE: ${BATCH_SIZE:-} + BUILD_BATCH_SIZE: ${BUILD_BATCH_SIZE:-} + +services: + opensearch: + profiles: ["opensearch"] + image: ${OPENSEARCH_IMAGE:-opensearch-cuvs-bench-opensearch} + build: + context: ${OPENSEARCH_BUILD_CONTEXT:-./opensearch} + dockerfile: ${OPENSEARCH_DOCKERFILE:-Dockerfile} + container_name: opensearch + restart: unless-stopped + ports: + - "${OPENSEARCH_BIND_ADDR:-0.0.0.0}:${OPENSEARCH_PORT:-9200}:9200" + - "${OPENSEARCH_PERF_BIND_ADDR:-127.0.0.1}:${OPENSEARCH_PERF_PORT:-9600}:9600" + environment: + <<: *aws-env + cluster.name: ${OPENSEARCH_CLUSTER_NAME:-opensearch-cuvs-bench} + node.name: ${OPENSEARCH_NODE_NAME:-opensearch-node-1} + discovery.type: single-node + bootstrap.memory_lock: "true" + OPENSEARCH_JAVA_OPTS: ${OPENSEARCH_JAVA_OPTS:--Xms16g -Xmx16g} + DISABLE_SECURITY_PLUGIN: ${DISABLE_SECURITY_PLUGIN:-true} + OPENSEARCH_INITIAL_ADMIN_PASSWORD: ${OPENSEARCH_INITIAL_ADMIN_PASSWORD:-} + ulimits: + memlock: + soft: -1 + hard: -1 + nofile: + soft: 65536 + hard: 65536 + volumes: + - opensearch-data:/usr/share/opensearch/data + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:9200/_cluster/health >/dev/null || exit 1"] + interval: 20s + timeout: 5s + retries: 30 + + remote-index-builder: + profiles: ["builder", "gpu"] + image: ${REMOTE_INDEX_BUILDER_IMAGE:-opensearchproject/remote-vector-index-builder:api-latest} + container_name: remote-index-builder + restart: unless-stopped + ports: + - "${REMOTE_INDEX_BUILDER_BIND_ADDR:-0.0.0.0}:${REMOTE_INDEX_BUILDER_HOST_PORT:-1025}:1025" + environment: + <<: *aws-env + healthcheck: + test: ["CMD-SHELL", "python3 -c 'import socket; socket.create_connection((\"localhost\", 1025), 2).close()'"] + interval: 5s + timeout: 5s + retries: 24 + start_period: 10s + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: ${GPU_COUNT:-1} + capabilities: ["gpu"] + + bench: + profiles: ["client"] + image: ${BENCH_IMAGE:-opensearch-cuvs-bench-bench} + build: + context: ${BENCH_BUILD_CONTEXT:-./bench} + dockerfile: ${BENCH_DOCKERFILE:-Dockerfile} + environment: + <<: *bench-env + extra_hosts: + - "host.docker.internal:host-gateway" + volumes: + - ${DATASET_PATH:-/tmp/datasets}:/data/datasets + + configure-remote-index-build: + profiles: ["configure"] + image: curlimages/curl:latest + environment: + <<: *bench-env + command: + - sh + - -ec + - | + : "$${S3_BUCKET:?S3_BUCKET must be set}" + : "$${REMOTE_INDEX_BUILDER_URL:?REMOTE_INDEX_BUILDER_URL must be set}" + + curl -fsS -X PUT "$${OPENSEARCH_URL}/_snapshot/$${REMOTE_VECTOR_REPOSITORY}" \ + -H 'Content-Type: application/json' \ + -d "{\"type\":\"s3\",\"settings\":{\"bucket\":\"$${S3_BUCKET}\",\"base_path\":\"$${S3_PREFIX}\",\"region\":\"$${AWS_REGION}\"}}" + + curl -fsS -X PUT "$${OPENSEARCH_URL}/_cluster/settings" \ + -H 'Content-Type: application/json' \ + -d "{\"persistent\":{\"knn.remote_index_build.enabled\":\"true\",\"knn.remote_index_build.repository\":\"$${REMOTE_VECTOR_REPOSITORY}\",\"knn.remote_index_build.service.endpoint\":\"$${REMOTE_INDEX_BUILDER_URL}\"}}" + +volumes: + opensearch-data: diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml index bbba494ace..4b6300ecb8 100644 --- a/deploy/docker-compose.yml +++ b/deploy/docker-compose.yml @@ -30,6 +30,8 @@ # REMOTE_INDEX_BUILD Override GPU/CPU mode detection (true/false); normally auto-detected # REMOTE_BUILD_SIZE_MIN Optional minimum segment size override for remote builds # REMOTE_BUILD_TIMEOUT Remote build wait timeout in seconds (default: 1800) +# REMOTE_VECTOR_REPOSITORY Optional snapshot repository name (default: vector-repo) +# S3_PREFIX Optional S3 prefix for staged vectors/indexes (default: knn-indexes) # DATASET Dataset name (default: sift-128-euclidean) # BENCH_GROUPS Parameter sweep group: test | base (default: test) # K Number of neighbors to search for (default: 10) @@ -118,7 +120,9 @@ services: REMOTE_INDEX_BUILD: ${REMOTE_INDEX_BUILD:-} REMOTE_BUILD_SIZE_MIN: ${REMOTE_BUILD_SIZE_MIN:-} REMOTE_BUILD_TIMEOUT: ${REMOTE_BUILD_TIMEOUT:-1800} + REMOTE_VECTOR_REPOSITORY: ${REMOTE_VECTOR_REPOSITORY:-vector-repo} S3_BUCKET: ${S3_BUCKET:-} + S3_PREFIX: ${S3_PREFIX:-knn-indexes} DATASET: ${DATASET:-sift-128-euclidean} DATASET_PATH: /data/datasets BENCH_GROUPS: ${BENCH_GROUPS:-test} diff --git a/deploy/opensearch/Dockerfile b/deploy/opensearch/Dockerfile index 06be41a26c..9ab8bfcdd8 100644 --- a/deploy/opensearch/Dockerfile +++ b/deploy/opensearch/Dockerfile @@ -7,6 +7,7 @@ RUN /usr/share/opensearch/bin/opensearch-plugin install --batch repository-s3 # entrypoint.sh populates the keystore from AWS credential environment # variables when provided, then execs opensearch. +COPY opensearch.yml /usr/share/opensearch/config/opensearch.yml COPY --chmod=755 entrypoint.sh /entrypoint.sh ENTRYPOINT ["/entrypoint.sh"] From 9cf5c45ea07d884e07242f4727c027354e5232b0 Mon Sep 17 00:00:00 2001 From: James Bourbeau Date: Mon, 22 Jun 2026 16:22:03 -0500 Subject: [PATCH 9/9] Update Signed-off-by: James Bourbeau --- deploy/AWS_MULTINODE_SETUP.md | 364 ++++++++++++++++++++++ deploy/AWS_MULTINODE_SINGLE_ROLE_SETUP.md | 354 +++++++++++++++++++++ deploy/DEPLOYMENT.md | 10 +- deploy/README.md | 3 - deploy/bench/Dockerfile | 2 +- 5 files changed, 724 insertions(+), 9 deletions(-) create mode 100644 deploy/AWS_MULTINODE_SETUP.md create mode 100644 deploy/AWS_MULTINODE_SINGLE_ROLE_SETUP.md diff --git a/deploy/AWS_MULTINODE_SETUP.md b/deploy/AWS_MULTINODE_SETUP.md new file mode 100644 index 0000000000..cff5d973e4 --- /dev/null +++ b/deploy/AWS_MULTINODE_SETUP.md @@ -0,0 +1,364 @@ +# Opensearch + cuVS multi-node benchmarking setup + +This guide sets up the three-node version of the OpenSearch GPU benchmark stack. + +```text +client node --> runs the benchmark submitter +opensearch node --> runs OpenSearch +builder GPU node --> runs the remote index build service +``` + +The examples use `us-west-2`, but that region is not required. Choose any AWS region where your needed instance types, GPU AMI, and quotas are available, then keep all resources in that same region. + +The goal is to keep the deployment simple while separating the three major components onto their own EC2 instances. Docker Compose still runs locally on each instance; it does not create a cross-host network. Cross-node communication uses EC2 private DNS names or private IPv4 addresses. + +## 1. Choose one AWS region + +In the AWS Console, set the region selector to your chosen region. The examples below use: + +```text +US West (Oregon) us-west-2 +``` + +Use this same region for S3, EC2, security groups, and the instances' IAM roles. The EC2 instances and security groups should also be in the same VPC so private DNS, private IPs, and security-group source rules work as expected. + +When running commands, set both AWS region variables from one value: + +```bash +export AWS_DEFAULT_REGION=us-west-2 +export AWS_REGION="$AWS_DEFAULT_REGION" +``` + +`AWS_REGION` is used when registering the OpenSearch S3 repository region. `AWS_DEFAULT_REGION` is used by AWS CLI and boto-style tooling. Setting both from the same value avoids accidental mismatches. + +## 2. Create the S3 bucket + +Go to **S3 > Create bucket**. + +Use: + +```text +Bucket name: globally unique name +Region: your chosen AWS region, for example US West (Oregon) us-west-2 +Object Ownership: ACLs disabled +Block Public Access: block all public access +Encryption: SSE-S3 is fine +Versioning: optional +``` + +This bucket stores the remote-build staging objects, including vectors and +generated index artifacts. Benchmark datasets and result plots stay on the +client node under `DATASET_PATH`. + +## 3. Create an IAM policy for S3 + +Go to **IAM > Policies > Create policy > JSON**. + +Use this policy: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": ["s3:ListBucket"], + "Resource": "arn:aws:s3:::opensearch-cuvs-bench" + }, + { + "Effect": "Allow", + "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"], + "Resource": "arn:aws:s3:::opensearch-cuvs-bench/*" + } + ] +} +``` + +Name it: + +```text +opensearch-cuvs-bench-s3-policy +``` + +The delete permission lets the snapshot repository and staging workflow clean up +temporary objects when needed. + +## 4. Create the EC2 IAM role + +Go to **IAM > Roles > Create role**. + +Choose: + +```text +Trusted entity: AWS service +Use case: EC2 +``` + +Attach: + +```text +opensearch-cuvs-bench-s3-policy +AmazonSSMManagedInstanceCore +``` + +Name it: + +```text +opensearch-cuvs-bench-ec2-role +``` + +This role gives the instances refreshable S3 credentials and enables Session Manager access. Prefer this over fixed `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` values. + +## 5. Create security groups + +Go to **EC2 > Security Groups > Create security group**. + +Create these three security groups in the same VPC: + +```text +sg-cuvs-client +sg-cuvs-opensearch +sg-cuvs-builder +``` + +Inbound rules for `sg-cuvs-client`: + +```text +No inbound rules needed +``` + +Inbound rules for `sg-cuvs-opensearch`: + +```text +TCP 9200 from sg-cuvs-client +TCP 9200 from sg-cuvs-builder +``` + +Inbound rules for `sg-cuvs-builder`: + +```text +TCP 1025 from sg-cuvs-opensearch +TCP 1025 from sg-cuvs-client +``` + +Leave outbound as the default `allow all`. Use Session Manager instead of SSH if possible. If you need SSH, add TCP `22` only from your own IP. + +## 6. Launch the OpenSearch node + +Go to **EC2 > Instances > Launch instance**. + +Use: + +```text +Name: opensearch-cuvs-db +AMI: Ubuntu 22.04 or Amazon Linux 2023 +Instance type: r7i.xlarge, r7i.2xlarge, or m7i.2xlarge +Security group: sg-cuvs-opensearch +IAM role: opensearch-cuvs-bench-ec2-role +Storage: 100+ GiB gp3, larger if needed +``` + +Under **Advanced details > Metadata options**, use: + +```text +IMDS endpoint: Enabled +IMDSv2: Required +Hop limit: 2 +``` + +After launch, copy the instance's **Private IPv4 DNS** or **Private IPv4 address**. You will use it as `OPENSEARCH_URL`. + +## 7. Launch the GPU builder node + +Launch a second EC2 instance: + +```text +Name: opensearch-cuvs-builder +AMI: AWS Deep Learning Base AMI with CUDA, Ubuntu 22.04 +Instance type: g5.xlarge or g6.xlarge +Security group: sg-cuvs-builder +IAM role: opensearch-cuvs-bench-ec2-role +Storage: 100+ GiB gp3 +``` + +Use the same metadata options: + +```text +IMDS endpoint: Enabled +IMDSv2: Required +Hop limit: 2 +``` + +After launch, copy the instance's **Private IPv4 DNS** or **Private IPv4 address**. You will use it as `REMOTE_INDEX_BUILDER_URL`. + +## 8. Launch the client node + +Launch the benchmark submitter instance: + +```text +Name: opensearch-cuvs-client +AMI: Ubuntu 22.04 or Amazon Linux 2023 +Instance type: c7i.xlarge or m7i.xlarge +Security group: sg-cuvs-client +IAM role: opensearch-cuvs-bench-ec2-role +Storage: 50-100 GiB gp3 +``` + +Use the same metadata options: + +```text +IMDS endpoint: Enabled +IMDSv2: Required +Hop limit: 2 +``` + +## 9. Install Docker and Compose on each node + +Connect to each instance with **EC2 > Instances > Connect > Session Manager**. + +Install Docker and Docker Compose if they are not already installed. Then verify: + +```bash +docker compose version +``` + +On the GPU builder node, also verify GPU access: + +```bash +nvidia-smi +docker run --rm --gpus all nvidia/cuda:12.9.0-base-ubuntu22.04 nvidia-smi +``` + +If you want to avoid `sudo docker`, add your login user to the `docker` group and reconnect: + +```bash +sudo groupadd docker 2>/dev/null || true +sudo usermod -aG docker "$(whoami)" +``` + +## 10. Copy the deployment checkout to each node + +On all three nodes, clone or copy the `deploy-opensearch-tmp` branch of the +`jrbourbeau/cuvs` fork: + +```bash +git clone --branch deploy-opensearch-tmp https://github.com/jrbourbeau/cuvs.git +cd cuvs +``` + +Make sure this file is present: + +```text +docker-compose.multinode.yml +``` + +## 11. Start OpenSearch + +On the OpenSearch node: + +```bash +export S3_BUCKET=opensearch-cuvs-bench +export AWS_DEFAULT_REGION=us-west-2 +export AWS_REGION="$AWS_DEFAULT_REGION" +docker compose -f docker-compose.multinode.yml --profile opensearch up -d +``` + +Verify: + +```bash +curl http://localhost:9200 +``` + +## 12. Start the remote index builder + +On the GPU builder node: + +```bash +export S3_BUCKET=opensearch-cuvs-bench +export AWS_DEFAULT_REGION=us-west-2 +export AWS_REGION="$AWS_DEFAULT_REGION" +docker compose -f docker-compose.multinode.yml --profile builder up -d +``` + +Verify the container is running: + +```bash +docker ps +``` + +From the OpenSearch node, verify that the builder is reachable. OpenSearch is +the service that calls the remote builder: + +```bash +python3 -c 'import socket; socket.create_connection(("BUILDER_PRIVATE_DNS_OR_IP", 1025), 5).close(); print("builder reachable")' +``` + +From the client node, run the same check. The benchmark container also waits +for the builder before starting a remote-build run: + +```bash +python3 -c 'import socket; socket.create_connection(("BUILDER_PRIVATE_DNS_OR_IP", 1025), 5).close(); print("builder reachable")' +``` + +## 13. Configure OpenSearch from the client node + +On the client node: + +```bash +export OPENSEARCH_HOST=OPENSEARCH_PRIVATE_DNS_OR_IP +export OPENSEARCH_URL=http://${OPENSEARCH_HOST}:9200 +export REMOTE_INDEX_BUILDER_URL=http://BUILDER_PRIVATE_DNS_OR_IP:1025 +export S3_BUCKET=opensearch-cuvs-bench +export AWS_DEFAULT_REGION=us-west-2 +export AWS_REGION="$AWS_DEFAULT_REGION" +export S3_PREFIX=knn-indexes +export REMOTE_VECTOR_REPOSITORY=vector-repo +``` + +Then run the one-shot configure profile: + +```bash +docker compose -f docker-compose.multinode.yml --profile configure run --rm configure-remote-index-build +``` + +This registers the S3-backed remote vector repository and tells OpenSearch where the remote index builder service lives. + +## 14. Run the benchmark + +Still on the client node: + +```bash +export REMOTE_INDEX_BUILD=true +export DATASET_PATH="$(pwd)/opensearch-cuvs-datasets" +export DATASET=sift-128-euclidean +export BENCH_GROUPS=test +export K=10 +export BATCH_SIZE= +export BUILD_BATCH_SIZE= + +mkdir -p "${DATASET_PATH}" + +docker compose -f docker-compose.multinode.yml --profile client build bench +docker compose -f docker-compose.multinode.yml --profile client run --rm bench +``` + +## 15. Debug checklist + +From the client node, these should all work: + +```bash +curl "$OPENSEARCH_URL" +python3 -c 'import os, socket; from urllib.parse import urlparse; url = urlparse(os.environ["REMOTE_INDEX_BUILDER_URL"]); socket.create_connection((url.hostname, url.port or 1025), 5).close(); print("builder reachable")' +aws s3 ls s3://$S3_BUCKET --region "$AWS_DEFAULT_REGION" +``` + +If S3 fails, check the IAM role and S3 policy. If OpenSearch or builder +connectivity fails, check private DNS/IP values and security group source rules. + +## Notes + +- Keep the S3 bucket and EC2 resources in the same AWS region. The examples use `us-west-2`, but the setup is not region-specific. +- Within that region, keep all three instances and security groups in the same VPC. Prefer the same Availability Zone for the first benchmark run. +- Use private DNS or private IPs, not public IPs, for cross-node service traffic. +- Docker Compose networks are local to one host; Compose service names do not resolve across EC2 instances. +- The EC2 IAM role credentials rotate automatically. Avoid freezing temporary credentials into `.env` unless the application absolutely requires literal `AWS_*` variables. diff --git a/deploy/AWS_MULTINODE_SINGLE_ROLE_SETUP.md b/deploy/AWS_MULTINODE_SINGLE_ROLE_SETUP.md new file mode 100644 index 0000000000..c9d555bde1 --- /dev/null +++ b/deploy/AWS_MULTINODE_SINGLE_ROLE_SETUP.md @@ -0,0 +1,354 @@ +# AWS single-role multinode setup + +This guide sets up the three-node version of the OpenSearch GPU benchmark stack in `us-west-2`: + +```text +client node runs the benchmark submitter +opensearch node runs OpenSearch +builder GPU node runs the remote index build service +``` + +The goal is to keep the deployment simple while separating the three major components onto their own EC2 instances. Docker Compose still runs locally on each instance; it does not create a cross-host network. Cross-node communication uses EC2 private DNS names or private IPv4 addresses. + +## 1. Set the AWS region + +In the AWS Console, set the region selector to: + +```text +US West (Oregon) us-west-2 +``` + +Use this same region for S3, EC2, security groups, and the instances' IAM roles. + +## 2. Create the S3 bucket + +Go to **S3 > Create bucket**. + +Use: + +```text +Bucket name: globally unique name +Region: US West (Oregon) us-west-2 +Object Ownership: ACLs disabled +Block Public Access: block all public access +Encryption: SSE-S3 is fine +Versioning: optional +``` + +This bucket stores the remote-build staging objects, including vectors and +generated index artifacts. Benchmark datasets and result plots stay on the +client node under `DATASET_PATH`. + +## 3. Create an IAM policy for S3 + +Go to **IAM > Policies > Create policy > JSON**. + +Use this policy: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": ["s3:ListBucket"], + "Resource": "arn:aws:s3:::opensearch-cuvs-bench" + }, + { + "Effect": "Allow", + "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"], + "Resource": "arn:aws:s3:::opensearch-cuvs-bench/*" + } + ] +} +``` + +Name it: + +```text +opensearch-cuvs-bench-s3-policy +``` + +The delete permission lets the snapshot repository and staging workflow clean up +temporary objects when needed. + +## 4. Create the EC2 IAM role + +Go to **IAM > Roles > Create role**. + +Choose: + +```text +Trusted entity: AWS service +Use case: EC2 +``` + +Attach: + +```text +opensearch-cuvs-bench-s3-policy +AmazonSSMManagedInstanceCore +``` + +Name it: + +```text +opensearch-cuvs-bench-ec2-role +``` + +This role gives the instances refreshable S3 credentials and enables Session Manager access. Prefer this over fixed `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` values. + +## 5. Create security groups + +Go to **EC2 > Security Groups > Create security group**. + +Create these three security groups in the same VPC: + +```text +sg-cuvs-client +sg-cuvs-opensearch +sg-cuvs-builder +``` + +Inbound rules for `sg-cuvs-client`: + +```text +No inbound rules needed +``` + +Inbound rules for `sg-cuvs-opensearch`: + +```text +TCP 9200 from sg-cuvs-client +TCP 9200 from sg-cuvs-builder +``` + +Inbound rules for `sg-cuvs-builder`: + +```text +TCP 1025 from sg-cuvs-opensearch +TCP 1025 from sg-cuvs-client +``` + +Leave outbound as the default `allow all`. Use Session Manager instead of SSH if possible. If you need SSH, add TCP `22` only from your own IP. + +## 6. Launch the OpenSearch node + +Go to **EC2 > Instances > Launch instance**. + +Use: + +```text +Name: opensearch-cuvs-db +AMI: Ubuntu 22.04 or Amazon Linux 2023 +Instance type: r7i.xlarge, r7i.2xlarge, or m7i.2xlarge +Security group: sg-cuvs-opensearch +IAM role: opensearch-cuvs-bench-ec2-role +Storage: 100+ GiB gp3, larger if needed +``` + +Under **Advanced details > Metadata options**, use: + +```text +IMDS endpoint: Enabled +IMDSv2: Required +Hop limit: 2 +``` + +After launch, copy the instance's **Private IPv4 DNS** or **Private IPv4 address**. You will use it as `OPENSEARCH_URL`. + +## 7. Launch the GPU builder node + +Launch a second EC2 instance: + +```text +Name: opensearch-cuvs-builder +AMI: AWS Deep Learning Base AMI with CUDA, Ubuntu 22.04 +Instance type: g5.xlarge or g6.xlarge +Security group: sg-cuvs-builder +IAM role: opensearch-cuvs-bench-ec2-role +Storage: 100+ GiB gp3 +``` + +Use the same metadata options: + +```text +IMDS endpoint: Enabled +IMDSv2: Required +Hop limit: 2 +``` + +After launch, copy the instance's **Private IPv4 DNS** or **Private IPv4 address**. You will use it as `REMOTE_INDEX_BUILDER_URL`. + +## 8. Launch the client node + +Launch the benchmark submitter instance: + +```text +Name: opensearch-cuvs-client +AMI: Ubuntu 22.04 or Amazon Linux 2023 +Instance type: c7i.xlarge or m7i.xlarge +Security group: sg-cuvs-client +IAM role: opensearch-cuvs-bench-ec2-role +Storage: 50-100 GiB gp3 +``` + +Use the same metadata options: + +```text +IMDS endpoint: Enabled +IMDSv2: Required +Hop limit: 2 +``` + +## 9. Install Docker and Compose on each node + +Connect to each instance with **EC2 > Instances > Connect > Session Manager**. + +Install Docker and Docker Compose if they are not already installed. Then verify: + +```bash +docker compose version +``` + +On the GPU builder node, also verify GPU access: + +```bash +nvidia-smi +docker run --rm --gpus all nvidia/cuda:12.9.0-base-ubuntu22.04 nvidia-smi +``` + +If you want to avoid `sudo docker`, add your login user to the `docker` group and reconnect: + +```bash +sudo groupadd docker 2>/dev/null || true +sudo usermod -aG docker "$(whoami)" +``` + +## 10. Copy the deployment checkout to each node + +On all three nodes, clone or copy the `deploy-opensearch-tmp` branch of the +`jrbourbeau/cuvs` fork: + +```bash +git clone --branch deploy-opensearch-tmp https://github.com/jrbourbeau/cuvs.git +cd cuvs +``` + +Make sure this file is present: + +```text +docker-compose.multinode.yml +``` + +## 11. Start OpenSearch + +On the OpenSearch node: + +```bash +export S3_BUCKET=opensearch-cuvs-bench +export AWS_REGION=us-west-2 +export AWS_DEFAULT_REGION=us-west-2 +docker compose -f docker-compose.multinode.yml --profile opensearch up -d +``` + +Verify: + +```bash +curl http://localhost:9200 +``` + +## 12. Start the remote index builder + +On the GPU builder node: + +```bash +export S3_BUCKET=opensearch-cuvs-bench +export AWS_REGION=us-west-2 +export AWS_DEFAULT_REGION=us-west-2 +docker compose -f docker-compose.multinode.yml --profile builder up -d +``` + +Verify the container is running: + +```bash +docker ps +``` + +From the OpenSearch node, verify that the builder is reachable. OpenSearch is +the service that calls the remote builder: + +```bash +python3 -c 'import socket; socket.create_connection(("BUILDER_PRIVATE_DNS_OR_IP", 1025), 5).close(); print("builder reachable")' +``` + +From the client node, run the same check. The benchmark container also waits +for the builder before starting a remote-build run: + +```bash +python3 -c 'import socket; socket.create_connection(("BUILDER_PRIVATE_DNS_OR_IP", 1025), 5).close(); print("builder reachable")' +``` + +## 13. Configure OpenSearch from the client node + +On the client node: + +```bash +export S3_BUCKET=opensearch-cuvs-bench +export OPENSEARCH_HOST=OPENSEARCH_PRIVATE_DNS_OR_IP +export OPENSEARCH_PORT=9200 +export OPENSEARCH_URL=http://${OPENSEARCH_HOST}:${OPENSEARCH_PORT} +export REMOTE_INDEX_BUILDER_URL=http://BUILDER_PRIVATE_DNS_OR_IP:1025 +export AWS_REGION=us-west-2 +export AWS_DEFAULT_REGION=us-west-2 +export S3_PREFIX=knn-indexes +export REMOTE_VECTOR_REPOSITORY=vector-repo +``` + +Then run the one-shot configure profile: + +```bash +docker compose -f docker-compose.multinode.yml --profile configure run --rm configure-remote-index-build +``` + +This registers the S3-backed remote vector repository and tells OpenSearch where the remote index builder service lives. + +## 14. Run the benchmark + +Still on the client node: + +```bash +export REMOTE_INDEX_BUILD=true +export DATASET_PATH=/data/opensearch-cuvs-datasets +export DATASET=sift-128-euclidean +export BENCH_GROUPS=test +export K=10 +export BATCH_SIZE= +export BUILD_BATCH_SIZE= + +mkdir -p "${DATASET_PATH}" + +docker compose -f docker-compose.multinode.yml --profile client build bench +docker compose -f docker-compose.multinode.yml --profile client run --rm bench +``` + +## 15. Debug checklist + +From the client node, these should all work: + +```bash +curl "$OPENSEARCH_URL" +python3 -c 'import os, socket; from urllib.parse import urlparse; url = urlparse(os.environ["REMOTE_INDEX_BUILDER_URL"]); socket.create_connection((url.hostname, url.port or 1025), 5).close(); print("builder reachable")' +aws s3 ls s3://$S3_BUCKET --region us-west-2 +``` + +If S3 fails, check the IAM role and S3 policy. If OpenSearch or builder +connectivity fails, check private DNS/IP values and security group source rules. + +## Notes + +- Keep all three instances in `us-west-2`. +- Prefer the same VPC and same Availability Zone for the first benchmark run. +- Use private DNS or private IPs, not public IPs, for cross-node service traffic. +- Docker Compose networks are local to one host; Compose service names do not resolve across EC2 instances. +- The EC2 IAM role credentials rotate automatically. Avoid freezing temporary credentials into `.env` unless the application absolutely requires literal `AWS_*` variables. diff --git a/deploy/DEPLOYMENT.md b/deploy/DEPLOYMENT.md index a04bc259d7..4084eb4cbf 100644 --- a/deploy/DEPLOYMENT.md +++ b/deploy/DEPLOYMENT.md @@ -53,7 +53,7 @@ sudo sysctl -w vm.max_map_count=262144 Set the required bucket name: ```bash -export S3_BUCKET= +export S3_BUCKET=opensearch-cuvs-bench ``` If you are using static credentials instead of a default AWS credential provider, also export: @@ -78,17 +78,17 @@ docker compose --profile gpu up --build -d --wait opensearch remote-index-builde ## Connecting OpenSearch to the GPU builder -Before any index can use GPU builds, you need to register your S3 bucket as a snapshot repository and apply the cluster settings that point OpenSearch at the builder service. Run these once against a live cluster. +Before any index can use GPU builds, you need to register an S3-backed snapshot repository and apply the cluster settings that point OpenSearch at the builder service. Run these once against a live cluster. **Register S3 repository:** ```bash -curl -X PUT http://localhost:9200/_snapshot/ \ +curl -X PUT http://localhost:9200/_snapshot/vector-repo \ -H "Content-Type: application/json" \ -d '{ "type": "s3", "settings": { - "bucket": "", + "bucket": "opensearch-cuvs-bench", "base_path": "knn-indexes", "region": "us-west-2" } @@ -103,7 +103,7 @@ curl -X PUT http://localhost:9200/_cluster/settings \ -d '{ "persistent": { "knn.remote_index_build.enabled": true, - "knn.remote_index_build.repository": "", + "knn.remote_index_build.repository": "vector-repo", "knn.remote_index_build.service.endpoint": "http://remote-index-builder:1025" } }' diff --git a/deploy/README.md b/deploy/README.md index 337b2866d3..ae516a8e03 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -214,9 +214,6 @@ docker compose run --rm --no-deps \ -e BUILDER_URL=http://remote-index-builder:1025 \ -e S3_BUCKET=${S3_BUCKET} \ -e AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION} \ - -e S3_ACCESS_KEY=${AWS_ACCESS_KEY_ID} \ - -e S3_SECRET_KEY=${AWS_SECRET_ACCESS_KEY} \ - -e S3_SESSION_TOKEN=${AWS_SESSION_TOKEN} \ bench \ pytest /opt/cuvs/python/cuvs_bench/cuvs_bench/tests/test_opensearch.py -v -m integration ``` diff --git a/deploy/bench/Dockerfile b/deploy/bench/Dockerfile index c709e5af20..20adbca588 100644 --- a/deploy/bench/Dockerfile +++ b/deploy/bench/Dockerfile @@ -10,7 +10,7 @@ RUN apt-get update \ # nvcc, and cuvs_bench declares a hard dep on the `cuvs` CUDA package). # The opensearch backend is pure Python and needs neither, so we add the # package directly to PYTHONPATH and install only the actual runtime deps. -RUN git clone --depth=1 --filter=blob:none --sparse --branch cuvs-bench-opensearch https://github.com/jrbourbeau/cuvs.git /opt/cuvs \ +RUN git clone --depth=1 --filter=blob:none --sparse https://github.com/rapidsai/cuvs.git /opt/cuvs \ && cd /opt/cuvs \ && git sparse-checkout set python/cuvs_bench