Fix GPU and Java-heap OOM when building large vector indexes

[nvzm123]

Summary

Fixes two separate out-of-memory failures when building accelerated HNSW
indexes over large datasets:

• GPU / device-memory OOM: the dataset matrix was built with
  CuVSMatrix.deviceBuilder, which eagerly loads the entire dataset onto the
  GPU. This switches to CuVSMatrix.hostBuilder, which keeps the dataset in
  host memory and lets CAGRA stream it host→device as needed, rather than
  requiring the full dataset resident on the GPU at once.

• Java-heap OOM: HNSW graph construction previously materialized the full
  dataset on the Java heap (List<float[]>). It now streams subsets from a
  native host matrix during graph construction, so the full dataset is never
  held on the heap.

Supersedes #141. This branch was cut from latest main for a clean rebase and
contains all of #141's commits (through 7c5c448) plus the review-feedback
consolidation described below.

Review feedback from #141

Addresses @narangvivek10's four review comments:

1. Duplicate createMultiLayerHnswGraph (List vs CuVSMatrix overloads) —
   consolidated into a single CuVSMatrix-based method in
   AcceleratedHNSWUtils. All callers (flush, merge, and the binary/scalar
   quantized writers) now build a matrix and pass it. Migrating the quantized
   writers surfaced a latent byte-width bug: the quantized branch hardcoded
   (dimensions + 7) / 8 (binary packing), which is wrong for scalar
   quantization (one byte per dimension). Width is now taken from the matrix's
   column count, correct for both binary and scalar.

2. Redundant size parameter — removed from writeFieldInternal and
   createMultiLayerHnswGraph; derived internally from dataset.size().

3. Duplicate writeFieldInternal — consolidated into a single
   writeFieldInternal(FieldInfo, CuVSMatrix) in
   Lucene99AcceleratedHNSWVectorsWriter, with a thin List adapter for the
   flush/sorting path.

4. Utils host/device flexibility — the CuVSResources parameter on the
   matrix builders was unused after the switch to hostBuilder, so it's
   removed, and the docs that still referred to "device memory" are corrected.
   Removing it updated call sites in CuVS2510GPUVectorsWriter and the
   binary/scalar quantized writers. Making the builders fully
   host/device-selectable seems broader than this PR — device construction
   still exists directly in CuVS2510GPUVectorsReader, so a proper
   host/device-aware API would want to account for that. Would you prefer I
   file a separate issue to track it, or leave it until something actually
   needs device-side construction through these helpers? Happy either way.

Testing

All format and merge suites pass locally:

• TestMerge (including testLargeScaleMerge with -DlargeScale=true)
• TestCuVSVectorsFormat
• TestLucene99AcceleratedHNSWVectorsFormat
• TestQuantizedVectorsFormats

Flaky test note: TestLucene99AcceleratedHNSWVectorsFormat.testCheckIntegrityReadsAllBytes
fails intermittently (~2 in 6 full-suite runs) via CheckIndex in
testRandomWithUpdatesAndGraph. It passes 100% in isolation and fails at the
same rate on the base branch without the consolidation, so it appears to be a
pre-existing, seed-dependent suite-level issue rather than something introduced
here. Flagging in case CI hits it.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[nvzm123]

cc: @narangvivek10 @kshitizgupta21 @singhmanas1 @cjnolet

[jamxia155]

I think the two main fixes you made are essential. Thanks, @nvzm123!

Re: the MAX_HNSW_LAYERS change, could you please mention it in the summary for the record?