feat(storage): define vector index metadata

[wenym1]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

In this PR, we define the vector index metadata in hummock. The vector index will have a separate map field in HummockVersion, keyed by the table_id of the vector index. We also define the VectorIndexDelta in HummockVersionDelta to represent the changes to apply on the vector index. VectorIndexDelta can be either Init or Adds. Init consists of the initial config of the vector index, while Adds consists of the data later added by the vector index writer.

The GC and time travel logic is also updated, so that the vector index files can be tracked and properly GCed, and we can reuse the time travel mechanism of LSM for vector index.

Checklist

• I have written necessary rustdoc comments.
• I have added necessary unit tests and integration tests.
• I have added test labels as necessary.
• I have added fuzzing tests or opened an issue to track them.
• My PR contains breaking changes.
• My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
• I have checked the Release Timeline and Currently Supported Versions to determine which release branches I need to cherry-pick this PR into.

Documentation

• My PR needs documentation updates.

Release note

[wenym1]

• feat(storage): support hummock FLAT index  #21820
• feat(storage): define vector index metadata #21885 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.