[META] Streams using Apache Arrow and Flight

[rishabhmaurya]

Please describe the end goal of this project

• In-memory columnar representation of any intermediate results from search
• • Data adjacency for sequential access (scans)
• • O(1) (constant-time) random access
• • SIMD and vectorization-friendly
• • Relocatable without “pointer swizzling”, allowing for true zero-copy access in shared memory.
• Interoperable representation of columnar data to be used across different engines like sharing between opensearch and datafusion, which is a rust based engine.
• RPC using bidirectional streams: making use of GRPC bidirectional streams handling backpressure from the client in realtime and producing batches of records on demand. Used both for internode communication (between data nodes and cordinator) as well as communication with end client.

Use cases

• Optimize memory overhead, cpu utilization and performance for -
• • Search pagination API
• • Aggregation (more details to follow) .
• • JOINs in Opensearch.

Apache Arrow will serve as a library for in-memory columnar representation on any transient results used for retrieval in these use cases. Arrow Flight to be used for stream RPC.

Supporting References

JOINs RFC making use of this integration - #15185

Issues

Basic Framework Changes:

• Library changes containing POJOs and Arrow vector APIs - Library changes for Apache Arrow integration #16691
• OpenSearch basic Arrow Flight server and client implementation as a separate module. [Feature Request] Arrow Flight server and client bootstrap logic #16963
• • Integration with server module via plugin interface [Feature Request] Arrow Flight server and client bootstrap logic #16963
• • TLS integration for Flight server and client [FEATURE] Support additional params in SecureTransportSettingsProvider to enable building of SslContext outside of security plugin security#5011 & [Feature Request] Arrow Flight server and client bootstrap logic #16963
• • Hot reload of SSL certificate - [Feature Request] Handle certificate hot reload for Flight server #16965
• • Security plugin changes to expose parameters to build SslContext using SecureTransportSettingsProvider [FEATURE] Support additional params in SecureTransportSettingsProvider to enable building of SslContext outside of security plugin security#5011 . PR Adapt to core changes in SecureTransportParameters security#5122
• • Auth for flight server - [FEATURE] Provide SSL configuration for auxiliary transports security#5104
• • Revisit defaults for server, client and allocator configs after benchmarks [Feature Request] Tune up server config for Flight server #16966
• General purpose FlightProducer supporting basic getStream() and getFlightInfo() APIs. [Feature Request] Base Flight producer for FlightServer #17065
• • ProxyStreamProducer acting as a proxy stream connecting the right data node holding the stream for a given ticket to the client.
• • [Feature Request] Enable Apache Arrow classes to be shared among plugins #17671
• • partitioned stream support - [Feature Request] Partitioned stream support in Stream framework #17493

Client support

• RFC [Feature Request] OpenSearchArrowClient (OSAC) to support consuming Arrow Flight Streams opensearch-java#1502

Migrate Search API

• Ideas to support any existing transport API to use flight transport [Feature Request] Enable transport request response to be returned in Arrow format #17695
• support search API using flight transport - [Feature Request] Integrate the Search API with Arrow based Flight Streams #17696

Aggregations

• RFC [RFC] Streaming Aggregation - A Memory-Efficient Approach #16774
• Meta task - [META] Streaming terms aggregation #17278

Operational excellence

• Stream cancellation, error handling and renewal.
• Metrics & Troubleshooting

Documentation

Related component

Search