Add stream search feature flag and auto fallback logic

[bowenlan-amzn]

Description

This PR changes the way to enable stream search feature.
Previously it's through query param, now change to a feature flag.

If enable stream transport and search feature flag at the same time, user can try out the stream search feature. Example gradle run command below.

./gradlew run -PinstalledPlugins="['arrow-flight-rpc']" \
-Dtests.opensearch.opensearch.experimental.feature.transport.stream.enabled=true \
-Dtests.opensearch.opensearch.experimental.feature.stream.search.enabled=true

• The auto fallback logic still pending. We need to understand what search and aggregation are supported.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 90ccce3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for bb3e8e2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for bb3e8e2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 3cb08c3: SUCCESS

[codecov]

Codecov Report

❌ Patch coverage is 68.75000% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.98%. Comparing base (2c6a404) to head (a74ef16).
⚠️ Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
...pensearch/rest/action/search/RestSearchAction.java	64.28%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19373      +/-   ##
============================================
+ Coverage     72.95%   72.98%   +0.03%     
- Complexity    70115    70123       +8     
============================================
  Files          5687     5688       +1     
  Lines        321703   321804     +101     
  Branches      46534    46550      +16     
============================================
+ Hits         234702   234883     +181     
+ Misses        68074    67917     -157     
- Partials      18927    19004      +77     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

❌ Gradle check result for 1fc5227: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[rishabhmaurya]

@bowenlan-amzn looks good, couple of minor comments.

[github-actions]

❌ Gradle check result for 4ece007: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8d53534: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 8d53534: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

❕ Gradle check result for a74ef16: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[rishabhmaurya]

@bowenlan-amzn on thinking more about this feature flag, it makes more sense to me to have it as a dynamic cluster setting with default as off. Let me know what do you think?
cc @harshavamsi