Merge branch 'main' into fix_loop_in_updateRep

Signed-off-by: kkewwei <[email protected]>

Author: kkewwei

.github/workflows/gradle-check.yml

@@ -24,7 +24,7 @@ jobs:
     - uses: actions/checkout@v4
     - name: Get changed files
       id: changed-files-specific
-      uses: tj-actions/[email protected].4
+      uses: tj-actions/[email protected].5
       with:
         files_ignore: |
           release-notes/*.md

CHANGELOG.md

@@ -5,81 +5,20 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 ## [Unreleased 3.x]
 ### Added
-- Change priority for scheduling reroute during timeout([#16445](https://github.com/opensearch-project/OpenSearch/pull/16445))
-- [Rule Based Auto-tagging] Add rule schema for auto tagging ([#17238](https://github.com/opensearch-project/OpenSearch/pull/17238))
-- Renaming the node role search to warm ([#17573](https://github.com/opensearch-project/OpenSearch/pull/17573))
-- Introduce a new search node role to hold search only shards ([#17620](https://github.com/opensearch-project/OpenSearch/pull/17620))
-- Fix systemd integTest on deb regarding path ownership check ([#17641](https://github.com/opensearch-project/OpenSearch/pull/17641))
-- Add dfs transformation function in XContentMapValues ([#17612](https://github.com/opensearch-project/OpenSearch/pull/17612))
-- Add tracking for long-running SearchTask post cancellation ([#17726](https://github.com/opensearch-project/OpenSearch/pull/17726))
-- Added Kinesis support as a plugin for the pull-based ingestion ([#17615](https://github.com/opensearch-project/OpenSearch/pull/17615))
-- Add FilterFieldType for developers who want to wrap MappedFieldType ([#17627](https://github.com/opensearch-project/OpenSearch/pull/17627))
-- [Rule Based Auto-tagging] Add in-memory rule processing service ([#17365](https://github.com/opensearch-project/OpenSearch/pull/17365))
-- [Security Manager Replacement] Create initial Java Agent to intercept Socket::connect calls ([#17724](https://github.com/opensearch-project/OpenSearch/pull/17724))
-- Faster `terms_query` with already sorted terms ([#17714](https://github.com/opensearch-project/OpenSearch/pull/17714))
-- Add ingestion management APIs for pause, resume and get ingestion state ([#17631](https://github.com/opensearch-project/OpenSearch/pull/17631))
-- [Security Manager Replacement] Enhance Java Agent to intercept System::exit ([#17746](https://github.com/opensearch-project/OpenSearch/pull/17746))
-- [Security Manager Replacement] Add a policy parser for Java agent security policies ([#17753](https://github.com/opensearch-project/OpenSearch/pull/17753))
-- [Security Manager Replacement] Implement File Interceptor and add integration tests ([#17760](https://github.com/opensearch-project/OpenSearch/pull/17760))
-- [Security Manager Replacement] Enhance Java Agent to intercept Runtime::halt ([#17757](https://github.com/opensearch-project/OpenSearch/pull/17757))
-- [Security Manager Replacement] Phase off SecurityManager usage in favor of Java Agent ([#17861](https://github.com/opensearch-project/OpenSearch/pull/17861))
-- Support AutoExpand for SearchReplica ([#17741](https://github.com/opensearch-project/OpenSearch/pull/17741))
-- Add TLS enabled SecureNetty4GrpcServerTransport ([#17796](https://github.com/opensearch-project/OpenSearch/pull/17796))
-- Implement fixed interval refresh task scheduling ([#17777](https://github.com/opensearch-project/OpenSearch/pull/17777))
-- [Tiered caching] Create a single cache manager for all the disk caches. ([#17513](https://github.com/opensearch-project/OpenSearch/pull/17513))
-- Add GRPC DocumentService and Bulk endpoint ([#17727](https://github.com/opensearch-project/OpenSearch/pull/17727))
-- Added scale to zero (`search_only` mode) support for OpenSearch reader writer separation ([#17299](https://github.com/opensearch-project/OpenSearch/pull/17299)
-- [Star Tree] [Search] Resolving numeric range aggregation with metric aggregation using star-tree ([#17273](https://github.com/opensearch-project/OpenSearch/pull/17273))
-- Added Search Only strict routing setting ([#17803](https://github.com/opensearch-project/OpenSearch/pull/17803))
-- Disable the index API for ingestion engine ([#17768](https://github.com/opensearch-project/OpenSearch/pull/17768))
-- Add SearchService and Search GRPC endpoint ([#17830](https://github.com/opensearch-project/OpenSearch/pull/17830))
-- Add update and delete support in pull-based ingestion ([#17822](https://github.com/opensearch-project/OpenSearch/pull/17822))
-- Allow maxPollSize and pollTimeout in IngestionSource to be configurable ([#17863](https://github.com/opensearch-project/OpenSearch/pull/17863))
 
 ### Changed
-- Migrate BC libs to their FIPS counterparts ([#14912](https://github.com/opensearch-project/OpenSearch/pull/14912))
-- Increase the floor segment size to 16MB ([#17699](https://github.com/opensearch-project/OpenSearch/pull/17699))
-- Unwrap singleton DocValues in global ordinal value source of composite histogram aggregation ([#17740](https://github.com/opensearch-project/OpenSearch/pull/17740))
-- Unwrap singleton DocValues in date histogram aggregation. ([#17643](https://github.com/opensearch-project/OpenSearch/pull/17643))
-- Introduce 512 byte limit to search and ingest pipeline IDs ([#17786](https://github.com/opensearch-project/OpenSearch/pull/17786))
-- Avoid skewed segment replication lag metric ([#17831](https://github.com/opensearch-project/OpenSearch/pull/17831))
-- Increase the default segment counter step size when replica promoting ([#17568](https://github.com/opensearch-project/OpenSearch/pull/17568))
 
 ### Dependencies
-- Bump `com.nimbusds:nimbus-jose-jwt` from 9.41.1 to 10.0.2 ([#17607](https://github.com/opensearch-project/OpenSearch/pull/17607), [#17669](https://github.com/opensearch-project/OpenSearch/pull/17669))
-- Bump `com.google.api:api-common` from 1.8.1 to 2.46.1 ([#17604](https://github.com/opensearch-project/OpenSearch/pull/17604))
-- Bump `ch.qos.logback:logback-core` from 1.5.16 to 1.5.18 ([#17609](https://github.com/opensearch-project/OpenSearch/pull/17609), [#17809](https://github.com/opensearch-project/OpenSearch/pull/17809))
-- Bump `org.jruby.joni:joni` from 2.2.3 to 2.2.6 ([#17608](https://github.com/opensearch-project/OpenSearch/pull/17608), [#17732](https://github.com/opensearch-project/OpenSearch/pull/17732))
-- Bump `dangoslen/dependabot-changelog-helper` from 3 to 4 ([#17498](https://github.com/opensearch-project/OpenSearch/pull/17498))
-- Bump `com.google.api:gax` from 2.35.0 to 2.63.1 ([#17465](https://github.com/opensearch-project/OpenSearch/pull/17465))
-- Bump `com.azure:azure-storage-blob` from 12.29.1 to 12.30.0 ([#17667](https://github.com/opensearch-project/OpenSearch/pull/17667))
-- Bump `tj-actions/changed-files` from 46.0.1 to 46.0.4 ([#17666](https://github.com/opensearch-project/OpenSearch/pull/17666), [#17813](https://github.com/opensearch-project/OpenSearch/pull/17813))
-- Bump `com.google.code.gson:gson` from 2.11.0 to 2.12.1 ([#17668](https://github.com/opensearch-project/OpenSearch/pull/17668))
-- Bump `com.github.luben:zstd-jni` from 1.5.5-1 to 1.5.6-1 ([#17674](https://github.com/opensearch-project/OpenSearch/pull/17674))
-- Bump `lycheeverse/lychee-action` from 2.3.0 to 2.4.0 ([#17731](https://github.com/opensearch-project/OpenSearch/pull/17731))
-- Bump `com.netflix.nebula.ospackage-base` from 11.11.1 to 11.11.2 ([#17734](https://github.com/opensearch-project/OpenSearch/pull/17734))
-- Bump `com.nimbusds:oauth2-oidc-sdk` from 11.21 to 11.23.1 ([#17729](https://github.com/opensearch-project/OpenSearch/pull/17729))
-- Bump `com.google.api.grpc:proto-google-common-protos` from 2.52.0 to 2.54.1 ([#17733](https://github.com/opensearch-project/OpenSearch/pull/17733))
-- Bump `ch.qos.logback:logback-classic` from 1.5.17 to 1.5.18 ([#17730](https://github.com/opensearch-project/OpenSearch/pull/17730))
-- Bump `reactor_netty` from 1.1.26 to 1.2.3 ([#17322](https://github.com/opensearch-project/OpenSearch/pull/17322), [#17377](https://github.com/opensearch-project/OpenSearch/pull/17377))
-- Bump `com.google.api.grpc:proto-google-iam-v1` from 1.33.0 to 1.49.1 ([#17811](https://github.com/opensearch-project/OpenSearch/pull/17811))
-- Bump `com.azure:azure-core` from 1.54.1 to 1.55.3 ([#17810](https://github.com/opensearch-project/OpenSearch/pull/17810))
-- Bump `org.apache.poi` version from 5.2.5 to 5.4.1 in /plugins/ingest-attachment ([#17887](https://github.com/opensearch-project/OpenSearch/pull/17887))
 
 ### Changed
 
 ### Deprecated
 
 ### Removed
-- Remove deprecated `batch_size` parameter from `_bulk` ([#14283](https://github.com/opensearch-project/OpenSearch/issues/14283))
-- Remove `FeatureFlags.APPROXIMATE_POINT_RANGE_QUERY_SETTING` since range query approximation is no longer experimental ([#17769](https://github.com/opensearch-project/OpenSearch/pull/17769))
 
 ### Fixed
-- Fix bytes parameter on `_cat/recovery` ([#17598](https://github.com/opensearch-project/OpenSearch/pull/17598))
-- Fix slow performance of FeatureFlag checks ([#17611](https://github.com/opensearch-project/OpenSearch/pull/17611))
-- Fix shard recovery in pull-based ingestion to avoid skipping messages ([#17868](https://github.com/opensearch-project/OpenSearch/pull/17868)))
 - Fix simultaneously creating a snapshot and updating the repository can potentially trigger an infinite loop ([#17532](https://github.com/opensearch-project/OpenSearch/pull/17532))
 
 ### Security
 
-[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/f58d846f...main
+[Unreleased 3.x]: https://github.com/opensearch-project/OpenSearch/compare/aa0e724e...main

README.md

@@ -61,3 +61,4 @@ Copyright OpenSearch Contributors. See [NOTICE](NOTICE.txt) for details.
 OpenSearch is a registered trademark of Amazon Web Services.
 
 OpenSearch includes certain Apache-licensed Elasticsearch code from Elasticsearch B.V. and other source code. Elasticsearch B.V. is not the source of that other source code. ELASTICSEARCH is a registered trademark of Elasticsearch B.V.
+

buildSrc/src/main/java/org/opensearch/gradle/agent/JavaAgent.java

@@ -0,0 +1,71 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.gradle.agent;
+
+import org.gradle.api.Plugin;
+import org.gradle.api.Project;
+import org.gradle.api.artifacts.Configuration;
+import org.gradle.api.tasks.Copy;
+import org.gradle.api.tasks.TaskProvider;
+import org.gradle.api.tasks.testing.Test;
+
+import java.io.File;
+import java.util.Objects;
+
+/**
+ * Gradle plugin to automatically configure the OpenSearch Java agent
+ * for test tasks in OpenSearch plugin projects.
+ */
+public class JavaAgent implements Plugin<Project> {
+
+    /**
+     * Plugin implementation that sets up java agent configuration and applies it to test tasks.
+     */
+    @Override
+    public void apply(Project project) {
+        Configuration agentConfiguration = project.getConfigurations().findByName("agent");
+        if (agentConfiguration == null) {
+            agentConfiguration = project.getConfigurations().create("agent");
+        }
+
+        project.afterEvaluate(p -> {
+            String opensearchVersion = getOpensearchVersion(p);
+            p.getDependencies().add("agent", "org.opensearch:opensearch-agent-bootstrap:" + opensearchVersion);
+            p.getDependencies().add("agent", "org.opensearch:opensearch-agent:" + opensearchVersion);
+        });
+
+        Configuration finalAgentConfiguration = agentConfiguration;
+        TaskProvider<Copy> prepareJavaAgent = project.getTasks().register("prepareJavaAgent", Copy.class, task -> {
+            task.from(finalAgentConfiguration);
+            task.into(new File(project.getBuildDir(), "agent"));
+        });
+
+        project.getTasks().withType(Test.class).configureEach(testTask -> {
+            testTask.dependsOn(prepareJavaAgent);
+
+            final String opensearchVersion = getOpensearchVersion(project);
+
+            testTask.doFirst(task -> {
+                File agentJar = new File(project.getBuildDir(), "agent/opensearch-agent-" + opensearchVersion + ".jar");
+
+                testTask.jvmArgs("-javaagent:" + agentJar.getAbsolutePath());
+            });
+        });
+    }
+
+    /**
+     * Gets the OpenSearch version from project properties, with a fallback default.
+     *
+     * @param project The Gradle project
+     * @return The OpenSearch version to use
+     */
+    private String getOpensearchVersion(Project project) {
+        return Objects.requireNonNull(project.property("opensearch_version")).toString();
+    }
+}

buildSrc/src/main/resources/META-INF/gradle-plugins/opensearch.java-agent.properties

@@ -0,0 +1,9 @@
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+#
+
+implementation-class=org.opensearch.gradle.agent.JavaAgent

buildSrc/src/test/java/org/opensearch/gradle/agent/JavaAgentTests.java

@@ -0,0 +1,66 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.gradle.agent;
+
+import org.opensearch.gradle.test.GradleUnitTestCase;
+import org.gradle.api.Project;
+import org.gradle.api.artifacts.Configuration;
+import org.gradle.api.tasks.Copy;
+import org.gradle.testfixtures.ProjectBuilder;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+
+public class JavaAgentTests extends GradleUnitTestCase {
+    private TemporaryFolder projectDir;
+    private final String PREPARE_JAVA_AGENT_TASK = "prepareJavaAgent";
+
+    @Before
+    public void setUp() throws IOException {
+        projectDir = new TemporaryFolder();
+        projectDir.create();
+    }
+
+    @After
+    public void tearDown() {
+        projectDir.delete();
+    }
+
+    /**
+     * This test is used to verify that adding the 'opensearch.java-agent' to the project
+     * creates the necessary agent configuration and tasks. This is basically
+     * a behavioral test of the {@link JavaAgent#apply(Project)} method.
+     */
+    @Test
+    public void applyJavaAgentPlugin() {
+        // Create an empty project and apply the JavaAgent plugin
+        Project project = ProjectBuilder.builder().build();
+        project.getPluginManager().apply(JavaAgent.class);
+
+        // Verify the agent configuration was created
+        Configuration agentConfig = project.getConfigurations().findByName("agent");
+        assertNotNull("Agent configuration should be created", agentConfig);
+
+        // Verify the prepareJavaAgent task was created and is of the right type
+        assertNotNull("prepareJavaAgent task should be created", project.getTasks().findByName(PREPARE_JAVA_AGENT_TASK));
+        assertTrue("prepareJavaAgent task should be of type Copy", project.getTasks().findByName(PREPARE_JAVA_AGENT_TASK) instanceof Copy);
+
+        // Verify the destination directory of the Copy task
+        Copy prepareTask = (Copy) project.getTasks().findByName(PREPARE_JAVA_AGENT_TASK);
+        assertEquals(
+            "Destination directory should be build/agent",
+            new File(project.getBuildDir(), "agent"),
+            prepareTask.getDestinationDir()
+        );
+    }
+}

distribution/packages/build.gradle

@@ -175,6 +175,9 @@ Closure commonPackageConfig(String type, boolean jdk, String architecture) {
           with jdkFiles(project, 'linux', architecture)
         }
       }
+      into('agent') {
+        with agentFiles()
+      }
       // we need to specify every intermediate directory in these paths so the package managers know they are explicitly
       // intended to manage them; otherwise they may be left behind on uninstallation. duplicate calls of the same
       // directory are fine

distribution/src/config/jvm.options

@@ -85,3 +85,4 @@ ${error.file}
 23:-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.asTypeUncached
 
 21-:-javaagent:agent/opensearch-agent.jar
+21-:--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED

gradle/ide.gradle

@@ -49,10 +49,6 @@ if (System.getProperty('idea.active') == 'true') {
     }
   }
 
-  buildScan {
-    server = 'https://127.0.0.1'
-  }
-
   idea {
     project {
       vcs = 'Git'

gradle/run.gradle

@@ -43,9 +43,17 @@ testClusters {
       installedPlugins = Eval.me(installedPlugins)
       for (String p : installedPlugins) {
         plugin('plugins:'.concat(p))
+        if (p.equals("arrow-flight-rpc")) {
+          // Add system properties for Netty configuration
+          systemProperty 'io.netty.allocator.numDirectArenas', '1'
+          systemProperty 'io.netty.noUnsafe', 'false'
+          systemProperty 'io.netty.tryUnsafe', 'true'
+          systemProperty 'io.netty.tryReflectionSetAccessible', 'true'
+        }
       }
     }
   }
+
 }
 
 tasks.register("run", RunTask) {

libs/agent-sm/agent-policy/src/main/java/org/opensearch/secure_sm/policy/PolicyFile.java

@@ -49,7 +49,8 @@ public class PolicyFile extends java.security.Policy {
         "org.bouncycastle.crypto.CryptoServicesPermission",
         "org.opensearch.script.ClassPermission",
         "javax.security.auth.AuthPermission",
-        "javax.security.auth.kerberos.ServicePermission"
+        "javax.security.auth.kerberos.ServicePermission",
+        "com.sun.tools.attach.AttachPermission"
     );
 
     private final PolicyInfo policyInfo;

libs/arrow-spi/build.gradle

@@ -1,9 +1,6 @@
 ---
 "get cluster stats returns analysis stats":
 
-  - skip:
-      version: " - 7.6.99"
-      reason:  "analysis stats are added for v7.7.0"
 
   - do:
       cluster.stats: {}

modules/analysis-common/src/yamlRestTest/resources/rest-api-spec/test/cluster.stats/10_analysis_stats.yml

@@ -22,9 +22,6 @@ setup:
 
 ---
 "Test use_field":
-  - skip:
-      version: " - 7.1.99"
-      reason: "Implemented in 7.2"
   - do:
       search:
         index: test

modules/analysis-common/src/yamlRestTest/resources/rest-api-spec/test/search.query/70_intervals.yml

@@ -83,9 +83,6 @@ setup:
         index: [test, other]
 ---
 "Simple Composite aggregation with GeoTile grid":
-  - skip:
-      version: " - 7.4.99"
-      reason: geotile_grid is not supported until 7.5.0
   - do:
       search:
         rest_total_hits_as_int: true
@@ -127,9 +124,6 @@ setup:
 
 ---
 "Simple Composite aggregation with geotile grid add aggregate after":
-  - skip:
-      version: " - 7.4.99"
-      reason: geotile_grid is not supported until 7.5.0
   - do:
       search:
         index: test

modules/geo/src/yamlRestTest/resources/rest-api-spec/test/geo_shape/230_composite.yml

@@ -1,8 +1,6 @@
 ---
 "Script errors contain position":
-  - skip:
-      version: " - 7.7.0"
-      reason: "position introduced in 7.7"
+
 
   - do:
       catch: /compile error/

modules/lang-painless/src/yamlRestTest/resources/rest-api-spec/test/painless/17_update_error.yml

@@ -500,9 +500,7 @@
 
 ---
 "Script Score With Highlight":
-  - skip:
-      version: " - 7.4.0"
-      reason: "highlight for script_score was introduced in 7.4.1"
+
 
   - do:
       indices.create:

modules/lang-painless/src/yamlRestTest/resources/rest-api-spec/test/painless/80_script_score.yml

@@ -1,10 +1,5 @@
 # Integration tests for ScriptScoreQuery using Painless
 
-setup:
-- skip:
-    version: " - 7.1.99"
-    reason: "random score function of script score was added in 7.2"
-
 ---
 "Random score function with _seq_no field":
     - do: