Merge branch 'main' into result-consumer-harden

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

Author: harshavamsi

CHANGELOG.md

@@ -17,6 +17,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Add a dynamic setting to change skip_cache_factor and min_frequency for querycache ([#18351](https://github.com/opensearch-project/OpenSearch/issues/18351))
 - Add overload constructor for Translog to accept Channel Factory as a parameter ([#18918](https://github.com/opensearch-project/OpenSearch/pull/18918))
 - Add subdirectory-aware store module with recovery support ([#19132](https://github.com/opensearch-project/OpenSearch/pull/19132))
+- [Rule-based Auto-tagging] Modify get rule api to suit nested attributes ([#19429](https://github.com/opensearch-project/OpenSearch/pull/19429))
+- [Rule-based Auto-tagging] Add autotagging label resolving logic for multiple attributes ([#19486](https://github.com/opensearch-project/OpenSearch/pull/19486))
 - Field collapsing supports search_after ([#19261](https://github.com/opensearch-project/OpenSearch/pull/19261))
 - Add a dynamic cluster setting to control the enablement of the merged segment warmer ([#18929](https://github.com/opensearch-project/OpenSearch/pull/18929))
 - Publish transport-grpc-spi exposing QueryBuilderProtoConverter and QueryBuilderProtoConverterRegistry ([#18949](https://github.com/opensearch-project/OpenSearch/pull/18949))
@@ -43,7 +45,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Implement GRPC GeoBoundingBox, GeoDistance queries ([#19451](https://github.com/opensearch-project/OpenSearch/pull/19451))
 - Implement GRPC Ids, Range, and Terms Set queries ([#19448](https://github.com/opensearch-project/OpenSearch/pull/19448))
 - Implement GRPC Nested query ([#19453](https://github.com/opensearch-project/OpenSearch/pull/19453))
+- Add sub aggregation support for histogram aggregation using skiplist ([19438](https://github.com/opensearch-project/OpenSearch/pull/19438))
 - Optimization in String Terms Aggregation query for Large Bucket Counts([#18732](https://github.com/opensearch-project/OpenSearch/pull/18732))
+- New cluster setting search.query.max_query_string_length ([#19491](https://github.com/opensearch-project/OpenSearch/pull/19491))
 - Harden the circuit breaker and failure handle logic in query result consumer ([#19396](https://github.com/opensearch-project/OpenSearch/pull/19396))
 
 ### Changed
@@ -97,6 +101,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Cache serialised cluster state based on cluster state version and node version.([#19307](https://github.com/opensearch-project/OpenSearch/pull/19307))
 - Fix stats API in store-subdirectory module's SubdirectoryAwareStore ([#19470](https://github.com/opensearch-project/OpenSearch/pull/19470))
 - Handle negative search request nodes stats ([#19340](https://github.com/opensearch-project/OpenSearch/pull/19340))
+- Remove unnecessary iteration per-shard in request cache cleanup ([#19263](https://github.com/opensearch-project/OpenSearch/pull/19263))
 
 ### Dependencies
 - Bump `com.gradleup.shadow:shadow-gradle-plugin` from 8.3.5 to 8.3.9 ([#19400](https://github.com/opensearch-project/OpenSearch/pull/19400))
@@ -108,7 +113,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `com.azure:azure-storage-common` from 12.29.1 to 12.30.2 ([#19016](https://github.com/opensearch-project/OpenSearch/pull/19016), [#19145](https://github.com/opensearch-project/OpenSearch/pull/19145))
 - Update OpenTelemetry to 1.53.0 and OpenTelemetry SemConv to 1.34.0 ([#19068](https://github.com/opensearch-project/OpenSearch/pull/19068))
 - Bump `1password/load-secrets-action` from 2 to 3 ([#19100](https://github.com/opensearch-project/OpenSearch/pull/19100))
-- Bump `com.nimbusds:nimbus-jose-jwt` from 10.3 to 10.5 ([#19099](https://github.com/opensearch-project/OpenSearch/pull/19099), [#19101](https://github.com/opensearch-project/OpenSearch/pull/19101), [#19254](https://github.com/opensearch-project/OpenSearch/pull/19254))
+- Bump `com.nimbusds:nimbus-jose-jwt` from 10.3 to 10.5 ([#19099](https://github.com/opensearch-project/OpenSearch/pull/19099), [#19101](https://github.com/opensearch-project/OpenSearch/pull/19101), [#19254](https://github.com/opensearch-project/OpenSearch/pull/19254), [#19362](https://github.com/opensearch-project/OpenSearch/pull/19362))
 - Bump netty from 4.1.121.Final to 4.1.125.Final ([#19103](https://github.com/opensearch-project/OpenSearch/pull/19103)) ([#19269](https://github.com/opensearch-project/OpenSearch/pull/19269)
 - Bump Google Cloud Storage SDK from 1.113.1 to 2.55.0 ([#18922](https://github.com/opensearch-project/OpenSearch/pull/18922))
 - Bump `com.google.auth:google-auth-library-oauth2-http` from 1.37.1 to 1.38.0 ([#19144](https://github.com/opensearch-project/OpenSearch/pull/19144))
@@ -140,6 +145,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `com.azure:azure-identity` from 1.14.2 to 1.18.0 ([#19361](https://github.com/opensearch-project/OpenSearch/pull/19361))
 - Bump `net.bytebuddy:byte-buddy` from 1.17.5 to 1.17.7 ([#19371](https://github.com/opensearch-project/OpenSearch/pull/19371))
 - Bump `lycheeverse/lychee-action` from 2.4.1 to 2.6.1 ([#19463](https://github.com/opensearch-project/OpenSearch/pull/19463))
+- Bump `io.grpc` deps from 1.68.2 to 1.75.0 ([#19495](https://github.com/opensearch-project/OpenSearch/pull/19495))
 
 ### Deprecated
 

gradle/libs.versions.toml

@@ -29,7 +29,7 @@ google_http_client = "1.44.1"
 google_auth       = "1.29.0"
 tdigest           = "3.3"
 hdrhistogram      = "2.2.2"
-grpc              = "1.68.2"
+grpc              = "1.75.0"
 json_smart        = "2.5.2"
 
 # when updating the JNA version, also update the version in buildSrc/build.gradle
@@ -54,6 +54,7 @@ commonscodec      = "1.18.0"
 commonslang       = "3.18.0"
 commonscompress   = "1.28.0"
 commonsio         = "2.16.0"
+commonscollections4 = "4.5.0"
 # plugin dependencies
 aws               = "2.32.29"
 awscrt            = "0.35.0"

modules/autotagging-commons/common/build.gradle

@@ -12,7 +12,7 @@ apply plugin: 'opensearch.publish'
 description = 'OpenSearch Rule framework common constructs which spi and module shares'
 
 dependencies {
-  api 'org.apache.commons:commons-collections4:4.4'
+  api "org.apache.commons:commons-collections4:${versions.commonscollections4}"
   implementation project(":libs:opensearch-common")
   compileOnly project(":server")
 

modules/autotagging-commons/common/licenses/commons-collections4-4.4.jar.sha1

@@ -0,0 +1 @@
+e5cf89f0c6e132fc970bd9a465fdcb8dbe94f75a

modules/autotagging-commons/common/licenses/commons-collections4-4.5.0.jar.sha1

@@ -0,0 +1,41 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.rule;
+
+/**
+ * Represents a feature value along with a matching score.
+ */
+public class MatchLabel<V> {
+    private final V featureValue;
+    private final float matchScore;
+
+    /**
+     * Constructs a FeatureValueMatch with the given feature value and score.
+     * @param featureValue the feature value
+     * @param matchScore the matching score
+     */
+    public MatchLabel(V featureValue, float matchScore) {
+        this.featureValue = featureValue;
+        this.matchScore = matchScore;
+    }
+
+    /**
+     * Returns the feature value.
+     */
+    public V getFeatureValue() {
+        return featureValue;
+    }
+
+    /**
+     * Returns the match score.
+     */
+    public float getMatchScore() {
+        return matchScore;
+    }
+}

modules/autotagging-commons/common/src/main/java/org/opensearch/rule/MatchLabel.java

@@ -17,6 +17,8 @@
 import java.nio.charset.StandardCharsets;
 import java.time.Instant;
 import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
@@ -29,6 +31,8 @@
  */
 @ExperimentalApi
 public class RuleUtils {
+    private static final String PIPE_DELIMITER = "\\|";
+    private static final String DOT_DELIMITER = ".";
 
     /**
      * constructor for RuleUtils
@@ -112,4 +116,36 @@ public static Rule composeUpdatedRule(Rule originalRule, UpdateRuleRequest reque
             Instant.now().toString()
         );
     }
+
+    /**
+     * Builds a flattened map of attribute filters from a {@link Rule}.
+     * This method reformats nested or prioritized subfields (e.g., values containing "|" for sub-attributes)
+     * into top-level attribute keys. For example, an attribute "principal" with value "username|admin" will
+     * become "principal.username" -> "admin" in the resulting map. Attributes without prioritized subfields
+     * remain unchanged.
+     * The resulting map is structured to make querying rules from the index easier.
+     * @param rule the rule whose attributes are to be flattened
+     */
+    public static Map<String, Set<String>> buildAttributeFilters(Rule rule) {
+        Map<String, Set<String>> attributeFilters = new HashMap<>();
+
+        for (Map.Entry<Attribute, Set<String>> entry : rule.getAttributeMap().entrySet()) {
+            Attribute attribute = entry.getKey();
+            Set<String> values = entry.getValue();
+            if (hasSubfields(attribute)) {
+                for (String value : values) {
+                    String[] parts = value.split(PIPE_DELIMITER);
+                    String topLevelAttribute = attribute.getName() + DOT_DELIMITER + parts[0];
+                    attributeFilters.computeIfAbsent(topLevelAttribute, k -> new HashSet<>()).add(parts[1]);
+                }
+            } else {
+                attributeFilters.put(attribute.getName(), values);
+            }
+        }
+        return attributeFilters;
+    }
+
+    private static boolean hasSubfields(Attribute attribute) {
+        return !attribute.getWeightedSubfields().isEmpty();
+    }
 }

modules/autotagging-commons/common/src/main/java/org/opensearch/rule/RuleUtils.java

@@ -13,7 +13,6 @@
 import org.opensearch.common.annotation.ExperimentalApi;
 import org.opensearch.core.common.io.stream.StreamInput;
 import org.opensearch.core.common.io.stream.StreamOutput;
-import org.opensearch.rule.autotagging.Attribute;
 import org.opensearch.rule.autotagging.FeatureType;
 import org.opensearch.rule.autotagging.Rule;
 import org.opensearch.rule.autotagging.RuleValidator;
@@ -34,7 +33,7 @@
 @ExperimentalApi
 public class GetRuleRequest extends ActionRequest {
     private final String id;
-    private final Map<Attribute, Set<String>> attributeFilters;
+    private final Map<String, Set<String>> attributeFilters;
     private final String searchAfter;
     private final FeatureType featureType;
 
@@ -45,7 +44,7 @@ public class GetRuleRequest extends ActionRequest {
      * @param searchAfter - The sort value used for pagination.
      * @param featureType - The feature type related to rule.
      */
-    public GetRuleRequest(String id, Map<Attribute, Set<String>> attributeFilters, String searchAfter, FeatureType featureType) {
+    public GetRuleRequest(String id, Map<String, Set<String>> attributeFilters, String searchAfter, FeatureType featureType) {
         this.id = id;
         this.attributeFilters = attributeFilters;
         this.searchAfter = searchAfter;
@@ -60,7 +59,7 @@ public GetRuleRequest(StreamInput in) throws IOException {
         super(in);
         id = in.readOptionalString();
         featureType = FeatureType.from(in);
-        attributeFilters = in.readMap(i -> Attribute.from(i, featureType), i -> new HashSet<>(i.readStringList()));
+        attributeFilters = in.readMap(StreamInput::readString, i -> new HashSet<>(i.readStringList()));
         searchAfter = in.readOptionalString();
     }
 
@@ -80,7 +79,7 @@ public void writeTo(StreamOutput out) throws IOException {
         super.writeTo(out);
         out.writeOptionalString(id);
         featureType.writeTo(out);
-        out.writeMap(attributeFilters, (output, attribute) -> attribute.writeTo(output), StreamOutput::writeStringCollection);
+        out.writeMap(attributeFilters, StreamOutput::writeString, StreamOutput::writeStringCollection);
         out.writeOptionalString(searchAfter);
     }
 
@@ -94,7 +93,7 @@ public String getId() {
     /**
      * attributeFilters getter
      */
-    public Map<Attribute, Set<String>> getAttributeFilters() {
+    public Map<String, Set<String>> getAttributeFilters() {
         return attributeFilters;
     }
 

modules/autotagging-commons/common/src/main/java/org/opensearch/rule/action/GetRuleRequest.java

@@ -15,6 +15,22 @@
  * @param <V>
  */
 public interface AttributeExtractor<V> {
+
+    /**
+     * Defines the combination style used when a request contains multiple values
+     * for an attribute.
+     */
+    enum LogicalOperator {
+        /**
+         * Logical AND
+         */
+        AND,
+        /**
+         * Logical OR
+         */
+        OR
+    }
+
     /**
      * This method returns the Attribute which it is responsible for extracting
      * @return attribute
@@ -26,4 +42,13 @@ public interface AttributeExtractor<V> {
      * @return attribute value
      */
     Iterable<V> extract();
+
+    /**
+     * Returns the logical operator used when a request contains multiple values
+     * for an attribute.
+     * For example, if the request targets both index A and B, then a rule must
+     * have both index A and B as attributes, requiring an AND operator.
+     * @return the logical operator (e.g., AND, OR)
+     */
+    LogicalOperator getLogicalOperator();
 }

modules/autotagging-commons/common/src/main/java/org/opensearch/rule/attribute_extractor/AttributeExtractor.java

@@ -14,11 +14,17 @@
 import org.opensearch.core.xcontent.XContentBuilder;
 import org.opensearch.core.xcontent.XContentParseException;
 import org.opensearch.core.xcontent.XContentParser;
+import org.opensearch.rule.MatchLabel;
+import org.opensearch.rule.attribute_extractor.AttributeExtractor;
+import org.opensearch.rule.storage.AttributeValueStore;
 
 import java.io.IOException;
+import java.util.HashMap;
 import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
 import java.util.Set;
-import java.util.TreeMap;
+import java.util.stream.Collectors;
 
 /**
  * Represents an attribute within the auto-tagging feature. Attributes define characteristics that can
@@ -36,10 +42,10 @@ public interface Attribute extends Writeable {
     String getName();
 
     /**
-     * Returns a map of subfields ordered by priority, where 1 represents the highest priority.
+     * Returns a map of subfields with its weight, which is used to calculate the match score for the attribute.
      */
-    default TreeMap<Integer, String> getPrioritizedSubfields() {
-        return new TreeMap<>();
+    default Map<String, Float> getWeightedSubfields() {
+        return new HashMap<>();
     }
 
     /**
@@ -106,4 +112,28 @@ static Attribute from(StreamInput in, FeatureType featureType) throws IOExceptio
         }
         return attribute;
     }
+
+    /**
+     * Evaluates the matching labels for the attribute
+     * @param attributeExtractor the extractor to get the attribute
+     * @param attributeValueStore in-memory value store for the attribute
+     */
+    default List<MatchLabel<String>> findAttributeMatches(
+        AttributeExtractor<String> attributeExtractor,
+        AttributeValueStore<String, String> attributeValueStore
+    ) {
+        Map<String, Float> scoreMap = new HashMap<>();
+
+        for (String value : attributeExtractor.extract()) {
+            List<MatchLabel<String>> matches = attributeValueStore.getMatches(value);
+            for (MatchLabel<String> entry : matches) {
+                scoreMap.merge(entry.getFeatureValue(), entry.getMatchScore(), Float::sum);
+            }
+        }
+        return scoreMap.entrySet()
+            .stream()
+            .map(e -> new MatchLabel<>(e.getKey(), e.getValue()))
+            .sorted((a, b) -> Float.compare(b.getMatchScore(), a.getMatchScore()))
+            .collect(Collectors.toList());
+    }
 }