FEATURE: Summarize the steps when max steps limit reached #4416

PauiC · 2025-11-12T10:26:26Z

Description

This feature provides intelligent summarization when ML Commons Chat Agents and PER (planner) agent reach their maximum iteration limit without completing tasks.

Related Issues

#4150

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

) Signed-off-by: Dhrubo Saha <[email protected]>

Signed-off-by: Yaliang Wu <[email protected]>

…roject#4331)" This reverts commit fb90b1b.

Signed-off-by: Yaliang Wu <[email protected]>

Signed-off-by: Dhrubo Saha <[email protected]>

Signed-off-by: Pavan Yekbote <[email protected]>

Signed-off-by: Yaliang Wu <[email protected]>

Signed-off-by: Pavan Yekbote <[email protected]>

Signed-off-by: Yaliang Wu <[email protected]>

…Agents (opensearch-project#4397) * add hooks in ml-commons (opensearch-project#4326) Signed-off-by: Xun Zhang <[email protected]> * initiate context management api with hook implementation (opensearch-project#4345) * initiate context management api with hook implementation Signed-off-by: Mingshi Liu <[email protected]> * apply spotless Signed-off-by: Mingshi Liu <[email protected]> --------- Signed-off-by: Mingshi Liu <[email protected]> * Add Context Manager to PER (opensearch-project#4379) * add pre_llm hook to per agent Signed-off-by: Mingshi Liu <[email protected]> change context management passing from query parameters to payload Signed-off-by: Mingshi Liu <[email protected]> pass hook registery into PER Signed-off-by: Mingshi Liu <[email protected]> apply spotless Signed-off-by: Mingshi Liu <[email protected]> initiate context management api with hook implementation Signed-off-by: Mingshi Liu <[email protected]> * add comment Signed-off-by: Mingshi Liu <[email protected]> * format Signed-off-by: Mingshi Liu <[email protected]> * add validation Signed-off-by: Mingshi Liu <[email protected]> --------- Signed-off-by: Mingshi Liu <[email protected]> * add inner create context management to agent register api Signed-off-by: Mingshi Liu <[email protected]> * add code coverage Signed-off-by: Mingshi Liu <[email protected]> * allow context management hook register in during agent execute Signed-off-by: Mingshi Liu <[email protected]> * add code coverage Signed-off-by: Mingshi Liu <[email protected]> * add more code coverage Signed-off-by: Mingshi Liu <[email protected]> * add validation check Signed-off-by: Mingshi Liu <[email protected]> * adapt to inplace update for context Signed-off-by: Mingshi Liu <[email protected]> * fix test Signed-off-by: Mingshi Liu <[email protected]> --------- Signed-off-by: Xun Zhang <[email protected]> Signed-off-by: Mingshi Liu <[email protected]> Co-authored-by: Xun Zhang <[email protected]>

…ut atm) Signed-off-by: Pavan Yekbote <[email protected]>

Signed-off-by: Dhrubo Saha <[email protected]>

…ing in index (opensearch-project#4403) * allow inline create context management without storing in agent register Signed-off-by: Mingshi Liu <[email protected]> * make ML_COMMONS_MULTI_TENANCY_ENABLED default is false Signed-off-by: Mingshi Liu <[email protected]> --------- Signed-off-by: Mingshi Liu <[email protected]>

Signed-off-by: Dhrubo Saha <[email protected]>

Signed-off-by: Yaliang Wu <[email protected]>

…#4407) Signed-off-by: Dhrubo Saha <[email protected]>

…roject#4408) * Fix POST_TOOL hook interaction updates and add tenant ID support Signed-off-by: Mingshi Liu <[email protected]> - Fix POST_TOOL hook to return full ContextManagerContext like PRE_LLM hook - Update MLChatAgentRunner to properly handle interaction updates from POST_TOOL hook - Ensure interactions list and tmpParameters.INTERACTIONS stay synchronized - Add tenant ID support to MLPredictionTaskRequest in ModelGuardrail and SummarizationManager Signed-off-by: Mingshi Liu <[email protected]> * fix error message escaping Signed-off-by: Mingshi Liu <[email protected]> * consolicate post_hook logic Signed-off-by: Mingshi Liu <[email protected]> --------- Signed-off-by: Mingshi Liu <[email protected]>

Hailong-am

cherry picked from original PR

Signed-off-by: Jiaru Jiang <[email protected]>

pyek-bot · 2025-11-14T19:08:22Z

ml-algorithms/src/main/java/org/opensearch/ml/engine/algorithms/agent/MLChatAgentRunner.java

+        }
+    }
+
+    public String extractSummaryFromResponse(MLTaskResponse response) {


try to avoid returning nulls inside methods, ideally this method would also handle the errors and return a fallback response.

If you return nulls in a method, it requires the caller to handle the null which can potentially lead to NPEs.

Not urgent, can address in future PRs

pyek-bot · 2025-11-14T19:11:03Z

.../main/java/org/opensearch/ml/engine/algorithms/agent/MLPlanExecuteAndReflectAgentRunner.java

+        }, finalListener::onFailure);
+
+        generateSummary(llm, completedSteps, allParams.get(TENANT_ID_FIELD), ActionListener.wrap(summary -> {
+            log.info("Summary generated successfully");


let's remove this log, seems redundant

pyek-bot · 2025-11-14T19:12:42Z

.../main/java/org/opensearch/ml/engine/algorithms/agent/MLPlanExecuteAndReflectAgentRunner.java

+            String fallbackResult = completedSteps.isEmpty() || completedSteps.size() < 2
+                ? String.format("Max Steps Limit (%d) Reached. Use memory_id with same task to restart.", maxSteps)
+                : String
+                    .format(
+                        "Max Steps Limit (%d) Reached. Use memory_id with same task to restart. \n "
+                            + "Last executed step: %s, \n "
+                            + "Last executed step result: %s",
+                        maxSteps,
+                        completedSteps.get(completedSteps.size() - 2),
+                        completedSteps.getLast()
+                    );


you can pass the fallback from the previous method which is generated, this can make the current method more reusable

it's fine for now!

pyek-bot · 2025-11-14T19:13:43Z

.../main/java/org/opensearch/ml/engine/algorithms/agent/MLPlanExecuteAndReflectAgentRunner.java

+        }
+    }
+
+    private String extractSummaryFromResponse(MLTaskResponse response) {


you can move this to agent utils in a later PR

pyek-bot

LGTM, let's merge and address the comments in the main branch PR (or in a follow up PR)

Thanks for taking this up!

dhrubo-os and others added 30 commits October 24, 2025 16:03

agentic memory integration with agent framework (opensearch-project#4331

fef8e70

) Signed-off-by: Dhrubo Saha <[email protected]>

support remote store for agentic memory

66860ab

Signed-off-by: Yaliang Wu <[email protected]>

auto create connector

c13a7af

Signed-off-by: Yaliang Wu <[email protected]>

auto create embedding model in remote store

39bd765

Signed-off-by: Yaliang Wu <[email protected]>

fix bulk load bug

f0d2be1

Signed-off-by: Yaliang Wu <[email protected]>

fix connector header for remote store

ee60d16

Signed-off-by: Yaliang Wu <[email protected]>

support ingest/search pipeline when create memory container

80ec214

Signed-off-by: Yaliang Wu <[email protected]>

Revert "agentic memory integration with agent framework (opensearch-p…

c369aec

…roject#4331)" This reverts commit fb90b1b.

refactor memory interface

10d822b

Signed-off-by: Yaliang Wu <[email protected]>

agentic conversation memory

b2f5354

Signed-off-by: Yaliang Wu <[email protected]>

fix remote store issues

94d97e7

Signed-off-by: Yaliang Wu <[email protected]>

extending memory refactoring to PER agent (opensearch-project#4348)

14b2206

Signed-off-by: Dhrubo Saha <[email protected]>

Merge agent-revamp into feature/memory as a single commit

c5ad5c3

Signed-off-by: Pavan Yekbote <[email protected]>

create internal connector in memory container

4aef03d

Signed-off-by: Yaliang Wu <[email protected]>

fix: support old style agent register + new style register

9a9a358

Signed-off-by: Pavan Yekbote <[email protected]>

fix: memory working with agent

df354b8

Signed-off-by: Pavan Yekbote <[email protected]>

support memory container id in execute agent request

d66e924

Signed-off-by: Yaliang Wu <[email protected]>

fix per body template for new execute api (per only supports text inp…

40c84e9

…ut atm) Signed-off-by: Pavan Yekbote <[email protected]>

added multi-tenancy to the add memory and search memory action

82f3444

added client.search instead of sdkclient for collection index

22e5ce4

Signed-off-by: Dhrubo Saha <[email protected]>

adding tenancy to the create session

d3bdf82

execute PER agent with multi-tenancy

51d5b47

Signed-off-by: Dhrubo Saha <[email protected]>

add tenant id when encrypt

e57a4cd

add tenant id to connector

68361cb

add tenant id for get model method

ffc38b0

Signed-off-by: Yaliang Wu <[email protected]>

remote conversation memory with REST call (opensearch-project#4406)

a7ffb02

not stashing context if index is not system index (opensearch-project…

d06bc8d

…#4407) Signed-off-by: Dhrubo Saha <[email protected]>

PauiC requested review from HenryL27, Zhangxunmt, austintlee, pyek-bot, sam-herman, xinyual, ylwu-amzn and zane-neo as code owners November 12, 2025 10:26

PauiC had a problem deploying to ml-commons-cicd-env-require-approval November 12, 2025 10:27 — with GitHub Actions Error

PauiC had a problem deploying to ml-commons-cicd-env-require-approval November 12, 2025 10:27 — with GitHub Actions Failure

Hailong-am approved these changes Nov 12, 2025

View reviewed changes

PauiC mentioned this pull request Nov 12, 2025

FEATURE: Summarize the steps when max steps limit reached #4184

Open

5 tasks

fix:recover file

62086c0

Signed-off-by: Jiaru Jiang <[email protected]>

PauiC requested a deployment to ml-commons-cicd-env-require-approval November 13, 2025 02:33 — with GitHub Actions Waiting

fix:recover file

e0dc13d

Signed-off-by: Jiaru Jiang <[email protected]>

PauiC requested a deployment to ml-commons-cicd-env-require-approval November 13, 2025 03:17 — with GitHub Actions Waiting

pyek-bot reviewed Nov 14, 2025

View reviewed changes

pyek-bot approved these changes Nov 14, 2025

View reviewed changes

pyek-bot force-pushed the feature/agentic_memory_integration branch from ce7f3f5 to cd738fa Compare November 14, 2025 19:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

FEATURE: Summarize the steps when max steps limit reached #4416

FEATURE: Summarize the steps when max steps limit reached #4416

PauiC commented Nov 12, 2025

Uh oh!

Hailong-am left a comment

Uh oh!

pyek-bot Nov 14, 2025

Uh oh!

pyek-bot Nov 14, 2025

Uh oh!

pyek-bot Nov 14, 2025

Uh oh!

pyek-bot Nov 14, 2025

Uh oh!

pyek-bot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

FEATURE: Summarize the steps when max steps limit reached #4416

Are you sure you want to change the base?

FEATURE: Summarize the steps when max steps limit reached #4416

Conversation

PauiC commented Nov 12, 2025

Description

Related Issues

Check List

Uh oh!

Hailong-am left a comment

Choose a reason for hiding this comment

Uh oh!

pyek-bot Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

pyek-bot Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

pyek-bot Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

pyek-bot Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

pyek-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants