Ccdihub 4.10.0 #59

shawnwangnih · 2025-10-15T19:11:07Z

No description provided.

to update opensearch to 2.x

Copilot

Pull Request Overview

This PR increases Elasticsearch/OpenSearch query/result size limits and introduces a scroll-based pagination path for large result sets. It also updates YAML loading to use LoaderOptions and adjusts an OpenSearch import for newer versions.

Added scrolling logic (collectPageWithScroll, rollToPage, collectScrollPage) with new constants for thresholds and sizes
Raised multiple hard limits (MAX_ES_SIZE, MAX_SIZE, AGGS_SIZE, maxValues) to 200000
Added LoaderOptions to SnakeYAML constructors and updated an OpenSearch Text import

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
service/ESService.java	Introduces scroll pagination, new constants, and helper methods for large result retrieval; adjusts scroll timeout.
model/search/yaml/type/SingleTypeYaml.java	Adds LoaderOptions to SnakeYAML constructor.
model/search/yaml/type/GroupTypeYaml.java	Adds LoaderOptions to SnakeYAML constructor.
model/search/yaml/type/GlobalTypeYaml.java	Adds LoaderOptions to SnakeYAML constructor.
model/search/mapper/TypeMapperImpl.java	Updates Text import to new OpenSearch package path.
controller/GraphQLController.java	Increases maximum accepted variable list size.
constants/Const.java	Raises MAX_SIZE and AGGS_SIZE limits.
ServletInitializer.java	Removes redundant servlet initializer class.
BentoApplication.java	Adds (unused) memory management imports.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-15T19:14:23Z

service/ESService.java

+    public static final int MAX_ES_SIZE = 200000; // Do not return more than this number of records
+    public static final int SCROLL_THRESHOLD = 10000; // Use scroll when trying to retrieve past this number of records
+    public static final int SCROLL_SIZE = 10000; // How big each scroll should be


Raising MAX_ES_SIZE to 200000 significantly increases worst-case memory and transfer costs; consider enforcing a tighter upper bound (e.g. configured via application properties) and streaming/processing chunks instead of materializing large result sets in memory.

Copilot · 2025-10-15T19:14:23Z

controller/GraphQLController.java

 			variables = gson.fromJson(rawVar, Map.class);
            // Verify that all parameter inputs are less than 1000 values
-			int maxValues = 1000;
+			int maxValues = 200000;


Increasing maxValues to 200000 can allow extremely large user-supplied lists, risking excessive query expansion and memory pressure; consider a lower configurable limit or pagination of input sets.

Copilot · 2025-10-15T19:14:24Z

constants/Const.java

    public static class ES_UNITS {
        public static final int DEFAULT_SIZE = 10;
-        public static final int MAX_SIZE = 10000;
+        public static final int MAX_SIZE = 200000;


Setting MAX_SIZE and AGGS_SIZE to 200000 may trigger very large from/size or aggregation buckets responses that are expensive for OpenSearch and the application; consider capping these or requiring scrolling/ composite aggs for large cardinalities.

Copilot · 2025-10-15T19:14:24Z

constants/Const.java

        public static final String SORT_DIRECTION = "sort_direction";
        public static final String CASE_IDS = "case_ids";
-        public static final int AGGS_SIZE = 1000;
+        public static final int AGGS_SIZE = 200000;


Setting MAX_SIZE and AGGS_SIZE to 200000 may trigger very large from/size or aggregation buckets responses that are expensive for OpenSearch and the application; consider capping these or requiring scrolling/ composite aggs for large cardinalities.

what are the suggestions

Copilot · 2025-10-15T19:14:24Z

service/ESService.java

+        query.put("size", SCROLL_SIZE);
+        String jsonizedQuery = gson.toJson(query);
+        request.setJsonEntity(jsonizedQuery);
+        request.addParameter("scroll", "1m");
+        // JsonObject page = rollToPage(request, pageSize, offset);
+        // return collectPage(page, properties, pageSize, offset % SCROLL_SIZE);
+        JsonArray page = rollToPage(request, pageSize, offset);
+        // checkMemoryInit();
+        return collectScrollPage(page, properties, pageSize, offset % SCROLL_SIZE);


collectPageWithScroll always requests SCROLL_SIZE regardless of (pageSize, offset) and materializes all intermediate hits, which can overshoot and allocate far more data than needed; compute an adaptive first batch size (e.g. remainingNeeded = offsetRemainder + pageSize) and slice only required hits instead of filling an oversized JsonArray.

Suggested change

query.put("size", SCROLL_SIZE);

String jsonizedQuery = gson.toJson(query);

request.setJsonEntity(jsonizedQuery);

request.addParameter("scroll", "1m");

// JsonObject page = rollToPage(request, pageSize, offset);

// return collectPage(page, properties, pageSize, offset % SCROLL_SIZE);

JsonArray page = rollToPage(request, pageSize, offset);

// checkMemoryInit();

return collectScrollPage(page, properties, pageSize, offset % SCROLL_SIZE);

// Compute the minimum number of hits needed for the first batch

int offsetRemainder = offset % SCROLL_SIZE;

int remainingNeeded = offsetRemainder + pageSize;

int firstBatchSize = Math.min(remainingNeeded, SCROLL_SIZE);

query.put("size", firstBatchSize);

String jsonizedQuery = gson.toJson(query);

request.setJsonEntity(jsonizedQuery);

request.addParameter("scroll", "1m");

JsonArray page = rollToPage(request, pageSize, offset);

// Slice only the required hits from the page

int startIdx = offsetRemainder;

int endIdx = Math.min(startIdx + pageSize, page.size());

JsonArray slicedPage = new JsonArray();

for (int i = startIdx; i < endIdx; i++) {

slicedPage.add(page.get(i));

}

return collectScrollPage(slicedPage, properties, pageSize, 0);

Copilot · 2025-10-15T19:14:25Z

service/ESService.java

            String scrollId = jsonObject.get("_scroll_id").getAsString();
            Map<String, Object> scrollQuery = Map.of(
-                    "scroll", "10S",
+                    "scroll", "1m",


The scroll keep-alive value '1m' is duplicated; extract a constant (e.g. SCROLL_KEEPALIVE) to avoid divergence and ease future tuning.

Copilot · 2025-10-15T19:14:26Z

service/ESService.java

+        query.put("size", SCROLL_SIZE);
+        String jsonizedQuery = gson.toJson(query);
+        request.setJsonEntity(jsonizedQuery);
+        request.addParameter("scroll", "1m");


The scroll keep-alive value '1m' is duplicated; extract a constant (e.g. SCROLL_KEEPALIVE) to avoid divergence and ease future tuning.

Copilot · 2025-10-15T19:14:26Z

service/ESService.java

+            scrollRequest = new Request("POST", SCROLL_ENDPOINT);
            Map<String, Object> scrollQuery = Map.of(
-                    "scroll", "10S",
+                    "scroll", "1m",


The scroll keep-alive value '1m' is duplicated; extract a constant (e.g. SCROLL_KEEPALIVE) to avoid divergence and ease future tuning.

Copilot · 2025-10-15T19:14:26Z

service/ESService.java

+    // public void checkMemoryInit() {
+    //     // Get the Java Runtime object
+    //     Runtime runtime = Runtime.getRuntime();
+
+    //     // Get the maximum heap size (in bytes)
+    //     long maxMemory = runtime.maxMemory();
+    //     // Get the initial heap size (in bytes)
+    //     long initialMemory = runtime.totalMemory();
+    //     // Get the current available memory (in bytes)
+    //     long freeMemory = runtime.freeMemory();
+
+    //     // Convert to MB for better readability
+    //     System.out.println("Initial Heap Size: " + (initialMemory / (1024 * 1024)) + " MB");
+    //     System.out.println("Maximum Heap Size: " + (maxMemory / (1024 * 1024)) + " MB");
+    //     System.out.println("Free Memory: " + (freeMemory / (1024 * 1024)) + " MB");
+    // }
+
+    // public void checkMemoryNow() {
+    //     // Optionally log the memory usage using MemoryMXBean
+    //     MemoryMXBean memoryMXBean = ManagementFactory.getMemoryMXBean();
+    //     MemoryUsage heapMemoryUsage = memoryMXBean.getHeapMemoryUsage();
+
+    //     System.out.println("Used Heap Memory: " + (heapMemoryUsage.getUsed() / (1024 * 1024)) + " MB");
+    //     System.out.println("Committed Heap Memory: " + (heapMemoryUsage.getCommitted() / (1024 * 1024)) + " MB");
+    // }


Large blocks of commented-out diagnostic code add noise; remove them or convert to conditional debug logging (logger.debug with guarded feature flag) to keep the class concise.

Copilot · 2025-10-15T19:14:27Z

BentoApplication.java

+import java.lang.management.ManagementFactory;
+import java.lang.management.MemoryMXBean;
+import java.lang.management.MemoryUsage;


These memory management imports are unused in this class; remove them to avoid clutter.

Suggested change

import java.lang.management.ManagementFactory;

import java.lang.management.MemoryMXBean;

import java.lang.management.MemoryUsage;

Copilot

Pull Request Overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-22T17:44:54Z

service/ESService.java

+     * @param properties The Opensearch properties to retrieve
+     * @param pageSize The desired number of results to obtain
+     * @param offset The desired offset of the results
+     * @return


The @return tag is missing a description. Please add a description of what this method returns (e.g., 'A list of maps containing the requested page of results').

Copilot · 2025-10-22T17:44:54Z

service/ESService.java

+     * @param request The Opensearch request
+     * @param pageSize How many records to obtain
+     * @param offset How many records to skip
+     * @return


The @return tag is missing a description. Please add a description of what this method returns (e.g., 'A JsonArray containing all hits from the scroll requests').

Suggested change

* @return

* @return A JsonArray containing all hits from the scroll requests, up to the requested page size and offset.

Copilot · 2025-10-22T17:44:54Z

service/ESService.java

+        request.setJsonEntity(jsonizedQuery);
+        request.addParameter("scroll", "1m");
+        // JsonObject page = rollToPage(request, pageSize, offset);
+        // return collectPage(page, properties, pageSize, offset % SCROLL_SIZE);


Remove commented-out code. These lines appear to be old implementation that has been replaced with the new JsonArray-based approach.

Suggested change

// return collectPage(page, properties, pageSize, offset % SCROLL_SIZE);

Copilot · 2025-10-22T17:44:55Z

service/ESService.java

+        // JsonObject page = rollToPage(request, pageSize, offset);
+        // return collectPage(page, properties, pageSize, offset % SCROLL_SIZE);
+        JsonArray page = rollToPage(request, pageSize, offset);
+        // checkMemoryInit();


Remove commented-out debugging code. If memory monitoring is needed in production, it should be implemented properly with appropriate logging levels.

Suggested change

// checkMemoryInit();

Copilot · 2025-10-22T17:44:55Z

service/ESService.java

+        // JsonObject outerHits = new JsonObject(); // Helper JSON object for the results
+        // JsonObject results = new JsonObject(); // The results to return


Remove commented-out code. These variables are no longer needed since the method now returns a JsonArray directly.

Copilot · 2025-10-22T17:44:55Z

service/ESService.java

+        // outerHits.add("hits", allHits);
+        // results.add("hits", outerHits);


Remove commented-out code. The old JsonObject wrapping logic has been replaced by returning the JsonArray directly.

Copilot · 2025-10-22T17:44:55Z

service/ESService.java

+    public List<Map<String, Object>> collectScrollPage(JsonArray searchHits, String[][] properties, int pageSize, int offset) throws IOException {
+        List<Map<String, Object>> data = new ArrayList<>();
+
+        //JsonArray searchHits = jsonObject.getAsJsonObject("hits").getAsJsonArray("hits");


Remove commented-out code. This old line extracts searchHits from a JsonObject, but the method now receives searchHits directly as a parameter.

Suggested change

//JsonArray searchHits = jsonObject.getAsJsonObject("hits").getAsJsonArray("hits");

David-YuWei and others added 25 commits September 14, 2023 13:00

Update all parameter inputs limitation to 6000

d460403

upgrade to use snakeyaml 2.0 package

6ab4c8c

Increase file size limit

e395db1

increase max ES size to 200000

4fd83ba

increase aggs size to 200000

5d2adbf

enable scroll when receiving records

e1bda5a

Update scroll threshold to 10000

e28e8a2

Increase graphqlcontroller max value to 200k

82fea93

Update scroll configuration

415ee27

update scroll configuration back

c898238

Add debug info

933e72e

Add debug info

3e00266

Add more debug info

5ba533c

correct type info

0d06c8d

Fix function parameter issue

afa7d36

Add debug info

71e6e47

Add more debug info

38da623

Add debug info

3997d04

add debug info

8639c8e

Add debug info

6d67faf

Add debug info

39bab6f

Add debug info

2347b6c

Revert debugging info

b4274ae

Delete ServletInitializer

e2f9903

Update TypeMapperImpl.java

5caff92

to update opensearch to 2.x

shawnwangnih requested review from AustinSMueller, David-YuWei and Copilot October 15, 2025 19:11

Copilot AI reviewed Oct 15, 2025

View reviewed changes

AustinSMueller changed the base branch from master to 4.11.0 October 22, 2025 17:43

AustinSMueller requested a review from Copilot October 22, 2025 17:43

Copilot AI reviewed Oct 22, 2025

View reviewed changes

	import java.lang.management.ManagementFactory;
	import java.lang.management.MemoryMXBean;
	import java.lang.management.MemoryUsage;

	* @return
	* @return A JsonArray containing all hits from the scroll requests, up to the requested page size and offset.

		// JsonObject outerHits = new JsonObject(); // Helper JSON object for the results
		// JsonObject results = new JsonObject(); // The results to return

		// outerHits.add("hits", allHits);
		// results.add("hits", outerHits);

Ccdihub 4.10.0 #59

Are you sure you want to change the base?

Ccdihub 4.10.0 #59

Uh oh!

Conversation

shawnwangnih commented Oct 15, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

shawnwangnih Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants