Germplasm Search Optimizations #49

jloux-brapi · 2025-03-19T17:19:10Z

A couple of optimizations, bug fixes and workarounds have been provided to improve the performance and usability of the germplasm search endpoint.

Hibernate was issuing a warning: firstResult/maxResults specified with collection fetch; applying in memory which was a big clue to why this search endpoint was performing so poorly. Tim essentially tried to implement as vlad described here, but there was a critical error, we passed through all of the query logic to the search query builder, which today runs paginated queries no matter what. This means that while we did the grunt work of fetching the IDs separately and giving them to another query, we ended up with the same applying in memory error as before. This code has been changed so that not only does the SearchQueryBuilder support non-paginated queries, there is also more support for the kind of double query that the paginated fetches require. (See GermplasmService.findGermplasmEntities()). The performance improvement is orders of magnitude. I went from about 4.5 seconds 100 records to about 500ms, 15 seconds on 1000 to 1 second on a dataset of 550k germs on a program.
A workaround was created for BI to send either a null page and pageSize or neither attributed be present in the request to kick off some new logic on the germplasm search POST endpoint which will utilize the new non-paginated query that SearchQueryBuilder supports to return all data at once, non-paginated. This has a breaking point, however, as at about 250k germ records per program java completely exhausts its heap trying to load all the data in entity objects and converting them to Json. It should be noted this also isn't particularly fast, as this is a large amount of data to transmit. 125k records takes about 30 seconds on average to get back. But this should work as a stop gap in the meantime.
While testing both of these features I also noticed that when more pages are requested than are currently available, the system can act in peculiar and sometimes disastrous behaviors, such as fetching all records for a query instead of paginating at all, which, if performed on a large data set can completely bring the server to its knees. To avoid this, additional logic was built into the BrAPIRepostitory impl to prevent these kinds of situations from arriving entirely by doing a max count first before fetching the query to compare the pageSize requested and the page number requested to the amount of available data. These situations were populated up the stack for all service methods that call them to send back 400 responses when they occur with information on how to avoid these situations (this explains how many files were touched). Additional optimizatons were made here to short-circuit the query lookup code if max count produced 0 results to avoid unnecessary long-running queries.
To totally eliminated the Could not prepare SQL statement error, we needed a way to completely refuse lookups that could produce more than 65k sql params, as this is the limit. These occurred mostly in my testing when I tried to paginate germplasms large than 65k records, because in order to fetch these records we need to pass IDs of found records in inital query to later join-fetch queries. I suppose there are other ways around this, like we could break up the queries into more queries, but the right solution feels like to incentivize the requester to actually request data from the search endpoint in a meaningful and more performant way. That is, we have configured a way to control the maximum allowable page size for page requests on the server. For now, it is 65k, and this applies to all entities, not just Germplasm. Specifically for BI, this will be a problem for the cache, which we have addressed for the germplasm entity but not for other larger entities they might have, like observations and observation units. I may revert this commit and put it somewhere else separately if it is a problem loading a cache for large datasets, or I might add to this body of code.

Note: This PR and its associated commits should also be merged to the prod server when it is verified on BI's end.

If not specified, this sort will be used to keep the endpoints idempotent.

…tion Added utility methods to SearchQueryBuilder and BrAPIRepositoryImpl to allow for proper paginating for hibernate fetch queries that don't suffocate memory. Also added methods to run queries without pagination entirely using the SearchQueryBuilder to prevent the use of pagination when it's not required, an issue that specifically had to be addressed for the BI cache, but one that introduced code that is reusable for other use cases. Modified the GermplasmApiController's searchGermplasmPost endpoint to accomodate two code paths: - One where no page and pageSize are supplied. In this scenario the code will grab all germplasm without the use of pagination. Good for large data grabs, but gets dangerous with excessively large amounts of data. This is entirely to meet BI's current use case, which we have strongly advised they move off of. - When page and/or pageSize are supplied, paginate as requested, default page size of 1000 if not requested.

…errors

Additionally make these configurable vars consistent and usable across BrAPIController and PagingUtility, which both utilize them.

BrapiCoordinatorSelby · 2025-03-21T19:03:05Z

src/main/java/org/brapi/test/BrAPITestServer/controller/germ/SeedLotsApiController.java

@@ -156,7 +156,7 @@ public ResponseEntity<SeedLotTransactionListResponse> seedlotsTransactionsGet(
 		validateAcceptHeader(request);
 		Metadata metadata = generateMetaDataTemplate(page, pageSize);
 		List<SeedLotTransaction> data = seedLotService.findSeedLotTransactions(transactionDbId, seedLotDbId,
-				germplasmDbId, germplasmName, crossDbId, crossName, commonCropName, programDbId, externalReferenceId,


externalReferenceId is the newer, correct spelling, introduced in BrAPI v2.1.
externalReferenceID is the deprecated one and could be deleted if there is no need for backwards compatibility with v2.0

These were autofixes for params that were being unused in these sigs, I can rename to externalReferenceId here and elsewhere

I will use the right one here and other places, but think we should wait to fully delete from ApiControllers and data models, or do it in another commit/MR.

I would either continue to support the deprecated externalReferenceID parameters (see my comment on CrossingProjectsApiController.java about how to do that), or fully remove them from controller signatures.

Ok, I would rather this not be a sticking point then. These changes are pretty far removed and out of scope of the things we actually want. I will just revert these changes.

If we actually want to remove those attributes from the codebase would rather that be done in a separate MR, bc it's a decent amount of changes to fully support.

jloux-brapi · 2025-03-21T20:13:46Z

src/main/java/org/brapi/test/BrAPITestServer/controller/germ/SeedLotsApiController.java

@@ -156,7 +156,7 @@ public ResponseEntity<SeedLotTransactionListResponse> seedlotsTransactionsGet(
 		validateAcceptHeader(request);
 		Metadata metadata = generateMetaDataTemplate(page, pageSize);
 		List<SeedLotTransaction> data = seedLotService.findSeedLotTransactions(transactionDbId, seedLotDbId,
-				germplasmDbId, germplasmName, crossDbId, crossName, commonCropName, programDbId, externalReferenceId,


These were autofixes for params that were being unused in these sigs, I can rename to externalReferenceId here and elsewhere

jloux-brapi · 2025-03-24T13:30:38Z

src/main/java/org/brapi/test/BrAPITestServer/controller/core/BrAPIController.java

@@ -33,29 +33,14 @@

 public class BrAPIController {
 	private static final Logger log = LoggerFactory.getLogger(ServerInfoApiController.class);
-
-	protected Metadata generateMetaDataTemplateForSearch(Integer originalRequestedPage, Integer newRequestedPage,


This was unused.

jloux-brapi · 2025-03-24T13:31:09Z

src/main/java/org/brapi/test/BrAPITestServer/controller/core/BrAPIController.java

@@ -81,16 +66,6 @@ protected Metadata generateMetaDataTemplate(Integer page, Integer pageSize) thro
 		return metaData;
 	}

-	private void validatePaging(Integer page, Integer pageSize) throws BrAPIServerException {


Moved to PagingUtility

jloux-brapi · 2025-03-24T13:42:15Z

src/main/java/org/brapi/test/BrAPITestServer/controller/germ/SeedLotsApiController.java

@@ -156,7 +156,7 @@ public ResponseEntity<SeedLotTransactionListResponse> seedlotsTransactionsGet(
 		validateAcceptHeader(request);
 		Metadata metadata = generateMetaDataTemplate(page, pageSize);
 		List<SeedLotTransaction> data = seedLotService.findSeedLotTransactions(transactionDbId, seedLotDbId,
-				germplasmDbId, germplasmName, crossDbId, crossName, commonCropName, programDbId, externalReferenceId,


I will use the right one here and other places, but think we should wait to fully delete from ApiControllers and data models, or do it in another commit/MR.

jloux-brapi · 2025-03-24T13:46:16Z

src/main/java/org/brapi/test/BrAPITestServer/factory/BrAPIComponent.java


 import java.util.List;

 public interface BrAPIComponent<T, R extends SearchRequest> {
-    List<T> findEntities(@Valid R request, Metadata metadata);
+    List<T> findEntities(@Valid R request, Metadata metadata) throws BrAPIServerException;


All these BrAPIAServerExceptions were added to catch and populate the InvalidPagingException up the stack to deliver a 400 to the user and give them a message instrucitng them how to page correctly.

src/main/java/org/brapi/test/BrAPITestServer/service/PagingUtility.java

jloux-brapi · 2025-03-24T13:52:30Z

src/main/java/org/brapi/test/BrAPITestServer/service/core/ListService.java

 		Pageable pageReq = PagingUtility.getPageRequest(metadata);
 		SearchQueryBuilder<ListEntity> searchQuery = buildQueryString(request);

-		Page<ListEntity> entityPage = listRepository.findAllBySearch(searchQuery, pageReq);
+		Page<ListEntity> entityPage = listRepository.findAllBySearchAndPaginate(searchQuery, pageReq);


All of these call changes are just signature switches. I wanted to utilize findAllBySearch to mean search using a search query without pagination. So far, just GermplasmService really uses this, and everything else still uses pagination for now. For all intents and purposes, findAllBySearchAndPaginate() is the old findAllBySearch()

mlm483

I still need to do functional testing, but I want to get my initial feedback to you as soon as possible.

src/main/java/org/brapi/test/BrAPITestServer/controller/germ/CrossesApiController.java

src/main/java/org/brapi/test/BrAPITestServer/controller/germ/CrossingProjectsApiController.java

src/main/java/org/brapi/test/BrAPITestServer/controller/germ/GermplasmApiController.java

src/main/java/org/brapi/test/BrAPITestServer/controller/germ/PlannedCrossesApiController.java

mlm483 · 2025-03-27T15:02:49Z

src/main/java/org/brapi/test/BrAPITestServer/controller/germ/SeedLotsApiController.java

@@ -156,7 +156,7 @@ public ResponseEntity<SeedLotTransactionListResponse> seedlotsTransactionsGet(
 		validateAcceptHeader(request);
 		Metadata metadata = generateMetaDataTemplate(page, pageSize);
 		List<SeedLotTransaction> data = seedLotService.findSeedLotTransactions(transactionDbId, seedLotDbId,
-				germplasmDbId, germplasmName, crossDbId, crossName, commonCropName, programDbId, externalReferenceId,


I would either continue to support the deprecated externalReferenceID parameters (see my comment on CrossingProjectsApiController.java about how to do that), or fully remove them from controller signatures.

src/main/java/org/brapi/test/BrAPITestServer/service/germ/SeedLotService.java

src/main/java/org/brapi/test/BrAPITestServer/service/germ/CrossingProjectService.java

src/main/java/org/brapi/test/BrAPITestServer/service/germ/GermplasmService.java

mlm483 · 2025-03-27T16:24:44Z

src/main/java/org/brapi/test/BrAPITestServer/service/germ/GermplasmService.java

 		});
 	}

+	private void fetchRemainingGermCollectionsUsingQuery(SearchQueryBuilder<GermplasmEntity> searchQuery, List<GermplasmEntity> germEntities) {


I think we've outgrown what the ORM can be reasonably used for, I would escape to SQL, either executing a database function (see this PR for an example) or creating a view that performs all the joins on the database side so that we can use a single query.

That said, I'm open to doing it this way if performance is acceptable (I still need to do functional testing) and we make a plan to stop doing this as soon as possible.

lol yea I kinda hate this whole method lol nothing technically wrong with it, its just not pretty
but it seems to be working for now, we'll keep an eye on it if future changes make it unnecessary

I understand the sentiment but disagree that the mentioned solution is a good fix.

The code is liable to break if the table or any associated tables undergo schema changes, and now you have to track down the procedure you made and fix it.

This ORM solution will stay intact regardless of schema changes (provided columns/relationships here are completely obliterated/changed).

A view tbh has a similar problem.

This is also breaks the convention that the application has total control of the SQL it needs to execute, which can be difficult to fix should the need arise.

All this said, yea this is an eyesore, but IMO this is more indicative that the actual schema/data model is the problem here, not the ORM.

And, to be fair, the only reason this eyesore method is needed is because you guys aren't paginating 😉

If you did, the updated optimizations to the paging version of this method will always deliver the performance you want, at least in my testing.

It seems reasonable to me to change data access code when the schema changes, but I realize nobody wants to maintain large PL/pgSQL functions or complex views.

I agree that pagination is the right approach when requesting a lot of data, ideally we can move to making paginated requests eventually.

BrapiCoordinatorSelby · 2025-03-27T18:49:40Z

src/main/java/org/brapi/test/BrAPITestServer/service/SearchQueryBuilder.java

+		if (sortClause.isEmpty()) {
+			// By default, sort on entity id to have query result remain idempotent
+			sortClause = " ORDER BY entity.id ASC ";
+		}


this could be moved to the constructor, line 25. Just to avoid maintaining the default in multiple places

I'm not sure that this check can be moved, but the default sort String definitely can, so I will do that.

This logic is meant to be called after a SearchQueryBuilder is constructed, because if the sort clause is non-empty, we should use that instead of this default.

These were simply IntelliJ suggestions I added in bc I was editing the surrounding code. I did not intend or want to change the way the spec or APIs work surrounding, at least not in this body of commits. To keep this work separate from the germ optimization work, I have reverted these changes. Also fixed a typo Matthew caught in ScaleService

mlm483

Tested locally along with the bi-api changes, it is working.

I did encounter an Out Of Memory Error when loading around 68k germplasm without pagination, but I had constrained the memory of the docker container to 4GB (and therefore the default JVM heap limit to 1GB, I think it defaults to 1/4 of available RAM in the docker image we're using). When I increased the available memory to 8GB (2GB JVM heap limit), it worked. Because we'll have 32GB (8GB JVM heap limit) in production, I think we'll have a decent amount of headroom, but it could still occur.

jloux-brapi added 8 commits March 19, 2025 11:56

Prevent pagination from occurring while join fetching on germs

8316721

Add SecurityUtils

cc5d9c6

Add default sort to all SearchQueryBuilder queries on entity id.

ec4055a

If not specified, this sort will be used to keep the endpoints idempotent.

Fix paging response metadata bug, patch UUID issue

1b06e85

Add error handling for bad pagination RQs to avoid giant queries and …

9ff25a5

…errors

Add configurable default page size and max allowed page size

2526b1e

Additionally make these configurable vars consistent and usable across BrAPIController and PagingUtility, which both utilize them.

Fix bug where pedigree attribute was misspelled

0f56f73

jloux-brapi requested review from dmeidlin, BrapiCoordinatorSelby, nickpalladino and mlm483 March 19, 2025 17:19

jloux-brapi changed the base branch from brapi-server-v2 to develop March 19, 2025 17:19

BrapiCoordinatorSelby reviewed Mar 21, 2025

View reviewed changes

jloux-brapi mentioned this pull request Mar 21, 2025

[BI-2579] Optimize Germplasm Import and Post Endpoint #51

Merged

jloux-brapi added 2 commits March 21, 2025 16:09

Remove erroneous imports from SearchRequest

d148e5d

Revert erroneous JsonbConverter change

55b1841

jloux-brapi commented Mar 24, 2025

View reviewed changes

Swap ref to externalReferenceID w externalReferenceId

407242c

jloux-brapi mentioned this pull request Mar 24, 2025

Germplasm search optimizations plantbreeding/brapi-Java-ProdServer#7

Closed

mlm483 requested changes Mar 27, 2025

View reviewed changes

BrapiCoordinatorSelby reviewed Mar 27, 2025

View reviewed changes

BrapiCoordinatorSelby approved these changes Mar 27, 2025

View reviewed changes

jloux-brapi added 2 commits March 28, 2025 12:23

Update comment, move default sort String to constructor

2de845c

jloux-brapi requested a review from mlm483 March 28, 2025 17:05

Merge branch 'develop' into germ-search-opts

38cf473

dmeidlin approved these changes Apr 3, 2025

View reviewed changes

mlm483 mentioned this pull request Apr 3, 2025

[BI-2578][BI-2489] - Optimize BrAPI Germplasm Search Breeding-Insight/bi-api#447

Merged

mlm483 approved these changes Apr 3, 2025

View reviewed changes

mlm483 self-assigned this Apr 3, 2025

nickpalladino approved these changes Apr 16, 2025

View reviewed changes

nickpalladino merged commit 828ed81 into develop Apr 16, 2025

Germplasm Search Optimizations #49

Germplasm Search Optimizations #49

Uh oh!

Conversation

jloux-brapi commented Mar 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mlm483 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mlm483 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!