[BI-2579] Optimize Germplasm Import and Post Endpoint

[jloux-brapi]

These changes should be merged on top of #49, as they are built on a branch based off those changes.

The thruput of the BrAPI Germplasm POST was very slow at the time I started working on this import.

An import of 30k germs on a clean DB or a program with no germs would take somewhere in the ballpark of 30-60 minutes.

Once the number of germs per program reached around 60-90k, the import would take hours or never finish at all.

After the host of changes made in the body of this MR, the 30k file can be imported in around 3-6 minutes on a DB with 550k germs on it with about 250k germs per program already loaded in.

It should be noted the import is much faster, but issues with the cache remain.

The associated bi-api changes edited the import process so that the cache is only refreshed at the end of the import. But as the number of records grows, the cache takes longer and longer to return, and this can impact the experience of the import.

The performance is still worlds better than it was before, but sometimes the cache getting can hold things up, and because it happens at the end of the import, users may find that when they try to view the germplasm records is takes a long time to load because the cache is still fetching.

I'll try to summarize the optimizations made:

• Optimized the Germplasm save so that it creates entities in batch, and also does lookups for these entities in batch, rather than one by one. This cut down about 2000 queries on an import size of 1000 records at a time.
• Removed saveAndFlushAlls everywhere. After some research, I found that flushalls can greatly hurt performance time because if entities being saved reference entities that were flushed (as a lot of this code seems to do) hibernate has to refresh the entities, or sometimes it doesn't even know what to do and gives an error.
• Batched Pedigree creation from Germplasm records. There were kind of a bunch of optimizations tied together in this. It's hard to estimate how many queries were killed from this batching. anywhere between 20-30k. Without the changes from Germplasm Search Optimizations #49, performance suffered even more because the pedigree batching code kicked off full germplasm search requests, which tried paginating and fetching in memory, not only slowing the performance but bringing memory allocation to a screeching halt.
  • Batch the creation of pedigree records from germplasm when no pedigree string came in
  • Batch the creation of pedigree records from germplasm when pedigree string came in.
  • Batch the creation of pedigree edges
  • Batch the creation of pedigree edges from parents
  • Batch the creation of pedigree edges from progeny (children)
• I found that after these optimizations were made, an extraordinary amount of deletions were occurring on the pedigree_node_external_references table, and brought the import to a screeching halt at scale. This ended up being due to the UpdateUtility unnecessarily clearing out existing external references that actual in fact already matched the ones that existed. A check was put in place to prevent this from ever occurring, however I only applied this check to the PedigreeNode code. Another solution to this problem could have been to just prevent setting pedigree external references entirely, as they likely do not need the same ones as germplasm. But this issue seemed so severe and worrisome for other entities I thought it would be helpful to keep the exref checking code for reuse.
• After this discovery, I hit another sticking point: Thousands of queries being created to convert the pedigree database data to DTOs which was being completely dropped on the floor in the case of the Germplasm POST. I created some new method signatures to prevent the dto conversion from ever happening for this use case. NOTE: This does not affect the data returned, as the data returned only returns germplasm data, and I kept those conversions untouched.
• Added batching spring.data properties to the application.properties.template. These must be replicated to the application.properties of the relevant application/deployment, as they are imperative for speedups relating to batch inserts