Refactor and enhancements to saving of ML pipeline results

[mihow]

By addressing #607, we opened a can of bugs and missing features. So this led to a significant amount of refactoring and enhancements that are long overdue.

Changes:

• Begin storing label lists / category maps that algorithms use. The model is called AlgorithmCategoryMap and maps the class index from the last model layer to the actual categories they represent. The categories have both a simple text label "Species name" as well as an object of metadata with an optional GBIF key, taxon rank, etc. These are used to create taxon entries in the Antenna database as well as show the top 3,5,10, N number of predictions from a model rather than just the top 1 which is saved as the occurrence's determination.
• Moves job logs to their own field on the job model to reduce DB writes and overwriting the status field when writing logs.
• Add "task_type" field to algorithms so we can determine if its a classification model vs. a detection model and so on.
• Can handle results from multiple algorithms (moth/non-moth only)
• Faster & refactored saving of results (batch saving, split up functions, etc)

Benefits of this change:

• Can apply post-processing filters of species by region (filter and re-weight the scores)
• Can do genus & higher taxon roll-ups
• Can get the top N (top 3, 5, 10) suggestions instead of just the first
• Labels are now entities with GBIF key, synonyms, etc. instead of just a text label - So can better map to Taxon entities in the Antenna database
• Can use confidence score algorithms other than softmax on the Antenna side (e.g. temperature calibrated)
• Allow seeing and agreeing with results from multiple models
• Uses best score from all (non-intermediate) algorithms to determine the species determination of an occurrence (not just the latest)
• Ready for detection tracking across frames!
• Can better send current results back to backend to determine reprocessing needs

While troubleshooting I should have fixed most of the issues reported in #310

TODO:

• More testing with live ML backend
• Test migration of existing detections & classifications (they shouldn't have any data in the affected fields)
• Add Algorithm & AlgorithmCategoryMap schemas to ML backends
• Ensure the schemes in the example ML backend and live ML backend are the same
• Update tests
• Add new tests for category maps
• Add tests for creation of algorithms & category maps (document how to update either)
• Fix detection of already processed images (since we now return moth/non-moth classifications, all detections are seen as already processed by every pipeline)
• Fix occurrences using moth/non-moth algorithm as the determination (likely because score is higher)
• Fix algorithms being registered multiple times
• Fix category maps from being registered multiple times
• Raise errors that happen when saving results in subtasks
• Fix reprocessing - always is processing all images
• Test on fresh export of data from production
• Verify that full category map + scores map to expected taxa results - Try showing top 3 (logits are empty again?)
• Issues with cache not updating?? - this was due to errors happening in the subtasks for saving results
• Fix tests

[netlify]

✅ Deploy Preview for ami-dev canceled.

Name	Link
🔨 Latest commit	2522063
🔍 Latest deploy log	https://app.netlify.com/sites/ami-dev/deploys/678af27213ed5a0008591ede

[mihow]

Closing in favor of #684