Create a distilled version of Raphael's LLM

[dcorney]

Overview

Currently, we pass each chunk of transcript to Gemini via the Vertex API. This is fine but slow and expensive.
We should distil our own version of an equivalent PaLM2 model so we can host this on our own cloud instance.

This work should follow on from the equivalent issue in the GenAI semantic enricher project.

Requirements

• Update the code here to use PaLM 2 instead of Gemini. Consider whether fine-tuning or in-context-learning is more appropriate.
• Check the new model's accuracy is still good
• Distil the PaLM 2 model
• Deploy to our own server and re-direct calls to it instead of Gemini