Consider moving to chat/completions API

[markmc]

From @gabe-l-hart here

it looks like these prompt templates are largely encoding the chat template for the model. For most models that I've seen, the chat template is actually encoded in the model metadata and applied server-side when using the chat/completions API. Has there been thought around using the chat API rather than applying the template client-side to the lower-level completions API?

[github-actions]

This issue has been automatically marked as stale because it has not had activity within 90 days. It will be automatically closed if no further activity occurs within 30 days.

[bbrowning]

Unstale - some of the research work we're merging in starts moving us over to the chat completions API instead of the generic completions API. So, we will be addressing at least the spirit of this issue.

[bbrowning]

#461 adds an LLMMessagesBlock that uses the chat completions API, as opposed to the LLMBlock in Pipelines that uses the older completions API. However, adding that block is just the first step towards moving us over - we still need to evaluate how/if we'd move our default pipelines over to the chat completions API instead of the completions API.

[bbrowning]

We have blocks to support both chat completions and old style completions, but still default to old style completions because it gives us the ability to send multiple prompts in a single inference call as a batch as opposed to sending a chat completion request per inference call. In testing it makes a meaningful difference in HTTP connection overhead if we can no longer send batches of prompts in per request, to the point where we have to start worrying about running out of open tcp sockets, file handles, maximum connections in the networking/routing layers, and so forth both on the client and teacher model server.

So, closing this as we won't just port our existing things over to chat completions but also may separately look into some of the batch inference APIs to allow us to accomplish similar server-side chat template management without also introducing a lot of networking overhead.