Has anybody integrate a LLM served with vLLM into Elastic( AI assistance for observability) with succes?

[DanielBeck93]

Hi,

I have tried to integrate models served with vLLM into Elastic AI assistance in Observerbality without success. I have at a point succeded with https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/main but now it is to slow even on 48GB GPU. And Llama-3.1-8B instruct does not work at all.

Have any of you succeded with this? Will be nice to share some experience.