Split LLM server

[ramashishb-yubi]

Hi,

First of all thanks for creating this wonderful repo! I was able to get privateGPT running on a local server. I need help to run it in a slightly different configuration as below.

• I want to split the LLM backend so that it can be run on a separate GPU based server instance for faster inference.
• I want to create one or more privateGPT instances which can connect to the LLM backend above for model inference and run the rest of the part (RAG, document ingestion, etc.) locally.

Has anyone tried this or has any idea of how this can be done? Appreciate any help!

Thanks,
Ramashish

[tlinnet]

Yes, install ollama and fetch model there. Then configure privategpt towards this.

[ramashishb-yubi]

Thank you @tlinnet for your quick response. I will check it out.