[Feature] Add LlamaCPP as server engine option

[rgerganov]

Description

llama.cpp is exposing OpenAI compatible APIs and works perfectly fine with genai-bench. This patch adds it as server engine option

[gemini-code-assist]

Summary of Changes

Hello @rgerganov, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces LlamaCPP as a new server engine option within the genai-bench framework. By integrating LlamaCPP, users can now leverage its OpenAI compatible APIs to benchmark models, thereby expanding the range of supported server technologies for performance evaluation. This enhancement provides greater flexibility and broader compatibility for testing various large language models.

Highlights

• New Server Engine Option: The LlamaCPP server engine has been added as a selectable option for genai-bench.
• Expanded Compatibility: This change allows genai-bench to utilize llama.cpp's OpenAI compatible APIs for benchmarking.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds LlamaCPP as a server engine option. While this is useful for metadata, the implementation is incomplete for providing full, user-friendly support for llama.cpp as a backend. For consistency with other OpenAI-compatible backends like vLLM and SGLang, LlamaCPP should also be added as an --api-backend option. I've left a comment with details on how to achieve this. Making these changes will significantly improve the usability and maintainability of this new feature.

[gemini-code-assist]

Adding LlamaCPP to --server-engine is a good step for metadata. However, to provide a complete and user-friendly integration for llama.cpp, it should also be treated as a first-class API backend, similar to vLLM and SGLang.

This would involve:

1. Adding llamacpp (lowercase, for consistency) to the click.Choice for the --api-backend option in api_options within this file.
2. In genai_bench/cli/validation.py, adding 'llamacpp': OpenAIUser to the API_BACKEND_USER_MAP.
3. Also in genai_bench/cli/validation.py, adding 'llamacpp' to the api_key_required list in validate_api_key.
4. In genai_bench/cli/cli.py, adding 'llamacpp' to the lists for auth_kwargs and auth_backend_map to handle authentication.

These changes would make LlamaCPP a fully supported backend, improving consistency and user experience.

[CatherineSue]

@rgerganov Can you fix the CI failure? And please take a look at the above comment from gemini.

[rgerganov]

@rgerganov Can you fix the CI failure? And please take a look at the above comment from gemini.

The CI should be fixed now but I cannot trigger it. The gemini comment doesn't make sense -- llamacpp is exposing OpenAI backend and doesn't require API keys, so no other changes are needed, similar to the TGI server engine option.