FEAT Add a custom REST LLM API template to Garak Config yaml file

[EricXQiu]

Summary

Garak supports REST APIs but I would like to know how to add the request template in the yaml config file.

Basic example

For example, the request template can be as follows:

{
    "messages": [
        {"role":"system", "content": "You are a helpful assistant."},
        {"role": "user", "content": <actual_text>}
    ]
}

How to add this to the config.yaml file?

Motivation

This will help for users to use garak with YAML file.

[jmartin-tech]

Configuration is loaded via json and yaml libraries, these treat the file formats as serialized object, that contain only basic types, primitives such as string, int, float as well as dict and list are supported. The restGenerator accepts such an object in req_template_json_object which it will serialize into a string for the request.

Your example snippet placed in a template if passed in a -G json file would be:

{
   "rest": {
      "RestGenerator": {
         "name": "example service",
         "uri": "https://example.ai/llm",
         "method": "post",
         "headers": {
            "X-Authorization": "$KEY"
         },
         "req_template_json_object": {
            "messages": [
               {
                 "role":"system",
                 "content": "You are a helpful assistant."
               },
               {
                 "role": "user",
                 "content": "$INPUT"
               }
            ]
         },
         "response_json": true,
         "response_json_field": "text"
      }
   }
}

Note the $INPUT is the keyword for <actual_text> that is replaced with the prompt from the probe.

As yaml this would be specified as:

---
plugins:
  generators:
    rest:
      RestGenerator:
        name: example service
        uri: https://example.ai/llm
        method: post
        headers:
          X-Authorization: "$KEY"
        req_template_json_object:
          messages:
          - role: system
            content: You are a helpful assistant.
          - role: user
            content: "$INPUT"
        response_json: true
        response_json_field: text

Note the additional structure for plugins -> generators needed for the yaml config file to clarify the configuration is for generators.

A helpful tool for this conversion can be found here and some general documentation on how configure garak can be found in the reference docs.

[EricXQiu]

Thanks Jeffrey. I think this is good to close.