feat: Test-based Evaluation with LLM-as-a-judge

[keshavramji]

This PR adds a flexible paradigm for evaluating a desired task(s) using LLM-as-a-judge evaluation on a set of tests for the task(s), with a corresponding (configurable) command line interface. Given a desired task and a set of instructions (guidelines) specifying intended model behavior, a user can provide a list of examples (consisting of inputs and (optionally) references / targets), and generate candidate responses from an LLM. A judge LLM is simultaneously instantiated and used to determine whether the candidates satisfy the guidelines.

• Added TestBasedEval component for evaluation with generator and judge sessions.
• Implements m eval run CLI command for configurable models and backends for the generator and juge.
• Supports flexible test format with multiple examples per test, each containing input-target pairs evaluated against custom instructions.
• Adds a Jinja template corresponding to the component -- TestBasedEval.jinja2 -- for pass/fail scoring.

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert|release)(?:\(.+\))?:

[avinash2692]

@keshavramji : Thanks for your contribution!

This might be due to my ignorance, but I'm having some trouble understanding the overall goal of the PR and what a "Test based Evaluation" entails. It might be helpful if you add:

• More details to your PR description that might later translate as an entry in the library docs.
• Some tests (which could include sample .json files to show the user what you are looking for in terms of overall schema of input).

Also left some comments on the files here.

[avinash2692]

Could this be removed? I think it's user specific

[avinash2692]

Is there a reason for this to be a package level import?

[keshavramji]

Just tested it without having the package level import, confirmed it's fine to remove.

[avinash2692]

This might be cause I don't have a proper understanding of what you're trying to achieve here, but could this be moved as a Jinja template under mellea/templates/prompts to generate a mellea.stdlib.base.TemplateRepresentation: ? You might also want to use the format_for_llm method in tandem with the TemplateRepresentation to create the Component

[keshavramji]

I have added a TestBasedEval.jinja2 template and changed test_based_eval.py to use TemplateRepresentation, per your suggestion.

[keshavramji]

Hi @avinash2692, thanks much for the feedback! I've looked to address each of your comments above, and have also updated the PR description for further clarity on the intended usage. For additional context, here is an example of the intended schema for a test file (in JSON format):

[
  { 
    "id": "test1"
    "source": "example_tests"
    "name": "Basic Addition"
    "instructions": "1. The answer should produce correct calculations. 2. The answer format should be consistent with the format described in the input question."
    "examples": [
        {
          "input_id": "test1.1",
          "input": [
             {
               "role": "user",
               "content": "2 + 2 = ?"
             }
          ],
          "targets": [
             {
               "role": "assistant",
               "content": "2 + 2 = 4"
             }
          ] 
        },
        {
          "input_id": "test1.2"
          "input": [
            {
              "role": "user",
              "content": "3 + 5 = ?"
            }
          ],
          "targets": [
            {
              "role":"assistant",
              "content": "3 + 5 = 8"
            }    
          ]
        }
     ]
  }
]

[avinash2692]

LGTM