Open source evaluation code?

[Sunishchal]

Thank you for this detailed survey of LLM judges!

My team is starting a project to build a unified framework for LLM judging and would like a way to automatically evaluate the human agreement and bias of each method. Would you consider open sourcing the code you used to get the results in section 5 of your paper?

We would also require a permissive license like MIT so we can use the code without copyright issues. That would make things much easier for our project :)

[Kuvvius]

Thank you for your interest in our work! We’re glad to hear that our survey was helpful for your project.

We are currently in the process of preparing the code for open-sourcing. Before releasing it, we plan to refine and improve certain aspects to ensure better usability and completeness, so it might take some time.

We appreciate your patience and look forward to sharing it soon! Please feel free to reach out if you have any specific requirements or suggestions.

[Sunishchal]

This is great to hear. Do you have an estimate for when it might be released? If it's available within the next month, I think it will be in time for us to use in our project.

Have you seen the new Verdict library? It may simplify the engineering effort for building the improved judges. I would also recommend building the evaluation using the Inspect framework to standardize the implementation with other evals and get nice logging features. They have basic support for LLM judges, but it may be a little challenging to fully support all the improvements you tested in your paper.

[Sunishchal]

Hi @Kuvvius I'm jut checking back for a status update here. Do you have a timeline for when the evaluation code might be released?

I'm currently in the process of implementing several LLM-as-a-Judge benchmarks in the Inspect framework and would be happy to help implement yours too if you're able to provide the necessary materials. Let me know!