feat: add support for custom evaluators

[radu-mocanu]

• add support for custom evaluators
• refactored coded evaluator files pushing to deduplicate code
• push custom evaluators (.py files) and evaluators types files

Add and register custom evaluator

1. (Optional) Add a new evaluator -> can be created manually in the evals/custom-evaluators directory

uipath add evaluator my_custom_evaluator

2. Implement the logic

3. Register the evaluator

uipath register evaluator my_custom_evaluator

4. Apply it to any dataset

The custom evaluator is self registering when the code is pushed

[Copilot]

Pull Request Overview

This PR adds support for custom evaluators by introducing CLI commands to create and register user-defined evaluators. Users can now extend the evaluation framework with custom logic tailored to their specific needs.

Key changes:

• Added uipath add evaluator command to scaffold new custom evaluators from a template
• Added uipath register evaluator command to generate evaluator specifications and type definitions
• Implemented dynamic loading of custom evaluators from the filesystem during evaluation runs

Reviewed Changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/uipath/eval/coded_evaluators/init.py	Exports base evaluator classes for custom evaluator implementation
src/uipath/_utils/constants.py	Defines custom evaluator prefix constant for file-based evaluator identification
src/uipath/_cli/cli_register.py	Implements registration command that parses custom evaluators and generates specs
src/uipath/_cli/cli_add.py	Implements add command that scaffolds new custom evaluators from template
src/uipath/_cli/_utils/_resources.py	Defines resource enum for CLI commands
src/uipath/_cli/_templates/custom_evaluator.py.template	Template file for generating new custom evaluators
src/uipath/_cli/_evals/_runtime.py	Fixed incorrect variable reference in eval execution
src/uipath/_cli/_evals/_evaluator_factory.py	Extended factory to dynamically load custom evaluators from files
src/uipath/_cli/init.py	Registers new CLI commands
samples/calculator/pyproject.toml	Updated dependency to local development version
samples/calculator/main.py	Added operator tracking for evaluation purposes
samples/calculator/evals/evaluators/types/correct-operator-evaluator-types.json	Generated type definition for custom evaluator
samples/calculator/evals/evaluators/correct-operator-evaluator.json	Generated spec for custom evaluator
samples/calculator/evals/eval-sets/default.json	Added custom evaluator to evaluation set
samples/calculator/evals/custom-evaluators/correct_operator.py	Example custom evaluator implementation
samples/calculator/README.md	Documentation for using custom evaluators

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}