feat(llmobs): support joining custom evaluations via tags

[lievan]

This PR implements LLMObs.submit_evaluation_for method, which gives users two options for joining custom evaluations

• by tag via the span_with_tag argument, which accepts a tuple containing a tag key/value pair
• by span via the span argument, which accepts a dictionary containing span_id and trace_id keys

There are also a couple behavior differences between submit_evaluation_for and submit_evaluation. In the new method, we

• throw whenever a required argument is the wrong value or type
• remove metadata argument
• move the warning log for missing api key to the eval metric writer's periodic method

Other changes:

Eval metric writer

Update the eval metric writer to write to the v2 eval metric endpoint. The main difference with this endpoint is that it accepts a join_with field that holds joining information instead of a top-level trace and span id fields.

Deprecate submit_evaluation

Deprecates submit_evaluation. I've set the removal version to be 3.0.0.

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

releasenotes/notes/submit-evaluation-for-01096d803d969e3e.yaml          @DataDog/apm-python
ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
ddtrace/llmobs/_writer.py                                               @DataDog/ml-observability
tests/llmobs/_utils.py                                                  @DataDog/ml-observability
tests/llmobs/llmobs_cassettes/tests.llmobs.test_llmobs_eval_metric_writer.send_score_metric.yaml  @DataDog/ml-observability
tests/llmobs/llmobs_cassettes/tests.llmobs.test_llmobs_eval_metric_writer.test_send_categorical_metric.yaml  @DataDog/ml-observability
tests/llmobs/llmobs_cassettes/tests.llmobs.test_llmobs_eval_metric_writer.test_send_metric_bad_api_key.yaml  @DataDog/ml-observability
tests/llmobs/llmobs_cassettes/tests.llmobs.test_llmobs_eval_metric_writer.test_send_multiple_events.yaml  @DataDog/ml-observability
tests/llmobs/llmobs_cassettes/tests.llmobs.test_llmobs_eval_metric_writer.test_send_score_metric.yaml  @DataDog/ml-observability
tests/llmobs/llmobs_cassettes/tests.llmobs.test_llmobs_eval_metric_writer.test_send_timed_events.yaml  @DataDog/ml-observability
tests/llmobs/llmobs_cassettes/tests.llmobs.test_llmobs_evaluator_runner.send_score_metric.yaml  @DataDog/ml-observability
tests/llmobs/llmobs_cassettes/tests.llmobs.test_llmobs_ragas_faithfulness_evaluator.emits_traces_and_evaluations_on_exit.yaml  @DataDog/ml-observability
tests/llmobs/llmobs_cassettes/tests.llmobs.test_llmobs_ragas_faithfulness_evaluator.test_ragas_faithfulness_emits_traces.yaml  @DataDog/ml-observability
tests/llmobs/test_llmobs_eval_metric_writer.py                          @DataDog/ml-observability
tests/llmobs/test_llmobs_service.py                                     @DataDog/ml-observability

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-01-08 23:33:22

Comparing candidate commit 86d204f in PR branch evan.li/submit-evaluation-for with baseline commit 75bed24 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 380 metrics, 2 unstable metrics.

[Yun-Kim]

Did a first quick pass, will do another pass on Monday!

[Yun-Kim]

Very minor comments/suggestions, but approving as none are super unblocking!

[datadog-dd-trace-py-rkomorn]

Datadog Report

Branch report: evan.li/submit-evaluation-for
Commit report: 86d204f
Test service: dd-trace-py

✅ 0 Failed, 87 Passed, 1468 Skipped, 3m 46.07s Total duration (35m 21.75s time saved)