feat(llmobs): improved LLM Observability data submission experience

[sabrenner]

What does this PR do?

This PR does a couple of things:

1. LLMObs evaluation metrics can be submitted through an agent proxy
2. submitEvaluation will not check for a present API key. If it is not set for agentless data submission, it will now be thrown by the module logic instead.
3. Adds a new feature to detect if an agent is running and has the appropriate endpoint to determine agentlessEnabled if it is not explicitly set by the user

Motivation

MLOB-2444

[github-actions]

Overall package size

Self size: 9.2 MB
Deduped: 101.51 MB
No deduping: 102.02 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.5.0 | 29.83 MB | 29.83 MB | | @datadog/native-appsec | 8.5.1 | 19.26 MB | 19.27 MB | | @datadog/native-iast-taint-tracking | 3.3.0 | 13.77 MB | 13.78 MB | | @datadog/pprof | 5.6.0 | 9.79 MB | 10.16 MB | | @opentelemetry/core | 1.30.1 | 908.66 kB | 7.16 MB | | protobufjs | 7.4.0 | 2.77 MB | 5.42 MB | | @datadog/wasm-js-rewriter | 3.1.0 | 2.37 MB | 2.52 MB | | @datadog/native-metrics | 3.1.0 | 1.06 MB | 1.46 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.13.1 | 117.64 kB | 839.26 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | lru-cache | 7.18.3 | 133.92 kB | 133.92 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.1 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 5.3.2 | 53.63 kB | 53.63 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.4.1 | 27.15 kB | 27.15 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | dc-polyfill | 0.1.6 | 24.56 kB | 24.56 kB | | shell-quote | 1.8.2 | 23.54 kB | 23.54 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

[datadog-datadog-prod-us1]

Datadog Report

Branch report: sabrenner/llmobs-writers-changes
Commit report: a77fceb
Test service: dd-trace-js-integration-tests

✅ 0 Failed, 926 Passed, 0 Skipped, 4m 5.81s Total Time

[codecov]

Codecov Report

Attention: Patch coverage is 60.52632% with 30 lines in your changes missing coverage. Please review.

Project coverage is 79.14%. Comparing base (b94052c) to head (d096d90).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
packages/dd-trace/src/llmobs/writers/util.js	32.14%	19 Missing ⚠️
packages/dd-trace/src/llmobs/writers/base.js	71.42%	8 Missing ⚠️
packages/dd-trace/src/llmobs/writers/spans.js	77.77%	2 Missing ⚠️
packages/dd-trace/src/llmobs/index.js	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5466      +/-   ##
==========================================
- Coverage   79.18%   79.14%   -0.04%     
==========================================
  Files         512      511       -1     
  Lines       23158    23189      +31     
==========================================
+ Hits        18338    18354      +16     
- Misses       4820     4835      +15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-04-02 13:44:52

Comparing candidate commit d096d90 in PR branch sabrenner/llmobs-writers-changes with baseline commit b94052c in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 946 metrics, 17 unstable metrics.

[sabrenner]

moved this telemetry metric to be inside the agent strategy callback - do we want to define our llmobs startup/enabled metric to include this async op, or just do it at the end of this function synchronously?

[sabrenner]

i've decided to special-case this here to make the message a bit clearer/more actionable, and still send the telemetry event as request_error. I figure maybe down the line we could special case some more things as applicable!

I also have telemetry submission here too but now that i think about it, if the api key and site don't match, and they're likely using agentless telemetry too, i doubt the telemetry event would even make it to our end...