|
1 | 1 | # LlamaIndex → TRACE |
2 | 2 |
|
3 | | -Emits a TRACE v0.2 Trust Record from LlamaIndex instrumentation events. |
| 3 | +Builds a first-party TRACE Trust Record from LlamaIndex tool observations. |
| 4 | +There are two distinct event routes: legacy instrumentation and the per-run |
| 5 | +workflow stream used by modern `FunctionAgent`. A global instrumentation |
| 6 | +handler alone does **not** observe `FunctionAgent` tool requests. |
| 7 | + |
| 8 | +## Modern FunctionAgent workflow |
| 9 | + |
| 10 | +The released-framework tests exercise `llama-index-core==0.14.24`, |
| 11 | +`llama-index-workflows==2.23.3`, and `llama-index-instrumentation==0.6.0`, with |
| 12 | +`agentrust-trace==0.9.0` and `agentrust-trace-tests==0.5.1`. Exact test pins live |
| 13 | +in [requirements-interop.txt](requirements-interop.txt). |
| 14 | + |
| 15 | +Dependency-audit limitation: the tested environment resolves LlamaIndex's |
| 16 | +transitive NLTK dependency to 3.10.3, affected by |
| 17 | +[GHSA-8mgp-746c-j5xp](https://github.com/advisories/GHSA-8mgp-746c-j5xp), |
| 18 | +with no patched release listed at verification time. These tests do not use |
| 19 | +NLTK's affected model-file APIs. A passing interoperability run is not a clean |
| 20 | +dependency-security audit; reassess dependencies for deployment. |
| 21 | + |
| 22 | +Use a fresh tracker for each run and pass events from that run's stream to |
| 23 | +`observe_workflow`. No global dispatcher registration is needed. Supply the |
| 24 | +model's identity explicitly from your configured agent; a process-global model |
| 25 | +observer could mix identities from concurrent runs. |
| 26 | +Once any event is passed to `observe_workflow`, record construction requires |
| 27 | +both explicit model fields, even for a run with no tool requests. |
4 | 28 |
|
5 | | -## First-party, like the LangChain adapter |
6 | | - |
7 | | -A `BaseEventHandler` runs in the agent's own process, so what it observes is the operator's own agent. The record carries **no `origin` block** — absence means `self`. It does not use [`agentrust-trace-adapters`](../../packages/agentrust-trace-adapters), which exists for *somebody else's* evidence and would mislabel this as a third-party transcription. |
8 | | - |
9 | | -## The risk here is different from LangChain's |
10 | | - |
11 | | -LangChain has a dozen typed callbacks. LlamaIndex has **one method** — `handle(event)` — and several event types carry payloads: |
12 | | - |
13 | | -| Event | Payload it carries | |
14 | | -|---|---| |
15 | | -| `AgentToolCallEvent` | `arguments` | |
16 | | -| `LLMChatStartEvent` | the whole `messages` list | |
17 | | -| `LLMCompletionEndEvent` | `prompt` and `response` | |
18 | | - |
19 | | -One entry point is easier to consume and harder to consume *safely*. So this handler reads an **explicit allow-list of fields** rather than the event object: tool name, event id, span id, and `model_dict` for identity. Nothing else is read, which means a payload-bearing field added upstream in a future version is ignored by default rather than captured. |
20 | | - |
21 | | -Four tests hold that line: an IBAN in tool `arguments`, an IBAN in chat `messages`, an IBAN in a field a later version might add, and an entire unrelated event type carrying prompt and response. |
| 29 | +```python |
| 30 | +from agentrust_trace.sign import sign_record |
| 31 | +from llamaindex_to_trace import TraceEventHandler |
22 | 32 |
|
23 | | -## Run it |
24 | 33 |
|
25 | | -```bash |
26 | | -pip install agentrust-trace llama-index-core |
| 34 | +async def run_with_record( |
| 35 | + agent, user_msg, *, subject, policy_bundle, workload_digest, |
| 36 | + model_provider, model_id, signing_key, |
| 37 | +): |
| 38 | + tracker = TraceEventHandler() |
| 39 | + handler = agent.run(user_msg=user_msg) |
| 40 | + try: |
| 41 | + async for event in handler.stream_events(): |
| 42 | + tracker.observe_workflow(event) |
| 43 | + result = await handler |
| 44 | + finally: |
| 45 | + if not handler.is_done(): |
| 46 | + await handler.cancel_run() |
| 47 | + |
| 48 | + unsigned = tracker.build_record( |
| 49 | + subject=subject, |
| 50 | + policy_bundle=policy_bundle, # bytes of the declared policy |
| 51 | + workload_digest=workload_digest, # digest of your artifact |
| 52 | + model_provider=model_provider, |
| 53 | + model_id=model_id, |
| 54 | + data_class="internal", |
| 55 | + ) |
| 56 | + return result, sign_record(unsigned, signing_key) |
27 | 57 | ``` |
28 | 58 |
|
| 59 | +The caller supplies its own configured agent, identity, policy bytes, artifact |
| 60 | +digest, and signing key. The offline tests supply a scripted local model and |
| 61 | +real local tools; they need no provider account, API key, or network requests. |
| 62 | +The stream has one consumer: if your application already processes it, call |
| 63 | +`observe_workflow` in that existing loop rather than starting a second consumer. |
| 64 | +The caller owns run cancellation and persistence. The adapter registers no |
| 65 | +hooks or background tasks, and does not label partial/cancelled observations |
| 66 | +as a successful run. |
| 67 | + |
| 68 | +### What enters the workflow transcript |
| 69 | + |
| 70 | +Each `ToolCall` contributes its `tool_name`, a SHA-256 fingerprint of `tool_id` |
| 71 | +in the existing `event_id` field, and `span_id: null`. Workflow events do not |
| 72 | +supply an instrumentation span. The call ID may be model-supplied, so its raw |
| 73 | +text is not retained. Fingerprints support correlation, not authentication or |
| 74 | +secrecy against guessing. Tool names remain visible metadata; do not put |
| 75 | +sensitive content in names. |
| 76 | + |
| 77 | +`ToolCallResult` and other events are ignored without reading their payloads. |
| 78 | +Arguments, results, prompts, responses, and arbitrary future fields never enter |
| 79 | +this transcript. One request plus one result counts once. Distinct requests |
| 80 | +with a repeated call ID still count separately: model IDs are not guaranteed |
| 81 | +unique. Replaying the stream is outside this observer's contract. |
| 82 | + |
| 83 | +In the tested framework, `ToolCall` is emitted **before lookup and execution**. |
| 84 | +The transcript therefore counts observed requests, including requests whose |
| 85 | +tool is unavailable or fails. It does not prove function-body execution, |
| 86 | +completion, business success, retry history, graph state, or exhaustive activity. |
| 87 | +Only events the caller actually passes to this tracker are observed. |
| 88 | + |
| 89 | +## Legacy instrumentation |
| 90 | + |
| 91 | +Existing `AgentToolCallEvent` handling is preserved. `LLMChatStartEvent` and |
| 92 | +`LLMCompletionStartEvent` can supply model identity through `model_dict`. |
| 93 | +The framework-free tests continue to check this route's explicit field |
| 94 | +allow-list. They do not establish support for every legacy LlamaIndex agent. |
| 95 | + |
29 | 96 | ```python |
30 | 97 | from llama_index.core.instrumentation import get_dispatcher |
31 | 98 | from llama_index.core.instrumentation.event_handlers import BaseEventHandler |
32 | 99 | from llamaindex_to_trace import TraceEventHandler |
33 | 100 |
|
34 | 101 | tracker = TraceEventHandler() |
35 | 102 |
|
| 103 | + |
36 | 104 | class Bridge(BaseEventHandler): |
37 | 105 | def handle(self, event, **kwargs): |
38 | 106 | tracker.observe(event) |
39 | 107 |
|
40 | | -get_dispatcher().add_event_handler(Bridge()) |
41 | | -# ... run your agent ... |
42 | 108 |
|
43 | | -record = tracker.build_record( |
44 | | - subject="spiffe://example.org/agent/index-bot", |
45 | | - policy_bundle=open("policy.cedar", "rb").read(), |
46 | | - # enforcement_mode defaults to "declared"; see below |
47 | | - workload_digest="sha256:...", |
48 | | - data_class="internal", |
49 | | -) |
| 109 | +bridge = Bridge() |
| 110 | +dispatcher = get_dispatcher() |
| 111 | +dispatcher.add_event_handler(bridge) |
| 112 | +try: |
| 113 | + # Run one legacy agent here, with no unrelated concurrent runs. |
| 114 | + ... |
| 115 | +finally: |
| 116 | + dispatcher.event_handlers.remove(bridge) |
50 | 117 | ``` |
51 | 118 |
|
52 | | -## `enforcement_mode` defaults to `declared` |
| 119 | +Global instrumentation is not per-run isolation. Do not feed legacy tool events |
| 120 | +and workflow tool requests into the same tracker: their identifiers cannot be |
| 121 | +reliably deduplicated. The first tool event selects a source; an event from the |
| 122 | +other source raises `MissingEvidence` before appending. Unknown event types |
| 123 | +are not recorded. Workflow requests with missing/non-string identity fields are |
| 124 | +also refused without retaining their values. |
| 125 | + |
| 126 | +## Evidence boundary and conformance |
| 127 | + |
| 128 | +These are in-process observations of the operator's own agent. Records have |
| 129 | +no `origin` block (self), default to `runtime.platform: software-only`, and |
| 130 | +retain `appraisal.status: none`. Signing binds the record to its signing key; |
| 131 | +it does not attest the observer, authenticate model-supplied tool identity, |
| 132 | +prove safe behavior, or establish hardware provenance or runtime integrity. |
| 133 | + |
| 134 | +`policy.enforcement_mode` defaults to `declared`: the caller's policy is named |
| 135 | +and hashed, but LlamaIndex has not evaluated or enforced it. Supplying another |
| 136 | +mode requires an actual external policy layer. Supplied attestation fields are |
| 137 | +passed through by the existing record builder; this adapter does not verify |
| 138 | +them or independently establish Level 1 assurance. |
53 | 139 |
|
54 | | -**LlamaIndex enforces no policy.** `enforce`, `advisory` and `silent` all presuppose that something *evaluated* the policy; for a bare run, nothing did. TRACE 0.9.0 added `declared` for that case, so the default is now truthful rather than the closest available overstatement. Needs `agentrust-trace>=0.9`. |
| 140 | +The real `FunctionAgent` tests sign and verify records and pass the released |
| 141 | +TRACE Level 0 conformance checks with the honest `declared` mode. Level 0 |
| 142 | +conformance does not raise this software-only evidence boundary. |
55 | 143 |
|
56 | | -## Conformance |
| 144 | +## Reproduce |
57 | 145 |
|
58 | | -**Level 0** without an attestation, **Level 1** with one. |
| 146 | +From the repository root in a fresh virtual environment: |
59 | 147 |
|
60 | 148 | ```bash |
61 | | -python -m pytest test_llamaindex_to_trace.py -q |
| 149 | +pip install pytest==9.1.1 agentrust-trace==0.9.0 |
| 150 | +python -m pytest integrations/llamaindex/test_llamaindex_to_trace.py -q |
| 151 | +pip install -r integrations/llamaindex/requirements-interop.txt |
| 152 | +python -m pytest integrations/llamaindex -q |
62 | 153 | ``` |
63 | 154 |
|
64 | | -20 tests. |
| 155 | +Or run `nox -s framework_adapters`, which preserves the framework-free pass |
| 156 | +and then installs the pinned released frameworks. CI runs both routes too. |
| 157 | +The real workflow regression is [test_llamaindex_interop.py](test_llamaindex_interop.py); |
| 158 | +it uses the released runner and a local scripted `MockFunctionCallingLLM`, not |
| 159 | +hand-constructed stand-ins for workflow delivery. Tests cover streaming and |
| 160 | +non-streaming model responses, request order, error and no-tool paths, |
| 161 | +concurrent run isolation, payload exclusion, and signed record validation. |
0 commit comments