Skip to content

Add ApplyGuardrailScope for security guardrail evaluation tracing#252

Merged
nikhilNava merged 3 commits into
mainfrom
nikhilc/addGuardrailSpan
May 20, 2026
Merged

Add ApplyGuardrailScope for security guardrail evaluation tracing#252
nikhilNava merged 3 commits into
mainfrom
nikhilc/addGuardrailSpan

Conversation

@nikhilNava

@nikhilNava nikhilNava commented May 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Introduces a new OpenTelemetry tracing scope (\ApplyGuardrailScope) that captures security guardrail evaluations as spans. This enables observability into content safety, policy enforcement, and risk assessment decisions made during agent operations.

Changes

New Contracts

  • *\GuardrailDetails* — Immutable contract capturing guardian evaluation metadata: target type, decision, provider info, policy details, and content hash for forensic correlation.
  • *\GuardrailFinding* — Represents an individual risk finding with severity, category, and confidence score.
  • *\GuardrailDecisionType* — Enum for well-known guardian decisions (Allow, Block, Modify, Flag, Error).
  • *\GuardrailRiskSeverity* — String constants for risk severity levels (Low, Medium, High, Critical).
  • *\GuardrailTargetType* — Extensible type for guardrail targets (UserInput, ModelOutput, ToolCall, etc.).

New Tracing Scope

  • *\ApplyGuardrailScope* — Disposable OpenTelemetry scope following the same pattern as \InvokeAgentScope, \InferenceScope, and \ExecuteToolScope. Supports recording findings post-creation and integrates with the existing span hierarchy.

ETW Support

  • *\ApplyGuardrailData\ / \ApplyGuardrailDataBuilder* — DTO and builder for Event Tracing for Windows logging of guardrail events.
  • Updated \IA365EtwLogger\ and \A365EtwLogger\ with guardrail-specific logging methods.

OpenTelemetry Constants

  • Added \microsoft.a365.guardrail.*\ attribute keys to \OpenTelemetryConstants.

Tests

  • Unit tests covering scope creation, finding recording, attribute emission, and edge cases.
  • Integration test validating end-to-end export of guardrail spans.

Design Notes

  • Guardian spans are children of the operation they protect (e.g., inference or tool execution spans).
  • Multiple guardian spans may exist under a single operation when guardrails are chained.
  • Follows the existing Core + Extensions pattern and disposable scope lifecycle.

Manual/Integration test
below is the generated span

 {
   "name": "apply_guardrail Azure Content Safety llm_input",
   "traceId": "4ceed0b2561683e35af42be964263022",
   "spanId": "dd203f5da4b9be28",
   "parentSpanId": "0000000000000000",
   "kind": "Internal",
   "startTimeUnixNano": 639148092770709545,
   "endTimeUnixNano": 639148092770758119,
   "attributes": {
     "gen_ai.operation.name": "apply_guardrail",
     "gen_ai.agent.id": "03806323-fbc0-4529-9b65-403bf2401661",
     "gen_ai.agent.name": "Guardrail Agent",
     "gen_ai.agent.description": "Agent for guardrail testing.",
     "microsoft.agent.user.id": "055398b7-037e-4b92-bb1f-2e569aa14a4a",
     "microsoft.agent.user.email": "guardrailagent@ztaittest12.onmicrosoft.com",
     "microsoft.a365.agent.blueprint.id": "9d3108da-236e-464d-953e-7f4b911af7a7",
     "microsoft.tenant.id": "a70eaef2-9c8d-49e8-b735-2931dff8fe68",
     "user.id": "guardrail-caller-123",
     "user.email": "guardrail-caller@ztaitest12.onmicrosoft.com",
     "user.name": "Guardrail Caller",
     "client.address": "192.168.1.42",
     "microsoft.security.decision.type": "deny",
     "microsoft.security.target.type": "llm_input",
     "microsoft.guardian.id": "azure-content-safety-001",
     "microsoft.guardian.name": "Azure Content Safety",
     "microsoft.guardian.provider.name": "Azure",
     "microsoft.guardian.version": "2.0.0",
     "microsoft.security.target.id": "msg-12345",
     "microsoft.security.decision.reason": "Content violates hate speech policy",
     "microsoft.security.decision.code": "HATE_SPEECH_001",
     "microsoft.security.policy.id": "policy-abc",
     "microsoft.security.policy.name": "Content Safety Policy",
     "microsoft.security.policy.version": "1.2.0",
     "microsoft.security.content.input.hash": "sha256:abc123def456",
     "microsoft.security.content.modified": false,
     "microsoft.security.external_event_id": "ext-event-789",
     "microsoft.security.content.input.value": "Check this content for safety",
     "microsoft.channel.name": "msteams",
     "microsoft.channel.link": "https://guardrail-channel.link",
     "microsoft.security.content.output.value": "sanitized-hash-output"
   },
   "events": [
     {
       "name": "microsoft.security.finding",
       "timeUnixNano": 639148092770742787,
       "attributes": {
         "microsoft.security.risk.category": "hate_speech",
         "microsoft.security.risk.severity": "high",
         "microsoft.security.policy.decision.type": "deny",
         "microsoft.security.policy.id": "policy-abc",
         "microsoft.security.risk.score": 0.95,
         "microsoft.security.risk.metadata": [
           "{\"category\":\"hate\",\"confidence\":0.95}"
         ]
       }
     }
   ]
 }

Introduce a new OpenTelemetry tracing scope (ApplyGuardrailScope) that captures
security guardrail evaluations as spans. This enables observability into content
safety, policy enforcement, and risk assessment decisions made during agent
operations.

New components:
- ApplyGuardrailScope: Disposable tracing scope for guardrail spans
- GuardrailDetails: Contract for guardian evaluation metadata (target, decision,
  policy, provider info)
- GuardrailFinding: Individual risk finding with severity and category
- GuardrailDecisionType, GuardrailRiskSeverity, GuardrailTargetType: Enums for
  well-known guardrail values
- ApplyGuardrailData/Builder: DTO and builder for ETW event logging
- OpenTelemetryConstants: New attribute keys for guardrail spans

Also includes unit tests and integration test coverage for the new scope.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nikhilNava nikhilNava requested a review from a team as a code owner May 19, 2026 17:26
Copilot AI review requested due to automatic review settings May 19, 2026 17:26
Add AddSdkAttributes method to BaseDataBuilder that includes
telemetry.sdk.name, telemetry.sdk.version, and telemetry.sdk.language
in ETW event payloads, ensuring parity with the OpenTelemetry span
pipeline where ActivityProcessor sets these same attributes.

Updated all concrete builders (InvokeAgent, ExecuteInference,
ExecuteTool, OutputMessages, ApplyGuardrail) to call AddSdkAttributes
at the start of BuildAttributes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented May 19, 2026

Copy link
Copy Markdown

⚠️ Deprecation Warning: The deny-licenses option is deprecated for possible removal in the next major release. For more information, see issue 997.

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a new ApplyGuardrailScope OpenTelemetry tracing scope (plus ETW DTOs/logging) to emit guardrail evaluation spans and finding events for security/guardrail observability.

Changes:

  • Introduces ApplyGuardrailScope and new guardrail contracts (GuardrailDetails, GuardrailFinding, decision/target/severity types) with new microsoft.security.* / microsoft.guardian.* attributes.
  • Adds ETW support via ApplyGuardrailData + builder and extends IA365EtwLogger/A365EtwLogger with LogApplyGuardrail.
  • Adds unit + integration tests covering span creation and finding event emission, and adds SDK attributes to multiple DTO builders.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
src/Tests/Microsoft.Agents.A365.Observability.Runtime.Tests/Tracing/Scopes/ApplyGuardrailScopeTest.cs New unit tests validating guardrail span tags, naming, decision updates, findings, and errors.
src/Tests/Microsoft.Agents.A365.Observability.Runtime.IntegrationTests/Agent365ExporterE2ETests.cs Adds an integration-style test that captures an Activity for ApplyGuardrailScope and asserts emitted tags/events.
src/Observability/Runtime/Tracing/Scopes/OpenTelemetryScope.cs Adds a protected helper to add ActivityEvent to the current activity.
src/Observability/Runtime/Tracing/Scopes/OpenTelemetryConstants.cs Adds operation name constant and guardrail-related semantic attribute keys.
src/Observability/Runtime/Tracing/Scopes/ApplyGuardrailScope.cs New scope implementation that sets guardrail/guardian/policy/content tags and emits finding events.
src/Observability/Runtime/Tracing/Contracts/GuardrailTargetType.cs New extensible target type wrapper with well-known values + implicit string conversions.
src/Observability/Runtime/Tracing/Contracts/GuardrailRiskSeverity.cs Adds well-known risk severity string constants.
src/Observability/Runtime/Tracing/Contracts/GuardrailFinding.cs New finding contract used to emit per-finding events/tags.
src/Observability/Runtime/Tracing/Contracts/GuardrailDetails.cs New guardrail evaluation details contract used to populate scope attributes.
src/Observability/Runtime/Tracing/Contracts/GuardrailDecisionType.cs New enum of guardrail decisions (allow/audit/deny/modify/warn).
src/Observability/Runtime/Etw/IA365EtwLogger.cs Adds LogApplyGuardrail API to the ETW logger interface.
src/Observability/Runtime/Etw/A365EtwLogger.cs Implements LogApplyGuardrail and assigns event id/name.
src/Observability/Runtime/DTOs/Builders/OutputDataBuilder.cs Adds SDK attributes to Output DTO attributes.
src/Observability/Runtime/DTOs/Builders/InvokeAgentDataBuilder.cs Adds SDK attributes to InvokeAgent DTO attributes.
src/Observability/Runtime/DTOs/Builders/ExecuteToolDataBuilder.cs Adds SDK attributes to ExecuteTool DTO attributes.
src/Observability/Runtime/DTOs/Builders/ExecuteInferenceDataBuilder.cs Adds SDK attributes to ExecuteInference DTO attributes.
src/Observability/Runtime/DTOs/Builders/BaseDataBuilder.cs Adds shared AddSdkAttributes helper.
src/Observability/Runtime/DTOs/Builders/ApplyGuardrailDataBuilder.cs New builder for ETW DTO attribute population for apply_guardrail.
src/Observability/Runtime/DTOs/ApplyGuardrailData.cs New ETW DTO representing apply_guardrail telemetry.

Comment thread src/Observability/Runtime/Tracing/Scopes/ApplyGuardrailScope.cs
Comment thread src/Observability/Runtime/Tracing/Contracts/GuardrailDetails.cs
Comment thread src/Observability/Runtime/Tracing/Contracts/GuardrailDetails.cs
Comment thread src/Observability/Runtime/Tracing/Contracts/GuardrailFinding.cs
Comment thread src/Observability/Runtime/Tracing/Scopes/ApplyGuardrailScope.cs
Comment thread src/Observability/Runtime/Tracing/Scopes/ApplyGuardrailScope.cs Outdated
- Throw ArgumentNullException instead of silently no-oping on null,
  matching the non-nullable parameter signature.
- Remove unused System.Runtime.Serialization import.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nikhilNava nikhilNava enabled auto-merge (squash) May 19, 2026 18:11
@nikhilNava nikhilNava merged commit e0bccef into main May 20, 2026
7 checks passed
@nikhilNava nikhilNava deleted the nikhilc/addGuardrailSpan branch May 20, 2026 16:47
nikhilNava added a commit to microsoft/opentelemetry-distro-dotnet that referenced this pull request May 27, 2026
* Add ApplyGuardrailScope for security guardrail evaluation tracing

Introduces a new OpenTelemetry tracing scope (ApplyGuardrailScope) that
captures security guardrail evaluations as spans. This enables
observability into content safety, policy enforcement, and risk
assessment decisions made during agent operations.

New contracts:
- GuardrailDetails: Immutable contract for guardian evaluation metadata
- GuardrailFinding: Individual risk finding with severity and score
- GuardrailDecisionType: Enum for guardian decisions (Allow, Deny, etc.)
- GuardrailRiskSeverity: String constants for risk severity levels
- GuardrailTargetType: Extensible type for guardrail targets

New tracing scope:
- ApplyGuardrailScope: Disposable OTel scope with RecordDecision,
  RecordContentOutput, and RecordFinding methods

Also adds:
- microsoft.security.* and microsoft.guardian.* OTel attribute keys
- AddEvent helper on OpenTelemetryScope base class
- AddSdkAttributes helper on BaseDataBuilder
- ETW logging support via LogApplyGuardrail
- ApplyGuardrailData DTO and builder
- 12 unit tests covering scope creation and attribute emission

Ref: microsoft/Agent365-dotnet#252

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add apply_guardrail to exporter allow-list and add test

The ApplyGuardrailOperationName was missing from the
GenAiOperationNames set in Agent365ExporterCore, which would cause
guardrail spans to be filtered out as NonGenAI in PartitionByIdentity.

Also adds a test verifying apply_guardrail spans are included in
exported batches.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add AddSdkAttributes call to ApplyGuardrailDataBuilder

Aligns with the other DTO builders (Output, InvokeAgent, ExecuteTool,
ExecuteInference) which all call AddSdkAttributes as the first step in
BuildAttributes to include telemetry.sdk.name/version/language.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Nikhil Chitlur Navakiran (from Dev Box) <nikhilc@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants