Need the option to mask the input and output of the LLM API in Datadog LLM observability

[Gekko0114]

The input and output of the LLM API contain sensitive information, so I don't want to send them to Datadog.
I would like to have an option to send data to Datadog with the input and output masked.
If it's okay, I want to create a PR for this.

Previously I raised the same question in this issue #10517 .
In this issue, I heard that APM integration dashboards should be used for these four providers (OpenAI/Bedrock/LangChain/Anthropic).
However, I would like to monitor Gemini.
According to #10971, Gemini will not be supported in APM integration because LLM obs will support Gemini.

Therefore I need an option to mask the input and output of the LLM API in Datadog LLM observability.
If it is okay, I would like to create a PR for this.

[Gekko0114]

Hi @Yun-Kim
Gently ping

[Gekko0114]

Hi,
It would be nice if you could give me any feedback.

[Kyle-Verhoog]

Hi @Gekko0114! Are you using VertexAI to use Gemini? If so, we have an integration coming in the next few weeks which will include controls to omit the input and output. Also, we're looking for people to try the integration as we develop it, would you be interested in partnering with us for this?

[Gekko0114]

Are you using VertexAI to use Gemini? If so, we have an integration coming in the next few weeks which will include controls to omit the input and output

Yes, I am using VertexAI. Sounds great!

Also, we're looking for people to try the integration as we develop it, would you be interested in partnering with us for this?

No problem! What should I do?

[Kyle-Verhoog]

Awesome, @Gekko0114 are you in our public Slack? If not could you join and we can follow up with you once we have a build ready for you to try!

[Gekko0114]

Hi @Kyle-Verhoog
Thanks, I've just joined the slack, but I couldn't find you because this channel has a lot of people named Kyle.
Which channel should I join? And could you let me know your account name?

[yj-ang]

Not sure if it's off topic. I tried with setting env DD_LANGCHAIN_SPAN_PROMPT_COMPLETION_SAMPLE_RATE="0.0" stated in the LangChain integration but I could still see all the content (input and output) visible under the LLM Obs Traces dashboard.

I want to collect the metrics but hide the input completely or having option to mask the sensitive information like phone number and email, I wonder if we could get flexibility to control over that.

[Kyle-Verhoog]

@Gekko0114 you should be able to find me under "Kyle Verhoog", send me a message!

[Kyle-Verhoog]

@yj-ang this sounds like a bug 🤔. I will do some investigation and get back to you.

[github-actions]

This issue has been automatically closed after a period of inactivity. If it's a
feature request, it has been added to the maintainers' internal backlog and will be
included in an upcoming round of feature prioritization. Please comment or reopen
if you think this issue was closed in error.

[underyx]

Kinda shocked this issue is fully ignored, make us feel like no serious company is using LLMObs by Datadog. We're on openai's zero data retention enterprise tier and need to ensure data is not logged for several of our customers. For now we'll try this applying this monkeypatching workaround but will likely switch to LangFuse or Logfire for a more robust solution.

original_llmobs_span_event = LLMObs._instance._llmobs_span_event

def obfuscating_llmobs_span_event(span: Span) -> dict[str, Any]:
    # TODO: check span tags to see if customer needs zero data retention; return intact event if not
    event = original_llmobs_span_event(span)
    if event["meta"].get("input", {}).get("messages"):
        event["meta"]["input"]["messages"] = [
            {
                "role": message["role"],
                "content": "<OBFUSCATED>"
                if message["content"] is not None
                else None,
            }
            for message in event["meta"]["input"]["messages"]
        ]
    if event["meta"].get("output", {}).get("messages"):
        event["meta"]["output"]["messages"] = [
            {
                "role": message["role"],
                "content": "<OBFUSCATED>"
                if message["content"] is not None
                else None,
            }
            for message in event["meta"]["output"]["messages"]
        ]
    return event

LLMObs._instance._llmobs_span_event = obfuscating_llmobs_span_event

[underyx]

We landed on this solution:

1. Somewhere in the trace, above the LLM span, set a boolean is_zdr tag with span.set_tag("is_zdr", is_zdr(user_id))

2. Use a util function that determines if an LLM span is affected by ZDR:

   def is_span_zdr(span: Span) -> bool:
       span_to_check_zdr_tag = span
       provisional_is_zdr = True
       while span_to_check_zdr_tag:
           requested_zdr_setting = span_to_check_zdr_tag.get_tag("is_zdr")
           if requested_zdr_setting == "True":
               return True
           elif requested_zdr_setting == "False":
               provisional_is_zdr = False
           span_to_check_zdr_tag = span_to_check_zdr_tag._parent
       return provisional_is_zdr

3. Patch LLMObs to prevent events being if a span is ZDR. We fully prevent the event from being sent by emptying the dict instead of obfuscating content because presumably LLMObs features would be confused by events appearing as if the LLM literally output <OBFUSCATED>.

   original_llmobs_span_event = LLMObs._instance._llmobs_span_event
    
   def obfuscating_llmobs_span_event(span: Span) -> dict[str, Any]:
       return {} if is_span_zdr(span) else original_llmobs_span_event(span)
    
   LLMObs._instance._llmobs_span_event = obfuscating_llmobs_span_event

4. Add a trace filter to APM to obfuscate content if span is ZDR:

   OPENAI_REQUEST_MESSAGE_REGEX = re.compile(r"openai\.request\.messages\.\d+\.content.*")
   OPENAI_RESPONSE_MESSAGE_REGEX = re.compile(
       r"openai\.response\.choices\.\d+\.message\.content.*"
   )
   class ZdrFilter(TraceFilter):
       def process_trace(self, trace: list[Span]) -> list[Span] | None:
           for span in trace:
               if span.name == "openai.request" and is_span_zdr(span):
                   for tag in span.get_tags():
                       tag_str: str = tag.decode() if isinstance(tag, bytes) else str(tag)
                       if re.match(OPENAI_REQUEST_MESSAGE_REGEX, tag_str):
                           span.set_tag(tag_str, "<OBFUSCATED>")
                       elif re.match(OPENAI_RESPONSE_MESSAGE_REGEX, tag_str):
                           span.set_tag(tag_str, "<OBFUSCATED>")
           return trace
   tracer.configure(trace_processors=[ZdrFilter()])