Your Agent Traces Are Shipping Customer Prompts to Your Observability Vendor

A team I was helping turned on GenAI content capture for a week to debug a tool-calling bug. One agent kept passing a malformed argument to a payments tool, and the metadata on the span did not show what the argument actually was. So they flipped the capture flag, saw the bad argument, and fixed the bug.

Nobody turned the flag back off.

Three months later a security review went looking for where customer data lived. It found that every prompt and every model reply for that service had been flowing into their observability vendor the whole time. Support ticket bodies pasted into prompts, customer names, order numbers, and in a few cases the last four digits of a card that a user had typed into a chat. All of it indexed, searchable from any dashboard login, replicated to whatever region the vendor stored traces in, and subject to the vendor's retention, not theirs.

The trace cost a fraction of a cent to emit. The cleanup, rotating access, working out which records to purge, and explaining the data flow to a reviewer, cost a lot more. None of it was necessary. The convention they were building on had a safe default, and they had quietly overridden it.

What the spec actually says about content capture

The OpenTelemetry GenAI semantic conventions draw a hard line between metadata and content. Metadata is the model name, the token counts, the latency, the operation type, the finish reason. Content is the actual messages: the system instructions, the user's input, the model's output.

Metadata is on by default. Content is not. The GenAI spans specification marks the content attributes, gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions, with the requirement level Opt-In, and states that instrumentations "SHOULD NOT capture them by default, but SHOULD provide an option for users to opt in." The same page carries a plain warning on the output attribute: it "is likely to contain sensitive information including user/PII data."

The OpenTelemetry project says the same thing in prose. Its GenAI observability write-up notes that by default "no prompt content or tool arguments are captured with GenAI telemetry, as these can contain sensitive data." Without the opt-in, a span tells you a call happened, how big it was, and how long it took, and nothing about what was said.

Two things follow from this. First, if prompts are in your traces, somebody enabled it, through an environment variable like OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, a config setting, or a vendor SDK default. Second, the content attributes are still at Development stability, which means their names and shape can change. You do not want a convention that is still moving to be the thing quietly writing customer data into a vendor's index.

Why "just turn it off" is not the whole answer

The reflex after an incident like the one above is to disable capture everywhere and move on. That works until the next hard bug, when an engineer needs to see what the model actually received and turns it back on in a hurry, often broadly, often without a plan to turn it off again. You end up oscillating between blind and leaky.

The real problem is that two decisions are tangled together. One is "do I want to see the content of this run." The other is "do I want the raw content to leave my cluster and land in a third party's storage." Those are not the same decision, and the capture flag treats them as one.

Metadata alone takes you further than people expect. The same traces you added to see why a run cost forty dollars one day and two the next already show the sequence of operations, which tool was called, how many tokens each step burned, and where the latency went. For a large share of failures, that structure is the answer. When a model loops, when a tool times out, when context grows without bound, you see it in the shape of the trace, not in the customer's words.

What you lose without content is the "what exactly did it say" class of bug: a prompt template that renders wrong, a tool argument that is subtly malformed, a reply that is almost right. Those are real, and worth solving. The fix is not all-or-nothing capture. It is to keep structure always, and make raw content a narrow, deliberate exception.

Redact at the collector, not in every service

The highest-leverage place to handle content is the OpenTelemetry Collector, on the path between your services and your exporter. Redacting there means one policy, enforced once, in a place you can audit, rather than trusting every team to scrub their own spans before they export.

The Collector's redaction processor is built for exactly this. It keeps an allowed list of attribute keys and deletes anything not on it, masks attribute values that match blocked regular expressions, and can hash a value instead of dropping it so you keep a stable identifier without the raw data.

# collector config: scrub GenAI content before it reaches the exporter
processors:
  redaction:
    allow_all_keys: true          # keep structural attributes like token counts
    blocked_values:               # mask anything that looks like PII in any value
      - '\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b'   # emails
      - '\b(?:\d[ -]*?){13,16}\b'               # card-like number runs
      - '\b\d{3}-\d{2}-\d{4}\b'                 # national id patterns
    summary: debug                # record how many values were masked, for audit
    # hash instead of mask where you need a stable, non-reversible id:
    hash_function: sha3

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [redaction, batch]
      exporters: [otlphttp]

This catches structured PII inside whatever content does flow through. It is a backstop, not a guarantee: free text will always contain things a regex misses. That is why redaction at the collector is the second line of defence, behind not capturing raw content in the first place.

Wiring opt-in capture so it is deliberate

The first line of defence is in the instrumentation, where you decide whether to attach content to a span at all. Make that decision explicit, bounded, and biased toward structure over raw text.

import hashlib, os

CAPTURE = os.getenv("GENAI_CAPTURE_CONTENT", "off")  # off | hashed | sampled

def set_message_attrs(span, messages, is_input):
    key = "gen_ai.input.messages" if is_input else "gen_ai.output.messages"
    # Always safe: structure without the words.
    span.set_attribute(f"{key}.count", len(messages))
    span.set_attribute(f"{key}.roles", ",".join(m["role"] for m in messages))
    if CAPTURE == "off":
        return
    if CAPTURE == "hashed":
        joined = "\n".join(m["content"] for m in messages)
        span.set_attribute(f"{key}.sha256", hashlib.sha256(joined.encode()).hexdigest())
        return
    if CAPTURE == "sampled" and span.get_span_context().trace_id % 100 < 2:
        # 2% of traces carry raw content, for a short, logged window only.
        span.set_attribute(key, [m["content"] for m in messages])

Three modes, one variable. In normal operation you run with structure only. When you need to find a run again without reading it, hashing lets you match a known input to its trace. When you genuinely need words, sampled capture puts raw content on a small fraction of traces, ideally scoped to one tenant and one time window, with that window recorded somewhere. The raw-content path should export to a restricted store, not the shared dashboard everyone logs into.

The shape of a captured message is worth seeing, because it explains the stakes:

{
  "gen_ai.operation.name": "chat",
  "gen_ai.input.messages": [
    {"role": "system", "content": "You are a support agent for ..."},
    {"role": "user", "content": "My order 4471-A never arrived, card ending 4242, email joe@acme.com"}
  ],
  "gen_ai.output.messages": [
    {"role": "assistant", "content": "I see order 4471-A for joe@acme.com ..."}
  ]
}

Everything sensitive about the customer is in those two fields. The surrounding metadata, the operation name, the token counts, the latency, is what you actually debug with most of the time, and it carries none of it.

Treat captured content as a regulated data flow

Once raw prompts leave your cluster, you have created a second place where personal data lives, with its own retention, access control, and storage region. That is a processing activity, and a reviewer will treat it as one. The data minimisation principle in GDPR Article 5 asks that personal data be "adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed." A trace exists to debug a run. The customer's literal words are rarely necessary for that purpose, which makes default-on capture hard to justify.

The convention itself is careful to stay out of these decisions. It defines where content can go and that it is opt-in; it does not set your retention period, your access policy, your regional storage, or your redaction rules. Those are yours to own, and the same discipline you apply to the egress guardrail that stops one customer's data reaching another applies to your telemetry: an observability vendor is an external recipient like any other.

This also connects to work you may already have in flight. If you are keeping an audit trail that records why an agent acted, that record and your traces are two stores of overlapping sensitive data, and they should share one redaction and retention policy rather than drifting apart. And if you are meeting the labelling duties the EU AI Act placed on AI output, you are already treating model inputs and outputs as regulated artifacts, which is the right frame for your telemetry too.

The parts that bite

Vendor SDK defaults are the first trap. Some LLM and framework instrumentations enable content capture out of the box, or behind a flag that reads like a convenience rather than a data decision. Read what your instrumentation attaches to a span before you trust it, because the OpenTelemetry default of off does not bind a third-party library that chose otherwise.

Regex redaction gives false confidence. It catches patterns, not meaning. A name, a home address written out in prose, a medical detail, these do not match a tidy pattern, and they are exactly the content a reviewer cares about. Redaction at the collector is a backstop for the content that slips through, never the reason it is safe to capture everything.

Logs are the forgotten channel. Teams scrub their spans and leave the same prompts in application logs at debug level, or in exception messages that serialize the whole request. The trace is one export path; your log pipeline is another, and it needs the same policy.

Sampling can still be too much. Two percent of a high-traffic service is a lot of real prompts if the window is open for a month. Scope sampled capture to a tenant, keep the window short, record when it is open, and have it default back to off. This is the same instinct as replaying a failed run deterministically: capture the minimum that lets you reproduce the problem, not everything in case it helps.

Finally, Development-stage attributes move. The content attribute names can change before they stabilise, so pin the instrumentation versions you depend on and watch the changelog, rather than discovering at an awkward moment that your redaction rules point at keys that no longer exist.

The takeaway

The safe behaviour is already the default. The OpenTelemetry GenAI convention keeps message content opt-in because prompts and replies routinely carry PII, and the moment you override that default you have decided, usually without meaning to, that customer data should live in a third party's searchable store. Keep structure always, make raw content a narrow and logged exception, and redact at the collector so one policy guards every export path. Metadata debugs most of what you will ever debug.

Separating "I want to see this run" from "I want the raw words to leave my cluster" is the whole move. Get those two decisions apart and the forgotten-flag incident stops being possible. If your traces capture prompts today and nobody on the team can say who can read them or for how long, that gap is worth closing before an audit finds it for you, and it is usually a short conversation to scope. If that is where you are, let's talk.