Your GenAI Dashboards Go Blank After an OpenTelemetry Upgrade

A team I was helping bumped their OpenTelemetry instrumentation by one minor version on a Tuesday. It was a routine dependency update, bundled with a dozen others, reviewed in thirty seconds, merged. Nothing in their agent changed. No prompt, no model, no tool. On Wednesday the finance channel asked why the GenAI cost dashboard had been flat at zero since the deploy.

The spend was not zero. The agents were running, the model calls were landing, the invoices at the end of the month would be real. What had gone to zero was the panel. It summed a token usage metric and multiplied by a price, grouped by tenant, and it had been reading correctly for months. The query still ran without error. It just matched nothing now. The metric it summed had been replaced, in the convention version the new library emitted, by a different set of metrics with different names. The dashboard was asking for a thing that no longer existed, and the honest answer to that question is zero.

This is the quiet failure mode of building observability on a spec that is still moving. A renamed attribute does not throw. Your collector accepts the new telemetry, your backend stores it, your query runs green. The only symptom is that the number is wrong, and wrong in the direction that looks like good news: spend dropped, errors vanished, latency improved. You do not get paged for a dashboard that says everything is fine.

The conventions are still moving under you

The GenAI semantic conventions are the shared vocabulary for how a model call, a tool call, or an agent step is described in telemetry: the attribute names like gen_ai.request.model, the token usage metrics, the span names. If you built agent tracing on the OpenTelemetry GenAI Semantic Conventions, you are depending on that vocabulary staying put. As of October 2026 it is not staying put. The conventions moved into their own repository so they could version independently of the core spec, and they are still marked Development. The repository's own Schema URL section, the thing that would let backends translate between versions, currently reads TODO.

Development status is not a formality. Read the breaking-change fragments in that repository and the churn is concrete. One change scopes gen_ai.request.top_k to decoding only and moves retrieval spans onto a new gen_ai.retrieval.top_k. Another removes gen_ai.provider.name from the internal invoke_agent span. Another removes gen_ai.agent.version from the same span and realigns execute_tool attributes across spans and metrics. The one that bit the team above replaces the gen_ai.client.token.usage histogram, and the gen_ai.token.type attribute it relied on, with separate per-direction, per-operation token histograms so distributions aggregate correctly.

Each of those is a reasonable improvement to the spec. Each is also a live grenade for anyone who wrote a query against the old shape. The agent and tool spans are churning more than the old model-call attributes, which matters because the agent layer is exactly where the interesting cost and reliability signal lives. You instrument the thing you most need to watch, and the thing you most need to watch is the part of the spec that is least settled.

Why the dashboard lies instead of breaking

A renamed attribute is not a schema violation. Telemetry is loosely typed by design: a span carries whatever attributes the instrumentation put on it, and a backend stores them without a migration step. So when gen_ai.client.token.usage becomes two new metrics, nothing in the pipeline objects. The old metric simply stops being produced. A query that sums it keeps summing an empty set.

Three classes of query break this way, and all three fail silent. Cost attribution is the first: a panel that converts tokens to dollars, like the one in a span tree that shows where the tokens went, now converts nothing. If you enforce per-tenant budget caps off that same metric, the caps stop firing too, which is worse than a blank panel because now a runaway tenant is uncapped. Second, alerts: a rule that pages when error rate on execute_tool crosses a threshold cannot cross a threshold it can no longer measure, so it sits green through an incident. Third, the SLOs and error budgets you report upward, which quietly start computing against partial data.

The reason this feels unfair is that you did nothing. There was no migration, no deprecation warning in your logs, no failing test. The change arrived through a transitive dependency, the way a tokenizer change you did not make can move a token budget you were counting on. The spec improved, your library tracked it, and the contract between your emitter and your queries broke in the gap.

A rename that blanks a cost panel

Make it concrete. Here is the kind of query a cost panel runs, before and after the token usage histogram was replaced. The query is valid in both worlds. It returns real numbers in the first and an empty series in the second.

# BEFORE: one histogram, token direction carried as an attribute.
# Reads correctly on the old convention version.
sum by (tenant) (
  rate(gen_ai_client_token_usage_sum{gen_ai_token_type="output"}[1h])
) * 15.0 / 1e6   # price per million output tokens

# AFTER the upgrade: gen_ai.client.token.usage no longer exists.
# gen_ai.token.type no longer exists. This query runs and returns nothing.
# The panel shows 0. Spend did not drop; the metric moved to, e.g.,
# gen_ai_server_output_token_usage_* with no token_type label at all.

No error, no alert, no clue in the query itself. The only way to catch this is to compare what the spans and metrics actually emit against what your queries assume, and to do that comparison on every library bump rather than when finance notices. That comparison is cheap to automate and expensive to skip.

Pin the version, keep the names in one place

The first two fixes are boring and they are the ones that pay. Do not float your instrumentation to the latest convention version; pin it, the same way you would gate a model migration behind a reindex and an eval rather than letting it ride in silently. A pin turns a surprise into a decision: when you choose to move the version, you move the dashboards in the same change.

Then stop scattering attribute name strings across your codebase and your dashboard JSON. Put every convention attribute in one module, emit from it, and query from the same constants where your query layer allows it. A rename becomes one edit in one file instead of a hunt through twenty call sites and thirty panels.

# genai_attrs.py  --  the convention vocabulary, in exactly one place.
# Pin the version you built against so a rename is a deliberate bump.
GENAI_SEMCONV_VERSION = "1.41.0"
SCHEMA_URL = f"https://opentelemetry.io/schemas/{GENAI_SEMCONV_VERSION}"

class GenAI:
    REQUEST_MODEL = "gen_ai.request.model"
    RESPONSE_FINISH = "gen_ai.response.finish_reasons"  # was output.messages[].finish_reason
    # Token usage moved from one histogram to per-direction metrics.
    TOKENS_INPUT = "gen_ai.server.input.token.usage"
    TOKENS_OUTPUT = "gen_ai.server.output.token.usage"

# A tracer tagged with the schema URL records which version it emits,
# so a consumer that ever supports translation knows what it is reading.
tracer = provider.get_tracer("agent", schema_url=SCHEMA_URL)

Tagging telemetry with a schema URL is the forward-looking half. OpenTelemetry's telemetry schemas are built for exactly this problem: a producer stamps its data with a version, and a consumer translates between versions using declared transformations. The rename_attributes transformation can map an old attribute name to the new one for spans and metrics, so in principle a backend converts old queries automatically. In practice the GenAI conventions do not publish a schema URL yet, so you cannot lean on automatic translation today. You set the version anyway, because knowing what you emit is the precondition for mapping it later, and for writing the test in the next section.

Test telemetry like the API it is

Your telemetry is an API. Dashboards, alerts, and budget enforcers are its consumers. An API with no contract test breaks silently on every change, which is precisely what happened on that Tuesday. So assert the contract: run your agent against a fixture, capture the spans and metrics it emits, and check that the exact attributes your queries depend on are present and populated.

# test_telemetry_contract.py
# The attributes every dashboard, alert and budget cap keys on.
REQUIRED = {GenAI.REQUEST_MODEL, GenAI.TOKENS_INPUT, GenAI.TOKENS_OUTPUT}

def test_spans_carry_the_attributes_dashboards_query(exporter):
    run_agent_once("summarize the incident report")  # fixture task
    spans = exporter.get_finished_spans()
    llm_spans = [s for s in spans if s.name.startswith("chat ")]
    assert llm_spans, "no model-call spans emitted at all"
    for s in llm_spans:
        missing = REQUIRED - set(s.attributes)
        assert not missing, f"span {s.name} dropped {missing}"
        assert s.attributes[GenAI.TOKENS_OUTPUT] > 0  # populated, not just present

Wire that into the CI gate that catches regressions before they ship. Now the next convention bump that renames TOKENS_OUTPUT fails this test the moment the dependency updates. The failure lands in a pull request, with a diff, where an engineer updates the constant, the mapping, and the affected panels together. That is the whole trick: move the discovery of a telemetry break from production, where it is invisible, into CI, where it is a red check.

The parts that bite

The schema URL being TODO is not a detail you can route around. Until the GenAI conventions ship one, no backend can auto-translate your old queries, so the rename_attributes machinery is a plan, not a tool you have today. Your mapping has to be manual, in your own module, and your contract test is the thing standing in for the translation the spec has not shipped.

Vendor auto-instrumentation is the second trap. If a framework or an agent SDK emits spans for you, you do not control which convention version it tracks, and its upgrade can land inside a release of theirs that looks unrelated. Pinning your own semconv constants does not pin theirs. The contract test still catches it, because it checks what actually came out regardless of who emitted it, which is the argument for testing emitted telemetry rather than trusting the library version string.

Third, dashboards as code drift from dashboards as clicked. If half your panels live in a JSON repo and half were edited by hand in the UI, your one-module discipline only covers the half under version control. The hand-edited panels are the ones that will read zero and stay that way, because nothing points a test at them. And finally, there is lag: telemetry written today is queried for weeks, so a rename can break a weekly cost rollup or a monthly SLO report long after the deploy that caused it, when the connection back to the dependency bump is cold. The contract test closes that gap by failing at bump time, not at report time.

The takeaway

The GenAI semantic conventions are the right thing to build on, and they are also unfinished, and both of those are true at once. The mistake is treating an in-Development spec as a fixed foundation and keying your money and your alerting on attribute names that the spec is openly still changing. The failure that follows is not loud. It is a panel that reads zero, a cap that stops capping, an alert that never fires, and a month of wrong numbers before anyone notices the floor moved.

None of the fixes are clever. Pin the version so a change is a choice. Keep the names in one module so a rename is one edit. Stamp a schema URL so you know what you emit. Test the telemetry like the contract it is, so the break happens in CI instead of in front of finance. If your GenAI dashboards feed cost decisions or alerting and you have never checked them against a library upgrade, that is a thirty-minute audit worth doing before the next routine dependency bump does it for you. If you want a second set of eyes on where your telemetry contract is quietly exposed, let us talk.