I’ve written before about instrumenting an AI agent with OpenTelemetry’s GenAI conventions and about what the MCP semantic conventions define. Those two posts covered manual instrumentation and the spec. What I had not done yet was install the auto-instrumentation package most people reach for first, run it against a real client and server, and look at what it produces.

So I did. I installed the most-downloaded MCP auto-instrumentation package on PyPI, pointed it at an MCP server and an MCP client running as two separate processes talking over stdio, and read the spans that came out. Some of it matches the spec. Some of it does not, and the gap is the useful part.

Everything below is either code I ran, or a claim I checked against the OpenTelemetry semantic conventions repository and Elastic’s documentation. Versions are pinned to exactly what I used, because in this area the version decides whether things work or crash.

Short version

• opentelemetry-instrumentation-mcp 0.62.3 crashes on import with mcp 2.x. Pin mcp<2.
• It emits traceloop.* attributes, not the OpenTelemetry MCP fields (mcp.method.name, gen_ai.tool.name and so on). Dashboards built on the spec field names will be empty.
• Trace context does propagate from client to server over stdio. Both processes end up in one trace.
• The Collector’s elasticsearch exporter (in the mode I used) writes to traces-generic-default. The data is in Discover, but the APM Service Inventory and Traces views stay empty.

The story, for anyone who is not a tech person

Imagine hiring an inspector to write a report every time a worker handles a customer request. The industry has an official report form with standard boxes. Your inspector, a popular auto-instrumentation package, fills in a different form of his own that predates the official one, so anyone searching the filing cabinet for the official box names finds nothing.

He also has a handbook problem. It is out of date for the newest version of the toolkit, so with version 2 of the MCP SDK he fails on his first day, and you need to pin the older version.

What he does well is the hard part. The client and the server run as two separate programs connected only by a pipe, yet his reports from both carry the same tracking number, so one journey shows up across both.

The reports are then filed in Elasticsearch. They are all there when you search in Discover, but the APM office does not look in that drawer, so the APM screens stay empty. The data is fine. It is simply filed where the APM app does not read.

That is the lesson of the article: test what a tool actually produces before building dashboards on the name of the package. A name or prefix with mcp in it does not guarantee that the official standard is being followed.

What it means in practice

• Pin the mcp package and the instrumentation library together, and re-test the pair when either changes.
• If you need the official field names, add them yourself on a manual client span around the tool call.
• Check what the instrumentation captures, because full tool inputs and outputs are recorded by default.

What changed in the spec recently

Some context first, because this area moves fast. GenAI observability in OpenTelemetry covers a few things: model call spans (gen_ai.operation.name, token usage, model name), agent and tool spans (invoke_agent, execute_tool), MCP-specific fields that link an agent’s tool call to the MCP server that handled it, optional prompt and completion content capture, two client-side histogram metrics, and provider-specific extensions for OpenAI, Anthropic and Bedrock.

The recent change most write-ups miss: on 12 June 2026, with core semantic-conventions v1.42.0, the GenAI and MCP conventions moved out of open-telemetry/semantic-conventions into a dedicated open-telemetry/semantic-conventions-genai repository, so they can change faster than the core spec’s stability rules allow. At the time of writing that repository has no tagged release, and both gen-ai-spans.md and gen-ai-metrics.md are marked Status: Development. If you build dashboards on gen_ai.* or mcp.* names, take that status seriously. Names have already changed once (gen_ai.system became gen_ai.provider.name in v1.37) and can change again.

The test setup

I wanted the simplest realistic setup: an MCP server exposing one tool, a client that calls it, and both running as separate processes connected only by stdio, which is how most local MCP servers run. No mocked transport.

python3.11 -m venv venv
source venv/bin/activate
pip install mcp opentelemetry-instrumentation-mcp \
            opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

The first problem showed up straight away. The official mcp Python SDK made a breaking rename in its 2.x line: FastMCP became MCPServer, imported from mcp.server.mcpserver instead of mcp.server.fastmcp. The import most examples use, including my earlier posts:

from mcp.server.fastmcp import FastMCP  # works on mcp<2, removed in mcp>=2

raises ModuleNotFoundError on mcp 2.2.0. To its credit, the error message tells you the new import. Check your installed version before copying code from anywhere, including this post:

pip show mcp opentelemetry-instrumentation-mcp | grep -E '^(Name|Version)'

The auto-instrumentation package does not support the current SDK

With mcp 2.2.0 installed, calling McpInstrumentor().instrument() crashes at startup:

wrapt.exceptions.PathResolutionError: unable to resolve attribute
'streamablehttp_client' in path 'streamablehttp_client' on
<module 'mcp.client.streamable_http'>

opentelemetry-instrumentation-mcp (the Traceloop/OpenLLMetry package, 0.62.3 at the time of writing) patches a function called streamablehttp_client. The 2.x SDK renamed it to streamable_http_client as part of the same cleanup that removed FastMCP, and the instrumentation has not caught up. You hit this the moment you pip install both packages without pins. Note that it fails even if you only use stdio, because the patch targets are resolved for every transport when you call instrument().

The combination that works:

pip install "mcp<2" opentelemetry-instrumentation-mcp==0.62.3

With mcp 1.30.0 and that instrumentation version, everything imports and runs. Pin both, and re-test the pair before you upgrade either one. A fix on either side could change this after publication.

The server and the client

Both processes need their own TracerProvider and their own McpInstrumentor().instrument() call, set up before anything is imported from mcp. The client injects trace context into each request and the server-side instrumentation extracts it. If only the client is instrumented you get a client-side trace and nothing from the server.

One stdio-specific trap on the server: stdout is the protocol channel. Anything you print to stdout, including the default ConsoleSpanExporter output, gets mixed into the JSON-RPC stream and breaks the connection. Send console output to stderr, or use an OTLP exporter.

Server (weather_server.py):

import sys
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
from opentelemetry.sdk.resources import Resource

provider = TracerProvider(resource=Resource.create({"service.name": "weather-mcp-server"}))
# stdout carries MCP JSON-RPC, so console spans must go to stderr
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter(out=sys.stderr)))
trace.set_tracer_provider(provider)

from opentelemetry.instrumentation.mcp import McpInstrumentor
McpInstrumentor().instrument()

from mcp.server.fastmcp import FastMCP

tracer = trace.get_tracer("weather.server")
mcp_server = FastMCP("weather-demo")

@mcp_server.tool()
def get_weather(city: str) -> str:
    """Return a fake but deterministic weather report for a city."""
    with tracer.start_as_current_span("weather.lookup") as span:
        span.set_attribute("weather.city", city)
        print(f"SERVER trace_id={span.get_span_context().trace_id:032x}", file=sys.stderr)
        return f"{city}: 24C, light rain, wind 12km/h"

if __name__ == "__main__":
    mcp_server.run(transport="stdio")

Client (weather_client.py):

import asyncio
import sys
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
from opentelemetry.sdk.resources import Resource

provider = TracerProvider(resource=Resource.create({"service.name": "weather-agent-demo"}))
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)

from opentelemetry.instrumentation.mcp import McpInstrumentor
McpInstrumentor().instrument()

from mcp import ClientSession
from mcp.client.stdio import stdio_client, StdioServerParameters

tracer = trace.get_tracer("agent.demo")

async def main():
    # sys.executable makes the server run in the same venv as the client
    server_params = StdioServerParameters(command=sys.executable, args=["weather_server.py"])
    with tracer.start_as_current_span("agent.turn") as turn_span:
        turn_span.set_attribute("gen_ai.agent.name", "weather-agent")
        print(f"CLIENT trace_id={turn_span.get_span_context().trace_id:032x}")
        async with stdio_client(server_params) as (read, write):
            async with ClientSession(read, write) as session:
                await session.initialize()
                result = await session.call_tool("get_weather", {"city": "Bengaluru"})
                print("TOOL RESULT:", result.content)

asyncio.run(main())
provider.shutdown()

agent.turn is my own span. It stands in for the point in a real agent loop where the model decides to call a tool. Every other client span in the trace comes from the instrumentation library.

What showed up: not the fields you would expect

Here is the tool-call span from the console exporter, trimmed to the relevant fields:

{
  "name": "get_weather.tool",
  "kind": "SpanKind.INTERNAL",
  "attributes": {
    "traceloop.span.kind": "tool",
    "traceloop.entity.name": "get_weather",
    "traceloop.entity.input": "{\"tool_name\": \"get_weather\", \"arguments\": {\"city\": \"Bengaluru\"}}",
    "traceloop.entity.output": "{\"result\": \"Bengaluru: 24C, light rain, wind 12km/h\"}"
  }
}

Look at what is missing. No mcp.method.name, no mcp.protocol.version, no gen_ai.tool.name, no gen_ai.operation.name, no jsonrpc.request.id. All of these are defined in the OpenTelemetry MCP semantic conventions, and none of them appear. The span kind is also INTERNAL, while the spec describes the client side of an MCP request as a CLIENT span. What you get instead is traceloop.span.kind, traceloop.entity.name, traceloop.entity.input and traceloop.entity.output. That is Traceloop’s own convention from OpenLLMetry, which predates the official spec and is separate from it.

This has a practical cost. If you install opentelemetry-instrumentation-mcp expecting spec fields because of the package name, and build a Kibana dashboard that filters on mcp.method.name, the dashboard will stay empty and nothing will tell you why. The data is there. It is under different names.

Also note that traceloop.entity.input and traceloop.entity.output contain full tool arguments and results by default. The spec treats gen_ai.tool.call.arguments and gen_ai.tool.call.result as opt-in because they can carry sensitive data. If your tools handle user data, check the package’s content-capture setting (TRACELOOP_TRACE_CONTENT=false turns it off in OpenLLMetry) before you ship this to a shared backend.

Defined in the OTel MCP spec	Emitted by opentelemetry-instrumentation-mcp 0.62.3
mcp.method.name	not present
mcp.protocol.version	not present
mcp.session.id	not present (and not applicable to stdio anyway, see below)
gen_ai.operation.name	not present
gen_ai.tool.name	not present (traceloop.entity.name instead)
gen_ai.tool.call.arguments	not present (traceloop.entity.input instead)
gen_ai.tool.call.result	not present (traceloop.entity.output instead)

The part that worked better than I expected

Connecting a trace across the client and server processes is the hard problem in MCP observability, so I tested it directly: both processes instrumented, a manual span inside the tool function, and the trace IDs compared.

CLIENT trace_id=3da21492226d47d5fc019087a596de8e
SERVER trace_id=3da21492226d47d5fc019087a596de8e

Same trace ID in two separate processes. Rebuilding the parent-child chain from both processes’ spans gives this waterfall:

agent.turn                          (client, root span)
├── initialize.mcp                  (client)
├── tools/list.mcp                  (client, issued by the SDK, not my code)
└── get_weather.tool                (client)
    └── ResponseStreamWriter        (client, sends the request)
        ├── ResponseStreamWriter    (server, handles it)
        └── weather.lookup          (server, my manual span)

That is W3C trace context crossing a stdin/stdout pipe, which has no built-in place for tracing headers. The mechanism is the MCP proposal for this (SEP-414): the client puts traceparent, tracestate and baggage into the _meta object of the JSON-RPC request, and the server reads them from there before starting its own spans. I did not inspect the raw JSON-RPC messages to confirm the exact keys this library writes, but the result is what that mechanism should produce: two processes, one trace ID, correct parent-child order.

So there are two findings pointing in opposite directions. The hard problem, cross-process context propagation, works today. The problem that sounds easier, using the field names the spec defines, is not handled by the package most people install first.

Adding the spec fields yourself

If your dashboards or team conventions expect the spec fields, add them with a manual CLIENT span around the call. This runs alongside the auto-instrumentation rather than replacing it. Two details worth getting right:

• Protocol version. Read it from the initialize result instead of hard-coding it. In my run the client and server negotiated 2025-11-25, which a hard-coded value would have got wrong.
• Session ID. mcp.session.id is meant to carry the real MCP session ID, which only exists on the Streamable HTTP transport (the Mcp-Session-Id header). Over stdio there is none, so leave it unset rather than inventing a UUID that no other component will recognise.

async def call_tool_with_semconv(session, protocol_version, tool_name, arguments):
    with tracer.start_as_current_span(
        f"tools/call {tool_name}", kind=trace.SpanKind.CLIENT
    ) as span:
        span.set_attribute("mcp.method.name", "tools/call")
        span.set_attribute("mcp.protocol.version", protocol_version)
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", tool_name)
        span.set_attribute("network.transport", "pipe")  # stdio maps to pipe
        result = await session.call_tool(tool_name, arguments)
        if result.isError:
            # a tool error is a normal JSON-RPC response, so nothing raises
            span.set_attribute("error.type", "tool_error")
            span.set_status(trace.Status(trace.StatusCode.ERROR))
        return result

# inside main():
init = await session.initialize()
result = await call_tool_with_semconv(
    session, init.protocolVersion, "get_weather", {"city": "Bengaluru"}
)

The isError check matters. When an MCP tool fails, the server returns a normal result with isError: true instead of raising, so without it every failed tool call shows up as a successful span.

For Streamable HTTP, the client gives you the session ID as the third value from the transport context manager (async with streamablehttp_client(url) as (read, write, get_session_id) on mcp<2). Set mcp.session.id from get_session_id() after initialize, and set network.transport to tcp.

With this wrapper the trace gets a new tools/call get_weather span under agent.turn, and the auto-instrumented get_weather.tool span becomes its child. The spec fields sit on the wrapper span and the Traceloop content fields sit on the child, so you keep both.

Sending it to Elasticsearch and Kibana

For this part I ran a local stack: Elasticsearch 9.5.2, Kibana 9.5.2 and the OpenTelemetry Collector, all in Docker (via colima, since I don’t have Docker Desktop installed). I ran the same demo against it and queried Elasticsearch directly to see what landed.

First, both scripts need to export over OTLP instead of to the console. Replace the console exporter in the client and the server with:

from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
)

With BatchSpanProcessor, call provider.shutdown() before the process exits, otherwise the last batch is dropped. The client script above already does this. The server is a child process that gets terminated when the client closes the stdio connection, so it may not flush in time. Either use SimpleSpanProcessor on the server for a demo like this, or register provider.shutdown with atexit.

# docker-compose.yml (trimmed to the essentials)
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.5.2
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=changeme
      - xpack.security.http.ssl.enabled=false
    ports: ["9200:9200"]

  kibana:
    image: docker.elastic.co/kibana/kibana:9.5.2
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=changeme
    ports: ["5601:5601"]

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.120.0
    command: ["--config=/etc/otelcol-config.yml"]
    volumes: ["./otelcol-config.yml:/etc/otelcol-config.yml"]
    ports: ["4317:4317", "4318:4318"]

# otelcol-config.yml
receivers:
  otlp:
    protocols:
      grpc: {endpoint: "0.0.0.0:4317"}
      http: {endpoint: "0.0.0.0:4318"}
processors:
  batch:
exporters:
  elasticsearch:
    endpoints: ["http://elasticsearch:9200"]
    user: elastic
    password: changeme
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [elasticsearch]

This is a local lab config. Security is on but TLS is off and the passwords are in plain text. Do not reuse it outside your laptop.

Two setup steps the official images don’t do for you. First, ELASTIC_PASSWORD only sets the elastic superuser, not kibana_system, so Kibana cannot connect until you set that password separately:

curl -u elastic:changeme -X POST "http://localhost:9200/_security/user/kibana_system/_password" \
  -H "Content-Type: application/json" -d '{"password":"changeme"}'

Second, if Elasticsearch, Kibana and the Collector run as separate docker compose projects (different directories, which happened to me while reorganising this lab), they end up on separate Docker networks and the Collector cannot resolve elasticsearch. The fix is docker network connect, but the failure is quiet: the Collector logs nothing until it tries to flush a batch, and then you see dial tcp: lookup elasticsearch: no such host.

After that, run the demo and check Elasticsearch:

$ curl -u elastic:changeme "http://localhost:9200/traces-generic-default/_count"
{"count":12,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}

Twelve span documents from the client and server combined. One of them:

{
  "_index": "traces-generic-default",
  "_source": {
    "@timestamp": "2026-09-29T03:19:47.571Z",
    "Name": "ResponseStreamWriter",
    "TraceId": "ac93c7ecbc20ba2a8c1442555ebcf6a4",
    "SpanId": "9faa1328d0a14a7a",
    "ParentSpanId": "d201b3bdf0c1abbf",
    "Resource.service.name": "weather-mcp-server",
    "Attributes.mcp.request.id": "0",
    "Attributes.mcp.response.value": "{\"protocolVersion\": \"2025-11-25\", ...}"
  }
}

Note mcp.request.id and mcp.response.value here. They look like spec fields because of the mcp. prefix, but they are not in the OpenTelemetry MCP conventions. They come from the instrumentation library. A prefix alone does not tell you a field is standard.

In Kibana, a data view on traces-generic-default shows all 12 documents in Discover, and this is where both conventions sit side by side: Attributes.traceloop.entity.input, Attributes.traceloop.entity.output and Attributes.traceloop.span.kind from the auto-instrumentation, next to Attributes.mcp.method.name, Attributes.gen_ai.tool.name and Attributes.gen_ai.agent.name from my manual spans. All of them are searchable with ordinary KQL, for example Attributes.gen_ai.tool.name : "get_weather", or TraceId : "ac93c7ecbc20ba2a8c1442555ebcf6a4" to pull every span of one trace from both processes.

Why the APM views stay empty

What does not show up, which I checked directly: the APM Service Inventory (/app/apm/services) and the Traces view (/app/apm/traces) are both empty for this data, even though the spans are in Elasticsearch and visible in Discover.

The reason is where and how the documents are stored. Collector 0.120.0’s elasticsearch exporter defaults to the none mapping mode, which writes flat Attributes.* documents into traces-generic-default. The Kibana APM app reads from its own configured index patterns (check APM > Settings > Indices), and traces-generic-default doesn’t match them. The data is fine. The APM app just doesn’t look there.

There are two ways to get the APM views:

1. Send the spans through Elastic’s APM ingest path. That means an APM Server or Elastic’s managed OTLP endpoint (Elastic Cloud), or a local Elastic Agent running the APM integration. The data then lands in the data streams the APM app reads by default.
2. Use the exporter’s otel mapping mode. This writes OTel-native documents to traces-generic.otel-default, and newer Collector releases use it as the default. Recent Kibana versions can read OTel-native trace data in the APM views, but confirm that your APM index settings include the .otel- pattern. I have not tested this path in this lab.

Because the mapping mode default changed between Collector versions, the image tag is part of the result. Run the same config on a newer Collector and your documents will land in a different index with different field names (attributes.* instead of Attributes.*). Pin the image, or set the mapping mode explicitly.

For option 1 with an Elastic Cloud endpoint, the Collector exporter looks like this:

exporters:
  otlp/elastic:
    endpoint: "YOUR_APM_OR_OTLP_ENDPOINT:443"
    headers:
      # use "Bearer YOUR_SECRET_TOKEN" for an APM Server secret token,
      # or "ApiKey YOUR_API_KEY" for an API key
      Authorization: "ApiKey YOUR_API_KEY"
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/elastic]

I have not re-run this demo through a live APM endpoint. That needs Elastic Cloud credentials or a local Fleet and Elastic Agent setup, which I covered in an earlier post. One thing I can say from Elastic’s documentation: the GenAI tab in the span flyout reads gen_ai.input.messages, gen_ai.output.messages and gen_ai.system_instructions. Those fields belong to a model-call span. A tool-call span like get_weather.tool doesn’t have them, so it will not fill that tab whichever exporter you use. To see conversation content there, instrument the LLM call itself, not just the MCP tool call.

Things that will trip you up

• Pin mcp and the instrumentation library together. Re-test the pair after either one changes version. The rename that broke this package is the kind of change that can happen again while the MCP SDK is still settling.
• Instrument before importing mcp. Set up the TracerProvider and call .instrument() first, in both processes. The library patches functions with wrapt, and code that imported the original function earlier keeps the unpatched version.
• Never write to stdout in a stdio MCP server. That includes debug prints and the default console span exporter. Use stderr.
• Don’t assume a package with “mcp” in its name emits the OpenTelemetry MCP conventions. Read the span attributes it produces once before building anything on them.
• Check what the instrumentation captures. Full tool inputs and outputs are recorded by default here, which may include user data.
• Pin the Collector image too. The Elasticsearch exporter’s default mapping mode decides which index and field names you get.
• Treat every gen_ai.* and mcp.* name as provisional. The conventions are still in Development status with no tagged release. That includes the names I set by hand in this post.

Frequently Asked Questions

I also ran the same demo against Grafana and Tempo instead of Elasticsearch, to check that the cross-process propagation result holds on a different backend. It does. The full write-up and config are in OpenTelemetry MCP Tracing to Grafana: Real Cross-Process Traces.

You can also read this article on Medium.

Connect with me on LinkedIn.

Related Articles:

Instrumenting an AI Agent with OpenTelemetry GenAI Conventions

OpenTelemetry Now Traces MCP Tool Calls

OpenTelemetry to Elastic End to End: What Actually Breaks