In the companion piece to this article I ran a real MCP client and server as two separate processes, instrumented both with OpenTelemetry, and traced exactly what a popular auto-instrumentation package does and does not capture, then sent the result into a local Elasticsearch and Kibana stack and checked what actually landed. This article is the same trace, same demo, same real spans, pointed at Grafana instead, because the two backends read OTLP differently enough that “it works in Elastic” tells you nothing about whether it works in Grafana.

I am not going to re-explain the MCP semantic conventions or the auto-instrumentation gotchas here, that piece already covers it. This one is entirely about getting the data out and seeing it rendered, with a config you can copy, change two values in, and run.

The story, for anyone who is not a tech person

Think of two couriers working in separate rooms, passing a parcel through a slot in the wall. The customer wants one tracking page that shows both couriers’ work as a single journey, not two unrelated entries.

In this article the two rooms are an MCP client and an MCP server, two separate programs that talk only through a pipe. The tracking number is the trace ID. When the client sends a request, it writes the tracking number on the note attached to it, so the server can pick up the same number and continue the same journey instead of starting a new one.

Tempo is the filing system that stores the journey, and Grafana is the screen that draws it. The test checks the whole chain: both programs send their steps to Tempo, and Grafana shows one timeline in which the server’s step sits correctly inside the client’s step.

The same result was already shown on Elasticsearch in the companion article. Repeating it on a different backend shows that the tracking works at the program level and does not depend on where the data finally lands.

Two small traps are worth knowing. Tempo needs about fifteen seconds after it starts before it accepts data, and another program already using port 3000 can quietly catch Grafana’s traffic. The Grafana Cloud setup is described here but was not run end to end.

What it means in practice

• Wait for Tempo to report ready before sending the first trace.
• Check what is already listening on port 3000 before blaming the Grafana container.
• For Grafana Cloud, copy the endpoint and token from your own stack’s OpenTelemetry connection tile.

What I actually stood up

Grafana’s trace backend is Tempo. Grafana itself is just the query and visualisation layer on top of it, whether that is a Tempo instance you run yourself or the one behind Grafana Cloud’s OTLP gateway. So the real test here is: does OpenTelemetry data sent straight to Tempo, no collector in between, actually render correctly in Grafana’s trace view for a trace that spans two separate operating system processes.

# docker-compose.yml
services:
  tempo:
    image: grafana/tempo:2.9.0
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
      - "3200:3200"   # Tempo query API

  grafana:
    image: grafana/grafana:11.6.0
    depends_on: [tempo]
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml
    ports:
      - "3000:3000"

# tempo.yaml
server:
  http_listen_port: 3200
distributor:
  receivers:
    otlp:
      protocols:
        grpc: {endpoint: 0.0.0.0:4317}
        http: {endpoint: 0.0.0.0:4318}
storage:
  trace:
    backend: local
    local: {path: /tmp/tempo/traces}
    wal: {path: /tmp/tempo/wal}
compactor:
  compaction: {block_retention: 24h}

# grafana-datasources.yml
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    isDefault: true

One real gotcha from actually running this, not a hypothetical: if you already have something else bound to port 3000 on your machine (I did, a leftover Next.js dev server from unrelated work), Grafana’s container will start fine and Docker will report the port mapping as active, but requests to localhost:3000 can still get routed to the other process instead, silently, with no error from either side. Check what’s listening before you assume the container isn’t working:

lsof -nP -iTCP:3000 -sTCP:LISTEN

If something’s already there, remap Grafana’s port in the compose file rather than fighting it.

Pointing the same demo at Tempo

Same client and server code as the Elasticsearch article, the only change is the exporter endpoint on both sides:

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True))
)

Running it and querying Tempo’s own search API directly, before even opening Grafana, to confirm ingestion happened:

$ curl -s "http://localhost:3200/api/search?limit=20"
{"traces":[{"traceID":"8be5ed6faecd0645b3f9b3873b83f29a","rootServiceName":"weather-agent-demo","rootTraceName":"agent.turn","startTimeUnixNano":"1790652357070222000","durationMs":477}]}

Real trace, real duration, ingested and searchable. Pulling the full trace back and printing every span’s service, name, and parent confirms the same cross-process structure I verified in the Elasticsearch piece, this time through Tempo’s own storage and API rather than a console exporter:

weather-agent-demo | agent.turn            | parent=None
weather-agent-demo |   tools/list.mcp      | parent=agent.turn
weather-agent-demo |   get_weather.tool    | parent=agent.turn
weather-mcp-server |     weather.lookup    | parent=get_weather.tool's stream span
weather-agent-demo |   initialize.mcp      | parent=agent.turn

Two separate Python processes, connected only by stdin and stdout, landing as one coherent trace in Tempo.

What it actually looks like in Grafana

Opening Grafana’s Explore view, selecting the Tempo data source, and pasting the trace ID in as a TraceQL query renders the full waterfall: weather-agent-demo: agent.turn as the root at 477.55ms, with initialize.mcp (376.99ms, dominated by MCP’s handshake), get_weather.tool (2.06ms), and the server-side weather-mcp-server: weather.lookup span nested directly underneath it at 11µs, exactly where it belongs in the timeline. No separate, disconnected trace for the server side. One waterfall, both processes, correct nesting, rendered by Grafana reading nothing but what Tempo stored from the OTLP data.

This is the same distributed-tracing-across-a-process-boundary result from the Elasticsearch article, confirmed a second time against a completely different backend. That is a meaningfully stronger claim than testing it once: the propagation mechanism (W3C trace context injected into the MCP request’s _meta field, per the MCP working group’s SEP-414 proposal) is working at the SDK and transport level, independent of which backend eventually stores the data.

Pointing this at Grafana Cloud instead of a local Tempo

I tested the mechanism above against a local, self-hosted Tempo because that is what I could actually verify without an account. Grafana Cloud’s OTLP ingestion is the same OTLP protocol hitting a hosted gateway with authentication in front of it, not a different pipeline, so the collector-side change is small. Grafana’s own current documentation gives the endpoint shape as https://otlp-gateway-<REGION>.grafana.net/otlp and authenticates with HTTP Basic auth, your numeric Grafana Cloud instance ID as the username, an API token as the password, base64-encoded together into the Authorization header.

Here is the collector config with exactly two things to replace, everything else works as-is:

exporters:
  otlp/grafanacloud:
    endpoint: "YOUR_GRAFANA_CLOUD_OTLP_ENDPOINT"   # e.g. otlp-gateway-prod-us-east-0.grafana.net/otlp:443
    headers:
      Authorization: "Basic YOUR_BASE64_INSTANCE_ID_AND_API_TOKEN"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/grafanacloud]

To generate that Authorization value, base64-encode your instance ID and API token together with a colon between them:

echo -n "YOUR_INSTANCE_ID:YOUR_API_TOKEN" | base64

Both values come from your Grafana Cloud stack’s own OpenTelemetry connection tile (Grafana Cloud Portal → your stack → Connections → OpenTelemetry), which generates the exact endpoint and token for your specific region and stack rather than a generic one, since Grafana’s own docs are explicit that the host can differ by when your stack was created. Do not hardcode the example endpoint above as if it were universal.

I have not personally pushed this exact demo’s spans all the way through a live Grafana Cloud account, that needs an account and a token I did not create for this session. What I have verified directly: the OTLP wire protocol, the span shapes, and the cross-process propagation are identical regardless of destination, since Tempo is the same trace store Grafana Cloud runs behind its gateway. The only new surface with Grafana Cloud specifically is the authentication header, which is a config change, not a different pipeline to test from scratch.

Things that will trip you up here specifically

Tempo’s ingester is not immediately ready after container start. It logs Ingester not ready: waiting for 15s after being ready even once the process is up; give it a genuine 15 to 20 seconds before sending your first batch, or your export will fail with connection errors that have nothing to do with your config.

Tempo’s /api/search endpoint uses TraceQL, not a simple tag filter string. ?tags=service.name=x silently returns an empty result set rather than an error on some setups; if you get nothing back, retry with no filter at all (/api/search?limit=20) before assuming ingestion failed.

If you are running both an Elasticsearch/Kibana stack and this Grafana/Tempo stack on the same machine for comparison, as I did, put them on separate Docker networks deliberately and map different host ports for anything that collides (Elasticsearch’s collector already claims 4317/4318 in the companion article’s setup; I ran Tempo’s OTLP receiver on 4319/4320 on the host to keep both stacks up side by side).

Frequently Asked Questions

You can also read this article on Medium.

Connect with me on LinkedIn.

Related Articles:

OpenTelemetry MCP Tracing to Elasticsearch: What Actually Works (Tested End to End)

Instrumenting an AI Agent with OpenTelemetry GenAI Conventions

OpenTelemetry Now Traces MCP Tool Calls