A Hands-On Guide To Every OpenTelemetry-to-Elastic Ingestion Path: Five Ways To Send OpenTelemetry Data To Elastic

I have been running the OpenTelemetry demo app against Elastic in five different configurations at the same time. Not one at a time for comparison. All five, simultaneously, fanning out from a single OTel collector.

Here are the actual doc counts from that session:

Flow 1a (ECH APM server): 18 million traces
Flow 1b (local APM server): 40 million traces
Flow 2+3b (Serverless managed OTLP): 5.9 million traces
Flow 3a (ECH managed OTLP): 17 million traces
Flow 4 (EDOT gateway): 21,680 traces
Flow 5 (Elastic Agent OTLP, tech preview): 5.4 million traces

Each path has different config, different limitations, and different things that break without telling you. This article walks through all five, shows the full working collector config, and covers browser RUM as the sixth path that always gets left off the architecture diagram.

Why are there so many paths?

Elastic APM existed before OpenTelemetry did. It had its own wire protocol, its own intake API, and its own ingest pipeline that translated incoming telemetry into ECS (Elastic Common Schema) fields before writing to Elasticsearch.

When OpenTelemetry became the standard, Elastic had to support two worlds in parallel: the existing APM pipeline that existing customers were already using, and a newer OTel-native approach where data lands in Elasticsearch in OTel format without any translation.

That split created two distinct destination patterns:

The APM path: data goes through the APM server, gets converted to ECS, lands in traces-apm.*. You see the service in APM Service Inventory in Kibana.
The OTel-native path: data goes through a managed OTLP endpoint, stays in OTel format, lands in traces-generic.otel-default. You see it in Discover and the newer OTel-aware views.

Both work. They just look different in Kibana, have different config, and have different things that go wrong.

Add the gateway pattern and the Elastic Agent OTLP receiver and you get five total paths. Here is what each one actually does.

The architecture

The diagram below shows all five flows plus the browser RUM path:

Left side: the OTel collector fans out to all five destinations simultaneously
Right side: the browser sends RUM events through a local CORS proxy to the APM servers
Each flow is colour-coded by type: orange for OTel-to-ECS, green for OTel-native, purple for gateway, red for Agent EDOT

One collector config file (otelcol-config-extras.yml) drives all five exporters at once. Each exporter in the service pipelines section receives the same traces, metrics, and logs and sends them independently to its destination.

The full collector config

This is the actual config I use. Replace the placeholder values with your own endpoints and credentials.

			
exporters:
  # Flow 1a: ECH APM server
  # OTel-to-ECS conversion. Data lands in traces-apm-default on ECH.
  # APM Service Inventory shows this service in Kibana.
  otlp/ech_apm:
    endpoint: "YOUR_CLUSTER_ID.apm.us-central1.gcp.cloud.es.io:443"
    compression: none
    headers:
      Authorization: "Bearer YOUR_APM_SECRET_TOKEN"
  # Flow 1b: Local APM server (elastic-agent managed)
  # Same OTel-to-ECS conversion, local ES and Kibana.
  # TLS insecure because local APM uses self-signed certs.
  otlphttp/local_apm:
    endpoint: "http://host.docker.internal:8200"
    tls:
      insecure: true
  # Flow 2+3b: Serverless managed OTLP
  # No APM pipeline. Data stays in OTel format.
  # Must use otlphttp. Serverless .ingest. endpoint does not support gRPC.
  otlphttp/serverless:
    endpoint: "https://YOUR-PROJECT.ingest.eastus.azure.elastic.cloud:443"
    headers:
      Authorization: "ApiKey YOUR_API_KEY=="
  # Flow 3a: ECH managed OTLP input
  # Same OTel-native result as serverless, but on a hosted ECH deployment.
  # ECH managed OTLP supports gRPC. Use otlp, not otlphttp.
  otlp/ech_managed_otlp:
    endpoint: "YOUR-DEPLOYMENT.ingest.us-central1.gcp.elastic-cloud.com:443"
    headers:
      Authorization: "ApiKey YOUR_API_KEY=="
  # Flow 4: EDOT collector gateway
  # Your collector sends here. The gateway holds the ES credentials and writes directly.
  # insecure: true is fine for container-to-container on a private Docker network.
  otlp/edot_gateway:
    endpoint: "edot-gateway:4317"
    tls:
      insecure: true
  # Flow 5: Elastic Agent embedded otelcol (tech preview)
  # elastic-agent ships an embedded otelcol on port 4320.
  # Connect the agent container to your demo network so it is reachable by hostname.
  otlp/agent_edot:
    endpoint: "elastic-agent:4320"
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters:
        - spanmetrics
        - otlp/ech_apm
        - otlphttp/local_apm
        - otlphttp/serverless
        - otlp/ech_managed_otlp
        - otlp/edot_gateway
        - otlp/agent_edot
    metrics:
      receivers: [otlp, spanmetrics]
      processors: [batch]
      exporters:
        - otlp/ech_apm
        - otlphttp/local_apm
        - otlphttp/serverless
        - otlp/ech_managed_otlp
        - otlp/edot_gateway
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters:
        - otlp/ech_apm
        - otlphttp/local_apm
        - otlphttp/serverless
        - otlp/ech_managed_otlp
        - otlp/edot_gateway

		

Three things to understand before reading on:

otlp uses gRPC. otlphttp uses HTTP. They are not interchangeable. Getting this wrong is the most common config mistake.
Auth format differs by destination: Bearer token for APM servers, ApiKey for managed OTLP endpoints.
compression: none on Flow 1a is intentional. Some ECH APM endpoints reject compressed gRPC payloads depending on version. If you hit unexplained connection errors, toggle this first.

Flows 1a and 1b: the classic APM path

This is the path most people start with. Your OTel collector sends to an APM server, which converts the incoming data from OTel format into ECS before writing to Elasticsearch.

What that conversion gives you:

OTel span attributes mapped to APM fields (span.name, transaction.type, service.name)
Data in traces-apm-default, the data stream the APM UI reads from
Your service visible in APM Service Inventory, with transaction waterfall, latency percentiles, and error grouping

If you want the full Kibana APM UI, this is the only path that gets you there. The OTel-native paths (Flows 2-5) skip the APM pipeline, so APM Service Inventory does not show your service by default.

Flow 1a (ECH APM server): get your APM endpoint from the deployment page in Elastic Cloud. It looks like YOUR_CLUSTER_ID.apm.us-central1.gcp.cloud.es.io. The secret token is on the same page. Use the otlp exporter (gRPC).

Flow 1b (local APM server): local elastic-agent manages the APM server at http://host.docker.internal:8200 when calling from inside Docker. No auth by default. Use otlphttp because the local APM server sometimes handles HTTP more reliably than gRPC in a local Docker setup.

What breaks: the endpoint must not include a path. The OTel exporter appends /intake/v2/events itself. If you include it, you get a double path and a 404. For local APM, also check that OTLP input is enabled in the Fleet APM integration policy. It is disabled by default.

Flow 2+3b: Serverless managed OTLP

Serverless Elastic does not have a separate APM server. There is one unified ingest endpoint at .ingest. that handles OTLP directly.

The result is different from Flows 1a/1b:

No APM pipeline conversion
Data lands in traces-generic.otel-default in OTel semantic conventions
Visible in Discover and the OTel-aware observability views, not in APM Service Inventory
Field names are OTel fields: resource.attributes.service.name, span.name, not the ECS equivalents

The .ingest. endpoint on Serverless only accepts HTTP, not gRPC. Use otlphttp, not otlp. Using the wrong exporter type gives a connection error with no explanation.

The endpoint format looks like: YOUR-PROJECT.ingest.REGION.elastic.cloud

The auth is an API key in the format ApiKey YOUR_KEY==. Not a Bearer token.

What breaks: mixing up Bearer token (APM) with ApiKey (OTLP endpoints). A wrong auth header gives a 401 with an empty body, which is not helpful for debugging. Also check that the endpoint URL does not have a trailing path segment.

Flow 3a: ECH managed OTLP input

This is the same OTel-native path as serverless, but on a hosted ECH deployment. ECH added a managed OTLP input endpoint so you can skip the APM pipeline on hosted too.

Same .ingest. endpoint pattern, same traces-generic.otel-default destination, same OTel fields in the output.

The one difference from serverless: ECH managed OTLP supports gRPC. Use the otlp exporter, not otlphttp.

			
otlp/ech_managed_otlp:
  endpoint: "YOUR-DEPLOYMENT.ingest.us-central1.gcp.elastic-cloud.com:443"
  headers:
    Authorization: "ApiKey YOUR_API_KEY=="

No tls: block needed. The connection uses your system’s CA certificates for *.elastic-cloud.com.

What breaks: using otlphttp instead of otlp here. Unlike serverless, ECH managed OTLP uses gRPC. The otlphttp exporter sends HTTP/1.1 or HTTP/2 protobuf, which hits a protocol mismatch and returns a 415 or just drops the connection.

Flow 4: the EDOT gateway pattern

This one is different architecturally. Instead of sending directly to an Elastic endpoint, your OTel collector sends to a second collector that you run yourself. That second collector (the gateway) holds the Elasticsearch credentials and writes to ES.

Why do this?

Your application collectors do not need to know about Elasticsearch or hold any credentials
You can add processing in the gateway (filtering, sampling, enrichment) without touching the application collector config
You can swap the backend destination without redeploying application collectors

The setup: run a second EDOT collector container on the same Docker network as your OTel demo. Your demo collector sends to it via otlp/edot_gateway. The gateway config has the ES credentials and the elasticsearch exporter.

The gateway config looks like this:

			
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
processors:
  batch:
    timeout: 5s
exporters:
  elasticsearch/ech:
    endpoints:
      - https://YOUR-DEPLOYMENT.es.us-central1.gcp.cloud.es.io
    api_key: YOUR_API_KEY==
    mapping:
      mode: otel
    flush:
      interval: 1s
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [elasticsearch/ech]

		

Add mapping: mode: otel to the elasticsearch exporter. Without it, the exporter defaults to writing to a plain index instead of a data stream, and every write fails with a require_data_stream error. This failure is silent at basic log verbosity.

What breaks: the gateway container must be on the same Docker network as your OTel demo collector. If you start the gateway after the demo is already running: docker network connect opentelemetry-demo edot-gateway. Also the elasticsearch exporter is not the same as the otlp exporter. It writes directly to ES using the data stream API, not through an APM server.

Flow 5: Elastic Agent as an OTLP receiver (tech preview)

Elastic Agent ships with an embedded OTel collector binary (otelcol) that listens on port 4320. You can point an OTel exporter directly at the elastic-agent container and let it handle the write to ES.

This is a tech preview. It works, but it is not production-supported at the time of writing.

			
otlp/agent_edot:
  endpoint: "elastic-agent:4320"
  tls:
    insecure: true

You need to connect the elastic-agent container to your OTel demo’s Docker network so it is reachable by hostname.

What breaks: the embedded otelcol on port 4320 only accepts gRPC. Use otlp, not otlphttp. If the agent container is not on the demo network, the connection times out with no helpful error message. Also check the elastic-agent version: older versions do not ship the embedded otelcol at all.

The path everyone leaves off the diagram: browser RUM

All five flows above are server-side. They cover traces, metrics, and logs from your backend services. None of them capture what happens in the browser.

Browser telemetry is different. It covers:

Page load timing and Core Web Vitals (LCP, FID, CLS)
JavaScript errors with stack traces
Which browser, OS, and device the user is on
Where the user is located

This data comes from RUM (Real User Monitoring) via the APM intake API at /intake/v2/events. The browser sends NDJSON directly to the APM server, not through an OTel collector.

I built a standalone tool that generates synthetic RUM events and sends them to one or more APM endpoints at once. It runs with a single Python command, no Node or npm needed. Download it and follow the README at github.com/rahulranjan22/rum-demo.

The RUM data lands in traces-apm.rum-default and shows up in Observability → User Experience in Kibana: page load distributions, Core Web Vitals, and the Visitor Breakdown charts showing browser, OS, device, and location.

One non-obvious thing: location data in the User Experience dashboard comes from context.request.socket.remote_address in the NDJSON body, not from X-Forwarded-For. ECH’s load balancer strips that header. If you set the IP in the header only, the location map is blank.

The failure modes no one documents

These are the issues I hit most in real testing:

gRPC vs HTTP exporter mismatch. Using otlp (gRPC) against a serverless ingest endpoint that only accepts HTTP. Or using otlphttp against ECH managed OTLP that expects gRPC. Both fail with unhelpful errors.
Path included in the endpoint URL. The OTLP exporter appends /v1/traces, /v1/metrics, /v1/logs itself. If your endpoint URL already has a path, you get a double path and a 404.
Wrong auth format. Bearer token for APM servers (Flows 1a/1b). ApiKey for OTLP ingest endpoints (Flows 2/3a/3b). Mixing these gives a 401 with an empty response body.
Gateway require_data_stream error. The elasticsearch exporter in the gateway tries to write to a data stream. If the target index is a plain index (not a data stream), every write fails silently at basic log verbosity. Fix: add mapping: mode: otel to the exporter config.
Gateway not on the right Docker network. The gateway container needs to be on the opentelemetry-demo network, not just running. Check with docker inspect edot-gateway and look at the Networks section.
Flow 5 connection timeout with no error. The elastic-agent container is not on the demo network. Connect it: docker network connect opentelemetry-demo elastic-agent.
RUM CORS failure. Browser RUM calls fail with CORS preflight errors if RUM is not enabled in the Fleet APM integration policy and allow_origins is not set. The local CORS proxy in the RUM demo tool bypasses this entirely.

Which path should you use?

You want the full Kibana APM UI: Flow 1a (ECH) or 1b (local). APM Service Inventory, transaction waterfall, latency percentiles, error grouping. This is the only path that gives you all of that.
You are on Serverless or want OTel-native fields in Elasticsearch: Flow 2+3b (Serverless) or 3a (ECH managed OTLP). Data stays in OTel format, no APM translation. You need to use the OTel-aware Kibana views.
You want to decouple credentials from your application collectors: Flow 4 (gateway pattern). Application collectors stay simple, the gateway handles auth and the ES write. Useful if you manage multiple teams’ collectors centrally.
You are exploring the future of Elastic Agent: Flow 5. Expect rough edges and check the version before you start.
You need browser telemetry in any of the above: add the RUM tool at github.com/rahulranjan22/rum-demo pointing at the same APM endpoint.

Closing

Running all five paths at the same time is the best way to understand what each one actually does. The config differences are small but the behavior differences are large: where data lands, which Kibana views show it, and what breaks silently.

The collector config above has all five exporters with placeholder values. Swap in your endpoints and credentials and you will have all five running at once. The doc counts at the start of this article are what a week of that setup produces.

For the browser side of the picture, the tool at github.com/rahulranjan22/rum-demo covers everything.

Leave a ReplyCancel reply