We’ll explore a Java application architecture that integrates OpenTelemetry for monitoring and observability, using Docker for deployment.

Project Overview

This project centers on a Java application instrumented with OpenTelemetry to capture and export telemetry data (metrics and traces). The Java application simulates multiple API requests and integrates with Prometheus for monitoring, Grafana for visualizing metrics, and Jaeger for distributed tracing.

Architecture

The architecture is built around the following components:

1. Java Application: A simple HTTP server that responds to API calls and simulates various operations while generating traces.
2. OpenTelemetry Collector: Collects telemetry data (metrics and traces) from the Java application and exports it to various backends.
3. Prometheus: A monitoring system that scrapes metrics from the OpenTelemetry Collector.
4. Grafana: A dashboard for visualizing the metrics collected by Prometheus.
5. Jaeger: A distributed tracing system that receives trace data from the OpenTelemetry Collector.

Project Detail

1. Dockerfile

The Dockerfile is divided into two stages: building the Java application using Maven and running the packaged JAR file in a slim OpenJDK image.

# Stage 1: Build the application
FROM maven:3.8.5-openjdk-11 AS build
WORKDIR /app

# Copy the source code and resources into the container
COPY TestHttpServer.java src/main/java/
COPY log4j2.xml src/main/resources/
COPY pom.xml .
RUN apt-get update && apt-get install -y iputils-ping

# Build the application
RUN mvn clean package -DskipTests

# Stage 2: Run the Java application
FROM openjdk:11-jre-slim
WORKDIR /usr/src/myapp

# Copy the OpenTelemetry Java agent and the packaged JAR file from the build stage
COPY opentelemetry-javaagent.jar opentelemetry-javaagent.jar
COPY --from=build /app/target/rranjan-java-app-1.0-SNAPSHOT.jar rranjan-java-app.jar

CMD ["java", "-javaagent:opentelemetry-javaagent.jar", "-jar", "rranjan-java-app.jar"]

2. OpenTelemetry Configuration (otel-config.yml)

The OpenTelemetry Collector serves as a pipeline for receiving, processing, and exporting telemetry data (both metrics and traces). It acts as the intermediary between the Java application and external observability tools like Prometheus and Jaeger.

• Receivers: It collects data through the OTLP receiver for metrics and traces, supporting both gRPC and HTTP protocols.
• Exporters:
• Prometheus exporter sends the data to Prometheus for metrics monitoring.
• OTLP exporter sends trace data to Jaeger for distributed tracing.

Data Flow:

1. The Java application sends OTLP-formatted telemetry data (metrics and traces) to the OpenTelemetry Collector.
2. The Collector processes the data and sends metrics to Prometheus and traces to Jaeger.

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8887"
  otlp:
    endpoint: "jaeger:4317"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]
    traces:
      receivers: [otlp]
      exporters: [otlp]

3. Docker Compose Configuration

The docker-compose.yml file orchestrates all services, ensuring they start and connect correctly.

version: '3.8'

services:
  java-app:
    build: .
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
      - OTEL_SERVICE_NAME=java-app
    depends_on:
      - otel-collector

  otel-collector:
    image: otel/opentelemetry-collector:latest
    command: [ "--config=/etc/otel-config.yaml" ]
    volumes:
      - ./otel-config.yaml:/etc/otel-config.yaml
    ports:
      - "4317:4317"
      - "8887:8887"

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    depends_on:
      - prometheus

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "14250:14250"
      - "16686:16686"

Java Application Implementation (TestHttpServer.java)

The Java application, a simple HTTP server, simulates multiple API endpoints like GET /api/users, POST /api/orders, and PUT /api/products. It also integrates with OpenTelemetry through the io.opentelemetry.api.trace library to trace and monitor each API call. When an API call occurs, a span is created, and the trace data (including metadata like errors) is sent to OpenTelemetry for processing and export.

The app uses the OpenTelemetry Java agent (opentelemetry-javaagent.jar), which is configured to automatically collect and send telemetry data to the OpenTelemetry Collector.

import java.io.IOException;
import com.sun.net.httpserver.HttpServer;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpExchange;
import java.net.InetSocketAddress;
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.context.Scope;

public class TestHttpServer {
    private static final Tracer tracer = GlobalOpenTelemetry.getTracer("TestHttpServer");

    public static void main(String[] args) throws IOException {
        HttpServer server = HttpServer.create(new InetSocketAddress(8001), 0);
        server.createContext("/api/simulate", new ApiHandler());
        server.setExecutor(null);
        server.start();
        System.out.println("Server is listening on port 8001");

        Span mainSpan = tracer.spanBuilder("Main Transaction").startSpan();
        try (Scope scope = mainSpan.makeCurrent()) {
            while (true) {
                simulateApiCall("GET /api/users");
                simulateApiCall("POST /api/orders");
                simulateApiCall("PUT /api/products/123");
                simulateApiCall("DELETE /api/items/456");
                simulateApiCall("GET /api/error");
                Thread.sleep(5000);
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt(); // Restore interrupted status
            System.out.println("Main thread interrupted: " + e.getMessage());
        } finally {
            mainSpan.end();
        }
    }

    private static void simulateApiCall(String apiEndpoint) {
        Span span = tracer.spanBuilder(apiEndpoint).startSpan();
        try (Scope scope = span.makeCurrent()) {
            span.setAttribute("api.endpoint", apiEndpoint);
            if (apiEndpoint.contains("error")) {
                throw new CustomException("Simulated error for " + apiEndpoint, 1001);
            }
            simulateWork();
        } catch (CustomException e) {
            span.recordException(e);
            span.setStatus(StatusCode.ERROR, "Error occurred: " + e.getMessage());
        } finally {
            span.end();
        }
    }

    private static void simulateWork() {
        try {
            Thread.sleep(1000);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt(); // Restore the interrupted status
            System.out.println("Work thread interrupted: " + e.getMessage());
        }
    }

    static class CustomException extends RuntimeException {
        private final int errorCode;

        public CustomException(String message, int errorCode) {
            super(message);
            this.errorCode = errorCode;
        }

        public int getErrorCode() {
            return errorCode;
        }
    }

    static class ApiHandler implements HttpHandler {
        @Override
        public void handle(HttpExchange exchange) throws IOException {
            String query = exchange.getRequestURI().getQuery();
            String response = "{\"message\": \"Simulated API call successful\"}";
            if (query != null && query.contains("endpoint=")) {
                String apiEndpoint = query.split("endpoint=")[1];
                simulateApiCall(apiEndpoint);
            }
            exchange.sendResponseHeaders(200, response.getBytes().length);
            exchange.getResponseBody().write(response.getBytes());
            exchange.close();
        }
    }
}

Prometheus Configuration (prometheus.yml)

Prometheus scrapes the OpenTelemetry Collector’s exposed metrics endpoint at 0.0.0.0:8887. It stores time-series data and allows the metrics to be queried for analysis.

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: "otel-collector"
    static_configs:
      - targets: ["otel-collector:8887"]

Grafana

Grafana is integrated with Prometheus to visualize the metrics in real-time. Grafana queries Prometheus and presents dashboards showing the health and performance of the Java application, including CPU usage, request rates, error rates, etc. The Grafana UI is exposed on port 3000.

Jaeger

Jaeger is responsible for distributed tracing. It receives traces from the OpenTelemetry Collector, showing the path of each request through the system. This helps in identifying bottlenecks, errors, and latency issues in the application’s microservices.

Jaeger’s UI is exposed on port 16686, allowing users to view and search for traces.

Docker Integration and Data Flow

The project is containerized using Docker with a multi-stage Dockerfile to build and run the Java application and a Docker Compose file to orchestrate the setup, ensuring seamless integration between all components.

Docker Compose Flow:

Java Application (Container java-app):

• Built from the provided Dockerfile, it runs the Java application, which exports telemetry data.
• Exports telemetry to the OpenTelemetry Collector at http://otel-collector:4317.

OpenTelemetry Collector (Container otel-collector):

• Receives telemetry from the Java app, processes it, and exports:
• Metrics to Prometheus.
• Traces to Jaeger.

Prometheus (Container prometheus):

• Scrapes the Collector’s Prometheus exporter and stores metrics data.

Grafana (Container grafana):

• Visualizes metrics from Prometheus in a customizable dashboard.

Jaeger (Container jaeger):

• Receives traces from the Collector and provides trace visualization for distributed transactions.

Detailed Data Flow

1. API Call Simulation:

The Java application simulates various API endpoints.

• Each API call is traced and metrics (e.g., response times, errors) are captured.

2. Telemetry Collection:

• The OpenTelemetry Java agent sends the collected telemetry data to the OpenTelemetry Collector via OTLP.

3. Data Processing:

• The Collector processes incoming telemetry and splits it:
• Metrics are sent to Prometheus.
• Traces are sent to Jaeger.

4. Visualization and Monitoring:

• Prometheus stores metrics, which can be visualized through Grafana.
• Jaeger stores traces, which can be analyzed through its UI to identify bottlenecks and errors in the application.

OpenTelemetry with ElasticSearch & Kibana

In addition to Prometheus, Grafana, and Jaeger, a powerful alternative for observability is to use Elasticsearch and Kibana. With this setup:

OpenTelemetry exporters can be configured to send both metrics and traces directly to Elasticsearch.

exporters:
  otlp/elastic:
    # !!! Elastic APM https endpoint WITHOUT the "https://" prefix
    endpoint: "11111111111.apm.xyz.xyz.cloud.es.io:443"
    compression: none
    headers:
      Authorization: "Bearer aaaaaaaaaaaaaaa"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [spanmetrics, otlp/elastic]
    metrics:
      receivers: [otlp, spanmetrics]
      processors: [batch]
      exporters: [otlp/elastic]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/elastic]

In Kibana, you can build dynamic dashboards, search traces, and explore logs, providing a unified platform for real-time observability and analysis with advanced filtering, querying, and visualization capabilities.

This approach offers centralized monitoring, combining the flexibility of Elasticsearch with Kibana’s rich visualization tools, ideal for users familiar with the Elastic stack. It integrates all data types — logs, metrics, and traces — under a single UI, simplifying troubleshooting and enhancing observability.

Conclusion

This project demonstrates the power of OpenTelemetry for collecting and exporting telemetry data from a Java application. It leverages Docker to manage and orchestrate services like the OpenTelemetry Collector, Prometheus, Grafana, and Jaeger, ensuring seamless integration and scalability. The architecture captures both traces and metrics, providing valuable insights into the performance and behavior of the application, making it a comprehensive solution for observability in modern microservices-based environments.

The project code is available on my GitHub.

OpenTelemetry with Elastic Observability

Elastic RUM (Real User Monitoring) with Open Telemetry (OTel)

OpenTelemetry: Automatic vs. Manual Instrumentation — Which One Should You Use?

Test and Analyze OpenTelemetry Collector processing

Instrumenting a Java application with OpenTelemetry for distributed tracing and integrating with Elastic Observability

Reach out at Linkedin for any questions.