Observability allows developers and operations teams to gain deep insights into the inner workings of their systems, diagnose issues efficiently, and maintain seamless operations. OpenTelemetry (OTel) has emerged as a cornerstone in observability by offering powerful instrumentation capabilities that capture traces, metrics, and logs from your applications. However, when implementing OTel, you face a fundamental choice: should you use automatic or manual instrumentation?

Before delving into the details of automatic and manual instrumentation, it is important to grasp the concept of OpenTelemetry instrumentation. OpenTelemetry is an open-source project that establishes a standardized method for gathering observability data (such as traces, metrics, and logs) from your applications, regardless of the technology stack being used. Instrumentation is incorporating code or tools into your application to capture this telemetry data, providing insight into the system’s performance, behavior, and user interactions.

Automatic Instrumentation

Automatic instrumentation simplifies the observability setup by requiring minimal code changes. It involves integrating an OpenTelemetry agent or SDK into your application, which automatically instruments supported libraries and frameworks to capture telemetry data.

How Automatic Instrumentation Works

When you activate automatic instrumentation, the OpenTelemetry agent connects with your application’s runtime environment by integrating with supported libraries and frameworks. For instance, if your application utilizes HTTP clients, database clients, or messaging systems, the agent will automatically collect telemetry data associated with these activities, like incoming and outgoing HTTP requests, database queries, or message publishing and consumption.

Advantages of Automatic Instrumentation

1. Ease of Use and Quick Setup: Automatic instrumentation is perfect for developers who need to quickly add observability to their applications. You simply include the OpenTelemetry agent or SDK in your application, configure it with minimal settings, and the agent takes care of the rest. This simplicity makes automatic instrumentation an attractive choice for those who want to get started with OpenTelemetry without delving into the intricacies of the APIs. For example, if you have a Spring Boot application that processes online orders, by adding the OpenTelemetry Java agent to your application’s startup script, you can automatically capture traces for incoming HTTP requests, database queries related to order processing, and outgoing HTTP requests to external payment gateways — without writing any additional code.
2. Comprehensive Coverage: Automated instrumentation automatically captures telemetry data from a wide variety of popular libraries and frameworks, providing comprehensive coverage without needing extra input from the developer. This ensures that important traces and metrics are captured without additional effort.
3. Consistency is key with the OpenTelemetry agent handling instrumentation, ensuring standardized data collection across various application components. This uniformity facilitates trace and metric correlation, resulting in improved insights and streamlined debugging.
4. Rapid Deployment in Existing Applications: If you need to add observability to an existing or legacy application, automatic instrumentation provides a quick and effective way to do so. It requires minimal code changes and can be integrated into production systems with minimal risk and effort.

Challenges of Automatic Instrumentation

1. Limited Customization: While automatic instrumentation provides broad coverage, it may not offer the level of customization required for more complex or unique use cases. You might find it challenging to tailor the telemetry data to meet specific business needs, such as capturing custom attributes, excluding certain data, or modifying how spans and metrics are created. For example, in the previously mentioned Spring Boot application, you might want to capture additional information such as the size of each order or the user’s loyalty status. With automatic instrumentation, you might not have the flexibility to easily add these custom attributes.
2. Potential Performance Overhead: The OpenTelemetry agent automatically instruments a wide range of components, which can lead to performance overhead, especially in high-traffic or latency-sensitive applications. The extra processing needed to collect and export telemetry data could potentially affect your application’s overall performance.
3. Remember that automatic instrumentation relies on having supported libraries and frameworks available. If your application uses unsupported or custom libraries, you may not have complete observability with automatic instrumentation alone. In such cases, you may need to resort to manual instrumentation for those components.

Manual Instrumentation

To capture the desired telemetry data using manual instrumentation, a specific code is added to the application. This approach offers developers complete control over the data collection process, as well as the reporting of the data, making it well-suited for applications with unique observability needs.

How Manual Instrumentation Works

Developers use manual instrumentation to write custom code for creating spans, metrics, and logs within the application. This code is embedded directly into the application’s logic, providing precise control over when and how telemetry data is captured. For instance, developers can manually instrument an HTTP request handler to capture not only the start and end of a request but also specific attributes such as the request payload size or user authentication status.

Advantages of Manual Instrumentation

1. Granular Control Over Data Collection: Manual instrumentation enables you to specify exactly which telemetry data is collected and when. You have the flexibility to instrument only the most crucial parts of your application, capturing specific attributes, custom spans, and detailed metrics that are tailored to your specific use case. For instance, in the context of building a high-frequency trading platform, you can manually instrument your order matching engine to record latency metrics at each stage of the order processing pipeline. Additionally, you can include custom attributes such as the type of financial instrument being traded or the trading strategy employed, thereby gaining deeper insights into the system’s performance.
2. Detailed Insights and Custom Attributes: Manually instrumenting your application allows you to gather detailed insights into specific parts of your system. This level of detail is especially helpful for monitoring critical components, pinpointing bottlenecks, and troubleshooting complex issues. Additionally, you can supplement spans and metrics with custom attributes, which offer valuable context for analysis.
3. Support for Custom Libraries: If your application uses proprietary or custom libraries not supported by OpenTelemetry’s automatic instrumentation, manual instrumentation is the way to go. You can instrument these libraries yourself, ensuring that all relevant telemetry data is captured and reported.
4. Optimized Performance: By selectively instrumenting only the most critical parts of your application, you can minimize the performance overhead associated with telemetry data collection. This approach ensures that your observability setup is as efficient as possible, focusing resources on capturing the most important data.

Challenges of Manual Instrumentation

1. Increased Development Time and Effort: When using manual instrumentation, developers must write and maintain the instrumentation code, which can be time-consuming, especially for large applications with many components. This method also requires a deep understanding of both the application’s architecture and the OpenTelemetry APIs. For example, in a microservices architecture with numerous services, manually instrumenting each service to capture detailed traces and metrics can be a significant task. It requires careful planning and coordination across teams to ensure consistent capture of all relevant data.
2. Higher Risk of Errors: When using manual instrumentation, there is a higher chance of errors or inconsistencies due to the need to write custom code. It can be difficult to ensure that all parts of the application are properly instrumented, especially in complex systems. Incorrectly configured spans or missing attributes can result in incomplete or misleading telemetry data.
3. Maintenance Requirements: As your application develops, you will need to consistently update and maintain the manual instrumentation code. This maintenance can contribute to the overall workload, particularly in rapidly changing or growing projects. Ensuring that manual instrumentation remains aligned with the application’s architecture and business logic is an ongoing challenge.

Which one is better for me?

The decision to use automatic or manual instrumentation depends on several factors, such as the complexity of your application, your performance requirements, the specific use cases you need to support, and your team’s expertise with observability tools.

When to Choose Automatic Instrumentation

• Need for Quick Setup and Deployment: If you need to quickly add observability to your application with minimal effort, automatic instrumentation is the way to go. It’s perfect for new projects, proof of concepts, or when monitoring a legacy system where introducing changes is challenging. For example, a startup building an MVP for a new SaaS product can use automatic instrumentation to rapidly gain insights into user behavior and system performance without investing significant time in writing custom instrumentation code.
• Standard Applications with Common Libraries: If your application uses popular libraries and frameworks supported by OpenTelemetry’s automatic instrumentation, you can achieve comprehensive coverage with minimal configuration, especially for web applications, microservices, or cloud-native environments.
• Low Complexity and Resource Constraints: For applications with simple observability needs, automatic instrumentation provides sufficient coverage without requiring extensive customization. It enables teams with limited resources or expertise in observability to gain valuable insights without deep knowledge of the underlying systems.

When to Choose Manual Instrumentation

• High Customization Needs: If your application has specific observability requirements that are not met by automatic instrumentation, manual instrumentation is the better choice. This is especially important for applications with custom business logic, proprietary protocols, or non-standard libraries. For example, an e-commerce platform may need manual instrumentation to capture detailed telemetry data related to user shopping behaviors, inventory management, and personalized recommendation algorithms. This data provides valuable insights for both engineering and business teams.
• Granular Control and Optimization: Manual instrumentation provides granular control over data collection. It captures custom attributes, optimizes performance, focuses on critical paths, and tailors observability setups to meet specific requirements.
• Complex and High-Performance Systems: In large-scale or high-performance applications where efficiency and detailed insights are crucial, manual instrumentation is important for ensuring effective and non-intrusive observability. This approach is ideal for mission-critical systems where understanding every millisecond of latency or every detail of a transaction is essential.

Often, the most effective approach involves a combination of automatic and manual instrumentation. Initiate the process with automatic instrumentation to promptly gain visibility into your application, then enhance it with manual instrumentation for the parts of the system that need more detailed insights or custom data collection.

Automatic and manual instrumentation each serves distinct purposes in OpenTelemetry. Automatic instrumentation is user-friendly, quick to deploy, and covers a wide range of scenarios, making it ideal for standard applications or when resources are limited. Manual instrumentation, however, offers precise control and customization, making it better suited for complex or highly specialized systems. Understanding their strengths helps you choose the right approach to balance ease of use with detailed insights for your application’s observability needs.

To start playing around with OpenTelemetry data and Visualize them in Elastic Observability check this detailed article.

Related Article:

OpenTelemetry with Elastic Observability

Elastic RUM (Real User Monitoring) with Open Telemetry (OTel).

Test and Analyze OpenTelemetry Collector processing

Configuration of the Elastic Distribution of OpenTelemetry Collector (EDOT)

Instrumenting a Java application with OpenTelemetry for distributed tracing and integrating with Elastic Observability

#otel #docker #kubernetes #devops #elasticsearch #observability #search #apm

Feel Free to Reach Out at Linkedin: