Alexis Kestler
Hey there! If you‘re like me, you want to understand what‘s happening under the hood of your complex applications and microservices. As systems get more distributed, it gets harder to track down issues and performance problems. We need better visibility – something experts are calling "observability". This is where OpenTelemetry comes in!
In this guide, I‘ll walk you through what OpenTelemetry is all about. I‘ll share my insights as a data analytics geek on how OpenTelemetry provides invaluable telemetry data and paves the way for next-generation observability. Let‘s get started!
Why Modern Apps Need Observability
Today‘s applications are complex beasts. Monolithic apps are being broken down into microservices and serverless functions. Systems are distributed across multiple servers, platforms, and cloud providers. We release updates continuously and change things on the fly. Debugging and monitoring these dynamic, distributed architectures requires a whole new approach.
Observability provides that approach. It‘s the ability to understand what‘s happening inside your systems based on their outputs and signals. Just logging messages and monitoring some metrics doesn‘t cut it anymore.
To properly observe modern apps, we need:
- Traces – Track requests as they flow across services, hosts, and environments.
- Metrics – Quantitative data about system performance, traffic, and errors.
- Logs – Logged events and statements with contextual data.
Analyzing these telemetry data points together provides true observability. It enables us to answer critical questions like:
- How is this user request traversing my distributed system?
- Which services or calls are causing latency bottlenecks?
- Are error rates spiking in a particular service?
- What events correlate to a rise in error logs?
This kind of insight is invaluable today. But collecting, correlating, and digesting all that telemetry data is no easy task. Every vendor seems to have their own custom solution, with proprietary instrumentation and Analytics. What a headache!
This fragmentation is where OpenTelemetry comes to the rescue.
OpenTelemetry: A Standard for Instrumentation and Data Collection
OpenTelemetry provides a single set of open source APIs, libraries, and services to standardize the collection and processing of telemetry data from apps and systems.
The project began in 2019 when the OpenTracing and OpenCensus projects merged. OpenTracing focused on distributed tracing, while OpenCensus provided a standard for metrics. Combined as OpenTelemetry, the goal is to create a truly open and universal framework.
OpenTelemetry is governed by the Cloud Native Computing Foundation. It‘s backed by major industry players including Microsoft, Google, Dynatrace and Splunk. AWS announced support in late 2020.
The key principles behind OpenTelemetry include:
- No vendor lock-in – Open source and open standards avoid proprietary solutions.
- Consistency – Standardized APIs and data formats work across languages and platforms.
- Interoperability – Integrates with popular monitoring backends and dashboards.
- Flexibility – Customizable with extendable SDKs, exporters, and backends.
- Compatibility – Supports legacy systems and allows gradual migration.
By providing a standard way to instrument apps and collect data, OpenTelemetry solves the current fragmentation in the observability space. It lets developers focus on creating telemetry without worrying about underlying vendor differences.
The project also provides flexibility on exporting and analyzing the telemetry. As an open standard, OpenTelemetry itself is not tied to any specific analytics platform or dashboard.
Next, let‘s explore how OpenTelemetry works and what it provides under the hood.
An Understandable Overview of OpenTelemetry
The OpenTelemetry project consists of a few key components:
APIs and SDKs
OpenTelemetry defines tracing, metrics, and logging APIs that allow instrumenting applications to generate telemetry data:
- Tracing API – Used to create, annotate, and record trace spans that represent operations.
- Metrics API – Provides instruments for generating metrics data like counters and gauges.
- Logging API – Allows adding correlation IDs and other context to log statements.
- Context Propagation – Propagates trace IDs and correlation data across processes.
- Semantic Conventions – Naming standards for consistency and interoperability.
In addition, OpenTelemetry provides SDKs with default implementations of these APIs in 10+ languages including Go, Java, JS, Python, .NET, PHP, Ruby, etc.
For example, the Python SDK allows you to easily generate traces and export them:
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("foo"):
print("Hello world!")
# Trace data is automatically exportedCollector
The OpenTelemetry Collector provides a vendor-neutral pipeline for receiving, processing, and exporting telemetry data. The Collector is like a router – it receives traces, metrics, and logs from instrumented apps and sends them to backends.
Having this intermediate collection tier solves some huge problems:
- Apps don‘t need to write directly to backends. The Collector handles it.
- Apps can send data to a single endpoint instead of multiple backends.
- The Collector batches, aggregates, and processes data before exporting.
The Collector can ingest data from OpenTelemetry SDKs as well as other legacy formats like Zipkin and OpenCensus traces. It supports exporting data to all popular tracing, metrics, and logging backends like Jaeger, Prometheus, ELK stack, etc.
Instrumentation Libraries
In addition to manual instrumentation, many third party libraries and frameworks come pre-instrumented out-of-box with OpenTelemetry. Some examples:
- Web frameworks – Django, Flask, Express, Spring Boot
- Data stores – Redis, MongoDB, MySql, PostgreSQL
- AWS SDK, gRPC, JDBC
- Kafka, Nginx, Envoy, Elasticsearch
These auto-instrumentations allow enabling telemetry from libraries and apps without changing any code. They use the OpenTelemetry APIs under the hood to automatically capture trace data, metrics, etc. Pretty neat!
Detailed Architecture of OpenTelemetry
Now that we‘ve seen the main components, let‘s explore how they fit together in the OpenTelemetry architecture:
Image Source: New Relic
There are three main steps:
1. Instrumentation
Applications use the OpenTelemetry APIs and SDKs to add instrumentation code that generates telemetry – traces, metrics, and logs.
For example, a Python web app would use the OpenTelemetry Python SDK to track incoming requests. This instrumentation automatically captures trace data for each request without any proprietary code.
2. Collection
The OpenTelemetry Collector receives telemetry data from instrumented apps over gRPC or HTTP protocols.
It acts as an intermediary buffer and processor. The Collector batches, aggregates, and applies transformations to the raw telemetry before exporting it.
3. Analysis Backend
Finally, the Collector exports preprocessed telemetry to various backends for visualization, alerting, and analysis.
Backends include tracing solutions like Jaeger, metrics databases like Prometheus, and log aggregators like Elasticsearch.
By standardizing the instrumentation and collection layers, OpenTelemetry provides flexibility on exporting to different analysis backends. There‘s no vendor lock-in!
The key benefits of this architecture include:
- Apps don‘t need to write directly to multiple backends
- Consistent instrumentation across languages and platforms
- Flexibility to export data anywhere for analysis
- Collector handles processing load instead of apps
- Existing apps can send telemetry without code changes
Next let‘s look at some cool advantages of using OpenTelemetry.
The Superpowers OpenTelemetry Provides
Here are some of the standout benefits of using OpenTelemetry:
- Portability – Instrument your apps once using OpenTelemetry APIs, run anywhere. No vendor lock-in.
- Interoperability – Integrates with all major tracing, metrics, and logging backends. Unified data.
- Flexibility – Mix and match SDKs, Collector, and exporters. Do what‘s best for your stack.
- Consistency – Standard semantic conventions and data formats across platforms.
- Automation – Many pre-instrumented libraries and frameworks for automatic telemetry.
- Efficiency – The Collector handles data processing instead of applications.
- Compatibility – Supports gradual migration from legacy systems and formats.
- Rich telemetry – Unified traces, metrics, and logs provide comprehensive observability into systems.
OpenTelemetry essentially provides a vendor-neutral but flexible and seamlessly interoperable standard for handling telemetry data from instrumentation to collection to analysis.
This elegant end-to-end solution creates a foundation for taming the complexity of observability.
Using OpenTelemetry Step-by-Step
Enough background, let‘s get our hands dirty! Here is how to start instrumenting your own apps with OpenTelemetry:
1. Install the OpenTelemetry SDK
First, install the OpenTelemetry SDK module for your app‘s language:
# For Python
pip install opentelemetry-sdk
# For NodeJS
npm install @opentelemetry/sdk
# For Java
gradle add opentelemetry-sdk2. Initialize a Tracer
Next, initialize a Tracer from the SDK to create spans:
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)3. Instrument Code to Generate Telemetry
Now instrument your application code to track operations. Use the Tracer to start and end spans to capture timing data:
with tracer.start_as_current_span("operation"):
# do some work
print("Hello world!")
# Span tracing data is automatically generated! You can also add attributes, events, and logs to spans for richer tracing data.
4. Configure an Exporter to Send Data
Finally, configure the SDK to export telemetry to the backend of your choice. For example:
# Export to Jaeger
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.jaeger import JaegerSpanExporter
jaeger_exporter = JaegerSpanExporter(
service_name="my-service"
)
trace.get_tracer_provider().add_span_processor(
BatchSpanProcessor(jaeger_exporter)
)That‘s the basics of how to instrument an app and export telemetry with OpenTelemetry!
The Collector can also be used to receive and process data before sending it to a backend.
OpenTelemetry is Quickly Gaining Massive Adoption
Since the initial release in 2019, OpenTelemetry adoption has skyrocketed. Tons of major tech players are onboard:
- CNCF backing with contributions from Splunk, Microsoft, Dynatrace, New Relic, and Google.
- AWS announced OpenTelemetry support in late 2020.
- 80+ releases and 160+ contributors on GitHub. Over 5500 stars.
- Instrumentation libraries for 10+ languages.
- Auto-instrumentation provided by popular libraries and frameworks.
OpenTelemetry enables observability for all kinds of systems:
- Languages – Python, Java, JS, Go, .NET, PHP
- Platforms – Kubernetes, AWS Lambda, Azure
- App Frameworks – Django, Rails, Flask, Express
- Data Stores – Redis, MongoDB, Cassandra
- API Gateways – Nginx, Envoy, HAProxy
All this adds up to a ton of momentum. With its flexible design and industry support, OpenTelemetry is fast becoming the standard for instrumentation and telemetry data collection.
Gaining Observability With OpenTelemetry: Next Steps
I hope this guide gives you a solid understanding of OpenTelemetry and how it enables observability! Here are some parting thoughts on next steps with OpenTelemetry:
- Play around with instrumenting test apps to see OpenTelemetry in action
- Consider a trial run on non-critical apps and services
- Check if your languages and frameworks are supported out-of-the-box
- Evaluate options like hosted OpenTelemetry Collector or SaaS analytics integrations
- Plan how to feed OpenTelemetry data into your existing metrics and tracing backends
The documentation and community resources out there are fantastic when you‘re ready to dive deeper. Please drop me a note if you have any other questions!
Observability is crucial for modern applications, and OpenTelemetry paves the way. This open standard solves real problems around fragmentation and telemetry collection. OpenTelemetry has huge momentum. I‘m betting we‘ll see it continue to take over as the default for instrumentation and observability.
Happy tracing!
