Using Open Telemetry with Datadog

Open Telemetry (OTEL) has been steadily gaining popularity in the observability space. OTEL is a collection of protocols, tools, and SDKs from the Cloud Native Computing Foundation (CNCF). In addition to being open-source and vendor neutral, the included libraries also support a wide range of languages and frameworks.

Datadog can be introduced seamlessly into an OTEL environment. Besides the use of vector, there are two primary ways to send OTEL data to the Datadog backend, either through the Datadog exporter or through the OTEL mode in the Datadog agent. 

Datadog agent OTEL mode

Out of the box, the Datadog agent does not automatically collect from OTEL-formatted metrics, traces, and logs. However, for agents above 6.48.0 and 7.48.0 (6.32.0 and 7.32.0 for no log support), Datadog can be switched to “OTLP mode” via the settings under the OpenTelemetry section in the datadog.yaml. With the following configuration, the Datadog agent will start listening for grpc traffic on 4317 and http traffic on port 4318. By default, the endpoint will support metrics and traces, but logs will have to be enabled manually like below:

otlp_config:
  receiver:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  logs:
    enabled: true

After enabling the above settings, running a Datadog status check will indicate that OTLP is enabled.

After enabling “OTLP mode”, metrics and traces can be sent with the use of the Open Telemetry SDK. The following example uses the automatic tracing instrumentation for a python script, which constructs a simple span using the Open Telemetry SDK. Auto instrumentation supports a lot of common frameworks and is a great way to get started:



OTEL_SERVICE_NAME=dd-oltp \ <- this will be the service name in Datadog
OTEL_TRACES_EXPORTER=console,otlp \
OTEL_METRICS_EXPORTER=console,otlp \
OTEL_LOGS_EXPORTER=console,otlp \
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=0.0.0.0:4317 \ <- by default the trace will be in GRPC format, so we will be using the GRPC endpoint
OTEL_EXPORTER_OTLP_INSECURE=true \  <- for when certs are not configured
opentelemetry-instrument /usr/local/bin/python3.11 trace_sender.py

The trace will then come into Datadog with additional OTEL-enriched metadata:

If installing the Datadog tracers is not a concern, Datadog tracing libraries can also switch to OTEL mode by running with the environment setting `DD_TRACE_OTEL_ENABLED=true`. An example trace sent via this method can be seen below:

Using Datadog exporter

For teams that are running the OTEL collector and want to avoid deploying Datadog agents, Datadog exporter is available for the collector. Datadog exporter is part of the opentelemetry-collector-contrib and can be found here. 

In the OTEL collector configuration file, add a new section in the exporter for Datadog. Environment variables are also supported.

exporters:
datadog:
api:
site:  (datadoghq.com, us3.datadoghq.com, us5.datadoghq.com ...)
key: 

After creating the exporter, update existing pipelines to use the exporter to start sending data to Datadog. The below example shows a host receiver for gathering system metrics, a batch processor for sampling, and a pipeline tying the receiver, processor and exporter together. The pipeline is designed to send system metrics to Datadog:

receivers:
# The hostmetrics receiver is required to get correct infrastructure metrics in Datadog.
hostmetrics:
collection_interval: 10s
    		scrapers:
paging:
        			metrics:
          				system.paging.utilization:
            				enabled: true
cpu:
        			metrics:
          				system.cpu.utilization:
            				enabled: true
      		disk:
      		filesystem:
metrics:
          				system.filesystem.utilization:
            				enabled: true
      		load:
      		memory:
      		network:
      		processes:

processors:
batch:
send_batch_max_size: 100
    		send_batch_size: 10
    		timeout: 10s

service:
pipelines:
metrics:
receivers: [hostmetrics]
      		processors: [batch]
      		exporters: [datadog]

After a few minutes, host metrics being collected will start coming into Datadog.

Have questions? Reach out to our team at chat@rapdev.io.