Datadog Agent ingestion

This feature is in Early AccessLaunchDarkly observability features are publicly available in early access. Enable observability in the billing page.Datadog Agent ingestion is not tied to a specific SDK. The LaunchDarkly Datadog-compatible endpoint accepts any traces, metrics, and logs that the Datadog Agent forwards, regardless of which Datadog tracing library, SDK, or integration instruments your applications. You do not need the LaunchDarkly observability SDKs to use it. To use ingested traces with guarded rollouts, your traces must include feature flag evaluation data. To learn how, read Attaching feature flag context to traces.

This topic explains how to send traces, metrics, and logs from the Datadog Agent to LaunchDarkly observability features.The LaunchDarkly OpenTelemetry Collector includes a Datadog-compatible receiver. If you already run the Datadog Agent in your infrastructure, you can configure it to send application performance monitoring (APM) traces, metrics, and logs to LaunchDarkly without changing your application instrumentation. This telemetry is available on the Traces, Logs, and observability pages in the LaunchDarkly UI.

Prerequisites

Before you configure Datadog Agent ingestion, you must:

Have LaunchDarkly observability enabled for your project
Have the Datadog Agent v6.0 or later installed and running in your infrastructure
Know your LaunchDarkly client-side ID

To find your client-side ID:

In the LaunchDarkly UI, click the project dropdown to open the project menu.
Select Project settings.
Click Environments.
Find the environment you want to use and copy the Client-side ID value.

Configure the Datadog Agent

To send telemetry to LaunchDarkly, configure the Datadog Agent to use the LaunchDarkly Datadog-compatible endpoint. You can replace the standard Datadog intake, or send data to both Datadog and LaunchDarkly simultaneously using dual shipping.Endpoint: datadog.observability.app.launchdarkly.com:8126

Configure using the agent configuration file

To configure the Datadog Agent using the datadog.yaml configuration file, include these lines:

apm_config:
  enabled: true
  apm_dd_url: https://datadog.observability.app.launchdarkly.com:8126

logs_config:
  logs_dd_url: datadog.observability.app.launchdarkly.com:8126
  use_http: true
  expected_tags_duration: "87600h"

dd_url: https://datadog.observability.app.launchdarkly.com:8126

Configure using environment variables

If you run the Datadog Agent in a container, you can use environment variables to provide the configuration:

DD_APM_ENABLED=true
DD_APM_DD_URL=https://datadog.observability.app.launchdarkly.com:8126
DD_DD_URL=https://datadog.observability.app.launchdarkly.com:8126
DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION=87600h

Configure using Docker Compose

If you run the Datadog Agent as a Docker container, add the configuration to your docker-compose.yml file:

services:
  datadog-agent:
    image: gcr.io/datadoghq/agent:latest
    environment:
      - DD_APM_ENABLED=true
      - DD_APM_DD_URL=https://datadog.observability.app.launchdarkly.com:8126
      - DD_DD_URL=https://datadog.observability.app.launchdarkly.com:8126
      - DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION=87600h
      - DD_API_KEY=placeholder

DD_API_KEY is requiredThe Datadog Agent requires a DD_API_KEY value to start, even when routing to a non-Datadog endpoint. You can use any non-empty placeholder string. LaunchDarkly does not validate or use this value.

Sending data to both Datadog and LaunchDarkly

The examples above replace the default Datadog intake with the LaunchDarkly endpoint. If you want to continue sending data to Datadog while also sending it to LaunchDarkly, you can configure the Datadog Agent to dual ship telemetry to both destinations.Dual shipping lets you keep your existing Datadog dashboards, alerts, and workflows while also using LaunchDarkly observability features and guarded rollouts.To learn how to configure dual shipping, read Dual Shipping in the Datadog documentation.When configuring dual shipping, use https://datadog.observability.app.launchdarkly.com:8126 as the additional endpoint for APM traces, metrics, and logs.

Associate telemetry with your LaunchDarkly project

LaunchDarkly uses the launchdarkly.project_id attribute to route telemetry to the correct project. Set it to your LaunchDarkly client-side ID. LaunchDarkly does not authenticate at the ingestion endpoint, so this attribute associates your telemetry with your project. The attribute must be present in your spans and logs, and not only as a host-level agent tag.How you set the attribute depends on how your services are instrumented:

Applications that use the Datadog tracer (dd-trace) should set the attribute as a global span tag. You can implement this without changing application code by setting the DD_TAGS environment variable on your service. This tags every span with the attribute, and the tags also flow through to your logs:

export DD_TAGS="launchdarkly.project_id:YOUR_CLIENT_SIDE_ID"

Applications that use an OpenTelemetry SDK should set the attribute as a resource attribute in the OTEL_RESOURCE_ATTRIBUTES environment variable:

export OTEL_RESOURCE_ATTRIBUTES="launchdarkly.project_id=YOUR_CLIENT_SIDE_ID"

If you run the OpenTelemetry Collector in front of the Datadog Agent, you can instead inject this attribute using a resource processor. To learn more, read OpenTelemetry in server-side SDKs.

The x-launchdarkly-project tag is also acceptedFor backward compatibility, LaunchDarkly also routes telemetry tagged with x-launchdarkly-project. New configurations should prefer launchdarkly.project_id.

Attaching feature flag context to traces

To use Datadog trace data with LaunchDarkly features like guarded rollouts and OpenTelemetry autogenerated metrics, your traces must include feature flag evaluation data. LaunchDarkly uses this data to correlate traces with specific flag evaluations and contexts.You can attach feature flag context to your Datadog traces by creating a custom SDK hook. The hook instruments each flag evaluation as a child span on your existing Datadog traces, adding attributes that LaunchDarkly uses to connect traces to flag evaluations.The hook must set the following span attributes on each flag evaluation:

Attribute	Description
`feature_flag.key`	The key of the evaluated flag.
`feature_flag.context.id`	The fully-qualified key of the LaunchDarkly context.
`feature_flag.contextKeys`	A JSON object mapping each context kind to its key, for example .
`feature_flag.provider.name`	Set to `LaunchDarkly`.
`feature_flag.result.value`	The string representation of the evaluation result.

Go example

This example uses a BeforeEvaluation hook to start a Datadog span with flag metadata, and an AfterEvaluation hook to record the result and finish the span.

Expand Go hook implementation

First, define the hook:

import (
	"context"
	"encoding/json"
	"fmt"

	"github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
	"github.com/launchdarkly/go-server-sdk/v7/ldhooks"
	"github.com/launchdarkly/go-sdk-common/v3/ldreason"
)

type DatadogHook struct {
	ldhooks.Unimplemented
	metadata ldhooks.Metadata
}

func NewDatadogHook() DatadogHook {
	return DatadogHook{
		metadata: ldhooks.NewMetadata("datadog-tracing"),
	}
}

func (h DatadogHook) Metadata() ldhooks.Metadata {
	return h.metadata
}

func (h DatadogHook) BeforeEvaluation(
	ctx context.Context,
	seriesContext ldhooks.EvaluationSeriesContext,
	data ldhooks.EvaluationSeriesData,
) (ldhooks.EvaluationSeriesData, error) {
	span, _ := tracer.StartSpanFromContext(ctx, "feature_flag.evaluation",
		tracer.ResourceName(seriesContext.FlagKey()),
		tracer.SpanType("launchdarkly"),
	)

	ldCtx := seriesContext.Context()
	contextKeys := make(map[string]string)
	for i := 0; i < ldCtx.IndividualContextCount(); i++ {
		if individualCtx := ldCtx.IndividualContextByIndex(i); individualCtx.IsDefined() {
			contextKeys[string(individualCtx.Kind())] = individualCtx.Key()
		}
	}
	if ldCtx.IndividualContextCount() == 0 && ldCtx.IsDefined() {
		contextKeys[string(ldCtx.Kind())] = ldCtx.Key()
	}
	contextKeysJSON, _ := json.Marshal(contextKeys)

	span.SetTag("feature_flag.context.id", ldCtx.FullyQualifiedKey())
	span.SetTag("feature_flag.contextKeys", string(contextKeysJSON))
	span.SetTag("feature_flag.key", seriesContext.FlagKey())
	span.SetTag("feature_flag.provider.name", "LaunchDarkly")

	return ldhooks.NewEvaluationSeriesBuilder(data).Set("span", span).Build(), nil
}

func (h DatadogHook) AfterEvaluation(
	ctx context.Context,
	seriesContext ldhooks.EvaluationSeriesContext,
	data ldhooks.EvaluationSeriesData,
	detail ldreason.EvaluationDetail,
) (ldhooks.EvaluationSeriesData, error) {
	if spanData, ok := data.Get("span"); ok {
		if span, ok := spanData.(*tracer.Span); ok {
			span.SetTag("feature_flag.result.value",
				fmt.Sprintf("%v", detail.Value.AsArbitraryValue()))
			span.Finish()
		}
	}
	return data, nil
}

Then register the hook when you initialize the LaunchDarkly client:

import (
	ld "github.com/launchdarkly/go-server-sdk/v7"
	"github.com/launchdarkly/go-server-sdk/v7/ldhooks"
)

ldConfig := ld.Config{
	Hooks: []ldhooks.Hook{NewDatadogHook()},
}

ldClient, err := ld.MakeCustomClient("YOUR_SDK_KEY", ldConfig, 5*time.Second)

When you evaluate flags, use the context-aware variation methods to pass the request context through to the hook:

value, err := ldClient.BoolVariationCtx(ctx, "my-flag-key", ldContext, false)

Configuring the Datadog tracer

Configure the Datadog tracer in your application to send traces to the LaunchDarkly Datadog-compatible endpoint. Set the launchdarkly.project_id global tag to your LaunchDarkly client-side ID so that traces are routed to the correct project. If you prefer not to change your application code, set the same tag using the DD_TAGS environment variable instead, as described in Associate telemetry with your LaunchDarkly project.

import "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"

tracer.Start(
	tracer.WithAgentURL("https://datadog.observability.app.launchdarkly.com:8126"),
	tracer.WithService("your-service-name"),
	tracer.WithEnv("production"),
	tracer.WithGlobalTag("launchdarkly.project_id", "YOUR_CLIENT_SIDE_ID"),
)
defer tracer.Stop()

Using flag data with guarded rollouts

After you configure the hook and tracer, LaunchDarkly automatically processes your Datadog trace data and associates it with the feature flags and contexts in your evaluations. This enables you to:

Use OpenTelemetry autogenerated metrics, such as HTTP error rates and latencies, with guarded rollouts
Monitor how flag changes impact your application performance in Traces
Correlate errors and latency regressions with specific flag evaluations and context attributes

To learn more about setting up guarded rollouts, read Creating guarded rollouts.

What data is collected

The Datadog Agent sends the following telemetry types to LaunchDarkly:

Traces: APM traces from your instrumented services, visible in Traces.
Metrics: Infrastructure and application metrics.
Logs: Application logs collected by the agent log collection feature, visible in Logs.

Verify that data is being received

After you configure the Datadog Agent, telemetry begins flowing to LaunchDarkly.To verify that traces are being received:

Open the Telemetry section and navigate to the Traces list.
Search for traces from your Datadog-instrumented services.

To verify that logs are being received:

Open the Telemetry section and navigate to the Logs list.
Search for logs from your services.

It may take a few minutes for data to appear after you first configure the agent.

Filtering ingested data

You can configure ingestion filters to control which logs are stored in LaunchDarkly. This is useful for reducing noise or staying within your observability quotas.To learn more, read Filters.

Get started

AgentControl

Feature flags

Releases

Observability

Experimentation

Metrics and events

Warehouse native

Infrastructure

Your account

Datadog Agent ingestion

Prerequisites

Configure the Datadog Agent

Configure using the agent configuration file

Configure using environment variables

Configure using Docker Compose

Sending data to both Datadog and LaunchDarkly

Associate telemetry with your LaunchDarkly project

Attaching feature flag context to traces

Go example

Configuring the Datadog tracer

Using flag data with guarded rollouts

What data is collected

Verify that data is being received

Filtering ingested data

​Prerequisites

​Configure the Datadog Agent

​Configure using the agent configuration file

​Configure using environment variables

​Configure using Docker Compose

​Sending data to both Datadog and LaunchDarkly

​Associate telemetry with your LaunchDarkly project

​Attaching feature flag context to traces

​Go example

​Configuring the Datadog tracer

​Using flag data with guarded rollouts

​What data is collected

​Verify that data is being received

​Filtering ingested data

Prerequisites

Configure the Datadog Agent

Configure using the agent configuration file

Configure using environment variables

Configure using Docker Compose

Sending data to both Datadog and LaunchDarkly

Associate telemetry with your LaunchDarkly project

Attaching feature flag context to traces

Go example

Configuring the Datadog tracer

Using flag data with guarded rollouts

What data is collected

Verify that data is being received

Filtering ingested data