Datadog Agent ingestion

Overview

This topic explains how to send traces, metrics, and logs from the Datadog Agent to LaunchDarkly’s observability features.

LaunchDarkly’s OpenTelemetry Collector includes a Datadog-compatible receiver. If you already run the Datadog Agent in your infrastructure, you can configure it to send APM traces, metrics, and logs to LaunchDarkly without changing your application instrumentation. Once received, this telemetry is available in the Traces, Logs, and observability dashboards in the LaunchDarkly UI.

Prerequisites

Before you configure Datadog Agent ingestion, you must:

• Have LaunchDarkly observability enabled for your project
• Have the Datadog Agent v6.0 or later installed and running in your infrastructure
• Know your LaunchDarkly client-side ID

To find your client-side ID:

1. In the LaunchDarkly UI, click the project dropdown to open the project menu.
2. Select Project settings.
3. Click Environments.
4. Find the environment you want to use and copy the Client-side ID value.

Configure the Datadog Agent

To send telemetry to LaunchDarkly, configure the Datadog Agent to use LaunchDarkly’s Datadog-compatible endpoint. You can replace the standard Datadog intake, or send data to both Datadog and LaunchDarkly simultaneously using dual shipping.

Endpoint: otel.observability.app.launchdarkly.com:8126

Configure using the agent configuration file

To configure the Datadog Agent using the datadog.yaml configuration file, include these lines:

1
apm_config:
2
  enabled: true
3
  apm_dd_url: http://otel.observability.app.launchdarkly.com:8126
4
5
logs_config:
6
  logs_dd_url: otel.observability.app.launchdarkly.com:8126
7
  use_http: true
8
9
dd_url: http://otel.observability.app.launchdarkly.com:8126

Configure using environment variables

If you run the Datadog Agent in a container, you can use environment variables to provide the configuration:

$
DD_APM_ENABLED=true
$
DD_APM_DD_URL=http://otel.observability.app.launchdarkly.com:8126
$
DD_DD_URL=http://otel.observability.app.launchdarkly.com:8126

Configure using Docker Compose

If you run the Datadog Agent as a Docker container, add the configuration to your docker-compose.yml file:

1
services:
2
  datadog-agent:
3
    image: gcr.io/datadoghq/agent:latest
4
    environment:
5
      - DD_APM_ENABLED=true
6
      - DD_APM_DD_URL=http://otel.observability.app.launchdarkly.com:8126
7
      - DD_DD_URL=http://otel.observability.app.launchdarkly.com:8126
8
      - DD_API_KEY=placeholder

Sending data to both Datadog and LaunchDarkly

The examples above replace the default Datadog intake with LaunchDarkly’s endpoint. If you want to continue sending data to Datadog while also sending it to LaunchDarkly, you can configure the Datadog Agent to dual ship telemetry to both destinations.

Dual shipping lets you keep your existing Datadog dashboards, alerts, and workflows while also using LaunchDarkly’s observability features and guarded rollouts.

To learn how to configure dual shipping, read Dual Shipping in the Datadog documentation.

When configuring dual shipping, use http://otel.observability.app.launchdarkly.com:8126 as the additional endpoint for APM traces, metrics, and logs.

Associate telemetry with your LaunchDarkly project

LaunchDarkly uses the launchdarkly.project_id resource attribute to route telemetry to the correct project. Set this to your LaunchDarkly client-side ID.

You can set this attribute in your application’s OpenTelemetry SDK configuration, or use the OTEL_RESOURCE_ATTRIBUTES environment variable for services that send data through the agent:

$
export OTEL_RESOURCE_ATTRIBUTES="launchdarkly.project_id=YOUR_CLIENT_SIDE_ID"

Alternatively, if you use the OpenTelemetry Collector in front of the Datadog Agent, you can inject this attribute using a resource processor. To learn more, read OpenTelemetry in server-side SDKs.

Attaching feature flag context to traces

To use Datadog trace data with LaunchDarkly features like guarded rollouts and autogenerated metrics, your traces must include feature flag evaluation data. LaunchDarkly uses this data to correlate traces with specific flag evaluations and contexts.

You can attach feature flag context to your Datadog traces by creating a custom SDK hook. The hook instruments each flag evaluation as a child span on your existing Datadog traces, adding attributes that LaunchDarkly uses to connect traces to flag evaluations.

The hook must set the following span attributes on each flag evaluation:

Go example

This example uses a BeforeEvaluation hook to start a Datadog span with flag metadata, and an AfterEvaluation hook to record the result and finish the span.

1
import (
2
	"context"
3
	"encoding/json"
4
	"fmt"
5
6
	"github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
7
	"github.com/launchdarkly/go-server-sdk/v7/ldhooks"
8
	"github.com/launchdarkly/go-sdk-common/v3/ldreason"
9
)
10
11
type DatadogHook struct {
12
	ldhooks.Unimplemented
13
	metadata ldhooks.Metadata
14
}
15
16
func NewDatadogHook() DatadogHook {
17
	return DatadogHook{
18
		metadata: ldhooks.NewMetadata("datadog-tracing"),
19
	}
20
}
21
22
func (h DatadogHook) Metadata() ldhooks.Metadata {
23
	return h.metadata
24
}
25
26
func (h DatadogHook) BeforeEvaluation(
27
	ctx context.Context,
28
	seriesContext ldhooks.EvaluationSeriesContext,
29
	data ldhooks.EvaluationSeriesData,
30
) (ldhooks.EvaluationSeriesData, error) {
31
	span, _ := tracer.StartSpanFromContext(ctx, "feature_flag.evaluation",
32
		tracer.ResourceName(seriesContext.FlagKey()),
33
		tracer.SpanType("launchdarkly"),
34
	)
35
36
	ldCtx := seriesContext.Context()
37
	contextKeys := make(map[string]string)
38
	for i := 0; i < ldCtx.IndividualContextCount(); i++ {
39
		if individualCtx := ldCtx.IndividualContextByIndex(i); individualCtx.IsDefined() {
40
			contextKeys[string(individualCtx.Kind())] = individualCtx.Key()
41
		}
42
	}
43
	if ldCtx.IndividualContextCount() == 0 && ldCtx.IsDefined() {
44
		contextKeys[string(ldCtx.Kind())] = ldCtx.Key()
45
	}
46
	contextKeysJSON, _ := json.Marshal(contextKeys)
47
48
	span.SetTag("feature_flag.context.id", ldCtx.FullyQualifiedKey())
49
	span.SetTag("feature_flag.contextKeys", string(contextKeysJSON))
50
	span.SetTag("feature_flag.key", seriesContext.FlagKey())
51
	span.SetTag("feature_flag.provider.name", "LaunchDarkly")
52
53
	return ldhooks.NewEvaluationSeriesBuilder(data).Set("span", span).Build(), nil
54
}
55
56
func (h DatadogHook) AfterEvaluation(
57
	ctx context.Context,
58
	seriesContext ldhooks.EvaluationSeriesContext,
59
	data ldhooks.EvaluationSeriesData,
60
	detail ldreason.EvaluationDetail,
61
) (ldhooks.EvaluationSeriesData, error) {
62
	if spanData, ok := data.Get("span"); ok {
63
		if span, ok := spanData.(*tracer.Span); ok {
64
			span.SetTag("feature_flag.result.value",
65
				fmt.Sprintf("%v", detail.Value.AsArbitraryValue()))
66
			span.Finish()
67
		}
68
	}
69
	return data, nil
70
}

1
import (
2
	ld "github.com/launchdarkly/go-server-sdk/v7"
3
	"github.com/launchdarkly/go-server-sdk/v7/ldhooks"
4
)
5
6
ldConfig := ld.Config{
7
	Hooks: []ldhooks.Hook{NewDatadogHook()},
8
}
9
10
ldClient, err := ld.MakeCustomClient("YOUR_SDK_KEY", ldConfig, 5*time.Second)

1
value, err := ldClient.BoolVariationCtx(ctx, "my-flag-key", ldContext, false)

Configuring the Datadog tracer

Configure the Datadog tracer in your application to send traces to LaunchDarkly’s Datadog-compatible endpoint. Set the X-LaunchDarkly-Project global tag to your LaunchDarkly project ID so that traces are routed to the correct project:

1
import "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
2
3
tracer.Start(
4
	tracer.WithAgentURL("http://otel.observability.app.launchdarkly.com:8126"),
5
	tracer.WithService("your-service-name"),
6
	tracer.WithEnv("production"),
7
	tracer.WithGlobalTag("X-LaunchDarkly-Project", "YOUR_PROJECT_ID"),
8
)
9
defer tracer.Stop()

Using flag data with guarded rollouts

After you configure the hook and tracer, LaunchDarkly automatically processes your Datadog trace data and associates it with the feature flags and contexts in your evaluations. This enables you to:

• Use autogenerated OpenTelemetry metrics, such as HTTP error rates and latencies, with guarded rollouts
• Monitor how flag changes impact your application performance in Traces
• Correlate errors and latency regressions with specific flag evaluations and context attributes

To learn more about setting up guarded rollouts, read Creating guarded rollouts.

What data is collected

The Datadog Agent sends the following telemetry types to LaunchDarkly:

• Traces: APM traces from your instrumented services, visible in Traces
• Metrics: Infrastructure and application metrics
• Logs: Application logs collected by the agent log collection feature, visible in Logs

Verify that data is being received

After you configure the Datadog Agent, telemetry begins flowing to LaunchDarkly.

To verify that traces are being received:

1. In the LaunchDarkly UI, expand Observe in the left navigation.
2. Click Traces.
3. Look for traces from your Datadog-instrumented services.

To verify that logs are being received:

1. In the LaunchDarkly UI, expand Observe in the left navigation.
2. Click Logs.
3. Look for logs from your services.

It may take a few minutes for data to appear after you first configure the agent.

Filtering ingested data

You can configure ingestion filters to control which logs are stored in LaunchDarkly. This is useful for reducing noise or staying within your observability quotas.

To learn more, read Filters.