Tracking AI metrics

Use the TrackRequest function to wrap your AI model provider call and record metrics when the model generates content.

The tracker is returned from your call to customize the AI Config and is specific to that AI Config. Make sure to call Config again each time you generate content from your AI model so that metrics are correctly associated with the customized AI Config variation.

Here’s how to call an AI model provider and record metrics from generation:

1
if (tracker.Config.Enabled == true) {
2
3
  // Use Task.Run here only as an example. You can wrap any async provider call.
4
  var response = tracker.TrackRequest(Task.Run(() =>
5
    {
6
      // Make request to a provider, which automatically tracks metrics in LaunchDarkly.
7
      // When sending the request to a provider, use details from tracker.Config.
8
      // For example, you can pass tracker.Config.Model and tracker.Config.Messages.
9
      // Optionally, return response metadata for additional logging.
10
      //
11
      // CAUTION: If the call inside Task.Run throws an exception,
12
      // the SDK will re-throw that exception.
13
14
      return new Response
15
      {
16
        Usage = new Usage { Total = 1, Input = 1, Output = 1 }, // Token usage data
17
        Metrics = new Metrics { LatencyMs = 100 } // Metrics data
18
      };
19
    }
20
  ));
21
22
} else {
23
24
  // Application path to take when tracker.Config is disabled
25
26
}

If you want to perform additional tracking beyond what LaunchDarkly provides automatically, populate the Response object with the metrics you want to record.

You can use the SDK’s other Track* functions to record metrics manually. The TrackRequest function expects a response, so manual tracking may be required for streaming use cases.

Each of the Track* functions sends data back to LaunchDarkly. The Monitoring tab of the AI Config in the LaunchDarkly UI aggregates metrics from all variations of the AI Config.

Here’s how to record metrics manually:

1
/// Track your own start and stop time.
2
/// Set duration to the time (in ms) that your AI model generation takes.
3
/// The duration may include network latency, depending on how you calculate it.
4
5
tracker.TrackDuration(response.Metrics.LatencyMs);

The SDK automatically flushes pending analytics events to LaunchDarkly at regular intervals. If you have a short-lived application, you may need to explicitly request that the underlying LaunchDarkly client deliver any pending analytics events to LaunchDarkly, using flush() or close().

Here’s how:

1
baseClient.Flush();

To learn more, read LDAIConfigTracker.

Use the TrackRequest() function to wrap your AI model provider call and record metrics from generation.

The tracker is returned from your call to customize the AI Config and is specific to that AI Config. Make sure to call Config() again each time you use the tracker and generate content from your AI model so that your metrics are correctly associated with the customized AI Config variation.

Here’s how to call an AI model provider and record metrics from generation:

1
if cfg.Enabled() {
2
3
  response, err := tracker.TrackRequest(func(config *Config) (ProviderResponse, error) {
4
5
    // Make request to a provider, which automatically tracks metrics in LaunchDarkly.
6
    // When sending the request to a provider, use details from config.
7
    // For example, you can pass a model parameter (config.ModelParam) or messages (config.Messages).
8
    // Optionally, return response metadata for additional logging.
9
10
    return ProviderResponse{
11
      Usage: TokenUsage{
12
        Total: 1, // Token usage data
13
      },
14
      Metrics: Metrics{
15
        Latency: 10 * time.Millisecond, // Metrics data
16
      },
17
    }, nil
18
  })
19
20
} else {
21
22
  // Application path to take when the configuration is disabled
23
24
}

You can use the SDK’s other Track* functions to record metrics manually. The TrackRequest function expects a response, so manual tracking may be required for streaming use cases.

Each Track* function sends data back to LaunchDarkly. To review the metrics that have been recorded, use GetSummary:

1
summary := tracker.GetSummary();
2
3
// recorded metrics available in summary.Duration, summary.Feedback,
4
// summary.Tokens, summary.Success, summary.TimeToFirstToken

To learn more, read GetSummary and MetricSummary.

The Monitoring tab of the AI Config in the LaunchDarkly UI aggregates data from the Track* functions from across all variations of the AI Config.

Here’s how to record metrics manually:

1
// Track your own start and stop time.
2
3
// Set duration to the time that your AI model generation takes.
4
// The duration may include network latency, depending on how you calculate it.
5
6
tracker.TrackDuration(10 * time.Millisecond);

The SDK automatically flushes these pending analytics events to LaunchDarkly at regular intervals. If you have a short-lived application, you may need to explicitly request that the underlying LaunchDarkly client deliver any pending analytics events to LaunchDarkly, using flush() or close().

Here’s how:

1
client.Flush();

To learn more, read Tracker.

If your AI Config uses completion mode, the Node.js (server-side) AI SDK provides several options for making a request to your generative AI provider and recording metrics from your AI model generation. You can use any of the following options:

• If you are working with OpenAI or Bedrock Converse, use the trackOpenAIMetrics or trackBedrockConverseMetrics functions, respectively, to record metrics. These functions take the result of your generative operation as a parameter.
• If you are working with Vercel, use either of the trackVercelAISDKGenerateTextMetrics or trackVercelAISDKStreamTextMetrics functions to record metrics. These functions take the result of a generative operation from any provider supported by the Vercel AI SDK as a parameter.
• If you are using a generative AI provider or framework for which the SDK does not provide a convenience function, use the SDK’s other track* functions to record metrics manually.

If your AI Config uses agent mode, you can access the instructions returned from the agent() call to send to your AI model. Use the tracker returned in this call to record metrics.

In the following examples, the tracker is from your call to customize the AI Config, and is specific to that AI Config. Make sure to call config again each time you use the tracker and generate content from your AI model, so that your metrics are correctly associated with the customized AI Config variation.

Here’s how:

1
const { tracker } = aiConfig;
2
3
if (aiConfig.enabled) {
4
5
  // Pass in the result of the OpenAI operation.
6
  // When you call the OpenAI operation, use details from aiConfig.
7
  // For instance, you can pass aiConfig.messages
8
  // and aiConfig.model to your specific OpenAI operation.
9
  //
10
  // CAUTION: If the call inside of trackOpenAIMetrics throws an exception,
11
  // the SDK will re-throw that exception
12
13
  const completion = await tracker.trackOpenAIMetrics(async () =>
14
    client.chat.completions.create({
15
      messages: aiConfig.messages || [],
16
      model: aiConfig.model?.name || 'gpt-4',
17
      temperature: (aiConfig.model?.parameters?.temperature as number) ?? 0.5,
18
      maxTokens: (aiConfig.model?.parameters?.maxTokens as number) ?? 4096,
19
    }),
20
  );
21
22
} else {
23
24
  // Application path to take when the aiConfig is disabled
25
26
}

Here’s how to make a request using the Vercel AI SDK’s generateText or streamText, and record the metrics:

1
const { tracker } = aiConfig;
2
3
// Pass in the result of the Vercel AI SDK's generateText function.
4
// When you call generateText, use details from the aiConfig,
5
// mapped to the input format required for the Vercel AI SDK.
6
//
7
// CAUTION: The toVercelAISDK function may throw an exception
8
// if a Vercel AI SDK model cannot be determined.
9
10
const completion = await tracker.trackVercelAISDKGenerateTextMetrics(
11
 generateText(
12
    aiConfig.toVercelAISDK(vercelProvider, vercelProviderOptions)
13
  )
14
)

To learn more, read trackVercelAISDKGenerateTextMetrics, trackVercelAISDKStreamTextMetrics, and toVercelAISDK.

You can use the SDK’s other track* functions to record these metrics manually. You may need to do this if you are using a model for which the SDK does not provide a convenience track[Model]Metrics function, and you are not using the Vercel AI SDK. The track[Model]Metrics functions are expecting a response, so you may also need to do this if your application requires streaming.

Each of the track* functions sends data back to LaunchDarkly. To review the metrics that have been recorded, use getSummary:

1
const summary = aiConfig.tracker.getSummary();
2
3
// recorded metrics available in summary.durationMs, summary.feedback,
4
// summary.tokens, summary.success, summary.timeToFirstTokenMs

To learn more, read getSummary.

The Monitoring tab of the AI Config in the LaunchDarkly UI aggregates data from the track* functions from across all variations of the AI Config.

Here’s how to record metrics manually:

1
// Track your own start and stop time.
2
3
// Set duration to the time (in ms) that your AI model generation takes.
4
// The duration may include network latency, depending on how you calculate it.
5
6
aiConfig.tracker.trackDuration(duration);

In completion mode, the tracker is returned from the config() call, so you can access it directly, as in the examples above. Make sure to call config() again each time you use the tracker and generate content from your AI model.

In agent mode, the tracker is part of the agent returned from the agent() or agents() call. If you are working in agent mode, replace tracker with agent.tracker in the examples above.

The SDK automatically flushes these pending analytics events to LaunchDarkly at regular intervals. If you have a short-lived application, you may need to explicitly request that the underlying LaunchDarkly client deliver any pending analytics events to LaunchDarkly, using flush() or close().

Here’s how:

1
ldClient.flush();

To learn more, read LDAIConfigTracker.

If your AI Config uses completion mode, use one of the track_[model]_metrics functions to record metrics from your AI model generation. The SDK provides separate track_[model]_metrics functions for several of the models that you can select when you set up your AI Config variations in the LaunchDarkly user interface.

If your AI Config uses agent mode, you can access the instructions returned from the customized AI Config to send to your AI model. Use the tracker returned as part of the agent() or agents() functions to record metrics.

The tracker is returned from your call to customize the AI Config, and is specific to that AI Config. Make sure to call config again each time you use the tracker and generate content from your AI model, so that your metrics are correctly associated with the customized AI Config variation.

Here’s how:

1
if config.enabled:
2
    # Pass in the result of the OpenAI operation.
3
    # When calling the OpenAI operation, use details from config.
4
    # For instance, you can pass config.model.name
5
    # and config.messages[0].content to your specific OpenAI operation.
6
    #
7
    # CAUTION: If the call inside of track_openai_metrics throws an exception,
8
    # the SDK will re-throw that exception
9
10
    messages = [] if config.messages is None else config.messages
11
    completion = tracker.track_openai_metrics(
12
        lambda:
13
          openai_client.chat.completions.create(
14
              model=config.model.name,
15
              messages=[message.to_dict() for message in messages],
16
          )
17
    )
18
else:
19
    # Application path to take when the config is disabled

You can use the SDK’s other track* functions to record metrics manually. This is useful when the SDK does not provide a convenience track_[model]_metrics function for your model, or when your application requires streaming, because track_[model]_metrics functions expect a response.

Each track* function sends data back to LaunchDarkly. To review the metrics that have been recorded, use get_summary:

1
tracker.get_summary()
2
3
# recorded metrics available in tracker.get_summary().duration, .feedback,
4
# .success, .usage, and .time_to_first_token

To learn more, read get_summary and LDAIMetricSummary.

The Monitoring tab of the AI Config in the LaunchDarkly UI aggregates data from the track* functions across all variations of the AI Config.

Here’s how to record metrics manually:

1
# Track your own start and stop time.
2
3
# Set duration to the time (in ms) that your AI model generation takes.
4
# The duration may include network latency, depending on how you calculate it.
5
6
tracker.track_duration(duration)

In completion mode, the tracker is returned from the config() call, so you can access it directly, as in the examples above. Call config() each time you generate content from your AI model so your metrics are associated with the correct AI Config variation.

In agent mode, the tracker is part of the agent returned from agent() or agents(). In the examples above, replace tracker with agent.tracker.

The SDK automatically flushes pending analytics events to LaunchDarkly at regular intervals. If you have a short-lived application, you may need to explicitly request that the underlying LaunchDarkly client deliver any pending analytics events to LaunchDarkly, using flush() or close().

Here’s how:

1
ldclient.get().flush()

To learn more, read LDAIConfigTracker.

Use one of the track_[model]_metrics functions to wrap your AI model provider call and record metrics from generation. The SDK provides separate track_[model]_metrics functions for several models that you can select when you set up your AI Config variations in the LaunchDarkly UI.

The tracker is returned as part of your call to customize the AI Config and is specific to that AI Config. Make sure to call config again each time you use the tracker and generate content from your AI model so your metrics are correctly associated with the customized AI Config variation.

Here’s how to use a provider-specific function to call OpenAI or Bedrock providers and record metrics from your AI model generation:

1
if ai_config.enabled
2
  # Wrap the OpenAI operation to record metrics.
3
  # When calling the OpenAI operation, use details from ai_config.
4
  # For instance, you can pass ai_config.model.name
5
  # and ai_config.messages to your specific OpenAI operation.
6
  #
7
  # CAUTION: If the call inside track_openai_metrics throws an exception,
8
  # the SDK will re-throw that exception.
9
10
  completion = ai_config.tracker.track_openai_metrics(
11
    openai_client.chat.completions.create(
12
      model: ai_config.model.name,
13
      messages: ai_config.messages.map(&:to_h)
14
    )
15
  )
16
else
17
  # Application path to take when the ai_config is disabled
18
end

You can use the SDK’s other track* functions to record these metrics manually. You may need to do this if you are using a model for which the SDK does not provide a convenience track_[model]_metrics function. The track_[model]_metrics functions are expecting a response, so you may also need to do this if your application requires streaming.

Each of the track* functions sends data back to LaunchDarkly. To review the metrics that have been recorded, use the summary property in the tracker:

1
ai_config.tracker.summary
2
3
# recorded metrics available in tracker.summary include: duration, feedback,
4
# success, usage, and time_to_first_token

To learn more, read summary and MetricSummary.

The Monitoring tab of the AI Config in the LaunchDarkly UI aggregates data from the track* functions from across all variations of the AI Config.

Here’s how to record metrics manually:

1
# Track your own start and stop time.
2
3
# Set duration to the time (in ms) that your AI model generation takes.
4
# The duration may include network latency, depending on how you calculate it.
5
6
ai_config.tracker.track_duration(duration)

Make sure to call config again each time you use the tracker and generate content from your AI model.

The SDK automatically flushes these pending analytics events to LaunchDarkly at regular intervals. If you have a short-lived application, you may need to explicitly request that the underlying LaunchDarkly client deliver any pending analytics events to LaunchDarkly, using flush or close.

Here’s how:

1
ld_client.flush()

To learn more, read ConfigTracker.