Node.js (server-side) AI SDK reference

Overview

This topic documents how to get started with the Node.js (server-side) AI SDK, and links to reference information on all of the supported features. You can use either JavaScript or TypeScript when working with the Node.js (server-side) AI SDK.

Get started

LaunchDarkly AI SDKs interact with AI Configs. AI Configs are the LaunchDarkly resources that manage model configurations and messages for your generative AI applications.

You can use the Node.js (server-side) AI SDK to customize your AI Config based on the context that you provide. This means both the messages and the model evaluation in your generative AI application are specific to each end user, at runtime. You can also use the AI SDKs to record metrics from your AI model generation, including duration and tokens.

Follow these instructions to start using the Node.js (server-side) AI SDK in your application.

Install the SDK

First, install the AI SDK as a dependency in your application using your application’s dependency manager. If you want to depend on a specific version, refer to the SDK releases page to identify the latest version. The Node.js (server-side) AI SDK is built on the Node.js (server-side) SDK, so you’ll need to install that as well.

Here’s how:

$
npm install @launchdarkly/node-server-sdk
>
npm install @launchdarkly/server-sdk-ai

Next, import init, LDContext, and initAi in your application code. If you are using TypeScript, you can optionally import the LaunchDarkly LDAIClient and LDAIConfig. These are implied, so are not strictly required.

Here’s how:

1
import { init, LDContext } from '@launchdarkly/node-server-sdk';
2
import { initAi, LDAIClient, LDAIConfig } from '@launchdarkly/server-sdk-ai';

Initialize the client

After you install and import the SDK, create a single, shared instance of LDClient. When the LDClient is initialized, use it to initialize the LDAIClient. The LDAIClient is how you interact with AI Configs. Specify the SDK key to authorize your application to connect to a particular environment within LaunchDarkly.

Here’s how:

1
const ldClient: LDClient = init('sdk-key-123abc');
2
3
try {
4
  await ldClient.waitForInitialization({ timeout: 10 });
5
  // initialization complete
6
} catch (error) {
7
  // timeout or SDK failed to initialize
8
}
9
10
const aiClient: LDAIClient = initAi(ldClient);

Configure the context

Next, configure the context that will use the AI Config, that is, the context that will encounter generated AI content in your application. The context attributes determine which variation of the AI Config LaunchDarkly serves to the end user, based on the targeting rules in your AI Config. If you are using template variables in the messages in your AI Config’s variations, the context attributes also fill in values for the template variables.

Here’s how:

1
const context: LDContext = {
2
  kind: 'user',
3
  key: 'user-key-123abc',
4
  firstName: 'Sandy',
5
  lastName: 'Smith',
6
  email: 'sandy@example.com',
7
  groups: ['Google', 'Microsoft'],
8
};

Customize an AI Config

Then, use config to customize the AI Config. You need to call config each time you generate content from your AI model.

This function returns the customized messages and model. Customization means that any variables you include in the messages when you define the AI Config variation have their values set to the context attributes and variables you pass to config. Then, you can pass the customized messages directly to your AI.

The customization process within the AI SDK is similar to evaluating flags in one of LaunchDarkly’s client-side, server-side, or edge SDKs, in that the SDK completes the customization without a separate network call.

Here’s how:

1
const fallbackConfig = {
2
  model: {
3
    name: 'my-default-model',
4
    parameters: { temperature: 0.8 }
5
  },
6
  messages: [ { role: 'system', content: '' } ],
7
  provider: { name: 'my-default-provider' },
8
  enabled: true,
9
};
10
11
const aiConfig: LDAIConfig = aiClient.config(
12
  'ai-config-key-123abc',
13
  context,
14
  fallbackConfig,
15
  { 'exampleCustomVariable': 'exampleCustomValue' },
16
);

To learn more, read Customizing AI Configs.

Call provider, record metrics from AI model generation

Finally, make a request to your generative AI provider and record metrics from your AI model generation.

The Node.js (server-side) AI SDK provides several options for this:

• If you are working with OpenAI or Bedrock Converse, use the trackOpenAIMetrics or trackBedrockConverseMetrics functions, respectively, to record metrics. These functions take the result of your generative operation as a parameter.
• If you are working with Vercel, use either of the trackVercelAISDKGenerateTextMetrics or trackVercelAISDKGenerateStreamMetrics functions to record metrics. These functions take the result of a generative operation from any provider supported by the Vercel AI SDK as a parameter.
• If you are using a generative AI provider or framework for which the SDK does not provide a convenience function, use the SDK’s other track* functions to record metrics manually.

Here’s how to make a request to OpenAI or Bedrock Converse and record metrics:

1
const { tracker } = aiConfig;
2
3
// Pass in the result of the OpenAI operation.
4
// When you call the OpenAI operation, use details from aiConfig.
5
// For instance, you can pass aiConfig.messages
6
// and aiConfig.model to your specific OpenAI operation.
7
//
8
// CAUTION: If the call inside of trackOpenAIMetrics throws an exception,
9
// the SDK will re-throw that exception
10
11
const completion = await tracker.trackOpenAIMetrics(async () =>
12
  client.chat.completions.create({
13
    messages: aiConfig.messages || [],
14
    model: aiConfig.model?.name || 'gpt-4',
15
    temperature: (aiConfig.model?.parameters?.temperature as number) ?? 0.5,
16
    maxTokens: (aiConfig.model?.parameters?.maxTokens as number) ?? 4096,
17
  }),
18
);

Here’s how to make a request using the Vercel AI SDK’s generateText or streamText, and record the metrics:

1
const { tracker } = aiConfig;
2
3
// Pass in the result of the Vercel AI SDK's generateText function.
4
// When you call generateText, use details from the aiConfig,
5
// mapped to the input format required for the Vercel AI SDK.
6
//
7
// CAUTION: The toVercelAISDK function may throw an exception
8
// if a Vercel AI SDK model cannot be determined.
9
10
const completion = await tracker.trackVercelAISDKGenerateTextMetrics(
11
 generateText(
12
    aiConfig.toVercelAISDK(vercelProvider, vercelProviderOptions)
13
  )
14
)

You can also use the SDK’s other track* functions to record these metrics manually. You may need to do this if you are using a model for which the SDK does not provide a convenience track[Model] function, and you are not using the Vercel AI SDK. The track[Model] functions are expecting a response, so you may also need to do this if your application requires streaming.

Make sure to call config each time you generate content from your AI model:

1
const aiConfig = aiClient.config(
2
  'ai-config-key-123abc',
3
  context,
4
  fallbackConfig,
5
  { 'exampleCustomVariable': 'exampleCustomValue' },
6
);
7
8
const { tracker } = aiConfig;
9
const completion = await tracker.trackOpenAIMetrics(...);

To learn more, read Tracking AI metrics.

Supported features

This SDK supports the following features:

• Anonymous contexts
• Context configuration
• Customizing AI Configs
• Private attributes
• Tracking AI metrics