AgentControlManage AgentControl

Configure token limits

This topic explains how to configure token limits for playground runs and offline evaluations in AgentControl.

As a way to manage AI costs, Admins can set a uniform daily token limit for each member’s playground and offline evaluation usage. Token usage counts towards the daily limit regardless of which large language model (LLM) provider API key the member uses. Each member’s token allotment resets daily at 00:00 UTC. Token limits default to disabled.

Scope of token limits

Token limits apply only to playground runs and offline evaluation, not to production SDK traffic. LaunchDarkly does not limit or meter tokens consumed when your application calls a model provider directly through an AI SDK.

Prerequisites

To configure token limits, you must have the following prerequisites:

  • An Admin or Owner role in your LaunchDarkly organization
  • Access to AgentControl

Viewing token usage

Only Admins and Owners can access the Evaluation tokens page. To view token usage:

  1. Log in to LaunchDarkly as an Admin or Owner.
  2. Click Settings. Under “Plans and usage,” click Evaluation tokens.

Your organization’s average token usage for the current day appears under “Today’s usage,” along with a breakdown showing each member’s usage.

All members, regardless of role, can track their own personal token usage from the Daily token usage counter in the playground sidebar. To reveal the sidebar, select the Show sidebar button. The counter reflects the member’s own usage, not the organization’s aggregate usage, and updates as the member runs prompts in the playground or starts offline evaluations.

Setting a daily token limit

Only Admins and Owners can set a daily token limit. To set a daily token limit for your organization:

  1. Log in to LaunchDarkly as an Admin or Owner.
  2. Click Settings. Under “Plans and usage,” click Evaluation tokens.
  3. Under “Daily token limit per member,” choose one of the following options:
    • Drag the slider to the token amount you want to set, or
    • Enter a token amount in the tokens field.
  4. Click Save.

Admins can allot a maximum of 10,000,000 tokens to each member, or set the limit to “No limit.” This limit applies to every member individually. You cannot set different limits for different members.

Reaching the token limit

Token limits help your organization manage AI costs predictably. Because the limit applies per member, one member reaching their limit does not affect other members. When a member exhausts their daily token limit, LaunchDarkly blocks that member from starting new playground runs and offline evaluation runs and displays a banner in the user interface (UI).

Restoring tokens

Token usage resets automatically at 00:00 UTC each day. Admins can also restore tokens before the daily reset in the following ways:

  • Increase the daily token limit
  • Disable the token limit

Token usage during concurrent evaluations

Because token usage is calculated asynchronously, members who are close to their daily token limit may exceed it slightly when they run a playground generation or an offline evaluation. For example, even if a member’s token limit is 10,000 tokens, the member’s ending usage might show 11,500 tokens. The member cannot start another playground run or offline evaluation after they exceed their token limit.

When working with multi-row datasets, the evaluation stops mid-run if the token usage is exceeded after a certain row. LaunchDarkly saves all completed rows.

Next steps