Experiment traffic assignment

This topic explains how LaunchDarkly assigns contexts to experiment variations and what causes those assignments to change. Understanding traffic assignment helps you predict whether the contexts in your experiment keep receiving the same variation, which affects both the experience of your end users and the reliability of your results.

Seeds and buckets

Every experiment iteration has a seed, which is a value that LaunchDarkly generates when you start the iteration.

To assign a context to a variation, LaunchDarkly combines the iteration’s seed with the context’s key, hashes the result, and maps that hash to one of 100,000 buckets. Each bucket belongs to one variation. Every context that maps to a given bucket receives that variation.

LaunchDarkly uses the key of the context whose kind matches the experiment’s randomization unit. If your experiment randomizes by user, LaunchDarkly uses the user key. If it randomizes by organization, LaunchDarkly uses the organization key, which is why every user in the same organization receives the same variation.

This calculation is deterministic. The same seed and the same context key always produce the same bucket, which means:

• A context receives a consistent variation for as long as the seed stays the same, no matter how many times it evaluates the flag or which SDK evaluates it.
• LaunchDarkly does not need to store a record of which variation each context received. It recalculates the assignment on every evaluation.
• If the seed changes, every context maps to a new bucket, and contexts move between variations.

Tracked and untracked buckets

The audience allocation you set for an experiment determines how many of the 100,000 buckets take part in it. To learn more, read Allocating experiment audiences.

Buckets fall into two groups:

• Tracked buckets are part of your experiment. The contexts in these buckets receive an experiment variation, and LaunchDarkly analyzes their metric events in your results.
• Untracked buckets are not part of your experiment. The contexts in these buckets receive the control variation, and LaunchDarkly excludes them from your results.

For example, if you allocate 30% of a flag rule’s traffic to an experiment with three variations, roughly 10,000 buckets receive each variation and are tracked. The remaining 70,000 buckets are untracked, receive the control variation, and do not appear in your results.

Untracked buckets are what LaunchDarkly draws on when you increase the amount of traffic in your experiment.

When LaunchDarkly reassigns contexts

Contexts keep their variations for as long as an iteration runs. Assignments change only when you start a new iteration, and what happens then depends on the change you made and on whether you allow traffic reshuffling. To learn more about the setting, read Traffic reshuffling.

Traffic increases

When you increase your audience allocation, LaunchDarkly needs more tracked buckets than the previous iteration used. It takes them from the untracked buckets first, which leaves the buckets that were already tracked where they are. This is why the contexts already in your experiment keep their variations when you ramp up.

If there are not enough untracked buckets remaining to cover the increase, LaunchDarkly cannot satisfy the new allocation without moving contexts. If you allow traffic reshuffling, LaunchDarkly generates a new seed for the iteration, which re-maps every context and redistributes them across all of the variations. If you disable traffic reshuffling, LaunchDarkly does not generate a new seed.

Traffic decreases

When you decrease your audience allocation, LaunchDarkly makes the appropriate number of tracked buckets untracked. The contexts in those buckets move out of the experiment, begin receiving the control variation, and no longer appear in your results.

If you then increase your allocation again, LaunchDarkly may assign contexts to different variations than they received before, even if you disabled traffic reshuffling.

Iteration starts and stops

When you start an iteration, LaunchDarkly writes the bucket assignments into your flag’s targeting rule as a percentage rollout and delivers the updated flag to your SDKs. Your SDKs perform the assignment calculation locally or receive the result from LaunchDarkly, depending on which SDK you use. To learn more, read Percentage rollouts.

When you stop an iteration, LaunchDarkly stops tracking its buckets and releases them for future use.

Traffic assignment in layers

Experiments in a layer are mutually exclusive, which means no context can enter more than one of them at a time. LaunchDarkly enforces this through the buckets.

All of the experiments in a layer share one seed, rather than each iteration generating its own. A shared seed means a given context maps to the same bucket in every experiment in the layer. LaunchDarkly can guarantee exclusivity by giving each experiment its own set of buckets. When you start an iteration of an experiment in the layer, LaunchDarkly allocates it a set of buckets matching its reservation amount, chosen from the buckets that no other experiment in the layer is using. Those buckets belong to that experiment until its iteration stops, at which point LaunchDarkly returns them to the layer for other experiments to use.

Within a layer, LaunchDarkly assigns buckets to variations in many small blocks scattered across the layer’s traffic, rather than in one continuous range per variation. For example, instead of assigning buckets 1 through 50,000 to variation A and buckets 50,001 through 100,000 to variation B, LaunchDarkly alternates small blocks of buckets between the two variations across the full range.

Scattering the blocks has two consequences:

• Adding an experiment to a layer that other experiments have already used does not carry over variation assignments from those earlier experiments, because the new experiment distributes its variations differently across the same buckets.
• LaunchDarkly can reshuffle one experiment in a layer by redistributing its blocks, without changing the seed the layer shares. Reshuffling one experiment does not reassign traffic in the other experiments running in that layer.

When you stop an iteration in a layer, LaunchDarkly stores a snapshot of the layer on that iteration. The results for a past iteration continue to reflect the experiments that were running alongside it at the time, even if the layer changes later.

To learn more, read Mutually exclusive experiments.

Traffic assignment in holdouts

A holdout changes which contexts are eligible to enter an experiment, but it does not change how that experiment splits traffic between its variations.

LaunchDarkly assigns contexts to the holdout using the holdout’s randomization unit, which must match the randomization unit of every experiment in the holdout. The percentage of traffic you hold out stays the same for the duration of the holdout, which keeps the held-out group and the rest of your traffic comparable.

When you add an experiment to a holdout, LaunchDarkly adds the holdout as a prerequisite to the experiment’s flag. The prerequisite blocks the contexts in the holdout before they reach the flag’s targeting rules. These contexts never receive an experiment variation and never appear in the experiment’s results. Contexts outside the holdout reach the targeting rules and enter the experiment as usual.

For example, if you hold out 10% of your traffic, and you allocate 50% of a flag rule’s traffic to an experiment with two variations, the experiment measures 45% of your total traffic, split evenly between the two variations.

When a holdout ends, LaunchDarkly removes the prerequisite from the experiment flags, and the contexts that were in the holdout can enter future experiments.

To learn more, read Holdouts.

Traffic assignment in multi-armed bandits

A multi-armed bandit (MAB) changes its traffic assignment while it runs, unlike a standard experiment that holds assignments steady for the length of an iteration.

At the interval you choose, a MAB estimates each variation’s probability of being the best and then reallocates traffic so each variation receives traffic matching that probability. LaunchDarkly applies the new allocation by redistributing which buckets belong to which variation.

MABs always reshuffle traffic when they reallocate, which prevents a MAB from running out of traffic as it shifts allocation toward a winning variation. As a result, a context can receive different variations over the life of a MAB. Consider this before you use a MAB for a change where a shifting experience would confuse your end users.

To learn more, read Multi-armed bandits.

Traffic assignment in stratified sampling

Stratified sampling chooses an iteration’s seed deliberately instead of generating one at random. The variations start out balanced across a covariate.

When you upload a CSV of context keys and covariate values, LaunchDarkly generates a series of candidate seeds. For each one, it simulates the assignment that seed would produce, compares the resulting distribution of covariate values against your CSV, and scores the balance. LaunchDarkly then stores the best-scoring seed on the iteration and uses it when you start the iteration.

After the iteration starts, traffic assignment follows the same rules as any other experiment.

To learn more, read Stratified sampling.