Creating experiments

This topic explains how to create and configure an experiment in LaunchDarkly. Creating an experiment requires several steps:

1. Deciding on the kind of experiment to build
2. Creating the flag or AgentControl config and its variations
3. Creating the right type of metric for your experiment
4. Building the experiment
5. Starting an iteration

These steps are explained in detail below.

​

Prerequisites

Before you build an experiment, you must complete the following prerequisites.

​

Decide on the kind of experiment to build

The kind of experiment you build is determined by the type of metric you attach to the experiment. You can use LaunchDarkly hosted metrics or metrics from an external warehouse you specify. You can run A/B experiments and A/A tests using any metric or standard metric group. You can also run A/B experiments using funnel metric groups or warehouse native metrics. You can also choose to use Data Export to export data from your experiments to a third-party analysis tool, which doesn’t require you to select any metrics when you create the experiment.

​

Create a metric

After you decide on an experiment type, you can create a new metric or reuse an existing metric in your experiment. To learn how, read Metrics. LaunchDarkly automatically creates metrics from AI SDKs, which you can use in experiments on configs. To learn more, read AgentControl config autogenerated metrics.

​

Create a flag or config

Before you begin an experiment, create a flag or config with the variations you plan to test. Configs are on by default, but flags are not. You do not need to toggle on the flag to create the experiment, but you must toggle it on before you start an experiment iteration. When you create the experiment, you will select this flag and one of its rules to run the experiment on. The context kind that your flag rule targets should match the randomization unit of your experiment. We recommend running the experiment on a rule that targets a subsection of your contexts rather than the default rule. This helps ensure consistent experiment results. To learn more, read Creating new flags, Creating flag variations, Create configs, and Create and manage config variations. If you are creating a warehouse native experiment using custom assignment sources, you do not need to choose a flag or config.

​

Limitations

You cannot run an experiment on a flag or config if:

• any of its rules have an active guarded rollout
• any of its rules have an active progressive rollout
• any of its rules are in a running Data Export experiment
• any of its rules are in a running experiment with warehouse native metrics
• the flag is a migration flag

You can build and run multiple experiments on the same flag or config as long as there is only one running experiment per rule. You cannot run multiple experiments on the same rule at the same time.

​

Build the experiment

The following procedures describe how to complete each section of the experiment builder:

• Name and hypothesis
• Variations and audience targeting
• Audience allocation
• Variations split
• Metrics
• Statistical approach and success criteria
• Advanced options

To begin building an experiment:

1. Click Create and choose Experiment. The “Create experiment” dialog appears.

2. Enter an experiment Name.
3. (Optional) Enter a Hypothesis. If you do not enter a hypothesis now, you must add one before you can save your experiment design.
4. (Optional) Click the tags icon to add new or existing tags to the experiment.
5. Click Create experiment. The experiment Design tab appears.

Complete each section of the builder following the procedures below.

​

Name and hypothesis

Edit the name and hypothesis as needed. If your LaunchDarkly account includes AI assist, you can use it to write your hypothesis.

​

Variations and audience targeting

To choose a flag or config for the experiment:

1. Choose an Assignment method:
   • Choose LaunchDarkly flag or config to run your experiment on a flag or config.
   • Alternatively, if you are building a warehouse native experiment using a custom assignment source, choose Warehouse assignment source and expand the procedure below.

Expand assignment source procedure

1. Choose your data source from the Assignment source menu.
2. Enter your Experiment ID.
3. Choose the context kind you want to analyze results by from the Randomize by menu. This must match the randomization unit your metrics use.
4. Enter the Variants you want to experiment on.
   • (Optional) Enter a Display name for each variant to give it a friendlier label in your results.
   • Select which variant is the Control.
5. Enter the percentage of traffic you expect each variant to receive in the Expected split section, or click Split equally.

2. Choose a Flag or config to use in the experiment.
   • (Optional) If you want to create a new flag or config, click Create flag or Create config.
3. (Optional) Check the Run as A/A test checkbox if you plan to compare two different, but identical, variations. You must define these variations in your codebase in the exact same way.
4. Choose a Targeting rule for the experiment audience.
   • If you want to restrict your experiment audience to only contexts with certain attributes, create a targeting rule on the flag or config you include in the experiment and run the experiment on that rule. The context kind that the flag rule targets should match the randomization unit of your experiment.
   • If the flag or config doesn’t have any targeting rules, the default rule is the only option.

5. Choose a context kind to Randomize by.

​

Audience allocation

To determine the audience for the experiment:

1. (Optional) If you want to exclude contexts in this experiment from other experiments you specify, click Add experiment to mutual exclusion layer and select a layer.

Expand layer options

A layer is a set of experiments that cannot share traffic with each other. All of the experiments within a layer are mutually exclusive, which means that if a context is included in one experiment, LaunchDarkly will exclude it from any other experiments in the same layer. All of the experiments within a layer must randomize by the same context kind, such as “user.”To add the experiment to an existing layer:

1. Click Select layer.
2. Search for and choose the layer you want to add the experiment to.
3. Enter a Reservation amount. This is the percentage of the contexts within this layer you want LaunchDarkly to include in this experiment.
4. Click Save layer.

If you need to create a new layer:

1. Click Create layer.
2. Add a Name and Description.
3. Click Create layer.
4. Enter a Reservation amount. This is the percentage of the contexts within this layer you want LaunchDarkly to include in this experiment.
5. Click Save layer.

2. Choose the Variation served to contexts outside this experiment. Contexts that match the selected targeting rule but are not in the experiment will receive this variation.
3. Select the percentage of contexts that match the experiment’s targeting rule to include in the experiment.
4. (Optional) Uncheck Disable reshuffling to allow traffic reshuffling. To learn more, read Traffic reshuffling.
5. (Optional) Click Apply stratified sampling to reduce covariate imbalance. Expand the procedure below for instructions on how to add covariate values.

Expand stratified sampling instructions

You can use stratified sampling to re-randomize the contexts in your experiment treatments. Re-randomization helps eliminate covariate imbalance and reduce false positives and false negatives in your experiment results.Here’s how:

1. Prepare a CSV file containing relevant context keys and covariate values. To learn how, read Stratified sampling in LaunchDarkly experiments.
2. Click Upload CSV. An “Upload CSV” dialog appears.
3. Click Select CSV and choose your CSV file.
4. Click Upload file.

Avoid PII in CSV filesLaunchDarkly stores the information you upload in your CSV files. For this reason, we recommend that you do not upload any personally identifying information (PII). If you have PII in LaunchDarkly that you want to delete, contact Support.

​

Variations split

To assign groups of contexts to variations:

1. (Optional) Click Edit to update the variation split for contexts that are in the experiment. The “Variation split” dialog appears.
   • Click Split equally to evenly divide contexts between variations, or use the input fields to assign a percentage of contexts to each variation.
   • Click Save audience split.
2. (Optional) Click the image icon next to a variation to upload an image associated with that variation. The “Upload image” dialog appears.
   • Click Upload image next to a variation you want to associate an image with.
   • Select a PNG, JPEG, or WebP file from your machine.
   • Click Save.
3. Select a variation to serve as the Control.

​

Metrics

If this experiment is a Data Export experiment, click the Data Export only checkbox. Data Export experiments do not require a metric or statistical approach, and don’t allow attribute selection. To add metrics to the experiment:

1. Select a Metric source.
   • If you are creating an experiment using warehouse native metrics, select your external warehouse. Otherwise, select “LaunchDarkly.”
2. Select one or more Metrics or metric groups.
   • (Optional) If you want to create a new metric, click Create to create and use a new metric or new metric group.
   • You can add only one funnel metric group per experiment.
3. (Optional) If you have added multiple metrics or a standard metric group and want to change the primary metric, click the metric’s Set as primary radio button.

4. (Optional) For Snowflake native experiments that use frequentist statistics, use the Analyze by menu to choose the context kind to use for analyzing each metric.

Expand metric analysis unit options

By default, each metric’s analysis unit matches the context kind of the experiment’s randomization unit. If you choose a different metric analysis unit, you must ensure that the selected context kind belongs to exactly one randomization unit. To learn more, read Randomization units and metric analysis units.

Analysis unit selection is not available for metric groupsThe Analyze by menu appears only for individual Snowflake metrics that you add to an experiment. Snowflake metrics that were added as part of a metric group always use the experiment’s randomization unit for analysis.

​

Statistical approach and success criteria

To choose a statistical method of analysis for the experiment:

1. Select a Statistical approach of Bayesian or frequentist. Expand the section below to read about the options for each.

Expand statistical approach options

You can select a statistical approach of Bayesian or Frequentist. Each approach includes one or more analysis options.We recommend Bayesian when you have a small sample size of less than a thousand contexts, and we recommend Frequentist when you have a larger sample size of a thousand or more. Frequentist is the default statistical approach in LaunchDarkly.The Bayesian options include:

1. Select a preset or custom success Threshold:
   • 90% probability to beat control is the standard success threshold, but you can raise the threshold to 95% or 99% if you want to be more confident in your experiment results.
   • You can lower the threshold to less than 90% using the Custom option. We recommend a lower threshold only when you are experimenting on non-critical parts of your app and are less concerned with determining a clear winning variation.
2. (Optional) Check Apply CUPED to all applicable metrics if you want to reduce variance in your experiment results.
3. (Optional) Check Use uninformative priors for treatments if you do not want to incorporate prior belief in your analysis.

The frequentist options include:

1. (Optional) Check Enable sequential testing: Sequential testing lets you act on the results of your experiment at any time, instead of waiting for a specific sample size. To learn more, read Fixed-horizon versus sequential.
2. Select a Significance level:
   • 0.05 p-value is the standard significance level, but you can lower the level to 0.01 or raise the level to 0.10, depending on whether you need to be more or less confident in your results. A lower significance level means that you can be more confident in your winning variation.
   • You can raise the significance level to more than 0.10 using the Custom option. We recommend a higher significance level only when you are experimenting on non-critical parts of your app and are less concerned with determining a clear winning variation.
3. Choose a Direction of hypothesis test:
   • Two-sided: We recommend two-sided when you’re unsure about whether the difference between the control and the treatment variations will be negative or positive, and want to look for indications of statistical significance in both directions.
   • One-sided: We recommend one-sided when you feel confident that the difference between the control and treatment variations will be either negative or positive, and want to look for indications of statistical significance only in one direction.
4. (Optional) Select a Multiple comparisons correction (MCC) option of Bonferroni or Benjamini-Hochberg. Then choose one of the following:
   • Apply across treatments to correct for additional comparisons from multiple treatments.
   • Apply across metrics to correct for additional comparisons across multiple metrics.
   • Apply across both metrics and treatments to correct for additional comparisons from multiple metrics and multiple treatments.
5. (Optional) Check Apply CUPED to all applicable metrics if you want to reduce variance in your experiment results.

To learn more, read Bayesian versus frequentist statistics.

​

Advanced

Advanced options include:

1. (Optional) If you want to include the experiment in a holdout, scroll to the “Advanced” section and select a Holdout name.

2. (Optional) If you want to be able to filter your experiment results by attribute, scroll to the “Advanced” section and select up to 20 context Attributes to filter results by.
3. Scroll to the top of the page and click Save.

If needed, you can save your in-progress experiment design to finish later. To save your design, click Save at the top of the creation screen. LaunchDarkly saves your in-progress experiment design and displays it on the Experiments list. To finish building the experiment, click the experiment’s name and continue editing. After you have created your experiment, the next step is to toggle on the flag. Configs are on by default. Then, you can start an iteration.

​

Start an experiment iteration

After you create an experiment and toggle on the flag, you can start an experiment iteration in one or more environments. To learn how, read Starting and stopping experiment iterations. Experiment iterations allow you to record experiments in individual blocks of time. To ensure accurate experiment results, when you make changes that impact an experiment, LaunchDarkly starts a new iteration of the experiment.