As a software engineer, you probably have lots of ideas about how to improve the products you work on. How can you convince other people your proposed changes have merit? A/B experiments are a way of gathering data about the impact of changes to your product.
Often, experiments are run by product managers. But maybe your product manager is too busy, or you work at a small company and need to wear a lot of hats, or you work on a product team such as Growth whose mandate is to quickly iterate and test lots of ideas. In those cases, running your own experiments can be a valuable skill.
In this tutorial you’ll learn how to run an A/B experiment to give your signup flow some 🍁seasonally themed flavor ☠️, using LaunchDarkly and ExpressJS.
Our example app is a costume shop. All experiments should start with a hypothesis. Will changing the design to be a bit spooookier increase our conversion rate? Let’s find out!
Prerequisites
- A free LaunchDarkly account - sign up for one here
- A developer environment with git, npm, and Node.js installed
Getting started with ExpressJS and LaunchDarkly
Some configuration is required before we get spooky. Clone the example repository. Open the project folder in your editor of choice.
git clone https://github.com/annthurium/expressjs-launchdarkly-signupIn the LaunchDarkly app, select the Project you’d like to work in or create a new one. Click on the dropdown with three dots next to the Production environment. Select “Copy SDK key.”
Paste the SDK key into the .env.example file. Rename the file to “.env” and save it. This step protects you from accidentally committing credentials to source control.
Run these commands to install dependencies and start the server:
cd expressjs-launchdarkly-signup
npm install
npm startGo to http://localhost:3000/signup.html in your browser. Behold the storefront.
Pretty cool, but it could be better.
Load http://localhost:3000/signup-2.html in your browser. Try not to scream with terror or delight.
Let’s add some code to wrap these signup flows in an A/B experiment.
Building a LaunchDarkly experiment
To run a LaunchDarkly A/B experiment, you need to understand the following concepts:
- Metrics are the user action we are trying to measure, which help us validate our hypothesis. In this case, we’re counting conversions.
- Flags enable you to conditionally change your app’s behavior without deploying or modifying the code. Flags are used in this demo to determine which experiment variation is shown to the end user.
Experiments connect flags and metrics to measure end-user / audience behavior.
Creating a metric
Head over to the LaunchDarkly app. Click in “Metrics” on the left hand menu. Click either of the “Create” buttons. Select “Create metric” from the dropdown menu.
Create a metric using the following configuration:
- Event kind: custom
- Event key: signup-conversion
- Occurrence (conversion:binary)
- Metric name: signup-conversion
- Metric key: signup-conversion
- Description: When the user inputs their credentials and clicks the submit button to sign up for a new account.
Creating an experiment
This metric isn’t connected to anything yet. Click “Experiments” in the left navigation menu, and click “Create experiment” on the following screen.
Configure your experiment as follows, then click Next:
- Name: spooky styling
- Hypothesis: Adding seasonal themed CSS will increase signup conversions.
- Experiment type: Feature change
In the subsequent section, choose these values:
- Randomization unit: user
- Attributes: leave this blank
From the metrics dropdown, select “signup-conversion.” Click Next, then click “Create flag.”
Creating a flag
Create a flag with the following configuration:
- Name: show-seasonal-css
- Description: Show seasonally themed CSS on the login page
- Flag type: String
- Variations: Control, Spooky CSS
After creating your flag, you’ll be taken back to the experiment creation flow. Now it’s time to set the audience. Select 100% as the audience size. In a real-life situation, you’d probably select a smaller percentage first, increasing size as you build confidence that the new variation doesn’t introduce bugs or regressions. Just for the purposes of this tutorial, setting the audience size to 100% will demonstrate the end results more quickly.
When you set the audience amount to 100%, each variation will automatically be served 50%. Your configuration should look like this:
Click Finish. You’ll be prompted to click “Open flag targeting” and turn the flag on.
Re flag rules: when targeting is On, serve Control – the Experiment configuration will determine which variants are shown to users.
Click “Review and save.” If your LaunchDarkly configuration requires confirmation, write a comment explaining these changes, such as “start experiment" or even "GETTING SPOOKY UP IN HERE 🩻." Click “Save changes.”
Finally, we’ll need to start the experiment. Click on “Experiments” in the left navigation menu, and then select “spooky styling.” Click “Start.” Tada! 👻
Adding LaunchDarkly event tracking and flag evaluation to the ExpressJS routing layer
Roll back over to your code editor. Open index.js. You can copy and replace all the code in the file with this, and check out the commented lines to see what has changed.
const express = require("express");
const path = require("path");
const serveStatic = require("serve-static");
const bodyParser = require("body-parser");
const LaunchDarkly = require("@launchdarkly/node-server-sdk");
// add express-session middleware to generate session IDs
const session = require("express-session");
require("dotenv").config();
const app = express();
app.use(serveStatic(path.join(__dirname, "public")));
app.use(bodyParser.urlencoded({ extended: false }));
// use the session function to generate session IDs for each request
app.use(
session({
secret: "session",
resave: false,
saveUninitialized: true,
cookie: { secure: false },
})
);
// add a GET function that redirects the root route
// depending on which experiment group the user falls into
app.get("/", async (req, res) => {
const context = {
kind: "user",
anonymous: true,
key: req.sessionID,
};
// Evaluate LaunchDarkly flag
const showSeasonalStyling = await ldClient.variation(
"show-seasonal-css",
context,
false
);
// redirect to a signup form with different styling based on flag value
let url;
if (showSeasonalStyling === "Spooky CSS") {
url = "signup-2.html";
} else {
url = "signup.html";
}
res.redirect(url);
});
// Initialize the LaunchDarkly client
const ldClient = LaunchDarkly.init(process.env.LAUNCHDARKLY_SDK_KEY);
// add a POST method to be called when the user finishes the signup flow
app.post("/login", async (req, res) => {
const context = {
kind: "user",
anonymous: true,
key: req.sessionID,
};
console.log("Session ID: ", req.sessionID);
// track the LaunchDarkly event
await ldClient.track("signup-conversion", context, 1);
res.send(
`Thanks for signing up! Look for the confirmation email in your inbox: ${req.body.email}`
);
});
// Add the waitForInitialization function to ensure the client is ready before starting the server
const timeoutInSeconds = 5;
ldClient.waitForInitialization({ timeout: timeoutInSeconds }).then(() => {
const port = 3000;
const server = app.listen(port, function (err) {
if (err) console.log("Error in server setup");
console.log(`Server listening on http://localhost:${port}`);
});
});
// Add the following new function to gracefully close the connection to the LaunchDarkly server.
process.on("SIGTERM", () => {
debug("SIGTERM signal received: closing HTTP server");
ld.close();
server.close(() => {
debug("HTTP server closed");
ldClient.close();
});
});Navigate to http://localhost:3000. You will see one of the signup variations. Go through the signup flow by putting in your email address, a password, and clicking “Submit.” Open a private browser and do the same. You have a 50/50 chance of seeing a different variation. 😉 Regardless, you should have a different session ID for each browser in the server logs.
Server listening on http://localhost:3000
Session ID: 5hLlQ1WnPyzxtoBb8z63ghxAROFYegsL
Session ID: fDpvOJbp_KdY8IIwjRQdJEBH6meqqCEgMetrics are visible in the LaunchDarkly app. If you click on “Metrics” in the left navigation menu, the same session IDs we saw in the logs appear here:
Once there is enough data, you can see how your experiment is performing:
So far, spooky CSS is winning. 🕸️ 🎃 We’d need a lot more data to confidently validate our hypothesis though. If you need a refresher on statistics, LaunchDarkly’s new sample size calculator can help you estimate sample size and duration.
Conclusion: Running experiments with LaunchDarkly to test the impact of seasonal CSS on signup conversions
If you’ve been following along, you’ve added a LaunchDarkly experiment to your web app that tests two versions of a signup flow to see if one converts at a higher rate.
This code demonstrates a simplified version of a signup experiment. There is so much more you can do with LaunchDarkly’s experimentation platform. For example, you can test more than two variations (multivariate experiments). You can use funnels to track how many steps a user has taken towards a goal.
The following posts could further assist you on your journey:
- A (developer)’s guide to experimentation in LaunchDarkly
- Running your first A/B Test in LaunchDarkly with JavaScript and Next.js
- Getting started with funnel experiments
Go forth and run experiments so you can argue with your boss or product manager, now that your opinions are backed up with data.
Thanks for reading! If you have any questions, or want to tell me what your favorite spooky emoji is, you can reach me via email (tthurium@launchdarkly.com), X/Twitter, Mastodon, Discord or LinkedIn.
