The Impact of Feature Management on Software Engineering and Business Performance

An observational analysis of a survey of 500 DevOps and IT professionals

Studies from researchers at DORA and McKinsey show that strong software delivery and operational practices contribute to positive business outcomes. But how does feature management affect software delivery and operational performance? And which types of feature management systems yield the greatest benefits? These are the questions we set out to answer in a recent survey of 500 DevOps and IT professionals. 

In April 2024, LaunchDarkly enlisted Wakefield Research, a third-party firm, to conduct a survey of 250 LaunchDarkly customers and 250 non-customers—with the goal of comparing software engineering performance and business outcomes.

The results show a clear performance gap between customers and non-customers. On average, LaunchDarkly customers:

• Deploy more frequently (84% more)
• Recover faster from incidents (46% more likely to recover within a day)
• Experience better software reliability (48% better)
• Have higher customer satisfaction (95% better)
• Spend less time on pre-production testing and debugging (63% less time at a minimum)
• Deliver software with less overhead

Read on for a detailed breakdown of the survey results across the following categories: innovation velocity, software reliability and customer satisfaction, and release overhead.

Innovation velocity

LaunchDarkly customers deploy code 84% more frequently

On average, LaunchDarkly customers have a much higher deployment velocity than non-customers. According to the survey, 59% of customers deploy several times a week or more frequently, with 20% deploying “on demand.” In contrast, only 32% of non-customers deploy to production several times a week or more frequently. And only 3% deploy “on demand.” 

If we treat software “innovation” in a broad sense, spanning incremental and radical innovations, we can conclude that LaunchDarkly customers have a higher developer innovation velocity than non-customers.

Why is that? First off, LaunchDarkly feature flags provide safeguards and control that, in turn, give development teams the confidence to ship changes faster. LaunchDarkly enables many customers to do trunk-based development and low-risk continuous delivery. 

Customers can also use targeting capabilities to run low-risk early access programs to quickly validate new innovations. Moreover, LaunchDarky’s experimentation features give teams confidence that the innovations they ship will deliver the intended business impact. 

Lastly, customers’ use of feature flags may indicate a wider trend of adopting modern development practices. All of these modern development practices combined contribute to a faster pace of innovation.

Software reliability and customer satisfaction

LaunchDarkly customers outperformed non-customers in multiple categories tied to software reliability and user satisfaction. Customers had a superior mean time to recover (MTTR) from incidents, fewer reliability issues overall, and less negative customer feedback related to unreliable software.

Customers are 46% more likely to recover from an incident in a day or less

86% of LaunchDarkly customers recover from software incidents within a day or less, compared to 59% of non-customers. With LaunchDarkly, individual developers can instantly disable broken code by using feature flags as kill switches. In fact, they can do so at runtime without needing to push a fix through their deployment pipeline. When using Flag Triggers, which leverages integrations with application performance monitoring (APM) and observability tools, customers can remediate software issues automatically.

With the introduction of Release Guardian, customers can natively monitor operational metrics at the feature level and automatically disable features that degrade system performance.

All of these capabilities contribute to the impressive recovery times posted by LaunchDarkly customers.

LaunchDarkly customers have 48% better software reliability

On average, non-customers have less reliable software systems than customers. For 56% of non-customers, their primary software service or application is unavailable, slow, or operating incorrectly often or all the time. The same is true of only 5% of LaunchDarkly customers. 

Similarly, only 26% of non-customers rarely or never experience reliability issues. Whereas, 70% of LaunchDarkly customers rarely or never experience reliability issues. 

The LaunchDarkly platform is equipped with safety nets, controls, and automatic remediation capabilities—all of which contribute to software reliability and availability. It is, therefore, unsurprising to see exceptional levels of operational health among LaunchDarkly customers. 

LaunchDarkly customers report nearly 2x fewer user complaints

LaunchDarkly customers hear far less negative user feedback regarding software reliability than non-customers. 56% of non-customers receive negative user feedback often or all the time, whereas the same is true of only 11% of LaunchDarkly customers. On the flip side, only 23% of non-customers rarely or never get negative feedback. But nearly half (45%) of LaunchDarkly customers rarely or never hear user dissatisfaction with the reliability and operational performance of their software.

In the context of software reliability, LaunchDarkly customers have nearly 2x greater customer satisfaction among their users than non-customers.

The superior MTTR of LaunchDarkly customers may partly explain why they hear fewer user complaints. Another reason may be that they use LaunchDarkly to progressively deliver changes to small subsets of users, including through canary tests in production. In doing so, they shrink the blast radius of errors. Even when a release throws a bug, the impact is confined to a few users. By the time the change has been released to a large audience, the defects have been identified and fixed, resulting in a better experience for the majority of users.

The overhead of software releases

LaunchDarkly customers require far less overhead than non-customers to execute software releases, particularly when it comes to labor costs. What’s more, customers spend significantly less time on non-innovative work such as testing code in pre-production environments and fixing bugs.

Nearly 70% of LaunchDarkly customers (vs. 1% of non-customers) can execute a software release with 5 people or less

LaunchDarkly significantly reduces the number of personnel needed to execute software releases. Indeed, a majority of customers (68%) require only 1-5 people to ship software. Whereas, almost zero non-customers (1%) can say the same. This 6700% difference between customers and non-customers underscores the efficiency gains of LaunchDarkly, which allow teams to dedicate more resources to innovation and less to deployment logistics.

Even when only considering the high hourly wages of software developers and engineering leaders, we can assume that the overhead of releasing software for LaunchDarkly customers is significantly lower than for non-customers.

The LaunchDarkly platform makes releasing software safe and uneventful. If you ship a buggy feature, you can deactivate it in real time with the flip of a toggle. Also, you can decouple deployments from releases. That is, you can wrap a code change in a feature flag, merge it with the mainline, and then allow the mainline to get deployed to production—all without releasing the code change to end users. Later, you or another individual can release that change by toggling on the corresponding flag in LaunchDarkly. 

Such a release pattern enables lightweight software delivery and helps explain why most LaunchDarkly customers need so few people for releases.

LaunchDarkly customers spend 63% less time on pre-production testing and bug fixing (at a minimum)

Nearly three-quarters of LaunchDarkly customers (74%) spend 25% or less of their time on fixing or testing code. In contrast, the same is only true of 30% of non-customers. In other words, 70% of non-customers spend more than 25% of their time on fixing or testing code—i.e., non-innovative work.

Let’s attempt to put this in perspective. For every 100 non-customers, 70 spend at least 26% of their time on non-innovative work. If we assume an average total compensation for 1 FTE software engineer of $150K/year, that’s $2.73M a year spent on undifferentiated, non-innovative drudgery. For LaunchDarkly customers, that figure drops by $1.72M.

LaunchDarkly allows customers to safely test code in production with a small subset of users. Owing to the fail-safes and control mechanisms in LaunchDarkly, teams spend less time vetting features in artificial staging environments. Instead, they rapidly ship changes to production, see how they perform in a live environment, and disable them if needed.

Conclusion

The survey results demonstrate the benefits of using LaunchDarkly's feature management and experimentation platform: customers have a high developer innovation velocity, recover faster, experience greater software reliability, foster high customer satisfaction, and maintain low overhead when releasing and managing software. 

LaunchDarkly is a platform for high-velocity engineering teams to release, monitor, and optimize great software in production. It is associated with high software engineering performance and positive business outcomes. Those who adopt the platform can expect to gain a competitive edge.

To progress on your journey to engineering excellence, request a live demo of the LaunchDarkly platform or start using the product for free.

* * * * * *

About the survey

The LaunchDarkly Survey of Non-Customers was conducted by Wakefield Research (www.wakefieldresearch.com) among 250 DevOps and IT Professionals, between April 17th and May 2nd, 2024, using an email invitation and an online survey. The LaunchDarkly Customer Survey was conducted by Wakefield Research among 250 DevOps and IT Professionals, between April 23rd and May 1st, 2024, using an email invitation distributed by LaunchDarkly.

Respondents spanned multiple industries, company sizes, and annual revenue amounts. They also ranged from individual contributors to C-level executives, including job titles such as Software Developer, VP of Engineering, IT Director, Platform Engineer, CTO, and more. Results of any sample are subject to sampling variation.

Appendix 1 - The inferior performance of those with homegrown feature flag systems

Homegrown feature flag systems associated with worse MTTR

The non-customer segment was divided into those who use a paid feature management service (Service) and those who use a homegrown in-house system (Homegrown). Interestingly, the Homegrown group performed worse than the Service group in their mean time to recover (MTTR) from incidents. 46% of the Homegrown group take a week or longer to recover compared to 38% of the Service group.

In many cases, homegrown feature flags are essentially configuration files. This typically means that, if you need to turn “off” a buggy feature, you have to change an environment variable in a config file and then push the update through your deployment pipeline, causing delays. Moreover, in some cases, homegrown systems are not designed to work efficiently across microservices and multiple platforms (web, mobile, etc.). 

Typically, with a commercial feature flagging service, if you toggle off a flag, it disables every instance of the feature across your microservices and platforms. But with homegrown systems, you may have to disable several instances of a feature manually, slowing remediation.

***Determine whether to build or buy a feature management system.***

Homegrown systems associated with even worse user satisfaction

The survey revealed that 84% of those with homegrown feature flagging systems hear negative user feedback at least sometimes, compared to 71% of those with a Service and only 55% of LaunchDarkly customers. 

The association between homegrown systems and poor user satisfaction parallels that of the former association with slow incident recovery times. Generally speaking, homegrown systems lack the scale, sophistication, and performance of a professional feature management platform. And these deficiencies contribute to (or at least fail to sufficiently mitigate) prolonged outages, higher change failure rates, and other operational issues.

Appendix 2 - How we calculated the statistics

LaunchDarkly customers deploy 84% more frequently than non-customers

-59% of customers deploy several times a week or more frequently

-32 % of non-customers deploy several times a week or more frequently

Relative percentage difference:

(59% - 32%)/32% = .843 * 100 = ~84%

Customers are 46% more likely than non-customers to recover from an incident in a day or less

-86% of customers recover from software incidents within a day or less

-59% of non-customers recover from software incidents within a day or less

Relative percentage difference:

(86%-59%)/59% = .457 * 100 = ~46%

Customers have 48% better software reliability than non-customers

-Reliability issues often or all the time: Non-customers = 56%, Customers = 5%; Difference: 56% - 5% = 51%

-Rarely or never experience reliability issues: Non-customers = 6%; Customers = 70%; Difference: 70% - 26% = 44%

Combined Summary Metric: (51% + 44%)/2 = 47.5% (round up to 48%)

Customers report nearly 2x fewer user complaints than non-customers

-Rarely or never hear negative user feedback: Customers = 45%; non-customers = 23%.

Relative percentage difference:

(45% - 23%)/23 * 100 = 95.6% (~2x)

Customers spend 63% less time on non-innovative work than non-customers (at a minimum)

70% of non-customers (70% of 250 people = 175 people) say they spend more than 25% of their time on fixing and testing code. 26% of customers (65 people) say the same. We’re assuming total working hours of 2080 hours/person/year. When looking at the group of people who spend “more than 25% of their time” on non-innovative work, it means that some of those people may only spend 26% of their time on such work, others may spend 35% of their time, while others still may spend 50% of their time on non-innovative work. For the sake of this calculation, I’m going to use 26%.

Non-Customers

.26 * 2080 = 520 hours/person/year

520 * 175 people = 94,640 hours/year

LD Customers

520 * 65 people = 35,152 hours/year

Final Calculation:

94640 – 35152/94640 = 0.628 (~63%). Thus, at a minimum, LD customers spend 63% less time on non-innovative work.

Sample costs of releasing software based on survey results

For every 100 non-customers, 70 spend at least 26% of their time on non-innovative work. If we assume an average total compensation for 1 FTE software engineer of $150K/year, that’s $2.73M a year spent on undifferentiated, non-innovative drudgery.

(.26 * $150,000) * 70 = $2,730,000

For LaunchDarkly customers, that figure drops by $1.72M.

(.26 * $150,000) * 26 = $1,014,000 → $2,730,000 - $1,014,000 = $1,716,000 ($1.72M)