Hireology builds safe, scalable, AI features with LaunchDarkly

Hireology helps take the hassle out of hiring, providing customers with a platform that streamlines the hiring process and delivers the information necessary for hiring the right talent efficiently. They serve a wide variety of industries, including healthcare, retail, and automotive. The platform automates rote hiring tasks including creating job postings, managing job applicant pipelines, and enabling better hiring decisions.

Hireology uses LaunchDarkly to implement safer software delivery practices. “We lean heavily on LaunchDarkly and its capabilities because we know how nimble it makes us,” says Gainous. When the team at Hireology began exploring integrating generative AI into their product in early 2023 with a small proof-of-concept Chrome extension, they recognized its potential to transform how their customers create job descriptions. However, using AI responsibly while maintaining high quality output across job descriptions for different industries with specific requirements and terminology requires careful planning.

"We think that generative AI in particular is a very meaningful thing for Hireology, because the models do a really great job at producing job descriptions, at producing content like email responses and things like that that are very much in the core workflow of what our users need to do," explains Scott Gainous, VP of Engineering at Hireology. “You can imagine the time it takes to pull all of that together, all the skills required, et cetera. To just be able to have a one-click ‘generate this job description,’ tweak the tone, tweak the expressiveness, is a really powerful solution to get more qualified candidates to apply for these jobs.”

The engineering team at Hireology isn’t made up of AI experts, but their confidence in strong engineering fundamentals helped them mitigate challenges along the way. For example, the team quickly discovered that newer versions of AI models don’t necessarily offer better performance. "Execute the same prompt 20 times and get 20 different answers and 10 hallucinations along the way," notes Gainous. "Newer versions of those models have not necessarily translated into better. Some are regressing over time. Some are getting better. Some are just different."

The non-deterministic nature of AI posed unique challenges for quality assurance and deployment. Job descriptions vary widely across industries and roles, and users would likely become frustrated if the platform generated irrelevant job descriptions that required significant edits. The team needed a way to validate how well models were working while confirming that quality was consistent across different industry verticals.

"Everything we do at Hireology, our first thought from an engineering perspective is: when you write code, what's the flag that controls it?" says Gainous. "Whether it's a teeny change, whether it's a massive change, it has to be behind a flag, because your commits are going to go straight to production."

Hireology uses LaunchDarkly as a key part of its AI development strategy to enable safe testing and deployment of features while maintaining high standards for quality. The team developed a comprehensive in-house testing framework that automatically evaluates different models and configurations, and uses LaunchDarkly to serve different versions of model configurations for testing.

Sam Elliott, Staff Quality Assurance Engineer at Hireology, developed an iterative approach to AI quality assurance. "We're using our automation to turn on flags for different test users, and within the different models, we are using temperatures. Each test user is tied to a specific model and temperature, and then we're running a job description bot that generates tests based on specific verticals."

The team uses its in-house AI evaluation tool to evaluate AI with a scoring system that assesses job descriptions based on specific criteria. "We are using AI to score and give quality ranks based on the things that we're expecting: is the job title present, are the locations present?" explains Elliott.

Without LaunchDarkly, “I feel like the process would be very heavy and slow, and there would be a lot of code changes. There would be a lot of code branches,” says Gainous. “LaunchDarkly really helps us streamline and react really fast. So I think it’s providing us a scalable solution to test many models at the same time with relative ease, and then reduce the amount of code we have to manage.”

Hireology also prioritizes responsible AI implementation, ensuring that human oversight remains central to their process. "Our point of view is: always be responsible with AI and make sure that humans are always in the loop," says Gainous. "They're making the decisions, and they're doing it in a way that bots aren't controlling and creating." 

He adds that Hireology wants its customers to have tools that help them reach their business goals: “We think, how can we have more than just great content, but (something) like a supporting team in our software?”

Hireology has enhanced its AI development process, achieving several significant improvements:

• Improved ability to test AI models at scale: LaunchDarkly makes it easier to serve model variations within the team's testing framework. They can now automatically evaluate different models and configurations across various industry verticals, with each test examining specific quality criteria such as proper job title placement and location information.
• Rapid deployment: The team now deploys multiple times per week, with the ability to quickly respond to any issues. Gainous explains, "Now the release process is fully independent from the deployment process. When we're ready to release something, the code is already deployed in production and it's just a matter of flipping flags."
• Enhanced user control and feedback: Hireology implemented a public preview system that gives customers control over adopting new AI features. "We want our customers to get their hands on the software as soon as possible, even if we know it's not feature-complete. We can start to get that continuous feedback," says Gainous. 
  
  This has proven more effective than traditional feature releases. "It's not great to just put new features out… if everything changes on you and you're trying to get your job done, it's jarring,” Gainous explains. “But if I can offer you, 'Hey, this thing's out there,' we remind you occasionally, 'Hey, check this thing out.' You adopt it now. You're more invested."
  
  This opt-in approach remade the way Hireology introduces new features to customers. The team has found that continuous feedback is "as essential as continuous deployment," and that it helps them refine and improve their AI features based on real user experiences.
• Controlled release process: Combining feature flags and opt-in capabilities has allowed Hireology to test AI features quickly, without risking stability. Gainous explains, "Coupling constant flagging changes with all the great rollout strategies that you know and love at LaunchDarkly, plus user-enabled opt-in, has given us a lot of control about how we launched this. Percentage-based rollouts of the software changes, and things like the preview feature enablement, helped inform us a lot and dialed things in as we went."

Hireology continues to innovate in the AI space, with plans for more advanced features like multi-agent AI communication and intelligent sourcing analysis. The team is exploring ways to automate model selection based on performance metrics so they can continue to provide the best-performing AI models for their customers.

"We want to scale AI to all the teams," says Gainous. "Our vision is not to basically build an AI product that we just put on the market, but to build a really intelligent platform that you just use and feel."

With LaunchDarkly as a key component of its development infrastructure, Hireology continues to lead the way in developing AI-powered hiring solutions while keeping its commitment to quality and responsible AI implementation.