Skip to main content
A/B testing in Naturalead lets you run controlled experiments across your AI agent configurations. Instead of guessing which system prompt, conversation stages, or qualification criteria work best, you can split incoming lead traffic between two or more agent variants and measure which one qualifies more leads, generates better conversations, or achieves higher reply rates. This guide walks you through creating an A/B test, running it, and interpreting the results.

​
How A/B testing works

When an A/B test is running, Naturalead automatically routes new conversations to different agent configurations based on the weights you assign. Each variant gets a proportional share of traffic, and all conversations are tracked independently so you can compare performance metrics side by side.
A/B tests operate at the conversation level. When a new conversation starts with a lead, the system selects a variant based on the configured weights and uses that variant’s agent configuration for the entire conversation.

​
Prerequisites

Before setting up an A/B test, you need:

Multiple Agent Configs

At least two agent configurations with different system prompts, stages, or qualification criteria. Create them via the Bot page or the Agent Config API.

Active Lead Flow

Incoming leads or an active campaign so the test variants receive conversations to compare. A/B tests need sufficient sample size to produce meaningful results.
A/B tests require a meaningful volume of conversations to produce statistically significant results. Running a test with fewer than 50 conversations per variant may lead to unreliable conclusions.

​
Step-by-step walkthrough

​
A/B test lifecycle

An A/B test moves through these statuses: Valid status transitions:
Variants can only be modified while the test is in draft status. Once a test has started running, changing variants would invalidate the results. If you need different variants, create a new test.

​
Tips for effective A/B testing

Change one variable

Modify only one dimension per test (e.g., system prompt tone, number of stages, or qualification criteria). Changing multiple variables makes it impossible to attribute performance differences.

Use equal weights

Start with 50/50 splits for the clearest comparison. Use unequal weights only when you want to limit exposure to an experimental variant.

Wait for significance

Let the test run until you have at least 50-100 conversations per variant. The results endpoint includes a confidence level indicator to help you decide when to conclude.

Apply the winner

Once you identify the best-performing variant, update your campaigns and default agent configuration to use the winning agent config. Then set up a new test to continue optimizing.