How to Run a Fair Split Test Between AI Texting and a Human Texter

You will find a lot of published comparisons between AI texting and human texters, ours included historically, and almost none of them describe how the test was run. A response-rate number with no protocol behind it is not evidence, it is decoration. So here is the protocol instead. Run it on your own leads and you will have an answer that is actually about your operation.

Start with random assignment, and mean it. Split incoming leads by a rule that has nothing to do with the lead, an alternating assignment on arrival order or a hash of the record id, and write the assignment into the CRM at creation time so nobody can quietly move a promising lead to their preferred arm. Assigning by source, by time of day, or by anyone's judgement will hand you a result that reflects the assignment rather than the treatment.

Hold the inputs steady. Same lead sources, same campaigns, same offer, same qualifying questions, same working definition of a booked appointment. If the human texter has a better script than the bot, you have tested scripts. If the bot runs at midnight and the human does not, you have tested coverage. Coverage is a real advantage and worth measuring, but measure it deliberately and report it separately rather than letting it hide inside a headline number.

Decide the sample size and the end date before you start, and do not look at the results and stop on a good day. Small samples in this business swing wildly, because one motivated seller in a batch of two hundred moves every percentage on the page.

Measure four things, not one. Response rate, which is the easiest and least important. Appointments booked, which is what you actually wanted. Appointments held, because a booking nobody shows up to is not an outcome. And fully loaded cost per held appointment, meaning wages and management time on one side, subscription and metered usage on the other.

Then write down what you found, including the sample size, the date range, the assignment rule, and the definitions. If you would not be comfortable publishing the method next to the number, the number is not ready to make a decision on, and that standard applies to every vendor's case study you are handed, ours included.