Short answer: an A/B test sends two versions of one message, at random, to two halves of the same list. It proves nothing if you change several things at once or call the winner too early. With 100 sends per version, a gap of a few points between two rates is still noise.
Why test a message instead of rewriting it by feel?
A message rewritten by feel teaches you nothing. You change three words, the reply rate moves, and you cannot tell whether the text, the list, the day of the week or plain luck did it.
An A/B test removes those explanations. Both versions go out at the same time, to people drawn at random from the same list. If a gap shows up, it comes from the message. It is the only situation where you can say "this sentence works better".
One limit: a test tells you which version wins on your audience, not why. The "why" stays a hypothesis that you test in the next round.
What should you test first?
Test what weighs most on the reply, in this order.
- The closing question of the first message. Open or closed, one question or two.
- The first sentence. A specific fact about the person, or an introduction of yourself.
- The length. Three lines against six.
- The invitation note. With or without a note. Our article on connection requests with or without a note gives a starting point.
Change one element between A and B. If the two versions differ in length, first sentence and question, you get a winner without knowing which of the three made the difference.
How many sends before you can conclude?
A lot more than you would expect. A rate observed on few people jumps around from one sample to the next.
The margin of error of a rate, at the 95% confidence level, is about 1.96 times the square root of (rate × (1 − rate) / number of sends) (Wikipedia, binomial proportion confidence interval, accessed October 8, 2026). The formula becomes unreliable on small samples or when the rate gets close to 0% or 100%.
A worked example, to read as an order of magnitude and not as a measurement:
| Sends per version | Observed rate | Margin of error (approx.) |
|---|---|---|
| 30 | 30% | 16 points |
| 100 | 30% | 9 points |
| 400 | 30% | 4.5 points |
With 100 sends per version, one version at 30% and the other at 36% cannot be told apart. And the gap between two versions is even less certain than each rate taken alone. A useful test will take several weeks, and that is normal.
How do you run a test, step by step?
- Fix the measure before you start. For an invitation note, it is the acceptance rate. For a message, it is the reply rate.
- Start from one list, split at random into two halves. Sending A on Monday and B on Tuesday mixes the message with the day.
- Change one thing between the two versions.
- Wait for the planned number of sends, even if one version takes the lead after three days.
- Keep the winner for new people, then pick the next element to test.
A reply is not always good news. A "no thanks" counts as a reply: if you want to measure real interest, also look at the meetings you booked. Our guide to acceptance and reply rates gives benchmarks to judge your numbers.
Which traps spoil an A/B test?
- Calling it too early. Stopping as soon as one version leads is the surest way to crown a winner that is not one.
- Testing on a mixed list. If A mostly reaches directors and B managers, you are comparing audiences.
- Running too many tests. Ten variants on 200 people produce no readable result.
- Retesting the same people. Someone who saw A should not get B later.
How does it work in Sales Farmer?
A/B tests are part of the Growth plan. On one step of the sequence, an invitation note or a message, you write a variant B. Each person receives A or B, half and half, and always keeps the same version. Results appear on the campaign page, and a verdict only shows once each version has 30 sends, at a 90% confidence level.
The 30-send rule is a floor for showing a verdict, not a guarantee: a result from 30 sends stays fragile. The "Keep the winner automatically" option then sends the clearly better version to new people. On the Starter plan the same method works by hand: two campaigns on two halves of the same list, compared in the campaign analytics.
Automatic mode adds a different kind of test, the openers test (Growth plan, trial included). Out of 20 to 90 people you tick, the AI writes two or three openers with different angles, each from a real fact about the person, and each person gets only one. The "Hook tests" section of the Results page compares them. For each test, it gives each opener's reply rate with its margin of uncertainty, and a chart of the cumulative rate day by day. It also says what share of your first messages from the last 30 days were in a test. No percentage before 20 invitations per opener, no winner on a small gap.
When one opener clearly stands out, the "Keep hook A" button (or B, or C) steers the next messages of the campaign toward its angle. You can go back, and messages already written do not change. A CSV export, with no personal data, pulls out the test numbers.
The limit: this test compares angles, not two sentences you wrote word for word. It is read on LinkedIn, so replies received by email are not counted. Records from before October 7 have no date: a test that contains some has no curve.
FAQ
How many messages do you need for an A/B test on LinkedIn?
At least 100 sends per version to read a gap of more than ten points, and more for a smaller gap. Below 30 sends per version, do not conclude anything.
Can you run an A/B test without a tool?
Yes. Draw your list at random into two halves, send A to the first and B to the second on the same day, then count the acceptances or replies of each group in a spreadsheet.
What should you test first in an outreach message?
The closing question of the first message, then the first sentence. They weigh most on the reply, and each changes in a single line.
Does an A/B test work with 50 prospects?
It only gives an indication. With 25 people per version, only a huge gap is readable. Better to add up several campaigns on the same audience before deciding.