You manage email and SMS for three to ten ecommerce brands. Every month someone suggests a new loyalty tier, another win-back flow, a quiz, an upsell popup, or a change to the loyalty programs. The list grows. Your sends stay the same. The question is not what you could test. It is which test moves the most revenue for the customers you already have, and the retention experiments ecommerce brands need most are usually hiding in the order export, not in a swipe file. This guide is for email and SMS managers and retention agencies who need a testing roadmap they can defend to a client.
Key takeaways
-
A retention experiment is a controlled change meant to increase repeat purchases from existing customers: a reorder reminder, a cross selling block, a win-back email, or a different first-purchase offer.
-
The backlog of ideas is not the bottleneck. Ranking the ideas by the revenue at stake in the order data is, and you can size each one as eligible customers times their typical order value.
-
Four test ideas hide in most order exports: reorder windows with no reminder, strong product pairs with no cross sell, overdue high-value customers nobody contacted, and first products that rarely lead to a second order.
-
A worked example with round numbers ranks four candidate experiments with nothing more than customers reached times order value.
-
A clean test needs a hold-out group, a single change, and a window at least as long as the natural reorder gap.
What counts as a retention experiment in ecommerce?
A retention experiment is any controlled change that aims to increase customer retention, repeat purchases, or customer lifetime value among existing customers, with customer lifetime value meaning the total revenue expected from a customer relationship. In ecommerce it usually means changing the timing, the content, or the offer between orders to better match what customers need, which matters because most customers expect a store to anticipate their needs and to treat them as individuals. For ecommerce email and SMS managers running one or more brands, and for retention or CRO agency practitioners who need a testing roadmap they can defend, the useful version is not theory but a clean test built from order history.
Concrete examples:
-
Sending a reorder reminder for pet food at day 35 instead of day 60.
-
Adding a cross selling block for coffee filters to the post-purchase email after someone buys coffee beans.
-
Testing a personalized offer for lapsed skincare customers who have not ordered in 90 days.
-
Changing the first-purchase sampler pack for dtc brands to a product with better second-order behavior.
Personalized follow-up and post-purchase content can reduce buyer's remorse, improve repurchase rates, and increase customer lifetime value.
This is different from acquisition tests, which focus on new customers visiting for the first time, such as a shorter checkout to reduce cart abandonment. It is also different from random tweaks that lack a hypothesis or a hold-out group. Repeat purchase rate measures the share of customers who come back within a timeframe, and a retention experiment is a deliberate attempt to move it. A repeat purchase rate where only a minority of customers come back can still mean strong retention for a store that sells durable goods, so judge it against your own history rather than another store's. The frame is always: for this set of existing customers, if we make one specific change, we expect more second or third orders within a defined window. In the sections that follow, the focus is on finding test ideas in order exports, such as reorder windows and product pairs, estimating revenue impact, running clean hold-out tests, coordinating overlapping experiments, and ranking what to launch first. Retention experiments grounded in your own orders are how brands and agencies stop guessing, prioritize the work most likely to lift repeat revenue, and build stronger customer loyalty over time.
Why rank by revenue at stake?
Because the real problem is not a shortage of ideas. It is that you do not know which idea has the most revenue behind it. You have order history. You can see your repeat purchase rate and revenue. What you cannot see at a glance is which specific products or weeks decide whether a first-time buyer becomes a loyal customer over the long term.
Retaining customers is usually cheaper than acquiring new customers, because the relationship already exists, which is why the order history of current customers deserves the same attention as the ad account. Companies that treat customer retention as a side project tend to run whichever test was suggested last, and their retention rates drift with the calendar. Companies with a retention strategy built on the export run the test with the most money behind it first, and the benefits compound: each result feeds the next ranking, and the business learns something it can reuse. Creating that ranked list is the whole retention strategy for the quarter; everything else is execution.
This article uses your own order export to find and rank a small number of experiments tied directly to customer lifetime value, for brands in supplements, coffee, pet food, skincare, and cleaning.

How do I find what to test next from order data?
The retention experiments ecommerce order data supports best show up as four patterns in the export. You find them by reading the export, not by copying generic tactics.
Required input: a CSV export of orders with at least these columns:
-
Order ID
-
Order date
-
Customer ID or email
-
Product ID or name
-
Quantity or line total
-
Order total value
This export can come from Shopify, WooCommerce, or any internal system. From it, you can count four sources of test ideas:
-
Customers due to reorder with no reminder
-
Strong product pairs that lack any cross selling
-
Overdue high-value customers who have quietly stopped buying
-
First products that rarely earn a second order
Each of these can be counted directly from the data, which gives a first-pass estimate of revenue at stake before any design work starts.
Source 1: customers due to reorder with no reminder
These are customers whose typical reorder window has arrived but who did not get any targeted reminder.
-
Filter your export to customers who bought the same replenishable product at least twice. Sort by customer and date, then subtract the previous order date from the current one to get days between orders per product.
-
Calculate the median gap for each product. Dog food A might come back every 32 days. Coffee blend B might come back every 17 days.
-
Now filter current customers who bought that product once, are past the median gap, and have not placed another order. These are your due-to-reorder audience.
-
Check your existing lifecycle flows. Is there an email or SMS triggered near that reorder window? If not, that gap is the experiment.
The timing of the reminder decides how much of this audience you retain. If you send a generic reminder at day 60 but the actual reorder cadence is day 32, you are contacting people nearly a month after they already bought from a competitor or forgot. The replenishment email timing guide covers how to set the send day from the gap.
Source 2: strong product pairs with no cross selling
Here you look for product pairs already bought together often, but with no systematic cross selling in emails, post-purchase flows, or onsite merchandising.
-
Scan your export for orders containing multiple products. Count how often each pair appears in the same order, and how often each product appears in any order at all.
-
Identify pairs where the chance that B is bought when A is in the basket is high: skincare serum plus moisturizer, cat litter plus odor neutralizer, coffee pods plus descaler.
-
If the post-purchase email after coffee pods does not mention the descaler, the experiment is to add a cross selling block and measure extra orders of the second product within 30 days.
This is market basket analysis in practice. Recommendations built on what customers purchase together tend to earn more repeat purchases than generic bestsellers, and a thank-you note or a small gift next to the cross sell suggestion makes the message feel like service rather than a sales push. It also makes sense to read the pair both ways: a strong same-order pair often works as a cross sell in either direction, so the descaler buyer can be the audience for coffee pods too.
Source 3: overdue high-value customers nobody contacted
These are customers with historically high spend who are late for their next purchase and have not received any targeted win-back communication.
-
From your export, calculate recency (days since last order), frequency (number of orders), and total spend per customer. Sort into simple buckets: Champions (high on all three), At-Risk (high spend but growing recency), Hibernating (long gone). This is RFM segmentation.
-
Set a clear overdue rule, for example 1.5 times a customer's usual order gap. Flag high-spend customers who cross that line without another purchase.
-
Count them. Note their historical average order value.
Churn in ecommerce is quiet: nobody cancels, they just stop ordering. Churn rate is the share of active customers who stop buying in a period, and even a modest monthly rate empties a customer base faster than most teams expect. Reading the overdue list every month is the feedback loop that catches churn signals before these customers are gone for good, and a support ticket or a complaint in the same period is often the reason. Different groups of customers respond to different incentives, so a win-back for Champions who went quiet should not be the same message as the one for a one-time buyer. A personalized email that names what the customer used to buy usually reactivates more lapsed customers, and lifts repurchase more, than a generic discount blast. Segmenting lapsed customers before you write the email is itself a retention strategy, and the other benefits of the segmentation, such as knowing which customers to protect from discounting, come along for nothing. An effective retention strategy reduces churn by finding the friction points that preceded the silence, such as a late delivery or a recipe change, and a loyalty program can turn an occasional buyer into an advocate once the incentive matches the segment.
Source 4: first products that rarely earn a second order
These are the first-purchased products that fail to turn new customers into returning customers.
-
Label each customer's first order. Note the primary product or category.
-
For each first product, compute the share of customers who ever place a second order and how many days it typically takes.
-
Focus on products with many first-time purchases but a low follow-up rate compared with the store average. A deep-discount sampler that never leads to a full-size purchase, or a niche coffee flavor that attracts one-time buyers only, are typical weak gateway products.
Test ideas: change the acquisition offer, adjust onboarding emails for buyers of that product, or steer paid traffic toward products with better second-order performance. Customer lifetime value is set largely at the first order, and a better gateway product at first touch can improve long term loyalty much more than a small boost in first-order conversion rate. Customers expect the follow-up to reflect what they bought, so one generic post-purchase flow for every first product misses the mark.
How do I estimate how much revenue a test could move before running it?
Multiply how many customers will see the change by their typical order value. That gives a rough figure for the revenue at stake, which is enough to rank tests.
-
Count eligible customers from each source.
-
Note their average order value from the export.
-
Multiply to get revenue at stake per experiment.
Ranking tests by customers reached times average order value gives a clear order of importance. An experiment that reaches several times as many customers at a somewhat lower order value usually has more at stake than a small, high-value audience, and the formula makes that visible before anyone builds an email. Agency practitioners can use the same sizing to justify the order of work to clients, and later to demonstrate which experiment produced the largest uplift. Cohort retention analysis, which follows groups of customers acquired in the same month, can sharpen the estimate once the first results are in, and it is the fairest way to show a client that retention rates moved because of the test rather than because of the season. Judge the success of the program on customer lifetime value and average order value alongside repeat purchase rates, not on a single open rate.
Worked example: ranking four retention experiments by revenue at stake
Illustration with round numbers, not a benchmark.
Imagine a dtc brand selling coffee beans, filters, tea, and a first-purchase sampler. Four candidate tests:
| Experiment | Customers reached | Typical order value | Revenue at stake |
|---|---|---|---|
| Follow-up for sampler buyers with a low second-order rate | 2,000 | 25 | 50,000 |
| Reorder reminder for overdue coffee bean buyers | 1,200 | 35 | 42,000 |
| Cross selling filters after coffee beans | 800 | 28 | 22,400 |
| Win-back for high-spend tea customers | 300 | 60 | 18,000 |
Ranking: sampler follow-up first, then the reorder reminder, then cross selling, then the win-back. The win-back has the richest customers and still comes last, because there are few of them. You can plug your own order export into the same formula to build your roadmap. If you want two more examples, use the same quick math on overdue pet food buyers and first-time skincare buyers to see which test moves repeat purchases sooner.
How do I run a retention test cleanly with a hold-out?
Select the audience, set aside a hold-out group that does not see the change, apply one change only, and wait at least one natural reorder window before comparing orders.
-
Audience: define criteria from your source. For a reorder reminder test: all customers who bought product A 25 to 35 days ago and have not reordered.
-
Hold-out: randomly set aside a clear minority, for example every fifth eligible customer sorted by customer ID, who will not receive the new message. Tag these in your spreadsheet or email tool.
-
Single change: if testing a new reorder reminder, do not also change the discount level or email frequency. A test that changes two things at once cannot tell you which one worked.
-
Window: if the median reorder gap is 30 days, monitor for at least 30 to 45 days before reading results. Compare what share of the test versus hold-out group ordered and what revenue they generated.
The readout is two numbers per group: the share who ordered, and the revenue per customer. The difference between test and hold-out is the effect of the change on repeat purchases, and nothing else. Success is a difference you would bet on again, not a difference that would vanish with a different week; when in doubt, let the window run one more cycle. The best readouts pair the numbers with customer feedback: a short survey of the test group tells you why the reminder worked, which the order data alone cannot.
How many retention experiments should run at once?
One to three concurrent retention experiments per brand, each on a different audience, so the results do not overlap.
Tests run safely in parallel when they focus on distinct segments: one test on first-purchase coffee buyers and another on long term skincare subscribers, as long as a single customer is not in both audiences. Overlapping experiments on the same segment, such as two different win-back offers for the same overdue customers, create noise and make it hard to communicate which change caused any improvement. Prioritize by expected revenue at stake and work through the top experiments in order. Fewer, better-defined retention experiments still increase retention rates and customer lifetime value over the long term, and they keep the test log readable for whoever runs the program in future. Creating one shared calendar of live tests per brand is how companies avoid double-booking a segment.
Doing this without the spreadsheet
Everything above works in a spreadsheet with an order CSV: reorder gaps from sorted order dates, product pairs from orders with two or more line items, overdue high-value customers from spend, order count and last order date per customer. It gets tedious across several brands every month.
Affinsy reads a store's order history from a CSV upload, or orders sent by webhook or API, and runs market basket analysis and RFM customer segmentation on it: which products act as openers, gateways and retainers, which pairs sell together or cannibalize each other, and which customer segments are Champions, At-Risk or Hibernating. From a rule or a reorder window you can build a ranked customer list, export it, and load it into your email tool yourself, with the hold-out split done in the sheet before the send.
Next steps
-
Export your orders this week. Compute the basics: reorder gaps per product, product pairs, recency, frequency and spend per customer.
-
List candidate tests from the four sources and estimate revenue at stake for each.
-
Pick one to three experiments to run in the next 30 to 60 days.
-
Frame each experiment as a simple question: if we add a reorder reminder at day 35 for pet food, how many more current customers place a next purchase by day 60 compared with those who see nothing?
-
Document each test: audience, hold-out, change, start and end dates, result. Your retention program becomes a repeatable system rather than scattered tactics.
Disciplined retention experiments compound into stronger customer loyalty, higher retention rates, and increased lifetime value over time, which is the kind of growth a business can plan around. If you would rather have the reorder windows, the product pairs and the overdue high-value customers worked out from your own orders, the 7-day trial of Affinsy delivers them.
FAQ
How small can my dataset be for meaningful retention experiments?
Meaningful tests usually need at least a few hundred eligible customers per experiment. Smaller brands can still run directional tests with fewer, as long as they accept more noise. Consider batching two or three months of first orders before running a first-product experiment to increase your audience size. For very small stores, focus on obvious gaps like missing reorder reminders rather than fine-grained segmentation, and track simple counts of repeat orders and total revenue between test and hold-out groups. Positive reviews on product pages help the first order; the experiments in this article start after it.
How do I coordinate retention experiments with my acquisition team?
Acquisition affects which first products people buy and which cohorts of customers acquired enter the funnel. Changes like pushing a new hero product should be communicated to the retention team before experiments start. Use a shared calendar where acquisition and retention mark large changes so you avoid starting key tests during major traffic shifts. Agree upfront on which metrics each team owns: first-order conversion for acquisition, second-order rate and time between orders for retention. Order data reflects both decisions, so interpret experiments in context, especially for dtc brands that change creative and offers frequently.
What if my retention team changes often because of employee turnover?
High turnover on a retention team slows learning even when customer retention remains a business priority, because the reasons behind each test leave with the person who ran it. Create a shared experiment log listing hypotheses, audiences, changes, timing, and outcomes so new hires and managers can see what has been tried. Document key customer segments, main replenishable products, and standard test templates, so training a new team member means learning an established system rather than re-creating one. Hiring managers who run one on one meetings with a new analyst in the first weeks shorten the ramp, and the training pays for itself the first time the log answers a client's question. Employee retention on the agency side has its own levers, such as career growth, competitive compensation, development and a sane work life balance, and they sit outside this article. What you can control is that the record of what drives repeat purchases survives the hiring cycle.
How do I keep from overwhelming existing customers with too many tests?
Set basic contact rules per customer: a maximum number of promotional sends per week and a clear priority order for lifecycle messages. If a customer is in a high-priority flow like a recent purchase flow, exclude them from lower-priority tests. Tag customers by lifecycle stage and design tests targeting only one or two stages at a time. Monitor unsubscribe and complaint signals alongside revenue outcomes. A retention experiment that lifts short-term orders but hurts the long term customer experience may not be worth keeping. Subscription offers deserve the same care. Customers who choose a subscription usually do it for convenience and a small saving, and subscribers tend to stay with a brand far longer than one-off buyers. A flexible subscription that is easy to pause tends to retain customers for longer than a rigid one, and it gives the business recurring revenue it can plan around, but it is still one change to test against a hold-out, not a default. Companies that write the contact rules down find that the customer experience improves before any test runs, and support and services teams stop fielding complaints about too many emails.
How often should I revisit my retention experiment roadmap?
Most ecommerce and dtc brands should revisit their roadmap every month, reviewing completed tests, updating revenue impact estimates, and re-ranking upcoming experiments on fresh order data. Seasonality and product launches change which experiments matter: refill timing tests before holidays, subscription save-rate tests early in the year. Keep a stable core of proven lifecycle flows while using the review to decide which single new experiment to add. Regular feedback from current customers, collected alongside the order data, tells you why a test worked and feeds the next round of research into what loyal customers and returning customers actually want. That is how a retention strategy stays a growth engine instead of a list of tactics, and how the benefits of each test outlive the quarter it ran in. Companies that review monthly keep their customer retention rates from drifting, and the business gets a future pipeline of tests instead of a scramble. Communication with the client works the same way: a monthly note on what was tested and what it retained is the clearest report an agency can send.