How do you measure the impact of conversion improvement on your site? Most marketers know the basics of how to calculate this, but might get confused when it came to details — or do they?
Astute readers will recall I began a series on calculating the impact of your continuous improvement efforts a few months ago, called “Making Millions From Losing Tests“.
Several readers wrote in privately asking for an even simpler example. It turns out that it’s the basic lift calculation that seems to be causing doubt, and therefore the detail work “feels shaky”, as one fellow put it. This is not uncommon.
Many of my clients seem to get “stuck” when measuring the impact to their bottom (and top) line from their conversion improvement and continuous optimization efforts. They’re actually doing the work and getti
ng results — but they manage to confuse themselves (and their bosses) when it comes to reporting the results.
Let me take this chance to clarify a simple way to measure lift. Often when you have the simple technique in your mind, the details fall into place by themselves.
A Hypothetical Example
Let’s assume your company started the year with some core numbers: 250,000 visitors and $1m in revenue. (I’m picking easy numbers on purpose.) That was for January. Now, during February, you started optimizing and visitors went up to 300,000 whereas revenue jumped to $1.5m. So, the simple lift question:
How much of the increase can rationally be argued to be part of your optimization efforts?
Surely it’s not the full half mil. Nor the extra 50k visitors. Where do you start in guestimating the impact of the improvement?
Here’s the simplest way to do it: First and most importantly, some amount of your traffic should be funneled into the control (or some call it “baseline”) part of your site. That is, your site as it was in January. This is what your visitors would have seen if you hadn’t done any improvement testing at all.
Allocate Some Traffic For A Control
As in all good testing programs, some percentage is allocated to the control. I’ll assume — and again, a “easy” number to keep the numbers simple — that you dedicated 10% of your traffic to the control version of your site. Ten percent of 300,000 visitors is 30,000 visitors.
Now you also know from your analytics what total revenue came from this group (or alternately, you know average order size, total carts completed, conversion rate, etc, from which you impute this total revenue number). Let’s say revenue from the control group was $125,000.
And that’s it.
How To Deal With Less-Than-Ideal Testing Situations
As an aside, you usually aren’t given 100% of the traffic to work with and you often don’t get to determine how much of the traffic is in the control group. You may need to dial up the percentage of the traffic over which you do have access, so you can be sure the critical control group is getting enough traffic to give you confidence in the results. That number can range from a third (on the high side) to as low as 5%. Again, it’s all dependent on your traffic.
If you are given less than 100% of the traffic on which to work your
continuous optimization efforts, don’t forget an important caveat: don’t let the traffic over which you do not have access be counted as your control — you must do a control group on your own. This is because you have to ensure that the conditions of the sample traffic being exposed to your control are approximately the same as those who are part of your ongoing tests. Otherwise there could be a skew in the sample traffic invalidating your measurement efforts.
For example, if you’re allowed to test against everything except affiliate traffic, then you can’t use the results from the affiliate traffic as your control — you need a control group from the non-affiliate traffic you do have access to. Otherwise you can’t be sure that your lift is due to your work, and you’re as likely to over-report your effort’s results as under-report them.
[This article is Cross-posted to my monthly column at MarketingLand, which is a great place to read all sorts of interesting content.]