It’s thrilling to set up your own online store, especially on a platform like Shopify, which makes it easy for sellers to create and maintain sleek e-commerce storefronts.
But despite the ease with which you can share to social media, blog about your product, and curate a website, there’s an additional step you need to think about.
To enjoy success, you need to do some Shopify A/B testing. This will help make sure your customers are responding to what you’re selling and how you’re presenting it.
What is Shopify A/B testing?
Have you ever posted a Facebook poll to your friends, asking which version of a product they like better?
That’s an A/B test — also called a split test — where you figure out which version of something an audience likes better. By posting the popular version to your storefront, you’ll enjoy increased conversions.
But the problem with your Facebook friends is that they aren’t an objective group. Your friends want to encourage you, and they’re probably not your target customers.
With Shopify A/B testing, your goal is to find out which version of your e-commerce storefront makes your customers convert.
How to split test with Shopify
In a classic A/B test, you direct a portion of your traffic to one version of the site and another portion to the variant. Various online software programs can help you run an A/B test on Shopify: Shogun, Convert, and Dynamic Yield are some of the top Shopify split-testing tools.
Keep in mind that these are all live split-testing tools, which means that they split traffic between two or more versions of a working web page and record data on which one earned the most conversions.
You can also run manual split tests by posting one version of your homepage for a week and recording the data, and then switching to the variant for a week.
Obviously, this is a tedious and more time-consuming way to conduct live testing. And whether you run manual tests or use a program, you’ll need to run tests for at least two weeks, often longer, for the best chance at getting conclusive data.
Another, faster way to split test is to run a poll entirely outside the Shopify marketplace. You’ll use the results from this poll to make changes on your site that you know will make a positive difference.
In this article, you’ll learn the benefits and drawbacks of both A/B testing styles — plus tips to help you succeed.
The benefits of live A/B testing
The most significant benefit of live testing on Shopify is that you get cold, hard data from your shoppers.
Say you run an A/B test between two versions of your Shopify home page for a t-shirt business. The old page features a serif font and the new one tests out a sans serif font.
Over the course of two weeks, your split test reveals that the sans-serif page drives more customers to your ‘Bestselling Tees’ page, which then leads to more people buying those t-shirts and greater revenue for you.
You’d then direct 100% of your traffic there in order to reap the full benefit of your improved font.
To make sure this testing is successful, follow these tips.
Set a sample size before beginning
Many brands running live tests stop the test once they achieve statistical signifi
In addition to reaching statistical significance, you also need to meet your predetermined sample size in order to achieve accurate results.
Use an online sample size calculator to punch in your baseline conversion rate and your minimum detectable effect, along with the desired statistical power and significance level.
Only once your sample size per variation has been reached should you stop your test. You may or may not achieve statistical significance at all, depending on how much the variants affect your customers’ purchasing decisions.
Run each variation for weeklong increments
Since you want data from each day of the week, avoid switching the variant every other day and instead change it every week. That way you collect data for Monday, Tuesday, Wednesday, and so on.
This is important because maybe you send out a coupon each week on Monday and customers tend to use it on Tuesday. By testing for full week increments, you’ll get full data for each variant on those two days.
Test for two full business cycles
For the most accurate results, you’ll want to test for at least one business cycle (usually a two-week time span) or two cycles (four weeks).
If you don’t reach the sample size within that time frame, you still should stop the test and consider more beneficial ways to get insight on your webpage. (Hint: there’s a solution for that coming right up!)
The sample size generally needs to be large in order to give you accurate results, as you can see in the screenshot above: 38,210 visitors per variation.
This means that if your website typically reaches 500 visits a week, you won’t meet your sample size and will either need to change the values (a lot) or try another option.
What should you test?
Before you commit to such a large-scale, pricey endeavor as live split testing, you’ll want to make sure that what you’re testing actually matters.
First, make sure that everything is loading well on your website across multiple device styles.
If your page loads easily on an iPad but lags or otherwise frustrates iPhone users, what can you do to fix it?
By getting a technical analysis of your site out of the way before testing, you’ll eliminate faulty software or loading from your future testing data. (And you’ll get more conversions before you even begin your test.)
Another way to get ideas on what to test is to run on-site surveys. If your visitors are filling them out, what are they saying could be improved about their initial shopping experience?
Focus your improvements and subsequent live testing on the most common or significant complaints.
Once visitors convert to customers, ask those customers to fill out full-length surveys.
These surveys will reveal further areas that could use brushing-up with improvements and live testing.
You can also glean insights from usability testing, where you observe the way users interact with your site from the initial loading of your webpage to the addition of a product to the shopping cart.
If at any point the users get confused or irritated, it’s time to improve upon that area of your website and run a live test.
Watch out for external validity threats
While the term external validity threats sounds scary, it’s really just a reminder to be aware of circumstances surrounding your live test.
Don’t, for example, start your live split test on December 15 and run it through mid-January. Chances are people are going to be purchasing a lot more in the initial week or two of the test and a lot less in January due to the holiday season.
To run a test during this window would jeopardize the validity of your test. You could have an apparent winner by December 30 only to watch that winner drop in popularity in early January because people were done with their holiday shopping spree.
The same goes for any notable sales days: Black Friday, Labor Day, the first day you launch a sales campaign, the change of seasons, and more.
Be aware of any significant events that contribute to increased or decreased shopping to reduce external validity threats.
This doesn’t mean you can’t test, but tha
Shopify live A/B testing: complicated but useful
As you can see, live A/B testing on Shopify isn’t something to leap into when you’re just starting your business and don’t have much traffic, or when you don’t have thousands of weekly visitors.
It’s often important to do once your business is established and growing.
This is where a major downfall of live split testing comes in: it’s only useful once you’re already enjoying some success.
What about those who need advice when they’re just starting, or who are seeing a lag in visits (and therefore conversions)?
This is where a different kind of split testing comes into play.
A/B testing with PickFu
For those who don’t yet have a ton of traffic on their Shopify storefront — or who simply want to get results in minutes or hours, not weeks — split testing on PickFu is an excellent option.
It’s also a lot less complicated than live split testing.
PickFu matches you with U.S.-based audiences who share valuable feedback on focused aspects of your Shopify store.
This is another significant benefit because while you can change several things about your site to conduct a live split test, you can’t direct customers to that single aspect alone in order to see how they react.
PickFu helps you create polls and select the type of person who will respond to them. For Shopify A/B testing, you can test things like:
- The overall look and user-f
riendliness of your storefront - Individual product listings as a whole
- Product titles
- Main images
- Product descriptions
- Product concepts
As you can tell, split testing on PickFu is more versatile than live split tests. You can post two versions of a t-shirt you’re thinking of selling on your Shopify store and ask, “Which one would you rather buy?”
Respondents will vote on their favorite, but that’s not all. They’ll also give you detailed feedback on why they chose the option they did. In a live split test, customers have no idea they’re participating in a test and therefore can’t give this type of feedback.
Benefits of Shopify A/B testing on PickFu
Because PickFu works entirely outside the Shopify marketplace, you won’t be directing shoppers to a potentially less compelling version of your Shopify site. Therefore, there’s no risk of lost sales or search ranking like there is in live testing.
Instead, you’ll take the feedback and votes gleaned from PickFu and use them to improve your page — often with definitively positive results.
Another benefit is your Shopify site or the new variants you’re considering don’t have to be fully functional. You can upload conceptual drawings or screenshots before you devote development time or resources to getting the back-end in order.
A case study
Take this case study, for example.
Steve Chou of Bumblebee Linens felt frustrated by how much time traditional live split tests take, especially because they often produced inconclusive results. So he figured he’d give PickFu a try. With his first PickFu poll, he asked 50 females which main images of a linen napkin they preferred.
See the two versions below.
The version on the right won by a large margin, so Steve changed the image on his site and watched sales skyrocket by 209%.
All from a simple PickFu split test that took less than 20 minutes.
Of course, live split tests offer data from real customers, which is something an outside test can’t do. But with PickFu, you can craft a target audience based on traits that are relevant to your product or store as a whole.
For example, if your Shopify store targets women with young children, you can choose respondents who are women with 1, 2, or 3+ kids.
Other targetable traits include dog/cat owners, homeowners, nutritional supplement users, cosmetics users, frequent exercisers, and much more.
Tips for testing on PickFu
For best results on PickFu, keep your questions simple. You don’t want to influence your audience’s response to your poll with a leading question, so something like “Which description do you like better?” is better than “Do you like Option B better than Option A?”
Also, avoid double-barreled questions, wherein you ask two different questions in one.
In other words, don’t try to stuff one poll with everything you want to know about your Shopify listing. You can always run a second or third poll to help you whittle down the listing to its best elements.
Get started today on PickFu, and feel free to reach out to us if you need any help whatsoever!
