If you’re looking for a fully customizable e-commerce platform, Magento 2 is the place for you.
Unlike other e-commerce sites like Shopify — which is an excellent choice for small businesses and those just beginning to venture into the e-commerce world — Magento allows for plenty of flexibility.
You just have to know how to code. As long as you’ve got basic coding knowledge and are willing to learn, you (or your team of web developers) can make your e-commerce website look exactly how you want it to look.
For example, you can use add-ons to customize your Shopify store, but your choices may be limited. Magento, on the other hand, offers thousands and thousands of add-ons for every aspect of your site via the Magento Marketplace.
And on
Take this e-commerce site for Redsea, a Middle Eastern appliance company. It created its first-ever online store with Magento.
Like WordPress, Magento is an open-source content management system (CMS) with a diverse and knowledgeable community to help answer any question you might have.
And you probably will have questions. After all, with so many customization choices, you’re faced with more decisions to make.
That’s why Magento A/B testing, also called split testing, is crucial to your e-commerce site.
What is Magento A/B testing?
The basic goal of any A/B test is to improve a metric such as:
- The click-through rate (CTR), or the percentage of shoppers who click on your product.
- The conversion rate, or the percentage of shoppers that purchase your product after seeing the listing.
With Magento A/B testing, you’re trying to figure out which version of your online storefront your target audience likes best.
What’s the best way to implement A/B testing in Magento?
The most well-known way to run a Magento A/B test is through live testing. In live testing, you
Various software programs like Best Worlds offer extensive live testing for Magento users. With Best Worlds, you can test factors like pricing, shipping deals, checkout extensions, and different online storefronts.
Other key live testing programs for Magento include Amasty, which enables you to run A/B and multivariate tests. Amasty helps you figure out which button sizes and colors,
Keep in mind that all live split-testing tools split traffic between two or more versions of a working web page, recording data on which one earned the most conversions.
You can only garner enough data to make meaningful conclusions about the variants if you have a steady stream of traffic. You also need to run the test for at least two weeks.
If you don’t gather enough data or give your test ample time, your split test won’t reflect daily fluctuations in sales or events outside of your control. Think of an increase in shopping before Mother’s Day. This can impact your traffic for the better but isn’t necessarily a portrayal of a typical day on your Magento e-commerce store.
Yes, it takes forever, but live testing can be a key tool because the data
Downsides to live Magento A/B testing
You might notice a couple of problems with live split testing. First, unless you have a thriving website already, it may not provide helpful results. You’ll have dished out cash on a program — unless you rotate the variants yourself every week and collect the data manually — and for what? Often the results are inconclusive.
Another downside? The two or more weeks you have to wait for results. What if one of the variants performs poorly, and as a result, the experiment costs you conversions?
And here’s another thing to consider: it’s important to split test your product descriptions. But changing the text on any listing can disrupt your keyword ranking. You don’t want to bump your listing out of a hard-earned ranking and indexing, especially if the change will lose you sales.
A customer who previously searched your product with a keyword that’s been displaced, rearranged, or eliminated can get an unfortunate surprise when they can’t find what they could the day or week before.
A frustrated customer isn’t going to want to come back later when you’ve figured out which variant works best.
Tips for live Magento A/B testing
If you’ve got the traffic and the time, the most significant benefit you’ll reap from live testing on Magento is that cold, hard data from your customers.
Say you run an A/B test between two versions of your Magento home page for a clothing business. The old page features a single hero image with mixed genders wearing the company’s clothing, while the new page splits the hero into two sections.
Over the next two weeks, your split test shows that the browser featuring the dual-image hero drives more customers to put clothes in their shopping carts.
You’d then direct 100% of your traffic to the dual-image hero variant in order to reap the full benefits of the chosen homepage.
Follow these tips to make sure your live testing is successful.
Set a sample size before beginning
On many live Magento A/B testing platforms, businesses running tests stop the test once they achieve statistical significance.
Here’s why we don’t recommend doing this: You don’t just need to aim for statistical significance, which isn’t always guaranteed. You also need to meet your predetermined sample size in order to achieve the most accurate results.
Use an online sample size calculator to punch in your baseline conversion rate and your minimum detectable effect, along with the desired statistical power and significance level.
Only once your sample size per variation has been reached should you stop your test.
You may or may not achieve statistical significance at all. It all depends on whether the variants greatly affect your shoppers’ purchasing decisions — or not.
Run each variation for weeklong increments
Since you want data from each day of the week, change it every week. This is less confusing than a daily change, and it can result in more accurate data.
For example, you’ll get a full week of data for each day of the week before moving on to the first variant.
This is important because maybe you send out a coupon each week on Friday and customers tend to use it on Saturday. By testing for full week increments, you’ll get full data for each variant on those two days.
Test for two full business cycles
For best results, test for at least one business cycle (usually a two-week time span) or two cycles (four weeks).
You may not reach the sample size within that time frame. In that case, you should stop the test and consider more beneficial ways to get insight on your webpage. (Hint: we have a solution for that, which you’ll learn about in a moment.)
Don’t get discouraged if you end up having to stop a live test.
A sample size generally needs to be large in order to give you accurate results, as you can see in the screenshot above: 38,210 visitors per variation.
Let’s say your website typically sees 500 visits a week. It will take forever to meet your sample size, and you will either need to change the values (a lot) or try another option.
What other software can I use to test on Magento 2?
If you’re not in a place to implement live Magento A/B testing, other programs that can eliminate the issues you’d face. Try PickFu.
PickFu matches you with U.S.-based audiences who share valuable feedback on focused aspects of your Magento store.
Here’s where PickFu’s greatest benefit shines: while you can change several aspects of your site in a live split test, you can’t ask customers to analyze that aspect alone.
In a live test, they have no idea they’re even participating.
PickFu helps you create polls and select the type of person who will respond to them. For Magento A/B testing, you can test areas like:
- The overall look and user-friendliness of your storefront
- Individual product listings as a whole
- Logos
- Hero images
- Product titles
- Main images
- Product descriptions
- Prices
As you can tell, split testing on PickFu is more versatile than live split tests. And it won’t mess with your keyword indexing!
For example, you can post several versions of
Or you can test a new logo, like this PickFu user recently did.
In this test, Option D garnered the highest score. Respondents left comments like:
- “I prefer Option D because I think the darker background with the globe stands out great.”
- “[Option] D is a nice logo. The words are easy to understand and the globe is on point. This one is my favorite one because it is simple, yet bold.”
- “The Zima Global logo with the shopping cart [Option A] is messy-looking with too many different things going on.”
There’s nothing like knowing why your target audience voted on the choice they did. That’s something you don’t get in a live Magento A/B test.
And because PickFu works entirely outside the Magento marketplace, you won’t be directing shoppers to a potentially less compelling version of your site so there’s no risk of lost sales or search ranking.
Instead, you can take the feedback and votes collected from PickFu and use them to improve your page — often with definitively increased returns, as you’ll see in this example.
Another benefit is your Magento site or the new variants you’re considering don’t have to be fully functional.
You can upload conceptual drawings or screenshots before you devote development time or resources to getting the back-end in order. When you’re working with a CMS program like Magento, this confidence can be meaningful to your business goals.
Targetable traits on PickFu
While you aren’t testing with an audience of your actual shoppers on PickFu, you can choose a target audience that resembles your ideal customer.
For example, you can target respondents by traits relevant to your products, such as dog/cat owners, homeowners, nutritional supplement users, cosmetics users, and more.
Get started with Magento A/B testing on PickFu today
It takes just a few minutes to set up your PickFu poll, and your results will start rolling in immediately.
Wait to do live testing until you and your site are ready for it, and in the meantime, get quick results using PickFu. Create your first poll today and start narrowing down all those choices you have to make on your Magento 2 site!
