Delving into Digital Marketing?: Your Intuition is Wrong – A/B Testing Results that Surprised the Experts

One of the things I absolutely love about split testing is its inherent ability to humble opinions of even the most seasoned testers. Sometimes even the most well researched hypotheses fail.

This is the strongest reason why companies must test everything from offer copy to page design, instead of relying on gut instinct or personal preferences. In this post, I’ll share some case studies that either saw huge opinion disparity among the WhichTestWon community, or whose results absolutely shocked our editorial team and Testing Awards’ judges.

Social Proof Is Not the End-All-Be-All

Can you guess which of these two versions of an in-line form generated more opt-ins for a well-known Web design blog?

As of today, 71% of the WhichTestWon community picked the version on the right – the one with social proof copy – as the winner. When I present this test to live audiences at conferences, usually 90-95% of attendees pick the social proof version.

However, they are all wrong. The variation at the left — without the subscriber count — saw 122% more newsletter opt-ins than its social proof counterpart. In a world where we are jaded by the user counts of Facebook, Twitter, etc…it seems that 14k visitors wasn’t compelling enough to get prospects to act.

Normally we see companies just add social proof without testing it because virtually every blog post and ‘social media expert’ has told them it will help conversions. Thankfully the team at this site tested this before implementation, or they would have missed out on a lot of opt-ins – more than double the opt-ins in fact.

Don’t get me wrong, there is a ton of value in social proof. For one, it is great at reducing a visitor’s anxiety and can give your brand a sense of authority. The trick is finding out where it is best to publish your social stats and what exactly is worth sharing. Should you post newsletter subscribers, Facebook likes, awards, or all of the above? Simply put: test it out. Never add something blindly; you don’t know how it will impact the bottom line.

Still not convinced? A VWO customer saw a similar result recently on its product page. Read the whole story here.

Icons May Be Trending, but they Might Hurt Conversions

Icons have been making a major comeback in web design. Overall, icons have been useful, especially when they are used as a replacement for bullets in stylized lists. The team at Build.com wanted to find out whether icons would be a useful navigational tool…the results surprised them.

Here are the two versions of the header they tested, one with icons and one without:

The challenger variation included icons that represented different category pages on the site. The team believed that an increased focus on navigation with their most visited categories would increase interactions and sales. However, the version without the icons saw 21% more product purchases.

Why? We suspect that although the icons provided a sleek navigation pane, overall they likely added more clutter that confused the visitor.

Security Seal on a Lead Gen Form Test

The highest point of friction on any lead generation page is the form itself. You need to identify the optimal number of form fields, choose an intuitive design, and add visible privacy policies and/or security icons to reduce anxiety.

These are well-known best practices that all lead gen marketers understand… and that’s probably why 74% of the WhichTestWon community guessed the wrong winner for this A/B test.

The variation without the TRUSTe logo next to the button got 12.6% more completed forms. Yes, the ‘submit’ button did shrink to accommodate the TRUSTe logo; but, we strongly suspect the primary cause for this lift has to do with the logo itself.

Trust seals can be essential to your conversion rate; the real trick is knowing where and when to place them.

In this particular circumstance, the TRUSTe logo was the wrong security seal at the wrong time. Visitors are used to seeing this seal, and others like it, directly in a shopping cart; not on a top funnel lead generation form. It’s quite likely that many of them suspected a payment transaction when they saw the trust seal here.

Instead of using a security seal, the team could have tested providing assurance by adding a simple fine-print text link to the privacy policy.

Remember, context is the key!

If Your Conversion Rates are Falling, Put on a Happy Face?

CRO specialists and designers love using faces! I get it; faces are the first thing the eye identifies when it looks at a web page. Numerous eye tracking studies support this claim.

However, sometimes a human face can be too much of a distraction. So, you need to test when you add faces to your page to make sure they aren’t competing with your important headlines and calls to actions (CTAs).

Here’s an example:

The version without the image won 24% more form completions. This wasn’t a perfectly clean test. There were some slight alterations in copy but nothing too dramatic. To tell you the truth, I’m more of a fan of the story behind this test than the actual unexpected results. However, it’s not the first or the last test we’ve seen where removing a face increased conversions.

By the way: I am happy that the team used the photo of an actual employee rather than a stock photo. Models and stock photos tend to get even worse conversions than “real” people.

What’s perhaps most amazing is that at the time of this split test HubSpot was about to make it a mandatory practice to include a person’s image on each of their landing pages. On some level this makes sense, they had found that some of their pages performed better with people’s pictures. However, what’s true for some pages may not always be true for all pages. Luckily, this test casted a seed of doubt and the company changed their design mandate.

Before you create any new landing page or design policies, please test beforehand…you have no idea just how many conversions you could leave on the table.

Video Icons – Product-Centric or Person-Centric?

Here is another case that tests whether using a face is appropriate.

The version of this Autodesk product page that used faces got 50% fewer video clicks. Nothing else on this page changed except for the video preview image. I am not anti-faces on websites; I simply want you to test before you implement!

Needless to say, the testing team was surprised by the results. So they ran a user survey to try to figure it out. The responses showed that Autodesk’s prospective buyers were more interested in seeing how the product worked over individuals talking about the product.

This just comes down to a case of knowing your audience and that best practices are not one-size-fits-all!

In Summary

Leaders in the testing field have all been stumped by the unexpected results before, and will be stumped again. The trick is to understand what to do after your test goes counter to your hypothesis or flat-lines.

Your next steps may include evaluating a litany of things such as your hypothesis, technology, source traffic, device…the list goes on. You need to learn if the test itself was flawed – or if your understanding of what your visitors really want from the page was flawed. Either way, you’ve learned something valuable.

Remember testing is an evolving process, future iterations are borne from our successes and our failures.

Keep testing my friends! There are so many variables to consider while running a test, it is no wonder that we often see lifts or losses where we least expect it.

The post Your Intuition is Wrong – A/B Testing Results that Surprised the Experts appeared first on Visual Website Optimizer Blog.