What to Measure and Test in E-Commerce Onsite Search

Peter Curran, President, Cirrus10 LLC, and Founding Board Member, PipelinePros
August 19, 2016

Search Matters

Assuming you aren’t a company with fewer than a hundred products or articles, and depending on a variety of factors including your industry, the breadth and depth of your assortment, and even the physical size and placement of your search bar; search should account for between 20% and 50% of your site traffic.

During a sales cycle with Oracle, I was once told by an IT person at a major specialty apparel brand with more than 10,000 products that “our customers just don’t seem to use search much, only about 3% of our traffic, so we just have the little magnifying glass icon in the upper right.”  Unfortunately, for that same prospect, as it is with almost every e-commerce site, search converted at twice the rate of browse, which means that they’re intentionally encouraging their visitors down a worse converting path.

In one household name retailer who we count as a customer, their search now converts at 8x browse!  So even if search accounts for 25% of your traffic it might still be contributing to over 50% of your PDP views.  Put simply, search matters, and if it matters then you should measure it, but what should you measure?

What to Measure

So, what should you measure with respect to e-Commerce search?  Here are my top three reports:

  1. Search Mix. This is a simple report showing how site traffic is split between browse, search, and other mechanisms such as finders.  If you can further segment the search traffic to those who clicked on a suggestion in type ahead versus those who entered keywords and clicked enter, that’s also useful to know because type ahead typically produces a higher quality resultset than raw keyword search does.  Look at this report monthly to quarterly and use it to prioritize your feature engineering.
  2. Search Performance. In its raw form, this report shows a list of terms (keywords or type ahead clicks) in descending frequency order with the following fields, if possible:
    • Frequency. The number of times was the term searched for in the report period.
    • Conversion. The percentage of the session population for the term went to a PDP.
    • Search Bounce. The percentage of the session population for the term searched again as their next action.
    • Bounce. The percentage of the session population for the term navigated away as their next action.  Ensure that this figure does not include clicks on guided navigation.  If it is possible for you to report on how and when guided navigation, sorts, and other search refinement features are used, that’s nice to have.
    • Exit. The percentage of the session population for the term left the site entirely as their next action.
    • The number of results returned to the user.
    • Screen. The split of traffic by screen width (e.g. phone, tablet, desktop).  If you prefer, you can also run three separate reports, one for each form factor.

This report should be reviewed in depth quarterly / seasonally, but the top 200 should be reviewed weekly.  This report can also be massaged and aggregated in dozens of ways to crystallize your understanding of how search is doing, generally.

  1. Zero Results. This report shows every term that resulted in zero search results sorted in descending frequency order.  Review this report no less often than weekly to ensure that you don’t need to do things such as, use the thesaurus to bridge a vocabulary gap between your products and the user’s term; handle a scarcity problem; fix an egregious data problem, or take some other corrective action.

Over the years, we’ve kept a list of all the different reports we’ve seen at our customers and we’re at over 60 since the company was founded in 2010, so there is definitely a lot more you can measure than the three things above, but it is astonishing how few companies look at any search-related reports at all (even if they have them!).

What to Test

Once you’re measuring search, what’s worth testing?

Just about every major decision related to how search behaves is worth testing.  A few functional tests that are worth it are:

  1. Horizontal vs. Vertical Guided Navigation. Many companies, particularly those with a heavily evergreen assortment, are finding that as their e-commerce capabilities mature, they’re able to improve the breadth and quality of product attribution.  This attribution often finds its way into guided navigation in the form of dimensions and refinements and as the list of dimensions grows, the efficacy of a traditional vertical arrangement is worth testing.
  2. Visual / Merchandized Type Ahead vs. Textual Type Ahead. Autocomplete suggestions in type ahead are completely under your control if you’re using Endeca.  There are myriad creative ways to generate autocomplete suggestions or other textual cues, but in my experience, it’s more valuable to fix crummy autocomplete suggestions than it is to add product imagery. That said, once you decide to add images, non-product suggestions, or anything else more exotic than good autocomplete, it’s prudent to test these layouts and outcomes.
  3. Presentation of Non-Product Content. After “it isn’t exactly rocket surgery”, my favorite saying is “snatching defeat from the jaws of victory” and it’s this saying that best describes the perils of mis-introducing non-product content to a traditional e-commerce site.  The traditionalist argument goes something like this: We’ve already achieved, or worse yet, paid for something improbable; we’ve gotten someone on our website instead of our competitor’s! And not only that, but this person has gone down the lucrative search path.  And now, on the 2 yard line in the Super Bowl with multiple downs and exactly enough time left in the 4th quarter to try a few safe plays, in a game in which you’re trailing, and with the best running back in football a man so difficult to tackle they call him “Beast Mode”, you’re going to choose to pass the ball over the middle into heavy coverage to a 2nd string wide receiver?!  You’re going to show them links to things that they can’t buy?! Madness!   The upstart opinion is some hippy dippy stuff about customer confidence, contextual calls to action, SEO gumdrops and unicorns, loyalty, blah blah blah.

No matter where you stand on the argument, it’s worth a test to see if the content is being presented properly and if that content is supporting the financial mission of the website.

As important as feature-related testing is relevancy testing.  Oracle Endeca Guided Search, and most easily when paired with Experience Manager, allows you to create multiple search interfaces which can select and order records in as many ways as you can imagine for any addressable search context on the site.  Your goal should be to organically (i.e. without excessive Boost & Bury or curation) order products (and non-product content) in search results in the way that converts the best in every segment for which you are influencing sort.

In conclusion, search is important not only because it represents a healthy percentage of site traffic, but moreover because it suggests an intent to purchase.  Measuring search performance, actively reviewing the measurements, and making the correct adjustments constitute a proven path to a more profitable e-commerce website.  Using testing tools in conjunction with site performance measurements can turn subjective arguments into objective hard data.

Go Deeper

Join me and the rest of the PipelinePros in our User Group’s Forum Session ID UGF7651 at Open World on Sunday September 18, 2016 where I’ll be speaking more about how to use Oracle Commerce to be an effective data-driven organization.

What to Measure and Test in E-Commerce Onsite Search