A client of ours has undertaken a Findability initiative.  A site’s “findability” determines the ease with which visitors can get from the page on which they arrive at the site to the page(s) containing the products or information they seek.

Funding for the project is conditional upon each phase proving its impact and value.  How to measure that impact, therefore, has become a focal point of debate and, naturally enough, contention.  Yes, we’re all familiar with the adage that numbers can make them say whatever we wish, but the complications go much further than that.  Over 50% of all failed visits to e-commerce websites happen because of findability issues.  Findability issues can encompass the site’s architecture and navigation scheme, its taxonomy, or its site search and meta-tagging strategy.  Whatever the reason, findability frequently impedes the visitor’s quest to find what they seek.  If they can’t find it, they can’t buy it.  So there’s a direct hit on conversion and revenue.

Assessing the effectiveness of the findability project should be a simple matter, should it not, of measuring conversion before and after?  Assuming nothing else has changed on the site, any difference can be reasonably ascribed to the findability initiative, right?  The problem lies in the assumption that nothing else changes, because that’s absurd.  In the online world everything changes at the speed of light.  Acquisition strategies are constantly being refined, resulting in dozens, possibly hundreds of different campaigns driving new and existing visitors to the site.  Landing pages are being tinkered with to optimize conversion.  Products are being added or subtracted.  Promotions descend with bewildering frequency at different times and within different categories or across categories.  A single review can launch or destroy a product. Items get moved into Sale or Clearance sections to make room for the next season’s inventory.  Pricing changes.  Recommendation engines place products in different contexts for different visitors.  A website never sleeps.  So the simple assumption that we undertake a findability initiative and compare pre/post conversion rates is, simply, not feasible.

We need to be nuanced in our approach to measurement and cognizant of matching the metrics to the measure.  Here’s what we mean.

Viewpoint

First, it’s essential to understand the viewpoint that a metric reflects.   Search success and search relevance, for example, should be metrics gathered from users through survey responses because the metrics reflect the viewpoint of the user.  The site’s metric for search success, however, often measures something completely different — the number of times the engine actually returns one or more results.  Similarly, the search function that ranks and scores results by “relevance” reflects the site’s definition of relevance, not the user’s.  Assigning the appropriate metric, therefore, depends on the viewpoint you wish to represent and what you  want to do with the data.  What are you measuring and what are you going to do with the results?

Thus, for the search aspects of the findability project, we might start by clarifying that we want to capture the user’s viewpoint of search success, because then we’ll know how effective it is from their point of view — which is what really matters.  One of our other clients followed the conventional way to measure search success – counting the number of times the engine produced one or more results.  Under that metric and from that viewpoint, it reported search success at 99.8%.  When they started asking visitors who used site search how successful their searches had been, the number fell slightly – 48%!  Be careful what you ask for!

Dimension

Beyond viewpoint, it’s important to distinguish between attitudinal and behavioral metrics.  We all know that what people say is often different from what they do.  During the redesign of the homepage of a national DIY site, the designers got positive reviews in usability testing of a shortcut button they added.  Subsequent path analysis demonstrated that the conversion rate of visitors clicking on the shortcut was 33% lower than that of visitors navigating through the left-hand menu.  Attitudinal data, in this instance, meant nothing in comparison to the behavioral data.  (The shortcut was never removed.  It had been the site manager’s idea.)

Relationship

The final step in metrics definition might be to explore the comparative value of direct versus indirect metrics.  Take a tree-test for site taxonomy.  You run a tree test on a single category.  Say, for argument’s sake, that 85% of users look in the correct categories for the products you ask them to find.  So you take the 15% of products that were not located and rework the taxonomy according to the feedback of where users thought the products belonged.  Then you rerun the test with the new taxonomy. Lo and behold, 94% of users find products this time.  You have a nine-point lift and a 10.6% improvement.  That’s a direct metric.  The cause-and-effect relationship between the taxonomy changes and findability scores are irrefutable.  The new taxonomy is better than the old.  But what happens if conversion falls when you introduce the new taxonomy to production?  Can you be sure that the cause-and-effect relationship is valid.  Because so many other factors contribute to conversion, the relationship between the taxonomy change and the lower conversion is indirect and therefore less reliable/trustworthy.  You have contradictory metrics.  What do you do?  Leave the new taxonomy in place or revert to the old?  You leave it in place.  Because direct metrics are indisputable and trump indirect metrics.  Whatever the cause of the conversion problem, the new taxonomy is certainly not it.

While you leave it in place, however, you should actually monitor a few other metrics.  The first is to look at the conversion of first-time visitors before and after the change.  This is as close to an apples-to-apples comparison of indirect metrics as you can get.  (You look at first-time visitors because visitors familiar with the site may or may not think the change is better, but it is different, and change can often provoke discomfort.)  The second is to look at site visitor survey data and compare the pre-/ post-change percentage of first-time visitors who cited Labeling issues as a reason for visit failure.

The take-away from all this metric-mashing?  Be clear about what you intend to do with the metrics.  Form follows function in metrics just as much as in architecture.  Balance the viewpoints, clarify the role of attitudinal and behavioral, assign direct and indirect appropriately.  Monitor and adjust.  You’ll be fine.

Pew Internet released a report late in January 2012 that quantified a behavior we either exhibit ourselves or see others exhibit ever more often – people using their smart phones as part of the in-store shopping process.  During the holiday shopping season:

• 38% of smart phone owners called a friend for advice
• 25% looked up prices online
• 24% looked up product reviews online

We all understand why people do this – it’s to feel confident that we’re making the right decision and getting the best price.  The word that best describes this state we seek is Certitude, defined as “freedom from doubt.”

Back in the day
Back in the day, reaching certitude in a store used to be difficult unless you had already been to several other stores to check out prices.  This is the way most of us learned to shop.  Reaching certitude took time and effort.  Then the Web introduced online shopping and we no longer physically had to scour the local retail landscape to compare prices and availability.  Comparison shopping was far more efficiently done through a browser.  Online certitude remained elusive, however, because we could not feel, smell, or get a true multi-sensory impression of the product we sought.  And online merchants often failed (and still do) to provide all the information we needed in order for each of us to reach our own points of certitude.

When we shop packing smart phones, however, we have found the fastest path to certitude.  The ability to access the Web while we’re mobile – via bar-code scanning apps and QR links, in particular — has effectively allowed us to be many places at the same time.  Add the social dimension into the mix, and we have the equivalent of certitude support that was previously provided by having a friend there to proffer their opinion.  Macy’s, Saks, and other retailers know their shoppers often send a photo from the fitting room, asking for feedback.

The implications of certitude for the online store
We each have our own pathways and our own levels of certitude, but the smart phone-equipped store shopper is likely to get there faster than the single-mode shopper.  And this realization begs a question for the managers of all e-commerce sites: Have you done everything you can to allow your visitors to reach their own level of certitude?

Answer that question first by looking at your internal search.  The fastest, surest way for a visitor to reach certitude online is by being able to type a product ID into the search box and have the results deliver exactly what’s being sought.  Take the case of jeans.  Females who like how a pair of jeans fits in the store will often go online later to buy more pairs in different colors.  They already know the jeans fit, they just need to see what other colors are available.  Simple, right?  No.  The product ID on a garment label may have nothing to do with the way the garment is referenced as online inventory.  The ID ascribed by the manufacturer may be different from that ascribed by the retailer.  Product descriptions themselves can and often do vary across channels.  Consistency in identification, therefore, is the first task in assuring findability via search and facilitating visitor certitude.

Once you’ve looked at search and meta-tagging, turn your attention to the browse path.  The “path to certitude” checklist depends on what’s being sold, but the typical pieces of information in a typical product-based site would include: features/benefits, demos/ videos, specs/dimensions, colors/ swatches, views/details, options/customization, comparisons, ratings/reviews.   If you haven’t provided information to address the certitude needs of every visitor, you have given many of them a reason to abandon your site.

But there is a back-up plan.  It’s called policy.  A site like Zappos realizes that the key element in shoe-buying is fit.  Zappos also knows that it is impossible to convey fit as part of the shopping experience, yet, fit is essential for a shopper to reach certitude in shoe buying.  So what does Zappos do?  Zappos eliminates all the risk involved in buying shoes that do not fit by offering free shipping and free returns – for a year!  Zappos’ success has proven that online shoppers do not have to reach certitude if the site’s policies shift the consequences of an erroneous decision from the shopper back to the site.

To take the issue of certitude full circle — which is to say, back to the store shopper armed with a smart phone – the site manager must also think of the mobile visitor’s need for certitude by optimizing his site for mobile access or, better still, creating a pure mobile site.

Takeaways
So, the takeaways for retail website managers are:

• Check your products for consistency of identification and align your meta-tags appropriately.
• Check that you have provided every piece of product information that any visitor type would need in order to reach certitude.
• Review your policies to see if you have done all you reasonably (and logically) can to shorten the path to certitude by shifting the burden of risk onto your own shoulders.
• Create a pure mobile experience to facilitate certitude for the mobile shopper, too.

–Todd Luckey, Senior Usability Analyst

–Roger Beynon, CSO