What's really going on with your website conversion rate?
It’s the voice of frustrated e-commerce managers all over the world:
“What just happened?”
Why that sudden downturn in digital revenues on your website? Why that unexpected drop in online conversion rates? Why is that new functionality on your web application failing to deliver the uplift in performance the business was promised? And what can be done about it?
The trouble is there may be no single, obvious reason why and where your conversion rate is being impacted – and no clear way forward to fix or optimise the site for better results.
But the digital team know there’s definitely a problem somewhere. Somewhere deep in the customer journey, perhaps, on certain browsers or device types. One or more bugs, maybe; usability issues missed before launch that are leaking revenue and potentially damaging your reputation with customers.
Still, there seems to be no quick way to figure out which of a myriad of possible issues might be affecting website performance.
“But didn’t we test all this before release?”
Your team might have performed basic functional and regression testing prior to release within a staging environment, but those tests may have been restricted in scope and limited in the number of devices that were used. Testing teams may simply have lacked the resource and time to conduct comprehensive exploratory tests or plan for ongoing testing to inform future optimisation efforts.
One round of testing is not enough
In this omni-channel world nothing remains static; websites and apps are moving targets for bug detection and usability testing. As release cycles shorten and hardware and software develops, customer experiences tomorrow will always be different from today. Make a change here and something will break there. In this world, everything is related to everything else, but nothing remains the same for long. In these circumstances, testing for and resolving bugs can simply become a reactive process of damage limitation, rather than a systematic and preventative regime.
At the same time conversion rate optimisation activity can become reduced to mere guesswork.
Why real-world website testing is the answer
The truth is real-world testing is often the only way to uncover and understand what’s going on in such a complex environment of continuous change, updates and deployment.
Emulators tell you only half the story, because they can’t properly replicate the touch screens or hard and soft keys of mobile devices, the conditions of use or the subtleties of the many OS versions that proliferate. Meanwhile, automated testing can be invasive, flakey and simply generate more work for brands, as results are analysed and tests re-run manually to confirm the existence of an issue that can then be properly addressed.
Manual testing is a daunting challenge for most
The amount of manual testing resource a single company has at its disposal compared to the speed of new software releases and the multitude of device/browser combinations, means it is practically impossible for a traditional team to test adequately through this method.
So, teams in ambitious companies find compromises are being made, having to choose between speed of innovation or the thoroughness of their approach to testing.
‘Delay – or push and pray’ have become default settings for some, presenting a real risk to revenue and reputation.
But for an e-commerce team working in a hypercompetitive world, neither scenario is really acceptable.
We can’t wait forever for new functionality to be released and lose competitive advantage, but we can’t afford for poor quality releases to spoil our conversion rate and brand reputation with customers.
Likewise, once re-platforming work is done or new functionality is released, we can’t afford for optimisation efforts to be done in a piecemeal way or simply be based around a series of hunches.
The stakes couldn’t be higher.
“DEPENDABLE QUALITY ASSURANCE AND TESTING HAVE ALWAYS BEEN OF HIGH IMPORTANCE IN THIS SECTOR – BUT WE ARE CURRENTLY WITNESSING TIMES WHEN THEY WILL BE ABSOLUTELY, MAKE-OR-BREAK, CRUCIAL.”
World Quality Report 2019
Why testing e-commerce websites and apps is central to conversion rate optimisation
In the era of accelerating digital opportunity and hypercompetition, e-commerce teams need websites and apps that deliver value to customers from the moment they’re released, with constant updates to functionality that open up more sales opportunities, drive loyalty and increase conversion.
Brands are involved in major re-platforming projects that promise richer customer experiences – but increasingly delivered against tight deadlines by smaller, more agile teams.
Those who aren’t constantly innovating are losing out, they are missing the opportunity to experiment, to ‘fail fast’ – to change dynamically with commercial demand.
But if innovation requires us to ‘fail fast’ on a commercial level, it also requires us to be constantly vigilant around quality, ensuring that new functionality is tested and optimised properly so that it is given maximum opportunity to succeed.
A few years ago Marks and Spencer’s notorious website relaunch that promised a new era for their online shoppers (cutting-edge digital experiences and easier paths to purchase) proved on release to be rife with bugs and usability issues. The £150-million rebuild, in fact, saw a 8.1% drop in online sales and 1.5% drop in share price, as millions of registered customers were forced to create new accounts with the retailer, while others reported a complete inability to check out.
Spectacular failures like these are the stuff of nightmares for brands, who, needing time to develop complex new offerings but faced with fierce commercial pressures to release, struggle to co- ordinate their testing efforts effectively and end up taking risks with their reputation.
At the same time brands lose business each year through system crashes as websites remain untested and unoptimised for the surges in traffic that are the reality of modern online retail. In 2019 alone, American retailer Costco’s Thanksgiving outage reportedly cost them $11 million, while in the UK Curry’s Black Friday crash led them into a PR nightmare as frustrated customers took to social media to berate them.
Efficient and effective testing has to be central to a process of bug trapping, usability and performance analysis that can protect a brand’s reputation.
The new e-commerce functionalities that are being developed and released are bringing more and more value to customers.
Personalisation, conversational AI and gamification are all areas where innovations are promising more memorable user experiences, increased loyalty, more dwell time and greater conversion.
According to research by Evergage, personalisation activity across websites and apps is continuing to bring measurable increases in conversion rates (63%) and visitor engagement (57%). Meanwhile in 2019 Shopify reported how these highly specific micro-moments of reward and delight are making or breaking visitor experiences. Shopify’s Kurt Elster describes how a customer journey originating in a personalised coupon resulted in an 18% increase in customer conversion rate against a non-tailored one.
“PRESENTING SOMEONE WITH WHAT APPEARS TO BE A DYNAMICALLY-GENERATED UNIQUE COUPON CODE SIGNALS TO THEM THAT THE EXPERIENCE IS PERSONALISED AND THAT THE URGENCY IS REAL. THE FORMATTING OF A COUPON CODE MAY SEEM LIKE A TINY MUNDANE DETAIL, BUT IT’S A BIG SIGNAL TO A SOPHISTICATED SHOPPER.”
But how thoroughly are these kind of innovations being tested? Features like this are creating new customer journeys and increasing customer expectations. But they’re also introducing new complexity and, equally, the risk of customer disappointment and commercial failure. Systems slowing down or crashing under pressure of increased usage can undo all the great PR and sales results such innovation promises.
With such specific customer journeys, dependent on multiple algorithms and integrations, only ongoing real-world testing can ensure that these different experiences are being realised effectively. And only ongoing, real-world testing can ensure that they continue to deliver as the sites and customer journeys themselves continue to evolve.
According to one piece of research by the DevOps Zone
- 47% of consumers expect a web page to load in 2 seconds or less
- 40% of consumers abandon a website that takes more than 3 seconds to load
- 44% of users will tell their friends about a bad experience online
- 79% of users who are dissatisfied with website performance are less likely to return
Meanwhile, the digital global opportunity brings distant and lucrative new markets tantalisingly close for eager, challenger businesses.
But can your website and apps really deliver the conversions you want in the regions you need to target? The challenge of testing these solutions – optimising for cultural and linguistic differences – is real and ongoing.
“Our global customers have different demands. We want products to ‘feel local’ and to support features that may be unique to specific markets. As we specialize our products for certain markets, it introduces more challenges for testing – like requiring special cultural knowledge.”
Patrick Copeland, Senior Engineering Director, Google
Google, Amazon and Facebook might have teams all over the world, creating and optimising their experiences by region and country, but the cost of keeping developers and testers in every territory for most companies can be prohibitively expensive.
Even so, brands who want to cut through in new markets, need to have the ability to ensure that their offerings are working on the ground – and that they can be continually tweaked and optimised to deliver the best results with potentially new and unfamiliar customer behaviours.
Mcommerce remains a vast opportunity, but many are not realising its massive potential because of the complexity of optimisation requirements across a huge range of changing devices.
Worldwide, mobile sales trail desktop by over one trillion dollars, and mobile conversion rates are less than half those of desktop.
How can businesses who need to engage with customers, wherever they are, be sure that the mobile experiences they are creating are able to do this? Are they tested in different conditions of connectivity, not just WiFi but using the EDGE, as well as 3, 4 and 5G? Is usability testing reflecting the device/browser combinations that are favoured by your ideal customers?
More to test; proliferating device/browser combinations
The digital world never stands still. Devices are constantly evolving, Operating System upgrades are always happening and new browsers are being launched. The digital landscape is in a constant state of flux.
According to BrowserStack, every year, across the world more than 4 billion people access the web through a combination of more than 9,000 distinct devices, 21 different operating systems and 8 major engines that power hundreds of different browsers.
That’s an estimated 63,000 possible browser/platform/device combinations that could be accessing your website at any given time!
This state of fragmentation means there are a myriad of reasons why a website might see a sudden drop in revenue, and multiple ways a site or an app might be failing to convert visitors to customers. But without testing the right combinations that reflect the likely variations in your market, and without being able to check the way they browse, you could be losing customers and revenue in surprising quantities.
E-commerce challenges have changed, testing and optimisation strategies must change with it
All of the above illustrate how the development challenge has changed and grown over the last 10 years for e-commerce companies. These are the pressures and opportunities that have necessitated moving away from a waterfall approach to development, and instead meant the embrace of devOps and other agile approaches.
Increasingly, in e-commerce companies development sprints are shortening. New functionality is being introduced in more rapid cycles – all designed to help brands keep their competitive edge.
According to the State of Agile report (2019) 97% of organisations are now practising agile development methods, while 52% report that more than half of their development teams are following agile practices.
Inevitably, in these conditions, the nature of testing has also had to change. Testing needs to uncover high-priority bugs earlier and more quickly to lower the risk of late fixes and increased costs. Speed and scale of response remains of the essence, but testing becomes a 365/24/7 operation, with the need for a constant feedback loop to exist between developers and testers.
Time and again brands who are working in an agile environment see the need for their testing to become more agile to keep pace with the velocity of changes they are making.
For example, the e-commerce brand Etsy has experienced first-hand the limitations of relying on traditional ‘waterfall’ testing regimes in a rapidly changing digital space.
“In practice, unexpected things kept happening in deployment. Users sometimes did unexpected things, like launching a flash sale that inundated the site with a huge number of users at once. Or there could be some unexpected interaction between the actual hardware and the software. Or sometimes in a bundle of many changes it was difficult to figure out the cause of the problem”
From the Age of Agile, Steve Denning
Etsy found that even relatively small changes could add or subtract ‘millions of dollars in sales’ from their bottom line, and if they couldn’t exactly account for or understand the impact of those changes, they couldn’t optimise rapidly enough to keep improving UX and protect their revenue.
So, instead, they moved to a system of continuous deployment, with a strategy of constant regression and exploratory testing, feeding back into developers to fix high priority issues.
“With an average of thirty changes to the website deployed each day, we are looking at extremely rapid innovation. Doing all these changes in tiny increments at warp speed within the framework of a central strategy enables extremely rapid innovation and learning, as well as much greater facility in spotting and fixing any problems that may emerge”
From the Age of Agile, Steve Denning
What are your options to protect and optimise conversion rate and e-commerce revenues?
“Digital business is driving a faster pace of delivery to support the continuous delivery of incremental changes. Traditional testing teams cannot meet this pace nor the expanded view of quality required.” Gartner
Who can you trust to deliver those real world insights that can help you protect your conversion rate and continue to optimise?
There was a time when CRO agencies and consultancies could promise easy answers for e-commerce brands. When e-commerce design was a nascent discipline and your customers were overwhelmingly on PCs and a small handful of mobile types.
Back then it was easy for a single expert to advise on best practice – to tell you what should be above or below the fold. But back then, of course, the fold stayed in one place. One size fits all psychological and design insights, observations about colour choices and size of boxes might have been relevant then, but now you need insight that crosses numerous device and browser
combinations, analytics that identify silent ‘killer’ bugs affecting certain customer journeys that incrementally make up significant revenue loss.
Times have changed. There is significant expertise in digital design present in most businesses, no developers are in the habit of designing poor customer experiences. What they need is access to more real world and more sophisticated data and insight – showing how customers are really interacting with their sites. They need analytics and test results that identify, prioritise and target invisible areas of weakness; the specific differences between the rendering of customer journeys on devices and operating systems that need addressing to maximise returns.
In-house testing team
Businesses like Amazon and Facebook have huge teams of testers across the globe that deliver QA around the clock, constantly feeding into the development and optimisation loop. This reflects their massive size and the rapid pace at which they iterate their software.
But most companies don’t have the luxury of this resource, and may not have changed the composition and function of their QA teams in years. And even as their development teams have stepped up the pace of their releases, their test regimes may have remained largely the same, geared towards a waterfall process of design, test and release.
Traditional QA set-ups may not be fit for purpose in the age of agile
In a world of waterfall development where big projects were developed, delivered, tested, then deployed only once or twice a year – using smaller, in house test teams for QA activity made sense.
The testing project was finite and manageable because builds were definitive and consumers were restricted in the types of devices and browsers they were using to access websites and apps – there was not so much fragmentation to account for.
But in an agile world of continuous deployment, within a fragmented digital landscape of seemingly infinite device, browser and OS combinations, a smaller QA team can become easily overwhelmed with the size of the challenge.
Browser blindness can easily set in for small teams battling to take in the many, subtle, but important differences between multiple interfaces.
In these conditions testing can all too often become perfunctory, limited to a few essential functional tests as quality becomes subservient to aims of constant innovation and delivery. Either that or the QA team itself becomes a block on release, unable to keep pace with the development team, but unwilling to relinquish control.
QA teams may themselves be understandably concerned about change, wanting to keep their testing monopoly, reluctant to outsource, unable to see how they could coordinate their teams effectively without conducting all necessary tests themselves. However, as many now acknowledge, this is often just compounding the problem.
In-house development teams
But what if there’s no separate QA function at all within a company? What if the burden of testing and optimisation falls entirely on developers themselves?
E-commerce teams are increasingly called upon to do more with less. Lean teams are delivering new functionality at a faster rate than ever before.
But dev teams are focused on and often selected for their innovative thinking, their drive and desire to move on to their next product. The mentality of the tester is different from that of the developer. You don’t want to be distracting them with time consuming testing duties, when they should be developing new and exciting functionality.
Overstretched development and QA teams can often be forced into cycles of delay and untested release. Bottlenecks may mean they are routinely missing bugs and approving suboptimal user experiences; surprised by unexpected crashes and constantly working to investigate a back-log of reported bugs.
Suspicion between digital managers, testing and development teams can grow as blame for sudden slumps in performance get shunted between them, while responsibilities for ongoing testing and optimisation is muddied, becoming a merely reactive and ad hoc activity.
There’s a risk that attention to existing sites and app optimisation can become piecemeal and uncoordinated if it’s only sparked by customer complaints and not focused, organised and systematic.
In many businesses there is no culture of systematic bug trapping and companies instead begin to rely on the ‘superhuman’ efforts of their engineers to prevent disaster – those prepared to work all night to fix the last few bugs, or investigating workarounds for a serious issue at the last minute.
“Occasionally there is a superstar of an engineer that can take one of these changes and run through the gauntlet of all the possible ways that it could screw up and make it a success. And then we make a hero out of that person. And everybody else who wants to be a hero says “Oh, that is what is valued around here.”
From – The Persistence of Firefighting in Product Development, MIT Center for Innovation in Product Development
But sometimes, even the best efforts of a company to set up adequate testing processes can fail in the context of modern development and release pressures.
“Despite many efforts to put quality assurance processes in place, it is a common practice among engineering teams to make exceptions in quality practices to meet deadlines or other goals. Although it’s certainly important to be flexible to meet ship dates or other deadlines, quality often suffers because of a lack of a true quality owner.”
— How We Test Software at Microsoft by Alan Pag
Testing should go hand-in-hand with optimisation efforts, but the challenges of delivering all this can often be too great for lean businesses, causing undue stress and breakdown of process.
How to build an e-commerce success story through a continual process of conversion optimisation
In an age of digital opportunity and agile development, the advantage is to the first movers and those who are committed to continuous improvement.
Ambitious e-commerce managers don’t want to be on the back foot constantly working out where it all went wrong.
At the same time to be successful QA has to be the concern of everyone in the business. The development and digital teams (and dedicated QA function, if there is one) all have to be pulling in the same direction. But to do that they need to be given the insights to power their decision making.
Real-world testing is central to this because it gives tangible, actionable feedback on the effect of changes and the existence of real bugs and errors. But delivering testing with the right coverage, and at the speed and scale necessary to make the difference when it most matters, is a complex task.
Outsourcing to professional testers
Outsourcing to professional testers has its pros and cons, but it can be a cost effective way to bring high quality insights to inform your ongoing optimisation activity.
It can bring the size and quality of testing resource that normally only hyperscale companies have access to within the reach of challenger e-commerce brands. In some cases thousands of testers can be made available to reduce functional and non-functional testing time down from days to hours.
These solutions can offer a high level of professional and sector expertise, as well as the required range of physical hardware to conduct thoroughgoing tests. Add to that the international reach these companies are able to offer and the coverage they give can easily eclipse most in-house capabilities.
With a scalable, ongoing testing function available 24/7, 365 days a year, challenger brands can really begin to echo the strategies of hyperscale competitors who are able to function in cycles of continuous deployment.
It enables the ongoing testing of new functionality – fuelling constant, incremental improvements in quality. QA teams who are able to work, confidently, with outsourced, professional testers can apply these working methods. Firstly using analytics to identify areas of weakness, then running ongoing regression, functional and exploratory manual tests at the scale and speed necessary to match the overall pace of their innovation.
And the kind of insights that real world testers can offer, represent real, qualitative differences from those of automated solutions – and at the scale and speed most in-house functions cannot rival.
Functional testing is vital for optimisation
Functional issues are the bugs that prevent vital user journeys from being completed, that lead to baskets being abandoned and customers stuck in loops or kicked off websites altogether, never to return.
Inappropriate redirects, 404 pages, incorrect filter results and checkout issues can all seriously damage revenue and reputation.
Functional testing identifies all these kind of issues using non-disruptive methods and then orders them using impact analytics.
Testers working at scale against scripts on a range of real world devices and browsers, can quickly uncover where certain browser, device or OS-specific bugs are present. And they can obviously emulate real human actions much more effectively than automation solutions.
Why test automation may be failing brands
In fact, companies using automation testing tools (such as Selenium) to perform their functional and regression tests often find them time consuming to set up and difficult to reconfigure quickly enough to meet their needs. These automation programmes can cause sites to crash and also need constant maintenance if they are not to fail mid test.
Some estimates say set up time for automation testing is 3 – 10 times longer than for manual tests.
This is not helpful for features and products that are still in development, as testing scripts need to change frequently to answer different questions.
As the functionality of websites becomes more and more complex – with increasing servitisation and personalised buying experiences, so testing needs are becoming more and more complex.
With UX characterised by highly engaging features such as carousels, configurators, AR and the like, the risk of bugs appearing rises, and the depth and complexity necessary for user acceptance testing must increase too.
This is where human testing can really excel. Non-functional testing includes the kind of usability, performance and exploratory testing that really brings profound customer insight to brands.
Website usability testing typically covers how easy it is to learn or understand to navigate the site, the efficiency with which a customer can perform desired actions, the amount of errors the customer makes in pursuing those actions, the level of satisfaction a site delivers and the ‘memorability’ of that experience.
Usability testing will flag up parts of customers journeys that are illogical, confusing or where required actions are unclear – as well as pick up on reactions to colour and other design choices that may be relevant for future optimisation.
Professional testing companies should deliver real world testers who match the target audience of your site – this can include age, gender and other demographic information, so that your testing feedback reflects the people whose behaviours and opinions matter most to your bottom line.
Conducted at scale, on relevant device and browser combinations, the right company will be able to quickly script and mobilise the right sized test group (from the appropriate demographic) to uncover the shifting and specific UX problems that are impacting revenue growth and holding you back.
Iterative usability testing is central to assessing the impact of changing designs and customer journeys of ever evolving feature-rich e-commerce sites. The insights delivered by these teams can help you make the right changes where they matter most.
For example, research by the Baynard Institute shows brands can increase conversion rate by more than 35% solely through better checkout design.
“Websites should deliver what your customers want or need in an engaging way that easily allows them to take action. A website that isn’t easy to use or understand… can result in lost sales, squandered internal resources and damaged brand reputation — costs that are too high for companies to absorb.”
Jane-Anne Mennella, Senior Director Analyst, Gartner
As brands become more focused on international opportunities, you need devices on the ground and real people with real local knowledge to help you spot issues that will impact conversion.
1. Payment expertise (different payment solutions prevail in different territories – common problems with integration need to be identified and remedied)
2. Testing the quality of your translations – don’t confuse users with clumsy, ambiguous working
3. Report on culturally inappropriate, insensitive or offensive content carried over from other sources – they will alienate users and could be a PR disaster at launch
4. Making sure the way translations are being rendered on local sites are not distorting boxes, making them stretched or otherwise unreadable
These needs, will of course, be continuous, as the speed of new releases of functionality increases and local conditions change.
It’s no good creating ever more exciting website features and user experiences if you can’t deliver those experiences fast and seamlessly. After all, research shows that more than half of visitors will leave a web page if they’re forced to wait longer than three seconds.
Retail red letter days like Black Friday and Cyber Monday, as well as unexpected news events and the rise of overnight trends, can bring sudden and unpredictable levels of custom to your website.
Human performance testing can concentrate on specific areas of the site to ensure it can cope with high levels of concurrent traffic with ease. A mass test on the payment section of a site can mimic the effect of a flash sale or other rush to your website, as the effect of multiple and simultaneous real world payment attempts are noted.
Effective stress testing like this also should let you isolate variables in order to determine the ability of your site to cope with particular events.
- Isolating all traffic to one particular geographic region
- Isolating performance across multiple browsers, platforms and devices
- Isolating the speed of image loading on specific devices and browsers
Continually optimising your website for performance is one of the ongoing strategic, testing priorities that ensures any business is able to cope with and thrive in periods of high, but unpredictable demand for your products or services.
Exploratory testing gives qualified, professional testers the freedom to follow their hunches, creativity, product and industry knowledge to uncover bugs and potential usability issues.
In scripted testing, you design test cases first and later proceed with test execution. But, exploratory testing is a simultaneous process of test design and test execution. It allows teams to replicate real world behaviour and decision making beyond a script, to take a thought wherever it leads them and help developers identify areas of weakness that can be tightened up.
Exploratory testing creates a constant feedback loop between testers and developers, it requires a high level of skill and it is a vital component of agile development.
Finding a professional testing partner who can deliver the full range of functional and non-functional testing in scripted and exploratory contexts, can be key to unlocking the full commercial potential of your site.
Every company is different. We all have unique customer offerings and want to communicate and deliver our distinct branding, features and benefits in compelling ways.
Digital experiences are the new battleground for this kind of cut through. Feature rich websites, constantly updated with new and exciting functionality deliver PR talking points, new opportunities for marketing engagement, and ultimately, more and more frictionless sales.
In this age of agile there is also a constant opportunity to make incremental changes that make a huge difference to the bottom line; to perform more informed A/B testing, personalising and refining customer experiences to deliver better results, customer loyalty and brand growth.
But, with so many moving parts, there is always the danger of bugs creeping in, slowing systems and frustrated customers who can’t do what they need to.
There are more reasons than ever to be on top of testing, to ensure quality keeps pace with the velocity of continual delivery and deployment – to prevent updates and changes wreaking havoc on your revenue stats.
But to do all this effectively, challenger brands need a flexible approach to testing which can lever its strategic benefits.
In the first place they need cost effective support for ongoing cycles of rock-solid, manual regression and other critical, functional testing.
This, in turn, will allow a company’s existing QA function and digital team to become more strategic, to focus on devising testing scripts, coordinating exploratory testing, and implementing conversion rate optimisation, rather than getting bogged down in the day-to-day business of juggling devices and handsets in rounds of never-ending testing.
Chosen wisely, an external, professional testing partner can give you the support you need for every kind of testing:
- Analytics can identify leaky browsers and revenue loss by browser/device combination
- Manual regression tests conducted at scale and speed across devices and operating systems to identify silent ’killer’ bugs and functional issues
- Ongoing testing at speed and scale to match the pace of release and innovation
- Professional testers can capture video of steps to illustrate issues and verify bugs
- Bugs and issues systematically identified and prioritised for actioning by developers
- Cycles of exploratory testing, with expert testers
Successful e-commerce stories are about the continual refinement of experiences, features and conversion, based on deep customer insight.
Choosing the right testing partner will mean you’ll never be left wondering ‘what just happened’ and you’ll always have a clear vision of what you need to do next to maximise sales and revenue.
Ultimately it will help you deliver greater operational efficiencies, more conversions, more revenue and all with less stress.