If you’ve read our blog post on problem statement vs the hypothesis

If you’ve read our blog post on problem statement vs the hypothesis, you should already know how to set the foundation to a successful experiment. However, analysing your data and results is also one of the most important stages of any experiment. 
Unfortunately our cognitive biases and wishful thinking can often impact how accurately we are able to interpret data. At CCX, we’re experts in experimentation and have fine-tuned our methods for reducing bias throughout the analysis process. This article will lay out some top tips and the process we undertake to ensure data is never misinterpreted. 

Misinterpreting data: what impact does this have?

To answer this question, let’s consider the impact that correctly interpreted data can have on a business and its customers. With any experiment, we’re looking to gain valuable insights from the data. These insights will tells you what sort of impact an implementation change would have on our revenue and other key metrics. Correctly interpreting the data means we can be confident in the insights we have deduced for making changes permanently, fuelling future iterations or even testing a completely new hypothesis. 
On the other hand, misinterpreting data can lead you to implement changes that are actually detrimental to a business, its customers and most importantly its bottom line. This is only made harder by our innate cognitive biases and often yearning for a successful experiment. 

So what is the best approach to analysing your experiments?

At CCX, we’ve analysed hundreds of experiments and typically follow a five-step process. Taking such a thorough approach helps ensure we are confident in the results and learnings we provide to our clients.
The five steps are as follows:

1. Decide on the outcome of your experiment

Very simply put; you need to determine what the result of your experiment was. Did it succeed? Or did it fail?
Of course, the answer isn’t always clear cut. We’ll break down how best to reach a definitive answer in a moment, but this is a very important question to ask yourself. A wrong answer at this stage could result in you implementing a website change that doesn’t end up benefiting your customers or your bottom line.

When should you consider the experiment outcome?

Contrary to most other experiment implementers, we like to start thinking about the performance of the experiment before it’s even gone live. This helps you define the criteria needed to declare your experiment a success, not only for your primary metrics but also for any secondary metrics you are tracking. 
Defining your success criteria upfront will make it easier to decide the outcome of your experiment. 

Experiment outcome example

Say we’re offering visitors who sign up to our newsletter a 10% discount. 
Our primary metric would be to measure and compare newsletter sign-ups and our secondary metrics would measure other aspects such as:

• 
  Journey progression
  
• 
  Conversion rate
  
• 
  Revenue per visitor
  
• 
  Engagement metrics with form fields on sign-up
  

Our results are in and they show that newsletter sign-ups have increased and more users are progressing through the funnel, which has resulted in more conversions. However, our revenue per visitor has decreased because of a 10% discount.