If you’ve read our blog post on problem statement vs the hypothesis

So would this be a success or a failure?
Setting your criteria before an experiment goes live allows you to answer this question quite quickly. For the example above, you might state that in order for the experiment to be classed as a success, there should be an increase in newsletter sign-ups of at least 20% and a rise in conversions of at least 5% to outweigh the reduction in revenue per visitor.
In another scenario, you might acknowledge that whilst a decrease in revenue per visitor is almost unavoidable, you’ll only deem the experiment a failure if revenue decreases below a certain threshold for each visitor.
So the earlier the better?
Either way, thinking about the different scenarios that might arise within your experiment and categorising them as successes or failures allows you to decide on the outcome of your experiment more objectively than if you were to simply start considering this question after you’d conducted the test.
In addition, asking this question before the experiment even begins helps you consider scenarios in more depth. This allows you to speculate on the impact your test will have on different audience segments.
For instance, if you were to create a pop-up telling the customer they can get a 10% discount if they sign up to the newsletter, this is likely to work well on desktop, but you might speculate that a pop-up wouldn’t work as well on mobile because they’re often more intrusive and annoying for end users. Knowing this, you might conduct another experiment to see how an embedded form would work on mobile instead.
As you can see, considering different experiment outcomes earlier in the process can impact the depth and variety of your experiments. This eliminates any “mining” of positive results when analysing data which often sees businesses declaring winners for limited or irrelevant audiences. 

2. Gather and compile all your data – both quantitative and qualitative

Before launching head-first into creating a story as to why the experiment results turned out the way they did, we like to take a step back once our experiments are complete and gather all the data we need to fully understand the results. 
Data will primarily come from your experimentation or analytics platforms. In most cases, this is usually more effective than using only the former as the latter will help capture any metrics your experimentation platform cannot.
However, in some cases, qualitative research will run parallel to your experiments. This will help you understand the behaviour of users within each A/B/n variation.
Gathering all of this data from these sources in a central space will allow you to gain a broader picture of the way your users behaved within the test. Some results might surprise you, which can be a great opportunity to explore your data further. Who knows, it could even lead to further tests and better results.
Naturally, the results you discover will lead you to formulate “stories” that try to unpick the data and create reasons for why you’re seeing certain results. We encourage you to write these thoughts down and refer back to them when you get to step three.