How do you prove statistical significance when determining the winning candidate of a given experiment?
- The winning combination (candidate) is tested to statistical significance. When testing each generation the system explores the all of the candidates. Upon detecting a level of separation between the candidates, the system then explores the candidates that seem to be winners. The science behind it is evolutionary algorithms.
- The winning candidate also benefits from the testing done to its individual elements in earlier generations
How do you know you haven’t made a mistake in pruning a candidate that could potentially be a winner?
- A candidate is not pruned permanently. It still has a chance for ‘survival’ when it’s elements are brought back as part of mutation. It would then be tested in combination with other elements in future generations to see if that combinatory effect makes it a winner. If it continues to fail the system determines that variation is a losing candidate.
Why do you not need to test all of the possible combinations?
- A/B tests or even current multivariate are limited to the number of variables they use because their testing approach would take too long to try them all.
- Ascend's genetic algorithm does not need to try every possible combination. Instead, it will intelligently get closer and closer to the best solution. Therefore, you can use far more variables. Ascend is searching dozens of parts of the search space simultaneously.
- Traditional solutions can only test one section of the search space at a time - climb one hill. If your marketing team is good (or lucky) maybe they are able to chose a big hill that will increase lift. Ascend is effectively climbing all hills, at the same time, to find the highest one. [Ascend users are much less likely to become stuck in "local minima.]
- You could try all combinations to be absolutely certain you have the best solution at that point in time. But the incremental gain you would get compared to the time needed to try them all isn't worth it. The algorithms are very efficient at finding gains in a comparably shorter amount of time.
How many generations does it take to determine a winning candidate?
- It depends on how many total values and elements you are testing, the quality of the creative, and so on. You can expect to see some directional results in generation three, possibly sooner.
How do you calculate how to prune candidates?
- The solution tests to see if a candidate is an elite (converting well) or if it isn’t performing compared to the control.
- The system will then evolve by combining and crossing better performing candidates.
- Some candidates will become elite if in combination with others, so less performing candidates are reintroduced in new combinations thanks to mutation.
How much can I expect to increase my conversion rate?
- We’re able to cite and provide case studies of our customers who used our solution. We have customers in the e-commerce, lead generation, and content verticals.
- Increasing conversion rate depends upon the following factors:
- Quality of the creative
- Number of hypotheses or ideas tested
- How visually different the variations are:
- (e.g. a dark background button color, a light background button color)
- How long you’re running your experiment