CXL Growth Marketing Minidegree Week 3 Review

Source: Photo by Thought Catalog on Unsplash

In the 3rd week of my CXL scholarship, I started the A/B testing mastery course.

Here I learned about:

• The value of A/B testing.
• The ROAR model for deciding when to run A/B tests.
• Key Performance Indicators.
• The FACT and ACT research model.
• The 6V Conversion Canvas for conducting research.

Let’s get right into the goodies.

1. Why and When Should You Do A/B Testing?

A/B testing helps you make better, more well-thought and trustworthy decisions.

If you have a business and you want to get valuable insights about your audience, you should do research using A/B tests.

“But Tanyo how much data do I need to run A/B tests?”

A great way to think about this question is using the ROAR model.

2. The ROAR (Risk, Optimization, Automation, Re-think) Model:

1) Risk

If you’re in this phase, you’re probably starting your business and don’t have a lot of customers. Likely, you won’t have enough data to run A/B tests.

As a general rule of thumb, under 1000 conversions, you can’t do split testing.

That leads you to try a lot of different ideas for increasing your conversions. If you find something that has a 15% impact on your conversions, then you have discovered a winning idea that will help you expand your business.

To get to the optimization phase, you need to have at least 1000 conversions per month to do proper A/B testing.

2) Optimization

After you get above the 1000 conversions threshold, you can experiment with different A/B tests. By this point, you already have an optimization team that can run A/B tests.

At this level, the winner of your A/B test needs to have at least a 5% impact. To get to the automation phase, you need to get a minimum of 10 000 conversions per month.

3) Automation

In the 3rd phase of the ROAR model, you need to have an automation team that can systemize your tests and automate them. At this point, your company is at a mature level where you’re making a lot of sales per month.

However, to get to the last phase of the ROAR model, you’ll need to have at least 100 000 conversions per month.

4) Re-think

At this level, you’re a huge company like Amazon or Apple, that can run thousands and thousands of experiments each month. Here, the winners of A/B tests have a small impact. However, due to the scale of the company that impact results in a lot of profit.

Here’s an excellent A/B test calculator.

Here’s how to use it:

• Put your significance at least 90% to avoid getting too many false positives.
• When you start testing, use at least 80% power to understand who are your winners.
• When you start, put your significance at 90% to remove too many false positives.

“Statistical power is the likelihood that an experiment will detect an effect when there is an effect there to be detected.”

3. Key Performance Indicators (KPI)

Each A/B test has different performance indicators like:

• Clicks — that are easy to improve but don’t have a huge impact on your business.
• Behaviour — easy to optimize but also doesn’t make a great impact.
• Transactions — if you truly want to optimize, you should start optimizing here.
• Revenue per user — it’s harder to optimize, but it has a positive impact on your business.
• Potential Lifetime Value — this is the one to optimize. However, it’s also the hardest indicator to improve.

If you’re a mature company, you should optimize things like revenue per user or potential lifetime value.

If you’re not a mature company (below 10 000 transactions per month), you should focus on clicks, behaviour, transactions.

Here are some things to keep in mind:

• A/B test tools and calculators are only comparable with binary variables (0/1) meaning you either convert or you don’t.
• KPIs can’t be user satisfaction, number of page views, etc. You should look for success indicators and avoid vanity metrics.
• Your company should have a good OEC (overall evaluation criteria) for everyone in the company. OEC is defined by using short-term metrics that predict long-time value.

4. The FACT and ACT Model

FACT stands for find, analyze, create, test.

In the finding and analyzing phase, the main question is “What to optimize?”

Key Insight: Without an analyzed reason to start, it doesn’t make sense to start at all.

Your customer behaviour research goals should give you:

• Insights in the most important customer journey.
• Understanding of the basic behaviour of your customers.
• An input for setting a hypothesis.

In the creating and testing phase of the model, you’re doing a lot of experiments based on the information you analyzed.

Then, you start implementing the ACT model:

• Analyzing the results.
• Combining them.
• Telling the results to your execs.

5. SIX V Conversion Canvas

• Value

Here, you should ask yourself two questions:

1. What company values are important and relevant?
2. What focus delivers the most business impact?

Key Insight: You need to know the product focus and the KPIs focus.

• Versus

Here, you’re doing a full analysis of your competitors and their best practices.

You can buy from them to understand more about their customer journey or track their website changes. Also, you can learn which tools they are using on their website and even find their A/B tests.

• View

Here, you’re looking at your own web analytics and behaviour data to learn more about your customers.

Questions to ask yourself when looking at your web analytics:

❓ Where do visitors stay on the site?

❓ Where do they come from?

❓ What is the flow of those visitors?

❓ Are there notable differences between segments or products?

❓ What’s the behaviour on the most important pages?

Questions to ask yourself when looking at your landing pages:

❓ Where do visitors start their journey?

❓ What’s the difference between new and existing customers?

❓ What’s the difference per devices?

Questions to ask yourself when looking at your traffic sources:

❓ Where do visitors come from?

❓ Do they already have a product in mind?

❓ Do they already know the brand?

Questions to ask yourself when looking at different customer journeys:

❓ What’s the CTR (and return rate) to the next step?

❓ What’s the exit rate and time spent on a page per step?

Here’s a typical e-commerce flow example:

💡 All users on your website with enough time to take action.

💡 All users on your website with at least some interaction.

💡 All users on your website with heavy interaction.

💡 All users on your website with clear intent to buy.

💡 All users on your website that are willing to buy.

💡 All users on your website that succeed in buying.

💡 All users on your website that return with the intent to buy more.

Here are different measurements you can use when conducting research:

📊 The number of templates.

📊 The time on a page.

📊 Heat and scroll maps.

📊 Screen recordings.

• Voice

Here you should ask yourself “what insights can be taken from the voice of the customer data such as surveys, feedback and service contact?”

Here’s what you can do to gather more data:

📝Talk to your customer service.

📝Check chat logs.

📝Check social media feedback.

📝Ask for feedback online.

📝Interview customers.

📝Create focus groups.

📝Make usability studies.

📝Recruit testers.

• Verified

Questions to ask yourself here are:

❓ What scientific research, insights and models are available?

❓ What do we know from scientific literature?

“If there’s an industry, there are studies done about them.”

Here, useful resources are things like Google Scholar, Semantic Scholar, DeepDyve.

• Validated

A question to ask yourself here is:

❓ What insights are validated in previous experiments and analyses?

At this point, you take into account things like assumptions on behaviour and motivation as well as assumptions on emotion & rationality.

That’s all for this week, folks.

Thank you for reading this post!

With this picture, I wish you a productive, healthy and happy week:

Source: Photo by Emma Matthews Digital Content Production on Unsplash