PageBuilder Editor: Bandito Content Testing Dashboard
Bandito Content Testing functionality is an add-on for Arc XP users. Contact Arc XP Customer Support if you want to use this functionality in PageBuilder Editor. For an overview of PageBuilder Editor's Bandito Content Testing feature, see PageBuilder Editor: Content Testing with Bandito Guide.
Overview
The Bandito Content Testing dashboard in PageBuilder Editor allows users to monitor all current and past content tests experiment and gain insights into your tests' performance with new metrics like spread, impressions, and lift.
How to access the Bandito Content Testing dashboard?
If your organization has Bandito content testing provisioned, you see the Optimizations tab in the PageBuilder Editor top navigation. Upon clicking it, a sub-navigation with the Content Testing option opens and takes you to the new Bandito Content Testing dashboard.
Note
All users with the Access PageBuilder Editor permissions can navigate to the dashboard.
How to use the Bandito Content Testing Dashboard
Upon opening the Content Testing Dashboard, you will see all of your active and past Bandito content test experiments in one place. When you create and publish a new experiment, it will dynamically appear in the dashboard. In this dashboard, you will see the following information for each test:
Column Title | Definition |
|---|---|
Test Block Name | Default block’s name. This will match whatever the block’s display name is for the page in Curate Workspace. |
Page Name | PageBuilder Editor page test is running on. |
Created Time | Time when Bandito content test was published and activated. |
# | Total number of variants, including the default block. Example: If you have a default block and 2 variants, the number in the column will be 3. |
Status | Details the latest status of the test. Options include: - In Progress - Paused - Inactive - Inactive: Winner Declared - Page Deleted For more information about the specific status definitions, see below. |
Default CTR | Click through rate for the default block (control block) |
Winner CTR | Click through rate for the winning block variant |
Spread | Calculates how much better the winning variant is doing compared to the default variant |
Impressions | Total number of times any variant was shown to users |
Lift | Number of clicks gained or lost by running the test. Based off if the test gained or lost clicks, the display will either be: - Positive number and green (clicks gained) - Negative number and red (clicks lost) For a more detailed explanation of lift, see FAQ. |
For active content tests, when you click on the test name, it will open the default block and the page it is on in PageBuilder Editor.
What are the different types of content test statuses?
The Bandito Content Testing Dashboard shows you both active and previous content tests. Content tests have one of the following statuses:
Status | Definition |
In Progress | The content test is published and active. There is no leading variant for the test yet. |
In Progress: Winner Declared | The content test is published and active. Convergence has occurred (the test has declared a winning variant), and it is ready to be closed. |
Inactive | The content test was published and closed, but no winner was declared. This may happen because a user closed the test manually by removing (deleting) the default block and/or a variant. |
Inactive: Winner Declared | The content test was closed and is no longer active because a winning variant was declared, and a user took some action to close the test. |
Paused | The content test has been paused. This happens when a page with an active test is unpublished. |
Page Deleted | The content test was in progress, but the page featuring the content test was deleted. |
FAQ
Q: What is lift, and how is it calculated?
Bandito is a multi-armed bandit test. Unlike traditional A/B testing, Bandito content testing routes traffic to the higher-performing variants and as a test continues, it will ultimately route nearly all visitor traffic to the winning block variant and automatically declare a winner. Lift refers to how many clicks were gained by running the test-- so consider it as what was gained or lost in terms of number of clicks (across the default and variants) by running the test versus if the test had never been executed and only the default variant (control block) was served to users.
Mathematically, lift is calculated as follows:
An example where lift is negative is if the default variant (control block) has an extremely high CTR and variants are being served to visitors, but there is low interaction with them. When the lift is low, it's important to remember that because of the nature of Bandito, the algorithm will automatically and dynamically route traffic to the higher-performing control block so your site visitors are still being served the optimized experience.
Q: The default CTR and Winner CTR for my content test are displaying the same information in the dashboard. Why is that?
If the Default CTR and Winner CTR are displaying the same number, this could mean one of two things:
The default block variant (the control block) is the leading/winning variant for your experience.
If the test has recently been published, it could mean that your default block variant (control block) and one of your variants are tied in terms of CTR. To see if this is the case, go to the block within PageBuilder Editor and hover over the beaker or eye icon.
Q: How do I remove Bandito Content Tests where the Page has been deleted from the Bandito Content Testing Dashboard?
Delete pages that had content tests appear in the dashboard.