PageBuilder Editor: Content Testing with Bandito Guide
Overview
In this guide, we will walk you through all that you need to know about Content Testing in PageBuilder Editor.
Content Testing functionality (also referred to as Bandito) is an add-on for Arc users. It is only available to organizations that have opted for it. If you are interested in using this functionality in PageBuilder Editor, contact Arc XP Customer Support.
Tests in PageBuilder Editor relate to Blocks, not article content (for instance, Content Testing allows you to test content on Blocks or Chained Blocks). The most common tests are the ones that change headlines, but you can change anything in the Feature configuration (for example, add different photos, change the byline, change the headline, change the story blurb).
The following criteria must be met to execute Content Tests in PageBuilder Editor successfully -
You can create Content Tests only on Pages and not on Templates.
Note
Content Testing only works on Published Pages. Tests created on Versions of a Page that are not Published will NOT start. You must first publish the Page before creating content tests.
You can create Content Tests on Blocks and Chained Blocks only.
Content Testing is not yet supported for:
Feed-based blocks
Chains
Linkable Blocks/Chains/Chained Blocks on parent Pages
Linked Blocks/Chains/Chained Blocks on child Pages
Note
Currently, Variants are supported for only non-static components.
How to create and initiate Content Tests in Editor
Here is a step-by-step guide for creating Content Tests.
Step 1: Create Variants for Content Testing
To create Variants for a Content Test on a block (that meets the above mentioned criteria), click on the three dots and select variant (previously New Variant). A new Variant “variant-1” will be created. variant-1 will have the same values as the original block by default.
Note
The original block itself is considered as the Control Variant - the default, and any subsequent Variants will have a number suffixed to them (for example, variant-1, variant-2 etc.). You will need at least one Variant in addition to the Control to create a Content Test.
Step 2: Update the Variant details
Click on the Control (Default) or Variant to get into the block configurations and update the details such as headline, blurb etc.
When you return to the Blocks Navigator, you will see a branch icon next to the block. Click on the branch icon to view the list of Variants.
Step 3: (Optional) Create and update more Variants
This step is optional. Create and update any additional Variants that you may need as described in step 1 and step 2 or by cloning a Variant (see Variant actions).
Step 4: Preview Variants
Hover on the Control or Variant to preview in the Editor Preview and make sure that all the Variants are updated as expected. For more details on previewing, refer to How to Preview Variants? section.
Step 5: Publish Page to initiate the Content Test
This is a very important step. Unless you publish your Page, Content Tests will only be created in the Editor but will not actually be initiated. To publish your Page, use “Share & Publish” (in most cases and the preferred option) or the advanced option - “Publish My Draft”.
Note
Before publishing, tool tip on the branch icon hover will indicate that the test has not been initiated.
How to update Content Tests?
You can update Content Tests when they have been created in PageBuilder Editor and are:
not yet initiated (not yet Published)
initiated (currently running)
To update Content Tests, you can:
add new variants to an existing Content Test by using either the Create Variant option on the Control or Clone option on the Variant.
edit existing variants - update Control or any existing Variant
click on the Control (Default) or Variant to get into the block configurations and update the details such as headline, blurb, image etc.
remove existing Variants (refer to Variant actions section).
Important
Do not forget to publish your Page for any Content Test updates to take effect.
As long as the Test is active, each visitor will be served appropriate Variants.
How to Preview Variants?
Click on the branch icon to expand and view all the Variants on a block. When you hover on the Control (Default) or Variant, the Editor Preview will reflect the corresponding details.
How to track your Content Tests?
Here is how you can track the progress of your Content Tests -
When a Test has been created but not yet initiated
 |
When a Test is in progress
 |
When a Test has a winner
 |
The leading Variant or the winning Variant when a Test has a winner displays the CTR% (click -through rate %)
If the Test has been running for a while and there still is no winner then you may either want to modify the Test to try more things or close the Test completely.
What happens when a Test has a winner?
While you are working on a Page and there is a winner on one of your Content Tests, this is what happens -
First, the branch icon on the block with a winner expands to display all the Variants and you will see a trophy icon next to the winning Variant. The winner could be the Control (Default) or any other Variant.
Next, you will see a modal indicating the winner and the CTR (click-through rate) details. You can choose Yes, Close Test to automatically close the Test or Cancel to do so manually.
If you choose to close the Test on the modal
the winner (Control or any other Variant) will be retained and rendered as the block
trophy and branch icon will be removed
all the other non-winning Variants will be removed from Block Navigator
Winner will be automatically rendered. There is no need to publish the Page.
If you click Cancel
No matter the reason for closing the Test manually, you will want to keep either the Control, or one of the Variants. Closing the Test manually is done slightly differently depending on the Variant you want to keep.
Note
In this state, you will not have Create Variant or Clone since the Test already has a winner.
Scenario: Keep the Default Variant and close the Test
When the Control is not the winner and you still want to retain it over the other winning Variant -
Click on the three dots next to each Variant you want to remove and choose Remove to manually remove each Variant.
when there are no other Variants, the branch icon on the Control will be removed. There will be no Content Test on that Feature.
Publish the Page to reflect any changes you made and close the test.
Scenario: Keep one of the Variants (non-Control) and close the Test
If instead of the Control you want to keep another Variant, click the three dots on the Variant you want to keep, and click on Copy Variant as Block. This will create a new block with the same Variant configurations.
Remove the Control (Default) block itself. All the Variants for the Block will also be removed.
Publish the Page to reflect any changes you made and close the test.
Note
Ideally, you should follow the “Yes, Close” flow and automatically close the tests, but the Cancel flow gives you the option of:
choosing a non-winning Variant and/or
converting Variants to Features after the Test has a winner.
If you do not remove all the Variants and close the Test, the Test will keep running in the background and those Variants/ Tests will linger on the Editor UI too.
It is recommended that you close the Test to avoid the overhead of maintaining the Test even after there is a winner and keeping the non-winning Variants in the Editor UI.
As a reminder, when you refresh your Page or return to the Page, you will be prompted to close any Test(s) that have not been closed even after a winner has been found.
Special scenarios
1. You can remove the Control (Default) block itself for a Test that has been created in Editor
a. but not yet initiated (not yet Published)
The Control (Default) and Variants are removed from the Editor.
- If the Test is only present in My Draft (not yet shared with others), the Test will be completely removed.
- If the Test has been shared with others - You need to Share Edits for others to see that you have removed the Test.
b. and initiated (currently running)
You will see a confirmation modal
- On confirming, the Control (Default) and Variants are removed without having to publish the Page.
- On canceling, there will be no change.
2. When you clone the Control (Default) block or add the Control (Default) to/from Clipboard,
a new block is created along with the Variants.
Display Name for the cloned block has a numbered suffix.
the Page needs to be Published for this new cloned block to have the Content Test initiated.
3. When you have a linkable Feature/ Chained block on a parent Page (i.e a Feature with Link ID in the Linking card),
you will not have the option to create new Variants, and thus you cannot create Content Tests on them.
 |
4. When you have Variants setup on a block/ chained block,
Linking card will not be available for the Control or the Variant.
 |
5. When you have linked block on a child Page,
you will not have the option to create Variants.
you will have to remove the link to be able to create Variants.
 |
6. When you copy a Page to a new Page or create a new Version of a Page that has Content Tests,
Variants on block/ chained blocks are not copied over to the new Page.
Variant actions
 |
Remove - You can remove a Variant by using this option.
Clone - You can create an exact copy of an existing Variant and update any necessary configurations.
Copy Variant as Block (previously Variant to Block) - You can change a Variant to be its own Block. The Variant is also retained at its original spot.
Future considerations
We plan to work on a few enhancements such as surfacing additional Content Testing analytics in PageBuilder Editor, and indicating which Features are enabled for Content Testing by Feature developers.
FAQ
Q.1 What is Bandito, and why do I need Content Testing?
A: See Getting Started with Content Testing and Bandito. PageBuilder Editor uses APIs from Bandito, an implementation of multi-armed Bandit to serve Variants to users and determine the leading and winning Variants.
Q.2 Can I create Content Tests on a Template?
A: No, Content Tests can only be created on Pages. You will not have the Content Testing options such as “Create Variant” available in the Editor UI for Templates.
Q.3 Do I need a Layout setup on my Page to create Content Tests?
A: No, Content Tests can be created on a Page with or without a Layout.
Q.4 Can I create Content Tests on blocks?
A: Yes, Content Tests can be created on blocks and chained blocks. Content Testing is not supported on chains and linkable or linked blocks/ chains/ chained blocks or feed-based blocks.
Q.5 Will Content Tests be initiated when I share edits?
A: On “Share Edits”, Content Tests will be shared with others similar to any other My Draft changes. But Content Tests will be initiated only when you publish your Page - “Share & Publish” (in most cases and the preferred option) or the advanced option - “Publish My Draft”.
Q.6 How do variants render on Pages?
A: Variants render on the client-side. This means that when a page initially loads, you may see the control variant flash and then the variant appears. This is expected behavior.
Q.7 What if a Test has a winner and I still prefer a different Variant?
A: See If you click Cancel.
Q.8 What if multiple tests converge at the same time while I’m working on a Page?
A: In the rare event that this happens, you will experience the flow similar to the winning scenario described above, except that there will be a modal confirmation for each Test with a result. Each modal will prompt you to close the specific test automatically or to keep the Variants to manage them manually.
Q.9 Why is Content Testing not working for me?
A: Here are a few things to review for Content Testing to work in PageBuilder Editor. Let us troubleshoot -
a) Do you see the Create Variant (previously New Variant) option on the three dots next to the Feature?
If not, do you know if your organization has opted for Content Testing functionality?
Content Testing is a functionality that comes at an additional cost to your organization. Contact Arc XP Customer Support if your organization is interested in using Content Testing.
b) Do you see “Test in progress” when you hover on the branch icon?
If you see the test is in progress for quite some time (the hover also indicates how long the test has been running), the test has not converged yet, meaning there is no clear winner. So, try modifying your Content Test (refer to the section on Update tests) or make sure there is enough traffic on the Page. Content Testing needs sufficient number of clicks before it can identify a winner. Try creating Content Tests on a Page with high traffic.
c) If all looks good, there is one more thing to check with Feature developers.
Feature developers need to include the registerSuccessEvent() method in the code for Features to enable Content Testing. For more details, check this documentation on RegisterSuccessEvent() Method.
Note
In the future, we are planning to extend support for Feature developers to specify when Features have Content Testing enabled and present that information in the Editor to indicate which Features are setup and available for Content Tests.
d) For more information on troubleshooting and additional scenarios, see Troubleshooting Content Testing with Bandito in PageBuilder Editor.
Q. 10 How does Bandito work with Micro Experiences?
A: Bandito works only on default PageBuilder experience. Secondary Micro Experiences are not compatible with Bandito.