A/B Testing Speed: The Lightning Mode makes A/B Testing 17.2x Faster
The challenge of any SaaS provider is to continuously find ways to improve the software, its performance and the user experience of its customers. As Convertize has evolved and acquired more customers in 2017, we have been faced with interesting challenges along the way. We have worked hard to find a solution that is secure reliable and provides faster AB Testing.
We have designed our software so that it requires a single piece of script (called a “pixel”) to serve AB testing experiments and behavioural messaging on any website.
How Can We Get Even Faster AB Testing Lightning Loading Speed?
Our pixel is now installed on thousands of sites and with over half a million requests per day, we need to answer 2 fundamental questions:
- How can we ensure our pixel is served as fast as possible
- How can we ensure our pixel runs an experiment as fast as possible
This article will give an overview of the technologies and concepts used in order to answer these questions. We conclude with the AB Testing benchmark we have conducted, showing that our pixel provides now up to 17.2 times faster AB Testing than a well-known alternative platform.
SmartEditor™: The smartest Visual Editor ever made
The process begins in our SmartEditor™, a suite of editing tools built using AngularJS on the front-end and Symfony on the backend, handling API requests.
As important as it is to make our pixel operate fast on our customers’ websites, we wanted to make sure the user experience of the Convertize application was also a pleasant one. We have therefore made a number of optimizations to ensure this is the case.
Optimization #1 – Loading a webpage into the SmartProxy™
The first step is to load the webpage into the application for the editing suite to be enabled. This happens using the Convertize SmartProxy™, which will retrieve the page and all assets using cUrl in order to have direct control over them for the purpose of editing.
Optimization #2 – Minimal Ajax requests
Each modification introduced by the user is called an Action. The user is able to introduce many changes to the webpage using the suite of tools provided.
Every time an action is made such as inserting an image or rearranging a bullet list, this is sent to the backend to be processed via an ajax request. To ensure we only send minimal amount of data to be processed, only the action being applied will be sent to the backend processor.
Optimization #3 – Merging decorator changes
Decorators describe DOM elements which exist in the original DOM tree. A single decorator represents one unique element (based on XPath). It contains information about the final version of the element.
If the user will introduce different changes on the same element, such as changing CSS color and then modifying the text, all modifications are merged into one representation of the element called a Decorator. This has the benefit that the pixel can quickly apply a set of changes on a single element and there is less data to store in the pixel, resulting in a smaller file to load.
Let’s assume that a user introduces the actions shown below (represented with JSON data). The example is simplified by introducing modifications on the same element, but the idea is the same as introducing modifications on multiple elements in a different order.
Figure 1. Example list of modifications of single element
All these changes will be merged into one single “decorator” action which represents the final version of the specified element. This object will be used inside the pixel and during initialization of the SmartEditor™.
Figure 2. Example decorator based on Figure 1
Optimization #4 – Merging repeat changes
For each element that has been modified in the editor, the pixel will apply that modification to the live webpage. It does this with O(1) complexity, meaning that if the same element is modified multiple times in history, the pixel will only store the very last modification.
Let’s assume that you have three paragraphs on your website and you introduce 30 text modifications for each of them (90 modifications in total). Rather than store all 90 modifications in the pixel, it will store only 3 (one modification per paragraph). If the paragraphs contain 500 characters, the pixel will grow by only 1500 bytes.
Optimization #5 – Merging DOM changes
DOM actions represent modifications which affect the structure of the Document Object Model. As an example, it can be rearranging a list, html editing and inserting an image merged into one unit called “DOM”. The key thing is that the backend merges all actions for the same element into one representation. See the following example JSON showing a number of editing actions: an inserted element, a couple of text changes followed by a css attribute update.
Figure 3. Example modifications based on insert action
All these changes will be merged into one action to be applied by the pixel, as follows:
Figure 4. Merge DOM modifications
Optimization #6 – Scalable modifications
It is important that the system is highly scalable. This not only means handling more traffic and load, but the system should not suffer performance issues when the amount of data increases.
When it comes to making modifications in the SmartEditor™, this means that if you edit the experiment and make 10,000 edits to it, the last edit will be processed just as fast as the first one .
How do we accomplish this? We create a “build” which contains the minimal amount of actions that the pixel must process. When a new edit is made, the optimization process will merge this into the existing build to generate a new updated build of data for the pixel. This process ensures data can be updated incrementally.
Optimization #7 – Handling dynamic content
After all modifications are applied, the system registers a Mutation Observer. When a new DOM node is added system attempts to match non-applied changes on this element and its children, which will allow users to apply the editing tools to the dynamic element.
Optimization #8 – Advanced editing mode
Advanced users who have worked with the HTML markup language are able to modify the HTML directly. The pixel is smart enough to only store the data for the elements that have actually been modified rather than text for the whole element.
This has the advantage of not storing unnecessary data, resulting in a smaller and faster pixel. No unnecessary changes are applied and therefore the page load time is decreased.
Let’s assume that a user has introduced the changes presented below. On the left side we have the original DOM structure of the specified element and on the right side we have the modified structure.
Figure 5. Example of advanced HTML modifications
The user has introduced many fundamental modifications. Both the HTML before editing and after editing will be compared using the diffDOM library. Let’s see an example of this using the online Pretty Diff tool.
Figure 6. Output of Pretty Diff
After a diff has been performed, only the elements that were changed will be added to the list of actions to process and add to our “build”.
Optimization #9 – Ensuring valid HTML
When the Advanced HTML editing feature is used, the application will make sure the HTML being entered is valid and not allow modifications to be saved if they break validity. You will see a red X where the problem occurs.
Figure 7. Example of HTML validator
SmartPixel Optimization: Creating a lightning fast loading pixel
Optimization #2 – Handling high load with Docker
As our customer base grew we knew it was time to re-architect our server infrastructure. We migrated from a Google Cloud environment to Amazon Web Services, at the same time introducing docker containers and autoscaling.
We will write about our move to AWS in another article and how it now allows us to handle any amount of traffic by scaling our services up automatically when required. Here is a diagram showing an overview of the AWS setup in regards to serving the pixel.
Figure 8. Server architecture of Pixel setup
Optimization #3 – Keeping all data in a single file
- All data, code and css required are embedded into a single file. Browsers can only handle so many concurrent connections per domain so the less HTTP connections the better.
- Any user detection information such as Geolocation and Device Detection are injected directly into the pixel by NginX so we don’t have to make further 3rd party API calls.
Optimization #4 – Dynamic pixel size
A user can enable or disable certain features for their experiment. For example, they may choose to modify the page only, or also add a SmartPlugin™.
To keep the pixel download size as small as possible we decided to take into account what is actually trying to be done in the experiment and only download the code necessary.
For example, as soon as the user turns off all experiments, the pixel becomes 0-byte in size. If they don’t use any SmartPlugins then the code for those plugins will not be part of the downloaded file.
Optimization #5 – Increase “perceived” page loading time
In order to avoid the FOOC issue (Flash of Original Content, also called Flickering effect), the pixel will briefly hide the content of the page until all SmartEditor™ modifications have been made.
Is this always necessary? What if the only change being made was to add a SmartPlugin™ like Scarcity onto the page? In this instance, there is no need to hide the original content because it is not being modified, we are only displaying a notification on top of the existing page.
The pixel is smart enough to detect the type of changes required for the experiment and perform as optimally as possible, ensuring the user will see the page appearing as soon as possible. Following is an example flow the pixel will follow to apply a split url variation.
Figure 9. Pixel flow for Split URL variation
Optimization #6 – Asynchronous conversion tracking
When an experiment is loaded onto a page, one of the tasks of the pixel is to record goals. This is accomplished by sending a HTTP request for a blank image with parameters to the tracking server and then listening for a successful status code.
It is important that these requests are non-blocking and aysnchronous where possible, to ensure the page load speed isn’t afffected and that the tracking happens behind the scenes without the user being aware. Following are examples of optimizations made by the pixel in relation to goal tracking:
- Page View goals are handled only when the experiment has been loaded into the page and the page has been displayed to the user. This is because the user experience has a much higher priority.
- When sending these tracking requests, we don’t need to wait for a response unless the variation being applied is a Split URL variation, in which case we must make sure all tracking was recorded successfully before redirecting the user.
Optimization #7 – Processing conversions with cookies and job queues
This is an area that has improved greatly to ensure backend processing of conversion data happens as efficiently as possible considering the high number of requests being sent per day.
With our previous setup we had an NginX server listening to conversion requests and logging the data into a file which was read by a Treasure Agent Daemon. The daemon would parse the log data for conversion and experiment participation entries and then spawn a script to insert the conversion into KairosDB (a timeline series interface to Apache Cassandra).
NginX was configured with the ngx_http_userid module to create 3rd party cookies automatically in order to identify clients by assigning them a unique identifier. The conversion script would query KairosDB with this unique identifier and determine whether the conversion was new or not (this is especially important to identify new vs returning visitors).
Following is a diagram showing our previous setup.
Figure 10. Old architecture for processing conversions
What we learnt as traffic grew was that this setup could not scale. Here were some of the problems we faced:
- Storing and querying KairosDB for unique identifiers was not a scalable solution. The more conversions we had, the slower the system became.
- Using Treasure Agent in this way was limited by the speed at which it could parse log files. We were unable to scale this out as the log entries had to be read in sequential order to handle cross-domain tracking.
After our engineers thought about what we needed to accomplish, we re-architected a solution that would automatically scale in our AWS infrastructure and apply conversion processing 200 times faster.
Our new solution has said goodbye to Treasure Agent and the NginX module for visitor identification. When the pixel will send a tracking request to the server, it is retrieved by NginX running inside a scalable docker service, which will proxy this through to a python WSGI application.
The application will now store all experiment participation and previous conversion data into a 3rd party cookie, meaning that identification of new vs returning users and conversions is done instantly without the need to query a database.
If the conversion is one we wish to track, it will be sent to an Amazon SQS queue for it to be handled by consumer python scripts that will pop items off the queue and process them, inserting data into KairosDB. As there is no querying to perform anymore for uniqueness, we simply insert data and write operations in KairosDB are lightning fast. We improved this speed further by creating a KairosDB and Cassandra cluster behind load balancers.
Here is our new setup showing the interaction between the browser making the conversion request and it finally getting tracked.
Figure 11. New architecture for processing conversions
Benchmark: Convertize.io now delivers 17.2x faster A/B testing than its competition
We have conducted a loading and speed benchmark against our competitors (for propriety sake we have omitted their name and use XXO throughout instead).
We first provide a benchmark against pixel size including just core functionality. The size affects how much data must be downloaded before your A/B test code can begin to run. Following on from this, we will make some different A/B tests, the typical modifications a user would make on their site, and compare 3 results:
Pixel Size – How much data is being downloaded
Pixel Load Time – How much time it takes to download and run the pixel code
Page Load Time – How long before your A/B test has been applied and the variation is displayed
Basic A/B testing benchmark information
- Internet connection speed: 20Mbps
- Website size: ~1223KB
- DOM tree size: 732 elements
- Page loading time has been tested using GTMetrix tool
- Original page load time (without any pixels): 4.40s
- GZip compression disabled
Loading all pixel code with no modifications
Our Pixel has a few key advantages compared to the competition:
- It doesn’t load any additional resources (JS/CSS) so its size is very small (56kB for all code). It requires only one HTTP request to our servers. XXO sends at least 4 different requests which has bad impact on the loading time.
- It virtually doesn’t slow down customer’s website. Loading time is very close to the original loading time
- It’s easy to install – one simple tag vs piece of JS code (XXO)
- Modify text of a 676 bytes long paragraph
- Insert new paragraph with 128 bytes of text
- Change content of the inserted paragraph (extend content with additional 600 bytes of data)
- Advanced HTML action on the 4645 bytes long container
- Add attribute to the element A (data-attribute=”test”)
- Change style for element B (add color and background color)
- Remove DOM element (40 bytes long)
- Rearrange elements
- Insert new image
- Modify text of element A – remove 100 bytes
- Modify text of element A – add 50 bytes
- Modify text of element A – replace content with new text (128 bytes)
- Advanced HTML on element A – add H1 (30 bytes)
- Remove element A
- Undo changes
- Modify text of element A – add 1024 bytes
- Rearrange element A after element B
- Rearrange element A after C
- Rearrange element A after D
- Rearrange element A before element B (original position)
- Advanced HTML on the parent of element A
- Rearrange element A after element B
- Add attribute to element A
- Change style of element C
- Change color
- Change background-color
- Add border
- Use advanced CSS editor for element C
- Remove all properties (should be original state)
- Modify text of element A
- Modify text of element A
- Modify text of element A
- Revert original text of element A (Pixel should do nothing)
Loading Time Results
Pixel load time
Page load time
You can see out of these tests that our pixel is now up to 17.2x faster than XXO, which is massive difference!
About the future
In this article, we have outlined SmartEditor™ and Pixel optimisations–the result of our hard work, and our dedication to providing you with a better experience. This development signifies enormous improvement and innovation to our optimisation platform, allowing our solution to work smarter and faster than any other optimisation product on the market. And yet, we are aware that this is still just a stepping stone along the way to perfect optimisation. When you’re committed to growth as much as we are at Convertize, “progress is inexhaustible” and change never stops. We will continue to look for ways to make your experiences smarter and faster.
With that in mind, enjoy Lightning mode and stay tuned!
Author note: I would like to thank David and John for their hard work and endless knowledge. Without their extensive contribution none of this would have been possible.