We hope you are excited by how Lightning Mode can boost your experiment speed significantly. This article aims to provide a benchmark against our competitors (for propriety sake we have omitted their name and use XXO throughout instead) and explain how our upgraded pixel is faster and smarter.

We will first provide a benchmark against pixel size including just core functionality. The size affects how much data must be downloaded before your A/B test code can begin to run. Following on from this, we will make some different A/B tests, the typical modifications a user would make on their site, and compare 3 results:

Pixel Size – How much data is being downloaded
Pixel Load Time – How much time it takes to download and run the pixel code
Page Load Time – How long before your A/B test has been applied and the variation is displayed

Happy Reading!

Basic benchmark information

  • Internet connection speed: 20Mbps
  • Website size: ~1223KB
  • DOM tree size: 732 elements
  • Page loading time has been tested using GTMetrix tool
  • Original page load time (without any pixels): 4.40s
  • GZip compression disabled

Loading all pixel code with no modifications

[A/B Testing benchmark] Convertize is now 17.2x faster than XXO

Our Pixel has a few key advantages compared to the competition:

  • It doesn’t load any additional resources (JS/CSS) so its size is very small (56kB for all code). It requires only one HTTP request to our servers. XXO sends at least 4 different requests which has bad impact on the loading time.
  • It virtually doesn’t slow down customer’s website. Loading time is very close to the original loading time
  • It’s easy to install – one simple tag vs piece of JS code (XXO)

Comparison benchmark

Test #1

Benchmark steps:

  • Modify text of a 676 bytes long paragraph
  • Insert new paragraph with 128 bytes of text
  • Change content of the inserted paragraph (extend content with additional 600 bytes of data)
  • Advanced HTML action on the 4645 bytes long container
    • Add attribute to the element A (data-attribute=”test”)
    • Change style for element B (add color and background color)
    • Remove DOM element (40 bytes long)
    • Rearrange elements
  • Insert new image

Test #2

Benchmark steps:

  • Modify text of element A – remove 100 bytes
  • Modify text of element A – add 50 bytes
  • Modify text of element A – replace content with new text (128 bytes)
  • Advanced HTML on element A – add H1 (30 bytes)
  • Remove element A
  • Undo changes
  • Modify text of element A – add 1024 bytes

Test #3

Benchmark steps:

  • Rearrange element A after element B
  • Rearrange element A after C
  • Rearrange element A after D
  • Rearrange element A before element B (original position)
  • Advanced HTML on the parent of element A
    • Rearrange element A after element B
    • Add attribute to element A
  • Change style of element C
    • Change color
    • Change background-color
    • Add border
  • Use advanced CSS editor for element C
    • Remove all properties (should be original state)

Test #4

Benchmark steps:

  • Modify text of element A
  • Modify text of element A
  • Modify text of element A
  • Revert original text of element A (Pixel should do nothing)

Results

[A/B Testing benchmark] Convertize is now 17.2x faster than XXO

Pixel size

[A/B Testing benchmark] Convertize is now 17.2x faster than XXO

Pixel load time

 

[A/B Testing benchmark] Convertize is now 17.2x faster than XXO

Page load time

 

[A/B Testing benchmark] Convertize is now 17.2x faster than XXO

Smart Persuasion starts here!

Convertize.io offers Behavioral Messaging, Personalisation, A/B Testing & Optimization for eCommerce, Travel and Saas websites

TRY FOR FREE