Now that I have a fairly robust way to track the unique visitors of my website, I need to explore a way to use that. A said earlier, I want to perform some split testing.

What is split or A/B testing?

In short, you split you user base randomly in 2 or more groups and you provide each group with a given version of your page or website. You also define some conversion goals, for example accessing a given page and then you track which group has the best conversion rate.

If you want to learn more, Jesse Farmer has a good introduction to A/B testing.

How to implement A/B testing?

You will need to define:

  • a test case;
  • and the corresponding conversion goal of the test;
  • for each test, some alternatives;
  • then you will need to log which alternative or treatment each visitor got;
  • and log for each user the conversion success.

Of course, you need to be able to track the individual users accessing your website and as such you need a good unique user tracking method.

A bit more details of what to store

The test case should store:

  • name and description;
  • the percentage of visitors to test;
  • status (running or not).

The conversion goal:

  • name;
  • URL of the goal or regex to match the goal;
  • a reference to the test.

The alternatives:

  • name;
  • the slot (see treatment below);
  • the alternative text in HTML or what is needed;
  • a reference to the test.

For the treatments, you need to think a bit more further than A/B testing and go into multivariable testing. Multivariable testing is the fact that instead of testing one or another alternative (A or B), you test several alternatives on the same page/website at the same time. It is basically an extended version of the A/B testing method. You can consider it as a huge A/B test with several alternatives based on all the combination of elements.

Imagine you have a webpage with a banner and a title (2 slots, one banner and one title) and you want to test 2 different banners B1 and B2 and 3 different titles T1 to T3. This means you have 6 possible treatments: B1T1, B1T2, B1T3, B2T1, B2T2 and B2T3.

You could avoid the definition of the slots by creating directly the 6 alternatives, but that would take some manual time and you will lose the ability to do a bit of statistical kung fu and take a shortcut with the number of tests to run to get some meaningful results.

So, a treatment should store:

  • a reference to the test;
  • which alternatives it includes.

Now you will need to deliver the treatment to the users and log which treatment has been given to whom and what was the outcome.

The treatment log should store:

  • a reference to the test;
  • the delivered treatment;
  • which user got it.

The conversion log should store:

  • a reference to the test;
  • a reference to the goal;
  • and a reference to the visitor.

This structure allows us to track different goals at the same time, this can possibly save time.

The A/B testing base workflow

The workflow is relatively simple, you basically do the following:

  • If the visitor is new for this test, find randomly if you need to test it based on the percentage of visitors to test;
  • if you do not test it, just send the base case and log it as "base treatment";
  • If you need to test it, select randomly a treatment, log it in the treatment log and deliver it.
  • If the visitor is not new for this test, send him the already assigned treatment.

On each page, check if an active goal is matched, if so, log the conversion success for this visitor.

What is important is to never send 2 different treatments for a given user.

What to do next?

Implement that to do some funny testing on the homepage of indefero.net and report you the results.