Dynamic allocation optimization in A/B tests using classification-based preprocessing

Emmanuelle Claeys, Pierre Gancarski, Myriam Maumy-Bertrand, Hubert Wassner
2021 IEEE Transactions on Knowledge and Data Engineering  
In traditional A/B testing, for instance on two webpages A and B, the objective is to decide which of these two pages is the best. To do that, a frequentist test can be used in which each page is randomly or alternatively chosen and applied to incoming website visitors for a given time. However, one problem with this approach is the non-adaptivity of the test. For example, if one page quickly appears as having a very stronger positive or negative impact than the other one, the test could be
more » ... ped earlier. One way to avoid this is to apply a bandit-based algorithm. Such an algorithm is able to automatically decides if a page should be chosen and applied more often than the other one. This approach, called dynamic allocation, allows to add adaptivity to the A/B test. However, bandit theory by traditional methods requires assumptions which are not always verified in reality. This is mainly due to the fact that the subjects tested are not homogeneous. We present our new method that finds the best variation for homogenous groups in a short period of time.
doi:10.1109/tkde.2021.3076025 fatcat:l6ifydtzdfgyjfyupwhajtit7a