16,500 Hits in 5.8 sec

Online Learning for Non-Stationary A/B Tests [article]

Andrés Muñoz Medina, Sergei Vassilvitskii, Dong Yin
2018 arXiv   pre-print
In this work we formulate this question as that of expert learning, and give a new algorithm Follow-The-Best-Interval, FTBI, that works in dynamic, non-stationary environments.  ...  This kind of A/B testing is ubiquitous, but suboptimal, as the monitoring requires heavy human intervention, is not guaranteed to capture consistent, but short-term fluctuations in performance, and is  ...  Conclusion In this work, we study the setting of monitoring non-stationary A/B tests, and formulate an expert learning framework for this problem.  ... 
arXiv:1802.05315v2 fatcat:tvxy25qsg5apbdmp7ynmdb4gji

Beyond A/B Testing

Timothy NeCamp, Josh Gardner, Christopher Brooks
2019 Proceedings of the 9th International Conference on Learning Analytics & Knowledge - LAK19  
However, traditional randomized experiments, like A/B tests, are limiting in large scale digital learning environments.  ...  We also provide practical advice for researchers who aim to run their own SRTs to develop adaptive interventions in scaled digital learning environments.  ...  In [10] , A/B tests were used to test the effectiveness of self-regulated learning strategies.  ... 
doi:10.1145/3303772.3303812 dblp:conf/lak/NeCampGB19 fatcat:y66f7hudjnb7ta56meir2ti3ua

Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework [article]

Chengchun Shi, Xiaoyu Wang, Shikai Luo, Hongtu Zhu, Jieping Ye, Rui Song
2021 arXiv   pre-print
The aim of this paper is to introduce a reinforcement learning framework for carrying A/B testing in these experiments, while characterizing the long-term treatment effects.  ...  A/B testing, or online experiment is a standard business strategy to compare a new product with an old one in pharmaceutical, technological, and traditional industries.  ...  First, we introduce a reinforcement learning (see e.g. Sutton & Barto, 2018) framework for A/B testing.  ... 
arXiv:2002.01711v5 fatcat:5vakh2geuffyhjkbn6dhen4v3i

Dynamic allocation optimization in A/B tests using classification-based preprocessing

Emmanuelle Claeys, Pierre Gancarski, Myriam Maumy-Bertrand, Hubert Wassner
2021 IEEE Transactions on Knowledge and Data Engineering  
In traditional A/B testing, for instance on two webpages A and B, the objective is to decide which of these two pages is the best.  ...  However, one problem with this approach is the non-adaptivity of the test.  ...  In summary, the proposed method Ctree-Ucb consists of two main steps: • an offline process for creating groups. • an online A/B test.  ... 
doi:10.1109/tkde.2021.3076025 fatcat:l6ifydtzdfgyjfyupwhajtit7a

ROI Maximization in Stochastic Online Decision-Making [article]

Nicolò Cesa-Bianchi, Tommaso Cesari, Yishay Mansour, Vianney Perchet
2021 arXiv   pre-print
We design an algorithm for learning ROI-maximizing decision-making policies over a sequence of innovation proposals.  ...  A significant hurdle of our formulation, which sets it aside from other online learning problems such as bandits, is that running a policy does not provide an unbiased estimate of its performance.  ...  Repeated A/B testing.  ... 
arXiv:1905.11797v6 fatcat:gpdesazdezg55bmi4piwzkub2q

Causality Testing: A Data Compression Framework [article]

Aditi Kathpalia, Nithin Nagaraj
2018 arXiv   pre-print
This measure is rigorously tested on simulated and real-world time series and is found to overcome the limitations of Granger Causality and Transfer Entropy, especially for noisy and non-synchronous measurements  ...  This motivates us to propose, for the first time, a generic causality testing framework based on data compression.  ...  We perform a straightforward extension of the above mentioned procedure (ET C(X)) for computing the joint ETC measure ET C(X, Y ) for a pair of input time series X and Y of the same length.  ... 
arXiv:1710.04538v2 fatcat:ckuatv7cezf7nhpr53g3n3n6hm

Classification methods for noise transients in advanced gravitational-wave detectors II: performance tests on Advanced LIGO data

Jade Powell, Alejandro Torres-Forné, Ryan Lynch, Daniele Trifirò, Elena Cuoco, Marco Cavaglià, Ik Siong Heng, José A Font
2017 Classical and quantum gravity  
This work provides an important test for understanding the performance of these methods on real, non stationary data in preparation for the second advanced gravitational-wave detector observation run,  ...  We show that all methods can classify transients in non stationary data with a high level of accuracy and show the benefits of using multiple classifiers.  ...  Acknowledgments We thank Salvatore Vitale, Reed Essick and the Burst and DetChar groups of the LIGO Scientific Collaboration for helpful discussions of this work.  ... 
doi:10.1088/1361-6382/34/3/034002 fatcat:p3d3nprxtvdvvigq4rpnteonn4

A t-test for synthetic controls [article]

Victor Chernozhukov, Kaspar Wuthrich, Yinchu Zhu
2022 arXiv   pre-print
Our t-test is easy to implement, provably robust against misspecification, valid with non-stationary data, and demonstrates an excellent small sample performance.  ...  An -package for implementing our methods is available.  ...  We write a b to denote a ≤ cb for some constant c > 0 that does not depend on the sample size. We write a b to denote a b and b a. For a set A, |A| is the cardinality of A.  ... 
arXiv:1812.10820v7 fatcat:hjt3qhgqjvadhhxynihacldlpq

An Empirical Testing of Autonomous Vehicle Simulator System for Urban Driving [article]

John Seymour, Dac-Thanh-Chuong Ho, Quang-Hung Luu
2021 arXiv   pre-print
In comparison with road tests, simulators allow us to validate the AV conveniently and affordably. However, it remains unclear how to best use the AV-based simulator system for testing effectively.  ...  On the other hand, we noted that the system failed to detect pedestrians or vehicles on the road in three out of four classes, accounting for 10.0% total number of scenarios tested.  ...  We also would like to thank the Organising Committee of IEEE Autonomous Driving AI Test Challenge for helpful training and support, as well as two reviewers for useful comments.  ... 
arXiv:2108.07910v2 fatcat:5bs5qzfmifhdpocdk2bq6ntlwe

Fast Unsupervised Online Drift Detection Using Incremental Kolmogorov-Smirnov Test

Denis Moreira dos Reis, Peter Flach, Stan Matwin, Gustavo Batista
2016 Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '16  
Two major features distinguish data stream from batch learning: stream data are generated on the fly, possibly in a fast and variable rate; and the underlying data distribution can be non-stationary, leading  ...  However, specifically for the classification task, the majority of such methods rely on the instantaneous availability of true labels for all already classified instances.  ...  |A| + |B| − 1 |A| + |B| oi o1 o2 . . . o |A|+|B|−1 o |A|+|B| G(oi) g1 g2 . . . g |A|+|B|−1 g |A|+|B| for each observation, we also have a corresponding value gi = G(oi).  ... 
doi:10.1145/2939672.2939836 dblp:conf/kdd/ReisFMB16 fatcat:cxpo2o7uz5a7bnzuutszbmrljy

Testing the efficient market hypothesis on the Nairobi Securities Exchange

Josephine Njuguna
2017 Investment Management & Financial Innovations  
Results indicate that we cannot accept the EMH for the NSE using the serial correlation test, unit root tests and the runs test. Overall, the Kenyan market is found to not be weak-form efficient  ...  To test weak-form efficiency in this market, this study uses the serial correlation test, unit root tests (ADF and Phillips-Perron) and runs test.  ...  Therefore, if the time series has a unit root, it is non-stationary.  ... 
doi:10.21511/imfi.13(3).2016.06 fatcat:cutnlrkxyfgazbomt4mi2eyopq

Rapid Regression Detection in Software Deployments through Sequential Testing [article]

Michael Lindon, Chris Sanden, Vaché Shirikian
2022 arXiv   pre-print
We present a statistical framework for rapidly detecting regressions in software deployments. Our approach is based on sequential tests of stochastic order and of equality in distribution.  ...  For Internet companies, this has the potential to degrade the user experience and increase user abandonment.  ...  This approach is related to an A/B test design found in online controlled experiments.  ... 
arXiv:2205.14762v1 fatcat:blfxolsygzbvjporqg6gigt3ku

Hierarchy, Morphology, and Adaptive Radiation: a Test of Osborn's Law in the Carnivora [article]

Graham J Slater, Anthony R Friscia
2018 bioRxiv   pre-print
Yet, despite much theoretical development and empirical testing, questions remain regarding the taxonomic levels at which adaptive radiation occurs, the traits involved, and its frequency across the tree  ...  Strong support for an early burst adaptive radiation is recovered for molar grinding area, a key proxy for diet.  ...  uncertainty in within-regime estimates, then Ψd(A, B) will be negative and a statistical test of whether the two regimes share a common R can be computed as P [Ψd(A, B) > 0].  ... 
doi:10.1101/285700 fatcat:q2zfxhfv75ewzbxnjumyep4gz4

Model-Powered Conditional Independence Test [article]

Rajat Sen, Ananda Theertha Suresh, Karthikeyan Shanmugam, Alexandros G. Dimakis, Sanjay Shakkottai
2017 arXiv   pre-print
We consider the problem of non-parametric Conditional Independence testing (CI testing) for continuous random variables.  ...  We then develop theoretical results regarding the generalization bounds for classification for our problem, which translate into error bounds for CI testing.  ...  Here, a, b ∈ R dz and a = b = 1. a,b are fixed while generating a single dataset. η 1 and η 2 are zero-mean Gaussian noise variables, which are independent of everything else.  ... 
arXiv:1709.06138v1 fatcat:4drgdcsnwvbbbh2h7svzwkdboe

Two-sample Test with Kernel Projected Wasserstein Distance [article]

Jie Wang, Rui Gao, Yao Xie
2022 arXiv   pre-print
We develop a kernel projected Wasserstein distance for the two-sample test, an essential building block in statistics and machine learning: given two sets of samples, to determine whether they are from  ...  We present practical algorithms for computing this distance function together with the non-asymptotic uncertainty quantification of empirical estimates.  ...  The authors would like to thank the Editor and the anonymous referees for the thoughtful comments and suggestions, which led to an improvement of the presentation.  ... 
arXiv:2102.06449v4 fatcat:4phnmly27jebrdgdgkj2haeqci
« Previous Showing results 1 — 15 out of 16,500 results