Ad Attributes and Attribution: Large-Scale Field Experiments Measure Online Customer Acquisition

David H. Reiley, Randall Aaron Lewis, Taylor A. Schreiner
2012 Social Science Research Network  
Using a large-scale experiment involving 3.7 million treated subjects on Yahoo!, we measure the ability of online display advertising to cause new account signups at an online business. We experiment with two dimensions of media choice: banner-shaped ads versus large rectangular ads, and Yahoo! Mail placement versus Yahoo! Run-of-Network placement. The Run-of-Network advertisements succeeded in generating a statistically significant increase in sign-ups of approximately 11% relative to the
more » ... ol group, but the Yahoo! Mail ads did not produce a statistically significant increase in sign-ups for this advertiser. By contrast with the results for choice of pages, we find no effect of the form factor of the ads (banner versus large rectangular) in this campaign. We note the somewhat surprising fact that even with experimental sample sizes in the millions of subjects, our estimates remain quite noisy: for example, the upper bound of the 95% confidence interval for the statistically insignificant Yahoo! Mail campaign represents a 15% increase in new customers. Most importantly, our estimates call into question click-only attribution models, as restricting attention to the number of users that clicked on an ad and converted is less than 30% of the estimated treatment effect on conversions. JEL Classification C93 -Field Experiments, M37 -Advertising, L81 -E-Commerce Keywords Online advertising, advertising effectiveness, field experiment, click attribution * Lewis and Reiley: Yahoo! Research, and . Schreiner: Yahoo! Marketing Insights, . We thank Meredith Gordon and Sergiy Matusevych for their work on the experiment and the data. Yahoo! Incorporated provided financial and data assistance as well as guaranteed academic independence prior to our analysis so that the results could be published no matter how they turned out. We acknowledge the helpful comments of Clara Lewis,
doi:10.2139/ssrn.2049457 fatcat:egjb5zuapnfglli67hsnixkdi4