Filters








22,088 Hits in 2.8 sec

Reducing Offline Evaluation Bias in Recommendation Systems [article]

Arnaud De Myttenaere, Boris Golden
2014 arXiv   pre-print
This paper analyses this evaluation bias and proposes a simple item weighting solution that reduces its impact.  ...  This adaptation process influences the way users interact with the system and, as a consequence, increases the difficulty of evaluating a recommendation algorithm with historical data (via offline evaluation  ...  A strong assumption we make is that in practice reducing offline evaluation bias for constant algorithms contributes to reducing offline evaluation bias for all algorithms.  ... 
arXiv:1407.0822v1 fatcat:vjrof7qe4jaufa5bml4rrfl5jq

Reducing offline evaluation bias of collaborative filtering algorithms [article]

Arnaud De Myttenaere , Bénédicte Le Grand
2015 arXiv   pre-print
This paper presents a new application of a weighted offline evaluation to reduce this bias for collaborative filtering algorithms.  ...  It thus influences the way users interact with the system and, as a consequence, bias the evaluation of the performance of a recommendation algorithm computed using historical data (via offline evaluation  ...  Thus those campaigns have strongly biased the collected data, leading to a significant bias in the offline evaluation score.  ... 
arXiv:1506.04135v1 fatcat:osqalsytu5gbrdpt5jbjhhs6ai

Recommending Dream Jobs in a Biased Real World [article]

Nadia Fawaz
2019 arXiv   pre-print
These biases impact the performance of various components of recommender systems, from offline training, to evaluation and online serving of recommendations in production systems.  ...  Specific techniques can help reduce bias at each stage of a recommender system.  ...  WHY REDUCING BIAS MATTERS Biases impact the performance of various components of recommender systems, from offline training, to evaluation, to online serving of recommendations in production systems.  ... 
arXiv:1905.06134v1 fatcat:gp4bbyjx4ncrrldhdrih3436bq

Study of a bias in the offline evaluation of a recommendation algorithm [article]

Arnaud De Myttenaere , Bénédicte Le Grand
2015 arXiv   pre-print
This paper describes this bias and discuss the relevance of a weighted offline evaluation to reduce this bias for different classes of recommendation algorithms.  ...  It thus influences the way users interact with the system and, as a consequence, bias the evaluation of the performance of a recommendation algorithm computed using historical data (via offline evaluation  ...  Thus those campaigns have strongly biased the collected data, leading to a significant bias in the offline evaluation.  ... 
arXiv:1511.01280v1 fatcat:s4j3pggganfibk4ninktcssubu

A Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation

Xueying Bai, Jian Guan, Hongning Wang
2019 Neural Information Processing Systems  
To reduce bias in the learned model and policy, we use a discriminator to evaluate the quality of generated data and scale the generated rewards.  ...  Reinforcement learning is well suited for optimizing policies of recommender systems.  ...  Offline evaluation The problems of off-policy learning [22, 25, 26] and offline policy evaluation are generally pervasive and challenging in RL, and in recommender systems in particular.  ... 
dblp:conf/nips/BaiGW19 fatcat:m5lf3d2t7jcvflv4c5cgidl6su

Comparing Offline and Online Recommender System Evaluations on Long-tail Distributions

Gabriel de Souza Pereira Moreira, Gilmar Alves de Souza, Adilson Marques da Cunha
2015 ACM Conference on Recommender Systems  
In this investigation, we conduct a comparison between offline and online accuracy evaluation of different algorithms and settings in a real-world content recommender system.  ...  accuracy results in offline and online evaluations.  ...  ACKNOWLEDGEMENTS Our thanks to CI&T for supporting the development of Smart Canvas R recommender system evaluation framework and to the ITA for providing the research environment.  ... 
dblp:conf/recsys/MoreiraSC15 fatcat:etn7rpylt5ggndgxmqm2tzx5by

Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation [article]

Xueying Bai, Jian Guan, Hongning Wang
2020 arXiv   pre-print
To reduce bias in the learned model and policy, we use a discriminator to evaluate the quality of generated data and scale the generated rewards.  ...  Reinforcement learning is well suited for optimizing policies of recommender systems.  ...  Offline evaluation The problems of off-policy learning [22, 25, 26] and offline policy evaluation are generally pervasive and challenging in RL, and in recommender systems in particular.  ... 
arXiv:1911.03845v3 fatcat:qgonaucopfavnms4cud34j6gry

Estimating Error and Bias in Offline Evaluation Results

Mucun Tian, Michael D. Ekstrand
2020 Proceedings of the 2020 Conference on Human Information Interaction and Retrieval  
Offline evaluations of recommender systems attempt to estimate users' satisfaction with recommendations using static data from prior user interactions.  ...  Substantial breakthroughs in recommendation quality, therefore, will be difficult to assess with existing offline techniques.  ...  CONCLUSIONS AND FUTURE WORK We have simulated user preference for items and resulting consumption observations in order to estimate error and bias in the results of offline evaluations of recommender systems  ... 
doi:10.1145/3343413.3378004 dblp:conf/chiir/TianE20 fatcat:dofm7765ircrzbk5tjyunrr2q4

Offline Evaluation and Optimization for Interactive Systems

Lihong Li
2015 Proceedings of the Eighth ACM International Conference on Web Search and Data Mining - WSDM '15  
evaluation policy Data , , ⋮ , , Biases of Direct Method • Sampling/selection bias • From production systems • Simpson's paradox • Modeling bias • Insufficient features to fully represent  ...  of a news recommendation system • click lift of a new user feature in ad ranking • reduction of time for user to find a relevant URL on SERP • … Sport User Article Click Movie Article Overall  ...  : • Can log randomization seed in and check offline to detect bugs Use standard t-test to detect ≠  ... 
doi:10.1145/2684822.2697040 dblp:conf/wsdm/Li15 fatcat:2ap6hcpimfh5xogmdzif6ar6ri

Recommendations as Treatments

Thorsten Joachims, Ben London, Yi Su, Adith Swaminathan, Lequn Wang
2021 The AI Magazine  
This article explains how these techniques enable unbiased offline evaluation and learning despite biased data, and how they can inform considerations of fairness and equity in recommender systems.  ...  In recent years, a new line of research has taken an interventional view of recommender systems, where recommendations are viewed as actions that the system takes to have a desired effect.  ...  This trade-off between bias and variance is central to offline evaluation.  ... 
doi:10.1609/aimag.v42i3.18141 fatcat:hdyi4nadijgp3fpieqojib5pfq

Revisiting offline evaluation for implicit-feedback recommender systems

Olivier Jeunen
2019 Proceedings of the 13th ACM Conference on Recommender Systems - RecSys '19  
Recommender systems are typically evaluated in an offline setting.  ...  CCS CONCEPTS • Information systemsRecommender systems; Evaluation of retrieval results.  ...  The biases present in these datasets pose a significant challenge when the data is used to evaluate other competing algorithms in an offline manner.  ... 
doi:10.1145/3298689.3347069 dblp:conf/recsys/Jeunen19 fatcat:tlm64i2mbza6hequt4xyrhl4zu

Learning to Rank Research Articles: A Case Study of Collaborative Filtering and Learning to Rank in ScienceDirect

Daniel Kershaw, Benjamin Pettit, Maya Hristakeva, Kris Jack
2020 International Workshop on Bibliometric-enhanced Information Retrieval  
We then describe offline and online evaluation, which are essential for productionizing any recommender.  ...  However, by learning from subjective user interactions with the recommender system, our relevance model reversed those trends.  ...  Designing offline evaluation raises the question of what the recommender system is designed to do.  ... 
dblp:conf/birws/KershawPHJ20 fatcat:lhqmte2j3ba7tfax2dws47jmiy

Do Offline Metrics Predict Online Performance in Recommender Systems? [article]

Karl Krauth, Sarah Dean, Alex Zhao, Wenshuo Guo, Mihaela Curmei, Benjamin Recht, Michael I. Jordan
2020 arXiv   pre-print
In this work we investigate the extent to which offline metrics predict online performance by evaluating eleven recommenders across six controlled simulated environments.  ...  Recommender systems operate in an inherently dynamical setting. Past recommendations influence future behavior, including which data points are observed and how user preferences change.  ...  Offline Evaluation -Algorithm developers use the datasets from step (1) to evaluate their recommender systems.  ... 
arXiv:2011.07931v1 fatcat:fre2cuepjzcv5gtnk3ulnblywu

Offline Reinforcement Learning for Mobile Notifications [article]

Yiping Yuan, Ajith Muralidharan, Preetam Nandy, Miao Cheng, Prakruthi Prabhakar
2022 arXiv   pre-print
Finally, we collect data through online exploration in the production system, train an offline Double Deep Q-Network and launch a successful policy online.  ...  We describe a state-marginalized importance sampling policy evaluation approach, which can be used to evaluate the policy offline and tune learning hyperparameters.  ...  [13] applied a Policy Gradient learning in YouTube recommender system with Off-policy correction for offline reinforcement learning. Ie et al.  ... 
arXiv:2202.03867v1 fatcat:v4yibo6htvc6jbwgh7s4f5rhyq

Overview of NewsREEL'16: Multi-dimensional Evaluation of Real-Time Stream-Recommendation Algorithms [chapter]

Benjamin Kille, Andreas Lommatzsch, Gebrekirstos G. Gebremeskel, Frank Hopfgartner, Martha Larson, Jonas Seiler, Davide Malagoli, András Serény, Torben Brodt, Arjen P. de Vries
2016 Lecture Notes in Computer Science  
The CLEF News-REEL challenge is a campaign-style evaluation lab allowing participants to tackle news recommendation and to optimize and evaluate their recommender algorithms both online and offline.  ...  In the intersection of these perspectives, new insights can be gained on how to effectively evaluate real-time stream recommendation algorithms.  ...  The research leading to these results was performed in the Crow-dRec project, which has received funding from the European Union Seventh Framework Program FP7/2007-2013 under grant agreement No. 610594  ... 
doi:10.1007/978-3-319-44564-9_27 fatcat:dtmwy2ipj5di7dhywxmd45i5vq
« Previous Showing results 1 — 15 out of 22,088 results