Filters








14,432 Hits in 6.6 sec

Do Offline Metrics Predict Online Performance in Recommender Systems? [article]

Karl Krauth, Sarah Dean, Alex Zhao, Wenshuo Guo, Mihaela Curmei, Benjamin Recht, Michael I. Jordan
2020 arXiv   pre-print
In this work we investigate the extent to which offline metrics predict online performance by evaluating eleven recommenders across six controlled simulated environments.  ...  We observe that offline metrics are correlated with online performance over a range of environments. However, improvements in offline metrics lead to diminishing returns in online performance.  ...  show how richer sets of offline metrics can be used to predict online performance Maksai et al. [2015] .  ... 
arXiv:2011.07931v1 fatcat:fre2cuepjzcv5gtnk3ulnblywu

Predicting Online Performance of News Recommender Systems Through Richer Evaluation Metrics

Andrii Maksai, Florent Garcin, Boi Faltings
2015 Proceedings of the 9th ACM Conference on Recommender Systems - RecSys '15  
We investigate how metrics that can be measured offline can be used to predict the online performance of recommender systems, thus avoiding costly A-B testing.  ...  In addition to accuracy metrics, we combine diversity, coverage, and serendipity metrics to create a new performance model.  ...  CONCLUSION We investigated predicting the online performance of news recommendation algorithms by a regression model using offline metrics.  ... 
doi:10.1145/2792838.2800184 dblp:conf/recsys/MaksaiGF15 fatcat:tv4gugb5ljgmdaufz5v35a4riq

Evaluating Recommendation Systems [chapter]

Guy Shani, Asela Gunawardana
2010 Recommender Systems Handbook  
In many cases a system designer that wishes to employ a recommendation system must choose between a set of candidate approaches.  ...  Recommender systems are now popular both commercially and in the research community, where many approaches have been suggested for providing recommendations.  ...  Simulating user behavior In order to evaluate algorithms offline, it is necessary to simulate the online process where the system makes predictions or recommendations, and the user corrects the predictions  ... 
doi:10.1007/978-0-387-85820-3_8 fatcat:tzxzm3utmzb5xkexpb43rv5som

A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation

Joeran Beel, Marcel Genzmehr, Stefan Langer, Andreas Nürnberger, Bela Gipp
2013 Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation - RepSys '13  
We conducted a study in which we evaluated various recommendation approaches with both offline and online evaluations.  ...  We discuss this finding in detail and conclude that offline evaluations may be inappropriate for evaluating research paper recommender systems, in many settings.  ...  Why do offline evaluations only accurately predict performance in real-world systems? For some recommendation approaches coverage was low.  ... 
doi:10.1145/2532508.2532511 dblp:conf/recsys/BeelGLNG13 fatcat:2bxioctjfrhgnne223bgjip5le

Revisiting offline evaluation for implicit-feedback recommender systems

Olivier Jeunen
2019 Proceedings of the 13th ACM Conference on Recommender Systems - RecSys '19  
Recommender systems are typically evaluated in an offline setting.  ...  Offline evaluation is much more efficient, but current methodologies often fail to accurately predict online performance.  ...  We have motivated the need for more effective offline evaluation strategies that are successful in predicting online performance.  ... 
doi:10.1145/3298689.3347069 dblp:conf/recsys/Jeunen19 fatcat:tlm64i2mbza6hequt4xyrhl4zu

AToMRS: A Tool to Monitor Recommender Systems

André Costa, Tiago Cunha, Carlos Soares
2016 Proceedings of the 8th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management  
Recommender systems arose in response to the excess of available online information. These systems assign, to a given individual, suggestions of items that may be relevant.  ...  This tool also aims to tackle the lack of mechanisms to enable visually assessment of the performance of a recommender systems' algorithm.  ...  Based on this feedback, the system performs online evaluation procedures.  ... 
doi:10.5220/0005992801330140 dblp:conf/ic3k/Costa0S16 fatcat:siujrzj5qvfotjcwq2h65yhqvq

Offline and online evaluation of news recommender systems at swissinfo.ch

Florent Garcin, Boi Faltings, Olivier Donatsch, Ayar Alazzawi, Christophe Bruttin, Amr Huber
2014 Proceedings of the 8th ACM Conference on Recommender systems - RecSys '14  
We can observe a reversal of the relative performance in comparison with the offline prediction.  ...  While there are many different metrics to assess the performance of a recommender systems [5, 9] , we focus on this metric because it can also be measured online and thus allows a comparison.  ... 
doi:10.1145/2645710.2645745 dblp:conf/recsys/GarcinFDABH14 fatcat:cyhmh3uv2fh2zbmn2sd5kcrpl4

A Comparison of Offline Evaluations, Online Evaluations, and User Studies in the Context of Research-Paper Recommender Systems [chapter]

Joeran Beel, Stefan Langer
2015 Lecture Notes in Computer Science  
Zheng et al. showed that CTR and relevance do not always correlate and concluded that "CTR may not be the optimal metric for online evaluation of recommender systems" and "CTR should be used with precaution  ...  However, criticism has been raised on the assumption that offline evaluation could predict an algorithm's effectiveness in online evaluations or user studies.  ...  do biased citingis used as ground-truth, a recommender system can never perform better than the imperfect status quo.  ... 
doi:10.1007/978-3-319-24592-8_12 fatcat:l6aklaw7bzb6piwd6dp3ya6fja

REDD 2014 -- international workshop on recommender systems evaluation

Panagiotis Adamopoulos, Alejandro Bellogín, Pablo Castells, Paolo Cremonesi, Harald Steck
2014 Proceedings of the 8th ACM Conference on Recommender systems - RecSys '14  
Yet, in the field of recommender systems, there still exists considerable disparity in evaluation methods, metrics and experimental designs, as well as a significant mismatch between evaluation methods  ...  Evaluation is a cardinal issue in recommender systems; as in any technical discipline, it highlights to a large extent the problems that need to be solved by the field and, hence, leads the way for algorithmic  ...   Can we predict the success of a recommendation algorithm with offline experiments? What offline metrics correlate better and under which conditions?  ... 
doi:10.1145/2645710.2645780 dblp:conf/recsys/AdamopoulosBCCS14 fatcat:fucjqxnv55bd7kqrz4ykk3qpau

Selecting Appropriate Metrics for Evaluation of Recommender Systems

Bhupesh Rawat, Sanjay K. Dwivedi
2019 International Journal of Information Technology and Computer Science  
Moreover, in order for a recommender system to generate good quality of recommendations, it is essential for a researcher to find the most suitable evaluation metric which best matches a given recommender  ...  To deal with this issue, a large number of recommender systems based on different recommender approaches were developed which have been used successfully in a wide variety of domains such as e-commerce  ...  to perform an online experiment and when it is a good option to perform offline experiments.  ... 
doi:10.5815/ijitcs.2019.01.02 fatcat:df2ar6nsj5dh7an7bdvlj4wed4

The Xbox recommender system

Noam Koenigstein, Nir Nice, Ulrich Paquet, Nir Schleyen
2012 Proceedings of the sixth ACM conference on Recommender systems - RecSys '12  
A recent addition to Microsoft's Xbox Live Marketplace is a recommender system which allows users to explore both movies and games in a personalized context.  ...  The system largely relies on implicit feedback, and runs on a large scale, serving tens of millions of daily users. We describe the system design, and review the core recommendation algorithm.  ...  SYSTEM DESIGN AND PREPROCESSING The architecture of the Xbox recommender system consists of an offline module and an online module (see figure 1).  ... 
doi:10.1145/2365952.2366015 dblp:conf/recsys/KoenigsteinNPS12 fatcat:eo3nkduc35f2jddjrmgptf3c4u

Accelerated learning from recommender systems using multi-armed bandit [article]

Meisam Hejazinia, Kyler Eastman, Shuqin Ye, Abbas Amirabadi, Ravi Divvela
2019 arXiv   pre-print
We showcase how we implemented a MAB solution as an extra step between offline and online A/B testing in a production system.  ...  We present the result of our experiment and compare all the offline, MAB, and online A/B tests metrics for our use case.  ...  Figure 3 : 3 Multi-armed Bandit Re-Allocation Timeseries Table 1 : 1 Offline and online performance of all selected recommender system models in this MAB campaign Offline results online results Recommender  ... 
arXiv:1908.06158v1 fatcat:7rp3l5ea25feliymdm6cyeuska

How good your recommender system is? A survey on evaluations in recommendation

Thiago Silveira, Min Zhang, Xiao Lin, Yiqun Liu, Shaoping Ma
2017 International Journal of Machine Learning and Cybernetics  
However, one of the current challenges in the area refers to how to properly evaluate the predictions generated by a recommender system.  ...  Recommender Systems have become a very useful tool for a large variety of domains. Researchers have been attempting to improve their algorithms in order to issue better predictions to the users.  ...  [23] , the authors use offline metrics to predict online performance, using concepts such as coverage, diversity and serendipity.  ... 
doi:10.1007/s13042-017-0762-9 fatcat:o77u7tg4yva47nlo6vto2xeaee

Recommender Systems: Past, Present, Future

Dietmar Jannach, Pearl Pu, Francesco Ricci, Markus Zanker
2022 The AI Magazine  
Ultimately, making recommendations is a human-computer interaction problem, where a computerized system supports users in information search or decision-making contexts.  ...  This special issue contains a selection of papers reflecting this multi-faceted nature of the problem and puts open research challenges in recommender systems to the fore-front.  ...  Several researchers however argue that a shift is needed in terms of how we do offline evaluations.  ... 
doi:10.1609/aimag.v42i3.18139 fatcat:apw3ho5ikzdjfm2ro7den5m2le

Towards 'Human/System Synergistic Development': How Emergent System Characteristics Change Software Development [chapter]

Helena Holmström Olsson, Jan Bosch
2016 Lecture Notes in Business Information Processing  
Recommender systems have gained a lot of popularity in recent times due to their application in the wide range of fields.  ...  So far, less interest is shown in the research so far on the evaluation of recommender systems in streaming environments.  ...  The online evaluation measures the performance of recommender algorithms using metrics such as click-through rate, while offline evaluation focuses on analyzing recommendation precision and technical complexity  ... 
doi:10.1007/978-3-319-40515-5_12 fatcat:fawb2qxl6benvlp5qqt3j5cjve
« Previous Showing results 1 — 15 out of 14,432 results