19,758 Hits in 8.3 sec

A Comparison of Offline Evaluations, Online Evaluations, and User Studies in the Context of Research-Paper Recommender Systems [chapter]

Joeran Beel, Stefan Langer
2015 Lecture Notes in Computer Science  
In this paper, we examine and discuss the appropriateness of different evaluation methods, i.e. offline evaluations, online evaluations, and user studies, in the context of research-paper recommender systems  ...  This is also true in the field of research-paper recommender systems, where the majority of recommendation approaches are evaluated offline, and only 34% of the approaches are evaluated with user studies  ...  In addition, we are first to discuss the appropriateness of the evaluation methods in the context of research-paper recommender systems, aside from our previous paper on recommender system evaluation  ... 
doi:10.1007/978-3-319-24592-8_12 fatcat:l6aklaw7bzb6piwd6dp3ya6fja

Report on the workshop on reproducibility and replication in recommender systems evaluation (RepSys)

Alejandro Bellogín, Pablo Castells, Alan Said, Domonkos Tikk
2014 SIGIR Forum  
Experiment replication and reproduction are key requirements for empirical research methodology, and an important open issue in the field of Recommender Systems.  ...  While the problem of reproducibility and replication has been recognized in the Recommender Systems community, the need for a clear solution remains largely unmet, which motivates the main questions addressed  ...  Furthermore, we acknowledge the efforts of the members of the program committee, including: Xavier Amatriain ( Special thanks are due to the paper authors, the invited speaker, and all the participants  ... 
doi:10.1145/2641383.2641389 fatcat:oizjlpkbkves3b5a2irwa53x2m

Research paper recommender system evaluation

Joeran Beel, Stefan Langer, Marcel Genzmehr, Bela Gipp, Corinna Breitinger, Andreas Nürnberger
2013 Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation - RepSys '13  
Among the evaluated approaches, 19% were not evaluated against a baseline. Of the user studies performed, 60% had 15 or fewer participants or did not report on the number of participants.  ...  Over 80 approaches for academic literature recommendation exist today. The approaches were introduced and evaluated in more than 170 research articles, as well as patents, presentations and blogs.  ...  Interesting in this context is the low number of online evaluations (7%) and the prevalence of offline evaluations (69%).  ... 
doi:10.1145/2532508.2532512 dblp:conf/recsys/BeelLGGBN13 fatcat:fenaekxxszaixda3xrq52i7ray

Offline and online evaluation of news recommender systems at

Florent Garcin, Boi Faltings, Olivier Donatsch, Ayar Alazzawi, Christophe Bruttin, Amr Huber
2014 Proceedings of the 8th ACM Conference on Recommender systems - RecSys '14  
In this paper, we report on the live evaluation of CT recommender systems.  ...  Previously, we evaluated the context-tree recommender systems in an offline setting and showed that there exists a trade-off between accuracy and novelty of recommendations [7] .  ... 
doi:10.1145/2645710.2645745 dblp:conf/recsys/GarcinFDABH14 fatcat:cyhmh3uv2fh2zbmn2sd5kcrpl4

Benchmarking News Recommendations

Frank Hopfgartner, Torben Brodt, Jonas Seiler, Benjamin Kille, Andreas Lommatzsch, Martha Larson, Roberto Turrin, András Serény
2016 SIGIR Forum  
In this report, we discuss the objectives and challenges of the NewsREEL lab, summarize last year's campaign and outline the main research challenges that can be addressed by participating in NewsREEL  ...  The CLEF NewsREEL challenge is a campaign-style evaluation lab allowing participants to evaluate and optimize news recommender algorithms.  ...  Acknowledgment The work leading to these results has received funding (or partial funding) from the European Union's Seventh Framework Programme (FP7/2007-2013) under grant agreement number 610594 (CrowdRec  ... 
doi:10.1145/2888422.2888443 fatcat:rgyv32ohmbh6rp2yypjygqohda

Insights on Social Recommender Systems

Wolney Leal De Mello Neto, Ann Nowé
2012 ACM Conference on Recommender Systems  
In view of that, this paper proposes insights on how to possibly achieve more efficient and accurate predictions for recommendations by exploring multiple dimensions of a RS architecture.  ...  Assuming a user has a dense social network, the cold-start problem can be easily tackled. Finally, rating prediction accuracy performs better when evaluated online than by offline cross-validation.  ...  This research is part of a master studies sponsored by Monesia: MObility Network Europe-Southamerica: an Institutional Approach, an Erasmus Mundus External Cooperation Window.  ... 
dblp:conf/recsys/NetoN12 fatcat:zp2xsbm7hjcvrelc532us7m3om

Stream-Based Recommendations: Online and Offline Evaluation as a Service [chapter]

Benjamin Kille, Andreas Lommatzsch, Roberto Turrin, András Serény, Martha Larson, Torben Brodt, Jonas Seiler, Frank Hopfgartner
2015 Lecture Notes in Computer Science  
The CLEF NewsREEL challenge is a campaign-style evaluation lab allowing participants to evaluate and optimize news recommender algorithms online and offline.  ...  In this paper, we discuss the objectives and challenges of the NewsREEL lab. We motivate the metrics used for benchmarking the recommender algorithms and explain the challenge dataset.  ...  Acknowledgments The research leading to these results was performed in the CrowdRec project, which has received funding from the European Union Seventh Framework Program FP7/2007-2013 under grant agreement  ... 
doi:10.1007/978-3-319-24027-5_48 fatcat:vucqtwml3rgtjbmr3x4mwafmqa

Research-paper recommender systems: a literature survey

Joeran Beel, Bela Gipp, Stefan Langer, Corinna Breitinger
2015 International Journal on Digital Libraries  
We also identified a lack of authority and long-term research interest in the field: 73% of the authors published no more than one paper on research-paper recommender systems, and there was little cooperation  ...  In the last sixteen years, more than 200 research articles were published about research-paper recommender systems.  ...  Some of the approaches were evaluated using both an offline evaluation and a user study.  ... 
doi:10.1007/s00799-015-0156-0 fatcat:3texqmxdozhglpfb5fflpyfopu

Evaluation Infrastructures for Academic Shared Tasks

Johann Schaible, Timo Breuer, Narges Tavakolpoursaleh, Bernd Müller, Benjamin Wolff, Philipp Schaer
2020 Datenbank-Spektrum  
In this paper, we elaborate on the benefits and shortcomings of four state-of-the-art evaluation infrastructures on search and recommendation tasks concerning the following requirements: support for online  ...  However, evaluating the performance of the underlying retrieval approaches remains a challenge.  ...  Acknowledgements The STELLA project is funded by the Deutsche Forschungsgemeinschaft (DFG) -Project number 407518790.  ... 
doi:10.1007/s13222-020-00335-x fatcat:f7lj6fr4n5hazoqtywrulw7b4u

Offline Evaluation of Reward-Optimizing Recommender Systems: The Case of Simulation [article]

Imad Aouali, Amine Benhalloum, Martin Bompaire, Benjamin Heymann, Olivier Jeunen, David Rohde, Otmane Sakhi, Flavian Vasile
2022 arXiv   pre-print
Both in academic and industry-based research, online evaluation methods are seen as the golden standard for interactive applications like recommendation systems.  ...  Here, we make the case that simulation-based comparisons provide ways forward beyond offline metrics, and argue that they are a preferable means of evaluation.  ...  In industry, if a sufficient number of offline metrics are promising, the method is further tested in online experiments such as A/B-tests. In this paper we argue: this is a sub-optimal methodology.  ... 
arXiv:2209.08642v1 fatcat:qaanspye7jdlrixnfn47dhghau

CLEF NewsREEL 2016: Comparing Multi-dimensional Offline and Online Evaluation of News Recommender Systems

Benjamin Kille, Andreas Lommatzsch, Frank Hopfgartner, Martha A. Larson, Jonas Seiler, Davide Malagoli, András Serény, Torben Brodt
2016 Conference and Labs of the Evaluation Forum  
Presented results illustrate potentials for multi-dimensional evaluation of recommendation algorithms in a living lab and simulation based evaluation setting.  ...  Running in its third year at CLEF, NewsREEL challenged participants to develop news recommendation algorithms and have them benchmarked in an online (Task 1) and offline setting (Task 2), respectively.  ...  Acknowledgments The research leading to these results was performed in the CrowdRec project, which has received funding from the European Union Seventh Framework Program FP7/2007-2013 under grant agreement  ... 
dblp:conf/clef/KilleLHLSMSB16 fatcat:ae6zwyy56vey3at3qhavtetry4

Dynamically Predicting the Quality of Service: Batch, Online, and Hybrid Algorithms

Ya Chen, Zhong-an Jiang
2017 Journal of Electrical and Computer Engineering  
This paper studies the problem of dynamically modeling the quality of web service. The philosophy of designing practical web service recommender systems is delivered in this paper.  ...  A general system architecture for such systems continuously collects the user-service invocation records and includes both an online training module and an offline training module for quality prediction  ...  The key idea of matrix factorization is assuming a latent lowdimensional space R on which, for each user , a user feature is defined and for each item (i.e., service in the context of this paper) , an  ... 
doi:10.1155/2017/9547869 fatcat:inftqp2wxvdfvniqxi36kmrlgi

Hood or Hypertext: A Comparison of Offline and Online Book Search Sessions

Maria Gäde, Vivien Petras
2016 Conference and Labs of the Evaluation Forum  
In particular, the focus lies on the observation and comparison of searching, browsing and recommending strategies.  ...  People searching for books make use of various sources and systems. While some users prefer online systems, others tend to visit bookstores in their neighborhood.  ...  Acknowledgments We would like to thank Manuela Wiggert for providing access to her bookstore Albertinen Buchhandlung as well as valuable insight into a physical bookstore environment.  ... 
dblp:conf/clef/GadeP16 fatcat:z2fumtuew5bmhju7l77gsb6v6y

Context-Aware Recommender System: A Review of Recent Developmental Process and Future Research Direction

Khalid Haruna, Maizatul Akmar Ismail, Suhendroyono Suhendroyono, Damiasih Damiasih, Adi Pierewan, Haruna Chiroma, Tutut Herawan
2017 Applied Sciences  
This paper presents a review of recent developmental processes as a fountainhead for the research of a context-aware recommender system.  ...  A number of existing context-aware systems have been surveyed in [41] . The survey outlined and explained the general processes and the design considerations in context-aware systems.  ...  Conflicts of Interest: The authors declare no conflict of interest.  ... 
doi:10.3390/app7121211 fatcat:p43pqjs7lbdcbknbo76anzawui

Escaping the McNamara Fallacy: Towards more Impactful Recommender Systems Research

Dietmar Jannach, Christine Bauer
2020 The AI Magazine  
However, in the recommender systems research community, we have fallen prey to a McNamara fallacy to a worrying extent: In the majority of our research efforts, we rely almost exclusively on computational  ...  Through our analyses, we identify a number of research gaps and propose ways of broadening and improving our methodology in a way that leads us to more impactful research in our field.  ...  Only a handful of papers combined offline experiments with a user study, and another small set of papers very briefly reported outcomes of a controlled field experiment (A/B test).  ... 
doi:10.1609/aimag.v41i4.5312 fatcat:fjimn4tlkrhdbo3elxhmbtwwlm
« Previous Showing results 1 — 15 out of 19,758 results