A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf
.
Fostering Coopetition While Plugging Leaks
2022
Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval
We articulate the design and implementation of the MS MARCO document ranking and passage ranking leaderboards. In contrast to "standard" community-wide evaluations such as those at TREC, which can be characterized as simultaneous games, leaderboards represent sequential games, where every player move is immediately visible to the entire community. The fundamental challenge with this setup is that every leaderboard submission leaks information about the held-out evaluation set, which conflicts
doi:10.1145/3477495.3531725
fatcat:y326lpdbdvgd3of6ny4yee5hv4