When conducting systematic reviews, medical researchers heavily deliberate over the inal query to pose to the information retrieval system. Given the possible query variations that they could construct, selecting the best performing query is diicult. This motivates a new type of query performance prediction (QPP) task where the challenge is to estimate the performance of a set of query variations given a particular topic. Query variations are the reductions, expansions and modiications of a

more »

... n seed query under the hypothesis that there exists some variations (either generated from permutations or hand crafted) which will improve retrieval efectiveness over the original query. We use the CLEF 2017 TAR Collection, to evaluate sixteen pre and post retrieval predictors for the task of Query Variation Performance Prediction (QVPP). Our indings show the IDF based QPPs exhibits the strongest correlations with performance. However, when using QPPs to select the best query, little improvement over the original query can be obtained, despite the fact that there are query variations which perform signiicantly better. Our indings highlight the diiculty in identifying efective queries within the context of this new task, and motivates further research to develop more accurate methods to help systematic review researchers in the query selection process. Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or ailiate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.