Internet Archive Scholar

Predicting reliability through structured expert elicitation with repliCATS (Collaborative Assessments for Trustworthy Science) [post]

Hannah Fraser, Martin Bush, Bonnie Wintle, Fallon Mody, Eden T. Smith, Anca Hanea, Elliot Gould, Victoria Hemming, Daniel George Hamilton, Libby Rumpff, David Peter Wilkinson, Ross Pearson (+19 others)
2021 unpublished

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (761.8 kB)
https://web.archive.org/web/20210410201132/https://files.osf.io/v1/resources/2pczv/providers/osfstorage/603353b134d3040401e3a7c3?action=download&direct&version=1

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2021; you can also visit the original URL. The file type is application/pdf.

Replication is a hallmark of scientific research. As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce a new technique to evaluating replicability, the repliCATS (Collaborative Assessments for Trustworthy Science) process, a structured expert elicitation approach based on the IDEA protocol. The repliCATS process is delivered through an underpinning online platform and applied to the evaluation of research claims in
more » ... ocial and behavioural sciences. This process can be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period. Pilot data suggests that the accuracy of the repliCATS process meets or exceeds that of other techniques used to predict replicability. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to assist with problems like understanding the limits of generalizability of scientific claims. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.
doi:10.31222/osf.io/2pczv fatcat:4c43abnltndw3lp2zi4uajtgce
Open Access
Citation

Hannah Fraser, Martin Bush, Bonnie Wintle, Fallon Mody, Eden T. Smith, Anca Hanea, Elliot Gould, Victoria Hemming, Daniel George Hamilton, Libby Rumpff, David Peter Wilkinson, Ross Pearson, Felix Singleton Thorn, raquel Ashton, Aaron Willcox, Charles T. Gray, Andrew Head, Melissa Ross, Rebecca Groenewegen, Alexandru Marcoci, Ans Vercammen, Timothy H. Parker, Rink Hoekstra, Shinichi Nakagawa, David R. Mandel, Don van Ravenzwaaij, Marissa McBride, Richard O. Sinnott, Peter Anton Vesk, Mark Burgman, Fiona Fidler. "Predicting reliability through structured expert elicitation with repliCATS (Collaborative Assessments for Trustworthy Science)." (2021)