Towards more reliable and fairer decision-making systems: pooling decisions decreases variation in accuracy and response bias [post]

Ralf Kurvers, Stefan Michael Herzog, Ralph Hertwig, Jens Krause, Max Wolf
2020 unpublished
Over the last decades, the relative benefits and costs of individual vs. collective decision-making systems have attracted ample attention in the behavioural sciences and beyond. This research however, has almost exclusively focused on accuracy as a performance criterion, neglecting another major performance dimension of decision-making systems, the variation in outcomes between decision-making agents. This is surprising as low outcome variation is a key goal in many high-stake contexts,
more » ... ng medical, judicial and political decision making. Employing a combined theoretical and real-world data-driven approach, we investigate how one of the most prominent systems of collective decision-making – the pooling of independent decisions using the majority rule – affects the variation in outcomes between agents. Using a general statistical argument and large-scale numerical simulations, we predict that pooling decisions robustly reduces variation in two key outcome variables: accuracy and response bias (i.e. the decision maker's tendency towards one response or the other). We test this prediction in real-world datasets on breast and skin cancer diagnostics, fingerprint analysis, geopolitical forecasting, and a general knowledge task, encompassing more than 350 decision makers making more than 125,000 decisions. As predicted, we find that pooling decisions robustly reduces variation in accuracy and response bias. Importantly, this reduction is accompanied by an increase in accuracy, showing that pooling independent decisions can simultaneously decrease variation and increase accuracy. Thus, while outcomes in individual decision-making systems are highly variable and at the mercy of individual decision makers, pooling decisions decreases this variation, thereby promoting more predictable, reliable and fairer outcomes.
doi:10.31234/osf.io/sv84r fatcat:dvoi6fl5k5cfrnla6hg2kmoa6y