Design Patterns for Hybrid Algorithmic-Crowdsourcing Workflows

Christoph Lofi, Kinda El Maarry
2014 2014 IEEE 16th Conference on Business Informatics  
Crowdsourcing has shown to be a powerful technique for overcoming many challenges in data and information processing where current state-of-the-art algorithms are still struggling. This is especially true for workflows that transparently combine algorithmic heuristics and dynamically crowdsourced tasks that are performed by human workers, and which promise to solve even more complex tasks effectively and efficiently. But still, such hybrid crowdsourcing workflows can be difficult to approach,
more » ... d they are often designed in an ad-hoc fashion. Therefore, in this paper, we extensively investigate such crowdsourcing workflows as described in the literature, and abstract generic design patterns, which codify commonly recurring challenges and their best-practice solutions. Each design pattern is described and discussed with a special focus on its requirements, constraints, and effects on the overall workflow. We illustrate the practicality of these patterns by providing realworld application examples where such patterns can or have been applied. Furthermore, we showcase how the individual design patterns can be extended and combined to support more complex workflows. Our design patterns provide an extensive overview of the hybrid crowdsourcing workflows' design space, and allow for a more efficient modeling, analysis, and documentation of such workflows.
doi:10.1109/cbi.2014.16 dblp:conf/wecwis/LofiM14 fatcat:2j3oxi36gfcejo3linc5h5z55a