Control Effectiveness: a Capture-the-Flag Study

Arnau Erola, Louise Axon, Alastair Janse van Rensburg, Ioannis Agrafiotis, Michael Goldsmith, Sadie Creese
2021 The 16th International Conference on Availability, Reliability and Security  
As cybersecurity breaches continue to increase in number and cost, and the demand for cyber-insurance rises, the ability to reason accurately about an organisation's residual risk is of paramount importance. Security controls are integral to risk practice and decisionmaking: organisations deploy controls in order to reduce their risk exposure, and cyber-insurance companies provide coverage to these organisations based on their cybersecurity posture. Therefore, in order to reason about an
more » ... ation's residual risk, it is critical to possess an accurate understanding of the controls organisations have in place and of the influence that these controls have on the likelihood that organisations will be harmed by a cyber-incident. Supporting evidence, however, for the effectiveness of controls is often lacking. With the aim of enriching internal threat data, in this article we explore a practical exercise in the form of a capture-theflag (CTF) study. We experimented with a set of security controls and invited four professional penetration testers to solve the challenges. The results indicate that CTFs are a viable path for enriching threat intelligence and examining security controls, enabling us to begin to theorise about the relative effectiveness of certain risk controls on the face of threats, and to provide some recommendations for strengthening the cybersecurity posture. CCS CONCEPTS • Security and privacy → Systems security; Network security; Intrusion/anomaly detection and malware mitigation.
doi:10.1145/3465481.3470095 fatcat:agmatzxh5ngxjcicglh66ejdhq