We provide a comprehensive analysis of the differences between two important standards for randomized benchmarking (RB): the Clifford-group RB protocol proposed originally in Emerson et al (2005) and Dankert et al (2006), and a variant of that RB protocol proposed later by the NIST group in Knill et al, PRA (2008). While these two protocols are frequently conflated or presumed equivalent, we prove that they produce distinct exponential fidelity decays leading to differences of up to a factor of

more »

... 3 in the estimated error rates under experimentally realistic conditions. These differences arise because the NIST RB protocol does not satisfy the unitary two-design condition for the twirl in the Clifford-group protocol and thus the decay rate depends on non-invariant features of the error model. Our analysis provides an important first step towards developing definitive standards for benchmarking quantum gates and a more rigorous theoretical underpinning for the NIST protocol and other RB protocols lacking a group-structure. We conclude by discussing the potential impact of these differences for estimating fault-tolerant overheads.