We explore the effect of a system's symmetries on fidelity decay behavior. Chaos-like exponential fidelity decay behavior occurs in non-chaotic systems when the system possesses symmetries and the applied perturbation is not tied to a classical parameter. Similar systems without symmetries exhibit faster-than-exponential decay under the same type of perturbation. This counter-intuitive result, that extra symmetries cause the system to behave in a chaotic fashion, may have important ramifications for quantum error correction.