We study the phase diagram of a model quantum spin Hall system as a function of band inversion and band-coupling strength, demonstrating that when band hybridization is weak, an interaction-induced nematic insulator state emerges over a wide range of band inversion. This property is a consequence of the long-range Coulomb interaction, which favors interband phase coherence that is weakly dependent on momentum and therefore frustrated by the single-particle Hamiltonian at the band inversion

more »

... . For weak band hybridization, interactions convert the continuous gap closing topological phase transition at inversion into a pair of continuous phase transitions bounding a state with broken time-reversal and rotational symmetries. At intermediate band hybridization, the topological phase transition proceeds instead via a quantum anomalous Hall insulator state, whereas at strong hybridization interactions play no role. We comment on the implications of our findings for InAs/GaSb and HgTe/CdTe quantum spin Hall systems.