Across Europe, the mainstream right is being eclipsed by more extreme rivals. The same fate could await the Tories, say academics Tarik Abou-Chadi and Simon van Teutem