By pandering to ever-smaller sections of the electorate, we Tories have failed to address the country as a whole, says a former immigration minister