Labour will need to overturn a Tory majority of nearly 20,000 to win - housing campaigners fear failure could see the party retreat from its policy