The right calls them ‘economic migrants’; progressives respond by focusing on women and children. I’ve fallen into that trap myself, says Guardian columnist Zoe Williams