The country grew rich from slavery, but those who directly benefited the most – and still do – were wealthy. They exploited everyone, says author and broadcaster Nels Abbey