The US academic and TV historian’s rejection of racial categories has drawn criticism from some but his sense of mischief and underlying belief in American decency remains undimmed