Former president suffered bad night in Philadelphia before being dealt further blow when Taylor Swift formally announced she is endorsing his rival Kamala Harris