Scottish government says judge-led public inquiry will look at ‘serious failings’ of police investigation, as killer appeals against conviction