The man who started making fish pies on his kitchen table served up healthy sales of £133m last year, despite staff and cost issues … and the problem of being outside the EU