… according to this documentary, at least. (and if it doesn't autoplay from the 7:00 mark, that's where you need to skip to.)
Included in (I think) the first DVD release of Series 1, the documentary talks about the process of getting the show on the air and casting the parts. Danny John-Jules was the first person to audition for the role of the Cat and the writers and producer had to sit there the whole day seeing other people knowing they had their man, but it's co-writer Doug Naylor's comments about casting Lister that was the most interesting.
We saw a lot of people. We saw Alan Rickman who really liked the script, said it was the best script he'd received within I think it was two or three months and he wanted to play Lister. Erm… and we saw, y'know, Hugh Laurie, there was a whole bunch of people we saw.
I just can't imagine anyone as Lister but Craig Charles, and Charles himself says later that the characters became caricatures of the actors playing them. Rickman wasn't yet a household name, the first series of Red Dwarf aired a few months before he would come to wider attention as Hans Gruber in that summer's Die Hard, but Hugh Laurie was a star, at least in Britain, after Blackadder and his work with Stephen Fry. I'm now thinking about an alternate universe where Rimmer and Lister were played by Fry and Laurie, and it's glorious :D