It was not his love for books or inability to fight, it was actually his fashion sense. As documented in the clothes Sam unfortunately did not wear when he returned for a nice and comfy family diner with his not wife and not son. (I guess they’re saving that outfit for the Disney version of GoT.)

Maybe the codpiece that makes it look like he’s got his dick out is a bit much.........