I suspect it started with votes for those with beards. Extending it to the beardless was a later move, of debatable soundness.