When I’ve heard it in relation to men, it tends to be when someone is described as a “silver fox” or “silver haired fox”.