You can be physically dominant and still have a face like a dropped pie. But you're right, physicality can be attractive.

But your argument about cavemen having large square jaws, even teeth and being rugby players doesn't stack up to me. It's to fact that physically dominant men are (generialisation) the type of personalities that go for what they want and some women respond well to that.