Depends what you mean by 'fit'. It is a factor that can work in your favour though. Quite simply it subcommunicates so many attractive qualities:

Indicator of good health
Muscle is an indicator of strength, power and masculinity
Alludes to an active lifestyle
Says you make an effort on your appearance,
Says you make an effort to improve yourself and work hard to reach goals

Plus you look better lol.

But like lots of other things, it's only a 'factor'.