I think the key is confidence. I did my dissertation on self-efficacy (yawn), all research is conclusive in that greater perceived confidence in whatever situation is directly related to outcome.

Looks aside, If going to the gym gives you greater confidence then keep at it. I have become much more confident over the last few months and have seen my success with women increase at the same time.