Fuck your insecurities about the situation is the best advice. Seriously stop caring about it and dont even react to it, you are you at thats that.

If someone questions you about something hes said, just be totally honest and say yeah and? If the guy trys to put you down infront of people just be non reactant and get on with what you're doing. He only has as much power over you as you let him.

As for making excuses or justifying when you lost your virginity, sod that.. It happened when it happened so what? You just need to realise that nothing that anyone thinks matters.