In your words, you mention you fall in the 'average' category, so why worry

personally think that you are looking at the wrong place to find confidence. you should be confident from the inside about who/what you are and can be. A confident man couldnt give a fuck about what the rest is doing let alone if they are better/bigger in any other aspect.

grow your confidence not your penis