you are more than likely doing nothing wrong
its normal for lots of people to go on the defensive when stopped in the street because they feel like you are either weird or trying to sell them somthing
if somone stops me in the street my immediate reaction is 'who the fuck are you'