"You look familiar to me"

I have heard the above quote at least once a week since I was in university. I don't know what happened in those years that made me look so "familiar". I've heard it so much that I don't even try to figure out if I actually know the person anymore. I just assume it's a mistaken identity and dismiss the claim even before it's been made.
And here's the funny thing....I am not always mistaken for the same person. I have been home, in Toronto and even Niagara Falls and been confused for someone else. And it is never the same person!
So, I know the theory about doppelgangers, but my question is can you have more than one? Or am I just so average looking that I look like many people? And have people who really do know me gone up to a stranger and thought she was me?
So many questions......