Don`t think that being an Elvis fan ever made me feel "limited", although it has sometimes lead to other people suggesting to me that I was,usually people who are followers of whatever is "cool" at the time or who buy music on the strength of music magazines etc. If anything,being an Elvis fan lead me to more different types of music than I would ever have come across otherwise...from gospel to country and bluegrass...in a lot of ways being an Elvis fan has given me a much broader appreciation of a lot of different music styles....and much more.
At the end of the day most people have their own little obsessions,whether it`s stamp-collecting or trainspotting..or whatever. Don`t worry about anyone giving you a hard time over it..... you chose well!!