Yes, he did, that's true.
But despite the fact that maybe he was going places the older white and black people wouldn't dream of going at the time, we also should keep in mind that when Elvis visited such places he was very young.
And he did not care either. My bet is he wasn't even thinking anything of it, and that was his strenght.
Back when he was a child and also in later years.