I agree with your last couple sentences. According to your argument preceding it, however, what this really boils down to is just word choice and semantics. If you feel as though your choice in semantics gives you a higher sense of moral righteousness, then congratulations! Good for you! You take very careful steps to make everyone 100% certain that you're absolutely not racist in any way whatsoever, and thus there is no confusion! Hoorah!

My point is that not everyone does this, and it does not automatically make them racist. It's very simple.