It isn’t racist now because it doesn’t suggest any prejudice or dislike and the last thing we would ever want to do is widen the class of comments that we decide express prejudice. We want to reduce the potential for accidental offense not increase it.
Hypersensitivity is a bad thing, and we'd all be better off without it. If you insist on turning yourself into a victim, perceiving slights at every turn, the world will offer ample opportunities to feed your paranoia. But why on Earth would you want to? Chances are: most people do not, in fact, hate you. But it's unnecessarily burdensome to expect them to constantly reassure you of this. Things would be much better if everyone could simply assume the best by default, and only take offense if someone was very clearly intending to insult them.
(Granted, it's perfectly understandable why someone who has suffered from others' malice in the past might be over-sensitized to it in future. I've had similar experiences myself. But it's still unfortunate, so we should want to help people to overcome their hypersensitivity, rather than encouraging it.)
Compare my response to Paul Gowder's suggestion that levelling down may be justified in cases where tolerating an inequality would express disrespect:
I agree that one shouldn't express disrespect. But we should increase freedom. Hence, to avoid unnecessary conflict here, it would be most inadvisable for us to adopt conventions of social meaning according to which increasing freedoms for some was understood as expressing disrespect for others. If such conventions are already present, we should do what we can to undermine and change them.