But I've since come to realize that I was overlooking a key practical difference: the greater harm of widespread exclusion. Scanlon states the problem especially clearly in ch.2. of his manuscript on permissibility:
One thing that seems crucial to racial discrimination in particular is that the prejudicial judgments it involves are not just the idiosyncratic attitudes of a particular agent, but are widely shared in the society in question and commonly expressed and acted on in ways that have serious consequences. The petty likes and dislikes of other individuals may be something we just have to live with, but it is another matter when the view that members of a certain group are inferior, and not to be associated with, becomes widely held in a society, with the result that members of that group are denied access to important goods and opportunities. (p.34)
If someone is idiosyncratically prejudiced against some arbitrary characteristic of mine, that kind of sucks, but it isn't such a huge deal. I can always just go to someone else instead. But if the prejudice is widespread in society, each act of discrimination is increasingly harmful, because I have nowhere else to go. If one grocer won't serve me, that costs me a few minutes as I head to another store down the road. But if no grocers will serve me, I starve.
On this view, acts of discrimination are wrong "because of their consequences", and so this impermissibility does not depend on vicious intentions. Scanlon continues:
Once a practice of discrimination exists, decisions that deny important goods to members of the group discriminated against and do so without sufficient justification, are wrong even if they express no judgments of inferiority on the agent's part. They are wrong even if done simply out of laziness, or a desire to avoid offending others by going against established custom.
Note that this objection only applies to discrimination that is part of a system of "widespread denigration and exclusion". A curious upshot: in a largely non-racist society, the odd racist is perhaps not acting as terribly as we might imagine. (Unless there is a risk of their repugnant attitudes becoming more widely shared once again.)