First, note that we must distinguish the state of blame from censorious expressions of blame:
To modify one's expectations and intentions toward a person in the way I have described, in response to that person's deficient attitudes -- to conclude that one can no longer interact with that person as a friend -- is to blame that person in the sense I have in mind, whether or not one also feels resentful or indignant. One might just feel sad. (pp.14-15)
This can't be a sufficient condition as stated, for we may withdraw from another in response to their deficient attitudes without blaming them in any sense. Tristram suggests a nice contrast here between disrespect and low self-esteem. If a (supposed) friend is consistently disrespectful and inconsiderate, this is the sort of failing we consider blameworthy. But if another's low self-esteem makes them difficult or unpleasant to interact with, we may consider it a character flaw which justifies us in revising our attitudes towards them - it may reduce our desire to befriend them, say - but we wouldn't blame them for it. It is more like a case of brute difference in tastes or interests, a mere "drifting apart" that - although a response to "something about the person" - does not reflect any violation of the relationship on their part.
There was much discussion in class about the "reasonable expectations" that people may have of each other, and whether it is necessarily blameworthy to disappoint these. It seems clear that it is not, at least if your expectation is merely based upon general epistemic clues and not any kind of promise or other normative commitment on the part of the other person. But I don't think this is a problem for Scanlon's account; at least, I read him as being peculiarly concerned about the reasonable expectations that are internal to a relationship, deriving from its constitutive norms, and not just any old expectation that you might (however reasonably) have come by and hope not to have disappointed. Let me emphasize a key passage from p.14:
[T]he way in which a friendship can end when the parties drift apart [does not involve any] violation of the standards of friendship, and this is what differentiates [it] from the kind of impairment I am concerned with. Impairment of that kind occurs when one party, while standing in the relevant relation to another person, holds attitudes toward that person that are ruled out by the standards of that relationship, thus making it appropriate for the other party to have attitudes other than those that the relationship usually involves.
The answer to Tristram's objection, then, is that disrespect is a violation of the internal standards of friendship, whereas low self-esteem is not. The latter is still a flaw, to be sure, and one that may have the effect of impairing your relationship in some sense, but only in an 'external' or instrumental kind of way. It is not intrinsically antithetical to the norms of friendship in the way that disrespect is. This difference explains why the one flaw, but not the other, warrants the friend's blame. Both, though, may be disappointments.