David Sobel's (2007) 'The Impotence of the Demandingness Objection' nicely develops this line of criticism (p.3):
Consider the case of Joe and Sally. Joe has two healthy kidneys and can live a decent but reduced life with only one. Sally needs one of Joe’s kidneys to live. Even though the transfer would result in a situation that is better overall, the Demandingness Objection’s thought is that it is asking so much of Joe to give up a kidney that he is morally permitted to not give. The size of the cost to Joe makes the purported moral demand that Joe give the kidney unreasonable, or at least not genuinely morally obligatory on Joe. Consequentialism, our intuitions tell us, is too demanding on Joe when it requires that he sacrifice a kidney to Sally.
But consider things from Sally’s point of view. Suppose she were to complain about the size of the cost that a non-Consequentialist moral theory permits to befall her. Suppose she were to say that such a moral theory, in permitting others to allow her to die when they could aid her, is excessively demanding on her. Clearly Sally has not yet fully understood how philosophers typically intend the Demandingness Objection. What has she failed to get about the Objection? Why is Consequentialism too demanding on the person who would suffer significant costs if he was to aid others as Consequentialism requires, but non-Consequentialist morality is not similarly too demanding on Sally, the person who would suffer more significant costs if she were not aided as the alternative to Consequentialism permits? What must the Objection’s understanding of the demands of a moral theory be such that that would make sense? There is an obvious answer that has appealed even to prominent critics of the Objection — that the costs of what a moral theory requires are more demanding than the costs of what a moral theory permits to befall the unaided, size of cost held constant. The moral significance of the distinction between costs a moral theory requires and costs it permits must already be in place before the Objection gets a grip. But this is for the decisive break with Consequentialism to have already happened before we feel the pull of the Demandingness intuitions.
We might similarly ask why deontological views are not considered excessively demanding when they prohibit Sally from saving her own life by stealing one of Joe's spare kidneys. In terms of raw (theory-neutral) cost to the agent, this is surely very demanding! Granted, as people standardly evaluate "demandingness", they might presuppose that moral prohibitions of this sort are not as relevantly "demanding" as positive obligations. But this is, in effect, to already be assessing questions of demandingness through deontologically-tinted glasses.
It seems, then, that there are no neutral grounds for considering impartial consequentialism to be "more demanding" than rival moral theories, at least in the sense of imposing excessively great costs on agents. One can only get this verdict by stacking the deck against consequentialism by implicitly defining "demandingness" in such a way as to only take a certain subclass of costs fully into account.
(Alternatively, we could take all this to motivate reconceptualizing demandingness as not really about costs at all, but rather something more like willpower. And then you just make these moves and -- voila! -- problem resolved.)
See also: Moral Demands and Compliance Effects