Opponents of consequentialism seem implicitly committed to the idea that the status quo is a morally privileged state of affairs. They abhor the idea of "utilitarian sacrifice", i.e. harming one person to help another more. But why? (It's not a failure to treat people as ends in themselves.) The resulting state of affairs is a better one. If deciding from a neutral position or "God's eye view" whether to actualize the former or the latter state of affairs, we should (ceteris paribus) prefer the latter. Why should things suddenly change merely because we're in the world, with the former state as the status quo? (I argue here that such a shift in context should make no difference.)
People simply assume that the status quo involves a just distribution. (Witness the absurd cries that redistributive taxation is "theft".) But this is often not the case. It is sheer luck what circumstances one is born into, and even later in life we never manage to wrest full control back from the whimsies of fortune. So it will often happen that someone is better off than another without especially deserving this to be so. So why not benefit another at a lesser cost to him? He has no special entitlement to the extra welfare fortune has granted him. It's good that a person be well-off, of course. We certainly wouldn't want to harm him unnecessarily. But it is even better for someone to receive a greater benefit. We shouldn't refrain from shifting to a better state of affairs merely due to a prejudice in favour of the existing distribution. Of course, it's easy to see why traditional elites would want to promote a "morality" which favours their entrenched interests and the status-quo. It's less clear why we should support such a bias.
Such concerns are bolstered once we recognize how hollow the so-called "doing/allowing distinction" is. There isn't any significant difference between harming someone to benefit another, and deliberately refraining from preventing such a harm. To prefer passivity is again to idolize "the natural way of things", what's historically "given", the world as it is rather than as it could be. It is, that is, to exhibit an unthinking deference to the status quo. Conservatism at its worst.
Finally, those of a more egalitarian bent might balk at the idea of imposing significant harms on one person in order to provide a vast number of individually smaller (but larger in aggregate) benefits to others. But if we are to be unbiased about this, we must consider the situation from a neutral perspective, i.e. without privileging the arbitrary historical distribution. So consider the situation in reverse: would you recommend imposing vastly many small harms in order to greatly benefit one person? If not, then the initial judgment rests on a conservative bias.
From a neutral perspective, there is no distinction to be made between imposing harms and withholding benefits. The only difference between the two cases concerns what we understand to be the status quo. If we refrain from idolizing the status quo, then we reject the idea of an absolute baseline that can ground the purported distinction. The only real assessments to be made here are relative ones. Suppose scenario A is better for you than B. Then to shift from A to B would be a "harm", while to prevent a shift from B to A would be to "withhold a benefit". This is a merely descriptive difference. If we deny that the historically given starting point provides a morally privileged baseline, then we must say that the relative harm done in either case is the same, namely the difference in utility of (A - B). It doesn't matter where we start from. (We know that, we're not tradition-bound conservatives.) What matters is how our end result compares with the various other alternatives that were open to us.
(Of course, as in the political case, there might be practical reasons to take tradition into account. I merely mean to deny it any intrinsic significance. There may be indirect utilitarian reasons for us to invent some form of doing/allowing or harm/withheld-benefit distinction, or the like.)