The core problem is that any action will interfere with others to some extent. So appeals to merely "negative" duties won't really restrict the scope of our obligations all that much. An example of this can be found in the problem of initial acquisition: whenever you claim a property right over previously common goods, you are peremptorily excluding others from its use. Your action is thus a kind of harmful "interference", and -- by the libertarian's own lights -- you ought to recompense others appropriately. (The implications of our actual history are explored here.)
Even after the establishment of "property rights", note that their enforcement is itself a form of coercive interference. So there is a conflict between (i) the liberty of the rich to use their surplus resources for luxury purposes without interference, and (ii) the liberty of the poor "not to be interfered with in taking from the rich what they require to meet their basic needs." If we take liberty as fundamental, the only reasonable resolution of this conflict is to recognize that liberty (ii) morally trumps (i). Hence the rich have the (still merely "negative"!) duty not to interfere with others' appropriation of their surplus resources for the sake of those in need.
That's obviously a very significant negative duty. Right-wing libertarians won't like it one bit. Which just goes to show the insincerity of any claims to be fundamentally concerned with liberty. They're more interested in preserving the privilege of the rich, and by force if need be. Freedom (for anyone but themselves) and justice have nothing to do with it.
Another application of the "core problem" is found in Thomas Pogge's work on human rights. He argues that we should conceive of rights in institutional terms:
By postulating a human right to X, one is asserting that any society or other social system, insofar as this is reasonably possible, ought to be so (re)organized that all its members have secure access to X, with 'security' always understood as especially sensitive to persons' risk of being denied X or deprived of X officially: by the government or its agents or officials. Avoidable insecurity of access, beyond certain plausibly attainable thresholds, constitutes official disrespect and stains the society's human rights record. Human rights are then moral claims on the organization of one's society.... Persons share responsibility for official disrespect of human rights within any coercive institutional order they are involved in upholding.
-- Thomas Pogge, 'How Should Human Rights Be Conceived?', p.64, emphasis added.
He further explains:
The most remarkable feature of this institutional understanding is that it can go well beyond minimalist libertarianism without denying its central tenet: that human rights entail only negative duties. The normative force of others' human rights for me is that I must not uphold and impose upon them coercive social institutions under which they do not have secure access to the objects of their human rights... Even if I owned no slaves or employed no servants myself, I would still share responsibility: by contributing my labour to the society's economy, my taxes to its governments, and so forth. I might honor my negative duty, perhaps, by becoming a hermit or an emigrant, but I could honor it more plausibly by working with others toward shielding the victims of injustice from the harms I help produce or, if this is possible, toward establishing secure access through institutional reform. (p.66, emphasis added)
By constraining ourselves to only negative duties, we find that "human rights give you claims not against all other human beings, but specifically against those who impose a coercive institutional order upon you." (p.67) But this is close enough to be practically the same thing. We are all participants, contributors, upholders, and hence imposers of the current global institutional order. Our actions thus cause harms to those who suffer unjustly under this order. It is wrong to impose such harms -- a violation even of merely negative duties -- so we have a corresponding duty to recompense the victims accordingly.
So we see that even negative duties make significant demands on us. Right-wingers may wish to ignore this. But they can no longer pretend to have the support of any half-way plausible theory of justice.