On the typical libertarian view, rights act as side-constraints on what a person may do. If an option open to you would involve violating someone's rights, then you should simply exclude that option from consideration. Rights serve to limit what is morally permissible. One might call this the 'deontological' interpretation of rights. As my name hints, there is an alternative, 'consequentialist', interpretation, according to which rights function as goals rather than constraints. The idea would be to try to minimize violations of people's rights. On this view, rights serve to inform rather than limit our moral decisions: they can tell us what to do, rather than merely what not to do. Further, it avoids absolutism: it might be moral to violate one person's rights in order to protect the rights of many other people.
I don't really understand why anyone would adopt the side-constraints view. It doesn't make sense to me. Perhaps the idea is grounded in the notion that some things (e.g. homicide) are really really bad and so ought not to be done (cf. Nozick). But then, as Cohen asks, "if such sacrifice and violation are so horrendous, why should we not be concerned to minimize their occurrence?"
The agent-relativity of deontological moralities leaves them at risk of being collectively self-defeating, as Derek Parfit has shown. According to such a morality M, our moral ("M-given") aims will vary from person to person. Perhaps each individual has an obligation to look after their family, and avoid violating others' rights themselves. We need not be so concerned about other people's families, or avoiding rights-violations committed by other people. This gives rise to a "prisoner's dilemma", whereby each of two people must independently decide between (1) meeting some of their own M-given aims; or (2) enabling more of the other person's M-given aims to be met.
(Perhaps both agents are currently on track to violate lots of rights. However, each can choose either to enable themselves to avoid some violations, or else enable the other to avoid even more violations.)
The agent-relative morality M tells us to choose option #1. But if everyone chose thusly, their M-given aims would be worse achieved than if everybody had instead chosen #2. (Each individual commits more rights-violations than they otherwise would have.) M is thus collectively self-defeating: if everyone follows it successfully, they each do worse, even by its own standards. And, as Parfit notes, "If there is any assumption on which it is clearest that a moral theory should not be self-defeating, it is the assumption that it is universally followed"!
I suppose the deontologist would respond that we have fundamentally misunderstood their morality. They do not have the agent-relative aim to minimize their own rights-violations. They do not understand rights as a sort of "goal" at all. Rather, as previously mentioned, they serve as side-constraints. If they further deny that we should have the moral aim of meeting these constraints, then they could dodge the above argument by denying that we have any "M-given aims" at all. Such talk of "aims" serves to illicitly smuggle in consequentialist premises which the deontologist would not accept. Or so one might argue. (I'm not entirely sure that it makes sense to deny that we have moral aims. It sounds very odd to me, at least.)
Perhaps, at the end of the day, all we can do is appeal to brute intuitions. It's just obvious that rights are not absolute. If the universe would be destroyed unless we sacrificed someone, isn't it quite clear that we ought to perform the sacrifice? Their death would be tragic, of course. But the death of billions would be even more tragic.
Understanding rights as side-constraints rather than goals is another case of prizing the merely formal over what has genuine substantive worth. Indeed, it seems to me that the very notion of rights is somewhat fetishistic. They lead one to value the (non-)performance of actions above actual human welfare. But the important aspect of a murder is the fact that a person died, not that someone else performed a rights-violating action. The moral status of an action does not alter its value, independently of facts about human welfare. It's the latter that really matter. It's not intrinsically worse to be killed by a person than by "natural causes". Either way, you've been harmed, and that's the tragedy.