A deontologist might respond by suggesting that our moral aims are not so impersonal: we have a special responsibility for our own (present) actions, and so must regard our not (now) ourselves causing harm / violating rights as a distinctive moral goal. Scheffler pushes back against this idea on pp. 415-6 of his 'Agent-Centred Restrictions, Rationality, and the Virtues':
[O]n standard deontological views, morality evaluates actions from a vantage point which is concerned with more than just the interests of the individual agent. In other words, an action will be right or wrong, on such a view, relative to a standard of assessment that takes into account a number of factors quite independent of the interests of the agent. And defenders of such views are unlikely to claim that the relevant standard of assessment includes agent-centred restrictions, but that it is a matter of indifference, from the vantage point represented by that standard, whether or not those restrictions are violated. For if it is not the case that it is preferable, from that vantage point, that no violations should occur than that any should, it is hard to see how individual agents could possibly be thought to have reason to observe the restrictions when doing so did not happen to coincide with their own interests or the interests of those they cared about. In other words, deontological views need the idea that violations of the restrictions are morally objectionable or undesirable if the claim that people ought not to commit such violations when doing so would be in their own interests is to be plausible. Yet if such views do regard violations as morally objectionable or undesirable, in the sense that it is morally preferable that none should occur than that any should, it does then seem paradoxical that they tell us there are times when we must act in such a way that a larger rather than a smaller number of violations actually takes place.
It's a fairly dense passage, so when teaching this topic last term I came up with a thought experiment to help illustrate.
Suppose that five innocent people whom you love are going to be murdered, unless you yourself murder a (distinct) innocent person. Is it wrong for you to murder an innocent person in order to save your five loved ones?
Standard deontological theories will insist that murder, even in this case, is wrong. But this may seem a difficult verdict to uphold, given that murdering the one seems preferable from both your personal standpoint and the impersonal standpoint.
Impersonally: five murders are worse than one. Personally: there is a special moral cost to you in committing a murder, sure, but it is not so great a cost (we may suppose) as losing your five loved ones. So, we may wonder, from what perspective does the deontological verdict have any normative force or appeal?
To get the verdict that murdering the one is wrong, the deontologist must hold that you are morally special (to override the impersonal verdict and get that your murdering one is morally worse than allowing five other murders to occur), but you're not so special that your interest in saving your loved ones overrides your putative moral obligations. It's an awkward combination of claims to assert simultaneously.
How do you think the deontologist might best respond to this challenge?