It's certainly true that "pro-life" extremists who can't tell the difference between a zygote and a person must deny that such circumstances can be relevant. (We wouldn't allow an amnesiatic mother kill her ten-year old son after uncovering the suppressed memory that he was a product of rape. The child's genesis is not his fault, after all, so it would still be an instance of killing an innocent person.) But since no-one in their right mind would deny a rape victim the right to an abortion, this just goes to show that no-one in their right mind is a pro-life extremist.
At the other extreme, you have those who think that people can do what they like with their bodies, for even the most fickle of reasons, and so the extra reasons for abortion provided by rape or incest would be entirely superfluous. As a consequentialist who can tell the difference between zygotes and persons, I'm reasonably sympathetic to this view. No harm is done, even by fickle abortions, so there are no grounds for thinking it's fundamentally immoral. (Perhaps we can derive reasons from indirect utilitarian reasoning, however. It's plausible that we really ought to be virtue ethicists in our everyday lives, so I can potentially endorse the arguments that emerge below.)
But again, most people aren't consequentialists. (And just as well, I say.) Most people probably don't have any coherent theoretical views which systematize their various moral intuitions. But we might charitably see them as virtue ethicists of a sort. They want to be good people, and having abortions for no good reason isn't the sort of thing a virtuous person would do. Special circumstances - such as rape or incest - can obviously have an impact here, then.
The ever-insightful Hilzoy of Obsidian Wings puts the matter thusly:
I think that we have an obligation to treat any form of human life, from conception onwards, with a certain sort of respect. What respect requires changes as a blastocyst develops into an embryo, a fetus, and then an infant, but I think it's there from the start. (And this doesn't require that I think that an embryo is "really" a person: I also think we ought to treat corpses with respect.)
At this stage, what respect seems to me to require is just this: that whatever we do to a blastocyst, we should have good reasons for doing, reasons that are not just a matter of our own pleasure or entertainment. Just as I would think it wrong to carve up a corpse for fun, I would think it wrong to kill a blastocyst for one's own amusement or convenience. (This is why, though I am in favor of legalized abortion, I would think that someone who decided, on a lark, to try to have abortions in each of the fifty states, or who had an abortion just so that she could go to a really great party in her favorite dress (which, let's suppose, wouldn't fit if she were pregnant), was doing something abhorrent. I think you just shouldn't kill embryos for these sorts of reasons. I wouldn't be in favor of making such abortions illegal, but that's partly due to the same sorts of reasons that lead me not to favor a legal ban on being a complete insensitive jerk to one's significant other: doubts not about whether it's wrong, but about whether it would be a good idea to have the state policing our motives in the ways that would be required if it were criminalized.)
But while I think it's wrong to kill a blastocyst for no good reason, I don't think it's wrong to kill one period. I just think that respect requires that one recognize that one is doing something that should not be done lightly. I feel similarly about corpses: I do not think that carving them up for fun is OK, but I do think that a person who thinks her corpse should be treated with respect can appropriately leave her body to science, where it will be carved up for a non-frivolous purpose.
I suspect that what really upsets most people about abortion is the idea that it might be done frivolously. We may be more confident that rape victims are not being frivolous in this respect. Hence, on the common-sense (or virtue-ethical) view, rape can be relevant to the morality of abortion.
As it happens, I think people's concerns here are probably empirically misguided. The frivolous hypothetical abortions described by Hilzoy are clearly not everyday occurrences. Surely no-one could seriously believe that most women who seek abortions are so fickle about it. Abortion is a serious choice, and I have trouble believing that many women would fail to see it as such. Conservatives often whine about "abortions of convenience", but I find it incredible to dismiss as mere "inconvenience" the massive responsibility inherent in bringing a child into the world.
(Perhaps some people are reckless about getting pregnant in the first place, and hence blameworthy for that. But once pregnant, abortion will often be the best and most responsible option. While it would certainly be preferable if there were no unwanted pregnancies, and hence no need for abortion, as things stand I rather suspect that people don't have abortions enough. This is for the general reasons discussed in my post Badness Without Harm, and illustrated in Parfit's "14 year old mother" example. The world would be a better place if people only had children when they were most capable of raising and providing for them. There's nothing "frivolous" about such concerns, and so I think it is misguided to decry such "abortions of convenience". In saying this, I grant that it might have been wrong for someone to get into this position in the first place. So, by all means, do feel free to condemn "unsafe sex of convenience", and advocate better sex education, contraceptive provision, etc. But leave off the irresponsible anti-abortion diatribes already.)