In Chapter 34, Parfit seems to accept the basic validity of the argument from disagreement. He writes that if...
even in ideal conditions, we and others would have deeply conflicting normative beliefs, [then] it would be hard to defend the view that we have the intuitive ability to recognize some normative truths. We would have to believe that, when we disagree with others, it is only we who can recognize such truths. But if many other people, even in ideal conditions, could not recognize such truths, we could not rationally believe that we have this ability. How could we be so special?
But what part of this argument depends on the "others" in question actually existing? It seems enough to consider a possible alternative psychology, like the Future Tuesday Indifferent agent (or similar, less gerrymandered, characters), who Parfit takes to be fully procedurally rational but merely substantively mistaken. When we consider this world of procedurally rational lunatics, we would have to believe that it is only we, and not they, who are capable of recognizing normative truths. But then the very same question arises: "How could we be so special?"
If we can answer this in relation to possible alternative views (e.g. by answering "we're just lucky!"), then presumably this very same answer will apply in case of actual disagreement. Yes, if I'd been raised differently, I might have ended up a Kantian, or a Creationist. So isn't it a fine stroke of luck that I wasn't raised in such a misguided (if well-meaning) community!
But Parfit doesn't take this tack. Instead, he seems to see the case of possible disagreement as less troubling than cases of actual disagreement. See, for example, his response to the worry that "different people might find conflicting beliefs self-evident":
If we claim that we have some ability, however, it is no objection that we might have lacked this ability. Different people might have conflicting visual experiences, which were like dreams and hallucinations, and were not a source of knowledge. But that is not in fact true. Different people's visual experiences seldom conflict, and believing what we seem to see is a fairly reliable way of reaching the truth. It may be similarly true that, after careful reflection, different people would seldom find conflicting beliefs self-evident. Believing what seems self-evident, after such reflection, may be another fairly reliable way of reaching the truth.
I'm not sure what Parfit has in mind here. Is he just making an externalist argument: that what matters is that our faculties are truly reliable, whether or not we can show it? But then why would it matter whether or not "different people", with different faculties from us, were constantly hallucinating? (That wouldn't change the fact that believing what we seem to see would be a reliable way for us -- if not for them -- to reach the truth.) It's odd. Can anyone else make sense of how to reconcile these quoted passages?
I think Parfit would do better to stick to his guns in the manner suggested by (one part of) his response to Street in an earlier chapter on epistemology (32 or 33, I think?):
Some whimsical despot might require us to show that some clock is telling the correct time, without making any assumptions about the correct time. Though we couldn't meet this requirement, that wouldn't show that this clock is not telling the correct time. In the same way, we couldn't possibly show that natural selection had led us to form some true normative beliefs without making any assumptions about which normative beliefs are true. This fact does not count against the view that [we know] these normative beliefs are true.
At the end of the day, the only way to avoid radical skepticism is to insist that we are, in a sense, epistemically lucky. There are alternative starting points that we might have found ourselves with, and no non-question-begging way to argue for the superiority of our actual starting points. Still, that in itself is no reason to abandon them. We might be wrong, but we may as well take ourselves to be right -- for that way we at least have a chance of being right.