Many people claim that morality is defined in terms of the beliefs that are widely accepted in a society. Thus Melbourne Philosopher, for example, suggests that "it cannot, by definition, be a moral act to do something immoral in the eyes of the wider populace." Now, I think this claim is quite easy to refute. Simply approach any competant speaker of the English language, and ask them whether the sentence: "My society approves of slavery, but slavery is wrong." is a self-contradiction. It clearly isn't. So morality must not be defined in terms of societal approval after all.
Why do so many people believe a theory that's so clearly mistaken? It could arise from the failure to distinguish between descriptive and normative "morals". The former concerns those norms that happen to be accepted within a society, as a matter of descriptive fact. Normative ethics, by contrast, concerns how people ought to act, or what norms ought to be accepted by society. Hopefully you can see that these are two very different concepts. And note that moral philosophers are interested only in the latter. (The former is a subject for anthropologists.)
Another important distinction is between moral beliefs and the "truthmaker" for those beliefs. Moral facts concern not what people believe, but rather, what makes those beliefs true (if indeed they are true). It seems clear to me, given what we mean when talking about normative ethics, that moral facts are independent of what anyone happens to believe about them.* Again, this linguistic intuition is supported by the "slavery" example above.
I suspect that what really motivates many relativists is their skepticism about moral realism. They do not believe that there are any mind-independent "moral facts" existing out there in the world. Fair enough. But relativism is not the most plausible form of anti-realism. There are other, better, options. The Ethical Werewolf offers a good introduction to two of them: error theory, and simple non-cognitivism. (According to the former, moral beliefs are simply all mistaken, like religious beliefs are if God does not exist. On the latter account, moral attitudes are not beliefs at all, but rather serve to express an individual's emotive attitudes, or such like.) Better yet, one might opt for a meta-ethical theory that's closer to the borderline between realism and anti-realism, such as constructivist non-cognitivism, or reductive ethical naturalism. These provide plausible truth-makers for our moral beliefs (rather than the patently absurd** "whatever society approves of"), without committing us to the bizarre metaphysical entities of non-reductive moral realism. Follow the links for more detail.
* (At least, people's actual beliefs are irrelevant. Perhaps some particular "ideal"/hypothetical beliefs could be relevant - see here.)
** (Again, it's just clearly false on linguistic grounds. The folk concept of "morality" does not involve agent- or cultural-relativity. Any analysis which suggests otherwise is simply changing the subject.)
P.S. Don't miss Hilzoy's excellent post on why people often think that they are moral relativists when really they're not. Relatedly, David Velleman explores some reasons why people claim to be attacking "moral relativism" when their real target lies elsewhere.