Presumably, what matters here are the reasons guiding the evaluation, or the standard against which the object is measured. But this simply shifts the question back a step: in virtue of what (we may ask) does a standard count as a peculiarly moral standard, rather than (say) an aesthetic one or an arbitrary personal choice? It is easiest to distinguish the latter; personal preferences make no claim to speak for a perspective larger than our own, whereas moral (and arguably aesthetic) judgments do claim this broader authority. Sidgwick puts it nicely in his Methods of Ethics:
The peculiar emotion of moral approbation is, in my experience, inseparably bound up with the conviction, implicit or explicit, that the conduct approved is 'really' right -- i.e. that it cannot, without error, be disapproved by any other mind. (I.iii.1)
If... I judge any action to be right for myself, I implicitly judge it to be right for any other person whose nature and circumstances do not differ from my own in some important respects. (III.i.3)
Precisely distinguishing the moral and the aesthetic (in a non-circular fashion) seems more difficult. We might say that morality is abstract and impartial,* essentially tied up with notions of universalizability, whereas judgments of beauty aren't beholden to anyone else's perspective, even as they exert a suggestive force (since everyone should appreciate what's truly beautiful). I'm not sure that's entirely adequate though. Any better ideas?
[* Disclaimer: the linked post is from 2005, and I would no longer endorse many of the assumptions found there. I just like the Railton quote.]