I previously suggested that an 'ideal agent' account would reduce down to my actual desire account. I thought the instrumental theory of rationality would basically guarantee that a fully informed and rational version of ourselves would simply choose to maximize the fulfillment of our desires. But as David pointed out in a comment:
It's clear that A cannot desire X as an object of ultimate desire until A becomes aware of it. Let's say that an "unknown object" (a "UO") is an object which A is unaware of and does not desire, but would desire as an ultimate end if A were made aware of it. If any UO's exist for A, then as A acquires more of the right kind of knowledge, A acquires more desires of objects as ultimate ends.
In other words, the idealized agent A+ might begin with only A's desires, but as he gained more knowledge, he might judge some previously unknown object X to be of worth, and thus come to have different values/desires from A. He might think it best for A to pursue X, even though A (due to ignorance) does not actually desire X.
Is this a problem for the actual desire theory? I'm not sure. I think we first need to answer the question, how does A+ come to desire X? How do we adopt new ultimate ends? It would seem unacceptably arbitrary if there was no basis for the new desire. But surely the only basis available is that of A's pre-existing desires?
As Railton writes in 'Taste and Value' (in Crisp & Hooker, Well-Being and Morality), p.55:
Of course, as a fool I have no antecedent desire identifiable as a desire to lead a more reflective or more Socratic life. But, if my motivational set contained no potential positive sentiment that could be 'recruited by' the information I gain about the Socratic life, then how could my novel exposure to the Socratic have any tendency to engage me...?
(It's important here that we understand desires in a broadly dispositional sense, rather than requiring an explicit 'token' to be present in the mind. I discussed an analogous treatment of beliefs here.)
So I would suggest that there is a sense in which any new desire must be grounded in our old ones. On this view, the new desire would be merely instrumental to begin with, since it arose for the sake of the more fundamental desires that were its basis. But perhaps over time it would develop into a 'self-standing' desire which you value for its own sake, independently of its original basis. I've no idea how this transition (from instrumental to intrinsic desire) is supposed to occur, but it does sound vaguely plausible to me that our desires can develop in such a way.
An intriguing issue arises when we trace this process all the way back to infancy. It seems that the regress must stop at some set of innate desires, presumably for very basic goods like food, pleasure, warmth, parental love (I guess that one's more complex!), etc. Our life experiences might then lead us to associate new objects with the fulfillment of our basic desires, eventually causing us to desire the new objects in their own right. Compounding development then opens us up to more and more complex goods, right up to the pleasures of philosophy itself! ;)
Well, it's a pretty picture, but this is all empty speculation on my part. So I'd like to put out a request to Chris to explain the cognitive science behind it all... Has there been much research into how people form desires/goals, etc.?