Suppose young Timmy mistakenly takes 'prime number' to be roughly synonymous with 'cool number'. So he goes around saying things like '666 is a prime number'. Does he believe that 666 is a prime number? Presumably not. He certainly doesn't have a de dicto belief involving the concept prime number, since he lacks this concept (he associates the words, 'prime number', with a different concept entirely). Nor does he have any de re beliefs about primes, i.e. beliefs which talk about this property under a different guise: he does not believe, for example, that 666 is divisible only by itself and 1. What Timmy believes is that 666 is a cool number (or, more likely yet, that '666' is a cool numeral), and he mistakenly takes the sentence '666 is a prime number' to express this belief.
What of Timmy's meta-beliefs? He might not have any, if he's very young, but let's suppose that he's aware of himself as a believing agent. What does he think he believes? Jack suggests to me the following: Timmy believes that he believes that 666 is a prime number. But this attribution seems mistaken for exactly the same reasons. Timmy lacks the concept prime number, so he can't have any (even meta-) beliefs involving it. And nor can he have any de re beliefs about primeness (under whatever guise), because he lacks any alternative grasp of the property in question. He's not capable of having primeness feature in his mental content at all.
Instead, I would suggest that Timmy has entirely accurate meta-beliefs. (We have no reason to doubt his introspective abilities.) He believes, truly, that he believes that 666 is a cool number. That's all. It's only his linguistic beliefs that are false. For example, he falsely believes that he can express his above (true) meta-belief by asserting, 'I believe that 666 is a prime number.' He can't; this assertion means something different from what Timmy thought. It means something that happens to be false, whereas all Timmy's non-linguistic beliefs are true.
The upshot of this is that sincere assertions do not always succeed in expressing your beliefs. Linguistic errors may mean that what you end up saying actually means something different from what you believe (i.e. what you meant to say).
This seems to me the tidiest way to make sense of what's going on in these cases. Is there any residual problem that the above analysis fails to deal with?