It's worth highlighting that this premise can't be right, for we can talk about things that do not causally affect us. We can even talk about things that don't exist, like unicorns, or God. (Or does Eliezer think that 'God' refers to the religious module of the brain, so that God exists after all?)
I'm reminded of Hilary Putnam's a priori semantic response to the skeptical scenario that I might be a brain in a vat (BIV). The idea is that if I were really a BIV then my terms like 'brain' and 'vat' would instead refer to the objects in the hallucinated 'image' (i.e. in the phenomenal world). So, the argument goes, whatever the words end up meaning, 'I am a brain in a vat' is guaranteed to come out false. I can "know" that I'm in the fundamentally real world, just because it's (allegedly) impossible for my term 'fundamentally real world' to refer to anything other than the phenomenal world that I'm presented with.
Clearly, something has gone wrong in these arguments. The appropriate response, I think, is to say "Stop redefining my words!" I can understand the BIV scenario perfectly well, and it's a scenario I cannot rule out with absolute certainty; hence 'I am a BIV' is not guaranteed to be false at all. Putnam's argument to the contrary is a mere semantic trick, playing with words.
The same is true of Eliezer. We know perfectly well what we mean by the term 'phenomenal consciousness'. We most certainly do not just mean 'whatever fills the role of causing me to make such-and-such utterances'. By suggesting this, he is playing Humpty Dumpty, redefining words to mean whatever he wants them to mean. It's simply changing the subject. (I never claimed that 'whatever fills the role of causing me to make such-and-such utterances' is physically irreducible. So to argue against this claim is not to address my claim that 'consciousness is irreducible'. It's just like arguing against atheism by redefining 'God' to mean 'the universe', or some such silliness.)
Eliezer has recently repeated the mistake in arguing for reductionism about identity (a view I actually share, though for different reasons). He writes:
Whatever-it-is which makes me feel that I have a consciousness that continues through time, that whatever-it-is was physically potent enough to make me type this sentence. Should I try to make the phrase "consciousness continuing through time" refer to something that has nothing to do with the cause of my typing those selfsame words, I will have problems with the meaning of my arguments, not just their plausibility.
Whatever it is that makes me say, aloud, that I have a personal identity, a causally closed world physically identical to our own, has captured that source - if there is any source at all.
And we can proceed, again by an exactly analogous argument, to a Generalized Anti-Swapping Principle: Flicking a disconnected light switch shouldn't switch your personal identity, even though the motion of the switch has an in-principle detectable gravitational effect on your brain, because the switch flick can't disturb the true cause of your talking about "the experience of subjective continuity".
This is a terrible argument, for the simple reason that whatever it is that makes me say 'X', is not necessary what I mean by 'X'. (Again: unicorns, God, etc.)
Cf. Linguistic Anti-Paternalism: each individual is the final authority on what they are using their words to express. To offer a brief sketch of how this might work: We may have a particular idea in mind, a bunch of descriptive platitudes about what it is to be X (which need not invoke causal relations at all, though it often will). Arguably, this is what decides the meaning of 'X', though there's no guarantee that it will successfully refer to anything in the world; that instead depends on whether there is any worldly thing which satisfies the platitudes.
Anyway, that's just a very rough and inadequate sketch of how an alternative view might go. But the purpose of this post is not to give a fully-fledged theory of meaning. It is simply to highlight one adequacy constraint on any such theory: it must make it possible for us to talk about, e.g., abstract or non-existent things, i.e. things which are not themselves the cause of our talk about them. (It needn't make epiphenomenalism true, but it had better be expressible!)