How many times as a female professor have I gone out to dinner parties with visiting speakers where there were several philosopher’s wives present (my other colleagues mostly being males), where the entire dinner table conversation was devoted to philosophical issues that excluded them? As a woman, I or perhaps simply as someone socialized to be more polite and empathetic, I face the choice then: should I try to join in with “the guys” and prove my mettle, thus ignoring half the people present at the table, or should I attempt to be more congenial and polite and talk to the women?
Now, from my perspective, the whole point of a bunch of philosophers going out to dinner with a visiting speaker is to discuss philosophy. That's what they're there for. To complain that "the entire dinner table conversation was devoted to philosophical issues" seems as bizarre to me as complaining that the entire seminar was dedicated to philosophy when some of the students might rather have discussed the local sports team. The problem does not necessarily lie with the topic of conversation; it could just be that the sports fans are in the wrong place.
More generally, it's nice to accommodate people and make them feel comfortable. But given that the lack of fit between 'nerds' and 'normals' is symmetrical, it's not clear why the norms of the latter group should always take precedence. I mean, there's no surer way to make me uncomfortable than to put me in a situation where one is expected to engage in small talk. That's just a fact about me and how I relate to others. Many people (outside of academia) seem to be just the opposite: uncomfortable with serious discussion, comfortable with small talk. That's a fact about them and how they relate to others. Each of these two personality types may find it difficult to relate to the other. Objectively speaking, that's the end of the story. But in practice the extroverts are socially dominant, so they lay fault on the nerds and introverts for failing to conform to their preferred (arbitrary) norms. What they don't seem to realize is that they are equally failing to conform to our preferred norms.