In theory, culture fit is a great thing to look for. Teamwork is vital, and high functioning teams produce better products. Culture fit is also more than just the do-you-get-along-well with the team. Culture fit can also be about moving rapidly, taking risks, speaking up, being creative, or not micromanaging.

You do want people who fit in well with your culture.

However, in practice, "culture fit" is often discriminatory, especially against asians, women, and older people.

Note: here, I'm not talking about the legal "discrimination." I'm using "discrimination" to mean "putting certain groups at a disadvantage, and probably unfairly."

When I've seen people express concerns about "culture fit," the culture has often not been defined, nor did anyone really try to drill in to the candidate's culture fit. The interviewer is saying "culture fit" but really they mean, "This person has a very different personality from me. I don't see them being my buddy."

This is problematic on multiple levels:

A diversity of personalities is healthy for a team. Diverse teams make better decisions. And plus, at some point, if your company keeps growing, you'll need to break out of a single personality mold. If you hire your first quieter nerd when you have 20 "brogrammers", you're going to have some problems. It only gets harder to make a team diverse, in terms of recruiting and integrating people.
There are ethical and legal issues with hiring for an attribute which is linked with race, gender, age, or another protected class. Some of these "personality" attributes are correlated with specific races, ages, or genders. For example, non-native English speakers are often "quieter" (in English) because they're not as comfortable in the language. (Note: I said issues. It's not necessarily illegal, but you do have to tread carefully.)
Your perception of their "culture fit" might be influenced by your own stereotypes (i.e., confirmation bias). Is that Chinese guy quiet and unassertive, or did you just assume he is because that's the stereotype in your head? Is that 50 year old slow to make decisions, or did you just assume that because of his age? Will that woman really not fit in with your team's environment, or are you just assuming that because she looks so different?

This doesn't mean that you can't look for a particular attribute if it places one group at a disadvantage.

Maybe you're working on a football game, so you really need people who love football. That might mean there will be fewer women and non-Americans who are good fits for the role.

That's okay -- sometimes. However, even there, you should be conscious of the decision you're making and the impact of it.

If you're going to look at "culture fit," proceed with caution:

Define, upfront, what you mean by culture. Are there specific attributes you want? Are there specific attributes you don't want? You should know what your culture is before asking if people fit in with it.
Sanity check your culture fit list. Do you actually want everyone like this? What does a company consisting of just people like this look like? Or do you want a mix? And if you care about diversity -- which you should (it's just good business) -- think carefully if this might put some groups at a disadvantage.
If culture fit is important, have designated people who will drill into it. Don't assume that someone is unassertive simply because they have a soft manner of speaking. These are not the same thing. Challenge the person's views. Do they back down?
Recognize that how someone acts in an interview does not necessarily mirror real life. People are generally on their best behavior during an interview. That person who strikes the perfect balance between being assertive but listening might well be rather pushy in real life. Remember, this is their I'm trying to make you like me act (and we all put on acts). Take their attitude with a grain of salt.

It's okay to prioritize culture fit, but you shouldn't prioritize anything unless you know what you're prioritizing, why you're doing that, and what the impact of this is.

Sadly, I've heard several friends -- both men and women -- remark that they would be wary of hiring a woman at their startup because of the "culture fit." If people are willing to admit this to themselves, admit it to people in general, and vocalize it to me, how much more discrimination happens subconsciously or privately?

Note: I work with companies and these other hiring issues. Come talk to me if you have questions.