Are AI detectors biased against non-native English writers?

The study is Liang, Yuksekgonul, Mao, Wu and Zou, "GPT detectors are biased against non-native English writers", published in Patterns in 2023. The Stanford team ran seven detectors on two sets of human writing: TOEFL essays by non-native speakers and essays by US eighth-grade students. The detectors misclassified over half of the TOEFL essays as AI-written. The US student essays were classified almost entirely correctly.

Key figures from Liang et al., 2023

Writing sample	Author group	Share wrongly flagged as AI
TOEFL essays	Non-native English speakers	More than half, across seven detectors
US eighth-grade essays	Native English speakers	Near zero
TOEFL essays after vocabulary enrichment	Non-native English speakers	Fell substantially, showing the detectors react to word choice

Why it happens

Many detectors lean on perplexity, a measure of how predictable each next word is to a language model. Writers working in a second language often use a smaller vocabulary and more common sentence patterns. That makes their text more predictable, which is exactly what the detectors associate with machine output.

The researchers tested this directly. When they used a model to enrich the vocabulary of the non-native essays, the false positive rate fell. When they simplified the vocabulary of native-speaker essays, the false positive rate rose. The tools were reacting to linguistic sophistication, not authorship.

What to do with that

• Do not use a detector score as the basis for an accusation, and be especially careful where the writer is working in a second language.
• Ask for process evidence such as drafts, notes or version history, which do not depend on how polished the prose is.
• Check your tool's documentation for any published testing on non-native writing. Many vendors do not publish it.
• Read the wider accuracy picture in how accurate are AI text detectors, really before setting any policy on scores.