AI detectors are far less accurate than their confidence scores suggest, and they fail in a specific, well-documented direction: they flag genuine human writing as AI, especially writing by non-native English speakers. The research on this is not ambiguous. A peer-reviewed Stanford study found that seven leading detectors labeled real human essays as AI-generated 61% of the time when the author was a non-native English speaker. If you have ever wondered whether a detector score can be trusted as proof, the short answer is no. Here is what the data actually shows.
Key facts
- A 2023 Stanford study (Liang et al., published in Patterns) found seven major AI detectors misclassified non-native English essays as AI-generated 61.3% of the time, while flagging native-English essays at near 0%.
- Turnitin's own stated false-positive rate is less than 1% at the document level, and around 4% at the sentence level.
- Vanderbilt University disabled Turnitin's AI detector in 2023, noting that at a 1% error rate, about 750 of the 75,000 papers it processed in a year could have been wrongly flagged.
- Independent analyses commonly report real-world false-positive rates well above the vendors' published figures, depending on the tool and the text.
- A December 2025 benchmark (Grammarly's BAID) found the bias against non-native writers still persists in current detectors.
How accurate are AI detectors, really?
No AI detector is reliable enough to treat as proof. Detectors do not read text and "know" whether a machine wrote it. They measure statistical patterns, mainly perplexity (how predictable each word is) and burstiness (how much sentence rhythm varies), and then guess. AI text tends to score low on both. The problem is that plenty of human writing scores low too, so the guess is wrong often enough to matter. Accuracy is highest on long, unedited AI text and drops sharply on edited text, short passages under 200 words, and any writing that mixes human and AI input.
What is a false positive, and how often does it happen?
A false positive is human writing that a detector flags as AI. It is the failure that does real damage, because it falls on people who did nothing wrong. Turnitin reports its own document-level false-positive rate at less than 1%. That sounds small until you apply it at scale: Vanderbilt pointed out that 1% of the 75,000 papers it ran through Turnitin in a single year would be roughly 750 students wrongly flagged. Independent testing tends to find higher real-world rates than the vendors publish, and far higher for specific groups of writers covered below.
Why do AI detectors flag non-native English writers?
This is the most serious and best-documented failure. The 2023 Stanford study by Weixin Liang and colleagues ran 91 essays written by non-native English speakers through seven detectors. On average the tools flagged them as AI 61.3% of the time, and every one of the seven agreed on the wrong answer for about 1 in 5 essays. Native-English essays were almost never misflagged. The cause is mechanical: non-native writers tend to use simpler, more common vocabulary and steadier sentence structure, which produces exactly the low-perplexity signal detectors read as "machine." The researchers proved the link by rewriting the essays with richer vocabulary, which dropped the false-positive rate from 61% to about 12%. A December 2025 benchmark confirmed the bias has not gone away.
Why did Vanderbilt disable Turnitin's detector?
Vanderbilt University turned off Turnitin's AI writing detector in August 2023 and said plainly: "we do not believe that AI detection software is an effective tool." Their reasoning was both mathematical and practical. At Turnitin's stated 1% false-positive rate, about 750 of the 75,000 papers Vanderbilt submitted in 2022 could have been incorrectly labeled as AI. They also noted that reliable AI detection is "a very difficult task for technology to solve (if it is even possible)." Other universities made similar calls. When the institutions buying these tools stop trusting them, that is a strong signal about their accuracy.
Why a 99% confidence score is not 99% proof
A detector that reports "98% AI" is not saying there is a 98% chance a machine wrote the text. It is reporting how closely the text matches the patterns in its training data. Those are different claims, and the gap between them is where people get hurt. Base rates make it worse. Imagine a batch of writing where 10% is actually AI and 90% is human, and a detector with a 5% false-positive rate. Out of 90 human pieces, it wrongly flags about 4 to 5. Out of 10 AI pieces, it correctly flags about 9 to 10. So roughly a third of everything it flags is human writing that was never AI. The score is a signal, not a verdict.
What do the detector companies themselves say?
Every major detector, including Turnitin, GPTZero, and Copyleaks, publishes language warning that its results should not be the sole basis for an accusation or a penalty. OpenAI went further: it retired its own AI text classifier in 2023, citing its low rate of accuracy. In practice that warning is routinely ignored: the tool says "AI," and a student gets a zero or a freelancer loses a client. The companies are, in effect, telling you their own numbers are not proof. It is worth taking them at their word.
What to do if your writing is falsely flagged
If genuine human work gets flagged, do not rewrite everything in a panic. A flag is a statistical guess, not evidence. Cross-check the text in two or three different detectors, since a flag that appears in one tool but not the others is almost certainly a false positive specific to that model. Keep your drafts, outlines, and version history, because evidence of a writing process is far more persuasive than any score. And if you are a non-native speaker or you write in a formal register, say so, because both are documented causes of false positives. For a step-by-step version of this, see our companion post on whether AI detectors can be wrong.
How to check your own writing before it gets flagged
A practical habit is to score your own work before you submit it, not to game anything, but to see what your writing looks like statistically. Metric37 offers a free AI detector that returns a human score from 0 to 100 with no account required. Paste any text, human-written or not, and see where it lands. If your own honest writing scores low, you now know it could trip a detector, and you can decide whether to vary your sentence length or add specific detail before it becomes someone else's problem. The score is information, not a guarantee, which is exactly how every detector score should be treated.
The bottom line
AI detection is useful as a rough signal and dangerous as a verdict. The published research, the detectors' own disclaimers, and the universities walking away from these tools all point the same way: the scores are unreliable, the failures land hardest on non-native and formal writers, and no one should lose a grade, a gig, or a reputation over a number a model is not even confident about.