How Do AI Detectors Work? Accuracy, False Positives, and What a Score Means

AI detectors don't read for meaning the way a person does. They look for statistical regularities that tend to separate human and AI writing in the examples a method was built or tested on. Published work keeps returning to two signals: how predictable each word is, given the words before it, and how much sentence length and rhythm vary across a passage. Language models are trained to predict likely next words, so their output tends to sit near the predictable end of that range. Human writing is lumpier: a long sentence, then a short one, then a word choice nobody else would have made.

One published method, DetectGPT, turns this idea into a test. It uses a second language model to make many slightly altered copies of a passage. Then it checks whether the model under test rates the original as clearly more probable than the copies. Text that model generated tends to pass this test. Human text is less likely to. On news articles generated by a 20-billion-parameter model, DetectGPT reached 0.95 AUROC, against 0.81 for the best earlier method of its type. AUROC is a 0-to-1 score for how well a method ranks AI text above human text. That result belongs to the text and model the authors tested. It isn't evidence that every detector does that well on every kind of writing, as the next section shows.

Less accurate, and less consistent, than a single score suggests. A 2023 study in the International Journal for Educational Integrity ran 12 publicly available detectors and two commercial systems used in education on 54 test documents. The set included human writing, human writing machine-translated from other languages, ChatGPT output, and ChatGPT output edited by hand or by a paraphrasing tool. The researchers found the tools "neither accurate nor reliable." Every tool scored below 80% accuracy, and only five scored above 70%. The tools also leaned toward labeling text as human-written. The authors estimate that about 20% of AI-generated texts would likely be attributed to humans. Accuracy fell further on the edited and paraphrased AI text, and the machine-translated human writing was wrongly flagged more often.

Yes, and in two directions. The University of San Diego Legal Research Center's guide to AI detection tools covers both: false positives and false negatives. The table defines them, along with two terms that decide how often each one happens.

The threshold ties the two errors together. Lower it, and the tool catches more AI text but also flags more human writing. Raise it, and fewer human writers get flagged while more AI text slips through. Any tool that turns a score into a yes-or-no label has this trade-off built in. The San Diego guide makes the same point: detection companies have to balance the false-positive rate against the false-negative rate.

A false-positive rate also has to be read against the number of texts checked. The University of Nebraska–Lincoln's Center for Transformative Teaching spells out what a rate under 1% means in practice: fewer than 1 in every 100 submissions would come back saying a student cheated when they did not.

Why different detectors give different results

Detectors differ in the examples they were trained or calibrated on, in where they set their threshold, and in the kinds of writing they were tested on. A 2025 survey of detection research in Computational Linguistics reports substantial drops in performance when detectors meet a new subject area. It also describes inconsistent results across languages. Detectors can do worse on text from generators they never saw in training, and current detectors struggle with short texts. New generators keep appearing, so a detector that handled older output well can fall behind. One tool's score says little about what another would report.

The Patterns study links its 61.3% false-positive rate to word choice. Many detectors rely on perplexity, a measure of how predictable each word is to a language model. Low perplexity reads as a sign of AI. The TOEFL essays that all seven detectors flagged had significantly lower perplexity. The authors warn that such detectors may penalize writers who use a more limited range of expressions.

The researchers then tested that explanation. They had ChatGPT rewrite the TOEFL essays to "sound more like that of a native speaker." The average false-positive rate across the seven detectors fell from 61.3% to 11.6%. Simplifying the word choices in the US essays had the opposite effect, and misclassification rose substantially. The tools were reacting to predictable phrasing, not to who wrote the text. That mechanism can reach beyond non-native English. Anything that narrows word choice or evens out sentence rhythm can produce a similar signal:

• Non-native English writing: a more limited range of expressions reads as more predictable, as the study found.
• Heavily edited or formulaic drafts: a templated lab report, or a draft revised again and again toward conventional phrasing, can lose the same variability.
• Short or structured text: a few lines leave little room for the sentence-to-sentence variation detectors look for.
• Technical and legal writing: fixed terminology and set phrasing are predictable by design.

None of this is evidence of AI authorship. It is evidence that a text is more predictable than average, which is what these tools actually notice.

A detector score tells you how closely a document's statistical profile matches the patterns the tool associates with AI writing. It can't tell you who wrote the document, whether its citations are genuine, or whether the writing is any good. A low score doesn't clear a text either, because false negatives happen too.

The Certentis benchmarks page states the same limit in its list of what it does not claim: an AI score does not prove authorship. Read a detector score the way you'd read any single test result. It gives you context, not a conclusion.

A flag opens a conversation. It doesn't close a case against you. Stony Brook University's published AI-flagged paper evaluation process has instructors start by checking the flag against the detector's known limits. A likely false positive can end there. Next they compare the flagged work with the assignment, the course AI policy and the student's earlier work. If concerns remain, they meet with the student before deciding on next steps. Your institution's process may differ. Read your course's AI-use policy before you respond, and keep your drafts, notes and document version history. Our guide on what to do if you're falsely accused of using AI walks through the full procedure step by step.

Our guide to signs of AI writing covers the patterns that show up more often in AI-generated text. AI and academic integrity explains how to disclose AI help and where assisting your writing ends and generating it begins. To see this kind of report for your own text, the Certentis AI Detector shows a document-level probability with sentence signals, as described above.