Turnitin recently rolled out a new update for its AI writing detection model, and it changed how papers get scored. Over the past few weeks we studied thousands of real reports from Turnitin's new model update and then updated our in-house classifier based on what we found. Here's what we learned, how closely we now match Turnitin, and what we're still improving.
What changed in Turnitin's October update
On October 14, Turnitin updated its English AI writing model, its second core English model update this year after one in February. The release note says only that it "improved AI detection" while minimizing false positives. They usually don't give anymore details than that, so we we investigated ourselves. We compared about 27,000 reports from their previous model update, with about 20,000 from the October 14th model update.
- It flags shorter stretches. On mixed papers, the typical run (amount in a row) of flagged sentences dropped from 7 to 4, and single-sentence flags went from 12% of runs to 27%. The gaps between flagged passages shrank too, from about 10 sentences to 4. The new model switches between flagged and unflagged text far more often, which makes mixed papers harder to predict as a third party.
- AI-heavy papers tend to score lower. Among papers scoring 20% or more, the typical score fell from 69% to 54%, and papers at exactly 100% went from 17% of reports to 5%. Some of this may come from the papers rather than the model, since the two sets are different papers from different months.
- Fewer flags on text that doesn't count. Lines without a full stop and reference sections are flagged less often than before (11% vs 13%, and 5.5% vs 8%), so the flags concentrate on full sentences.
- One highlight colour. Since a minor interface update on August 4, Turnitin reports now only show a single blue highlight. The separate purple layer for AI-paraphrased text was removed, and that text is now highlighted the same as any other AI flagged writing.
Some things didn't change. The Turnitin AI percentage score is still derived from the number of flagged sentences divided by the number of counted sentences (x100). Between 1% and 19%, Turnitin still shows an asterisk and highlights nothing, which for end users, is effectively a 0%.
These changes are partly why our detector which was trained on hundreds of thousands of Turnitin's earlier reports, drifts on the new ones. However, the amount of drift was not meaningful enough to declare our old model "useless" by any means, it was still extremely accurate on most documents.
How we updated our platform
Our previous model was trained before Turnitin's update, and on reports from the new model it drifted from Turnitin's scores. So we updated it.
- Training on Turnitin's newest reports.
We retrained our in-house classifier on tens of thousands of real Turnitin reports, including thousands from the last few days, ensuring it was able to learn how to predict sentence classification the same way as the new Turnitin update. - It reads sentences in context.
Instead of judging each sentence on its own, the classifier reads long stretches of a paper at once, which is how it is able to replicate Turnitin flags with a high degree of accuracy. - Turnitin's sentence counting.
We apply the same rules Turnitin uses to decide which sentences count, so our percentage is worked out the same way theirs is. We also apply a curve that has been fine tuned to increase accuracy even in cases where we may not come to the same sentence count conclusion as Turnitin.
How accurate it is now
After all updates, we test our model on real reports from Turnitin's new model that our model never saw in training. The figures in this section come from ~570 unseen reports:
- Our average gap from Turnitin's score is 2.2 points, down from 12.6 with our old model.
- Every one is within 10 points of Turnitin, 502 are within 5 points, 299 are within 2 points, and 155 match Turnitin's score exactly.
- Papers come from every part of the range: 99 that Turnitin scored 0%, 181 at 1–19%, 196 mixed papers at 20–79%, and 93 at 80–100%.
When we say 0%, you can trust it
The question most of you care about is: if we show 0%, will Turnitin agree? Our money-back guarantee is centered upon his, so it matters a ton...
When we show 0%, Turnitin comes in under 20% on every paper in the test (131 of 131), and at exactly 0% on 70% of them.
A clean 0% is one with no warning on it. Now, with our new update when a paper scores 0% but has a few sentences close to the line, we now show a warning under the score, so you know Turnitin may show a low score rather than 0%. With a clean 0%, Turnitin showed exactly 0% on 78% of papers; with the warning, on 18%. And because Turnitin shows no percentage and no highlights below 20%, a Turnitin result under 20% is a 0% in theory.
Of the 99 papers Turnitin scored 0%, we showed 0% on 92, with an average gap of just 0.2 points.
Fully AI-written papers
At the opposite end of the spectrum, fully AI-written papers are where our model update is strongest. Of the 24 papers in the test that Turnitin scored 100%:
- we showed exactly 100% on 20, and 95% or more on all 24;
- we flagged 99.0% of their sentences (2,273 of 2,296).
Across heavily AI-written papers more broadly (Turnitin 80–100%), our average gap is 2.3 points, down from 17.7 with our old model.
Mixed papers (20–79%)
Papers that mix human and AI writing are the hardest for any detector, us included. On the 196 mixed papers in the test, where Turnitin scored between 20% and 79%, our average gap from Turnitin's score is 2.9 points, down from 15.3 with our old model.
Matching Turnitin's sentence splitter
Because Turnitin's percentage is flagged sentences divided by counted sentences, how a paper is split into sentences matters almost as much as which sentences get flagged. If we count 80 sentences where Turnitin counts 100, even perfectly flagging every sentence still would result in vastly different document AI percentage.
With this new update, we trained our own sentence counter on millions of sentences from the latest Turnitin reports, exactly as Turnitin classified them. It now reproduces 95.4% of Turnitin's sentences exactly (same first word, same last word), up from 90.3% with the general rule based splitter we used before.
What's next
It would be pointless to stop striving for higher accuracy. Our goal is to bring our sentence-level accuracy on papers to the same level as our percentage score accuracy. We plan to continue:
- collecting more of Turnitin's newest reports, focused on the mixed papers where we miss most;
- running targeted tests to learn more of Turnitin's splitting and counting rules;
- refining the splitter and the classifier together.
p.s. I can also confidently say that we are once again the most accurate Turnitin AI detector on the market (not sure if that ever changed). Anyone claiming near perfect accuracy (sentence and percentage) across all document types is a blatant liar. That sort of statistic is actually just impossible, Turnitin even makes mistakes that are impossible to replicate. They also can classify identical word and pdf documents differently due to a perhaps a broken hidden sentence splitter, something that is once again basically impossible to replicate. Unless you have millions, or even tens of millions of reports as training data (including target documents), it would be impossible to get a model to learn enough of what their classifier produces in order to mimic the small discrepancies that can change scores. A reliable (constant result) cap on these sort of models is maybe in the mid 90's% (percentage and sentence accuracy) across all document types, and even that is extremely high.
However, I can still confidently say that Turnitin's false positive rate is extremely low from testing. Something you should never be worried about. If our platform, Turnitin, (or Pangram even, another great detector) scores your document high, it should be pretty safe to assume that its not human written.
If you see a result that looks off, let us know through any of our support channels.
