Since launching our in-house AI detector, we’ve been working on two things: how accurately the model flags AI-written sentences, and how clearly your report shows those results.
This post covers our February and March updates, including new training runs, changes we made after noticing a shift in Turnitin’s results, and a new way to show sentences the model isn’t sure about.
Keeping up with changes in Turnitin
Around February 13, we noticed Turnitin scoring documents lower for AI than it had before. On February 15, we released a preliminary model update to account for it, then followed with another update on February 18 as we collected more examples.
Initial testing on February 19 showed about 90% sentence-level accuracy against Turnitin. In that test, the two systems agreed on whether a sentence was AI-written about 90% of the time.
Improving the score and the highlights
Your overall percentage and your sentence highlights are two different parts of the result. Turnitin calculates the percentage by dividing AI sentences by total sentences, but it leaves some text out of that count, like references and calculations. When our sentence count doesn’t match, the percentage can differ even when the highlights match.
For example, if Turnitin counts 20 sentences and we count 23, the same 16 flagged sentences come out to 80% on Turnitin and about 70% on Academi.cx.
On March 3, sentence highlights were testing at over 85% accuracy against Turnitin, while overall percentages still needed work. That’s why we recommend focusing on which sentences are flagged, not the percentage alone.
On March 19, we finished another training run that brought accuracy to about 90%, and started testing a way to show the model’s confidence in your report.
A clearer display for uncertain sentences
Our March 19 test placed a raw prediction score beside sentences above a certain threshold. On March 20, we changed how it works. Sentences the model is confident are AI-written get highlighted, with no score beside them. Sentences it’s less sure about show their score and stay unhighlighted.
That way you can tell a clear flag from a borderline sentence at a glance.
Continuing to refine the model
We’re still working on bringing both the overall score and the sentence highlights closer to Turnitin, and we’ll post updates as bigger changes go live.
