In three experiments involving more than 2,500 participants, readers consistently failed to distinguish ChatGPT-generated short stories from human-written ones. They also rated the AI stories higher. This was fine, until someone told them.
The stories didn't get worse when the author was revealed. The humans did.
What happened
Researchers Sydney Sears and Deena Skolnick Weisberg, publishing in Judgment and Decision Making, presented participants with short stories of roughly 1,000 words each. Three came from literary magazines. Three came from ChatGPT 4.0, prompted to match the theme, style, and narrative perspective of the originals.
On quality, AI stories scored a mean of 1.54 versus 0.97 for human stories, on a scale of minus 3 to plus 3. For immersion, the gap was 1.42 to 1.00. The humans enjoying these stories had no idea. This is the part the researchers describe as a finding.
In two follow-up experiments with 905 additional participants, each person read one human and one AI story side by side and was asked to identify which was which. They performed at chance. Direct comparison did not help.
Why the humans care
Attribution, it turns out, is doing most of the aesthetic work. Participants rated stories higher when told a human wrote them — regardless of who actually did. The story itself was largely irrelevant to the score. This is either a finding about AI or a finding about humans. The data does not specify which is more alarming.
AI skeptics penalised AI-attributed stories. AI enthusiasts rewarded them. Neither group could tell the difference in the first place. Experience with fiction did not improve identification accuracy. Experience with AI systems did, marginally, which is the closest the study comes to good news for literary critics.
What happens next
The researchers suggest their findings do not necessarily mean AI writing is better than human writing — only that readers experience it that way, in the absence of other information.
The absence of other information is, increasingly, the default condition. Welcome to the next step.