Detect and Punish
Should writers police AI-generated prose?
Welcome to The Third Hemisphere, where I try to make sense of how AI is reshaping work, thinking, and creativity, often by watching my own assumptions get upended.
If you were forwarded this and want to subscribe, click below. If you want to support a real human writing about AI, upgrade to paid.
I’ve given a few talks on AI to journalists where I run a simple exercise. First, I break the writing process into stages and ask: which would you feel comfortable using AI for? Many are fine using it for research; people are mixed on using it for editorial feedback; few are comfortable using it to generate prose.
I then hand each reporter one of two AI-generated research reports on collagen supplements. Same underlying studies, same data. Report A opens with positive clinical findings and mentions industry funding as a limitation in paragraph thirteen. Report B opens with the funding bias analysis, loudly labels which results are industry funded. I ask: what story are you primed to write? The answers are different, depending on which report they read. Report A primes a “does collagen work?” story. Report B primes a “why you can’t trust collagen research” story. Both reports are defensible summaries of the same literature. The AI just decided how to order and present the same information.
A reporter who reads Report A before doing any of her own research is more likely to spend the rest of her reporting anchored to the frame that collagen basically works and industry funding is a footnote. A reporter who reads Report B first will spend her reporting anchored to the frame that the research is corrupted. Both will write every word of their articles themselves. Both will pass any AI detector. Meanwhile, a reporter who did her own research and then asked an AI to help clean up her prose would get flagged. I’d argue she was the least influenced of the three, yet the moral intuitions of writers are that this last reporter betrayed the craft the most.
I’ve been thinking of this little exercise a lot lately because over the past month there’s been a new burst of outrage about AI and authorship, this time aimed not at the AI industry but at individual authors. Hachette pulled a horror novel called Shy Girl by Mia Ballard after readers and detectors flagged it as fishily machine-generated. The New York Times cut ties with a freelance book critic, who admitted an AI editing tool regurgitated passages from a Guardian article verbatim into his draft. The Atlantic reported that Kate Gilgan’s “Modern Love” column was flagged at over sixty percent likely AI-generated; Gilgan said she hadn’t copy-pasted anything but had used ChatGPT, Claude, Copilot, Gemini, and Perplexity for “inspiration and guidance and correction.”1
In certain corners of social media, AI detector screenshots get shared like mugshots. Pile-ons begin, and the whole affair has the grim energy of a public stoning.
I want to take the anger behind this seriously because I share it. Generative AI is being foisted on society by people with enormous power and very little concern about eliminating the market value of creative and intellectual work, work that gives meaning to my life. I do not think this is a moral panic. AI companies trained their models on writers’ work without permission or compensation. Freelance rates have collapsed. Publications that I value deeply are under even more economic stress than they were in the pre-AI internet era, and the people profiting from the collapse are among the richest humans who have ever lived. These same people, who five years ago performed progressive values, now wear their MAGA hats to dinner. For writers who identify with the political left, which is a lot of them, using AI to write your column isn’t a lapse in judgment. It’s cavorting with (or caving to) a political foe. So the punishment has the zeal of excommunication, the sting of betrayal, because in this mindset, that’s what it is.
But I’m not sure the impulse to detect and punish is productive. Sure, shaming and exiling can be framed as a sort of internal housekeeping, of maintaining editorial standards in the face of what some view as a corrosive technology. However, detection and punishment of individual writers using AI-generated prose polices one visible point of AI involvement and distracts from what I think might be the more consequential upstream impacts of the technology.2
My intellectual life straddles two worlds. I’m a longtime writer (I’ve published in Slate, the New York Times, and elsewhere) and my social circle is populated by writers and artists. But I’m also a scientist at an academic medical institution (NYU), and I’ve spent the past several years deep in AI research, education, and implementation across research and clinical settings. The conversations about AI and writing in these two worlds could not be more different.
When scientists and doctors talk about AI in their writing process, the conversation can get heated but does not tend to devolve into tribal sorting of the same fervor. Where in the workflow does AI help, where does it hurt, what are you giving up cognitively when you let a bot draft for you? I’ve argued extensively that writing itself has real cognitive value in science, since working through ideas in prose is part of how researchers clarify their thinking, and there’s probably a real loss when that gets offloaded. Is that a net gain or loss for science? Time will tell. In the clinic, AI documentation error rates are a real concern, but so are clinician error rates. How can we build a human-AI system that improves patient care without introducing safety issues, or is the deeper problem the sheer volume of documentation clinicians are expected to produce?
These conversations are possible in part because the stakes are different than they are in media and publishing. Scientists aren’t facing the economic threat from generative AI in the way that writers are. And a scientist’s professional identity doesn’t depend on writing in the way a novelist’s does. Those differences give science and medicine room to ask hard questions and actually study the answers. A study in Nature this year analyzed 41.3 million research papers and found that AI-using scientists publish three times as many papers and accumulate nearly five times as many citations, but that collectively, AI adoption shrank the range of scientific topics by about five percent and decreased engagement between researchers by twenty-two percent. Good for individual scientists; maybe not so great for science. A recent study on AI-assisted endoscopy feeds into broader concerns about clinicians losing diagnostic skill after relying on the tools. Because writing is one instrument among many in science and medicine, the question of AI assistance gets looked at with cool empiricism. Which uses help or harm individual scientists or doctors? Which uses help or harm the practice of science or medicine?
Although I don’t buy the claim that AI is “inevitable” in some manifest destiny sense, I also feel like the current incentives in media and publishing, as Max Read recently argued, make it highly unlikely it won’t be used by some writers at some stage of the writing process. A study published last month in Science Advances caught my eye because it tested what happens when people co-write with AI assistants that carry a built-in political bias. Over 2,500 participants wrote short essays with AI help. Their opinions shifted towards the bias of the tool. A majority didn’t notice the shift had happened, calling the AI’s opinions “reasonable and balanced.” This effect is strong: Not even warning the participants of the bias helped at all; they still drifted towards the model’s opinions. Just like in my collagen exercise I gave to journalists, I think it’s unavoidable that interacting with these tools at any stage of the writing process will deeply shape the final output, whether or not AI was used to generate the literal prose.3
I think it’s fine not to use AI tools and think it is healthy for the journalistic and literary communities to have abstainers; my contention is that some people will use AI, and it would be better to understand and shape that use through evidence and realistic professional norms rather than drive it underground by shaming.4 Take using AI for research, which many reporters seem to support: I’d like to know how AI-generated research summaries shape story framing compared to, say, searching for original papers via academic databases like PubMed. I’d like to know whether the framing effects are stronger or weaker than what writers are already exposed to via Google’s search algorithm or PR pitches. AI’s influence on media and publishing is likely to be profound and long-lasting, and that is a world I hold dear. But I’m concerned that the efforts spent poring over text for detector-fueled pile-ons—all that time, energy, and fury to assemble a circular firing squad—is aimed at the form of AI influence that, in the long run, may be least important.
I have to be honest: for a "Modern Love" column, I'm not sure the AI version is noticeably worse than the median human-written entry. The column has been running on the same emotional template for years: compressed redemption arc, therapy-speak, tidy epiphany. Gilgan's alleged crime was producing a version of a product that was already, in some sense, oppressively formulaic. In my admittedly snob-adjacent opinion, complaining that AI made a "Modern Love" column worse is like complaining that a Happy Meal is unhealthy because the fries are fried in seed oil.
Two decades of internet publishing and social media incentives have already profoundly reshaped how journalists write and probably how they think. A recent study analyzed roughly 40 million headlines over twenty years and found shifts toward sensationalism and negativity at every quality level, across every major outlet. Will AI increase this trend, decrease it, or what?
I could make a McLuhan type argument about how even non-users of AI will be affected by AI, but this essay is getting long enough.
A quick note about AI detectors: They aren’t snake oil, and quite good at the level of large corpuses of text. But they aren’t the slam dunk posters think they are because they are far less reliable for article length chunks of text. And it’s worth noting that AI detection companies have skin in the game. Take Pangram, the detection company whose technology flagged both Gilgan and Shy Girl and whose data powered the Atlantic piece. They worked on research that found AI-generated text in roughly nine percent all newspaper articles, including about 4.5 percent of all New York Times opinion pieces. The company’s CEO is now on LinkedIn urging publishers to “strictly moderate AI-generated content” and “draft and enforce robust AI-use policy.” How convenient that the company that helps generate the evidence, and fuels the outrage, is also selling the moderation tools. Make of that what you will.
You write, " Scientists aren’t facing the economic threat from generative AI in the way that writers are."
Perhaps scientists don't face a threat from generative AI in particular. That seems reasonable. But they would definitely seem to face a threat from AI generally.
I've spent recent weeks watching a great many interviews with the Nobel Prize winning experts who invented AI, and others of similar status. These experts seem pretty united in predicting that machines smarter than humans are coming, and where they differ it seems to be mostly about the schedule for that outcome. Again and again they claim that the primary threat AI presents at this time is to knowledge workers.
And, if it becomes true that AI brings sweeping changes to the economy by automating millions of jobs, there is likely to be a considerable push back against the science community in general, who will be perceived to be the ultimate source of the painful disruption.