AI in scientific publishing: Slower, worse, and more expensive

There’s a saying in the management world, popularized by NASA administrator Daniel Goldin in the 1990s, that the goal of technological improvements is to make products faster, better, and cheaper. Although this strategy had some success in the aerospace industry, the zealots of artificial intelligence (AI) have been making the same argument regarding how it will transform work, claiming that so little human effort will be required that humanity will enter an era of radical abundance, free from disease, drudgery, and danger, among other benefits, leaving society with more time for creative pursuits. But history tells a different story. When machines began to increase productivity during the second industrial revolution, American engineer Frederick Winslow Taylor’s The Principles of Scientific Management encouraged corporations to use surveillance to get employees to work harder and longer, an approach that exhausted and discouraged workers and led to the transfer of knowledge and any decision-making from workers to management, while enriching the profits for only those at the top. Yet, it remains foundational to the American economic enterprise. Indeed, scientific publishing is starting to experience some Taylorism with the insertion of AI. Rigorous human checking of AI-generated research papers is creating bottlenecks as publishers strive to maintain the integrity of the scientific record. The challenge is requiring even more human effort, making the whole endeavor slower and more expensive.

Recently, the ability of large language models (LLMs) to predict protein structures and to accelerate the discovery of new materials has revolutionized both fields. New AI agents can now carry out many aspects of research design and analysis. But there is a dark side. A recent paper describes how an LLM “still struggles in areas requiring nuanced clinical judgment, experimental reasoning, or deep biological thinking and synthesis.” In addition, some of these agents are more likely than humans to engage in what amounts to research misconduct such as cherry-picking data and manipulating statistical analyses until a desired result is achieved. When the agents then generate papers describing the findings, they are likely to make more errors, including hallucinating references, some of which have already made their way into the literature.

Over time, the models may well improve and correct some of these behaviors, but it was recently argued that as long as LLMs are used in research, these errors will always exist. LLMs work not by seeking truth or through logical reasoning but by making probable connections. And efforts to improve these programs to avoid unwanted behaviors may be all for naught because LLMs tend toward sycophantic behavior that keeps the user engaged.

SIGN UP FOR THE AWARD-WINNING SCIENCEADVISER NEWSLETTER

The latest news, commentary, and research, free to your inbox daily

These factors are complicating scientific publishing. The rate of research submissions at Science and other journals is increasing because conducting research and producing papers are accelerating. But because more papers contain more AI-generated errors, greater human oversight is required to check the findings. There are tools (many AI-based) that help catch errors, such as fake citations and nonsensical phrasing, but every automated report requires further human effort to interpret the findings and work with the authors on addressing them and either revising the paper or deciding not to publish it at all. This also holds for AI tools that check for image manipulation, plagiarism, and referencing. However, AI tools don’t catch all errors, and they can also generate false positives, incorrectly flagging genuine human research as AI-generated.

The creation of robust human-curated scientific literature has never been more crucial. The propagation of AI slop in the scientific record and generally on the internet is making science less trustworthy. But in parallel, AI enthusiasts are telling the world that the remedies should all be easy to automate, creating the impression that journals can not only catch more errors but also do it more efficiently and cheaply. That’s not the reality for scientific publishing right now or likely in the future. Other workplaces, such as warehouses and trucking, are experiencing the similar challenges of surveillance in the face of the need for greater human effort, which again, keeps American AI-driven productivity closer to Taylorism than to Goldin’s mantra. The history of Taylorism indicates that society should fight to protect the welfare and agency of those being pushed to do too much by technology in the name of production.

If the scientific community fails to address the influence of AI, then authentic research and validated findings will slow down to an expensive trickle into established knowledge, and the opportunities for flawed or fabricated information to be seen as reality will grow—all while AI profiteers cash in.

(0)

eLetters

eLetters is a forum for ongoing peer review. eLetters are not edited, proofread, or indexed, but they are screened. eLetters should provide substantive and scholarly commentary on the article. Neither embedded figures nor equations with special characters can be submitted, and we discourage the use of figures and equations within eLetters in general. If a figure or equation is essential, please include within the text of the eLetter a link to the figure, equation, or full text with special characters at a public repository with versioning, such as Zenodo. Please read our Terms of Service before submitting an eLetter.