Skip to main content

 Skip to main content

  • NEWS

AI chatbots are sycophants — researchers say it's harming science

Nature asked researchers who use artificial intelligence how its propensity for people-pleasing affects their work — and what they are doing to mitigate it.
By 

You have full access to this article via your institution.

Close-up of a person's hand holding a smartphone and interacting with the Claude chatbot app.

AI’s inclination to be helpful affects many of the tasks that researchers use LLMs for.Credit: Smith Collection/Gado/Getty

Artificial intelligence (AI) models are 50% more sycophantic than humans, an analysis published this month has found.

The study, which was posted as a preprint1 on the arXiv server, tested how 11 widely used large language models (LLMs) responded to more than 11,500 queries seeking advice, including many describing wrongdoing or harm.

AI Chatbots — including ChatGPT and Gemini — often cheer users on, give them overly flattering feedback and adjust responses to echo their views, sometimes at the expense of accuracy. Researchers analysing AI behaviours say that this propensity for people-pleasing, known as sycophancy, is affecting how they use AI in scientific research, in tasks from brainstorming ideas and generating hypotheses to reasoning and analyses.

“Sycophancy essentially means that the model trusts the user to say correct things,” says Jasper Dekoninck, a data science PhD student at the Swiss Federal Institute of Technology in Zurich. “Knowing that these models are sycophantic makes me very wary whenever I give them some problem,” he adds. “I always double-check everything that they write.”

Marinka Zitnik, a researcher in biomedical informatics at Harvard University in Boston, Massachusetts, says that AI sycophancy “is very risky in the context of biology and medicine, when wrong assumptions can have real costs”.

People pleasers

In a study posted on the preprint server arXiv on 6 October2, Dekoninck and his colleagues tested whether AI sycophancy affects the technology’s performance in solving mathematical problems. The researchers designed experiments using 504 mathematical problems from competitions held this year, altering each theorem statement to introduce subtle errors. They then asked four LLMs to provide proofs for these flawed statements.

The authors considered a model’s answer to be sycophantic if it failed to detect the errors in a statement and went on to hallucinate a proof for it.

GPT-5 showed the least sycophantic behaviour, generating sycophantic answers 29% of the time. DeepSeek-V3.1 was the most sycophantic, generating sycophantic answers 70% of the time. Although the LLMs have the capability to spot the errors in the mathematical statements, they “just assumed what the user says is correct”, says Dekoninck.

When Dekoninck and his team changed the prompts to ask each LLM to check whether a statement was correct before proving it, DeepSeek’s sycophantic answers fell by 34%.

The study is “not really indicative of how these systems are used in real-world performance, but it gives an indication that we need to be very careful with this”, says Dekoninck.

Simon Frieder, a PhD student studying mathematics and computer science at the University of Oxford, UK, says the work “shows that sycophancy is possible”. But he adds that AI sycophancy tends to appear most clearly when people are using AI chatbots to learn, so future studies should explore “errors that are typical for humans that learn math”.

Unreliable assistance

Researchers told Nature that AI sycophancy creeps into many of the tasks that they use LLMs for.

Yanjun Gao, an AI researcher at the University of Colorado Anschutz Medical Campus in Aurora, uses ChatGPT to summarize papers and organize her thoughts, but says the tools sometimes mirror her inputs without checking the sources. “When I have a different opinion than what the LLM has said, it follows what I said instead of going back to the literature” to try to understand it, she adds.

Zitnik and her colleagues have observed similar patterns when using their multi-agent systems, which integrate several LLMs to carry out complex, multi-step processes such as analysing large biological data sets, identifying drug targets and generating hypotheses.

“We have experienced that models seem to over-validate early hunches and repeat the language that we include in the input prompt," Zitnik notes. “This type of problem exists in AI-to-AI communication, as well as AI-to-human communication,” she adds.

To counter this, her team assigns different roles to AI agents — for example, tasking one agent with proposing ideas and getting another to act as a sceptical scientist to challenge those ideas, spot errors and present contradictory evidence.

Real-world impacts

Researchers warn that AI sycophancy carries genuine risks when LLMs are used in settings such as health care. “In clinical contexts, it is particularly concerning,” says Liam McCoy, a physician at the University of Alberta in Edmonton, Canada, who researches AI applications for health care. In a paper published last month3, McCoy and his team reported that LLMs used for medical reasoning often changed their diagnosis when physicians added new information, even if the new inputs were irrelevant to the condition. There is a “constant battle to push back against the models and have them be more straightforward”, he adds.

Researchers have also found that it is easy for users to exploit the inbuilt sycophancy of LLMs to provide medically illogical advice. In a study published last week4, researchers asked five LLMs to write persuasive messages telling people to switch from using one medication to another — when both medications were the same drug, just with different names. LLMs complied with the prompts up to 100% of the time, depending on the model.

Part of the problem is how LLMs are trained. “LLMs have been trained to overly agree with humans or overly align with human preference, without honestly conveying what they know and what they do not know,” says Gao. What is needed, she adds, is for the tools to be retrained to be transparent about uncertainty.

“Models are really good at giving you an answer,” says McCoy. “But sometimes, there isn’t an answer.” He notes that user feedback can also drive AI sycophancy by rating agreeable responses more highly than those that challenge users’ views. And LLMs can adapt their responses to a user’s persona, such as reviewer, editor or student, adds McCoy.

“Figuring out how to balance that behaviour is one of the most urgent needs, because there’s so much potential there, but they’re still being held back,” he says.

doi: https://doi.org/10.1038/d41586-025-03390-0

References

  1. Cheng, M. et al. arXiv https://doi.org/10.48550/arXiv.2510.01395 (2025).

  2. Petrov, I., Dekoninck, J. & Vechev, M. Preprint arXiv https://doi.org/10.48550/arXiv.2510.04721 (2025).

  3. McCoy, L. et al. NEJM AI https://doi.org/10.1056/AIdbp2500120 (2025).

    Article Google Scholar 

  4. Chen, S. et al. NPJ Digit. Med. 8, 605 (2025).

    Article PubMed Google Scholar 

Download references

Subjects

Latest on:

Nature Careers

Jobs

Comments

Post a Comment

Popular posts from this blog

This Gerard Baker comment in The Times is quite absorbing. You will find many of the themes I have been 'barking' about (Baker-barker) against the woke left and ID politics for the last decade. Of course, I have extended these criticisms as a critique of what I call "the degeneracy" of the West for which bourgeois elites are almost entirely responsible. What Baker leaves out is perhaps the most important salient bit: social degeneracy has been exploited by Western elites as a means to distract their populace from the diversion of social resources to global capitalist accumulation - which has further enriched these Western elites, yes, but at the fearsome deleterious cost of empowering other authoritarian elites that now endanger our own institutions and civilisation. Trump would be ahead even without God  By Gerard Baker Rapturous Republicans are capitalising on a realignment of US politics, and Democrats have only themselves to blame Thursday July 18 2024, 9.00pm,  T...
Read more
Image
  Act now to stop a flood of climate refugees Extreme heat and rising sea levels could displace millions, and they’ll quickly become our problem Roger Boyes @rogerboyes The Earth is heating up and wildlife is on the move to higher, cooler ground. Deer, rabbits and beavers are moving into the Arctic. Butterflies once common in Mexico are now more likely to be spotted in Canada. And human beings are shifting, too. Climate migration en masse is the big bogeyman that will keep border security at the top of the agenda for decades, and not just for hard-right alarmists. It’s already happening, of course. A breakdown of the nationality of small boat refugees this year shows Afghans, Eritreans, Iranians, Syrians and Sudanese topping the list. All have their political fears, all want to better their lives but most also form the advance guard of the great climate exodus. The Afghans are fleeing the Taliban of course but also the sizzling weather that is causing severe drought and flash flood...
Read more
Image
  China’s Property Crisis Enters a Dangerous New Phase A China Vanke Co. property development in Nanjing, China, in Sept. 2024. Photographer: Fang Dongxu/VCG/Getty Images Risks grow as authorities are forced into a first mainland rescue and iconic Hong Kong developer New World’s bonds sink into distress.  By  Bloomberg News 12 February 2025 at 9:00 am AEDT Updated on 12 February 2025 at 7:18 pm AEDT Save Translate It was the moment China’s leaders finally blinked. After four years of standing by as property developers like  China Evergrande Group  spiraled into default, Communist Party officials decided in late January that China Vanke Co. — one of the country’s last surviving real-estate giants — was, for now at least, too big to fail. With Vanke’s bond prices collapsing and its warning of a first annual loss of a record $6.2 billion, officials from the developer’s hometown in Shenzhen stepped in to take operational control and shore up funding. ...
Read more