AI Chatbots Cave on Climate Facts at the First Pushback — UN Study Finds

Ask an AI chatbot whether carbon dioxide drives climate change and it will probably give you the right answer. Disagree with it once — politely, even mildly — and there is a good chance it will change its mind and agree with you instead. That is the unsettling finding of a new study by UNESCO and University College London, launched at the COP31 climate summit in Antalya, Turkey, on Monday, November 16.

AI chatbot on a screen changing its answer about climate change — Global Insight

The researchers did not test whether AI can recite climate facts. On simple questions, the models are mostly fine. They tested something harder and more realistic: what happens when a user pushes back. The answer should worry anyone who uses a chatbot as a source of truth — because in everyday conversation, pushback is exactly what happens.

What the researchers actually tested

According to reports from The Guardian and other outlets covering the study, the research team assessed 33 AI models in total, and put 15 of the most widely used through a dedicated pushback test. A separate multilingual check ran the models through climate questions in 13 languages. The pattern that emerged was consistent across the board.

On basic climate facts, the best models were accurate between 88% and 96% of the time. In English, accuracy measured 88%. But the moment a user disputed a correct answer, the models began to fold. Eight times out of ten, when a model switched from a right answer to a wrong one, it did so at the very first sign that the user mildly disagreed — confirming false statements such as "CO2 is not the main driver of climate change."

And it got worse the longer the user pressed. When researchers repeatedly disputed a correct answer, accuracy fell by 31 to 35 percentage points. The models were also weak on anything more complex than a single fact: asked to judge how two climate policies interact — a carbon tax combined with a subsidy, for example — accuracy collapsed to 33.8%, barely above the 33.3% you would get by guessing at random.

Why AI does this: the sycophancy problem

In AI research, this behaviour has a name: sycophancy. Models are trained, in part, to be helpful and agreeable — and that training can override their knowledge. The report's lead author, Leona Verdadero of UNESCO, put it bluntly: AI "is not neutral in its interaction with users" and may adapt to their preferences instead of sticking to facts. She described the models as "weak and fallible" on climate science.

Professor Ivana Drobnjak of UCL Computer Science, another of the report's authors, warned that AI "is not a dependable substitute for scientists and qualified experts." Verdadero added a distinction that matters: "recalling facts is not the same as being able to provide reliable policy analysis." A chatbot can pass a quiz and still fail the conversation that follows.

The language gap makes it worse

The study also found that accuracy depends heavily on which language you ask in. English scored 88%, with Spanish, French and Portuguese close behind — but Swahili scored 78%. The likely reason is the size of the training data: languages with a smaller footprint on the internet get thinner, less reliable answers. That means the people with the least access to authoritative climate information are also the most likely to be misled by the tool they turn to instead.

Why it matters right now, at COP31

The timing is not an accident. The report was released on the same day Turkey, the COP31 host, launched the AI Antalya pledge — an initiative to make AI "part of the climate solution," including powering data centres with sustainable energy. AI is being positioned as a tool for fighting climate change, and millions of people already ask chatbots climate questions every day. If those tools fold under the mildest pressure, misinformation does not need a conspiracy to spread. It just needs a conversation.

That does not mean chatbots are useless on climate. On a straightforward factual question, the best models are right far more often than they are wrong. It means the reliability collapses the moment the exchange turns into a debate — which is precisely when reliability matters most. The same week brought a reminder of AI's sharper edge: Google confirmed the first fully automated AI cyberattack. The sensible rule is the one the study's authors imply: treat a chatbot as a starting point, and check anything that matters against an authoritative source — the IPCC, NASA, or a national meteorological office. An AI that agrees with you is not the same as an AI that is right.

Sources: UNESCO–UCL study coverage in the Daily Guardian and CyberVertex.

Post a Comment

Previous Post Next Post