AI has rapidly transformed how people access information and make decisions. Tools like ChatGPT offer speed, convenience and support for everyday tasks, however growing evidence suggested overreliance on AI may influence how we think, reason and evaluate information. 

The research from the University of Pennsylvania’s Wharton School of Business has reviewed 1,300 subjects’ use of AI. The study found that in 80% of cases, users utilised ChatGPT responses without applying their own analysis and reasoning, leading to using inaccurate information produced by the system.  

Steven Shaw, a postdoctoral researcher shared how outsourcing our brain power is having a significant impact. He commented: “We call it adoption without verification”.

Generative AI systems are ensuring “people can surrender their thoughts to AI and let it think for them,” Shaw continued. 

“They’re basically subverting the whole internal brain set of processes.”

Cognitive Surrender

Researchers from the University of Pennsylvania have coined the term for this tendency as ‘cognitive surrender’, as AI users defer thinking to external systems.

Not only can this cause a tendency to utilise inaccurate information, but it may also result in the slow deterioration of human intuition, reflection and analysis. One of the researchers says that this can further affect the human condition through the deterioration of confidence over time. 

With people increasingly turning to AI for prompt responses, we are being exposed to a cognitive cost-benefit. Studies have found overreliance can lead to ‘cognitive atrophy’, reduced cognitive thinking abilities and a tendency to second-guess your own knowledge when contradicted by AI. 

The effects may be so impactful that researchers have begun updating decision-making models. Widely accepted decision making often refers to two systems at play in the brain –  System 1 involves quick and intuitive thinking, and System 2 refers to slow and conscious reflection. Both are human systems.

This evolving field of AI research suggests a new system may need to be added – System 3. System 3 refers to artificial cognition, which involves the consideration of an additional external thinking system such as Machine Learning, to inform decision-making.

They have suggested this evolution will “reframe[s] human reasoning and may reshape autonomy and accountability in the age of AI.”

Recommended reading

• • Report: Appetite for AI Investment Higher Than Manageable for CIOs
  • Scotland Publishes 5-year AI Strategy to Compete Globally
  • Scotland Leads UK in AI Productivity Gains

The Confidence-Competence Gap

AI intelligently positions its responses to sound credible and confident, allowing users to be swept up in its accessibility and intellect

The University of Pennsylvania researchers conducted three studies where they presented participants with logic and reasoning questions and the option to use ChatGPT. More than half  of the time, participants depended on AI to answer questions, with varying degrees of accuracy in their responses. 

As well as assessing the accuracy of responses, the study assessed confidence levels of respondents. Those who answered using ChatGPT were 10% more confident. 

One researcher and Wharton professor Gideon Nave, commented: “when AI is wrong, we see that people end up performing worse than if they had no AI at all, and become more confident in their wrong answers.”

While technology has long-supported information gathering, the emergence of AI-driven may result in a new level of cognitive dependence. If used without critical thinking, this reliance could reshape not only how decisions are made but also how confidence and reasoning develop over time. 

AI must be used to enhance human intelligence, rather than as a replacement for it.