Study warns of growing blind trust in AI systems and the risks of apparent self-assurance

Artificial intelligence is increasingly becoming not just a tool for processing information, but also an entity in which people apparently place more trust than in other individuals. A recent study by the University of Waterloo and University College London shows that users rate responses from AI systems such as OpenAI ChatGPT or Google Gemini as more credible even when their content is identical to human responses.

The findings raise fundamental questions about the relationship between humans and machines – particularly in safety-critical areas where trust, decision-making quality and accountability play a central role.

AI creates the impression of objective certainty

The researchers observed that many people attribute greater neutrality, precision and competence to machine-generated responses than to human statements. Even when AI and humans formulate exactly the same answer, the machine-generated version is often rated as more reliable.

This effect appears to be less related to the actual content of the information than to the way AI systems communicate. Modern dialogue systems usually formulate responses fluently, in a structured manner and with a high degree of linguistic confidence. It is precisely this linguistic self-assurance that creates the impression of professional authority in the user.

The problem is that linguistic persuasiveness is not automatically synonymous with factual accuracy.

Generative AI systems in particular are known for their ability to present erroneous information with a high degree of persuasiveness. In AI research, this phenomenon is often referred to as ‘hallucination’ – that is, the generation of content that sounds plausible but is incorrect.

Trust becomes a safety-critical variable

This development has far-reaching implications for businesses, public authorities and operators of critical infrastructure. This is because AI systems are increasingly being integrated into safety-critical decision-making processes – for example, in cybersecurity, risk analysis, situation assessment, surveillance systems or industrial automation.

The more people rely on machine assessments in this context, the greater the risk of automated misjudgements or the uncritical acceptance of false information.

This appears particularly critical when combined with time pressure and information overload. In complex operational environments, people tend to accept supposedly objective systems more readily than human judgements. AI is thus gradually becoming an authority figure.

This can be particularly problematic in the security sector. False alarms, incorrect risk analyses or manipulated AI outputs could influence operational decisions without being sufficiently scrutinised.

Anthropomorphisation amplifies the effect

The researchers see a further reason for the growing trust in the increasing humanisation of modern AI systems. Language models communicate in a dialogue-oriented, polite and context-sensitive manner. This subconsciously creates the impression of social competence or even emotional intelligence among many users.

This so-called anthropomorphisation fundamentally alters the perception of machine systems. Users no longer treat AI systems like traditional software, but increasingly as competent conversation partners.

This becomes particularly problematic when users can no longer clearly recognise the systems’ limitations. AI often gives the impression of comprehensive knowledge, even though it primarily processes statistical probabilities and possesses no actual ‘ability to understand’.

AI is changing information hierarchies

The study’s findings also highlight a broader societal shift: information hierarchies are shifting.

For a long time, human expertise was regarded as the central basis of trust. Today, a paradoxical situation is increasingly emerging: many people regard machines as more objective and rational than other people – even when the content is identical.

In the long term, this development could have significant implications for education, the media, science and decision-making processes. For the more AI systems are accepted as the primary source of knowledge, the greater their influence on society becomes.

There is also another risk: AI systems can not only be flawed, but can also be deliberately manipulated.

Data poisoning, manipulated training data or algorithmic biases could lead to erroneous information being systematically disseminated – with potentially far-reaching consequences.

Self-assurance is mistaken for competence

A key finding of the study concerns the perception of self-assurance. People often automatically interpret clear, structured and decisive answers as a sign of competence.

According to the researchers, this is precisely where one of the greatest dangers of modern AI systems lies. This is because AI often communicates with a high degree of linguistic confidence – regardless of whether the underlying information is correct.

This gives rise to a new form of digital authority that is not based on actual expertise, but on rhetorical consistency and technical presentation.

Particularly in times of a growing flood of information, this can lead to people increasingly delegating critical thinking to machines.

Human-in-the-loop remains crucial

The study therefore underscores the importance of so-called ‘human-in-the-loop’ approaches. AI systems should support decision-making, but not replace it unchecked.

Human judgement remains indispensable, particularly in safety-critical areas – not despite, but precisely because of the growing capabilities of modern AI.

For the more convincingly AI communicates, the more important it becomes to critically assess machine-generated statements and recognise their limitations.

The study by the University of Waterloo and University College London thus reveals more than just a psychological phenomenon. It highlights a central challenge facing digital society: trust is increasingly shifting from human expertise towards algorithmic self-assurance – with potentially far-reaching consequences for safety, the quality of decision-making and societal resilience.