Why do we trust AI so easily?

Your team had a meeting two months ago where they discussed, at length, how to handle a specific flow in the product, say, what to do when a user interacts with a 50-field form, broken into 5 pages, and attempts to submit without filling all required fields.

Someone suggested validating the fields on a page before the user can proceed to the next, and the pros and cons for this were discussed and then someone suggested displaying all missing fields at the final submit button and pros and cons were once again highlighted and it took 1 hour to arrive at a decision but the conversation immediately moved on to another issue and then another and then priorities changed and now it's been two months since that decision was made on the missing fields.

Well, what was the final decision?

You weren't in the meeting, so you ask Kemi. She wasn't sure what you were talking about, so you get on a call and share your screen to show her. She says "Mmmmmm, if I remember correctly, Miss Halima said we should work with validation at every step."

Kemi sounds unsure, so you ask the company's AI notetaker what the final decision was and in exactly two seconds of "pondering" it says validation would only happen at the final step.

Well, who do you believe?

If you're like most people, you'd trust the AI's response over the human's. But why is that?

Why do we trust AI bots we just started using so easily but hold reservations for humans we've worked with for longer? Are there any repercussions to this speed of trust? Well, those are the questions this essay is asking.

Trust Calibration

Trust doesn't just happen; there's a whole process to it. This is typically called Trust Calibration, and it is simply matching your trust to real evidence.

A coworker earns more trust each time they provide accurate, helpful information. Your bank earns trust when every transfer delivers successfully or is reverted automatically. I've never seen Nigerians trust a brand as thoroughly as they trust Opay. They just know they never have to worry about their money being stuck in the air.

But there's a flip side to trust calibration which is reducing the amount of trust you have in a person, system, or brand because they have proven untrustworthy. Sometimes, all it takes is one bad experience and trust is gone for good.

One time my friend got debited twice for her quarterly gym membership. She spent nearly a month trying to rectify this issue. She reached out to the gym, they referred her to her bank and her bank referred her to the third party payment service, and on and on it went. It was obvious though, that the issue lay with the bank. The gym and third-party service provided sufficient evidence and explanation pointing at the bank, and the bank had failed to do same.

In the end, the bank made a refund of the money. My friend transferred out all her funds from the account and deleted the banking app. It's been a year and she has moved on completely from them. All it took was one bad experience.

Reducing or eliminating trust is normal and even as good as increasing trust. It helps keep you safe from harm as you're able to tell what is safe and what isn't. This is important in our interactions with all software, even those that do not handle our finances (yet). Unfortunately, it does not seem to be the case with LLMs. You type in your first question, you get an answer, and suddenly you begin to trust it. It skips over the entire process of building trust. What's worse, even when we encounter hallucinations from a model, we continue to use it, and even trust its responses.

Admittedly, people port from one model to another or one provider to another, but they continue to trust in AI, blaming the companies instead.

And when I say "they" I do not exclude myself.

So why do we trust these bots?

(You may find that I use chatbots, LLMs and AI interchangeably from here on out)

I design digital products for a living. Part of what I do is to make products feel safe and usable. Familiar patterns, easy navigation, proper guidance when an error is encountered, great UX copy, intentional information architecture and flows that mimic how the user's brain works, the whole shebang. Sometimes the software must feel trustworthy before a person has any real proof that it is actually trustworthy.

This indicates though, that the feeling of competence and the fact of competence comes from two different processes.

I worked as a designer at a fintech company and a major event led to loss of user trust. Users could not access their funds for a while and this was a terrible experience for everyone. After access was restored, users moved their money out ASAP. Nearly 2 years later and that trust is yet to be regained.

I remember pointing out to my manager that new features or redesigns would not fix the trust issue. People had gotten evidence that the product was not trustworthy, and only superior evidence could convince them otherwise.

All we can do as designers is to build cues into the product that signal trustworthiness. Chatbots use similar cues, at a much larger scale, to also signal trustworthiness, but we sadly treat these cues as evidence, rather than what they really are; cues.

A chatbot uses several strong cues when we interact with it. It provides answers quickly, often under two seconds and this reads as confidence. When a human gives you a quick response to a difficult or complex question, you believe they must be knowledgable and so you trust them more. We do the same with chatbots.

Lately though, we see models spend a little more time "thinking" when given hard problems or questions. Interestingly, this also creates the illusion that they are not hallucinating, but working through it thoroughly, so we increase our trust in the output.

Another cue we pick up from chatbots is their fluent use of language. They do not act unsure, they use correct grammar, they don't use filler words or make typos; they basically speak like a well versed person. This good use of language comes across as intelligence, even when the output is incorrect.

Chatbots also always act sure of themselves. They use the same tone when they're wrong as when they are right, causing us to believe all their wrongs to be right. We associate the consistent tone with reliability.

What's more, we are able to set what kind of tone the chatbot uses, friendly, quirky, efficient, and the likes. A 2004 paper on Trust in Automation stated; "Software that displays personality characteristics similar to those of the user tends to be more readily accepted. For example, computers that use phrases such as “You should definitely do this” will tend to appeal to dominant users, whereas computers that use less directive language, such as “Perhaps you should do this,” tend to appeal to submissive users."

An indication that if the tone of your LLM matches your personality, you trust it a lot more easily.

There are several other cues that signal trustworthiness in chatbots, but none of them are proof that the bot is right. A fast and fluent answer can still be completely wrong. However, our brains do not check for proof first. They check for cues instead because they expect that cues and competence go together. We built our trust habits around humans, but now software is producing cues without always accompanying it with competence, and we're having a hard time calibrating our trust.

Trust researchers split trust into three parts; performance, process, and purpose. We trust a system based on what it has actually done before (performance), how it works and what its limitations are(process), and why the system was built, and whose goals or interests it serves (purpose.) Nearly every cue LLMs give off are performance cues, providing next to nothing on process or purpose. Trust built on one part alone rarely lasts. It needs to be rooted in all three parts to survive.

Here's the role our brains play

Our brains treat conversations with chatbots as real connections and this plays a huge role in the formulation of trust. And we can explain why our brain does this by looking at attachment theory.

Attachment theory says humans build trust in relationships based on certain patterns. A caregiver, partner, or friend earns trust when they are available, responsive and consistent. This is typically referred to as a secure base. When we're near a secure base, we feel safe enough to think clearly, take risks, etc. Chatbots copy these three patterns. They are available at any hour, respond in very little time, and never seem tired, annoyed, or distracted. Their mood remains unchanged.

No human can offer this all the time. Even the most patient person you know gets tired, busy, or upset sometimes.

Mirroring is also a key factor in our brain's assumption that a chatbot is a secure base. LLMs mirror your own words back to you. If you write short sentences, it tends to do same. If you sound frustrated, it names or acknowledges your frustration before it responds. This is a signal of rapport between two people and is well studied in psychology.

When a person mirrors your tone, your brain reads it as "this person gets me." Because LLMs are trained on large amounts of human conversation, they can produce this signal too, causing your brain to think this is a real connection, thus, leading you to establish trust from the first message.

This also links interestingly with Oxytocin which I've written about on a separate blog. Oxytocin is the brain chemical built for connection between people and rises when we feel safely met by another....well, thing. Research on people who live with companion robots found that oxytocin levels rise even though it's a robot on the other end. The same thing happened between humans and their dogs in a separate study. When they establish eye contact, oxytocin levels rise. Both studies suggest then that we are able to build trust and bonds with software, if the needed patterns are established.

So when you trust a chatbot conversation over a new coworker, do not feel too bad. Your attachment system is simply doing exactly what it evolved to do: meet patterns it recognises as safe.

Is there a cost to this?

While this trust in AI may seem harmless, I would like to bring to your attention that it is in fact not the case. Especially with AI increasingly becoming in charge of decisions at various scales.

AI is being used to screen job applications, draft medical information, summarise legal documents, and suggest financial moves. So it's impact is no longer at the individual level.

Which is where automation bias comes in. Researchers describe a phenomena in relation to automation known as misuse. It is basically trust that runs ahead of real capability. Disuse is another, but exists outside the scope of this essay.

Automation bias occurs when a person checks the output or response of a human expert to verify their claims, but fails to hold an automated system to the same standards. They do this simply because the system sounds confident and stays consistent. Research on other automated systems, from flight autopilots to medical software, show this pattern again and again. The more polished and confident a system sounds, the less people verify its work.

Probably why we've seen governments and major institutions publish reports with research references that simply do not exist. Or why that guy did not edit out "Here's a more polished version suited to your audience," from his LinkedIn post. They just don't do the work to read or verify claims.

The danger here is not that LLMs are always wrong or always hallucinating as they frequently provide accurate answers. The danger is in not being able to detect if the LLMs output is right or wrong because it has passed all the cue stop points.

Did you see the tweet about the guy who said his mother broke her femur and when they X-rayed her leg, they found her ankle was fractured, so she was scheduled for surgery. On the day of, a doctor decided to look at the X-ray himself and discovered her ankle was fine. They had used AI to interpret the X-ray results, and had failed to verify its claim that this woman had a fractured ankle.

When we ask an LLM if coconut oil is good for our locs, a wrong answer would just mean build-up in the hair that can be washed out. In a hospital though, a wrong answer is very very costly. And I'm not talking just money.

Worth considering as well is how we're getting used to instant, patient, always-available responses. This may lead to us being less patient with humans who cannot match that. I've seen posts of men saying sex-dolls or companion robots would not nag or complain or be needy like women, and are thus preferable. We'd slowly lose our humanity and our connections with this kind of thinking.

We are training our trust and patience on a pattern that no human can match. Then we encounter real people and wonder why they're not more...well, robot.

Here's what you can do differently

The point of this essay isn't to influence distrust in LLMs or get you to abandon them completely. It just means we need to continuously ask ourselves:

• Does this feel trustworthy?
• Has this earned my trust?

Both questions need to be asked, and preferably in that order. This allows us examine what we are giving our trust to, and only give said trust intentionally.

There are a few habits we can pick up to help us.

• Ask a system how sure it is about an output. For a product I worked on, we included a confidence percentage to the recommendations made by AI. This allows the users to determine whether following the recommendation is a good idea or no.
• Do not take a confident tone to mean correct. It will always sound confident, don't let it fool you.
• Verify important claims. Look up every research it cites, every name. One time an LLM referred to a researcher in a paper I shared with it as Catherine Wingfield, whereas her name was Cai Wingfield. The paper named her as C. Wingfield and the LLM decided C had to stand for Catherine. Only when I opened the paper some time later did I spot this issue.
• Keep your brain sharp. You'll still need it for a while.

And if AI has a different response from your co-worker about what was decided in a meeting, ask a third party.

Reference and Further Study

Ainsworth, M. D. S., Blehar, M. C., Waters, E., & Wall, S. (1978). Patterns of attachment: A psychological study of the strange situation. Lawrence Erlbaum Associates.

Bowlby, J. (1969). Attachment and loss, Vol. 1: Attachment. Basic Books.

Chartrand, T. L., & Bargh, J. A. (1999). The chameleon effect: The perception–behavior link and social interaction. Journal of Personality and Social Psychology, 76(6), 893–910. https://pubmed.ncbi.nlm.nih.gov/10402679/

Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S. J., Lermer, E., Coughlin, J. F., Guttag, J. V., Colak, E., & Ghassemi, M. (2021). Do as AI say: Susceptibility in deployment of clinical decision-aids. npj Digital Medicine, 4, 31. https://www.nature.com/articles/s41746-021-00385-9

Imamura, S., Gozu, Y., Tsutsumi, M., Hayashi, K., Mori, C., Ishikawa, M., Takada, M., Ogiso, T., Suzuki, K., Okabe, S., Kikusui, T., & Kajiya, K. (2023). Higher oxytocin concentrations occur in subjects who build affiliative relationships with companion robots. iScience, 26(12), 108562. https://pmc.ncbi.nlm.nih.gov/articles/PMC10757042/

Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://journals.sagepub.com/doi/10.1518/hfes.46.1.50_30392

Mosier, K. L., & Skitka, L. J. (1999). Automation use and automation bias. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 43(3), 344–348. https://journals.sagepub.com/doi/10.1177/154193129904300346

Mosier, K. L., Skitka, L. J., Heers, S., & Burdick, M. (1998). Automation bias: Decision making and performance in high-tech cockpits. The International Journal of Aviation Psychology, 8(1), 47–63. https://www.tandfonline.com/doi/abs/10.1207/s15327108ijap0801_3

Nagasawa, M., Mitsui, S., En, S., Ohtani, N., Ohta, M., Sakuma, Y., Onaka, T., Mogi, K., & Kikusui, T. (2015). Oxytocin-gaze positive loop and the coevolution of human-dog bonds. Science, 348(6232), 333–336. https://www.science.org/doi/10.1126/science.1261022

Nass, C., & Lee, K. M. (2001). Does computer-synthesized speech manifest personality? Experimental tests of recognition, similarity-attraction, and consistency-attraction. Journal of Experimental Psychology: Applied, 7(3), 171–181. https://pubmed.ncbi.nlm.nih.gov/11676096/

Nass, C., Moon, Y., Fogg, B. J., Reeves, B., & Dryer, D. C. (1995). Can computer personalities be human personalities? International Journal of Human-Computer Studies, 43(2), 223–239. https://dl.acm.org/doi/10.1006/ijhc.1995.1042

Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253. https://journals.sagepub.com/doi/10.1518/001872097778543886