Going Along with the Machine: Explanation, Trust, and User–AI Agreement

When an AI system recommends an answer, do people check it or go along with it? We built an explainable video search system that, instead of simply ranking motion-capture clips for a text query, generates its own example clips of the activity it thinks you mean. This lets people see how the model understood the query before accepting its answer. In a study with 44 participants, we measured accuracy and also user–machine synchronization: how often a person’s final answer matched the AI’s, whether or not either was right.

Explanations did not make people better judges of the AI. Participants who trusted the system agreed with it on about 60% of tasks, with or without explanations. We also measured skepticism: choosing a correct answer when the AI’s suggestion was wrong. Skepticism appeared in only about 13–15% of tasks, and showing explanations did not change it. Trust appeared to be set by instinct rather than by evidence. People who distrusted the explanations stopped following the AI and their accuracy fell, while those who trusted them did about as well as with the black-box system, without getting better at spotting when it was wrong.

As AI tools are proposed to help people judge whether images, video, or other evidence is authentic, this study is a caution: showing people the model’s reasoning does not by itself lead to critical assessment. How closely people follow a tool, and whether they follow it at the right times, matters as much as the tool’s own accuracy.

Blog Post

Case Study: Querying a Video Dataset Using Explainable A.I.