AI Perspectives
For the second installment of our AI Perspectives series, CEO Bonnie Dibling, Tech Moderator Chris Dethloff, and COO/Director of Technology Research John Dibling discuss the possibility of using chatbots to moderate qual market research. We’d love to hear your thoughts on the topic, too.
Chatbots do not lend themselves to moderating live interviews and focus groups largely because of how they’re used and their limitations. AI and large language models (LLMs), however, do offer some potential applications for assisting — not replacing — moderators.
I’ve heard about chatbots having in role in ethnographies, where videos and images get uploaded. In those scenarios, chatbots can be used to ask the respondent simple open-ended questions, such as, “What does that image mean to you?” or “What were you experiencing in that moment? Describe that in detail for me.” Or, “What’s the most important thing about that picture for you?”
For basic questions like these, I see chatbots as being very helpful to the moderator and especially to the analyst. Because, while you still have to type in the response, those types of questions aren’t really moderating. They’re just pre-programmed probes, so the moderator’s skills and time aren’t really needed for these.
That being said, I can also anticipate situations where the respondent doesn’t really understand the chatbot’s probe or how to respond to it, and the chatbot won’t know how to clarify. Say the respondent is shown a picture of their dog, and the chatbot asks, “Why is this photo important to you?” The respondent’s answer is, “Well, because it’s my dog,” and that’s it.
A moderator would know how to steer the conversation to delve deeper into that response, whereas a chatbot wouldn’t, and simply move on to the next question. So to me, this is a very limited use of a very powerful tool.
In qualitative research, I think in general AI is a solution looking for a problem. Humans are already the most effective moderators, and I’m not sure a chatbot could improve on that. Don’t get me wrong, I’m a big proponent for using AI in other applications. I think it can truly make our lives better in some ways, but qualitative research just isn’t one of them. It would be like having a chatbot be your therapist. Not a good idea, or a good use of this technology.
There are companies out there that are using chatbots in innovative ways for open-ended surveys. For example, the chatbot asks a customer, “What did you love about coming to our store today?” And the customer writes, “It was fun.” Then there are AI programs that will follow up on that response to get more detail, asking for instance, “What made it fun?” or “You said xyz, can you tell me more about that?”
In terms of using chatbots to moderate qual research, I’m less certain about that. Will it ever get to a point where I am a hologram AI person conducting focus groups or one-on-one interviews? Probably not — at least, not as AI currently stands.
I will say that I think chatbots can be used be used very successfully as an aid. The narrower the sub-topic it’s given, the better it can perform. If it has to moderate the entire interview on its own, that’s when it might struggle. A chatbot can very quickly and easily get thrown off by the human feedback it receives, especially if the human interacting with it is deliberately trying to wreak havoc.
Chatbots running off large language models (LLMs) as we currently understand them can be prompt engineered and manipulated. When given clear boundaries and guidelines, a chatbot can do a decent job. For example, we can tell the chatbot that if the response they get from a human respondent is under 400 words, the bot will follow up to get more detail.
The issue comes when the human respondent has back-and-forth interaction with the chatbot — input, output, input, output, like a conversation. This leaves the chatbot vulnerable to manipulation and getting thrown off track.
I think the answer is just no, because fundamentally the interface is not conducive to running a qualitative interview. The interface for a chatbot requires you to type stuff in. No one’s going to be able to do that while conducting a live interview. It’s challenging enough to focus on the interview, watch the backroom, and keep track of all the input from the stakeholders.
So no, I don’t think that there’s any place for chatbots in a qualitative market research interview. AI or LLMs in general could potentially have a role in live interviews and focus groups. AI can be used for streaming transcripts, for example. We do that today, so that’s already a solved problem.
Looking toward the future, you could conceive of an AI model being fed the discussion guide, then “listening” to and transcribing the live interview or focus group discussion in real time. The moderator could perhaps use this as tool to check and make sure they’re hitting all the important topics and questions from the discussion guide. AI could help the moderator keep the conversation on track, stay aware of how much time is left and what ground still needs to be covered. But that capability doesn’t exist yet, and it’s going to take some work to get it there.
So, again, I don’t believe chatbots have a place in moderating qual market research. But I do think AI has the potential to be trained and used as a moderator’s assistant. Maybe one day it could even do the entire interview, but it will never be as good as a human who can understand nuance.