AI Perspectives

Using AI to generate market research reports.

Any qual market researcher will tell you that creating insightful, actionable, and attractive reports is a labor of love. For this AI Perspectives piece, CEO Bonnie Dibling, Tech Moderator Chris Dethloff, and COO/Director of Technology Research John Dibling weigh in on whether AI tools like ChatGPT can help make report generation less time consuming, or even take over the task of writing the report from start to finish.

Have you tried using ChatGPT or another AI tool to help you write report? We’d love to hear about your experience with it.

Disclaimer: Protecting the privacy and security of people, data, and organizations remains the top priority when using AI tools like ChatGPT. Please adhere to any policies around the use of AI, privacy, and compliance established by your clients, your own organization, or relevant governing bodies.
The Question
Can AI be used effectively to write qual market research reports?

While AI might one day be able to generate usable reports, today’s LLMs like ChatGPT offer limited potential for summarizing content but not for generating the reports themselves.

AI Newbie Perspective
“AI could conceivably be useful in qual research as a tool to synthesize and categorize interviews, which would make analysts’ lives so much easier.”Bonnie Dibling, CEO

If AI is going to help qualitative research in any way, it’s going to be on the analysis side. AI could conceivably be used to synthesize and categorize interviews, which would make our analysts’ lives so much easier.

I see it working like this. You upload 50 interviews you’ve conducted, and then give ChatGPT or whatever LLM you’re using prompts that instruct it to find and pull out supporting quotes around a specific topic. This would be an efficient way to provide rationale for asserting a particular finding. And that, I think, is a wonderful use of AI in qualitative research.

Using AI this way to free up analysts’ time means that we as researchers can do more (more interviews, more research, more projects) with less (fewer human resources and less time).

AI Curious Perspective
“I think yes. From my point of view, reports are probably the easiest entrance for using AI in qual research, because it excels in summarizing content and data.” Chris Dethloff, Tech Moderator

AI can have a role in helping create reports, but I do have some caveats.

To me, one of the interesting aspects of market research is in understanding the degree of a “no” given by a respondent. Sometimes a “no” is a hard, resolute, absolute “that will never happen.” But there’s also a spectrum of “no,” when the “no” could be a “probably not,” or a “no, under these circumstances,” or a “no except for this,” or even “no, but when I think about it, maybe yes.” AI may be good at synthesizing data, but it’s not good at understanding these shadings.

Could I use an LLM like ChatGPT to write a report? Probably, but purely for summarization. Can I use it to understand all the nuance of human language, and give me the degree of a “no?” From my experience, not currently — and not unless the AI model has had hundreds of thousands of dollars thrown at it to train it. Which makes it cost-prohibitive.

Keep in mind that AI performs the tasks it’s been trained on. So, if the question is, can AI write a report, the answer is yes, if it has been trained for that. Can ChatGPT write a report? Not by itself.

In terms of synthesizing and filtering language and giving you output, yes ChatGPT can do those tasks. And this can offload a significant chunk of your workload in terms of the time, effort, and energy that it takes to transcribe interviews and focus groups. But if I as a human still need to read through 20 hours of transcripts captured by ChatGPT, the question becomes: is it really saving me time and effort?

AI OG Perspective
“One day, yes. Today, no. ChatGPT, certainly not.” John Dibling, Technology Lead

As of today, there are a couple of big challenges not just for ChatGPT but for LLMs in general that keep it from being an effective tool for generating reports. One of these challenges is the size of its context window, meaning how much data it can retain in its short-term memory and pull from. Right now, that limit is upwards of 10,000 words, although the more recent GPT-4 is upping that limit. Anything beyond that, and the request becomes too complex for the AI to handle.

That’s a problem, because in order to write a report you need to have all 30 or 50 interviews at your fingertips. You need to be able to aggregate all those conversations and all that data for an entire research project. Because of the context window limitations, that’s an impossibility for ChatGPT today. And I don’t see that ever changing.

I’m not even touching on privacy concerns with an LLM like ChatGPT. You can’t trust it with sensitive or confidential information that gets revealed over the course of interviews or focus groups. A custom AI model that runs on premise could potentially address that issue. If it’s on premise, you have control over the data and can make sure it never gets shared with anyone (or with OpenAI), so privacy is assured.

But there’s still the problem of the context window size. You could try to get around this limit by segmenting the interviews, and tackling small sets of interviews at a time. But that likely won’t work either, because interviews usually end up going all over the place — making it difficult to group them into topics or categories.

Bottom line: I can see potentially see a future for using AI in writing reports, but today it’s just a hard no for me.