AI Perspectives

If AI is going to help qualitative research in any way, it’s going to be on the analysis side. AI could conceivably be used to synthesize and categorize interviews, which would make our analysts’ lives so much easier.

I see it working like this. You upload 50 interviews you’ve conducted, and then give ChatGPT or whatever LLM you’re using prompts that instruct it to find and pull out supporting quotes around a specific topic. This would be an efficient way to provide rationale for asserting a particular finding. And that, I think, is a wonderful use of AI in qualitative research.

Using AI this way to free up analysts’ time means that we as researchers can do more (more interviews, more research, more projects) with less (fewer human resources and less time).

AI can have a role in helping create reports, but I do have some caveats.

To me, one of the interesting aspects of market research is in understanding the degree of a “no” given by a respondent. Sometimes a “no” is a hard, resolute, absolute “that will never happen.” But there’s also a spectrum of “no,” when the “no” could be a “probably not,” or a “no, under these circumstances,” or a “no except for this,” or even “no, but when I think about it, maybe yes.” AI may be good at synthesizing data, but it’s not good at understanding these shadings.

Could I use an LLM like ChatGPT to write a report? Probably, but purely for summarization. Can I use it to understand all the nuance of human language, and give me the degree of a “no?” From my experience, not currently — and not unless the AI model has had hundreds of thousands of dollars thrown at it to train it. Which makes it cost-prohibitive.

Keep in mind that AI performs the tasks it’s been trained on. So, if the question is, can AI write a report, the answer is yes, if it has been trained for that. Can ChatGPT write a report? Not by itself.

In terms of synthesizing and filtering language and giving you output, yes ChatGPT can do those tasks. And this can offload a significant chunk of your workload in terms of the time, effort, and energy that it takes to transcribe interviews and focus groups. But if I as a human still need to read through 20 hours of transcripts captured by ChatGPT, the question becomes: is it really saving me time and effort?

As of today, there are a couple of big challenges not just for ChatGPT but for LLMs in general that keep it from being an effective tool for generating reports. One of these challenges is the size of its context window, meaning how much data it can retain in its short-term memory and pull from. Right now, that limit is upwards of 10,000 words, although the more recent GPT-4 is upping that limit. Anything beyond that, and the request becomes too complex for the AI to handle.

That’s a problem, because in order to write a report you need to have all 30 or 50 interviews at your fingertips. You need to be able to aggregate all those conversations and all that data for an entire research project. Because of the context window limitations, that’s an impossibility for ChatGPT today. And I don’t see that ever changing.

I’m not even touching on privacy concerns with an LLM like ChatGPT. You can’t trust it with sensitive or confidential information that gets revealed over the course of interviews or focus groups. A custom AI model that runs on premise could potentially address that issue. If it’s on premise, you have control over the data and can make sure it never gets shared with anyone (or with OpenAI), so privacy is assured.

But there’s still the problem of the context window size. You could try to get around this limit by segmenting the interviews, and tackling small sets of interviews at a time. But that likely won’t work either, because interviews usually end up going all over the place — making it difficult to group them into topics or categories.

Bottom line: I can see potentially see a future for using AI in writing reports, but today it’s just a hard no for me.