The study, titled Determining the Validity of Large Language Models for Automated Perceptual Analysis, highlights the growing capabilities of LLMs in generating data comparable to that obtained from traditional human surveys.

Led by Peiyao Li and Zsolt Katona from University of California Berkeley, along with Noah Castelo from the University of Alberta and Miklos Sarvary from Columbia University, the research delves into the realm of automated market research utilising LLMs.

“LLMs can be used to generate text when given a prompt on certain generative Artificial Intelligence (AI) platforms,” said Li. “Our research focused on perceptual analysis and the use of automated market research for certain product categories.”

By using data already available on the internet, the researchers devised a novel methodology enabling market researchers to solely rely on LLMs for conducting market research. Their findings reveal that LLM-powered market research can yield meaningful results akin to those obtained from human participants, with agreement rates ranging from 75% to 85%.

“Although LLMs eliminate the need for direct human interviews, it’s crucial to recognize that the underlying data still originates from human experiences,” said  Katona. “LLMs have been fine-tuned to accurately replicate human responses by learning from real human perceptions, attitudes, and preferences.”

Castelo continues, “Through prompt-based interactions, LLMs can assess various brands or products within a category, producing results that closely align with traditional research involving human participants.”

Recommended reading

• Chatbot Numbers Rise as LLMs Take the Tech World by Storm
• Cybersecurity Firm Acumen Enters UK Market With Glasgow Office
• How Concerned Are Marketers That Tech Will Replace Their Jobs?

The researchers foresee significant implications for market research efficiency and cost reduction, particularly in product and brand categories conducive to LLM-based analysis. However, they caution against fully automated market research devoid of human input, citing potential inaccuracies across certain product categories.

“While our findings are promising, it’s essential to acknowledge that this is just the beginning,” says Sarvary. “As the field of market research continues to explore the capabilities of LLMs, we anticipate addressing more nuanced inquiries and further refining the methodology.”