\

We’re experiencing a conflict between the economic pressure to produce quick and cheap research results and the scientific demand for rigor. Hundreds, if not thousands, of lifelike personas can be generated within minutes by vendors promising strong results. But these often operate as methodological black boxes, producing outputs that can’t be validated, may contain hidden bias and can quietly mislead decision-making.

The synthetic data market is growing quickly, with valuations projected to surge from approximately $267 million in 2023 to over $4.6 billion by 2032. Driven by demand for instant insights in an always-on economy, 95% of insight leaders plan to use synthetic data within the next year and the appeal is clear. Speed, scale, cost efficiency and the ability to generate insights from niche audiences are key drivers.

To move synthetic testing from a purely experimental approach to a reliable, scalable practice, organizations need to address these risks directly. Several approaches can help overcome skepticism and create a more sustainable model. It’s important to identify the key problem areas and address them directly.

While cost savings and speed to insights are compelling reasons for adoption, several challenges remain. The most successful organizations understand the strengths and weaknesses of different synthetic tools and when to use them.

Common challenges with synthetic research approaches

Why general LLMs fail to live up to expectations 

Why can’t you just ask ChatGPT your research questions? A common misconception in synthetic research is that providing an LLM with a detailed backstory ensures a representative output. Recent large-scale experiments suggest the opposite.

Initial studies show that prompting an LLM such as ChatGPT, Claude or Gemini to produce more content per persona increases bias/homogeneity instead of creating a diverse set of outputs. For example, personas used to predict the results of the 2024 U.S. presidential election (with detailed backstories provided by an LLM) swept every state for the Democrats and failed to reflect the political diversity of the population.

This phenomenon highlights a problem known as bias laundering, a pervasive issue in AI that affects everything from facial recognition to synthetic research, as LLMs are trained on internet data that disproportionately reflects a Western, educated, industrialized, rich, democratic (WEIRD) worldview. Asking models to be diverse personas produces a statistical mean filtered through this bias, laundering exclusion as AI neutrality.

Furthermore, synthetic respondents can suffer from the Pollyanna Principle, or the tendency for LLMs to be overly agreeable and positive in their responses to user prompts. Most users of generative AI chat interfaces have likely encountered this: ideas are met with encouragement like ‘great idea’ or ‘good choice’ rather than objective evaluation.

For instance, in a usability test comparing synthetic with human respondents, synthetic users reported completing all online courses. Where human users might report dropping out of most online courses, synthetic users reported completion.

High dropout rates among real users confirmed that synthetic respondents were trying to say what they thought experimenters wanted to hear. This sycophancy can lead to lousy product concepts being affirmed by helpful AI agents.

Fine-tuning provides context that synthetic approaches lack

Aren’t LLMs trained on a broad enough set of information to produce realistic use cases in almost any scenario? The most effective way to align synthetic respondents with reality is fine-tuning using proprietary data. While general LLMs provide decent baseline estimates for existing products, they struggle with new issues and underrepresented segments.

In one experiment, a team queried a base GPT model about a fictitious pancake-flavored toothpaste and ran into the Pollyanna Principle head-on. Without training data, the model anticipated people would like it — in other words, it hallucinated a preference for novelty. Once researchers fine-tuned the model on past survey data about toothpaste preferences, the output correctly shifted to negative.

In another study on the desirability of a built-in projector in laptops, the base model overestimated willingness to pay by a factor of three. After fine-tuning with survey data on standard laptops, the error was corrected, aligning synthetic results with human benchmarks.

Getting the best results with synthetic

The competitive advantage in synthetic research is not the model itself — which is becoming a commodity — but the proprietary context that conditions it. For instance, Dollar Shave Club used synthetic panels grounded in category data to validate new customer segments in days rather than months, achieving results that mirrored human behavior at a fraction of the effort.

A few approaches can help you get the best results from synthetic research.

Train synthetic, test real

To address some of these challenges, the market research industry has proposed an industry-wide validation methodology known as train-synthetic, test-real (TSTR). In this approach, models are trained on synthetic data and tested for predictive validity against a held-out sample of real-world data. Early results have been positive.

In research spearheaded by Stanford University and Google DeepMind, digital agents trained on interview data replicated human survey answers with 85% accuracy and social forces with 98% correlation. 

This approach acknowledges the shortcomings of relying solely on off-the-shelf LLMs as a starting point, as well as the risks of taking synthetic results at face value without validation. By using synthetic methods early and validating with real data, teams can realize time and cost savings while building confidence in results.

Utilizing governance and transparency

Being successful with synthetic research means researchers and readers can’t embrace the synthetic persona fallacy — the belief that LLMs possess the equivalent of human psychology and persona traits.

Instead, a more rigorous validation approach is needed, supported by governance guardrails, well-documented processes and transparency into the methods used.

A persona transparency checklist can guide researchers as they engage with synthetic personas: 

• Application domain: The specific task the persona is meant to perform.
• Target population: The demographic target group the persona is meant to represent, as opposed to relying on generic descriptions.
• Data provenance: Whether existing datasets were reused or modified to construct the personas.
• Ecological validity: Whether the experimental interaction reflects real-world usage contexts.

Transparency solves two challenges. It addresses ethical concerns around disclosure and builds trust by showing how synthetic approaches work and where they fall short. As synthetic influence grows, distinguishing between real and synthetic content will become critical.

Trust but verify

A realistic approach to synthetic research means abandoning the belief that LLMs inherently reflect human psychology and instead focusing on empirical benchmarking, fine-tuning and transparency.

Synthetic research works if you respect its limits

Synthetic research shows great potential but is a promise with a catch. The promise is unprecedented speed and scale and the catch is the risk of bias and hallucination.

Acknowledging these challenges and building governance and guardrails to mitigate them will help you succeed. This also turns internal skepticism into a structured governance approach that balances efficiency with outcomes, creating a win-win.

Source link