British Prime Minister Benjamin Disraeli apparently once said: “There are three kinds of lies: lies, damned lies, and statistics.” And I don’t disagree with ol’ Ben.

Numbers, as trustworthy as they seem, can be twisted, cherry-picked, and misrepresented until they say whatever you want them to say. And if you think you’re too savvy to be hoodwinked, there’s a bridge I’d like to sell you.

I’m not saying most people deliberately mislead with statistics. I’m saying research is, at best, flawed—and at worst, manipulated, whether by design or incompetence, to serve an agenda. It’s flawed because the act of asking questions alters responses (the Hawthorne Effect), sample sizes are often too small, or questions are framed to skew results. It’s manipulated when data is presented as fact, but shaped to tell a convenient fiction.

The COVID-19 pandemic was a goldmine for bad predictions. Early in the crisis, research firms confidently declared that remote work would permanently replace office culture, that people would stop dining out, and that retail was dead. Fast forward a few years, and many office buildings are full, restaurants have waitlists, and retail is thriving. It turns out that asking people what they think they’ll do in the future is wildly unreliable.

Then there’s the streaming industry, which completely misread subscriber loyalty. Netflix, Disney+, and HBO Max all assumed their research showed that customers would tolerate price hikes. Instead, millions of users canceled their subscriptions. People say they’ll stay, but when their credit card gets charged, reality kicks in.

Political polling is another prime example of research gone wrong. The last U.S. presidential election was yet another reminder that pollsters can get it horribly wrong—again. Pre-election numbers confidently predicted one outcome, only for election night to tell a different story. The same thing happened in the UK, where polls repeatedly underestimated Conservative Party support. Why? The same reasons marketing research fails: bad sampling, self-selection bias, misleading questions, and overconfidence in small differences. I only bring this up because if the top US pollsters—armed with decades of experience and millions in funding—can still botch an election forecast, do you really think your brand’s toothpaste survey is airtight?

So, given my skepticism, why am I writing an article about how marketers should use statistics correctly? Because too many rely on research without really understanding how stats work—and if you’re going to use them, you should at least use them right.

What Research Can (and Can’t) Do for Marketers

Research can be useful for spotting broad trends over time. If your customer satisfaction score has been dropping steadily for two years, that’s a real trend worth investigating. But if it dipped by 3% last month, don’t panic—it’s probably just noise. It’s also great for tracking shifts in behavior over time. If your brand was beloved in 2022 but is now struggling, research can help pinpoint when and why perceptions started to change.

It’s also a somewhat useful tool for generating hypotheses—showing you what’s worth investigating, rather than dictating your next move. If data suggests Gen Z is spending more on skincare, it’s worth exploring, but it doesn’t mean you should launch a moisturizer tomorrow.

That brings us to an important distinction: research is best for testing messaging, not ideas. Many of the best marketing ideas would have been murdered in a focus group. However, if research shows that people respond more positively to the word “premium” than “luxury,” that’s a useful insight you can apply without gutting a creative concept.

The real problem starts when marketers misuse research, expecting it to serve as a crystal ball. People are notoriously unreliable when predicting their future behavior. Ask them if they plan to go to the gym three times a week next year, and most will say yes. Check back in February, and their gym card is collecting dust.

Another major misuse of research is using it to justify a bad decision after the fact. If you’re cherry-picking data to validate a choice you already made, you’re not doing research—you’re doing PR. The worst offenders, however, are those who let research replace instinct and creativity altogether. No survey would have led to Tesla ditching traditional advertising altogether, and no focus group would have approved Ryanair’s counterintuitive, unfiltered (and very popular) strategy of openly mocking its own customers on social media. Some of the greatest marketing decisions have gone against the data.

How Research Gets Misused: Common Mistakes and How to Spot Them

If you’re going to use research, you’d better know how to conduct and analyze it properly. Here are a few things to know to make your research more useful and some of the most common (and most dangerous) pitfalls to watch out for.

1. The More Agreement, the More Reliability
If you conduct a survey and nearly everyone agrees on something, congratulations—you might actually have a reliable result. The Law of Large Numbers tells us that as a sample size grows, the average of the collected data gets closer to the true average of the population. In plain English, if you survey enough people and they overwhelmingly agree, you’re probably onto something. But if the responses are all over the place, your data is about as reliable as a drunk guy guessing lottery numbers. That’s why a survey where 90% of respondents agree on something is far more useful than one where opinions are scattered.

2. The Magic Number for Any Universe: 1,067
Bigger sample sizes don’t always mean better accuracy. Shouldn’t you survey millions of people for the best results? Not really. The magic number you need is 1,067. Why this oddly specific number? Because 1,067 respondents give you a ±3% margin of error at a 95% confidence level, no matter how big the total population is. Whether you’re surveying an entire country or everyone who’s ever rage-tweeted about airline delays, you don’t need millions of responses—just the right 1,067. For reference: 500 respondents = ±4.5% margin of error (less precise). 10,000 respondents = ±1% margin of error (more precise but with diminishing returns). This is why 1,067 is the sweet spot—big enough for accuracy, small enough to avoid lighting your research budget on fire. Of course, the sample has to be representative in terms of gender, age, etc.

By now your eyes may be glazing over. But if you’re interested, here is the formula for margin of error.

3. The Great Data Slice-and-Dice Disaster
Breaking data into smaller subgroups is like slicing a cake—you start with a full, delicious picture, but the more pieces you cut, the smaller and less satisfying each bite becomes

A UAE-wide survey of 1,067 people gives you a reliable ±3% margin of error. But after you conduct that survey, if you break it down by each emirate, suddenly your sample sizes shrink along with its reliability. If Dubai accounts for 40% of respondents, that’s only 427 people, pushing the margin of error closer to ±5%. Adu Dhabi, with 15% of respondents, now has just 160 people, sending the margin of error past ±8%. Keep slicing further—say, by age, gender, or left-handedness (because why not?)—and suddenly, your data is less science, more astrology. The smaller and more obscure the sample, the shakier the conclusion.

4. The Oven-and-Ice Problem: Why Percentages Can Lie
One of the worst offenses in marketing research is playing fast and loose with percentages

There’s an old joke about a statistician who has his head in an oven and his feet in ice, yet insists that, on average, he feels fine. This is exactly how marketers present data—averaging wildly different numbers and pretending they tell a coherent story. If a survey claims “30% of users clicked our ad,” the obvious first question should always be: 30% of what? Five people out of 15? 3,000 out of 10,000? A percentage without raw numbers is meaningless. A company might say its conversion rate jumped 50%, but if that just means it went from two customers to three, that’s not exactly cause for celebration.

5. Margin of Error: Why Tiny Differences Don’t Matter
Marketers love obsessing over small shifts in numbers, but if the change is within the margin of error, it’s not a real change—it’s noise. If a survey finds that “37% of customers prefer Brand X (±3%)”, that means the real number could be anywhere from 34% to 40%. If you’re making decisions based on a 1% or 2% difference when your margin of error is ±3%, congratulations—you’re reading statistical tea leaves. If the margin of error is bigger than the difference, your conclusion is built on statistical quicksand.

6. Sampling Error: Why Small Changes Can Be Misleading
Even if you follow all the rules, research isn’t perfect. If you run a survey today and repeat it with a different (but equally structured) group tomorrow, you’ll still get slightly different results. That’s called sampling error, and it’s why minor shifts in survey data shouldn’t send you into a panic. Marketers often freak out over tiny dips in survey numbers. If customer satisfaction was at 82% last quarter and dropped to 80% this quarter, is that a real issue? Not necessarily. It could just be noise in the data. Instead of obsessing over minor fluctuations, watch for trends over time. A slow, steady decline? That’s a red flag. A one-time 2% drop? Probably not worth losing sleep over.

7. The Danger of Self-Selection Bias
If you let people choose whether to participate in a survey, congratulations—you’ve just created biased data. People who willingly take surveys are not a random cross-section of the population. They’re typically: 1. Extremely happy 2. Extremely angry 3. Extremely bored. This is why online reviews are terrible for accurate research. The neutral majority? They don’t bother. The same thing happens with political polling. Certain voter groups are less likely to respond, skewing results. This is why pollsters have been embarrassingly wrong in multiple elections. I mean, ask yourself this: who answers a call from an unknown number these days? 

8. The Perils of Leading Questions and Confirmation Bias
“Wouldn’t you agree that our product is revolutionary?” That’s not research. That’s manipulation. Leading questions push respondents toward a particular answer, whether they realize it or not. It’s one of the easiest ways to ruin a survey, and marketers do it all the time. Then there’s confirmation bias, where marketers interpret data in ways that support what they already believe. If you want to prove that customers love your product, you’ll find data that supports it—ignoring the parts that don’t. Good research asks neutral questions and accepts results—even when they aren’t what you want.

9. Statistical Significance vs. Practical Significance
Just because a result is statistically significant doesn’t mean it’s useful. Marketers often get excited about a tiny percentage difference and assume it means something important. Imagine a brand tests two ad headlines, and Headline A gets 51% engagement while Headline B gets 49%. Statistically, that might be significant. But does a 2% difference justify an entire campaign overhaul? Probably not. Statistical significance simply means the result isn’t random—not that it’s meaningful. The better question is: Does this difference actually matter for business? If not, move on.

10. Correlation vs. Causation: The Oldest Statistical Sin
Just because two numbers move together doesn’t mean one causes the other. Ice cream sales and drowning deaths both rise in summer, but no one’s blaming the Rocky Road. Marketers fall for this all the time. A study may find that people who drink coffee are more successful, but does that mean coffee creates success? Or are ambitious, hardworking people just more likely to need caffeine? Any claim that “X causes Y” should immediately raise red flags. If the data doesn’t prove direct causation, assume someone’s trying to make a weak argument sound impressive.

How to Not Get Tricked by Bad Stats

Use research to spot trends, not predict the future. Never base big decisions on tiny differences. Always check sample size and margin of error. Be skeptical of overly specific breakdowns. Correlation is not causation. If the numbers look weird, they probably are. And just remember: 87.5% of all statistics are made up. Including that one.

Now that AI has permeated every aspect of business and the pace of its development suggests it’s coming for all our jobs, it’s easy to imagine a future where marketing research is entirely automated. With its vast computing power, AI could soon dominate this field, processing data faster and more accurately than human teams ever could.

I should probably preface this article by admitting that I’ve always harbored a distinct mistrust for much of the research brands conduct around the marketing space, particularly regarding its impact on creative output. In my experience, research is often a sanity check—a way to justify or rule out ideas. It may suggest what not to do but rarely provides actionable insights on what to pursue. If research were that effective, we’d live in a world of only hit songs, blockbuster movies, best-selling novels, and advertising campaigns that never miss.

I’m also drawn to the Observer Effect, which suggests that the simple act of observation may influence the subject being observed. Simply put, “The act of asking the question changes the answer.” This inherent limitation underscores the challenges of relying solely on research, even as AI promises to eliminate human biases (though maybe insert a few of its own).

AI has already disrupted industries by automating tasks previously thought to require human intuition and creativity. In marketing research, it promises to transform how we collect, analyze, and apply data. From consumer sentiment analysis to campaign optimization, the technology seems unstoppable. But can it truly replace human oversight?

The Current Necessity of Human Expertise in Marketing Research

While AI excels at data processing, humans remain crucial for interpreting nuanced insights, especially in qualitative research. For instance, Procter & Gamble relies on human analysts to decode cultural nuances in global campaigns, ensuring messages resonate across diverse markets.

Even in numbers-driven industries, human involvement is critical. According to a 2023 Forrester report, 68% of companies that use AI for market research still rely on human teams to validate insights and translate them into actionable strategies.

Moreover, AI algorithms often inherit biases from their training data. Human oversight is essential to identify and correct these biases. For example, Facebook’s ad-targeting AI was found to skew results based on racial and gender biases in a 2023 audit, prompting the platform to double its hum an oversight team.

Creativity thrives on intuition and risk, areas where research alone falls short. Nike’s campaigns balance AI-driven insights with human-led storytelling, crafting narratives that connect emotionally with audiences worldwide. Their 2023 “You Can’t Stop Us” campaign integrated AI data to identify trending themes but relied heavily on human creativity to deliver its powerful, unifying message.

How AI is Revolutionizing Marketing Research
Platforms like Oomiji integrate open-ended questions and provide real-time analysis, challenging traditional focus groups by offering deeper, more immediate insights. In a recent case study, Oomiji helped a mid-sized retailer boost customer satisfaction by 35% by identifying previously overlooked concerns.

Similarly, Qualtrics reported that companies using their AI-driven tools cut research timelines by 40% while increasing actionable insights by 25% in 2023. SurveyMonkey has rolled out advanced AI analytics, allowing businesses to generate reports in under an hour, a process that once took weeks.

The future of focus groups may lie in virtual environments led by AI, offering cost savings and scalability. In 2023, Coca-Cola piloted VR-based focus groups, which provided deeper behavioral insights, reducing campaign misalignment by 30%.

AI tools offer instant tabulation of survey data, delivering polished reports in minutes. However, despite their efficiency, these reports often miss contextual nuances. A 2023 study by McKinsey revealed that 48% of companies using automated tools faced misinterpretations in at least one major campaign, underscoring the need for human oversight.

The Potential for Full Automation in the Future
Advanced AI can predict consumer trends with remarkable accuracy, potentially eliminating the need for human hypothesis testing. Case in point: Amazon’s predictive analytics not only improved product recommendations but also increased conversion rates by 22% in 2023.

AI platforms are also starting to develop creative content based on data-driven insights, such as ad copy and design variations. Levi Strauss experimented with AI-generated designs last year, saving 15% on production costs while maintaining high consumer approval rates.

However, the leap to full automation isn’t without hurdles. Gartner projects that only 12% of organizations will achieve fully automated marketing research workflows by 2025, primarily due to trust and ethical concerns.

The Limits of AI Today: Trust, Accuracy, and Ethics
Despite its capabilities, AI struggles with transparency. How it arrives at conclusions is often a black box, raising trust issues. According to a recent PwC survey, 72% of business leaders expressed concern over the opacity of AI-driven insights.

Errors in automated sentiment analysis can mislead campaign strategies. For example, Wendy’s faced significant backlash in 2023 after AI misinterpreted customer sentiment, leading to a poorly received promotion that cost the company $10 million. The misstep highlighted the dangers of relying too heavily on AI without human checks.

Embracing a Hybrid Future
While AI will continue to enhance marketing research, human expertise will remain critical for ensuring accuracy, creativity, and ethical compliance. Businesses must embrace AI while maintaining a strong human element, ensuring the best of both worlds in their marketing research endeavors.

Sources:
• Statista, “AI Use in Marketing,” 2024.
• Forrester, “AI in Market Research,” 2023.
• PwC, “Trust in AI Systems,” 2024.
• McKinsey, “State of AI,” 2023.