What happens when you ask ChatGPT for statistics? Can you trust the data at face value? According to my experiment ...
Recently, to strengthen a blog post I was writing, I searched for stats related to case studies.
While I normally would turn to Google, I decided to try out ChatGPT.
In response to my prompt ("Share stats and cite sources about the power of case studies"), I received seven stats and sources. A few of them appeared to be VERY strong, and from reputable sources. I got excited!
And then ... I clicked on the URLs to each citation.
The results were rather disturbing — and they serve as a cautionary tale about using Artificial Intelligence for research and information gathering.
My ChatGPT experiment results: the TL;DR version
The longer version of my ChatGPT experiment is below this table. As for the short story?
ChatGPT provided seven stats + seven links
Not one of the links went to the recommended stat
I couldn't find any of the stats anywhere on the internet
ChatGPT "source" | Where did the link go? | Is the ChatGPT stat available elsewhere on the internet? |
|---|---|---|
Harvard Business Review | Error page | No |
Demand Gen Report (twice) | Wrong page, both times | No |
Content Marketing Institute | Error page | No |
Search Engine Land | Error page | No |
HubSpot | Error page | No |
Journal of Education for Business | Error page | No |
Stat #1: Harvard Business Review
The stat, verbatim from ChatGPT
"According to a study by the Harvard Business Review, 79% of B2B buyers said that they have consumed at least one case study in the past 12 months. (Source: Harvard Business Review)"
The link provided by ChatGPT
The result
404 Error
What I did next
I copied and pasted the entire stat, verbatim, into Google. No results. I then plugged in a variety of search terms related to 79%, B2B, case study / case studies, 12 months / one year, and Harvard Business Review.
What I discovered
I could not find the stat ANYWHERE. I gave up after 10 minutes.
Can I use the ChatGPT stat?
Absolutely not.
Stat #2: Demand Gen Report
The stat, verbatim from ChatGPT
"Case studies are the second most popular type of content among B2B marketers, after blog posts. (Source: Demand Gen Report)"
The link provided by ChatGPT
The result
The URL took me to this page, which has nothing to do with the stat:
What I did next
As I did with Harvard Business Review, I copied and pasted the entire stat, verbatim, into Google. The top result took me to this page, which cited the stat. However, that stat was from 2013, not 2018 as the name of ChatGPT's original URL suggested.
I then added "2018" to my Google search, and this page came up as the first result. To check if the ChatGPT case study stat was in THAT report, I downloaded it.
What I discovered
Case studies are certainly mentioned in the DGR's Content Preferences report. However, the wording of the ChatGPT stat is somewhat misleading. This report doesn't ever explicitly say that case studies are the second-most popular type of content for B2B. And even when you compare some of the case study stats in the report to blog stats, there is at least one instance where case studies score HIGHER than blogs.
Can I use the ChatGPT stat?
I don't think so. The Content Preferences 2018 report does have some interesting nuggets in it about B2B content marketing preferences. But there's nothing in there to confidently support ChatGPT's original stat.
Not to mention, through a quick Google search, I found a 2022 Content Preferences report, which includes more up-to-date stats.
Stat #3: Content Marketing Institute
The stat, verbatim from ChatGPT
"A survey of 63 marketing and sales professionals found that 72% believe that case studies are effective in generating leads and sales. (Source: Content Marketing Institute)"
The link provided by ChatGPT
The result
Error page:
What I did next
I first copied and pasted the ChatGPT stat, verbatim, into Google. The top result took me to this CMI report.
What I discovered
Hmmm ... there are a few mentions of case studies in the report. But there is literally no reference to 73 percent, as per ChatGPT's recommended stat. Rather, 61% of B2B marketers stated they had used case studies in the previous 12 months (but there's no mention of this "belief" in case studies generating leads and sales).
Back to Google! I searched around a bit more for the source of the stat, using different search terms. Finally, I found this page, which quoted a 73% stat from the Content Marketing Institute. But it was related to applying content marketing to an organization's overall marketing strategy. No mention of case studies.
Can I use the ChatGPT stat?
Honestly, I don't think so. The CMI report has some interesting stats about case studies, but none that align exactly (or even somewhat closely) with what ChatGPT provided.
Stat #4: Search Engine Land
The stat, verbatim from ChatGPT
"Case studies can increase website traffic by up to 300%. (Source: Search Engine Land)"
The link provided by ChatGPT
The result
This error page:
What I did next
I looked everywhere on Google! I used the exact phrase. I used the title provided in the URL. Keywords. And many other search strategies. I also went to the Search Engine Land website to see if I could find it.
What I discovered
Nothing. I found absolutely nothing. No articles online referencing this 300% increase in web traffic due to case studies.
Can I use the ChatGPT stat?
Certainly not. Next!
Stat #5: Demand Gen Report (again)
The stat, verbatim from ChatGPT
"In a study of 900 B2B buyers, 82% said that they found case studies to be influential in their decision-making process. (Source: Demand Gen Report)"
The link provided by ChatGPT
The result
Just like Stat #2 above, ChatGPT sent me to the wrong page.
What I did next
Another spin on Google! Using the stat verbatim, Google took me back to the 2022 Content Preferences Report. Although I looked for the stat within the entirety of the report, there was nothing about case studies pertaining to 82%.
What I discovered
A pattern, lol! Once again, no amount of digging via Google could produce that stat for me.
Can I use the ChatGPT stat?
Hell no.
Stat #6: HubSpot
The stat, verbatim from ChatGPT
"Case studies have been shown to improve retention of information by up to 70% compared to other forms of content. (Source: HubSpot)"
The link provided by ChatGPT
The result
Surprise (but not really)! Another error page. Albeit a cute one.
What I did next
Yet another Google search. Copying and pasting the stat. Plugging in keywords. The usual.
What I discovered
Despite all the other misses above, I still held out hope I'd find this stat somewhere across the internet. But sadly, no luck.
Can I use the ChatGPT stat?
What do you think?
I began wondering if I'd ever be able to trust ANY ChatGPT stats.
Stat #7: Journal of Education for Business
The stat, verbatim from ChatGPT
"A meta-analysis of 40 studies found that case-based instruction is more effective than traditional lecture-based instruction in promoting critical thinking skills. (Source: Journal of Education for Business)"
The link provided by ChatGPT
The result
Error page
What I did next
Nothing. Did you notice the stat isn't even about case studies? Thanks, ChatGPT!
What I discovered
ChatGPT takes liberties with language when interpreting human prompts.
Is ChatGPT full of BS when it comes to "facts"?
Maybe I'm naive, but I sincerely believed that the AI tool would provide some accurate data! Instead, ChatGPT provided not only seven incorrect URLs, but seven seemingly fabricated "facts."
I went down a research rabbit hole to see if the chatbot might have misinterpreted some of the numbers it provided. After all, stats can often be difficult to qualify, even for humans. But these weren't misinterpretations. We're talking about made-up numbers.
As a freelance content writer and editor, I have to do a lot of research for my B2B, government, and non-profit clients. But what if I had used ChatGPT for a client project and taken the AI's responses at face value? What if I hadn't clicked on the links? Or fact-checked each and every one of those stats?
Publishing false or misleading recklessly can have severe impacts
We've seen how "fake news" can cause confusion, fear, and even harm to individuals or groups that are targeted. From a business perspective, spreading misinformation can also damage corporate reputations, not to mention organizations' ability to garner public trust.
Obviously, case studies aren't a life-or-death type of subject matter. Still, my little experiment reinforces the need to use AI with great caution.
Lessons learned: using AI for research
I'm still learning about AI and how it relates to my profession. And so far, I've seen some great things come from tools like ChatGPT and Jasper. It's especially helpful for powering up my creative muscle when my writing brain is on overload.
But as for my little experiment using AI to conduct research? The whole experience has made me feel a bit uncomfortable.
I'll probably try doing a similar experiment in the future — hopefully next time with better luck and accuracy. But for now, here's what I've learned when using ChatGPT as a research tool:
Tip #1: Never trust AI as a standalone research tool
If you want to use AI for fact-finding, consider it as ONE tool. It's not THE tool. Use it as part of your research arsenal. In other words, ChatGPT is not replacing the library any time soon.
Tip #2: Always ask for citations and sources
You have a right to be cautious with AI. When you prompt it for information, don't blindly accept its response as gospel. Always ask the AI for citations and sources to back up what it's telling you.
Tip #3: Click on all the links AI provides
When the AI shares citations, look at them! Click on Every. Single URL.
Tip #4: AI-recommended links don't work? Do a bit of digging online
ChatGPT's responses come from a knowledge base with a cut-off date of September 2021. Therefore, the tool is limited in terms of providing the most up-to-date information.
It's possible the URLs that AI suggests could simply be outdated (that is, the links changed since late 2021). If you get an error page, do multiple Google searches using key phrases and numbers.
Need accurate, well-researched, public-facing content?
In an age when immense masses of data are readily available at our fingertips, it's more crucial than ever to be able to analyze and evaluate that information (and misinformation) critically — well before it is published.
Tools like ChatGPT can spit out numbers, but they can't be entirely relied upon to create accurate content. Not to mention, AI does not have the same critical thinking skills that humans do.
So if your organization is looking to develop truly accurate public-facing content, take extreme caution in (1) who you employ to do the job, (2) what AI tools they use, and (3) how they use those tools.
Want to do business with a talented freelance content writer who has 20+ years of solid expertise, a journalism background, and solid research and follow-up skills? Book your FREE, no-strings-attached, 30-minute consultation with me today.