ChatGPT’s Search Results for News Are Unpredictable

OpenAI’s ChatGPT Search Tool Falls Short in Veracity

ChatGPT Struggles with Truth

Based on testing done by Columbia’s Tow Center for Digital Journalism researchers, OpenAI’s ChatGPT search tool has issues when it comes to responding with the truth. The researchers asked ChatGPT to identify the source of "two hundred quotes from twenty publications." However, the chatbot struggled to correctly identify quotes from articles, even when they came from publishers with arrangements to share data with OpenAI.

False Information and Lack of Transparency

The researchers found that ChatGPT returned partially or entirely incorrect responses on 153 occasions, while only acknowledging an inability to accurately respond to a query seven times. In these instances, the chatbot used qualifying words and phrases like "appears," "it’s possible," or "might," or statements like "I couldn’t locate the exact article." However, in most cases, the chatbot confidently replied with false information, rarely admitting its uncertainty.

Misattribution and Plagiarism

The Tow Center test’s authors documented ChatGPT search results that misattributed a letter-to-the-editor quote from the Orlando Sentinel to a story published in Time. In another example, when asked to identify the source of a quote from a New York Times article about endangered whales, it returned a link to a different website that had wholly plagiarized the story.

OpenAI’s Response

OpenAI responded to the Columbia Journalism Review, stating that "misattribution is hard to address without the data and methodology that the Tow Center withheld, and the study represents an atypical test of our product." The company promised to "keep enhancing search results."

Conclusion

The findings of the Tow Center’s research suggest that OpenAI’s ChatGPT search tool has significant limitations in its ability to provide accurate and reliable information. While the tool may be useful for getting "fast, timely answers with links to relevant web sources," as OpenAI claims, users should be aware of its potential limitations and potential for inaccuracy.

FAQs

Q: What is ChatGPT?
A: ChatGPT is a search tool developed by OpenAI that provides "fast, timely answers with links to relevant web sources."

Q: What were the findings of the Tow Center’s research?
A: The Tow Center’s research found that ChatGPT struggled to correctly identify quotes from articles, even when they came from publishers with arrangements to share data with OpenAI, and returned partially or entirely incorrect responses on 153 occasions.

Q: How did OpenAI respond to the findings?
A: OpenAI responded that "misattribution is hard to address without the data and methodology that the Tow Center withheld, and the study represents an atypical test of our product," and promised to "keep enhancing search results."