ChatGPT Search Integration Raises Concerns
Inefficient Search Process
In October, OpenAI integrated ChatGPT Search into ChatGPT, promising an experience in which users could browse the web and access the latest news from its news partners and sites that have not blocked OpenAI’s web crawler. However, a new review by Columbia’s Tow Center for Digital Journalism shows that the process may not be as efficient as it sounds.
Testing Publisher Content Representation
The Tow Center performed a test to determine how well publisher content is represented on ChatGPT. It selected 10 articles from 20 random publishers who partnered with OpenAI, are involved in lawsuits against OpenAI, or unaffiliated publishers who either allowed or blocked the web crawler.
Methodology
The researcher then extracted 200 quotes, which, when run among search engines like Google or Bing, pointed back to the source in the top three results. Finally, it was time to let ChatGPT identify the quotes’ sources. Ultimately, the goal was to see if the AI accurately serves publications, giving them credit for their work.
Results
The results varied in accuracy, some entirely correct or incorrect, and some partially correct. Yet, nearly all answers were presented confidently, without the AI saying it couldn’t produce an answer even from publishers who had blocked its web crawler. Only in seven of the outputs did ChatGPT say to use words or phrases that insinuated it was unclear.
Concerns
Beyond misleading users, ChatGPT’s false confidence could risk causing reputational damage to publishers. That statement was backed up by an example in which ChatGPT inaccurately attributed a quote from the Orlando Sentinel to a Time article, with over a third of ChatGPT’s responses with incorrect citations being of that nature.
Other Issues
Other problematic findings from the experiment include ChatGPT citing an article from The New York Times, which has blocked it, from another website that had plagiarized the article, or the citing of a syndicated version of a piece from MIT Tech Review instead of the original article, although MIT Tech Review does allow crawling to take place.
Conclusion
Ultimately, this research points to a larger question of whether or not partnering with these AI companies offers publishers more control and whether creating new AI search engines truly benefits publishers or hurts their businesses in the long run.
FAQs
Q: What are the concerns with ChatGPT Search Integration?
A: The concerns are that the integration may not be as efficient as promised, and that it may cause reputational damage to publishers due to inaccurate citations and false confidence.
Q: What was the methodology used in the test?
A: The researcher extracted 200 quotes, which were then run through search engines like Google or Bing, and then analyzed to see if ChatGPT accurately identified the quotes’ sources.
Q: What were the results of the test?
A: The results varied in accuracy, with some entirely correct or incorrect, and some partially correct. Nearly all answers were presented confidently, without the AI saying it couldn’t produce an answer even from publishers who had blocked its web crawler.
Q: What are the implications for publishers?
A: The implications are that partnering with AI companies may not offer publishers more control, and that creating new AI search engines may hurt their businesses in the long run.
