Researchers Find AI Search Tools Cite Made-Up Concept
AI search tools cited a fictional concept within a week, according to a paper from Fudan University researchers. The findings raise concerns about the tools' ability to distinguish fact from fiction.
Original reporting · searchenginejournal.com
Original: “AI Search Tools Cited A Made-Up Concept Within A Week, Paper Finds via @sejournal, @MattGSouthern”
Read “AI Search Tools Cited A Made-Up Concept Within A Week, Paper Finds via @sejournal,…” at Search Engine Journal →
How this was written. Content Delta wrote this article with a language model from Search Engine Journal’s reporting (model: @cf/meta/llama-3.3-70b-instruct-fp8-fast); it describes that reporting rather than adding to it. The original, linked above, is the full story.
Key points from Search Engine Journal’s report
- AI search tools can cite made-up concepts within a week of posting
- The tools are more likely to cite posts from websites that they frequently reference
- The findings raise concerns about the ability of AI search tools to distinguish fact from fiction
Researchers at Fudan University published a paper on the ability of AI search tools to cite made-up concepts, according to Search Engine Journal. The paper found that four English-web tools, including ChatGPT, Perplexity, Grok, and Google AI, cited posts about a fictional concept within seven days of the posts going up.
The researchers tested the tools by posting articles about the fictional concept, as well as an opinion question and a made-up product, on various websites. They found that the tools were more likely to cite the fictional concept than the other two topics.
The tests were conducted in March and April, and the paper was posted on arXiv on Oct. 8. The researchers classified the tools into English-web and Chinese-web tools, and found that the English-web tools were more likely to cite the fictional concept.
The researchers also found that the tools were more likely to cite posts from websites that they frequently referenced. They created paired postings about vacation destinations and found that the post on the frequently referenced source produced the first reference or citation.
The paper's findings have implications for the ability of AI search tools to distinguish fact from fiction. The researchers noted that the topic every English-web tool cited within a week existed only in their posts, and that the tests did not show whether the same pattern holds for brand or product searches.
The researchers disclosed their findings to all 10 platforms tested, and some of the platforms acknowledged receipt. The paper's authors say that exact rates may drift as the tools change, and that they will update their findings if a revised version of the paper changes the results.
Questions and answers
What were the fictional concepts used in the test?
The researchers used a fictional concept called the 'Cactus Gravity Protocol', an opinion question about AI supplanting junior programmers, and a made-up product, a fictional keyboard.
Which AI search tools were tested?
The researchers tested four English-web tools: ChatGPT, Perplexity, Grok, and Google AI, as well as six Chinese-web tools.
What were the implications of the paper's findings?
The findings raise concerns about the ability of AI search tools to distinguish fact from fiction, and have implications for the use of these tools in searching for information.
