AI hallucinations: Retail technology experts sound off on the risks
By: Mark Hollmer – July 06, 2025
Supporters of generative artificial intelligence — technology that learns from its mistakes and appears to think like humans do — pitch it as a historic advance that can boost efficiency and improve customer interaction for automotive retail and other industries around the world.
But it also can lapse into disruptive hiccups known as hallucinations, which create problems for businesses using the technology if left unchecked.
“Hallucinations are a well-known challenge with broad generative AI models,” Devin Daly, CEO of retail technology company Impel, told Automotive News via email. “And hallucinations absolutely matter, especially in high-stakes, consumer-facing industries like automotive where even a single inaccurate or misleading response can erode trust or create significant financial and reputational risk for businesses.”
AI hallucinations have been an issue from the beginning
Hallucinations have existed since generative AI debuted in 2023. They occur when chatbots or computer vision tools see patterns or objects that don’t exist, “creating outputs that are nonsensical or altogether inaccurate,” according to an IBM web posting from that year.
In other words, AI hallucinations can manifest as strange answers to questions or fictional data that has no connection to the question asked. They’re also not going away, said Steve Greenfield, CEO of Automotive Ventures, a venture capital fund that invests in retail and mobility technology.
“In the future, consumers will be interacting with some sort of AI and they’ll get a false answer and a dealer will be at risk,” Greenfield said. “Like humans, these things aren’t infallible. They’ll make errors.”
Some dealerships have told Inga Maurer, a senior partner with analyst firm McKinsey & Co., about a type of hallucination they have faced with customer relationship management platforms driven by AI chatbots. It occurs, she said, when the chatbot is asked to draft a set of new messages for customers but uses information from prior sales outreach campaigns instead.
“It is an informed hallucination. It doesn’t make stuff up, but it didn’t realize that the campaign was over three months ago, and it now tells the customer there’s a campaign they’re eligible for,” Maurer said.
Wider statistics on hallucinations are hard to find. In May, the New York Times and other media outlets reported that ChatGPT developer OpenAI found its newest generative AI iteration hallucinated 48 percent of the time when answering questions about public figures versus 33 percent with the previous version. That older version, in turn, performed worse than its predecessor.
Minimizing hallucinations is possible
Aharon Horwitz is CEO of Fullpath, an automotive retail technology company in Florida and Israel that provides a customer data and marketing automation platform for dealerships. Part of its services include ChatGPT-driven chatbots. It was one of the first retail technology companies to use them.
In December 2023, customer inquiries to some of those chatbots managed by Fullpath went off the rails because of an inundation of prankster requests. A customer persuaded one chatbot to sell it a 2024 Chevrolet Tahoe for $1 (the deal was not binding) in what some view as an early example of AI hallucination.
Horwitz disagreed with the term “hallucination” in those cases, framing it more as the “function of more primitive guardrails.” The AI followed users’ instructions and parroted back information. After the $1 incident and related issues, Fullpath added programming that limited the ability of users to convince AI chatbots to make wonky decisions. It also helps reduce hallucinations.
“When we went live initially, it was unexplored territory,” Horwitz said. “Since then, the space has evolved tremendously, and there are a number of different techniques that companies use to protect against hallucinations, some of which we use as well.”
Chatbots checking chatbots
For Fullpath, that meant installing other ChatGPT chatbots to monitor the primary ones in its system to determine whether what they are telling users is true. Those chatbots compare the information to a given dataset, and if it doesn’t match, it gets flagged for human users to check. Another innovation: diversifying the number of chatbots doing calculations so no single one handles too much information.
Improvements also now make for better “memory retrieval,” Horwitz said, allowing users to be more direct with a chatbot request.
“You can tell a chatbot, ‘Now don’t just go and answer this from your general knowledge. Go and look at the specific information I am giving you, and from that, answer the question,’” Horwitz said.
Keeping AI focused
Impel, of New York City, produces an AI platform that handles marketing, merchandising, sales and service operations for dealerships. Daly said the company has reduced hallucination risks, in part, by building “a layered architecture” into its platform, including an automotive-specific AI engine that’s trained to understand what is and isn’t relevant for automotive retail.
“These layers help the AI stay focused, avoid off-topic or out-of-policy responses and detect attempts to ‘trick’ it with malicious or misleading prompts,” Daly said.
In addition, Impel’s AI relies on what Daly calls a “proprietary knowledge bank” — a database that lets dealerships customize the AI function by inputting their specific strategies, services, policies, pricing structures and preferred messaging plus brand-specific content from manufacturers. Daly said this helps reduce hallucination risks by giving the AI the data and context it needs to respond accurately.
Impel also has worked on improving how AI functions in a “real-world retail environment” with continued refinement and sharing of information, Daly said.
“Our AI continues to learn over time and we continue to refine and improve it continuously,” he said. “It’s also part of a broader research effort that’s being shared publicly so developers across industries can learn from it, test it and build safer and more trustworthy AI systems for everyone.”
Retail automotive experts: Managed hallucinations an acceptable part of doing business
Greenfield said generative AI’s benefits outweigh any risks from hallucinations, comparing them to Google searches that turn up some relevant results and others that miss the mark.
“If you search on Google and it shows you 10 links in the organic search, then we don’t take for granted that all of them are relevant,” Greenfield said. “We decide which of those 10 links are worthy of clicking on.”
Having a human monitor also can help minimize problems, he said.
Maurer agreed.
“If you let AI without a human into the loop, then hallucinations are a real problem because then nobody catches it,” Maurer said. “It has potentially negative implications for the customers and how they are engaging with the dealership.”
A human can best intervene with proper training, Maurer said.
“It’s more about training the human in the loop on how to spot hallucinations and how to correct them because some AI hallucinations can be very convincing,” she said.
Maurer doesn’t expect hallucinations to go away, but she said AI has continued to improve, learning to better ingest emails and communicate back in a style that mimics how an individual salesperson would engage with a customer.
Further improvements also are likely to reduce hallucination risks, she said.
“Do I believe that we come to the point where … hallucinations can be minimized? Yes, absolutely,” Maurer said.
Daly offered similar optimism.
“With the right safeguards and an industry-specific approach,” he said, “they can be significantly reduced and managed responsibly.”
