AI's Accuracy Woes: A Challenge for Holiday Shopping

Artificial intelligence is not yet ready to fully manage your holiday shopping, according to a recent analysis.
A study conducted by Product.ai, a startup specializing in verifying product claims, indicates that AI-powered shopping assistants frequently offer inconsistent details regarding pricing, product specifications, and current availability. The company evaluated free and premium versions of popular AI models including ChatGPT, Claude, Gemini, and Perplexity. They posed 220 shopping-related questions about various items such as laptops, televisions, mattresses, sunscreens, and robotic vacuum cleaners, running each query five times. This process generated a total of 8,794 responses. Product.ai's findings revealed that 86% of these inquiries resulted in verifiable factual discrepancies, defined as discernible conflicts in information like prices, product models, or specifications across multiple responses.
The study highlights a core problem with AI's current capabilities: its difficulty in providing precise information essential for both human and AI-driven shopping experiences. Among the AI tools tested, Gemini exhibited the highest proportion of significant errors, with its free version showing a 56% error rate and its paid version at 54%. While Claude's performance improved with its paid offering, reducing critical errors from 44% to 21%, Perplexity recorded the lowest rate of such errors at 14% for its paid service, followed by ChatGPT's paid version at 17%. Furthermore, the research found that these AI tools sometimes contradicted themselves when given the same question multiple times, with Gemini's free tier demonstrating this inconsistency in 29% of cases. When inaccuracies occurred, the price discrepancies often amounted to a median of $300, a substantial difference that should prompt consumers to exercise caution, as noted by Dakota Nunley, head of search product at Product.ai.
Given these findings, consumers are advised against blindly trusting AI agents or allowing them to handle payment details without oversight. It is recommended that users independently verify information provided by AI tools. This might involve cross-referencing answers from several AI engines or, more simply, checking the vendors' official websites directly. For the upcoming holiday season, AI can serve as an effective tool for product discovery and brainstorming, but it is crucial to remain vigilant and avoid completely relinquishing control over purchasing decisions. This approach ensures a more informed and reliable shopping experience.