Shopify Catalog Tested With AI Models
Three AI models were tested on Shopify's catalog, each picking a different product, according to Search Engine Journal. The test highlights the importance of AI models in agentic commerce.
Original reporting · searchenginejournal.com
Original: “3 AI Models Shopped Shopify’s Catalog – Each Picked A Different Winner via @sejournal, @slobodanmanic”
Read “3 AI Models Shopped Shopify’s Catalog – Each Picked A Different Winner via @sejournal,…” at Search Engine Journal →
How this was written. Content Delta wrote this article with a language model from Search Engine Journal’s reporting (model: @cf/meta/llama-3.3-70b-instruct-fp8-fast); it describes that reporting rather than adding to it. The original, linked above, is the full story.
Key points from Search Engine Journal’s report
- Three AI models were tested on Shopify's catalog, each picking a different product.
- The models were consistent with themselves, but not with each other.
- The catalog's ranking mattered less than expected, with the models preferring products with the most reviews.
- The test highlights the potential for inconsistencies in the data, with different review counts shown in the catalog and on the store's page.
- The choice of AI model can affect which product is chosen, and ultimately, which store gets the customer's order.
Search Engine Journal reported that Shopify announced Shop MCP, a feature that allows agents or apps to search the Shopify catalog and complete checkout through a single connection. This feature covers over a billion products and does not require an API key, making it accessible to any company building a shopping agent.
The test was conducted to see how different AI models would pick products from the same list, with the same instruction and request. The models were given a list of 20 products, along with their title, store, price, star rating, and review count, and were asked to choose one product to buy.
The models used in the test were Claude Haiku 4.5, Sonnet 5, and Opus 5, run through Claude Code with default settings and no web access. Each model was given the same list of products and the same request, and the results were compared to see if the models would pick the same product.
The results showed that the three models did not agree on the top product in any of the six searches. Haiku and Sonnet agreed on three products, Sonnet and Opus agreed on two, and Haiku and Opus agreed on one. Each model was consistent with itself, but not with the other models.
The test also showed that the catalog's ranking mattered less than expected, with Haiku picking the top-ranked product only 8% of the time, Sonnet never, and Opus 20% of the time. The models seemed to prefer products with the most reviews, even if the review count was not shown on the store's page.
In one test, the models were given a request for a water bottle, and Haiku and Sonnet picked the same product, citing its high review count. However, the review count shown in the catalog was different from the count shown on the store's page, highlighting the potential for inconsistencies in the data.
The results of the test have implications for agentic commerce, where agents use AI models to find and buy products for shoppers. The choice of AI model can affect which product is chosen, and ultimately, which store gets the customer's order.
Questions and answers
What is Shop MCP?
Shop MCP is a feature that allows agents or apps to search the Shopify catalog and complete checkout through a single connection, according to Search Engine Journal.
How did the AI models pick products?
The models were given a list of 20 products, along with their title, store, price, star rating, and review count, and were asked to choose one product to buy.
What were the implications of the test?
The test highlights the importance of AI models in agentic commerce, and how the choice of AI model can affect which product is chosen, and ultimately, which store gets the customer's order.