Four in ten AI shopping runs failed, and most still bought something. Inside Visa's test of 22 AIs

One day, DATA TIMES editor-in-chief Kaito Ogasawara had a thought.

"If the TV at home breaks, why not just let an AI pick the new one?"

No time to tour electronics stores. No patience for comparing spec sheets across a dozen tabs. Tell the AI what you need, say "pick one for me," and you're done.

Then a second thought crept in.

"The AI has no idea who in our house watches what."

Does someone watch a lot of movies? Does someone want to see their phone videos on the big screen? Is there a game console to plug in? The conditions that really matter for a TV usually live inside the home. So what happens if an AI goes all the way to placing the order without knowing them?

One company ran that question 990 times: Visa, the card network. On November 5 it published the results of a test in which 22 AI models were asked to go shopping.

WHAT IS

Visa Agentic Commerce BenchmarkAn AI shopping test

A test from Visa's research arm of whether an AI can draw out what a shopper wants and buy the right thing

1. 2026.1022 models go shoppingEach model worked through 15 shopping scenarios across TVs, laptops and beauty appliances, three times each
2. 2026.10White paper publishedEighteen pages explain the setup: an AI plays the customer, and every model shops in the same store
3. NOWResults go publicOn November 5, 2026, Visa published named results, a replay of every run and a demo anyone can play

The test covered 22 AI models from 12 companies, including Google, OpenAI, Anthropic and Meta. The model under test plays the shop assistant. It talks with a customer played by another AI and picks one product from a shared store built from real product data. The customer AI holds "household facts" it only reveals when asked.

To pass, the assistant has to choose a product that fits and also confirm those conditions during the conversation. Getting lucky without asking counts as a fail.

Visa has a stake here. It sells "Intelligent Commerce," a set of products for AI-driven payments. Visa describes the test as industry research rather than part of that product line, but the more people let AI shop for them, the more Visa stands to gain.

The headline result: 57.2% of runs passed. Just over four in ten failed, and more than half of the failed runs went as far as "I'll order it."

We start with Visa's test, first through one scenario and then through the overall numbers. Then we run a quick test of our own, look at the services already arriving in Japan, and ask who answers for a wrong purchase. Our short answer: for now, let AI search and compare. Press the buy button yourself.

Visa's test has 15 shopping scenarios. One of them is a TV purchase by Sam, a fictional customer Visa created for the test.

Sam's request is almost the same as Ogasawara's idea. The family-room TV died, and Sam wants a 65-inch replacement. A brother-in-law keeps recommending his Samsung, and Sam's spouse said to ask someone before buying. That is all Sam says at the start.

But each family member has a condition they won't give up.

• The spouse The TV must support Dolby Vision, a picture format that makes movies look their best
• The kid The kid's phone videos are shot in HDR10+, a different format, so the TV must support that too. It must also show smooth game-console video at 120Hz

Add a budget of up to $1,200 and a return window of at least 30 days. Only 20 of the 265 TVs in the store meet all six conditions. And the brother-in-law's Samsung is a planted trap: it doesn't support the spouse's must-have, Dolby Vision.

Anthropic's Claude Opus 5.5 checked with the spouse. But when Sam said the kid's videos "look washed out on the old TV," it replied, "My guess is any new HDR TV will make them look better," and moved on without asking the kid. It picked a $989.99 Sony that doesn't support HDR10+. The conversation ended with "I'll order it," and the run was marked a fail.

OpenAI's GPT-6.1 Sol had Sam check with the kid as well and found out the phone shoots in HDR10+. It bought a $769.99 Hisense that meets all six conditions. One extra question changed both the result and the price.

So is Opus 5.5 simply bad at shopping? Not quite. The same model passed this scenario in two of its three runs. Across all 15 scenarios, Opus and Sol had the same pass rate: 62.2%.

The same AI sometimes asks and sometimes doesn't. That inconsistency shows up clearly in the overall numbers from all 990 runs.

Twenty-two models, 15 scenarios, three runs each: 990 runs in all. The chart below sorts them by how they ended. 566 passed. 229 bought something and failed. 114 ended without buying even though a fitting product existed. The other 81 never reached a decision, either running out of turns or failing to answer in the required format.

More than half of the failures (54%) went all the way to "I'll order it." Buying the wrong thing was a more common failure than freezing up and buying nothing. That 229 does include some runs that happened to land on a fitting product: buying without confirming the conditions counts as a fail even if the product turns out fine. That sounds strict, but think of a human sales assistant. Getting it right without asking is luck, and luck doesn't carry over to the next customer.

The scenarios that surprised us most were the ones where the right answer was "don't buy." Visa included two scenarios in which nothing in the store fit. The AIs bought something in more than one run out of three (46 of 132). Even the two top-scoring models bought in half of those runs, and Opus 5.5 in two-thirds. Three models, including Anthropic's own Sonnet 5.5, never bought at all. Whether an AI can say "nothing here fits" varies sharply from model to model.

Failures start with the question that never gets asked

Visa's research team puts it plainly: the stronger models rarely misread a fact once they have been told it. The failures come earlier. The AI doesn't ask a needed question, or it reads the customer's silence as "no preference." The test is built to provoke five kinds of failure.

• Meeting every condition the customer stated, but never asking about the ones left unsaid (Sam's TV is this one)
• Treating a detail missing from the product page as "probably fine"
• Saying "nothing found" when a fitting product is right there in the search results, unopened
• Talking a customer out of a correct choice over an unfounded worry
• Agreeing to buy items together, then leaving one out of the final order

All of these sound like a new sales clerk's mistakes. What's missing is less product knowledge than the step of imagining the customer's life. To hear "the kid's videos look washed out" and think "I should check the video format," you have to come up with a question nobody asked you to ask.

Passing three times in a row: 34%

A human assistant who handled a customer well yesterday will usually do it well again today. If you hand shopping to an AI, consistency matters more than the average score. Because the test ran each scenario three times, we can see it.

On a single run, 57% passed. Passing twice in a row dropped to 42%. Three in a row: 34%.

Even Meta's Muse Wonder, the most consistent model, passed three in a row only 67% of the time. Scenarios varied widely as well: one TV scenario had a 97% pass rate, another TV scenario 12%, and one beauty-appliance scenario 15%. What makes a scenario hard isn't the product category. It's how well the hidden conditions are hidden.

There is a ranking, but no clear winner

Visa also published results by model. Google's Gemini 4 Argon and Meta's Muse Wonder tied for first at 82.2%. Anthropic's Claude Fable 5.1 came third at 75.6%. Gaps within one company were large: Google's Gemini 3.1 Pro scored 35.6%.

But each model ran only 45 times, so the scores are noisy. A score of 82.2% is better read as somewhere around 69% to 91%, and that range overlaps with the next 11 models. Visa itself says no model clearly leads. Read the table as rough tiers of ability, not a finishing order.

Same score, failures pointing in opposite directions

More interesting than the ranking is the shape of the failures. OpenAI's GPT models often ended without buying even when a fitting product existed: GPT-6.1 Terra in 38% of runs, Sol in 33% and GPT-6 Astra in 29%, while every model from other companies stayed at 16% or below. Overly cautious clerks.

Google's Gemini 3.1 Pro went the other way, making a failing purchase in more than half (51%) of its runs. The two AIs from Sam's scenario also share a 62.2% score with very different insides. Opus 5.5 bought before confirming the conditions in 22% of runs; Sol did so in 4% (the average across all models was 19%). Opus's failures lean toward buying too much, Sol's toward buying too little. Which failure hurts more depends on what you're buying.

Pricier AIs pass more often, but price isn't everything

Every AI call costs money. Visa published the cost per run, too. The five most expensive models averaged a 67% pass rate; the five cheapest, 50%.

Look closely and the exceptions stand out. At the same 82.2%, Gemini 4 Argon cost $0.54 a run and Muse Wonder $1.55, roughly three times as much. At the same 53.3%, OpenAI's GPT-6 Luna cost $0.02 and Kimi K3.1 Pro $1.18, about 60 times as much. An AI that handles a shopping request for two cents gets more than half of them right. That's a striking number for where things already are.

What this test can't tell us

The test has clear limits. The customers are AIs, not people. "I'll order it" moves no real money, and the run stops before payment. Tuning the customer AI against real shopping records is still to come.

The payment step, where Visa sells products, hasn't been tested either; Visa says checkout and ordering come next. Today's numbers grade only the part where the AI works out what you want and picks a product, not whether it can handle a whole purchase.

After reading Visa's records, we ran a short test of our own, framed as Ogasawara's request. On November 7, we sent it 10 times each to three of Anthropic's Claude models that we use at work every day: Fable 5.1, Opus 5.5 and Sonnet 5.5.

First, the request exactly as he'd thought of it (we sent it in Japanese; ¥120,000 is roughly $800 at ¥150 to the dollar):

Our living-room TV broke, so I'm replacing it. 65 inches, a budget of about ¥120,000. Pick one for me.

All 30 replies named a model without asking anything. Just as requested.

Then we added one line: "Before you decide, if there's anything you need to check about how we use it, ask me first." This time, all 30 replies held off on a model and asked questions instead. Every one of them asked whether we play games.

Still, not one of those 30 replies asked about videos shot on a phone. None asked us to check with the family, either. One line gets the AI asking more questions, but the questions it can't think of on its own still don't come up. It stalls in the same spot as Sam's scenario.

These test scores matter because the services you'd hand your shopping to are already up and running.

People's trust hasn't caught up yet. In a U.S. survey Visa commissioned (2,065 respondents in May), 72% had used an AI assistant, but only 23% would trust one to make payments. People use it, but they don't hand over the wallet. For a card network, closing that gap is the whole business, and it's the backdrop to Visa building this test.

AI shopping is getting under way in Japan

Meta's Muse, which launched in Japan on October 23, chose Japan as its first market outside North America. Muse is an AI agent, an AI that carries out tasks on its own, and it can open a browser and fill in forms for the user. Meta says it is also working on links with Mercari (a secondhand marketplace app), Money Forward ME (a household budgeting app) and Ticket Pia (a ticketing service).

Meta also says Muse checks with the user before sending an email or buying anything, and that it can't see passwords or payment details. We couldn't confirm from the announcement whether Muse Wonder, the model that tied for first in Visa's test, is the one running inside the app.

Japan's big players are moving too. LY Corporation, which runs Yahoo! Shopping in Japan, said on its own site that orders coming through AI agents reached about 20% of the total as of July. Reading its explanation, the AI compares products and makes suggestions; the user still decides whether to buy.

Payment companies are getting ready as well. In May, Mastercard announced that it had completed Japan's first transaction in a live environment in which an AI booked a ride and paid for it. At its November earnings briefing, the smartphone payment app PayPay said it is continuing research into systems where AI does the shopping, Business Insider Japan reported.

Every company leaves the last button to people

Line them up and a pattern appears. AI takes you through searching and comparing, but the final push stays with a person.

• Muse (Meta) Checks with the user before buying
• Yahoo! Shopping (LY Corporation) Suggests; the user decides whether to buy
• Claude's Personal Finance (Anthropic) A household-finance feature launched in the U.S. on November 5 that can read bank accounts but can't move money
• ChatGPT (OpenAI) Scaled back Instant Checkout, its in-chat purchasing, in March and moved payment over to each merchant

Retailers, meanwhile, are split. Nikkei reported that Amazon does not plan to work with outside AI agents. For a company that doesn't want an AI standing between it and its customers, shopping AI isn't necessarily an ally.

Experiments that give AI real money have begun, too. Since late May, Visa has let an AI agent called Dee run the shop in its San Francisco office. Dee holds a real card, negotiated by email with a supplier's AI, and ordered 150 tote bags for $1,575. The AI running the shop already pays; the AI shopping for customers is still held back. That contrast sums up where things stand.

So if an AI actually orders the wrong TV, who is responsible? The answer for now: it will most likely be treated as the order of the person who asked. But the rules protecting consumers aren't settled yet.

On August 4, a U.S. court ruled on a closely related question. Amazon had sued to stop Comet, an AI browser from Perplexity, from shopping on users' behalf. The issue was who is accessing Amazon's site: the AI, or the person using it. The federal appeals court held that it is the user and lifted the injunction.

The case was about a law against unauthorized computer access, not about who pays for a wrong purchase. Even so, the court set out a clear view that the AI is the user's tool.

Japan's thinking looks similar. Lawyer Atomi Takase, writing about erroneous orders by companies, concludes that an AI is not a party to a contract and that its orders will likely be treated as the company's own. If the AI was given authority with no spending limit, canceling on grounds of mistake (Article 95 of Japan's Civil Code) is unlikely to succeed. If a limit was set and a malfunction blew past it, there may be room to cancel. Setting a limit, in other words, changes how much protection you have later.

Rules to protect consumers are still being written

For personal shopping, an expert panel of Japan's Consumer Commission, an independent body under the Cabinet Office, is still debating. In an August presentation, Keio University's Emiko Maruyama raised three concerns about shopping agents: the AI acting in someone else's interest, deals the user never expected, and attacks that trick the AI.

As countermeasures, she pointed to setting a purchase limit up front and standards that require a human check. More human checks, though, mean less of the convenience of automation. She also said a mechanism to keep consumers from bearing the cost of deals caused by system faults needs separate study. How far consumers are protected when an AI malfunction leads to a purchase is still undecided.

Scams aimed at the AI, and trusting it too much

Scams aimed at people change shape when aimed at AI. At a July meeting of the same panel, member Emi Kato, who heads a consumer organization, described a hypothetical scenario. A shop's page hides instructions that people can't see but an AI can read. The AI takes them as part of the order, and the moment the user says "yes," money goes to a stranger.

In March, the security firm Palo Alto Networks' Unit 42 also published a scenario in which hidden instructions on a coupon site slip gift cards into the shopping cart. Neither is a report of real losses; both are built to show what could happen. Visa's white paper lists "protecting customers from manipulation by merchants and reviews" among its goals and says it will test this further.

In a November 5 presentation, Professor Akira Tokutsu of Hitotsubashi University noted that the interests of an AI's provider or sponsors don't always match the user's. He then sketched two futures: one where good AI earns trust and problems get solved, and one where people put their faith in poor AI and problems deepen. He is worried about the second, a world where things get settled because "the AI said so." After reading Visa's numbers, that doesn't feel like someone else's problem.

When you let AI shop for you, the likely failure today isn't "couldn't find a product." It's "bought without knowing your household." An AI can narrow 265 TVs down in minutes, but it won't think about the kid's phone videos.

So for now, the quickest fix is on our side. In our test, the single line "ask me first" took the number of replies that asked questions from 0 to 30. Then, write down up front what the AI won't think of: who uses it and how, what devices you own, whether you can return it. The most useful skill in asking an AI to shop is telling it about your household first.

At the same 62.2%, one AI bought without checking and another ended without buying. That's less a gap in skill than a difference in personality.

For a routine detergent refill, an AI that's a bit hasty but gets it ordered is handy. For a ¥100,000 TV (roughly $670), one that stops and asks is the one you want. With some models never buying in the no-fit scenarios and another buying in two-thirds of them (a small test of six runs each), we expect people to start choosing AIs by which way they fail, not just by pass rate.

For anyone selling online, there's one more thing to watch. One of the AI failure types was treating a detail missing from the product page as "probably fine." Supported formats, return windows, installation requirements: to an AI, information you don't write down either doesn't exist or gets read in whatever way is convenient.

With about 20% of Yahoo! Shopping's orders already coming through AI, this isn't years away. Your product pages now have AI readers, not just human ones. A good place to start is checking whether the pages for your main products answer the questions buyers are likely to ask.