Artificial Intelligence1 min read

We asked AI the same 210 buying questions 2 days in a row,84% of answers changed

By · Published by Everything Blog

In short

The study evaluated responses from ChatGPT, Gemini, Google AI Overviews, and Google AI Mode to measure how reliably platforms recommend products. Researchers discovered that 27 percent of suggested brands completely disappeared after just 24 hours due to massive source citation turnover. Established market leaders retained 73 percent of their recommendation spots, whereas smaller brands kept only 51 percent. This discrepancy highlights how unstable automated summaries can be when substituting for traditional, static search engine rankings.

Key points

  • AI engines act as real-time summaries of volatile web sources rather than maintaining sta…: AI engines act as real-time summaries of volatile web sources rather than maintaining stable product rankings.
  • Dominant market brands experience significantly better recommendation retention than smal…: Dominant market brands experience significantly better recommendation retention than smaller, less established competitors.
  • Category stability varies widely, with accounting software showing the highest brand rete…: Category stability varies widely, with accounting software showing the highest brand retention rate at 79 percent.

Sources checked by our analysis pipeline: 1

Yesterday we showed that AI product recommendations mostly come from "best X" lists. The most common question we got was simple: do the answers stay the same?

So we asked again. Same 210 questions, same four engines, same settings, one day later. Then we compared each answer with the one from the day before.

Most of them changed.

- Answers that changed overnight

- 84%701 of 839

- Brands gone the next day

- 27%named on day one, not day two

- #1 picks still named the next day

- 90%but only 63% still #1

- Spots kept by smaller brands

- 51%vs 73% for category leaders

The short version

- 84% of answers named a different set of brands the next day. On Gemini it was 95%.

- 27% of the brands named on day one were gone on day two.

- The top spot is fairly safe. A brand ranked #1 was still named the next day 90% of the time, though it was still #1 only 63% of the time.

- Lower down, it's close to a…

Artificial Intelligence