34.8% of the variance in 'brand mentions in AI answers' is resampling noise
A KPI that's showing up on more and more agency dashboards this year is "brand mention percentage in AI answers", the share of AI-generated responses that mention your brand when asked a relevant question. It's an intuitive number, treated the way a search-ranking position used to be treated. We wanted to know how much of the movement in that number, week to week, is signal versus noise, so we ran a variance decomposition across 12,933 AI-answer responses.
The answer is that most of it is noise. Resampling the identical question against the identical model, nothing about the brand or the prompt changed, accounted for 34.8% of the variance we measured. The actual brand-mention signal we were trying to track accounted for 0.7%. Put plainly: if you ran the same query twice in a row and got different answers, and you usually would, most of what moved between those two answers had nothing to do with your brand at all.
That's not a small caveat on an otherwise-useful metric. It's most of the variance in the number. A dashboard tracking this week over week, without accounting for resampling noise, is mostly reporting the AI model's own inconsistency back to you dressed up as a brand trend. A real change in your visibility can sit inside a swing that size and be indistinguishable from nothing having happened at all.
This doesn't mean the underlying question, does an AI model mention your brand when it's relevant, is a bad one to ask. It means asking it once and charting the answer isn't measuring what people think it's measuring. Getting a usable number out of it means sampling enough times per period to separate the signal from the noise the model itself introduces, which is a different and considerably less convenient exercise than a weekly dashboard refresh.