Loading...
ChatGPT answers change because large language models generate responses probabilistically, and the output shifts with prompt wording, model version, session context, and whether live retrieval is used. A single screenshot therefore captures one draw from a distribution, not your brand's real position. Reliable reporting needs many relevant questions, run repeatedly across engines, and summarised as trends, averages, and share of voice. In this guide I explain why the variation happens, what it means for the numbers you put in front of leadership, and how to build a sampling method that survives scrutiny, limits included.
You ask ChatGPT for the best tool in your category, and your brand is third on the list. You ask again ten minutes later and you are first, or missing entirely. The wording has shifted, the order has changed, and a competitor you had never seen mentioned has appeared. The natural reaction is to suspect something has broken or that your brand has been dropped. In almost every case, neither is true.
A language model produces text one token at a time, sampling each next word from a probability distribution rather than looking up a stored answer. Temperature and related sampling settings control how much randomness enters that process, so identical inputs can legitimately yield different outputs. OpenAI's guidance on reproducible outputs says that outputs are nondeterministic by default. Even when a developer fixes the seed parameter and the request settings, determinism is only mostly achievable, not guaranteed, and the API returns a system_fingerprint precisely because backend changes can alter results. That guidance concerns the API, but the same sampling principle applies to the consumer ChatGPT product.
Variation is simply how generative AI works. It is not a defect and it is not evidence of a penalty. The idea I want you to carry through the rest of this article is that each answer is one observation from a range of possible answers. Our job as analysts is not to capture a single observation and treat it as truth. It is to estimate the range: how often your brand appears, how prominently, and how that pattern moves over time.
Sampling randomness is only the first layer. Several other factors move answers, and a credible method has to control or at least record each one.
One caveat on the evidence: the published research I have cited concerns prompt robustness and reproducibility in general. I am not aware of a rigorous, public figure for how often consumer ChatGPT brand recommendations overlap between repeated runs, and I would treat any such statistic you see quoted with caution until you have checked the original methodology.
The habit is familiar. Someone asks ChatGPT a question, sees the brand ranked first (or conspicuously absent), screenshots it, and the image lands in the board deck. I understand the appeal, because a screenshot is vivid and quick. It is also statistically weak, and it fails in two ways.
A screenshot also hides everything that makes a measurement interpretable: the sample size (one), the exact prompt, the model, the date, whether search was on, and the competitor context that determines share of voice. The credibility cost arrives when a director re-asks the question, gets a different answer, and concludes that the whole channel is unreliable. That distrust is hard to reverse.
So does a single mention in ChatGPT mean your brand is visible? No. One mention shows that appearing is possible, not that it is likely. Visibility is consistent presence: how often, how prominently, and across how many relevant questions your brand shows up over time.
Two ideas replace the screenshot. The first is query coverage, meaning the breadth of realistic customer questions in which your brand could appear. These include category questions ('what is the best accounting software'), comparisons ('X versus Y'), problem-led queries ('how do I stop chasing late invoices'), and purchase-intent queries ('accounting software for a UK limited company'). Checking one question tells you nothing about the others.
The second is repeated sampling: running each question many times so you can report an appearance rate instead of a yes or no. Before the arithmetic, one design point matters. Your data has two different sources of change, and they should not be mixed.
My recommendation is to treat each day (or each scheduled run) as its own batch with conditions held constant, record the date and conditions alongside the resulting rate, and then compare batches across time. That way, spread within a batch is sampling noise, and drift between batches is what you investigate.
Here is a simple binomial illustration of why batch size matters. Suppose your brand's true appearance rate for a question is 40%, and suppose each run in a batch behaves as an independent draw. That independence is an assumption, not a guarantee. Runs sharing a model snapshot, a retrieval index, or a cached source may be correlated, which would make the real uncertainty wider than shown. Using the normal approximation for a 95% confidence interval, the margin of error is roughly:
| Runs in a batch | Approximate margin of error (illustrative) |
|---|---|
| 1 | Meaningless (result is 0% or 100%) |
| 10 | about ±30 points |
| 20 | about ±21 points |
| 50 | about ±14 points |
| 100 | about ±10 points |
So 8 appearances in 20 runs gives a 40% rate with a plausible range of around 19% to 61%. That is still wide, but it is far more informative than one run. The interval narrows only with the square root of the sample, so returns diminish. Treat the table as an illustration of the arithmetic, not a measurement of any particular engine. The normal approximation is also rough at small batch sizes, and a Wilson interval is a better choice if you want to report precise ranges.
There are other signals a screenshot rarely captures. Position-weighted citations recognise that being the first recommendation or first cited source is worth more than appearing fifth. Soft mentions, where the brand is named without a link, matter because many answers recommend a company without citing it. And because OpenAI states that ChatGPT's search answers draw on sources retrieved for that query, I recommend logging cited domains and URLs alongside the mention itself.
How many runs do you need? There is no validated threshold. As a pilot heuristic, I suggest 10 to 20 runs per important question per engine per batch, with more for questions that flip between presence and absence and fewer for stable ones. Use the interval above to judge whether that is precise enough for the decision in front of you, and always publish the rate together with the run dates, engine, mode, and sample size.
Once you are collecting repeated samples, the question becomes how to read them. These are the principles I apply.
A short worked example, with invented numbers, shows how this plays out. Suppose that over the first three weeks your 28-day average appearance rate for a question group sits between 22% and 30%, which becomes your baseline band. In week six the 7-day average reaches 38% and stays there for three weeks, on two engines, across three question groups. That meets my heuristic: sustained, outside the band, and visible in more than one place. If the rate had touched 38% for a single day on one engine, I would log it and wait.
What counts as a real change versus noise? Sustained, beyond your band, and visible in more than one place. Do engines vary by the same amount? Not necessarily, so chart each separately before combining them.
It is also worth keeping the wider context in proportion, and being clear about its geography. Pew Research Center's 2025 analysis of 68,879 Google searches by 900 U.S. adults found AI summaries in about 18% of searches, and users clicked a traditional result in 8% of visits with a summary versus 15% without. Gartner forecast in 2024 that traditional search engine volume would fall 25% by 2026, though that is a global forecast, not an outcome. Neither measures UK behaviour or ChatGPT brand mentions, and UK patterns may differ. They do help explain why answer-engine visibility deserves a place alongside conventional metrics. Google's documentation on AI features in Search explains that this traffic is counted within Web search in Search Console. It does not tell you what ChatGPT says about you, so do not conflate the two.
You can run a credible pilot with nothing more than a spreadsheet. Here is the method I recommend.
I should be honest about the limits. Thirty questions, three engines, and ten runs each comes to 900 responses per batch, and every one needs to be read and coded. That is feasible for a one-off pilot. It gets time-consuming very quickly, though, and inconsistency creeps in as different people code answers differently. A short coding guide and a second reviewer on a sample of answers help considerably.
The tipping point arrives when you track more than a handful of questions, more than one engine, or need weekly reporting. At that stage the labour of manual coding outweighs the cost saving, and consistency suffers. Daily AI visibility tracking also produces enough data for rolling averages and variation bands to mean something.
| Dimension | Manual spreadsheet sampling | Automated daily monitoring |
|---|---|---|
| Effort | High and recurring | Set up once, runs continuously |
| Consistency | Varies by person and day | Same questions, same process |
| Engine coverage | Usually one or two | Several engines in parallel |
| Trend quality | Sparse, hard to smooth | Daily data, rolling averages |
| Reporting readiness | Built by hand each time | Digests and exports on tap |
| Cost | Staff time | Subscription |
Disclosure: MentionOwl is our product, so read this paragraph as a vendor-specific example, not neutral advice. MentionOwl crawls your website, generates customer questions, and runs them daily against ChatGPT, Claude, Gemini, Copilot, and Perplexity, recording answers, citations, sentiment, and competitor mentions. Its 0-100 visibility score is a proprietary index built from query coverage, position-weighted citations, share of voice, and soft mentions, so it is not interchangeable with other vendors' scores. It also has weekly digests, a REST API, and AI legibility audits. Please check current features and pricing with the vendor before relying on any of this.
Whichever tool you evaluate, I would ask five questions. Can you see and edit the question set? Does it record the engine, mode, and date for every run? Does it show component metrics as well as a headline score? Can you export raw answers to audit the coding? And is the scoring method documented well enough that you could explain it to a sceptical director?
So, can you track brand mentions in ChatGPT automatically? Yes, and LLM monitoring tools exist for exactly this, at sample sizes manual work cannot sustain.
The data only helps if leadership trusts it. This is the structure I use.
When a director says, 'But I asked ChatGPT and we weren't there,' you might respond:
'That's a fair observation, and it's one possible answer out of many. ChatGPT gives different answers each time, so we don't rely on a single result. Across 40 questions run daily, we appear in around X% of answers, against Y% for our nearest competitor, and that figure has been stable or rising for four weeks. Your result is consistent with us not appearing in some runs, which is exactly what we measure.'
That answer turns an anecdote into a conversation about rates, which is where the useful decisions get made.
Because the model samples its text probabilistically rather than retrieving a fixed answer. Wording, session context, model version, and live retrieval add further variation.
There is no validated threshold, and one run is never enough. Ten to twenty runs per important question, per engine, per batch is a pilot heuristic. Check the resulting interval against the decision you need to make, and report run dates and conditions alongside the rate.
No. One mention shows that appearing is possible, not that it is likely. Visibility is how often, how prominently, and in how many relevant questions you appear over time.
Report trends, not snapshots: a visibility score with a rolling average and variation band, share of voice against named competitors, the sample size, and a methodology note.
Not necessarily. Variation depends on each engine's mode and how much it relies on live retrieval, and both change over time, so track each engine separately before combining them.
A change that is sustained, exceeds your baseline band, and appears across several questions or engines. My two-to-four-week persistence rule is a heuristic, not a standard.
Yes. LLM monitoring tools, including our own MentionOwl, run customer questions on a schedule across several engines and record mentions, citations, sentiment, and competitors. Judge any tool by how transparent its method is.
Learn how an AI visibility score works: explore query coverage, citations, AI share of voice and soft mentions, and discover what changes mean.
Wondering if you can just ask ChatGPT yourself each week? We break down the real time cost of manual AI visibility checks versus automated LLM monitor
Learn how agencies can turn brand monitoring into an AI visibility retainer with practical deliverables, pricing, tools, and a plan to win clients ear