Does the specific ChatGPT model version a customer is using change whether we show up, and should that affect our playbook?
ChatGPT's model version changes which answer a shopper sees, because ChatGPT routes prompts automatically across several models. OpenAI describes GPT-5 as a single auto-switching system and names GPT-5.4 mini as the fallback when paid users hit rate limits (per OpenAI's model release notes, August 2026). Visibility tracking that records no model version records an unrepeatable result.
The mechanism
Three things vary between two shoppers asking the same question:
- Which model answers. OpenAI's release notes describe automatic routing, list GPT-5.6 Sol, GPT-5.4 Thinking and GPT-5.3 Instant among current models, and name GPT-5.4 mini as the fallback when paid users hit rate limits (per OpenAI, August 2026).
- Whether the model searched. A routed answer that triggers retrieval cites live sources; one answered from parametric knowledge cites nothing and may name brands from training data alone.
- Personalization and memory. A returning shopper's prior context is part of the prompt, and none of it is visible to you.
The implications for your playbook
Version-aware tracking stops being a nicety. A prompt panel that logs the answer but not the model, the plan tier, the date and whether sources were cited produces numbers that cannot be compared month over month. That is the difference between a measurement and an anecdote.
Practical minimum for a mid-market panel: fixed prompt set, logged model name (ChatGPT reports the active model when asked, per OpenAI, August 2026), logged date, logged citation presence, same account tier each run, and a stated sample size.
Edge cases worth calling out
- API results are not consumer results. A panel built on the API measures a different system than the one your customers use. Useful as a control, misleading as the headline number.
- Single-run screenshots prove nothing. Routing and personalization make one run a sample of one.
- A model update is a legitimate explanation for a swing. Before rewriting content in response to a drop, check whether the model changed under you.
What this does not change. None of the above alters what makes a page citable. Routing changes the odds of retrieval firing; it does not change which page gets picked once it does.
The Deploi point of view
Our own position, from building on Shopify. Separate from the facts above.
- Our take: Track the version or do not track. An AI-visibility number without a model, a date and a sample size is a screenshot with ambition.
- What we’ve seen: Prompt sets written by the marketing team rarely match how shoppers actually ask. Refreshing the panel's prompts from real query data each quarter changes the results more than any content edit made in the same period.
- Where we disagree: Vendor dashboards report a single AI visibility score. Aggregating across models, plan tiers and personalization states into one number hides the variance that is the whole story.
- What this page adds: that answer variance has three named sources (routing, retrieval firing, personalization) and what a defensible panel has to log.
Reviewed by Martin Dejnicki, Director of SEO & AI Search. Facts verified 2026-09-13.