Measurement guide

Why AI answers change between runs

Generated answers are variable by design. The question, context, retrieved sources, model configuration and product updates can all change the response, even when the visible prompt looks identical.

4 min read

Brand illustration for why AI answers change, with abstract interface shapes and no text.

Key takeaways

  • Models select from possible continuations. Equivalent inputs can still produce different wording.
  • Wording, history, location, language and personalisation can change what is treated as relevant.
  • Search-enabled answers may retrieve different pages as indexes and availability change.
  • Provider updates and API settings can change conditions that a consumer interface does not expose.
  • Report appearance rates and stability. Do not treat one response as a permanent position.
Contents

Why the same question can produce a different answer

  • Models select from possible continuations rather than retrieving one fixed answer. Even equivalent inputs can produce different wording or selections.
  • Small changes in wording, conversation history, location, language or personalisation can change what the system considers relevant.
  • Search-enabled answers may retrieve different pages as indexes, page content, ranking signals and live availability change.
  • Providers update models, system instructions, tools and serving infrastructure. The same visible question can therefore run under different conditions.
  • Temperature, seed, reasoning effort and other controls can affect API output. Consumer interfaces may not expose or fix these settings.
A hub diagram of five reasons answers change: continuations, context, retrieved pages, provider updates and API controls.
The visible prompt can stay the same while continuations, context, retrieved pages, updates and API controls change.

What to record before you interpret a change

  1. Keep the exact question and intent classification.
  2. Record system, interface, model when visible, account state, market, language and date.
  3. Run repeated observations for important questions.
  4. Preserve complete answer evidence and visible citations.
  5. Report appearance rates and stability instead of one permanent position.
  6. Version the prompt panel and mark methodology changes before comparing periods.
Five steps for recording a question, the system, repeated runs, citations and stability.
Repeated observations and complete evidence are what separate ordinary variability from a trend.

API controls help, but do not remove uncertainty

For API experiments, pin the model version and keep inputs and parameters fixed when supported. OpenAI documentation notes that model outputs are variable by nature. Even best-effort deterministic sampling is not a guarantee. Consumer AI products also include retrieval, interface and account context that an external measurement system may not control.

How to report a change

  • Do report sample size, repetitions, dates and missing observations.
  • Do not combine unlike systems or markets into one unexplained score.
  • Do not attribute a change to your work without a comparison design.
Two panels contrasting what to report, such as sample size and dates, with what not to claim, such as one answer as a rank.
A different answer is not, by itself, evidence that a brand lost a ranking.

Frequently asked questions

Why does ChatGPT give a different answer to the same question?

Generated output is variable by nature, and the surrounding context, available sources, model configuration and provider updates may also differ. Identical visible wording does not guarantee identical execution conditions.

Does a different answer mean a brand lost its ranking?

Not necessarily. One changed response may be ordinary answer variability. A meaningful trend requires repeated observations under comparable conditions.

How many times should a prompt be repeated?

There is no universal number. Use more repeats when the decision is important or the answer is unstable, and publish the chosen repetition rule with the result.

Can temperature zero guarantee the same API output?

No. Lower randomness may improve consistency, but providers document that determinism is not guaranteed and backend or model changes can still affect output.

Sources

The variability claim for API output is grounded in OpenAI documentation on model behaviour, versioning and reproducible outputs. Consumer products add retrieval and interface context those pages do not control.

  1. OpenAI API compatibility and model versioning
  2. OpenAI: reproducible outputs with the seed parameter

Start a free AI visibility audit

Open ViziGEO and review a documented sample of questions, mentions, recommendations and citations. The audit records what was observed. It does not guarantee a mention, a citation or a recommendation.