Why does AI give Different Answers to the same question?

AI gives different answers to the same question because it does not retrieve a stored reply. It writes each answer from scratch, one small piece at a time, choosing among several likely options with a little chance built in. Add the settings, the history of your chat and the model version behind it, and two identical questions rarely follow an identical path.

This article explains how that choosing works, what temperature does, and why even the most cautious setting does not give the same answer every time. It then separates harmless variation from variation that causes real problems, and sets out practical ways to get steadier results. It is written for owners and managers who have asked a tool the same thing twice, got two different replies, and wondered which one to trust.

Key takeaways

• A large language model predicts likely next words and picks among them with some randomness. Variation is part of how it works, not a sign that your account is broken.
• Temperature controls how much chance goes into that choice. Lower is steadier, higher is more varied, and most chat apps do not let you change it.
• Zero is not a guarantee. A technical write-up that ran one model 1,000 times at temperature zero got 80 different completions, and a major supplier’s developer guide says its seed setting gives outputs that are only mostly identical.
• Wording, chat history, saved memory, custom instructions and the model version all change the reply, so two colleagues asking the same words can see different results.
• A different wording is usually harmless. A different fact, figure or decision is the problem, and that is where you need a source, a template or a person.
• The best fixes are practical: a fixed format, worked examples, source material in the request, one task per request and a test on realistic inputs before anyone relies on the tool.

How does an AI Model decide what to say?

A large language model does not hold a list of answers. It holds patterns learned from a vast amount of text, and it uses them to work out which word, or part of a word, is most likely to come next. It then adds that piece to the reply and repeats the process until the answer is finished.

At each step the model produces a spread of possible next words with a likelihood attached to each. “The capital of France is” has one overwhelmingly likely continuation. “A good opening line for this email is” has dozens that are all reasonable. The tool then samples from that spread instead of always taking the top choice. If it always took the top choice, replies would be repetitive and often stilted.

Because the choice is repeated hundreds of times in a reply, small differences early on grow. A single different word near the start sends the rest of the answer down another path. Two replies can start identically, diverge at one word and end with different structure, emphasis and sometimes different conclusions.

This is also why a spreadsheet and an AI tool feel so different in use. A spreadsheet formula gives the same output for the same input every time. A language model gives a plausible output for the same input, and plausible has more than one form.

What is temperature, and does it change the Answers?

Temperature is the setting that controls how much chance goes into the choice described above. At a low temperature the model leans hard towards the most likely word at each step, so replies are steadier and more predictable. At a higher temperature less likely words get through more often, so replies vary more and tend to read as more creative.

Two practical points follow.

Most people cannot change it. Temperature is a setting that developers use when they build with a model through its API. In a standard chat app it is normally fixed by the supplier and not shown. If you work in a chat window, you are using whatever the supplier chose.

It is not the only source of variation. Even if temperature were fixed, your wording, the earlier messages, attached files and saved preferences would still change the reply. Temperature is one dial on a machine with several.

For business use, the sensible rule is simple. Where you need repeatable output, such as sorting, labelling, extracting figures or following a fixed template, a lower temperature suits the job. Where you want options, such as headline ideas or alternative phrasing, variation is the feature you are after.

Why does AI give Different Answers even at temperature zero?

Many people set the temperature to zero and expect identical replies. It improves consistency a great deal, but it does not deliver it with certainty, and the people who build these systems say so.

A technical write-up from an AI research lab tested this directly. It asked one model for 1,000 completions of the same prompt at temperature zero. It got 80 unique completions, the most common of which appeared 78 times. The first 102 tokens, which are the small word-pieces a model works in, were the same in every run. At the 103rd they split: 992 runs continued one way and eight another. Once the path split, the rest of the answers differed.

The explanation in that write-up is not the one most people expect. It is not mainly about chance in the model. It is about how the servers do their arithmetic. The result of a calculation can depend on how many requests the server is handling at the same moment, because the order in which numbers are combined changes with the batch size. Your request is bundled with other people’s, the load varies from second to second, and tiny numerical differences can tip a close call between two words. The researchers showed that when the calculation was made independent of batch size, all 1,000 completions came out identical.

The supplier side tells the same story. A developer cookbook from another major supplier describes its seed setting as a way to get “mostly deterministic” outputs, and states that determinism is not guaranteed because of the inherent non-determinism of its models. It also notes that the identifier for the exact model and infrastructure combination can change, which is another reason a repeated request may not match.

The lesson for a business is not that temperature zero is useless. It is that “set it to zero” is not a control you can put in a procedure and then stop checking.

Does the way you word a Prompt change the Answer?

Yes, and often more than any setting does. A prompt is the whole input the model sees, and the model has no way of knowing what you meant, only what you wrote.

Three things matter most.

Small wording changes are not small to a model. “Summarise this contract” and “List the risks in this contract” ask for different things. “Give me three options” and “Recommend one” lead to different shapes of reply. A request that leaves room for interpretation will be interpreted differently on different runs.

Missing detail gets filled in. If you do not state the audience, length, tone or format, the model chooses for you and may choose differently next time. A request that says who the reader is, how long the reply should be and what the layout is leaves far fewer ways to vary.

Everything earlier in the chat counts. The model reads the whole conversation so far, up to the limit of its context window. A follow-up question asked in a long chat is answered in light of everything above it, including a half-correct reply you did not correct. The same question asked in a fresh chat starts without that baggage. If you want to understand where that limit sits and what happens beyond it, our article on what a context window is covers it in plain terms.

This is why the briefing habit matters. Our guide to how to brief an AI tool like a new hire sets out the details a good request should carry, and it is the cheapest consistency fix on this page.

Why does ChatGPT give a colleague a Different Answer?

You and a colleague type the same sentence into the same product and get different replies. Variation from sampling explains some of it, but several other things differ between two accounts.

• Saved memory. Many chat tools can remember facts about you from earlier conversations and use them in later ones. Your tool may know your sector, your preferred spelling and your role. Your colleague’s may know something else.
• Custom instructions. If one of you has set standing custom instructions, such as “answer briefly” or “write for a non-technical reader”, every reply is shaped by them before you type a word.
• Different plans and models. Accounts on different plans can be given different models or different limits. Two people may not be talking to the same model.
• Connected sources. One account may have search, files or a company knowledge base switched on, and the other may not.
• Different chat history. As above, the earlier part of the conversation changes how the next question is read.

For a team, this is the argument for shared setup rather than individual improvisation. Agreed instructions, a shared prompt library and a short note on which tool and plan to use remove most of the account-level differences. Your AI use policy is the right place to record which tools are approved and how they should be configured.

Can an AI Model change without you noticing?

Yes. The model behind a tool name is not a fixed object. Suppliers update, replace and retire models, adjust the system instructions that sit around them, and change the infrastructure that serves them. Some tool names point at whatever the current model is, so the same product can behave differently in March and in September without you doing anything.

The developer cookbook mentioned earlier makes the same point in technical terms: the fingerprint that identifies the combination of model weights and infrastructure changes when the supplier changes its setup, and outputs from different fingerprints may not match.

For a business, this is the reason to think about drift. A workflow that gave you reliable results last quarter can slowly change character after an update, and nobody is told. The protection is dull but effective: record the tool and model version where you can see it, keep a small set of test cases, and re-run them after any announced change or any time results seem off.

Is a Different Answer the same as a wrong Answer?

No, and the distinction saves a lot of worry.

Different wording, same substance. Two summaries of one document may use different sentences, order the points differently and still contain the same facts. This is ordinary and harmless. A reader would see the same thing from two human writers.

Different substance. One reply says a clause allows termination on 30 days’ notice and another says 60. One classifies an invoice as travel and another as training. One recommends option A and another option B. At least one of those is wrong, and you cannot tell which from the replies alone.

The second case overlaps with hallucination, where a model produces something plausible that is not true. A change between runs is often the first visible sign of it, because a fact the model actually knows tends to come out the same each time, while an invented detail varies. Our piece on how to reduce AI hallucinations covers the causes and the checking routine in detail.

So when you see two different replies, do not ask which one sounds better. Ask which parts changed, and check those parts against a source.

When do inconsistent Answers become a Business problem?

It depends on what the reply is used for. Variation costs nothing when you want ideas and a great deal when you want a decision.

The pattern is that the more an output feeds a decision, a record or a customer, the more consistency you need. For the high rows, the sensible design is not “get the AI to be consistent”. It is to take the decision out of the AI’s hands, or to make sure a person reviews each output before it is used. Our guide on how to check AI output before it reaches a customer describes how to set that review up without making it a bottleneck.

There is a governance point too. If a tool influences hiring, credit or similar decisions about people, inconsistent results are not just inefficient. They are difficult to defend if someone asks why two similar people were treated differently. That is a good reason to keep a record of what was asked and what came back.

How do you get more consistent Answers from AI?

You cannot remove the variation, but you can shrink it. These are the steps that make the most difference, roughly in order of effort.

1. Say exactly what you want. State the task, the audience, the length and the format. Vague requests produce varied results because the model fills the gaps differently each time.

2. Fix the output format. Ask for a table, a numbered list or a set of named fields. One major supplier’s developer guide on consistency puts specifying the output format first among its techniques, and it also recommends giving examples of the output you want, which it says is more effective than abstract instructions.

3. Show worked examples. Include two or three examples of a good answer to a similar input. The model follows a pattern it can see far more reliably than a description of one.

4. Put the source material in the request. If the reply should come from your policy, price list or contract, attach it or paste the relevant section and tell the tool to use only that. This is the idea behind grounding, and when it is built into a system it is usually done with retrieval. The comparison of RAG vs fine-tuning explains when each approach fits.

5. Give one task per request. The same developer guide suggests breaking complex work into smaller subtasks, with each step covering one job. Fewer decisions per step means fewer places to diverge.

6. Use a low temperature where you can. If the tool is built on the API, set it low for extraction, labelling and templated writing. Do not rely on it alone.

7. Save what works. Keep tested requests in a shared place, in a prompt library, and use saved custom instructions for standing preferences. A tested request used by everyone beats a different improvised request from each person.

8. Keep a person checking the parts that matter. For figures, names, dates and commitments, make it a rule that a person checks the output before it goes out. Consistency settings reduce the problem, and review catches what remains.

Should you ask the AI twice and compare the Answers?

It is a useful habit, with limits. Published research on a technique called self-consistency found that when a model is asked to reason through a problem several times and the most common final answer is taken, accuracy improved on arithmetic and reasoning tests, with a gain of 17.9% reported on one set of maths word problems. The idea behind it is simple: a correct reasoning path is more likely to be reached repeatedly than a wrong one.

You can use a lighter version at work:

• Run an important question two or three times in separate chats.
• If the substance matches, your confidence goes up a little.
• If the substance differs, treat that as a flag and check the point against a source.

Keep the limits in view. The research covered structured problems with clear right answers, not open business judgements. Two replies can agree and both be wrong, especially when both lean on the same missing fact. Agreement is a signal, not a proof, so it does not replace the source check.

How do you test if an AI Workflow is consistent enough?

Before a tool does the same job every day, test it on purpose. This takes an afternoon and prevents a lot of later surprises.

1. Collect real inputs. Gather 20 to 30 examples of the actual material the workflow will handle, including awkward ones. Remove personal data you should not be pasting into a tool.
2. Write down what a good result looks like. For each input, note the facts and the format that must be right. Decide in advance what counts as a pass.
3. Run each input several times. Five runs each is a reasonable start. Use fresh chats, so earlier replies do not influence later ones.
4. Compare substance, not wording. Mark any run where a fact, figure or decision changed.
5. Set a threshold. For low-risk work, you might accept a few differences. For work that touches customers or records, accept none without a human check.
6. Record the tool, version and date. Then you know what you tested.
7. Re-run after changes. Repeat the test when the model, the tool or the request changes, and every few months regardless.

This is the same discipline as the one in a well-scoped AI workflow, where the task, the inputs and the success measure are written down before anything is built. It is also one of the things an AI audit looks at: which tools your team already relies on, how they were checked, and who notices if their behaviour changes.

What should your team know about inconsistent AI?

A short briefing does more than a long policy. Cover these points:

• The first answer is a draft. It is not a record of what the tool “thinks”. Asking again can legitimately give a different reply.
• Do not treat a screenshot as proof. A reply that appeared once may not appear again, so it is a poor basis for telling a customer something.
• Check facts, not phrasing. Compare figures, names, dates and recommendations when two replies differ.
• Use the shared setup. Use the approved tool, the agreed instructions and the saved requests, so results are comparable between colleagues.
• Write down the tool and date for important work. If a decision relied on an AI reply, note which tool produced it and when.
• Say so when something looks off. A reply that contradicts last week’s belongs in a shared note, not in one person’s memory.

If you want the team to practise this on real examples, the art of prompting training day covers wording requests, setting formats and checking outputs, using your own tools and documents.

What follows

Pick one task your team gives to an AI tool every week. Run the same request five times in fresh chats and compare the substance of the replies. If the facts hold, fix the format and the examples and save the request where everyone can use it. If the facts change, add the source material to the request, or move the decision to a person. Then write down the tool and version you tested, and run the check again after the next update.

Sources

• AI research lab technical blog, “Defeating Nondeterminism in LLM Inference”: ran 1,000 completions of one prompt at temperature zero and found 80 unique results, the first divergence at the 103rd token, traced the cause to server batch size changing the order of calculations, and showed all 1,000 completions identical once the calculation was made independent of batch size.
• Major AI supplier developer cookbook, “How to make your completions outputs reproducible with the new seed parameter”: describes a seed setting that gives “mostly deterministic” outputs, explains the system fingerprint that identifies the model and infrastructure combination, and states that determinism is not guaranteed because of the inherent non-determinism of its models.
• Major AI supplier developer guide, “Increase output consistency”: recommends specifying the output format, constraining with examples, using retrieval to keep replies grounded in a fixed set of information and chaining prompts so each covers one subtask.
• Wang et al., “Self-Consistency Improves Chain of Thought Reasoning in Language Models” (ICLR 2023): proposes sampling several reasoning paths and taking the most consistent answer, reporting gains of 17.9% on GSM8K, 11.0% on SVAMP, 12.2% on AQuA, 6.4% on StrategyQA and 3.9% on ARC-challenge over standard chain-of-thought prompting.