Azure AI · Copilot Studio · LLMs
TL;DR – AI agents are non-deterministic by design. Two settings called Temperature and Top P control how much creative randomness the model applies when generating a response. Crank them up and you get variety; dial them down and you get consistency. Understanding these levers – and knowing where to find them in Azure AI Foundry and Copilot Studio – is the practical fix for agents that give you a different answer every single time you ask the same question.
You have a Copilot agent deployed. It looks great in the demo. Then someone asks it the same question twice in the same day and gets two noticeably different answers. Not wrong answers, necessarily. Just… different. The wording is different. The structure is different. In one case it gives three steps, in another it gives five. Your stakeholder raises an eyebrow. You start wondering whether the thing is broken.
It is not broken. It is doing exactly what it was designed to do. The problem is that the defaults were not tuned for your use case. Let me explain why this happens and, more usefully, what you can actually do about it inside the Microsoft AI stack.
First: how an LLM actually generates a response
Before touching any settings it helps to have a rough mental model of what is happening under the hood. Large language models like the GPT-4o model family that powers Microsoft Copilot do not look up answers in a database. They do not run a rule engine. What they do is predict the next token – basically the next chunk of text – one piece at a time, given everything that came before it.
At each step the model produces a probability distribution across its entire vocabulary. Think of it as a massive ranked list of candidates for the next word, each with a score attached. “The answer is” might be followed by “yes” with a high score, “no” with a slightly lower score, “complicated” with a moderate score, and tens of thousands of other words trailing off toward near-zero probability.
The model does not automatically pick the highest-ranked word every time. That strategy, called greedy decoding, produces technically correct but often robotic and repetitive text. Instead, the model samples from that distribution. Which candidate it picks is probabilistic. Run the same prompt twice and you will likely land on different tokens at one or more steps, which cascades into a different sentence, a different paragraph, a different answer.
That is the source of the variability you are seeing. It is not a bug. It is the mechanism that makes the model feel natural and generative rather than mechanical. The question is whether the dial is set correctly for what you are trying to do.
Temperature: the most important dial you probably have not tuned
Temperature is the primary control for how much randomness gets applied during sampling. It is a multiplier that reshapes the probability distribution before the model picks its next token.
A high temperature (closer to 2.0) flattens the distribution, making lower-probability candidates more competitive. The model is more likely to take unexpected turns. Responses feel creative, varied, sometimes surprising – and sometimes inconsistent in a way that frustrates users expecting a reliable answer.
A low temperature (closer to 0.0) sharpens the distribution, concentrating probability mass on the top candidates. The model becomes more deterministic. Set it to exactly 0 and most implementations will give you the greedy result: the single highest-probability token at each step, every time. The response will be consistent, predictable, and potentially a bit bland.
In practice the sweet spot for most enterprise Copilot agents sits somewhere in the 0.2 to 0.5 range if you want reliable, repeatable answers. Anything above 0.8 and you should expect meaningful variability between runs. The Azure AI Foundry default is typically 1.0, which is fine for experimentation but higher than you probably want for a production support agent.
Top P: the other lever, and how it interacts with Temperature
Top P (also called nucleus sampling) is a companion setting that works differently but achieves similar ends. Rather than scaling the whole distribution, it defines a probability mass cutoff. The model considers only the smallest set of candidates whose combined probability adds up to P, and ignores everything else.
At Top P 1.0, the entire vocabulary is on the table. At Top P 0.9, only the tokens covering the top 90% of probability mass are considered. At Top P 0.1, you are looking at a very small nucleus of high-confidence candidates only.
Here is where it gets interesting. Temperature and Top P interact. A high Temperature with a low Top P is a strange combination – you are flattening the distribution and then narrowing the pool. A low Temperature with a high Top P mostly defers to Temperature because the distribution is already sharp. In general you should tune one and leave the other at a sensible default rather than trying to use both simultaneously. Microsoft’s own guidance for Azure OpenAI Service recommends altering Temperature or Top P but not both at once, which is good practical advice.
For most agent scenarios, set Temperature to your desired value and leave Top P at 0.95 or 1.0.
There is more to it than just those two settings
Temperature and Top P are the main controls, but they are not the whole story. A few other factors contribute to response variability:
System prompt wording. Vague or broad system prompts give the model enormous latitude to interpret the task. A tighter, more specific system prompt constrains the output space even before sampling comes into play. If your agent prompt says “be helpful with HR questions” you will get more variance than if it says “answer HR policy questions using only the provided knowledge base, in plain English, in three sentences or fewer.”
Context window content. If the agent pulls in retrieved documents or conversation history that varies between runs – even slightly – those differences feed into the generation and produce different outputs. Retrieval-augmented generation setups in Azure AI Search can surface different top-ranked chunks depending on query phrasing, which then flows into the response.
Model version. Microsoft periodically updates the underlying model versions for GPT-4o and other models in Azure OpenAI Service. Behaviour can shift between versions even with identical parameters. Pinning to a specific model deployment version in Azure AI Foundry is worth doing for production workloads where consistency matters.
Token limits. A low Max Tokens setting can truncate responses inconsistently depending on how long a given generation runs. This is not a temperature issue but it produces variability that can look like one.
Where to actually change these settings in the Microsoft AI stack
Let me be concrete about where these controls live, because that question comes up every time I run a workshop.
Azure AI Foundry (formerly Azure OpenAI Studio). In the Chat Playground you will find Temperature and Top P in the Configuration panel on the right-hand side. In the Deployments view, when you create or edit a deployment, you can set default parameters that apply to API calls unless overridden. For production agents calling the API directly, you pass temperature and top_p as parameters in the request body. These are first-class fields in the Azure OpenAI API.
Copilot Studio. In a published Copilot Studio agent, go to Settings > Generative AI and look for the Generative answers configuration. You will find a slider labelled something like “How creative should your responses be?” – this maps directly to Temperature. Selecting “More precise” moves Temperature lower; “More creative” moves it higher. There is not yet a numeric input in the Copilot Studio UI as of mid-2026, so you are working with broad bands rather than exact values. For fine-grained control you need to go upstream to the Azure AI Foundry model deployment that backs the agent, or use a custom connector to an Azure OpenAI endpoint where you control the parameters explicitly.
Azure AI Agent Service. If you are building agents programmatically via the Azure AI Agent Service (the managed agent runtime that landed in general availability in early 2025), you set these parameters on the model configuration object when creating or updating the agent. The Azure AI SDK for Python and the REST API both expose temperature and top_p at the model level.
Practical solutions: what to actually do
Here is the practical checklist I run through when someone tells me their agent gives inconsistent answers:
1. Lower Temperature first. Start at 0.3 and test. If responses feel too rigid or start repeating the same phrasing verbatim, nudge up to 0.5. For factual Q&A agents, FAQ bots, and policy assistants, 0.2 to 0.4 is usually the right range. For agents doing summarisation or creative tasks, 0.6 to 0.8 might be more appropriate.
2. Tighten your system prompt. Add explicit constraints on format, length, and source. “Answer only based on the provided documents. If the answer is not in the documents, say so. Respond in plain English in no more than four sentences.” This reduces the output space dramatically and often has more impact than tweaking Temperature alone.
3. Pin your model deployment version. In Azure AI Foundry, when you create a deployment choose a specific model version rather than “auto-update to latest.” Set a reminder to review the version quarterly. Unexpected behaviour changes after an automatic model update are a support ticket waiting to happen.
4. Check your retrieval layer. If the agent uses Azure AI Search or a similar RAG setup, make sure your top-K retrieval is stable and that chunk sizes are consistent. Erratic retrieval produces erratic generation even with a low Temperature.
5. Use evaluation runs in Azure AI Foundry. The Evaluation feature lets you run a fixed set of test questions against your agent and score the responses automatically. Run this after any parameter change. It is the fastest way to know whether your Temperature tweak actually improved consistency or just changed what kind of inconsistency you are getting.
6. Consider structured output mode. For agents where you need a very specific response format – JSON, a numbered list, a fixed template – Azure OpenAI’s structured outputs feature forces the model to conform to a JSON schema. It does not change the sampling behaviour directly, but it eliminates a large class of format-level variability entirely.
When variability is actually fine
One thing worth saying before wrapping up: not all response variability is a problem. A creative writing assistant, a brainstorming agent, or a tool that generates multiple versions of a piece of marketing copy should produce different outputs each time. That is the whole point. The mistake is applying the same configuration to a compliance FAQ bot that you would to an ideation tool.
The question to ask is: does the user of this agent expect the same input to produce a reliably consistent output? If yes, lower Temperature and tighten your prompt. If the use case tolerates or benefits from variation, leave it alone and save yourself the tuning time.
The short version
Your AI agent gives different answers because it is sampling from a probability distribution, not looking up a fixed answer. Temperature and Top P control how wide that sampling net is cast. In the Microsoft AI stack – Azure AI Foundry, Copilot Studio, Azure AI Agent Service – these are configurable parameters. Lower Temperature toward 0.3 for consistency, tighten your system prompt, pin your model version, and validate with evaluation runs. That combination will get most production agents to a point where the variability is acceptable and explainable to your stakeholders.
