Designing Determinism in Artificial Intelligence Systems

We cannot completely eliminate the probabilistic nature of LLMs, but we can design systems that make business decisions sufficiently predictable, traceable, and reliable.

Large language models (LLMs) operate probabilistically, so the same input does not necessarily produce the same response every time. In some situations, that variability is useful, especially when we want to generate content or interpret information. But when we begin to incorporate these models into business processes, we encounter situations where we need much more predictable behaviour.

Consider an application that analyses technical queries and determines whether a product configuration is viable. We can submit the same query ten times and receive nine affirmative answers, while on the tenth the model interprets a constraint differently. If that decision affects a quotation, manufacturing, or a commitment to a customer, we have a problem that cannot be solved simply by accepting that models work this way. We need to reduce that variability to a level the process can tolerate.

One of the first measures is usually to set the model temperature to zero, in an attempt to reduce randomness during generation. That can help, although it does not guarantee identical results, because factors related to inference infrastructure, numerical precision, or model versions also come into play. There is, however, another element over which we have far greater control and that I believe deserves particular attention: the context we provide to the LLM.

When we work with RAG architectures, the same query can retrieve different documents depending on how the search is performed, the versions available, or even the order in which the chunks are added. If we are evaluating compatibility between two components and one run retrieves their technical specifications while another also includes a commercial exception, the model is no longer working with exactly the same information. That is why, before modifying prompts or changing models, it is worth checking whether we are really keeping the context stable and whether we are using the right sources.

Nor does it seem reasonable to solve this by providing all available documentation. Adding redundant information, conditions that do not apply to the case, or contradictory documents can introduce new ambiguities. In a technical validation, for example, it is probably more useful to retrieve the specific characteristics of the components, their constraints, and the applicable exceptions than to load an entire collection of data sheets. To achieve this, we need to control the authorised sources, the versions of the information, and the retrieval criteria, ensuring that the model has what it needs without adding elements that could alter its interpretation.

An agent is not always the answer

Today, platforms such as Claude and OpenAI make it much easier to create agents capable of consulting information, using tools, and automating processes. They are solutions that allow us to move very quickly and can be perfectly valid for many scenarios. But when we need consistent decisions, traceability, and assurances about the outcome, using them directly may not be enough. An agent can interpret a request, retrieve documentation, and execute actions, but it can also vary the sequence of steps, select different tools, or apply different criteria between runs. Completing a task does not necessarily mean we have control over how it reached that result.

For certain processes, it may be preferable to build an orchestration layer that controls which information is used, which operations must be executed, and which validations are mandatory. This is not about giving up on agents or building from scratch capabilities that these platforms already provide. It is about deciding which responsibilities we can delegate and which we need to keep under our control.

Imagine, for example, a system that receives commercial queries about product configurations. We can use an LLM to interpret the request and identify references, dimensions, finishes, and special conditions, transforming that information into structured data that we then validate. From there, a service retrieves specifications and constraints from authorised, versioned sources, while a rules engine checks compatibility and determines whether the configuration is viable. Finally, the model can use the result to draft an understandable explanation, without changing the decision that was made. The orchestration controls the process and can stop it when information is missing or contradictions appear. It also records the data and rules used, so we can reconstruct what happened.

This separation does not eliminate every risk, because an incorrect interpretation in the first stage can affect the ones that follow. But it allows us to introduce checks and avoid making every decision depend exclusively on the response generated by the model.

Determinism as a system property

This is where I think it is worth changing the perspective slightly. Instead of trying to make every component deterministic, we can work to make the system’s final behaviour sufficiently predictable, while understanding that this consistency must be measurable rather than merely expected.

We can run the same query repeatedly under controlled conditions and check whether we receive identical responses, whether we reach the same decisions, and whether those decisions are correct. These are different questions: a system can generate different explanations while maintaining consistent operational outcomes, just as it can repeat an incorrect answer in every run. We also need traceability to investigate discrepancies, preserving not only the prompt and the response, but also the model versions, the retrieved information, and the rules applied.

Not every process needs the same guarantees. We can accept different wording in a commercial proposal, but we should hardly accept arbitrary variations in prices, compatibility assessments, or approvals. That is why, when designing AI solutions, I believe it is worth investing as much effort in controlling context, defining the architecture, and measuring consistency as we do in selecting the model we will use.

We are not going to eliminate the probabilistic nature of LLMs completely, but we can design systems that reduce its impact on business decisions. And that is probably one of the most important differences between getting an automation to work and being able to trust it once it becomes part of our everyday processes.