Anyone who assumes that the same prompt, the same model and the same settings always lead to the same answer is in for a surprise. That is the core of the study Same Prompt, Different Outcomes: Evaluating the Reproducibility of LLM-Based Data Analysis (February 2026). In that study's 480 runs across multiple models, temperatures and prompting strategies, identical configurations still produced divergent analyses and conclusions. The study argues that reproducibility must be measured explicitly, using repeated runs and evaluation of metrics such as completion, concordance, validity and consistency. The study can be found on arXiv.
Measuring reproducibility instead of assuming it
Multiple sources describe how variation persists even with tight settings. The study Reproducibility and Robustness of Large Language Models for Clinical Mobility Extraction (April 2026) quantified the reproducibility and robustness of LLMs in clinical text extraction through repeated runs — 13,200 inferences across multiple temperatures — using agreement as a statistical measure. It reports that reproducibility with identical prompts and text depends strongly on model, task and temperature, and that even at temperature 0 variation occurs. The authors underline that multiple runs and reporting of variability are therefore needed in care contexts. The full text is available on PMC.
The essay Randomness in Large Language Models: What Researchers Need to Know (July 2026) broadens this. It argues that LLM output should in practice be understood as draws from a distribution rather than fixed measurements, and describes sources of randomness including sampling, silent model updates and routing. It reports empirical evidence that temperature 0 does not remove all variability, and introduces reporting standards for prompt and model configurations so that replication and auditing of LLM-based research become possible. The piece is available via papers.cool.
Prompt and harness design as an experimental setting
Reproducibility lies not only in the model, but in the entire test harness. The preprint Prompt Design at Scale (July 2026) introduces VeyraBench and describes a harness that, together with its corpus generator and raw results, enables byte-identical reproduction in its experimental setup. It reports that variations in prompt format, number of instructions and context length have a systematic effect on instruction adherence and hallucinations, and that full disclosure of harness, data and results greatly increases the reproducibility of prompt experiments. The preprint can be read on arXiv.
The practical guide How can AppSec teams make AI findings consistent enough to use? (August 2026) describes how application security teams can make AI findings consistent enough to be usable, with an emphasis on a stable test harness, fixed prompt templates, versioning of prompts and evaluation sets, and explicit measurement of run-to-run variation and drift. The guide says these practices map to NIST and NIST CSF controls for configuration management and process consistency. The article is available on nhimg.org.
From a single prompt to a verifiable chain
Together these sources describe reproducibility as a chain of decisions that must be visible: which prompt version was used, with which model configuration, against which dataset and test cases, how many runs were carried out, how large the variation was, and which answers were ultimately accepted as trustworthy enough. For professionals with sensitive or high-trust information, this means that prompts and AI answers should be treated as experimental actions that should be repeatable: with fixed prompt templates, version control, repeated runs and statistical variability measurements.
Here lies the role of a verification console such as IamVera.ai. Vera does not build a new model, but is a verification layer that can make the chain around prompts and AI answers visible: which prompt versions, model settings and test cases led to which answers and how consistent those answers are over time. In this way it can help demonstrate that AI answers are reproducible enough for audit and decision-making, while the professional final judgement always remains with the user.