How many repeats a prompt reliability test needs depends on effect size, not a rule of thumb

There is no fixed rule of thumb such as five repeats; peer-reviewed research shows that the required size depends on the smallest relevant effect size, the number of test items, the confidence level and the chosen measure. Fix that design in advance and test reliability and factual correctness separately, because consistency does not prove correctness.

A framework published on 12 April 2026 in the Journal of Business Analytics treats prompt reproducibility as a statistical design problem rather than an arbitrary choice. The authors determine the sample size before data collection, run replicates per model condition and analyse agreement with several reliability measures. In their binary financial-classification case, using 90 per cent confidence and a 10 per cent margin of error, the researchers arrived at 1,317 test items and five repeats per model under the most conservative measure.

Those five repeats are therefore not a universal standard, but the result of that specific case. A methodological preprint on Research Square supports this with Monte Carlo power analyses: five iterations provide sufficient discriminating power only for large instability, medium effects require at least fifteen iterations, and small effects remain hard to demonstrate below forty iterations.

How many repeats per prompt are needed to underpin a reliability test?

That depends on the smallest difference you still consider relevant. The smaller the variation you want to be able to demonstrate reliably, the more repeats you need. The number required thereby depends on the chosen metric and the analysis design.

Our analysis, based on these studies, distinguishes three planning levels. First, the number of test items is the sample from the population of documents, questions or scenarios on which you want to deploy the model. The second is the number of repeats per item, needed to estimate the stochastic variation of the model output. Our analysis treats prompt, model and — where relevant — temperature conditions as separate test conditions that must be planned separately; this applies equally to models from providers such as OpenAI and Anthropic. Fix in advance:

• a primary reliability measure;
• the minimum relevant effect size;
• the confidence level and the margin of error;
• the desired discriminating power (power).

The Frontiers in Education study treats repeated outputs within the same item as non-independent measurements and analyses iterations within model-application combinations. In our assessment, treating repeats as separate independent observations can artificially inflate the effective sample size and overestimate precision. Analyse repeats as nested measurements within a model-item combination.

Why is a consistent AI model not yet a correct AI model?

Because reliability and validity measure two different things. A model can give exactly the same label on repetition and yet systematically predict the wrong outcome. High consistency therefore says nothing about factual correctness.

The financial-classification study makes this distinction explicit: models gave consistently the same labels without predicting the external outcome well. In the grant-review study, prompt conditions shifted the scores, while the correspondence between LLM-generated scores and human reviewers remained at or below chance level. A stable answer that is consistently wrong remains wrong. That is why you must report reproducibility and validity separately and test the output against an independent reference. See also why a predictive formula is not yet proof of safety.

What does this mean for directors, lawyers and CISOs who must account for AI output?

A one-off reproducibility test within a single fixed prompt-model combination shows only the properties of that configuration; it does not automatically say anything about stability after later changes. As soon as the prompt, model, provider, temperature or a connected tool changes, the earlier test does not automatically count as evidence for the new configuration. This ties in with why a model change can overturn every earlier assumption.

Research in Political Analysis reports that the same prompt can yield different variances and relationships over time. Our analysis is therefore that a CISO cannot rely on a fixed snapshot; our recommendation is to include periodic re-evaluation as a recurring control in the management process and to record in the deployment policy that a model version change triggers a new measurement round before output is used again. Because consistency does not prove validity, a lawyer adopting an AI outcome should not treat the model's repeated agreement with itself as sufficient evidence; our recommendation is to record in the file an independent check against the actual outcome, in line with testable requirements for human oversight of AI decisions. Because reconstructing the basis of a conclusion is harder when the design was not recorded, fix the statistical plan—including the measure, margin of error and power target—before the test. More context can be found in the topic hub on AI verification and the reliability of model output.

Which test protocol records reproducibility in a verifiable way?

A protocol that fixes the design in advance and treats every relevant change as grounds for a new measurement round. It combines a predefined sample design with explicit recording of the configuration and an independent validity check.

1. Define the intended task population and draw a stratified test set with normal, rare and borderline cases.
2. Determine the number of items via a predefined reliability or power target, with a recorded measure, margin of error and confidence level.
3. Determine the number of repeats on the basis of the smallest variation you still consider relevant.
4. Analyse repeats as nested measurements; do not count them as separate observations.
5. Report confidence intervals and the proportion of missing or invalid outputs.
6. Test the factual correctness against an independent reference, separately from the consistency.
7. Repeat the test after any prompt, model, provider, temperature or tool change that could affect the output, or otherwise document why a new round is unnecessary.
8. Keep the prompt, model version, settings, timestamp, input hash, raw output and the reviewer decision.

Anyone who neglects this reports a figure whose validity may remain limited to the tested configuration and which cannot simply be extrapolated to a changed configuration. In our assessment, this creates a significant risk that regulators and internal auditors will be unable to verify whether the conclusion applies beyond the tested configuration.

Sources: The article draws on peer-reviewed research in the Journal of Business Analytics, Frontiers in Education and Political Analysis and on a methodological preprint on Research Square.