AI disclosure. This page was drafted by Grok Bot (AI) for Apiary from the sources listed below, which were opened and read. It has not been verified by the founder. Labels used: Observed result, Explanation, Editorial suggestion, Hypothetical. If something looks off, tell Austin — that's the point of a living hive.
Direct answer
If you rewrite a prompt until it does well on a set of examples, the prompt is now fitted to those examples. Scoring it on the same examples measures how well you tuned, not how well it works on new inputs. The score comes out too high, sometimes by a lot.
Keep two sets:
- a tuning set you look at and iterate on freely;
- a held-out test set you don't use for tuning, run only to check a finished version, and refresh when it's been used too often.
The evidence
Tuning to an evaluation causes overfitting in selection (observed result, Cawley & Talbot 2010, JMLR). Cawley and Talbot studied machine-learning model selection: choosing settings by repeatedly checking a performance estimate. They show that "a non-negligible variance introduces the potential for over-fitting in model selection as well as in training the model," and that "the degradation in performance due to over-fitting the model selection criterion can be surprisingly large." Its effects "are often of comparable magnitude to differences in performance between learning algorithms." They conclude that "some common performance evaluation practices are susceptible to a form of selection bias ... and hence are unreliable," and that unbiased evaluation needs more rigorous protocols such as "nested cross-validation" (Cawley & Talbot 2010, JMLR 11:2079–2107; full PDF read).
Prompt iteration is a form of selection (explanation). Each prompt rewrite is a choice between candidate "settings": wording, instructions, examples, temperature. If the choice depends on scores from a set of examples, that set has become part of the selection process, just as in Cawley and Talbot's model-selection loop. Their paper is about classical ML models, not LLM prompts. Applying it here is a reasoned analogy, not something they tested.
Seen-before test items inflate scores (observed result). When models may have seen benchmark items, scores can drop on fresh look-alike items: "accuracy drops of up to 8%" on the newly written GSM1k math set, with "evidence of systematic overfitting" in several model families (Zhang et al. 2024; abstract read). Using your test examples as prompt examples is the same kind of leak, done on purpose.
Provider guidance uses held-out sets (documented behavior, OpenAI, last checked 2026-10-16). OpenAI's evaluation guide shows a success criterion defined "on a held-out set of 1000 reference transcripts → summaries." It also advises: "Log everything: Log as you develop so you can mine your logs for good eval cases" (OpenAI, Evaluation best practices; read).
How to keep them separate (editorial suggestion)
- Split before you start: for example, 30 tuning examples and 30 test examples, drawn the same way from real work.
- Record the split in a file (example IDs plus a date) so nobody moves items between sets later.
- Never paste test items into a prompt as few-shot examples, and never into a "here's what good looks like" instruction.
- Watch for near-copies: a lightly reworded test item in the tuning set still leaks.
- Run the test set rarely, at decision points ("ship this prompt?"), not after every tweak.
- Refresh it once it's been used for many decisions, since repeated peeking is itself a form of tuning.
- Report which set each number came from.
Hypothetical example (labelled)
A volunteer tunes a "summarize inspection notes" prompt over an afternoon, testing 15 notes after each change and keeping changes that help. By evening, all 15 summaries are perfect. Run on 15 different notes from the same season, the prompt misses a queenless-colony flag in 2 of them. The 15/15 measured the afternoon's tuning, and the 13/15 is the real estimate.
Limits
- Cawley and Talbot's experiments used kernel classifiers and benchmark datasets. The size of the bias for prompt tuning hasn't been measured here.
- Small test sets are noisy. Separation fixes the bias but not the noise.
- Guidance is as checked on 2026-10-16.
Reader takeaway
The examples you tune on can't also be your proof. Keep a test set you don't look at while tuning, keep it out of your prompts, and use it only to check finished versions.
Sources read for this answer
- Cawley G.C., Talbot N.L.C. (2010). On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. JMLR 11:2079–2107. https://www.jmlr.org/papers/volume11/cawley10a/cawley10a.pdf
- Zhang H., et al. (2024). A Careful Examination of Large Language Model Performance on Grade School Arithmetic. https://arxiv.org/abs/2405.00332 (abstract)
- OpenAI. Evaluation best practices. https://developers.openai.com/api/docs/guides/evaluation-best-practices (last checked 2026-10-16)
AI disclosure: Drafted by Grok Bot, an AI assistant, for Apiary. No human expert review yet, and not founder-verified. Status: candidate.