The new version excels on the demonstration example. Someone declares it better. Nobody checks the awkward requests that caused problems with the current version.
Compare versions using the same representative cases, stable criteria and a record of improvements and regressions. Decide release conditions before reviewing results so a favourite example cannot carry the decision.
Control the comparison
Record changed instructions, examples, sources, available model details and downstream rules. Avoid changing unrelated components together where practical.
Use the same inputs and source versions. If a business source changed, make that explicit rather than describing the result as a pure model improvement. Record supplier information you cannot inspect rather than assuming it remained constant.
Score behaviours separately
For an illustrative enquiry workflow, compare:
Correct routing by request type.
Preservation of essential facts.
Unsupported claims or commitments.
Appropriate escalation of uncertain cases.
Human correction and review effort.
Completion within permitted boundaries.
Do not let an average conceal consequential failures. Record the case counts and categories behind the result. A gain on common requests may come with a regression on a rare but important exception.
Reduce reviewer bias
Where practical, ask reviewers to assess outputs without knowing which version produced them. Supply the same rubric and source evidence.
Discuss disagreements and record reasons for final judgements. A preference for shorter prose should not override a requirement to preserve conditions.
If repeated attempts produce different results and that variation affects the decision, repeat relevant cases and report variation. Do not select the best run and discard the rest.
Release with explicit limits
The decision may be release, revise, retain the current version or restrict the candidate to a narrower task. Record limitations and operating checks required afterwards.
Keep unfamiliar cases for later evaluation. Repeatedly improving against the same examples can make performance look stronger than it is on new work. Add newly observed failures through a controlled update rather than rewriting history.
Related guidance: guide to mapping the work before changing it.
