A demonstration uses the cleanest enquiry in the inbox. It has complete details and one obvious request. Your real inbox also contains forwarded threads, missing information and customers asking for three things.
Photo by Cht Gsml on Unsplash.
An AI evaluation set is a controlled collection of examples used to judge whether a workflow performs the task you need. Reflect the work and exceptions the team actually handles, with expected outcomes defined before reviewing results.
Describe the population first
Write down what can enter the workflow. Include source, format, language where relevant, request types and known exceptions. Exclude work the pilot is not authorised to handle.
For an illustrative classifier, the population might include website enquiries for three services and existing-client requests. It might exclude recruitment messages and information the pilot is not approved to process.
Select cases by category
Build a coverage checklist:
Straightforward examples of each intended category.
Missing or contradictory information.
Requests containing more than one intent.
Unusual formats genuinely received.
Cases requiring human judgement or refusal to classify.
Previous failures retained as regression checks.
Use authorised, appropriately prepared examples. Synthetic cases can test a boundary, but label them and avoid treating them as evidence of real-world frequency. A set dominated by invented edge cases will not describe normal workload proportions.
Agree the expected result
For each case, record the correct route, essential facts and unacceptable mistakes. Some tasks permit several good answers. Define what must remain true rather than demanding one exact sentence.
Ask another knowledgeable person to check ambiguous labels. Disagreement may reveal an unclear business rule. Resolve that before blaming the tool for choosing differently.
Keep evaluation separate from improvement
If you repeatedly rewrite instructions around the same examples, those examples become familiar design material. Keep some aside for a later check and add newly observed exceptions through a controlled process.
Record the set's version and limitations. A small collection can reveal failures without proving every future case will work. Increase coverage when the workflow expands to new inputs or decisions.
Related guidance: guide to mapping the work before changing it.
