Agents
How to evaluate an AI agent before putting it to work
By Julián Medina · Director of SourcingUp and creator of CommerceUp
Test an agent on representative tasks with acceptance criteria defined in advance. Include incomplete information, tool failures and situations that require a person’s input.
Build a set of test cases
Collect routine examples, known exceptions and requests outside the agent’s scope. Write down the acceptable result and any actions that must not happen for each case. This makes comparisons less dependent on the impression left by the latest test. Keep the data and configuration used as well.
Measure the work left for people
Track tasks completed correctly, errors, human interventions and rework. An agent may finish many tasks while leaving the team with too much to check. For example, when classifying customer inquiries, getting the category right matters, but so does recognizing when an ambiguous message needs human review.
Retest when the system changes
A change of model, instructions or integration can change the agent’s behavior. Rerun the relevant cases and add failures found in day-to-day use. This makes evaluation a way to preserve learning and clarify which tasks the agent can handle and where it needs support.
Put it into practice
- Define the expected result for each test case.
- Include tool failures and missing information.
- Record interventions and rework.
- Repeat the evaluation after significant changes.
The thinking behind this guide
A guide by Julián Medina. The scenarios are illustrative, not measured customer outcomes. My work at SourcingUp.