Prompt tuning without evaluation is taste
The feeling that it got better usually comes from the last example you looked at.
Contents
Thirty cases is enough
Teams often try to build a perfect evaluation set and end up building none. What you actually need is thirty inputs that failed. Pulling them from production logs takes an afternoon.
- 20 that actually failed
- 5 near the boundary
- 5 completely ordinary (fixing hard cases does break easy ones)
Direction matters more than the score
"87% correct" on its own is not useful. What is useful is whether v3 beats v2, and that comes from running the same thirty twice.
$ ./eval --prompt v3 --cases cases.jsonl
passed 24 / 30 (v2: +3, newly broken: 1)"newly broken: 1" is the most important cell in that table. A total score hides it.