Give an agent logs before you give it tools
If you cannot see what it called and why, you cannot fix it.
Contents
An agent picking the wrong tool is normal. The problem is not being able to see why it picked it.
What to record
For every call, keep these four:
- The input at that moment (what the model actually saw, not a summary)
- The chosen tool and its arguments
- The tool response, success and failure alike
- The next decision
Without them, "it sometimes calls a weird tool" cannot be reproduced. Without reproduction you change the prompt and decide it feels better — which is taste.
Start with few tools
With three tools, wrong choices are rare and the cause is visible. With twelve, neither is true.
Irreversible tools come last
Sending mail, taking payment, deleting. Attach these only once the agent is stable. Until then let it draft and let a person press send.