Stopping and retrying
This block explains how the loop ends and which failures are worth trying again.
The Northwind agent calls issue_refund while the payment service is down. The model tries again, gets the same error and tries again. Nothing guarantees it will stop.
Who says "enough"
Back to the phone. Your colleague keeps asking you to ring the payment office, and the office does not pick up. They will go on asking for as long as you keep answering. So the decision to stop is yours, and it has to be a rule you can count: five tries, then this goes to someone else.
A loop has two ways out.
The normal exit is a reply with no tool calls. On Anthropic's API it arrives with stop_reason set to end_turn. A reply with max_tokens was cut off, so do not treat it as an answer.
The forced exit is a budget that your code counts:
Budget | What it catches |
|---|---|
Model calls per run | Tool use that never ends |
Time per run | Runs that drag on |
Tokens or money per run | Runs that grow expensive |
Identical failures | One call failing the same way again |
Count budgets in the loop. A limit that exists only in the prompt is a request to the model.
A hanging tool is you stuck on hold. The loop is waiting, so no counter gets checked. Put a timeout on each tool call as well.
What the customer gets
When a budget fires, the customer gets a short, honest message and a person takes over with the history attached:
We could not finish your refund automatically. A colleague has your case and will reply in this chat.
No stack trace and no internal error text. If a refund already went through, say so, and read that fact from your payment records, because the model's text is not a record.
Busy, or no such number
Two things can happen when you ring someone. The line is busy, so you try again in a minute. Or the number does not exist, and ringing again gets you the same result forever.
Failures split the same way:
Failure | What to do |
|---|---|
429 rate limit | Busy. Wait, honor |
5xx on a model call, including Anthropic's 529 | Busy. Retry with backoff |
Connection error or timeout on a model call | Busy. Retry with backoff |
400, 401, 403 | No such number. Fix the request, key, permissions or spend limit |
429 from a spend or credit limit | No such number. Retries will not help: change the limit or wait for it to reset |
Refund rejected by a business rule | No such number. Return it to the model as a tool error |
The two kinds of 429 look the same from the status alone, so check the error code.
Waiting the right way
Backoff makes the pause grow after each attempt, up to a cap: one second, then two, then four.
Backoff alone has a flaw. If a thousand clients lost the line at the same moment, they all ring back at the same moment, and the line is busy again. Jitter adds randomness to each pause so they spread out.
The official Anthropic and OpenAI Python SDKs already retry twice by default on connection errors, 408, 409, 429 and 5xx. A retry loop of yours around them multiplies the attempts.
A timeout or a 5xx on a write is different: you cannot tell whether the refund happened. The next block deals with that.
Conclusion
- Your code ends the run, with budgets it counts itself.
- The customer gets an honest message and a person.
- Only failures that can clear by themselves are retried, with growing, randomised pauses.