An AI API timeout is an unknown outcome, not a failed action
· agents, engineering, reliability, analysis
How to separate generation retries from tool execution, recover after an interrupted stream, and test duplicate effects before shipping an agent.
An agent asks a service to create a support ticket. The service creates it, but the connection closes before the acknowledgement reaches the agent. The agent sees an error, repeats the request, and creates a second ticket.
The model may have followed its instructions perfectly. The application lost the distinction between a failed connection and a failed action.
The same ambiguity appears one step earlier in an AI API call. A generation can begin or finish upstream while the client receives no complete result. Repeating the call can create another billable generation. That does not, by itself, prove that one request was billed twice. It may mean the application submitted two independent requests for one intended operation.
Three events that need separate records
For an agent that can change external state, keep three questions separate:
| Boundary | What success establishes | What it does not establish |
|---|---|---|
| Model response | A complete model result reached the application | A proposed tool action actually happened |
| Tool execution | The destination accepted or completed the action | The user received the acknowledgement |
| User acknowledgement | The application reported the result | Every retry or side effect was accounted for |
A single green “agent succeeded” badge cannot explain which boundary failed. An operation identifier should connect the records without making them the same record. A generation attempt may have its own request ID; a business action should retain a stable operation ID across recovery attempts.
For example, create-ticket:case-123 identifies the user's intent. It should not be regenerated merely because a new model response proposed the action again. A fresh operation ID on every retry defeats deduplication.
A timeout belongs in the unknown state
HTTP already makes this distinction. RFC 9110, section 9.2.2 says automatic retries of non-idempotent requests require knowledge that the operation is effectively idempotent or that the original request was never applied. Receiving no response is not that knowledge.
A useful application state sequence is prepared → submitted → confirmed. If acknowledgement is lost after submission, transition to outcome_unknown. Treat that as recoverable work with its own path, rather than translating it immediately into “try again.”
Recovery might query the destination by a stable operation ID, retrieve an existing result, or ask a person to resolve an ambiguous action. If the destination offers neither deduplication nor a way to identify the prior action, the application cannot manufacture an exactly-once guarantee by retrying more carefully.
Idempotency is a contract at the destination
An arbitrary Idempotency-Key header is not a universal solution. The server must implement and document its semantics. As one concrete example, Stripe's API documents how it saves and reuses a result for a key, checks parameters, and handles key retention. Those are properties of that API, not properties automatically inherited by an AI gateway.
For a tool service you control, define the key's scope, bind it to a validated request, serialize concurrent attempts, and retain a durable result reference. Do not store prompts or private payloads in an unrestricted deduplication log. If the business write and deduplication record are committed separately, test the crash between them: recording success too early loses work; recording it too late can duplicate work.
A model-generated tool-call ID is useful for tracing a particular response. It is not necessarily a stable business-operation ID across a regenerated response. The application needs to own that relationship.
Streaming makes partial success visible
A stream that delivered useful text and then disconnected is different from a call that never connected. Replacing it with a second generation can produce contradictory continuations. Concatenating both attempts can create an answer that no single model run produced.
Keep partial output marked as incomplete. Do not execute partially received tool arguments. When restarting a generation, keep its output separate from the previous attempt, and retain the relationship for diagnosis. If a completed tool action already occurred, pass its verified result into recovery instead of asking the agent to perform it again.
Retry policy also needs one owner. An application, SDK and gateway can each retry without the others knowing. With three layers allowing three total attempts each, one logical operation can reach 27 downstream attempts. That is an illustrative upper bound, not a measurement of any particular SDK or gateway. Inspect the actual retry settings before assigning blame to a model or provider.
Five failure cases to test before production
Use a disposable destination and synthetic data. Inject failures at known boundaries rather than waiting for random network problems.
- Reject before submission. Confirm that no external object exists and that recovery produces one object.
- Commit the action, then drop the acknowledgement. Confirm that recovery finds the existing object instead of creating another.
- Deliver part of a model stream, then disconnect. Confirm that incomplete tool arguments never execute.
- Run two recovery workers for one operation simultaneously. Confirm that the destination enforces deduplication under concurrency.
- Restart the worker after the business write but before its local success record. Confirm that durable destination evidence resolves the outcome.
Record logical operations, generation attempts, tool attempts, confirmed external objects, unknown outcomes, and reconciliation time separately. A lower error rate is not an improvement if it hides more duplicate effects. A quick successful retry is not a complete recovery if the first attempt remains unaccounted for.
What this means when choosing a gateway
Ask where request IDs are available, how incomplete streams are represented, which layers retry, and what evidence remains after a timeout. Check the documented scope of any idempotency promise. Compare the application's actual request history with billing records (on TopxAI, the usage log, one row per request with its status) before calling a second charge a duplicate charge.
At TopxAI, our first-request guide starts with a small generation and a Usage check. That verifies one narrow path. It does not establish exactly-once behavior for your agent's external tools. We do not claim that adding an idempotency header makes every TopxAI endpoint deduplicate requests.
The practical design question is what the application can prove after an acknowledgement disappears. If the answer is only “the client timed out,” recovery has more work to do.
Disclosure: published by the TopxAI team with AI-assisted drafting. The workflow and test cases above are engineering recommendations, not reported benchmark results or a claim of end-to-end testing against every provider.