The user has started reading the first sentence when the provider connection fails. Could the gateway call another model and finish the answer?
It could start another request. Presenting that request's output as a continuation of the first answer, however, would combine two independent generations into something the user sees as one response.
The first chunk is a commitment
Streaming delivers content incrementally instead of waiting for the complete answer. OpenAI documents this delivery model in its streaming responses guide.
Once a chunk has been delivered, some of the answer is outside the gateway. Discarding it from server memory does not remove it from the screen or undo work that the client has already performed.
Modelion therefore does not fail over between providers after the first chunk has been sent. A switch can be considered before response content is delivered, and any replacement candidate must still satisfy the policy constraints.
What does silent stitching break?
The first model might start a three-part plan. Another model, given the same context, might produce a different five-part plan. Even if the resulting prose reads smoothly, it is no longer one coherent generation.
Structured output exposes the problem more directly. Appending a newly generated object to half of an existing JSON object can create invalid output. If a tool call has already been processed, a fresh attempt might also repeat the external operation.
The network failure alone cannot decide whether repetition is acceptable. That depends on what the client has received and what effects have already occurred.
Make client states explicit
Even a small interface should distinguish these states:
| State | User-facing behaviour |
|---|---|
| No content received | Answer is being prepared |
| Content arriving | An in-progress partial answer |
| Stream interrupted | Partial answer with an explicit failure state |
Normal completion is separate again. A closed connection does not, by itself, prove that the response arrived in full. Handle the completion markers and error events of the protocol you are using.
Keeping those states distinct also improves support reports. "The request failed" otherwise hides the difference between an empty response and an answer that stopped after several paragraphs.
Give a retry its own identity
If the interface offers retry, decide whether to preserve the partial answer or clear it before starting a new one. Either can work when the behaviour is clear to the reader.
Record a separate request identifier for the new attempt so that the interrupted response and the replacement can be traced independently. For operations with side effects, such as tool execution, design deduplication around the operation's identity as well.
Measure integrity alongside speed
Time to first token describes how quickly an answer starts. It does not describe how often answers finish. A system that starts quickly and frequently disconnects can look healthy if only that first metric is tracked.
Review first-chunk latency, total duration, completed streams and interrupted attempts together. They separate startup failures that may benefit from another eligible candidate from interruptions that must be handled after the answer has already begun.
