Model APIs
Streaming and connection handling
Implement SSE parsing, cancellation, timeouts and recovery without duplicate generation or false success states.
Updated
Parse SSE by protocol
Streaming emits events while generation is in progress, usually as text/event-stream. A TCP read is not an event boundary: one read may contain half a line or several events. Buffer incomplete data and split events using the SSE blank-line delimiter.
| Protocol | Content and completion |
|---|---|
| Chat Completions | choices[].delta; typically terminates with data: [DONE] |
| Responses | Handle typed text deltas, completion and failure events |
| Anthropic | content_block_delta, message_delta, message_stop and related events |
| Gemini | Candidate chunks and finishReason |
Client and proxy configuration
- Use curl -N to disable CLI output buffering. When proxying, avoid reading the entire upstream body before responding.
- Set sensible initial-response and overall task deadlines and let users cancel.
- Connect browsers to your backend; never put model keys in SSE URLs.
- Use an SDK stream iterator, or handle split UTF-8 sequences, empty events and terminal events in your own parser.
A closed stream is not necessarily complete
Provider errors and disconnects can happen after HTTP 200. Record the request id, mark the output incomplete and let the user choose whether to regenerate. A disconnection does not guarantee that upstream processing stopped or that no usage occurred.
Avoid stacking retries in the SDK, proxy and task runner. Retrying after partial output can duplicate content, charges or tool actions.