v0.8.4
Home GitHub

Streaming

Last updated

How to receive an answer as a stream of normalized events and how stream failures are reported.

Stream the answer

Streaming is an optional capability reached by type assertion:

streamAgent, ok := agent.(llmrouter.StreamAgent)
if !ok {
    log.Fatalf("%s does not support streaming", agent.Name())
}

events, err := streamAgent.SendStream(ctx, messages, nil, llmrouter.ReasoningHigh, llmrouter.ModeDefault)
if err != nil {
    log.Fatal(err)
}

for evt := range events {
    switch evt.Type {
    case llmrouter.StreamEventText:
        fmt.Print(evt.TextDelta)
    case llmrouter.StreamEventUsage:
        fmt.Printf("\n[usage] in=%d out=%d\n", evt.Usage.Input, evt.Usage.Output)
    case llmrouter.StreamEventError:
        log.Fatal(evt.Err)
    }
}

Event types

Chat Completions and Responses SSE both normalize onto the same event type, so callers never tell the two upstream formats apart.

const (
    StreamEventText      StreamEventType = "text"
    StreamEventReasoning StreamEventType = "reasoning"
    StreamEventToolCall  StreamEventType = "tool_call"
    StreamEventUsage     StreamEventType = "usage"
    StreamEventDone      StreamEventType = "done"
    StreamEventError     StreamEventType = "error"
)

Stream errors

When the upstream answers with non-SSE content, the failure is wrapped in a *llmrouter.StreamError carrying Provider / Code / Body and unwrapping to llmrouter.ErrStreamUnsupported, so errors.As for the status code and errors.Is for the capability gap both work. Stream bodies are capped at 64 MiB and frames quoted in errors at 512 bytes.

中文