Streaming

Streaming lets you render tokens as the model produces them instead of waiting for the whole response. Set stream: true on any /v1/chat/completions request.

Always set stream_options.include_usage: true when streaming. Without it you cannot bill the request correctly on your side, and per-key spend tracking on ours silently loses information. Every server-sent event framework we recommend below does this by default.

Non-streaming requests run under a fixed time budget; stream for long outputs. Your max_tokens / max_completion_tokens is honored in full on both paths — output is never silently truncated to a fixed ceiling. The difference is time: a request without stream: true returns nothing until the whole generation is finished, so it must complete within a fixed server-side time budget (roughly 95 seconds). A generation that can’t finish in time returns a timeout error, not partial content. Streaming has no such budget — each token keeps the connection alive and resets the clock — so set stream: true for any response that may be long (a large max_tokens, a reasoning model, or long structured output).

Example

curl https://api.flex.ai/v1/chat/completions \
  -H "Authorization: Bearer $FLEXAI_API_KEY" \
  -H "Content-Type: application/json" \
  -N \
  -d '{
    "model": "Llama-3.3-70B-Instruct-FP8",
    "messages": [{"role":"user","content":"Count to five."}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

import os
from openai import OpenAI
client = OpenAI(base_url="https://api.flex.ai/v1", api_key=os.environ["FLEXAI_API_KEY"])

stream = client.chat.completions.create(
    model="Llama-3.3-70B-Instruct-FP8",
    messages=[{"role": "user", "content": "Count to five."}],
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        print(f"\n\nprompt={chunk.usage.prompt_tokens} completion={chunk.usage.completion_tokens}")

import OpenAI from "openai";
const client = new OpenAI({
  baseURL: "https://api.flex.ai/v1",
  apiKey: process.env.FLEXAI_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "Llama-3.3-70B-Instruct-FP8",
  messages: [{ role: "user", content: "Count to five." }],
  stream: true,
  stream_options: { include_usage: true },
});

for await (const chunk of stream) {
  if (chunk.choices[0]?.delta?.content) {
    process.stdout.write(chunk.choices[0].delta.content);
  }
  if (chunk.usage) {
    console.log(`\n\nprompt=${chunk.usage.prompt_tokens} completion=${chunk.usage.completion_tokens}`);
  }
}

Anatomy of the stream

The response is text/event-stream. Each event is a line beginning with data: followed by a JSON object, then a blank line:

data: {"id":"chatcmpl-abc","choices":[{"delta":{"content":"One"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-abc","choices":[{"delta":{"content":", "},"index":0,"finish_reason":null}]}

...

data: {"id":"chatcmpl-abc","choices":[{"delta":{},"index":0,"finish_reason":"stop"}]}

data: {"id":"chatcmpl-abc","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":8,"total_tokens":20}}

data: [DONE]

Three things worth knowing:

delta.content is the incremental piece — concatenate them to reconstruct the full response.
The penultimate chunk carries finish_reason (stop, length, tool_calls, or content_filter). Its delta is typically empty.
The final-before-[DONE] chunk carries usage (only when you set stream_options.include_usage: true). This is the one source of truth for token counts; do not try to count tokens client-side.

Runaway streams

A model can get stuck repeating itself — the same phrase, over and over, until it hits its context window. Left alone, that stream would run for many minutes, hold a slot on the model, and bill every repeated token. The gateway watches the streamed text for this. When the last few kilobytes stop containing anything new for several checks in a row, it ends the stream: you receive a normal terminal chunk with finish_reason: "length", and that chunk (and the usage chunk after it) carries flexai_stopped: "degeneration" so you can tell it apart from a stream that simply reached max_tokens. On /v1/responses the terminal event is response.incomplete with incomplete_details.reason: "max_output_tokens", the same flexai_stopped field on the response object, and the text streamed so far collected into one assistant message in output (so a client that reads the final response object rather than the deltas still gets what it was billed for). A second, looser bound cuts a stream that has grown past roughly half the model’s context window while also repeating itself (flexai_stopped: "ceiling"); a long stream that is still saying new things is never cut by it. Treat flexai_stopped as a signal to change something — usually the sampling parameters (a temperature of 0 with a long open-ended prompt is the common trigger), sometimes the prompt — rather than to retry as-is. The tokens you received are billed; the tail the model would have gone on to produce is not generated.

data: {"id":"chatcmpl-abc","choices":[{"delta":{},"index":0,"finish_reason":"length"}],"flexai_stopped":"degeneration"}

data: {"id":"chatcmpl-abc","choices":[],"usage":{"prompt_tokens":120,"completion_tokens":2210,"total_tokens":2330},"flexai_stopped":"degeneration"}

data: [DONE]

Tool calls while streaming

Tool call arguments arrive as delta.tool_calls[].function.arguments fragments that you accumulate the same way as delta.content. See the tool use guide for a full example.

Handling disconnects

No resume protocol

There is no Last-Event-ID-style resume. A dropped TCP connection ends the stream — reconnecting issues a brand-new request with a fresh chatcmpl-… id, not a continuation of the old one. Tokens emitted before the disconnect are not retransmitted. If you need durability across flaky networks, persist the partial output your client has already received and re-prompt with the conversation so far when you reconnect; the model will not pick up exactly where it left off, but it will continue the conversation.

Cancellation billing

When the client closes the connection mid-generation, the server propagates the cancellation to the model backend and bills only the tokens actually emitted up to the disconnect — not the requested max_tokens. The model stops generating when you stop reading. This means it is safe to disconnect early to cap latency or cost. The matching warehouse row will reflect the tokens you received, not the budget you requested.

Retrying

A retry is a fresh, independently-billed request — there is no server-side idempotency key, so we cannot deduplicate. Pragmatic guidance:

Short prompts: retry directly; the duplicate cost is bounded.
Long generations: surface the partial output to the user and let them decide whether to continue. Echo what you already have back as context if they choose to retry.

See Concurrency for the per-key rate limit headers that govern how aggressively you can retry.

Get Started

Guides

Agents

Reference

Changelog

Chat

Completions

Embeddings

Models

Usage

Example

Anatomy of the stream

Runaway streams

Tool calls while streaming

Handling disconnects

No resume protocol

Cancellation billing

Retrying

​Example

​Anatomy of the stream

​Runaway streams

​Tool calls while streaming

​Handling disconnects

​No resume protocol

​Cancellation billing

​Retrying

Example

Anatomy of the stream

Runaway streams

Tool calls while streaming

Handling disconnects

No resume protocol

Cancellation billing

Retrying