> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flex.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# LangChain & LangGraph
> Point a LangChain chat model or a LangGraph agent at FlexAI with four lines of config, and pick a model that has been measured to hold up in a tool-calling loop.
LangChain and LangGraph work against FlexAI's OpenAI-compatible API today: `ChatOpenAI` takes a `base_url`, so you set it to `https://api.flex.ai/v1`, pass your FlexAI key, name a served model, and every LangChain surface — chat, streaming, tool calling, agents, structured output — runs unchanged. There is no package to install beyond LangChain's own.
The headline for agent builders: every model in the [table below](#which-models-to-use) completed a two-tool LangGraph agent loop against production as written here, and a model that failed any check does not appear on this page.
New to FlexAI? [Create an account](https://tokens.flex.ai/signup?utm_source=docs\&utm_medium=referral\&utm_campaign=langchain), add billing, and create an `sk-…` key on the dashboard's **API Keys** page — the [quickstart](/inference-api/quickstart) walks through it. Export the key as `FLEXAI_API_KEY` for the snippets on this page.
## Connect
```bash theme={null}
pip install -U langchain langchain-openai langgraph
```
```python theme={null}
import os
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="DeepSeek-V4-Flash-0731",
base_url="https://api.flex.ai/v1",
api_key=os.environ["FLEXAI_API_KEY"],
max_tokens=4096,
)
print(llm.invoke("In one sentence, what is a GPU?").content)
```
That is the whole integration. `model` is an id that `GET /v1/models` returns right now — the serving subset of the [catalog](https://flex.ai/models); a model offered for dedicated deployment or not currently serving is absent from that list and fails immediately. Ids are case sensitive. `max_tokens=4096` matters as soon as tools are involved; [what to plan for](#what-to-plan-for) explains why.
If you would rather not pass the URL and key in code, LangChain reads the OpenAI SDK's environment variables:
```bash theme={null}
export OPENAI_BASE_URL=https://api.flex.ai/v1
export OPENAI_API_KEY=$FLEXAI_API_KEY
```
and `ChatOpenAI(model="DeepSeek-V4-Flash-0731", max_tokens=4096)` connects the same way.
## Stream
Streaming works with `.stream()` and `.astream()`. Set `stream_usage=True` so token usage arrives at the end of the stream — that is the `stream_options.include_usage` flag the [streaming guide](/inference-api/guides/streaming) asks for, and it keeps your own accounting in step with your bill. Usage rides on its own chunk after the last piece of text (LangChain then emits one empty closing chunk), so add the chunks together and read `usage_metadata` off the total rather than off the last object:
```python theme={null}
llm = ChatOpenAI(
model="DeepSeek-V4-Flash-0731",
base_url="https://api.flex.ai/v1",
api_key=os.environ["FLEXAI_API_KEY"],
max_tokens=4096,
stream_usage=True,
)
full = None
for chunk in llm.stream("Write two short sentences about GPUs."):
print(chunk.content, end="", flush=True)
full = chunk if full is None else full + chunk
print(full.usage_metadata) # {'input_tokens': ..., 'output_tokens': ..., 'total_tokens': ...}
```
The *Streaming* column in the table passes only when this snippet streamed text in more than one chunk and usage arrived after the last text chunk.
## Call tools
Define tools with `@tool`, bind them, and feed results back as `ToolMessage`s. The round trip below is the pattern LangGraph automates in the next section.
```python theme={null}
import json
from langchain_core.messages import HumanMessage, ToolMessage
from langchain_core.tools import tool
@tool
def get_weather(city: str) -> dict:
"""Return current weather for a city."""
return {"city": city, "temperature_c": 21, "conditions": "clear"}
llm_with_tools = llm.bind_tools([get_weather])
messages = [HumanMessage("What's the weather in Paris?")]
ai = llm_with_tools.invoke(messages) # the model asks for get_weather
messages.append(ai)
for call in ai.tool_calls:
result = get_weather.invoke(call["args"])
messages.append(ToolMessage(content=json.dumps(result), tool_call_id=call["id"]))
final = llm_with_tools.invoke(messages) # the model answers with the result
print(final.content)
```
Two details of the gateway make this loop simpler than on some providers. An assistant turn that is a pure tool call may carry `content` as `null`, omitted, or `""` — the gateway accepts all three and normalizes the empty string, so whatever your LangChain version serializes is fine. A `user` message directly after a `tool` message is the other one, and here the honest answer is narrower: the gateway does bridge that gap, but only for the specific models whose chat template is known to insist on strict role alternation, and that list is short. LangGraph's own loop does not produce the shape — it sends the tool result and lets the model answer — so most agents never hit it. If you are building a multi-turn chat where the user types again straight after a tool result and you see a `400` about message ordering, insert an assistant turn between them. This page's probe does not measure that case, so the table says nothing about it.
To force a tool call, bind with `tool_choice="any"` (LangChain's name for OpenAI's `"required"`). Every model in the table honored it in the last run, with one thing worth knowing about how that was measured: the probe binds a single tool, and for a single-tool request the gateway can rewrite a forced call into a named one, which the serving stack enforces more reliably. On `gpt-oss-120b` that rewrite is what the column is reporting — with two or more tools bound, generic `required` is not enforced at the engine, and a turn where the model answers in prose instead of calling a tool comes back as a `4xx`. If you need a forced call across several bound tools on that model, name the tool you want rather than relying on `required`. Leave `tool_choice` at its default when you want the model to decide.
## Run a LangGraph agent
`create_agent` (LangChain 1.x, built on LangGraph) gives you the ReAct loop — model turn, tool calls, tool results, repeat until the model answers. With two tools the model has to chain them:
```python theme={null}
from langchain.agents import create_agent
from langchain_core.tools import tool
@tool
def get_weather(city: str) -> dict:
"""Return current weather for a city."""
return {"city": city, "temperature_c": 21, "conditions": "clear"}
@tool
def convert_c_to_f(celsius: float) -> float:
"""Convert a Celsius temperature to Fahrenheit."""
return round(celsius * 9 / 5 + 32, 1)
agent = create_agent(
llm,
tools=[get_weather, convert_c_to_f],
system_prompt="You are a concise assistant. Use the tools when they help.",
)
state = agent.invoke({"messages": [{"role": "user", "content": "What's the weather in Paris, in Fahrenheit?"}]})
print(state["messages"][-1].content)
```
`state["messages"]` holds the full trace — the two tool calls, their results and the final answer — so you can log or replay it. The *Agent loop* column in the table passes only when that trace shows the chain in order: one turn calling `get_weather`, its result, a later turn calling `convert_c_to_f` with the temperature the weather tool returned, its result, and a final answer that carries the converted value. Both calls in one turn, the converter first, or an answer that ignores the conversion do not pass. If you have graphs built on the older `langgraph.prebuilt.create_react_agent`, they run against FlexAI with the same `llm`; LangGraph 1.x marks that entry point deprecated in favour of `create_agent`, but nothing about the provider changes either way.
## Get structured output
`with_structured_output` returns a Pydantic object instead of text. Which method to use depends on what the model advertises in `supported_parameters` on `GET /v1/models` — see [model discovery](/inference-api/guides/model-discovery):
* `structured_outputs` listed → strict JSON Schema, `method="json_schema"` (LangChain's default for `ChatOpenAI`). The gateway enforces your schema, so a plain instruction is enough.
* only `response_format` listed → JSON mode, `method="json_mode"`. The model promises valid JSON, not your field names, so spell the keys out in the prompt.
```python theme={null}
from pydantic import BaseModel, Field
class Weather(BaseModel):
city: str
temperature_c: float = Field(description="Temperature in Celsius")
conditions: str
# Models advertising `structured_outputs`: the schema is enforced for you.
structured = llm.with_structured_output(Weather, method="json_schema")
print(structured.invoke("It's 21°C and clear in Paris. Extract the weather."))
# Models advertising only `response_format`: name the keys, then parse.
json_mode = llm.with_structured_output(Weather, method="json_mode")
print(json_mode.invoke(
"It's 21°C and clear in Paris. Extract the weather as JSON with exactly these keys: "
"city (string), temperature_c (number), conditions (string)."
))
```
The *Structured output* column in the table records which method each model passed with. The json\_mode rows were measured with the keyed prompt above — without it, every json\_mode model returned valid JSON under its own field names and the Pydantic parse failed, which is the behaviour JSON mode promises — and then with a second prompt that never asks for JSON at all under `response_format: json_object`, which only an enforced grammar can turn into JSON. A model whose runtime accepts JSON mode without enforcing it fails that second check and is left off the table. A model that lists neither parameter returns `400` with `param: "response_format"` rather than guessing — the [compatibility matrix](/inference-api/reference/openai-compatibility) has the exact contract.
## Which models to use
Every row below comes from running this page's snippets against production with `scripts/agents_langchain_probe.py` in the token-service repository: one chat call, a stream, the tool round trip above, the two-tool `create_agent` loop above, structured output with the method the model advertises, and a forced tool call. A model appears here only if it passed every one of those checks; the roster is FlexAI's priced production catalog, and models that were not available on the serverless API at run time, or that failed a check, are left off and counted in the footer. **yes** means it worked as written; the *Structured output* column also names the method that passed. Models are listed by id.
| Model | Chat | Streaming | Tool calling | Agent loop | Structured output | `tool_choice="required"` | First call |
| ---------------------------------- | ---- | --------- | ------------ | ---------- | ------------------ | ------------------------ | ---------- |
| `DeepSeek-V4-Flash-0731` | yes | yes | yes | yes | yes (json\_schema) | yes | 0.5 s |
| `GLM-5.2` | yes | yes | yes | yes | yes (json\_mode) | yes | 4.9 s |
| `GLM-5.3-Flash` | yes | yes | yes | yes | yes (json\_mode) | yes | 2.6 s |
| `MiniMax-M2.7` | yes | yes | yes | yes | yes (json\_mode) | yes | 2.1 s |
| `Qwen3-30B-A3B-Thinking-2507-FP8` | yes | yes | yes | yes | yes (json\_mode) | yes | 3.2 s |
| `Qwen3-Coder-30B-A3B-Instruct-FP8` | yes | yes | yes | yes | yes (json\_mode) | yes | 0.6 s |
| `Qwen3.6-27B-FP8` | yes | yes | yes | yes | yes (json\_mode) | yes | 10.9 s |
| `Qwen3.6-35B-A3B-FP8` | yes | yes | yes | yes | yes (json\_schema) | yes | 3.3 s |
| `Qwen3.8-27B` | yes | yes | yes | yes | yes (json\_mode) | yes | 1.7 s |
| `Step-3.7-Flash` | yes | yes | yes | yes | yes (json\_mode) | yes | 7.0 s |
| `gemma-4-26B-A4B-it` | yes | yes | yes | yes | yes (json\_schema) | yes | 0.5 s |
| `gemma-4-31b-it` | yes | yes | yes | yes | yes (json\_schema) | yes | 0.8 s |
| `gpt-oss-120b` | yes | yes | yes | yes | yes (json\_schema) | yes | 0.7 s |
Measured on 2026-09-17 against `https://api.flex.ai/v1` with `scripts/agents_langchain_probe.py` in the token-service repository, whose full run record — every model probed, including those this table omits — is committed beside it under `scripts/agents_langchain_probe_runs/`. "First call" is one cold request from a laptop and is indicative only, not a benchmark. 8 served models that failed a check in this run are left off the table and kept in the run record. 22 further rows in the priced catalog could not be measured because they were not answering on the serverless API at run time. Do not read that as 22 more models you could buy: the pricing catalog keeps a priced row for every model that has ever billed, so that late usage still settles, and most of those are retired models rather than current dedicated-deployment offerings.
There is no context column here on purpose. A model's window is catalog
metadata rather than something this page measures, and it moves on its own
schedule, so any number printed here would start drifting the day it was
written. Read `context_length` from `GET /v1/models` and size prompts against
that. One caveat worth knowing: the advertised window can sit above what the
engine currently accepts, so treat a `400` on a very long prompt as the
engine's limit rather than a bug in your client.
Prices are on [flex.ai/pricing](https://flex.ai/pricing); this page does not repeat them so they cannot go stale here.
## What to plan for
* **Give tool turns output headroom.** Several models write tool calls in a verbose native markup that the engine parses into `tool_calls`. If `max_tokens` cuts it off you get `finish_reason: "length"` and no `tool_calls` — to an agent that looks like the model narrated an action and stopped. `max_tokens=4096` on the `ChatOpenAI` constructor is the floor for tool turns; treat `length` without `tool_calls` as a truncation to retry with more budget.
* **Stream long agent turns.** Non-streaming requests are bounded by a roughly 95-second generation budget, and `llm.stream(...)` lifts it. `agent.stream(...)` on its own does not: LangGraph's default stream mode streams graph state, and the agent node still calls the model without streaming, so the same budget applies to each turn. Pass `stream_mode="messages"` (or use `astream_events`) to make LangChain issue a streaming request to the model.
* **Read the window from the API.** Prompts over a model's `context_length` return `400`. Size against the served number, not the model card.
* **`tool_choice="required"` is per model.** The table says which served models honor a forced call. Default `tool_choice` works everywhere tool calling does.
* **The agent-loop column is about chaining, not parallelism.** It passes only when a model calls the two tools in order, feeding the first tool's result into the second. A model that emits both calls in one turn fails it and is left off this table — which is a statement about dependent chains, not a defect: emitting several tool calls at once is a capability, and most tool-capable models we host can do it (the [tool-use guide](/inference-api/guides/tool-use) covers that case). Separately, a few models are served with single-call tool parsing and return `400` with `This model only supports single tool-calls at once!` when asked for two at once. So if your agent needs parallel tool calls, this table is the wrong filter — use it when your tools depend on each other.
* **Reasoning defaults differ by model, and reasoning tokens bill as output.** Models fall into four groups, and the group a given model is in can change as the fleet moves, so confirm with one plain call rather than trusting a list. *Off unless asked:* `DeepSeek-V4-Flash-0731` turns on with `ChatOpenAI(reasoning_effort="high")`, and the hybrid-thinking models — `gemma-4-26B-A4B-it`, `gemma-4-31b-it` and `Qwen3.6-35B-A3B-FP8` among them — with `ChatOpenAI(extra_body={"thinking": {"type": "enabled"}})`. *Its own dial:* `gpt-oss-120b` takes `reasoning_effort` at `low`, `medium` or `high`. *On by default but switchable:* `GLM-5.2` reasons unless you pass `ChatOpenAI(extra_body={"chat_template_kwargs": {"enable_thinking": False}})`. *Thinking-only:* some models reason unconditionally and reject an attempt to disable it with a `400` saying the model is thinking-only — a name containing `Thinking` is the usual hint, and for those the only lever is `max_tokens`. To place a model yourself, send one plain call and look for `reasoning_content` in `response_metadata`, then try the disable switch once and see whether it is accepted. Whenever reasoning is on, raise `max_tokens` to cover it. Use `extra_body` for these switches, not `model_kwargs`: the OpenAI SDK rejects unknown top-level parameters, and `extra_body` is how they reach the request body. The [tool-use guide](/inference-api/guides/tool-use#agent-loops-on-deepseek-v4-flash-0731) covers the DeepSeek switches in detail.
* **Older base URL.** If you already call `https://tokens.flex.ai/v1`, it keeps serving the identical API; `api.flex.ai` is simply the advertised host, and it is what the snippets here use.
## FAQ
Yes. LangGraph agents talk to the model through LangChain's `ChatOpenAI`, and FlexAI serves the OpenAI chat-completions API at `https://api.flex.ai/v1`. Set `base_url` and `api_key` on `ChatOpenAI`, name a model from the table, and `create_agent` (or an existing `create_react_agent` graph) runs unchanged.
Any model in the table completed the two-tool agent loop against production; `DeepSeek-V4-Flash-0731` is the model most of FlexAI's coding-agent traffic runs on. The table is measured, so trust it over a model card.
No. LangChain's `ChatOpenAI` with `base_url="https://api.flex.ai/v1"` is the integration; `pip install langchain-openai` is the only provider dependency.
The model ran out of output budget while writing the call. Set `max_tokens=4096` or more on `ChatOpenAI` for tool turns and retry; see [what to plan for](#what-to-plan-for).
The raw OpenAI-SDK round trip, parallel calls, and the agent-loop notes for DeepSeek-V4-Flash-0731.
The exact fields and endpoints the gateway implements.