# CLAUDE Source: https://docs.flex.ai/CLAUDE # CLAUDE.md — flexaihq/docs This is the FlexAI documentation site (Mintlify, published at `https://docs.flex.ai`). Config is `docs.json`; content is `.mdx` under the section folders (`getting-started/`, `core-services/`, `platform-services/`, `cli/`, `blueprints/`, `inference-api/`, …). ## Voice & tone All published docs follow the house voice & tone guide at `.github/VOICE-AND-TONE.md`: **capability-first** framing, benefit-before-caveat, second person, confident-not-apologetic. Read it before writing or reviewing a page — tone work never changes a fact, command, limit, or URL. ## ⚠️ `inference-api/` is synced from `flexaihq/token-service` — prose round-trips, generated files don't The `inference-api/` + `snippets/inference-api/` trees are **synced from `flexaihq/token-service`** by its `.github/workflows/sync-docs.yaml` (the *forward* sync), which opens/updates the PR on branch `auto/sync-token-service-docs`. The source-of-truth is `token-service/docs/content/`. **Prose ROUND-TRIPS — safe to edit here, including in the Mintlify UI.** Hand-written pages — `overview.mdx`, `quickstart.mdx`, `guides/*.mdx`, and `reference/*.mdx` *except* `errors.mdx` — are mirrored back to `token-service/docs/content/` by `reverse-sync-prose.yaml` whenever they change on `main`. So a Mintlify edit committed to `main` publishes immediately **and** opens a reflect-back PR upstream (`auto/sync-docs-prose`) for engineers to review. `docs/content/` stays canonical; the next forward sync is a no-op. (Publish-first: the Mintlify editor must be configured to commit to `main`.) **Generated files NEVER round-trip — never edit them anywhere:** * `inference-api/reference/errors.mdx` — rendered by `scripts/generate-docs.py` * `inference-api/openapi.yaml` — copied from `token-service/docs/openapi.yaml` * `snippets/inference-api/billing-constants.mdx` — generated from backend constants `guard-synced-docs.yaml` fails any PR that touches the **generated** files above (prose PRs pass). To change generated content, edit the upstream source/generator in `token-service`. Full playbook: `token-service/docs/authoring-inference-api-docs.md`. * There is **no model catalog page** — the public model list lives at [https://flex.ai/models](https://flex.ai/models) (the old `/inference-api/reference/models` URL redirects there via `docs.json`). ## ⚠️ Two different hosts — the API is `api.flex.ai`, the browser app is `platform.flex.ai` Getting these the wrong way round is the easiest mistake to make on this site. | What you are writing | Host | Example | | - | - | - | | **API base URL** — anything a machine calls | `https://api.flex.ai` | `curl https://api.flex.ai/v1/chat/completions` | | **API base URL for an EU workspace** | `https://eu.api.flex.ai` | `curl https://eu.api.flex.ai/v1/chat/completions` | | **Dashboard / signup / billing** — anything a person opens in a browser | `https://platform.flex.ai` | `https://platform.flex.ai/dashboard/keys` | | **The portal's own `/api/*` surface** — not part of the Inference API | `https://platform.flex.ai` | `https://platform.flex.ai/api/models` | The reason the split is strict is that `api.flex.ai` serves **`/v1/*` only**. Everything else on that host — `/`, `/dashboard`, `/signup`, `/api/*` — is rejected at the edge with a bare 404 before it reaches any backend. So a dashboard link or an `/api/*` path pointed at `api.flex.ai` is not merely unconventional, it is a dead link. **The trap runs the other way too, and that direction is worse, because it does not look like a failure.** `platform.flex.ai` routes `/api/*` plus the single exact path `/v1/models`. Every other `/v1` path falls through to the browser app: a GET returns `200` with an HTML page, and a POST returns `405`, also HTML. An SDK pointed at `https://platform.flex.ai/v1` is handed a web page instead of an error, so it fails somewhere further down with a confusing message. This is exactly why probing one endpoint is never enough to call a host an API base. Never write `platform.flex.ai/v1` in a snippet. `tokens.flex.ai` is the original host and is **not dead**: `tokens.flex.ai/v1/*` keeps serving the identical API surface **unchanged and indefinitely**, because customers have it hardcoded and nothing about it is deprecated. Only its browser shell redirects to `platform.flex.ai`. The change is purely in what we *advertise*, so never "fix" a customer's existing `tokens.flex.ai/v1` usage, and never rewrite a dashboard link to `api.flex.ai`. As of PRO-101 the name is no longer used anywhere in customer-facing docs. The OpenAPI `servers:` list carries exactly `https://api.flex.ai` and `https://eu.api.flex.ai`, in that order, which is what makes generated SDKs and "Try it" default to the canonical host. **Retired browser hosts.** `console.flex.ai` (301s to `platform.flex.ai`, path preserved) and `auth.flex.ai` (`/ui/login`, `/ui/recovery`, `/registration`) are superseded by `platform.flex.ai/login`, `/recovery` and `/signup`. They still resolve, so a stale link is a consistency problem rather than a broken one, but do not write new ones. ## ⚠️ Never reference staging These are **customer-facing** docs. The internal non-prod surfaces — `tokens.flexsystems.ai`, `tokens.staging.flex.ai`, and `tokens.dev.flex.ai` (the staging and dev hosts after the flexsystems.ai retirement, ENG-1538) — must **never** appear in any published page: no base URLs, examples, links, or OpenAPI servers. The CI guards reject all three. * This applies site-wide (every published `.mdx` and `inference-api/openapi.yaml`), not just the Token Factory (Inference API) tab. * For `inference-api/**` content, the canonical home is **upstream** in `flexaihq/token-service` (`docs/content/`, `docs/openapi.yaml`, or the generator). Generated files edited here are overwritten by the sync; prose edits round-trip upstream — so never introduce a staging URL either way. Token-service's `sync-docs.yaml` guard blocks a staging URL from reaching the synced payload, and `reverse-sync-prose.yaml` re-checks prose on the way back. * `no-staging-refs.yaml` is the receiver-side backstop: it fails any PR (and push to `main`) whose published content contains `tokens.flexsystems.ai`. ## `docs.json` (navigation) IS owned here The nav is **not** synced — it lives only in this repo. So: * Adding/removing/reordering an Inference API page (e.g. a new guide) is a manual edit to `docs.json` **here**, paired with the page existing upstream (the sync delivers the `.mdx`, `docs.json` makes it appear). * When a page is withdrawn upstream, the sync deletes the `.mdx` but leaves its `docs.json` entry, and Mintlify still builds and serves a 404. `nav-integrity.yaml` fails any PR — the sync PR included — whose `docs.json` lists a missing page, so auto-merge holds the sync and fails its own run. Drop the entry in a separate PR to `main`; once it lands, auto-merge updates the sync branch and merges it. For a rename, drop the old entry first and add the new one after the sync lands (the check rejects an entry whose file isn't on `main` yet). Leave the sync PR itself alone: a comment, review or manual branch update makes auto-merge hold it for a human. * The Token Factory tab (the Inference API) is the primary tab; `Platform` is second. Its `openapi` key points at `inference-api/openapi.yaml`. ## Everything else All other sections (`platform`, `core-services`, `cli`, `blueprints`, `console`, `faq`, `best-practices`, …) are **authored directly in this repo** — edit the `.mdx` here normally. Only the `inference-api/` + `snippets/inference-api/` trees are externally owned. # Create a chat completion Source: https://docs.flex.ai/api-reference/chat/create-a-chat-completion /inference-api/openapi.yaml post /v1/chat/completions Given a list of messages, returns a model-generated response. Supports streaming (SSE), tool use, and (on some models) vision input. **Streaming:** set `stream: true` and `stream_options: { include_usage: true }` to receive token-by-token deltas. The final chunk (the one with `finish_reason` set) carries the `usage` block; you need it for accurate cost tracking. # Create a (legacy) text completion Source: https://docs.flex.ai/api-reference/completions/create-a-legacy-text-completion /inference-api/openapi.yaml post /v1/completions Prefer `/v1/chat/completions` for new integrations. `/v1/completions` is kept for compatibility with older OpenAI SDK callers. # Create an embedding Source: https://docs.flex.ai/api-reference/embeddings/create-an-embedding /inference-api/openapi.yaml post /v1/embeddings Generate a dense vector representation of one or more input strings. Use the vector to rank documents by similarity, cluster, or as a retrieval index. # List available models Source: https://docs.flex.ai/api-reference/models/list-available-models /inference-api/openapi.yaml get /v1/models Returns every model currently being served to your account — chat, code, reasoning, vision, embedding, image, audio and video — in OpenAI's `Model` shape. Early-access models appear only for organizations granted access to them. This is the catalog to build against: each entry says which endpoint to call (`category`), how much prompt it accepts (`context_length`), the longest completion it will produce (`max_output_length`), which request parameters it honors (`supported_parameters`) and what it costs (`pricing` for per-token models, `media_pricing` for per-unit models). Each entry is enriched with OpenRouter-spec metadata so third-party catalog ingesters can consume `/v1/models` directly. The OpenAI-compatible top-level keys (`id`, `object`, `created`, `owned_by`) stay byte-identical for back-compat with the OpenAI Python SDK. `owned_by` is always `"flexai"` — clients branching on the legacy `"openai"` placeholder must update. The OpenAI SDK's `models.list()` surfaces only those four keys as typed attributes; read the raw JSON (or `model.model_extra` in the Python SDK) for the rest. **Prices.** Per-token models carry `pricing`: `input_per_mtok` / `output_per_mtok` are USD per million tokens as numbers, and `prompt` / `completion` are the same rates in USD **per single token** as decimal strings (OpenRouter's convention), so `"0.0000002"` is $0.20 per million. Per-unit models carry `media_pricing`, in USD per `billing_unit`. Both state `currency`. **`null` means not published, never zero or unlimited.** `context_length`, `max_output_length`, `pricing` and `media_pricing` are present on every entry. A `null` limit means we have not published one — the model still has limits — and a `null` price means the rate is not published yet, not that the model is free (`is_free` is always `false`). `pricing` is always `null` on per-unit models and `media_pricing` on per-token models. Other optional fields are omitted when we have no value for them. The list is live: a model that is scaled down or restarting drops out until it is serving again, so treat a missing id as "not available right now" rather than "retired". If the service has no readiness signal at all, it returns the full catalog instead and sets the `X-FlexAI-Models-Degraded: no-readiness-signal` response header — the list is still fine for discovery, but not a liveness verdict. Responses are `Cache-Control: private, max-age=30`. # List currently-effective per-model pricing Source: https://docs.flex.ai/api-reference/models/list-currently-effective-per-model-pricing /inference-api/openapi.yaml get /v1/pricing FlexAI extension. Returns the currently-effective pricing rows for every model currently being served. A model that is temporarily not serving — scaled down or restarting — drops out until it is back, so keep your own copy of a rate rather than reading an absent row as a price change. **Requires a Bearer API key** — the same key you use for inference. Pricing is identical for every caller, but the endpoint is authenticated so the live model catalogue isn't exposed to anonymous probes. Cache-Control is `private, max-age=60, stale-while-revalidate=120`, so a client may cache its own copy while shared/CDN caches will not. `/v1/models` carries the same rates on each model entry (`pricing` per token, `media_pricing` per unit) and is the catalog to build against. This endpoint is the flat, per-million-token view of the same numbers, with `pricing_type` and `effective_at` on every row, for billing and cost tooling that wants prices without the rest of the model metadata. # Retrieve a model Source: https://docs.flex.ai/api-reference/models/retrieve-a-model /inference-api/openapi.yaml get /v1/models/{model} Returns one entry of the `GET /v1/models` catalog — the same object, with the same fields and the same values, that the list returns for that model to the same caller. The OpenAI SDK's `client.models.retrieve("")` calls this. `model` may be the model's `id` or any of its `aliases`; the entry returned always carries the canonical `id`. A model that is not in your live catalog right now — unknown, retired, scaled down, or not available to your account — returns `404` with `code: "model_not_found"`. Responses carry the same `Cache-Control` and `X-FlexAI-Models-Degraded` headers as the list. # Limits and remaining credit for the calling key Source: https://docs.flex.ai/api-reference/usage/limits-and-remaining-credit-for-the-calling-key /inference-api/openapi.yaml get /v1/flex/key Returns the calling key's configured limits and credit state: rate limits (RPM/TPM), cumulative spend, budget ceiling, and remaining budget — so a client can distinguish "the model is unhealthy" from "I'm rate-limited / out of credits" without admin access. The key in the `Authorization` header sees only itself. This endpoint is **budget-exempt**: it keeps answering when the key is out of credits (that is exactly when you need it). Inference endpoints remain fully budget-gated. `remaining_budget_usd` is the same value the gateway's 402 `insufficient_balance` check enforces, floored at 0. `rate_limit` reports the key's *configured* limits — the same values the `x-ratelimit-*` headers on inference responses advertise; for live headroom in the current window, read `x-ratelimit-api_key-remaining-requests` from a successful non-streaming response (or `x-ratelimit-remaining-requests` from a 429). # Programmatic spend for the calling key Source: https://docs.flex.ai/api-reference/usage/programmatic-spend-for-the-calling-key /inference-api/openapi.yaml get /v1/flex/usage FlexAI-specific shape (intentionally not OpenAI's `/v1/organization/usage/*` — that surface requires org/team scoping which FlexAI does not yet model). Returns total spend, total tokens, per-model breakdown, and a daily history for the API key in the `Authorization` header. Cached per-key for 60 seconds. `total_spend` is the sum of the charges applied to your account for the requests in the window, so it matches your invoice for that period exactly. Note that it is not recoverable from `total_tokens` and the published per-token rates: a cached prompt is billed at the lower cached rate, so a token-based re-derivation will read high. Group by `canonical_model` to reconcile per model. # Usage and cost for a single request Source: https://docs.flex.ai/api-reference/usage/usage-and-cost-for-a-single-request /inference-api/openapi.yaml get /v1/flex/usage/request/{request_id} Looks up one request's usage row for exact per-request cost attribution — the per-turn complement to the whole-key aggregate `/v1/flex/usage` returns. `request_id` is the value of the `x-litellm-call-id` response header on the original inference call (**not** the `id` field in the response body). Capture that header per call, then join it here. The lookup is scoped to the calling key: a request made with a different key returns the same `404` as an id that does not exist. Rows are written when a request completes, so a lookup immediately after the call may briefly `404` — retry after a few seconds. `total_spend` is the charge applied to your account for this request — the same figure the `/v1/flex/usage` aggregates sum, so per-request values reconcile against those totals exactly. It is not derived from the token counts returned alongside it: a cached prompt is billed at the lower cached rate, so a token-based re-derivation reads high. A request whose billing has not been written yet reports `0.0` rather than an estimate; billing settles within seconds, so retry if you need the final figure immediately after the call. Only LiteLLM-routed (text) requests carry a call id, so rows always have `modality=text`. # Best Practices Source: https://docs.flex.ai/best-practices Recommended practices for working with the FlexAI platform Explore proven best practices and recommendations for working effectively with FlexAI. These guides provide practical solutions for common integration challenges and help you optimize your workflow across various environments and use cases. Learn how to configure the FlexAI CLI to work properly in virtual machines or containers using host networking for authentication callbacks. Workaround for authenticating with FlexAI CLI on headless servers or VMs without web browsers by copying configuration files. Essential guidelines for naming FlexAI resources including Inference Endpoints, Fine-tuning and Training Jobs, Datasets, and Secrets. Learn the required formats and constraints. # Logging in with the CLI on a local Virtualized Environment Source: https://docs.flex.ai/best-practices/alternative-methods/cli-on-a-virtualized-env Configure the FlexAI CLI for use in virtual machines and containers If the environment you're using to run the FlexAI CLI is a local VM or container, you can set-up your environment to use *host networking* to allow the FlexAI CLI to get the authentication callback message from your host system. #### References * [Docker Networking: Host Network Driver](https://docs.docker.com/engine/network/drivers/host/) * [Podman: network create documentation](https://docs.podman.io/en/latest/markdown/podman-network-create.1.html) * [VirtualBox: Virtual Networking](https://www.virtualbox.org/manual/ch06.html) # Logging in with the CLI from a non-graphical environment Source: https://docs.flex.ai/best-practices/alternative-methods/logging-from-a-remote-env Authenticate the FlexAI CLI on headless or remote environments You can authenticate the FlexAI CLI on a headless or non-graphical environment — such as a Virtual Machine (*VM*) — by signing in on a machine that has a web browser and copying the credentials across. The browser step is needed because `flexai auth login` authenticates you through your GitHub account. Install the FlexAI CLI on a machine equipped with a web browser, log in using the `flexai auth login` command, and then copy the FlexAI CLI configuration file contents to the VM's `$HOME/.flexai/config.yaml` configuration file: 1. Log into FlexAI from a machine equipped with a web browser 2. Open the `$HOME/.flexai/config.yaml` file on that machine 3. Copy its contents 4. Open the `$HOME/.flexai/config.yaml` file on your machine without a graphical environment 5. Paste the contents of the first configuration file into the second one The FlexAI CLI on the machine without a graphical environment should now be authenticated and ready to use. # Resource Naming Conventions Source: https://docs.flex.ai/best-practices/resource-naming-conventions Naming rules and constraints for FlexAI resources FlexAI resources ([Training Jobs](/cli/reference/training/), [Datasets](/cli/reference/dataset/), [Code Registries](/cli/reference/code-registry/), [Secrets](/cli/reference/secret/), [Inference Endpoints](/cli/reference/inference/), [Storage Providers](/cli/reference/storage/) and [Checkpoints](/cli/reference/checkpoint/)) have specific naming conventions that must be followed. These conventions ensure that resource names are unique, consistent, and easy to understand. ## Resource names * Each name must be unique within its namespace — attempting to create a resource with the same name as an existing one will return an error. * Must start and end with a letter or a number. * Resource names can use letters (a-z, A-Z), numbers (0-9), hyphens (`-`), and underscores (`_`). * Must be between 3 and 63 characters long. ## Environment Variables and Secrets ### Key=Value pairs The **Key** in a Key=Value pair used to describe a Secret or Environment Variable must follow the conventions below: * Each **Key** must be unique within a command execution: — attempting to use the same **Key** multiple times will return an error. * The first character must be a letter (a-z or A-Z), a hyphen (`-`), an underscore (`_`), or a period (`.`). * The rest of the **Key** can include letters (a-z, A-Z), numbers (0-9), hyphens (`-`), underscores (`_`), or periods (`.`), and it can also end with any of these characters. * Must be **at least** 1 character long. # Blueprints Source: https://docs.flex.ai/blueprints A set of end-to-end experiments to explore FlexAI — from your first training job to advanced fine-tuning, inference, and multi-agent applications. This section walks you through **a set of hands-on experiments** with **FlexAI** — from running your very first training job to fine-tuning language, diffusion, and text-to-speech models with techniques like QLoRA and LoRA, and integrating FlexAI with other platforms such as experiment trackers. ## Getting Started ### Prerequisites **FlexAI CLI**: Install the FlexAI CLI by following the steps shown in the [Installing the FlexAI CLI](/cli/install) guide. ## The Experiments The following table lists out the experiments available. Each experiment is designed to walk you through a specific use case and contains its required code, as well as detailed instructions on how to run it on FlexAI: | No. | Section | Description | | - | - | - | | 1 | [A Simple Training Job on FlexAI](/blueprints/running-a-simple-training-job) | Step-by-step guide to get your first Training Job on FlexAI running | | 2 | [A Simple Distributed Data Parallel (DDP) Training Job on FlexAI](/blueprints/running-a-simple-training-job-with-ddp) | Demonstrates that you only need to add 2 flags in order to start a DDP Training Job | | 3 | [Resuming a Training Job from a Checkpoint](/blueprints/continuing-a-training-job-from-a-checkpoint) | Learn how to resume a Training Job from a previously saved checkpoint | | 4 | [Streaming Large Datasets During a Training Job](/blueprints/streaming-datasets) | Train a model on a large dataset using streaming | | 5 | [Training Job & Experiment Tracking](/blueprints/integrating-a-experiment-tracker) | Using Weights and Biases with FlexAI for *experiment tracking* | | 6 | [Fine-Tuning a Language Model with QLoRA](/blueprints/qlora-ft-on-a-language-model) | Fine-tune a causal language model efficiently using QLoRA | | 7 | [Fine-Tuning a Diffusion Model with LoRA](/blueprints/lora-ft-on-a-diffusion-model) | Fine-tune a diffusion model efficiently using LoRA | | 8 | [Fine-Tuning a Text-to-Speech Model](/blueprints/ft-on-a-tts-model) | Fine-tune a text-to-speech (TTS) model | | 9 | [Fine-Tuning a Language Model using Flash Attention](/blueprints/flash-attention-ft-on-a-language-model) | Fine-tune a causal language model efficiently using the flash-attn package | | 10 | [Train and Serve a French LLM on FlexAI with LlamaFactory](/blueprints/llama-factory) | Fine-tune Qwen2.5-7B on French data using LlamaFactory and deploy as a production-ready inference endpoint | | 11 | [Train and Serve Language Models with Axolotl](/blueprints/axolotl) | Fine-tune language models on domain-specific data using Axolotl framework with custom dataset configurations and FSDP | | 12 | [Reinforcement Learning Fine-Tuning with EasyR1](/blueprints/easyR1) | Train language models with RL using GRPO and DAPO algorithms for enhanced reasoning capabilities | | 13 | [GRPO Training on Vision-Language Models](/blueprints/vlm-grpo) | Fine-tune vision-language models using GRPO reinforcement learning with HuggingFace TRL, LoRA, vLLM, and DeepSpeed ZeRO-3 | | 14 | [Language Model Evaluation with LM-Evaluation-Harness](/blueprints/lm-evaluation-harness) | Comprehensive evaluation of language models across 300+ tasks and benchmarks | | 15 | [RAG Application with LangChain and FlexAI Inference Endpoints](/blueprints/rag-application) | Interactive interface for users to ask questions based on provided documents using Retrieval-Augmented Generation | | 16 | [Speech-to-Text Application Using FlexAI Inference Endpoints](/blueprints/speech-to-text-inference) | Interactive interface for recording audio messages and receiving transcriptions | | 17 | [Multi-Agent Orchestration with LangGraph](/blueprints/multi-agent) | Build a multi-agent system where specialized AI agents work together under a central supervisor | | 18 | [Text-to-Image Inference with FlexAI Endpoints](/blueprints/text-to-image-inference) | Deploy and use Stable Diffusion 3.5 Large for high-quality image generation via FlexAI inference endpoints | | 19 | [Text-to-Audio Inference with FlexAI Endpoints](/blueprints/text-to-audio-inference) | Deploy and use Stable Audio Open 1.0 for high-quality audio generation via FlexAI inference endpoints | | 20 | [Text-to-Speech Inference with FlexAI Endpoints](/blueprints/text-to-speech-inference) | Deploy and use the Kokoro model for natural voice synthesis via FlexAI inference endpoints | | 21 | [Text-to-Video Inference with FlexAI Endpoints](/blueprints/text-to-video-inference) | Deploy and use Wan2.2-T2V-A14B for high-quality video generation via FlexAI inference endpoints | | 22 | [Object Detection and Computer Vision with Ultralytics YOLO11](/blueprints/ultralytics) | Train and deploy state-of-the-art object detection, segmentation, and pose estimation models using YOLO11 | | 23 | [Text-to-Speech Inference with Qwen3-TTS and FlexAI Endpoints](/blueprints/text-to-audio-qwen3-tts) | Interactive Qwen3-TTS interface for high-quality text-to-speech synthesis | *** ## Keep in mind ### Notes on HuggingFace Accelerate Integration This section includes experiments that utilize the **HuggingFace Accelerate library** for efficient training. The **FlexAI CLI** simplifies running training scripts by automatically determining the appropriate execution method: * **`python`**: Used for single-accelerator training. * **`torchrun`**: Automatically used for multi-accelerator distributed training. If you're accustomed to using the **`accelerate launch`** command, you can seamlessly run the same scripts on FlexAI without modification. Simply provide the script to FlexAI, and it will handle execution. As highlighted in the [Accelerate documentation](https://huggingface.co/docs/accelerate/en/basic_tutorials/launch#using-accelerate-launch), the `accelerate launch` command is optional. Instead, the Accelerate functionality integrates directly into your script, making it compatible with other launchers like `torchrun`. Unlike `accelerate launch`, `torchrun` does not use the YAML configuration file generated by `accelerate config`. If your training setup relies on specific configurations from the YAML file, you may need to adjust your script to explicitly define these settings using the `Accelerator` class. By doing so, you ensure seamless execution across different environments while maintaining flexibility for various training setups. # Fine-Tune LLMs with Axolotl on Domain-Specific Data Source: https://docs.flex.ai/blueprints/axolotl Use Axolotl to fine-tune language models on your own data, then deploy as a production inference endpoint on FlexAI. FSDP, custom datasets, one-click deploy. This blueprint demonstrates how to use FlexAI to fine-tune language models on domain-specific data using Axolotl, then deploy them as production-ready inference endpoints. For illustration purposes, we'll fine-tune for maximum command of French using the `Qwen2.5-7B` model and the `openhermes-fr` dataset. You will see that this process requires configuring Axolotl's training parameters, leveraging FlexAI's managed training infrastructure, and deploying the fine-tuned model as a scalable inference endpoint. If you haven't already connected FlexAI to GitHub, run `flexai code-registry connect` to set up a code registry connection. This allows FlexAI to pull repositories directly using the repository URL in training commands. First, ensure your domain-specific dataset is properly configured in your Axolotl YAML file. For our French language example, we'll use the `openhermes-fr` dataset. Navigate to `code/axolotl/qwen2/fft-7b-french.yaml` and verify the dataset configuration: ```yaml theme={null} datasets: - path: legmlai/openhermes-fr type: system_prompt: "" field_instruction: prompt field_output: accepted_completion format: "{instruction}" no_input_format: "{instruction}" ``` For your own use case, replace this with your domain-specific dataset. The `openhermes-fr` dataset is specifically designed for French language tasks and serves as an excellent example of domain specialization. The `qwen2/fft-7b-french.yaml` file contains the training configuration for domain-specific fine-tuning. Key settings include: * **Model**: `Qwen/Qwen2.5-7B` - Excellent multilingual base model suitable for domain adaptation * **Stage**: Full fine-tuning - Perfect for task-specific and domain-specific adaptation * **Dataset**: `openhermes-fr` - Example domain-specific dataset (replace with your own) * **Training**: Full fine-tuning with FSDP (Fully Sharded Data Parallel) for optimal performance ```yaml theme={null} base_model: Qwen/Qwen2.5-7B load_in_8bit: false load_in_4bit: false strict: false datasets: - path: legmlai/openhermes-fr type: system_prompt: "" field_instruction: prompt field_output: accepted_completion format: "{instruction}" no_input_format: "{instruction}" dataset_prepared_path: val_set_size: 0.05 output_dir: /output-checkpoint sequence_len: 2048 sample_packing: true eval_sample_packing: true gradient_accumulation_steps: 2 micro_batch_size: 1 num_epochs: 3 optimizer: adamw_torch_fused lr_scheduler: cosine learning_rate: 1.0e-5 bf16: auto tf32: true gradient_checkpointing: true flash_attention: true warmup_ratio: 0.1 evals_per_epoch: 4 saves_per_epoch: 1 weight_decay: 0.0 fsdp: - full_shard - auto_wrap fsdp_config: fsdp_limit_all_gathers: true fsdp_sync_module_states: true fsdp_offload_params: true fsdp_use_orig_params: false fsdp_cpu_ram_efficient_loading: true fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP fsdp_transformer_layer_cls_to_wrap: Qwen2DecoderLayer fsdp_state_dict_type: FULL_STATE_DICT fsdp_sharding_strategy: FULL_SHARD ``` ## Create Secrets To access the Qwen2.5-7B model and OpenHermes-FR dataset, you need a HuggingFace token. Use the [`flexai secret create` command](https://docs.flex.ai/cli/reference/secret/create) to store your *HuggingFace Token* as a secret. Replace `` with your desired name for the secret: ```bash theme={null} flexai secret create ``` Then paste your *HuggingFace Token* API key value. ## \[Optional] Pre-fetch the Model To speed up training and avoid downloading large models at runtime, you can pre-fetch your HuggingFace model to FlexAI storage. For example, to pre-fetch the `Qwen/Qwen2.5-7B` model: 1. **Create a HuggingFace storage provider:** ```bash theme={null} flexai storage create HF-STORAGE --provider huggingface --hf-token-name ``` 2. **Push the model checkpoint to your storage:** ```bash theme={null} flexai checkpoint push qwen25-7b --storage-provider HF-STORAGE --source-path Qwen/Qwen2.5-7B ``` During your training run, you can use the pre-fetched model by adding the following argument to your training command: ```bash theme={null} --checkpoint qwen25-7b ``` ## Training For a 7B model, we recommend using **1 node (4 × H100 GPUs)** to ensure reasonable training time and avoid out-of-memory issues. ### Standard Training (without prefetch) ```bash theme={null} flexai training run axolotl-french-sft \ --accels 4 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --env FORCE_TORCHRUN=1 \ --requirements-path code/axolotl/requirements.txt \ -- axolotl train code/axolotl/qwen2/fft-7b-french.yaml ``` ### Training with Model Prefetch To take advantage of model pre-fetching performed in the [Optional: Pre-fetch the Model](#optional-pre-fetch-the-model) section, use: ```bash theme={null} flexai training run axolotl-french-sft-prefetched \ --accels 4 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --checkpoint qwen25-7b \ --env FORCE_TORCHRUN=1 \ --requirements-path code/axolotl/requirements.txt \ -- axolotl train code/axolotl/qwen2/fft-7b-french.yaml ``` ## Monitoring Training Progress You can check the status and lifecycle events of your Training Job by running: ```bash theme={null} flexai training inspect axolotl-french-sft ``` Additionally, you can view the logs of your Training Job by running: ```bash theme={null} flexai training logs axolotl-french-sft ``` ### Training Observability with Weights & Biases For advanced monitoring and visualization of training metrics, Axolotl supports Weights & Biases (wandb) integration. You can leverage wandb logging for detailed insights into training progress, loss curves, and model performance. To enable wandb logging, update your YAML configuration: ```yaml theme={null} wandb_project: your-project-name wandb_entity: your-wandb-entity wandb_watch: gradients wandb_name: qwen25-7b-sft wandb_log_model: checkpoint ``` For additional monitoring, you can also access FlexAI's hosted [TensorBoard](https://tensorboard.flex.ai/) instance, which provides organization-wide real-time insights into your training workload progress. TensorBoard is enabled by default for every organization - simply log in using your FlexAI account credentials to track metrics and visualize performance. For more details on observability features, see the [Axolotl documentation](https://github.com/OpenAccess-AI-Collective/axolotl#logging). ## Getting Training Checkpoints Once the Training Job completes successfully, you will be able to list all the produced checkpoints: ```bash theme={null} flexai training checkpoints axolotl-french-sft ``` Look for checkpoints marked as `INFERENCE READY = true` - these are ready for serving. ## Serving the Trained Model Deploy your trained model directly from the checkpoint using FlexAI inference. Replace `` with the ID from an inference-ready checkpoint: ```bash theme={null} flexai inference serve axolotl-french-sft-endpoint --checkpoint ``` GPU specification for inference endpoints is currently managed automatically by FlexAI. Future versions will allow explicit GPU count specification for inference workloads to optimize cost and performance based on your specific requirements. You can monitor your inference endpoint status: ```bash theme={null} # List all inference endpoints flexai inference list # Get detailed endpoint information flexai inference inspect axolotl-french-sft-endpoint # Check endpoint logs flexai inference logs axolotl-french-sft-endpoint ``` ## Testing Your Domain-Specific Model Once the endpoint is running, you can test it with domain-specific prompts. For our French language example, the model should demonstrate strong French language understanding, proper grammar and syntax, and cultural context awareness. ### Before and After Training Comparison To illustrate the improvement from fine-tuning on French data, here's a comparison using the question: **"Qui a gagné la Coupe du monde 2018 ?"** (who won the 2018 world cup?) **Base Model Response (Qwen/Qwen2.5-7B before training):** ``` La Coupe du monde de football 2018 a été remportée par la Russie. ``` *Issues: Incorrect answer (says Russia instead of France)* **Fine-tuned Model Response (after full fine-tuning on openhermes-fr):** ``` La France a remporté la Coupe du monde de football 2018, en battant le Croatie lors de la finale disputée à Moscou le 15 juillet 2018. ``` *Improvements: Correct answer (France), excellent French grammar, accurate details, proper structure* This example demonstrates the dramatic improvement in both factual accuracy and French language quality after domain-specific fine-tuning. ### Example API Call ```bash theme={null} curl -X POST "https://your-endpoint-url/v1/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_API_KEY" \ -d '{ "prompt": "Expliquez-moi les avantages de l'\''intelligence artificielle en français:", "max_tokens": 200, "temperature": 0.7 }' ``` Adapt the prompt and evaluation criteria to match your specific domain and use case. ## Expected Results After fine-tuning on domain-specific data, your model should achieve: * **Domain Expertise**: Specialized knowledge and terminology understanding for your target domain * **Task-Specific Performance**: Enhanced capabilities for domain-relevant tasks and workflows * **Maintained General Capabilities**: Preserved reasoning, problem-solving, and general language skills For our French language example: * **Strong French Language Understanding**: Natural conversation flow, proper grammar, cultural context * **High Performance on French Tasks**: Question answering, text summarization, creative writing ## Technical Details ### Training Configuration Breakdown: * **Full Fine-tuning with FSDP**: Enables training of 7B model on 1 node efficiently * **Mixed Precision (bf16)**: Accelerates training while maintaining numerical stability * **Gradient Accumulation**: Effective batch size of 2 (2 steps × 1 per device) * **Learning Rate Schedule**: Cosine decay with 10% warmup for stable convergence * **Context Length**: 2048 tokens, optimized for conversation tasks * **Sample Packing**: Efficient batch utilization for variable-length sequences * **Flash Attention**: Optimized attention mechanism for faster training ### Resource Requirements **Recommended Configuration for Qwen2.5-7B:** * **Nodes**: 1 node (cost-effective for 7B models) * **Accelerators**: 4 × H100 GPUs per node * **Memory**: \~200GB+ GPU memory total * **Training Time**: \~2-4 hours for 3 epochs * **Storage**: \~30GB for checkpoints **Command Line Parameters Explained:** * `FORCE_TORCHRUN=1`: Ensures proper distributed training setup ### Scaling Options: * For faster training: Increase to 2 nodes (16 × H100) * For larger datasets: Adjust `num_epochs` parameter * For longer context: Increase `sequence_len` (requires more memory) * For memory efficiency: Switch to QLoRA with `load_in_4bit: true` and `adapter: qlora` * For other models: Use configs in `code/axolotl/` directory (Llama, Mistral, Gemma, Phi, etc.) ## Additional Training Examples Axolotl provides extensive configuration examples for various models and training strategies: ### Llama 3.1 8B with LoRA ```bash theme={null} flexai training run axolotl-lora-llama3-8B \ --accels 8 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --env FORCE_TORCHRUN=1 \ --secret HF_TOKEN= \ --requirements-path code/axolotl/requirements.txt \ --runtime nvidia-25.06 \ -- axolotl train code/axolotl/llama-3/lora-8b.yml ``` ### Mistral 7B with QLoRA ```bash theme={null} flexai training run axolotl-qlora-mistral-7B \ --accels 8 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --env FORCE_TORCHRUN=1 \ --secret HF_TOKEN= \ --requirements-path code/axolotl/requirements.txt \ --runtime nvidia-25.06 \ -- axolotl train code/axolotl/mistral/qlora.yml ``` Explore the `code/axolotl/` directory for more examples including Gemma, Phi, Qwen2, multimodal models, and advanced configurations. ## Troubleshooting **Training Job Fails to Start:** ```bash theme={null} # Check FlexAI authentication flexai auth status # Verify repository access git clone https://github.com/flexaihq/blueprints ``` **Out of Memory Errors:** * Reduce `micro_batch_size` from 1 to lower value (not recommended below 1) * Increase `gradient_accumulation_steps` to maintain effective batch size * Consider switching to QLoRA: set `load_in_4bit: true` and `adapter: qlora` for memory efficiency * Enable `fsdp_offload_params: true` for additional memory savings **Checkpoint Not Inference Ready:** * Wait for training to complete fully (check with `flexai training inspect`) * Ensure Axolotl configuration saves model in compatible format * Verify training completed successfully without errors **Endpoint Deployment Issues:** * Verify checkpoint shows `INFERENCE READY = true` status * Check FlexAI cluster availability with `flexai inference list` * Review detailed logs with `flexai inference logs ` **Dataset Loading Issues:** * Verify dataset path is correct in YAML configuration (e.g., `legmlai/openhermes-fr`) * Ensure HuggingFace token has access to private datasets * Check dataset format matches the specified type (sharegpt, alpaca, etc.) ## Code ### `requirements.txt` ```text theme={null} packaging==23.2 setuptools==75.8.0 wheel ninja flash-attn>=2.6.3 # Install deepspeed deepspeed>=0.15.0 # Install axolotl from GitHub git+https://github.com/flexaihq/axolotl ``` ### `qwen2/fft-7b-french.yaml` ```yaml theme={null} --- base_model: Qwen/Qwen2.5-7B # Automatically upload checkpoint and final model to HF # hub_model_id: username/custom_model_name datasets: - path: legmlai/openhermes-fr type: system_prompt: '' field_instruction: prompt field_output: accepted_completion format: '{instruction}' no_input_format: '{instruction}' dataset_prepared_path: val_set_size: 0.05 output_dir: /output-checkpoint sequence_len: 2048 sample_packing: true eval_sample_packing: true wandb_project: wandb_entity: wandb_watch: wandb_name: wandb_log_model: gradient_accumulation_steps: 2 micro_batch_size: 1 num_epochs: 3 optimizer: adamw_torch_fused lr_scheduler: cosine learning_rate: 1.0e-5 bf16: auto tf32: true gradient_checkpointing: true gradient_checkpointing_kwargs: use_reentrant: false resume_from_checkpoint: logging_steps: 1 flash_attention: true warmup_ratio: 0.1 evals_per_epoch: 4 saves_per_epoch: 1 weight_decay: 0.0 fsdp: - full_shard - auto_wrap fsdp_config: fsdp_limit_all_gathers: true fsdp_sync_module_states: true fsdp_offload_params: true fsdp_use_orig_params: false fsdp_cpu_ram_efficient_loading: true fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP fsdp_transformer_layer_cls_to_wrap: Qwen2DecoderLayer fsdp_state_dict_type: FULL_STATE_DICT fsdp_sharding_strategy: FULL_SHARD special_tokens: # save_first_step: true # uncomment this to validate checkpoint saving works with your config ```

🚀 Run this on FlexAI

Managed checkpoints mean you never lose a run to preemption. Jobs launch in under 60 seconds — no infra setup, built-in observability.

Get started → Talk to us
# Resume Training from a Checkpoint on FlexAI — Never Lose a Run Source: https://docs.flex.ai/blueprints/continuing-a-training-job-from-a-checkpoint Learn how to resume a training job from a saved checkpoint on FlexAI. Managed checkpoint storage means preemption never kills your progress. This experiment will continue a training from a Checkpoint emitted by the Training Job in the Running a simple Training Job on FlexAI experiment, so make sure to complete it and download its output artifacts before proceeding. Extract the contents of the `output_0.zip` file into a directory named `fetched_checkpoints`: ```bash theme={null} unzip output_0.zip -d fetched_checkpoints ``` This `fetched_checkpoints` directory contains the different checkpoints that have been saved in the `/output-checkpoint` of the Training Job's runtime environment during execution. Let's use the checkpoint (saved at step 500) located in `fetched_checkpoints/output/checkpoint-500/`. Create the FlexAI checkpoint to be passed to the next run that will resume the training: ```bash theme={null} flexai checkpoint push gpt2-ckpt500 --file fetched_checkpoints/output/checkpoint-500 ``` Resume training from your checkpoint with the following command: ```bash theme={null} flexai training run gpt2training-resume --repository-url https://github.com/flexaihq/blueprints --dataset gpt2-tokenized-wikitext --checkpoint gpt2-ckpt500 --requirements-path code/causal-language-modeling/requirements.txt \ -- code/causal-language-modeling/train.py \ --do_eval \ --do_train \ --dataset_name wikitext \ --tokenized_dataset_load_dir /input/gpt2-tokenized-wikitext \ --model_name_or_path /input-checkpoint \ --resume_from_checkpoint /input-checkpoint \ --output_dir /output-checkpoint \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 8 \ --logging_steps 50 \ --save_steps 500 \ --eval_steps 500 \ --eval_strategy steps \ --num_train_epochs 6 ``` Compared to the experiment that starts training from the base model, note that: * `--checkpoint gpt2-ckpt500` has been added - referring to the checkpoint created above, the content of the `checkpoint-500` folder will be mounted on `/input-checkpoint` * `--model_name_or_path` has been updated - pointing to the new checkpoint location together with additional HuggingFace args to resume the training from the checkpoint: * `--resume_from_checkpoint /input-checkpoint` * `--num_train_epochs 6` ## Code ### `code/causal-language-modeling/train.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import math import os import sys from dataclasses import dataclass, field from typing import Optional import evaluate import numpy import torch import transformers from transformers import ( AutoConfig, AutoModelForCausalLM, AutoTokenizer, HfArgumentParser, Trainer, TrainingArguments, default_data_collator, ) SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) sys.path.append(os.path.dirname(SCRIPT_DIR)) from dataset.prepare_save_dataset import DatasetArguments, load_and_tokenize from utils.experiment_tracking import set_wandb transformers.logging.set_verbosity_info() @dataclass class ModelArguments: model_name_or_path: str = field(default=None) torch_dtype: Optional[str] = field(default=None) attn_implementation: Optional[str] = field(default=None) @dataclass class AdditionalArguments: max_train_samples: Optional[int] = field(default=None) max_eval_samples: Optional[int] = field(default=None) def parse_args(): parser = HfArgumentParser( (DatasetArguments, ModelArguments, TrainingArguments, AdditionalArguments) ) return parser.parse_args_into_dataclasses() def _load_model_and_tokenizer(model_args, print_model=False): config = AutoConfig.from_pretrained(model_args.model_name_or_path) tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path) tokenizer.pad_token = tokenizer.eos_token tokenizer.pad_token_id = tokenizer.eos_token_id torch_dtype = ( model_args.torch_dtype if model_args.torch_dtype in ["auto", None] else getattr(torch, model_args.torch_dtype) ) extra_model_args = {} if model_args.attn_implementation is not None: extra_model_args["attn_implementation"] = model_args.attn_implementation model = AutoModelForCausalLM.from_pretrained( model_args.model_name_or_path, config=config, torch_dtype=torch_dtype, **extra_model_args, ) if print_model: print(model) return model, tokenizer def train(dataset_args, model_args, training_args, additional_args): set_wandb(training_args) print(f"Training/evaluation parameters {training_args}") train_dataset, eval_dataset = load_and_tokenize( tokenizer_model_name=model_args.model_name_or_path, do_eval=training_args.do_eval, **vars(dataset_args), ) max_train_samples = float("inf") max_eval_samples = float("inf") if not dataset_args.dataset_streaming: max_train_samples = len(train_dataset) if training_args.do_eval: max_eval_samples = len(eval_dataset) if additional_args.max_train_samples is not None: max_train_samples = min(max_train_samples, additional_args.max_train_samples) train_dataset = train_dataset.take(max_train_samples) if additional_args.max_eval_samples is not None: assert training_args.do_eval, "Cannot set max_eval_samples without do_eval" max_eval_samples = min(max_eval_samples, additional_args.max_eval_samples) eval_dataset = eval_dataset.take(max_eval_samples) model, tokenizer = _load_model_and_tokenizer(model_args, print_model=True) metric = evaluate.load("accuracy") def preprocess_logits_for_metrics(logits, labels): if isinstance(logits, tuple): logits = logits[0] return logits.argmax(dim=-1) def compute_metrics(eval_preds): preds, labels = eval_preds mask = (labels != tokenizer.pad_token_id) & (labels != -100) labels = numpy.concatenate([label[mask[i]][1:] for i, label in enumerate(labels)]) preds = numpy.concatenate([pred[mask[i]][:-1] for i, pred in enumerate(preds)]) return metric.compute(predictions=preds, references=labels) trainer = Trainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset, tokenizer=tokenizer, data_collator=default_data_collator, compute_metrics=compute_metrics, preprocess_logits_for_metrics=preprocess_logits_for_metrics, ) train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint) trainer.save_model() metrics = train_result.metrics metrics["train_samples"] = max_train_samples trainer.log_metrics("train", metrics) trainer.save_metrics("train", metrics) trainer.save_state() if training_args.do_eval: metrics = trainer.evaluate() metrics["eval_samples"] = max_eval_samples try: perplexity = math.exp(metrics["eval_loss"]) except OverflowError: perplexity = float("inf") metrics["perplexity"] = perplexity trainer.log_metrics("eval", metrics) trainer.save_metrics("eval", metrics) if __name__ == "__main__": train(*parse_args()) ``` ### `code/causal-language-modeling/requirements.txt` ```text theme={null} accelerate>=1.8.1 datasets>=2.21.0 evaluate>=0.4.3 scikit_learn>=1.5.2 transformers>=4.43.3 wandb>=0.18.1 ```

🚀 Run this on FlexAI

Managed checkpoints mean you never lose a run to preemption. Jobs launch in under 60 seconds — no infra setup, built-in observability.

Get started → Talk to us
# RL Fine-Tuning with EasyR1: GRPO & DAPO for Better Reasoning Source: https://docs.flex.ai/blueprints/easyR1 Fine-tune LLMs with reinforcement learning using EasyR1 on FlexAI. GRPO and DAPO algorithms, better reasoning, distributed training with FSDP and vLLM. This experiment demonstrates how to use FlexAI to fine-tune language models using reinforcement learning (RL) techniques with [EasyR1](https://github.com/hiyouga/EasyR1), a framework for training reasoning-capable models using GRPO (Group Relative Policy Optimization), DAPO, and REINFORCE algorithms. For illustration purposes, we'll fine-tune the `Qwen2.5-7B-Instruct` model on mathematical reasoning tasks using the `math12k` dataset with GRPO algorithm to improve reasoning capabilities. If you haven't already connected FlexAI to GitHub, run `flexai code-registry connect` to set up a code registry connection. This allows FlexAI to pull repositories directly using the repository URL in training commands. ## Quick Start Run GRPO training on Qwen2.5-7B with this single command: ```bash theme={null} flexai training run grpo \ --accels 8 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --env FORCE_TORCHRUN=1 \ --secret WANDB_API_KEY= \ --secret HF_TOKEN= \ --requirements-path code/easyR1/requirements.txt \ --runtime pytorch-28-vllm-0110-nvidia \ -- python3 -m verl.trainer.main \ config=code/easyR1/config.yaml \ worker.actor.model.model_path=Qwen/Qwen2.5-7B-Instruct ``` Replace `` and `` with your actual values. ## What is EasyR1? EasyR1 is a reinforcement learning framework specifically designed for training language models with enhanced reasoning capabilities. It implements several RL algorithms optimized for LLMs: * **GRPO (Group Relative Policy Optimization)**: Efficient policy optimization using group-based advantage estimation * **DAPO (Data-Augmented Policy Optimization)**: Enhanced training with data augmentation strategies * **REINFORCE**: Classic policy gradient method for LLM fine-tuning The framework is built on top of [VERL (Versatile Efficient Reinforcement Learning)](https://github.com/volcengine/verl), providing distributed training capabilities with FSDP and vLLM integration. ## Directory Structure The `code/easyR1/` directory contains: * `config.yaml` - Main GRPO training configuration * `format_prompt/` - Jinja templates for prompt formatting * `reward_function/` - Custom reward scoring functions For baseline training scripts and additional examples, refer to the [EasyR1 GitHub repository](https://github.com/hiyouga/EasyR1). EasyR1 uses a comprehensive YAML configuration file that controls all aspects of RL training. The main configuration file is located at `code/easyR1/config.yaml` in this repository. ### Key Configuration Sections #### Data Configuration ```yaml theme={null} data: train_files: hiyouga/math12k@train val_files: hiyouga/math12k@test prompt_key: problem answer_key: answer format_prompt: ./code/easyR1/format_prompt/math.jinja max_prompt_length: 2048 max_response_length: 2048 rollout_batch_size: 512 ``` #### Algorithm Settings ```yaml theme={null} algorithm: adv_estimator: grpo # GRPO, DAPO, or REINFORCE use_kl_loss: true kl_coef: 1.0e-2 ``` #### Worker Configuration ```yaml theme={null} worker: actor: model: model_path: Qwen/Qwen2.5-7B-Instruct optim: lr: 1.0e-6 rollout: n: 5 # number of rollout samples per prompt temperature: 1.0 reward: reward_type: batch reward_function: ./code/easyR1/reward_function/math.py:compute_score ``` For pre-configured training scripts and baseline examples, refer to the [EasyR1 repository](https://github.com/hiyouga/EasyR1). The repository provides multiple baseline configurations for different models and tasks: ### Available Baselines (in EasyR1 repo) * **Mathematical Reasoning**: `qwen2_5_7b_math_grpo.sh`, `qwen3_4b_math_grpo.sh` * **Geometric Reasoning (Vision-Language)**: `qwen2_5_vl_7b_geo3k_grpo.sh`, `qwen2_5_vl_7b_geo3k_dapo.sh`, `qwen2_5_vl_7b_geo3k_reinforce.sh` * **Multi-Image Tasks**: `qwen2_5_vl_7b_multi_image.sh` You can adapt these examples to work with FlexAI by following the training commands in this blueprint. For your specific use case, you may want to create a custom configuration. Here's how to customize the `config.yaml`: ### Custom Dataset Replace the dataset configuration: ```yaml theme={null} data: train_files: your-username/your-dataset@train val_files: your-username/your-dataset@test prompt_key: question # adjust based on your dataset answer_key: solution # adjust based on your dataset ``` ### Custom Reward Function Create your own reward function in `code/easyR1/reward_function/custom.py`: ```python theme={null} def compute_score(prompts, responses, answers): """ Args: prompts: List of input prompts responses: List of model responses answers: List of ground truth answers Returns: List of reward scores (float) """ scores = [] for response, answer in zip(responses, answers): # Your custom reward logic here score = your_evaluation_function(response, answer) scores.append(score) return scores ``` Then update the config to reference your custom reward function: ```yaml theme={null} worker: reward: reward_function: ./code/easyR1/reward_function/custom.py:compute_score ``` ### Custom Prompt Format Create a custom Jinja template in `code/easyR1/format_prompt/custom.jinja`: ``` {{ "{{" }} problem {{ "}}" }} Please solve this step by step and provide your final answer. ``` Update the config: ```yaml theme={null} data: format_prompt: ./code/easyR1/format_prompt/custom.jinja ``` ## Create Secrets To access HuggingFace models and datasets, you need a HuggingFace token. Use the [`flexai secret create` command](https://docs.flex.ai/cli/reference/secret/create) to store your *HuggingFace Token* as a secret: ```bash theme={null} flexai secret create ``` Then paste your *HuggingFace Token* API key value. Use the same command to store your Weights & Biases (wandb) API key as a secret: ```bash theme={null} flexai secret create ``` Then paste your Weights & Biases API key value. ## \[Optional] Pre-fetch the Model To speed up training and avoid downloading large models at runtime, you can pre-fetch your HuggingFace model to FlexAI storage: 1. **Create a HuggingFace storage provider:** ```bash theme={null} flexai storage create HF-STORAGE --provider huggingface --hf-token-name ``` 2. **Push the model checkpoint to your storage:** ```bash theme={null} flexai checkpoint push qwen25-7b-instruct --storage-provider HF-STORAGE --source-path Qwen/Qwen2.5-7B-Instruct ``` ## Training For RL training with EasyR1, we recommend using **1 node (8 × H100 GPUs)** for 7B models to handle the actor, reference model, and rollout workers efficiently. The commands below use this repository which contains all necessary configuration files in the `code/easyR1/` directory. ### Standard Training: Mathematical Reasoning with GRPO ```bash theme={null} flexai training run grpo \ --accels 8 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --env FORCE_TORCHRUN=1 \ --secret WANDB_API_KEY= \ --secret HF_TOKEN= \ --requirements-path code/easyR1/requirements.txt \ --runtime pytorch-28-vllm-0110-nvidia \ -- python3 -m verl.trainer.main \ config=code/easyR1/config.yaml \ worker.actor.model.model_path=Qwen/Qwen2.5-7B-Instruct ``` ### Training with Model Prefetch ```bash theme={null} flexai training run grpo-prefetched \ --accels 8 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --checkpoint qwen25-7b-instruct \ --env FORCE_TORCHRUN=1 \ --secret WANDB_API_KEY= \ --secret HF_TOKEN= \ --requirements-path code/easyR1/requirements.txt \ --runtime pytorch-28-vllm-0110-nvidia \ -- python3 -m verl.trainer.main \ config=code/easyR1/config.yaml \ worker.actor.model.model_path=/input-checkpoint/qwen25-7b-instruct ``` ### Training with Custom Configuration To use a modified configuration or different dataset, override config values: ```bash theme={null} flexai training run grpo-custom \ --accels 8 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --env FORCE_TORCHRUN=1 \ --secret WANDB_API_KEY= \ --secret HF_TOKEN= \ --requirements-path code/easyR1/requirements.txt \ --runtime pytorch-28-vllm-0110-nvidia \ -- python3 -m verl.trainer.main \ config=code/easyR1/config.yaml \ worker.actor.model.model_path=Qwen/Qwen2.5-7B-Instruct \ data.train_files=your-username/your-dataset@train \ data.val_files=your-username/your-dataset@test \ trainer.experiment_name=custom-experiment ``` ## Monitoring Training Progress You can check the status and lifecycle events of your Training Job: ```bash theme={null} flexai training inspect grpo ``` View the logs of your Training Job: ```bash theme={null} flexai training logs grpo ``` ### Training Observability with Weights & Biases EasyR1 supports Weights & Biases (wandb) integration for detailed training metrics visualization. The configuration already includes wandb logging: ```yaml theme={null} trainer: logger: ["file", "wandb"] project_name: easy_r1 experiment_name: qwen2_5_7b_math_grpo ``` ## Getting Training Checkpoints Once the Training Job completes successfully, you can list all produced checkpoints: ```bash theme={null} flexai training checkpoints grpo ``` Look for checkpoints marked as `INFERENCE READY = true` - these are ready for serving. ## Serving the Trained Model Deploy your RL-trained model directly from the checkpoint using FlexAI inference. Replace `` with the ID from an inference-ready checkpoint: ```bash theme={null} flexai inference serve easyr1-reasoning-endpoint --checkpoint ``` Monitor your inference endpoint status: ```bash theme={null} # List all inference endpoints flexai inference list # Get detailed endpoint information flexai inference inspect easyr1-reasoning-endpoint # Check endpoint logs flexai inference logs easyr1-reasoning-endpoint ``` ## Testing Your RL-Trained Model Once the endpoint is running, you can test it with reasoning tasks. For our mathematical reasoning example, the model should demonstrate improved step-by-step reasoning and accurate problem-solving. ### Before and After Training Comparison To illustrate the improvement from RL fine-tuning, here's a comparison using a math problem: **Problem**: "If a train travels 120 miles in 2 hours, what is its average speed in miles per hour?" **Base Model Response (Qwen2.5-7B-Instruct before RL training):** ``` The average speed is 60 mph. ``` *Issues: Correct answer but no reasoning steps shown* **RL Fine-tuned Model Response (after GRPO training on math12k):** ``` Let me solve this step by step: Step 1: Identify the given information - Distance traveled = 120 miles - Time taken = 2 hours Step 2: Apply the speed formula Speed = Distance / Time Step 3: Calculate Speed = 120 miles / 2 hours = 60 miles per hour Therefore, the average speed of the train is 60 mph. ``` *Improvements: Clear reasoning steps, structured approach, educational value* This demonstrates how RL training encourages the model to show its reasoning process, making it more reliable and transparent. ### Example API Call ```bash theme={null} curl -X POST "https://your-endpoint-url/v1/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_API_KEY" \ -d '{ "prompt": "Solve the following problem step by step: A rectangle has a length of 15 cm and a width of 8 cm. What is its area?", "max_tokens": 500, "temperature": 0.7 }' ``` ## Expected Results After RL fine-tuning with EasyR1, your model should achieve: * **Enhanced Reasoning**: Step-by-step problem-solving with clear explanations * **Improved Accuracy**: Higher success rate on reasoning tasks * **Better Generalization**: Ability to apply learned reasoning patterns to new problems * **Structured Outputs**: More organized and educational responses For mathematical reasoning tasks: * **Explicit Step-by-Step Solutions**: Clear breakdown of problem-solving process * **Higher Success Rate**: Improved accuracy on math benchmarks * **Better Error Detection**: Ability to identify and correct mistakes ## Technical Details ### Training Configuration Breakdown **Reinforcement Learning Components:** * **Actor Model**: The model being trained (policy network) * **Reference Model**: Frozen copy for KL divergence computation * **Rollout Workers**: Generate multiple responses for each prompt (n=5) * **Reward Function**: Evaluates response quality (custom per task) **Distributed Training:** * **FSDP (Fully Sharded Data Parallel)**: Efficient memory usage for large models * **vLLM Integration**: Fast inference during rollout generation * **Tensor Parallelism**: For rollout workers (size=2) **Optimization:** * **GRPO Algorithm**: Group-based advantage estimation for stable training * **KL Penalty**: Prevents model from deviating too far from base model * **Gradient Checkpointing**: Reduces memory usage during backpropagation ### Resource Requirements **Recommended Configuration for Qwen2.5-7B:** * **Nodes**: 1 node (sufficient for RL training with actor + reference + rollout) * **Accelerators**: 8 × H100 GPUs per node * **Memory**: \~400GB+ GPU memory total (actor, reference, and rollout workers) * **Training Time**: \~8-12 hours for 15 epochs * **Storage**: \~50GB for checkpoints **Command Line Parameters Explained:** * `FORCE_TORCHRUN=1`: Ensures proper distributed training setup * `--runtime pytorch-28-vllm-0110-nvidia`: PyTorch 2.8 with vLLM 0.11.0 optimized for EasyR1 * `--repository-url`: Points to the FlexAI blueprints repository * `config=code/easyR1/config.yaml`: Main configuration file path relative to repository root ### Key Configuration Parameters **Data Settings:** * `rollout_batch_size: 512`: Number of prompts per training iteration * `max_prompt_length: 2048`: Maximum input length * `max_response_length: 2048`: Maximum output length **Algorithm Settings:** * `adv_estimator: grpo`: Choice of RL algorithm * `kl_coef: 1.0e-2`: Strength of KL penalty * `use_kl_loss: true`: Enable KL divergence loss **Training Settings:** * `total_epochs: 15`: Number of training epochs * `n_gpus_per_node: 8`: GPUs per node * `val_freq: 5`: Validation every 5 epochs * `save_freq: 5`: Save checkpoint every 5 epochs ### Scaling Options * **For faster training**: Increase to 2 nodes (16 × H100) * **For larger models**: Increase `tensor_parallel_size` for rollout * **For better exploration**: Increase `rollout.n` (more samples per prompt) * **For memory efficiency**: Enable CPU offloading (`enable_cpu_offload: true`) * **For different tasks**: Modify reward function and prompt templates ## Advanced Examples ### Vision-Language Model with Geometric Reasoning ```bash theme={null} flexai training run grpo-VL-Geo \ --accels 8 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --env FORCE_TORCHRUN=1 \ --secret WANDB_API_KEY= \ --secret HF_TOKEN= \ --requirements-path code/easyR1/requirements.txt \ --runtime pytorch-28-vllm-0110-nvidia \ -- python3 -m verl.trainer.main \ config=code/easyR1/config.yaml \ worker.actor.model.model_path=Qwen/Qwen2.5-VL-7B-Instruct \ data.train_files=hiyouga/geometry3k@train \ data.val_files=hiyouga/geometry3k@test \ data.format_prompt=./code/easyR1/format_prompt/r1v.jinja \ worker.reward.reward_function=./code/easyR1/reward_function/r1v.py:compute_score \ trainer.experiment_name=qwen2_5_vl_7b_geo3k_grpo ``` ### Using DAPO Algorithm ```bash theme={null} flexai training run Dapo-14B \ --accels 8 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --env FORCE_TORCHRUN=1 \ --secret WANDB_API_KEY= \ --secret HF_TOKEN= \ --requirements-path code/easyR1/requirements.txt \ --runtime pytorch-28-vllm-0110-nvidia \ -- python3 -m verl.trainer.main \ config=code/easyR1/config.yaml \ worker.actor.model.model_path=Qwen/Qwen3-14B \ algorithm.adv_estimator=dapo \ algorithm.online_filtering=true \ data.train_files=hiyouga/dapo17k@train \ data.val_files=hiyouga/dapo17k@test \ data.format_prompt=./code/easyR1/format_prompt/dapo.jinja \ worker.reward.reward_function=./code/easyR1/reward_function/dapo.py:compute_score \ trainer.experiment_name=qwen3_14b_dapo17k_dapo ``` ## Troubleshooting **Training Job Fails to Start:** ```bash theme={null} # Check FlexAI authentication flexai auth status # Verify repository access git clone https://github.com/flexaihq/blueprints ``` **Out of Memory Errors:** * Reduce `rollout_batch_size` from 512 to 256 * Reduce `rollout.n` from 5 to 3 (fewer samples per prompt) * Enable CPU offloading: `enable_cpu_offload: true` in FSDP config * Reduce `tensor_parallel_size` for rollout workers **Reward Function Errors:** * Verify reward function path is correct in config * Test reward function locally before training * Ensure reward function returns float scores for all inputs * Check for NaN or infinite reward values **Checkpoint Not Inference Ready:** * Wait for training to complete fully * Check `save_model_only: false` in config to include all necessary files * Verify training completed without errors **Endpoint Deployment Issues:** * Verify checkpoint shows `INFERENCE READY = true` status * Check FlexAI cluster availability * Review detailed logs with `flexai inference logs ` **Dataset Loading Issues:** * Verify dataset path format: `username/dataset@split` * Ensure HuggingFace token has access to datasets * Check prompt\_key and answer\_key match your dataset schema **vLLM Rollout Errors:** * Adjust `gpu_memory_utilization` (default 0.6) * Reduce `tensor_parallel_size` if GPUs are insufficient * Enable `enforce_eager: true` for debugging ## References * **EasyR1 GitHub**: [https://github.com/hiyouga/EasyR1](https://github.com/hiyouga/EasyR1) * **VERL Framework**: [https://github.com/volcengine/verl](https://github.com/volcengine/verl) * **FlexAI Documentation**: [https://docs.flex.ai](https://docs.flex.ai) * **HybridFlow Paper**: [https://arxiv.org/abs/2409.19256](https://arxiv.org/abs/2409.19256) * **GRPO Algorithm**: Introduced in DeepSeekMath paper - [https://arxiv.org/abs/2402.03300](https://arxiv.org/abs/2402.03300) * **GRPO Documentation**: [https://huggingface.co/docs/trl/grpo\_trainer](https://huggingface.co/docs/trl/grpo_trainer) ## Code ### `requirements.txt` ```text theme={null} git+https://github.com/hiyouga/EasyR1.git@d146d24e990c8102fee44e61e5ca389907712960 ``` ### `config.yaml` ```yaml theme={null} --- data: train_files: hiyouga/math12k@train val_files: hiyouga/math12k@test prompt_key: problem answer_key: answer image_key: images video_key: videos image_dir: video_fps: 2.0 max_prompt_length: 2048 max_response_length: 2048 rollout_batch_size: 512 mini_rollout_batch_size: val_batch_size: 1024 format_prompt: ./code/easyR1/format_prompt/math.jinja override_chat_template: shuffle: true seed: 1 min_pixels: 262144 max_pixels: 4194304 filter_overlong_prompts: true algorithm: adv_estimator: grpo disable_kl: false use_kl_loss: true kl_penalty: low_var_kl kl_coef: 1.0e-2 online_filtering: false filter_key: overall filter_low: 0.01 filter_high: 0.99 worker: actor: global_batch_size: 128 micro_batch_size_per_device_for_update: 1 micro_batch_size_per_device_for_experience: 2 max_grad_norm: 1.0 padding_free: true dynamic_batching: true ulysses_size: 1 model: model_path: Qwen/Qwen2.5-7B-Instruct enable_gradient_checkpointing: true trust_remote_code: false freeze_vision_tower: false optim: lr: 1.0e-6 weight_decay: 1.0e-2 strategy: adamw lr_warmup_ratio: 0.0 fsdp: enable_full_shard: true enable_cpu_offload: false enable_rank0_init: true offload: offload_params: true offload_optimizer: true rollout: n: 5 temperature: 1.0 top_p: 1.0 limit_images: 0 gpu_memory_utilization: 0.6 enforce_eager: false enable_chunked_prefill: false tensor_parallel_size: 2 disable_tqdm: true val_override_config: temperature: 0.6 top_p: 0.95 n: 1 ref: fsdp: enable_full_shard: true enable_cpu_offload: true enable_rank0_init: true offload: offload_params: false reward: reward_type: batch reward_function: ./code/easyR1/reward_function/math.py:compute_score trainer: total_epochs: 15 max_steps: project_name: easy_r1 experiment_name: qwen2_5_7b_math_grpo logger: [file, wandb] nnodes: 1 n_gpus_per_node: 8 max_try_make_batch: 20 val_freq: 5 val_before_train: true val_only: false val_generations_to_log: 3 save_freq: 5 save_limit: 3 save_model_only: false save_checkpoint_path: /output-checkpoint load_checkpoint_path: find_last_checkpoint: true ``` ### `format_prompt/math.jinja` ``` {{ "{{" }} content | trim {{ "}}" }} You FIRST think about the reasoning process as an internal monologue and then provide the final answer. The reasoning process MUST BE enclosed within tags. The final answer MUST BE put in \boxed{}. ``` ### `reward_function/math.py` ```python theme={null} # Copyright 2024 Bytedance Ltd. and/or its affiliates # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 import re from typing import Any from mathruler.grader import extract_boxed_content, grade_answer def format_reward(response: str) -> float: pattern = re.compile(r".*.*\\boxed\{.*\}.*", re.DOTALL) format_match = re.fullmatch(pattern, response) return 1.0 if format_match else 0.0 def accuracy_reward(response: str, ground_truth: str) -> float: answer = extract_boxed_content(response) return 1.0 if grade_answer(answer, ground_truth) else 0.0 def compute_score( reward_inputs: list[dict[str, Any]], format_weight: float = 0.1 ) -> list[dict[str, float]]: if not isinstance(reward_inputs, list): raise ValueError("Please use `reward_type=batch` for math reward function.") scores = [] for reward_input in reward_inputs: response = re.sub( r"\s*(<|>|/)\s*", r"\1", reward_input["response"] ) format_score = format_reward(response) accuracy_score = accuracy_reward(response, reward_input["ground_truth"]) scores.append( { "overall": (1 - format_weight) * accuracy_score + format_weight * format_score, "format": format_score, "accuracy": accuracy_score, } ) return scores ```

🚀 Run this on FlexAI

Managed checkpoints mean you never lose a run to preemption. Jobs launch in under 60 seconds — no infra setup, built-in observability.

Get started → Talk to us
# Fine-Tune LLMs with Flash Attention on FlexAI — Faster Training Source: https://docs.flex.ai/blueprints/flash-attention-ft-on-a-language-model Fine-tune a causal language model using Flash Attention on FlexAI. Faster training, lower memory, managed checkpoints, full walkthrough from setup to deploy. This experiment is temporarily disabled. This experiment demonstrates how easy it is to leverage **FlexAI** to run a Training Job making use of *Flash Attention* through the [flash-attention](https://github.com/Dao-AILab/flash-attention) package with a couple of commands. We will use an example of training a causal language model (LLM) on the `wikitext` dataset using the `GPT-2` model. You will see that this straightforward process only requires two components: a training script and a dataset. The training script is responsible for defining the model, setting up and applying hyperparameters, running the training loop, and applying its respective evaluation logic, while the dataset contains the information that will be used to train the model. If you haven't already connected FlexAI to GitHub, you'll need to set up a code registry connection: ```bash theme={null} flexai code-registry connect ``` This will allow FlexAI to pull repositories directly from GitHub using the `-u` flag in training commands. In this experiment, we will use a pre-processed version of the the `wikitext` dataset that has been set up for the `GPT-2` model. > If you'd like to reproduce the pre-processing steps yourself to use a different dataset or simply to learn more about the process, you can refer to the [Manual Dataset Pre-processing](#manual-dataset-pre-processing) section below. 1. Download the dataset: ```bash theme={null} DATASET_NAME=gpt2-tokenized-wikitext && curl -L -o ${DATASET_NAME}.zip "https://bucket-docs-samples-99b3a05.s3.eu-west-1.amazonaws.com/${DATASET_NAME}.zip" && unzip ${DATASET_NAME}.zip && rm ${DATASET_NAME}.zip ``` 2. Upload the dataset (located in `gpt2-tokenized-wikitext/`) to FlexAI Storage as a new dataset: ```bash theme={null} flexai dataset push gpt2-tokenized-wikitext --file gpt2-tokenized-wikitext ``` Now, it's time to train your LLM on the dataset you just *pushed* in the previous step, `gpt2-tokenized-wikitext`. This experiment uses the `GPT-2` model, however, the training script we will use leverages the HuggingFace Transformers `Trainer` class, which makes it easy to replace `GPT-2` with another model compatible with `flash-attention`. To start the Training Job, run the following command: ```bash theme={null} flexai training run flexai-experiments-flash-attention --repository-url https://github.com/flexaihq/blueprints --dataset gpt2-tokenized-wikitext --requirements-path code/causal-language-modeling/requirements-flash-attn.txt \ -- code/causal-language-modeling/train.py \ --do_eval \ --do_train \ --dataset_name wikitext \ --tokenized_dataset_load_dir /input/gpt2-tokenized-wikitext \ --model_name_or_path openai-community/gpt2 \ --output_dir /output-checkpoint \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 8 \ --logging_steps 50 \ --save_steps 500 \ --eval_steps 500 \ --attn_implementation flash_attention_2 \ --torch_dtype float16 \ --eval_strategy steps ``` The first line defines the 3 main components required to run a Training Job in FlexAI Storage: 1. The Training Job's name (`flexai-experiments-flash-attention`). 2. The URL of the repository containing the training script (`https://github.com/flexaihq/blueprints`). 3. The name of the dataset to be used (`gpt2-tokenized-wikitext`). The second line defines the script that will be executed when the Training Job is started (`code/causal-language-modeling/train.py`). After the second line come the script's arguments, which are passed to the script when it is executed to adjust the Training Job hyperparameters or customize its behavior. For instance, `--max_train_samples` and `--max_eval_samples` can be used to tweak the sample size. You can check the status and life cycle events of your Training Job by running: ```bash theme={null} flexai training inspect flexai-experiments-flash-attention ``` Additionally, you can view the logs of your Training Job by running: ```bash theme={null} flexai training logs flexai-experiments-flash-attention ``` Once the Training Job completes successfully, you will be able to download its output artifacts by running: ```bash theme={null} flexai training fetch flexai-experiments-flash-attention ``` This will download a `zip` file containing the trained model artifacts to your current working directory. You can now have a look at other Experiments within this repository to explore other use cases and techniques. ## Optional Extra Steps ### Manual Dataset Pre-processing To prepare and save the `wikitext` dataset for the `GPT-2` model run the following command: ```bash theme={null} python code/dataset/prepare_save_dataset.py \ --dataset_name wikitext \ --tokenized_dataset_save_dir gpt2-tokenized-wikitext \ --dataset_config_name wikitext-2-raw-v1 \ --tokenizer_model_name openai-community/gpt2 \ --dataset_group_text true ``` The generated dataset will be created in the directory set as the value of `--tokenized_dataset_save_dir`, in this case: `gpt2-tokenized-wikitext`. Keep in mind that you can use other combinations of datasets and models available on HuggingFace. ## Code ### `code/causal-language-modeling/train.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import math import os import sys from dataclasses import dataclass, field from typing import Optional import evaluate import numpy import torch import transformers from transformers import ( AutoConfig, AutoModelForCausalLM, AutoTokenizer, HfArgumentParser, Trainer, TrainingArguments, default_data_collator, ) SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) sys.path.append(os.path.dirname(SCRIPT_DIR)) from dataset.prepare_save_dataset import DatasetArguments, load_and_tokenize from utils.experiment_tracking import set_wandb transformers.logging.set_verbosity_info() @dataclass class ModelArguments: model_name_or_path: str = field(default=None) torch_dtype: Optional[str] = field(default=None) attn_implementation: Optional[str] = field(default=None) @dataclass class AdditionalArguments: max_train_samples: Optional[int] = field(default=None) max_eval_samples: Optional[int] = field(default=None) def parse_args(): parser = HfArgumentParser( (DatasetArguments, ModelArguments, TrainingArguments, AdditionalArguments) ) return parser.parse_args_into_dataclasses() def _load_model_and_tokenizer(model_args, print_model=False): config = AutoConfig.from_pretrained(model_args.model_name_or_path) tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path) tokenizer.pad_token = tokenizer.eos_token tokenizer.pad_token_id = tokenizer.eos_token_id torch_dtype = ( model_args.torch_dtype if model_args.torch_dtype in ["auto", None] else getattr(torch, model_args.torch_dtype) ) extra_model_args = {} if model_args.attn_implementation is not None: extra_model_args["attn_implementation"] = model_args.attn_implementation model = AutoModelForCausalLM.from_pretrained( model_args.model_name_or_path, config=config, torch_dtype=torch_dtype, **extra_model_args, ) if print_model: print(model) return model, tokenizer def train(dataset_args, model_args, training_args, additional_args): set_wandb(training_args) print(f"Training/evaluation parameters {training_args}") train_dataset, eval_dataset = load_and_tokenize( tokenizer_model_name=model_args.model_name_or_path, do_eval=training_args.do_eval, **vars(dataset_args), ) max_train_samples = float("inf") max_eval_samples = float("inf") if not dataset_args.dataset_streaming: max_train_samples = len(train_dataset) if training_args.do_eval: max_eval_samples = len(eval_dataset) if additional_args.max_train_samples is not None: max_train_samples = min(max_train_samples, additional_args.max_train_samples) train_dataset = train_dataset.take(max_train_samples) if additional_args.max_eval_samples is not None: assert training_args.do_eval, "Cannot set max_eval_samples without do_eval" max_eval_samples = min(max_eval_samples, additional_args.max_eval_samples) eval_dataset = eval_dataset.take(max_eval_samples) model, tokenizer = _load_model_and_tokenizer(model_args, print_model=True) metric = evaluate.load("accuracy") def preprocess_logits_for_metrics(logits, labels): if isinstance(logits, tuple): logits = logits[0] return logits.argmax(dim=-1) def compute_metrics(eval_preds): preds, labels = eval_preds mask = (labels != tokenizer.pad_token_id) & (labels != -100) labels = numpy.concatenate([label[mask[i]][1:] for i, label in enumerate(labels)]) preds = numpy.concatenate([pred[mask[i]][:-1] for i, pred in enumerate(preds)]) return metric.compute(predictions=preds, references=labels) trainer = Trainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset, tokenizer=tokenizer, data_collator=default_data_collator, compute_metrics=compute_metrics, preprocess_logits_for_metrics=preprocess_logits_for_metrics, ) train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint) trainer.save_model() metrics = train_result.metrics metrics["train_samples"] = max_train_samples trainer.log_metrics("train", metrics) trainer.save_metrics("train", metrics) trainer.save_state() if training_args.do_eval: metrics = trainer.evaluate() metrics["eval_samples"] = max_eval_samples try: perplexity = math.exp(metrics["eval_loss"]) except OverflowError: perplexity = float("inf") metrics["perplexity"] = perplexity trainer.log_metrics("eval", metrics) trainer.save_metrics("eval", metrics) if __name__ == "__main__": train(*parse_args()) ``` ### `code/causal-language-modeling/requirements-flash-attn.txt` ```text theme={null} accelerate>=1.8.1 datasets>=2.21.0 evaluate>=0.4.3 # flash-attn==2.7.4.post1 disable for now. scikit_learn>=1.5.2 transformers>=4.43.3 wandb>=0.18.1 ``` # Fine-Tune a Text-to-Speech Model on FlexAI Source: https://docs.flex.ai/blueprints/ft-on-a-tts-model Fine-tune a TTS model on FlexAI with managed checkpoints and GPU access. Full tutorial from dataset prep to deployment, no infra setup. The goal of this experiment is to fine-tune the `parler_tts_mini_v0.1` model to create a French version. The model generates high-quality speech from input text, which can be controlled using a description prompt (e.g., gender, speaking rate, etc.). The training uses the a `text-to-speech` dataset in French, enabling the model to produce natural and expressive speech in this language. If you haven't already connected FlexAI to GitHub, you'll need to set up a code registry connection: ```bash theme={null} flexai code-registry connect ``` This will allow FlexAI to pull repositories directly from GitHub using the `-u` flag in training commands. You can download the pre-processed version of the dataset by running the following command: ```bash theme={null} DATASET_NAME=text-to-speech-fr && curl -L -o ${DATASET_NAME}.zip "https://bucket-docs-samples-99b3a05.s3.eu-west-1.amazonaws.com/${DATASET_NAME}.zip" && unzip ${DATASET_NAME}.zip && rm ${DATASET_NAME}.zip ``` > If you'd like to reproduce the pre-processing steps yourself to use a different dataset or simply to learn more about the process, you can refer to the [Manual Dataset Pre-processing](#manual-dataset-pre-processing) section below. Next, push the contents of the `text-to-speech-fr/` directory as a new FlexAI dataset: ```bash theme={null} flexai dataset push text-to-speech-fr --file text-to-speech-fr ``` ## Training To start the Training Job, run the following command: ```bash theme={null} flexai training run text-to-speech-ddp --repository-url https://github.com/flexaihq/blueprints --dataset text-to-speech-fr --secret WANDB_API_KEY= --requirements-path code/text-to-speech/requirements.txt \ --nodes 1 --accels 8 \ -- code/text-to-speech/run_parler_tts_training.py ./code/text-to-speech/french_training.json ``` Instead of passing a `.json` file as input, you can also set the arguments manually. For example: ```bash theme={null} flexai training run text-to-speech-ddp --repository-url https://github.com/flexaihq/blueprints --dataset text-to-speech-fr --secret WANDB_API_KEY= --requirements-path code/text-to-speech/requirements.txt \ --nodes 1 --accels 8 \ -- code/text-to-speech/run_parler_tts_training.py \ --model_name_or_path=parler-tts/parler_tts_mini_v0.1 \ --save_to_disk=/input/text-to-speech-fr \ --temporary_save_to_disk=./audio_code_tmp/ \ --wandb_project=parler-francais \ --feature_extractor_name=ylacombe/dac_44khZ_8kbps \ --description_tokenizer_name=google/flan-t5-large \ --prompt_tokenizer_name=google/flan-t5-large \ --report_to=wandb \ --overwrite_output_dir \ --output_dir=/output-checkpoint \ --train_dataset_name=PHBJT/cml-tts-20percent-subset \ --train_metadata_dataset_name=PHBJT/cml-tts-20percent-subset-description \ --train_dataset_config_name=default \ --train_split_name=train \ --eval_dataset_name=PHBJT/cml-tts-20percent-subset \ --eval_metadata_dataset_name=PHBJT/cml-tts-20percent-subset-description \ --eval_dataset_config_name=default \ --eval_split_name=test \ --target_audio_column_name=audio \ --description_column_name=text_description \ --prompt_column_name=text \ --max_eval_samples=10 ``` ## Optional Extra Steps You can run these extra steps in a FlexAI Interactive Session or in a local env (e.g. `pipenv install --python 3.10`), if you have hardware that's capable of doing inference. ### Inference A simple inference script that you can easily adapt to your needs is available at `code/text-to-speech/predict.py`. ### Manual Dataset Pre-processing If you'd prefer to perform the dataset pre-processing step yourself, you can follow these instructions. #### Clone this repository If you haven't already, clone this repository on your host machine: ```bash theme={null} git clone https://github.com/flexaihq/blueprints.git blueprints --depth 1 --branch main && cd blueprints ``` #### Install the dependencies Depending on your environment, you might need to install - if not already - the experiments' dependencies by running: ```bash theme={null} pip install -r code/text-to-speech/requirements.txt ``` #### Dataset preparation Prepare the dataset by running the training command with the `--preprocessing_only` flag in `./code/text-to-speech/french_training.json`. For large datasets, it is recommended to run the preprocessing on a single machine to avoid timeouts when running the script in distributed mode. The content will be saved to the destination specified in `--save_to_disk=./text-to-speech-fr/`. Run the dataset preparation using: ```bash theme={null} python code/text-to-speech/run_parler_tts_training.py ./code/text-to-speech/french_training.json ``` Make sure to remove the `--preprocessing_only` flag before attempting to run the script for training purposes. ## Code ### `code/text-to-speech/french_training.json` ```json theme={null} { "model_name_or_path": "parler-tts/parler_tts_mini_v0.1", "save_to_disk": "/input/text-to-speech-fr", "temporary_save_to_disk": "./audio_code_tmp/", "wandb_project": "parler-francais", "feature_extractor_name": "ylacombe/dac_44khZ_8kbps", "description_tokenizer_name": "google/flan-t5-large", "prompt_tokenizer_name": "google/flan-t5-large", "report_to": [ "wandb" ], "overwrite_output_dir": true, "output_dir": "/output-checkpoint", "train_dataset_name": "PHBJT/cml-tts-20percent-subset", "train_metadata_dataset_name": "PHBJT/cml-tts-20percent-subset-description", "train_dataset_config_name": "default", "train_split_name": "train", "eval_dataset_name": "PHBJT/cml-tts-20percent-subset", "eval_metadata_dataset_name": "PHBJT/cml-tts-20percent-subset-description", "eval_dataset_config_name": "default", "eval_split_name": "test", "target_audio_column_name": "audio", "description_column_name": "text_description", "prompt_column_name": "text", "max_eval_samples": 10, "max_duration_in_seconds": 30, "min_duration_in_seconds": 2.0, "max_text_length": 600, "group_by_length": true, "add_audio_samples_to_wandb": true, "preprocessing_num_workers": 8, "do_train": true, "num_train_epochs": 100, "gradient_accumulation_steps": 4, "gradient_checkpointing": false, "per_device_train_batch_size": 6, "learning_rate": 0.00095, "adam_beta1": 0.9, "adam_beta2": 0.99, "weight_decay": 0.01, "lr_scheduler_type": "constant_with_warmup", "warmup_steps": 500, "logging_steps": 100, "freeze_text_encoder": true, "do_eval": true, "predict_with_generate": true, "include_inputs_for_metrics": true, "evaluation_strategy": "steps", "eval_steps": 1000, "save_steps": 1000, "per_device_eval_batch_size": 4, "audio_encoder_per_device_batch_size": 24, "dtype": "bfloat16", "seed": 456, "dataloader_num_workers": 8, "attn_implementation": "sdpa" } ``` ### `code/text-to-speech/predict.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT # This is a basic inference script you can easily modify for your needs import soundfile as sf import torch from parler_tts import ParlerTTSForConditionalGeneration from transformers import AutoTokenizer checkpoint_path = "YOUR CHECKPOINT PATH" device = "cuda:0" if torch.cuda.is_available() else "cpu" model = ParlerTTSForConditionalGeneration.from_pretrained(checkpoint_path).to(device) tokenizer = AutoTokenizer.from_pretrained(checkpoint_path) prompt = "Salut, comment vas-tu aujourd'hui ?" description = "A man speaking at a moderate speed with moderate pitch, very clear audio recording that has no background noise." input_ids = tokenizer(description, return_tensors="pt").input_ids.to(device) prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device) attention_mask = input_ids.clone().fill_(1) generation = model.generate( input_ids=input_ids, prompt_input_ids=prompt_input_ids, attention_mask=attention_mask, ) audio_arr = generation.cpu().numpy().squeeze() sf.write("audio.wav", audio_arr, model.config.sampling_rate) ``` ### `code/text-to-speech/requirements.txt` ```text theme={null} accelerate==1.0.1 datasets>=2.21.0 evaluate>=0.4.3 jiwer>=3.0.4 parler-tts @ git+https://github.com/huggingface/parler-tts.git@5d0aca9753ab74ded179732f5bd797f7a8c6f8ee wandb>=0.18.1 ```

🚀 Run this on FlexAI

Managed checkpoints mean you never lose a run to preemption. Jobs launch in under 60 seconds — no infra setup, built-in observability.

Get started → Talk to us
# Track Training Experiments with Weights & Biases on FlexAI Source: https://docs.flex.ai/blueprints/integrating-a-experiment-tracker Integrate Weights & Biases experiment tracking with FlexAI training jobs. Monitor loss curves, compare runs, and share results — zero extra infra. ## Experiment Tracking *Experiment tracking* involves systematically recording and managing details of machine learning experiments, such as code, data, configurations, parameters, metrics, and results. It ensures reproducibility, comparability, and accountability across experiments, aiding in efficient model development and deployment. Weights & Biases (*wandb*) is one approach to achieving this. Follow the next instructions to log experiments to your *wandb* account. ## Setting Up the Weights and Biases Secret To enable seamless integration with *wandb* in your experiments, follow these steps to create the *wandb* secret: 1. **Retrieve Your API Key** Visit your [Weights & Biases Settings page](https://app.wandb.ai/settings) to find your API key. Copy the key for use in the next step. 2. **Create the Secret** Use the [`flexai secret create` command](https://docs.flex.ai/cli/commands/secret/) to store your *wandb* API key as a secret. Replace `` with your desired name for the secret: ```bash theme={null} flexai secret create ``` Then past your *wandb* API key value. 3. **Note on Project Name** Keep in mind that the project name used in your *wandb* setup does not need to be an FlexAI Secret. Additionally, the project name does not need to be pre-created in *wandb* — it will be automatically created if it doesn't exist when you log your first experiment. ## Log to Weights and Biases You can now log experiments to your *wandb* account by adding the following flags to any `flexai training run` command: ```bash theme={null} --secret WANDB_API_KEY= --env WANDB_PROJECT= ``` You can optionally set your *run name* using the `--run_name ` HuggingFace arg. For more ways to customize and configure your *wandb* environment, check out the [Weights & Biases Environment Variables Guide](https://docs.wandb.ai/guides/track/environment-variables/). ## Setting Up the Experiment If you haven't already connected FlexAI to GitHub, you'll need to set up a code registry connection: ```bash theme={null} flexai code-registry connect ``` This will allow FlexAI to pull repositories directly from GitHub using the `-u` flag in training commands. In this experiment, we will use a pre-processed version of the the `wikitext` dataset that has been set up for the `GPT-2` model. 1. Download the dataset: ```bash theme={null} DATASET_NAME=gpt2-tokenized-wikitext && curl -L -o ${DATASET_NAME}.zip "https://bucket-docs-samples-99b3a05.s3.eu-west-1.amazonaws.com/${DATASET_NAME}.zip" && unzip ${DATASET_NAME}.zip && rm ${DATASET_NAME}.zip ``` 2. Upload the dataset (located in `gpt2-tokenized-wikitext/`) to FlexAI: ```bash theme={null} flexai dataset push gpt2-tokenized-wikitext --file gpt2-tokenized-wikitext ``` ## Running the Training Job with Experiment Tracking Now that all the pieces are in place (*wandb* Secret, Source, and Dataset), you can run the training job with experiment tracking enabled. ```bash theme={null} flexai training run gpt2training-tracker --repository-url https://github.com/flexaihq/blueprints --dataset gpt2-tokenized-wikitext --secret WANDB_API_KEY= --env WANDB_PROJECT= --requirements-path code/causal-language-modeling/requirements.txt \ -- code/causal-language-modeling/train.py \ --do_eval \ --do_train \ --dataset_name wikitext \ --tokenized_dataset_load_dir /input/gpt2-tokenized-wikitext \ --model_name_or_path openai-community/gpt2 \ --output_dir /output-checkpoint \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 8 \ --logging_steps 50 \ --save_steps 500 \ --eval_steps 500 \ --eval_strategy steps \ --run_name ``` You can now visit your *wandb* dashboard and look for your project's name to follow the progress of the Training Job and analyze its results in near real-time. ## Code ### `code/causal-language-modeling/train.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import math import os import sys from dataclasses import dataclass, field from typing import Optional import evaluate import numpy import torch import transformers from transformers import ( AutoConfig, AutoModelForCausalLM, AutoTokenizer, HfArgumentParser, Trainer, TrainingArguments, default_data_collator, ) SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) sys.path.append(os.path.dirname(SCRIPT_DIR)) from dataset.prepare_save_dataset import DatasetArguments, load_and_tokenize from utils.experiment_tracking import set_wandb transformers.logging.set_verbosity_info() @dataclass class ModelArguments: model_name_or_path: str = field(default=None) torch_dtype: Optional[str] = field(default=None) attn_implementation: Optional[str] = field(default=None) @dataclass class AdditionalArguments: max_train_samples: Optional[int] = field(default=None) max_eval_samples: Optional[int] = field(default=None) def parse_args(): parser = HfArgumentParser( (DatasetArguments, ModelArguments, TrainingArguments, AdditionalArguments) ) return parser.parse_args_into_dataclasses() def _load_model_and_tokenizer(model_args, print_model=False): config = AutoConfig.from_pretrained(model_args.model_name_or_path) tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path) tokenizer.pad_token = tokenizer.eos_token tokenizer.pad_token_id = tokenizer.eos_token_id torch_dtype = ( model_args.torch_dtype if model_args.torch_dtype in ["auto", None] else getattr(torch, model_args.torch_dtype) ) extra_model_args = {} if model_args.attn_implementation is not None: extra_model_args["attn_implementation"] = model_args.attn_implementation model = AutoModelForCausalLM.from_pretrained( model_args.model_name_or_path, config=config, torch_dtype=torch_dtype, **extra_model_args, ) if print_model: print(model) return model, tokenizer def train(dataset_args, model_args, training_args, additional_args): set_wandb(training_args) print(f"Training/evaluation parameters {training_args}") train_dataset, eval_dataset = load_and_tokenize( tokenizer_model_name=model_args.model_name_or_path, do_eval=training_args.do_eval, **vars(dataset_args), ) max_train_samples = float("inf") max_eval_samples = float("inf") if not dataset_args.dataset_streaming: max_train_samples = len(train_dataset) if training_args.do_eval: max_eval_samples = len(eval_dataset) if additional_args.max_train_samples is not None: max_train_samples = min(max_train_samples, additional_args.max_train_samples) train_dataset = train_dataset.take(max_train_samples) if additional_args.max_eval_samples is not None: assert training_args.do_eval, "Cannot set max_eval_samples without do_eval" max_eval_samples = min(max_eval_samples, additional_args.max_eval_samples) eval_dataset = eval_dataset.take(max_eval_samples) model, tokenizer = _load_model_and_tokenizer(model_args, print_model=True) metric = evaluate.load("accuracy") def preprocess_logits_for_metrics(logits, labels): if isinstance(logits, tuple): logits = logits[0] return logits.argmax(dim=-1) def compute_metrics(eval_preds): preds, labels = eval_preds mask = (labels != tokenizer.pad_token_id) & (labels != -100) labels = numpy.concatenate([label[mask[i]][1:] for i, label in enumerate(labels)]) preds = numpy.concatenate([pred[mask[i]][:-1] for i, pred in enumerate(preds)]) return metric.compute(predictions=preds, references=labels) trainer = Trainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset, tokenizer=tokenizer, data_collator=default_data_collator, compute_metrics=compute_metrics, preprocess_logits_for_metrics=preprocess_logits_for_metrics, ) train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint) trainer.save_model() metrics = train_result.metrics metrics["train_samples"] = max_train_samples trainer.log_metrics("train", metrics) trainer.save_metrics("train", metrics) trainer.save_state() if training_args.do_eval: metrics = trainer.evaluate() metrics["eval_samples"] = max_eval_samples try: perplexity = math.exp(metrics["eval_loss"]) except OverflowError: perplexity = float("inf") metrics["perplexity"] = perplexity trainer.log_metrics("eval", metrics) trainer.save_metrics("eval", metrics) if __name__ == "__main__": train(*parse_args()) ``` ### `code/causal-language-modeling/requirements.txt` ```text theme={null} accelerate>=1.8.1 datasets>=2.21.0 evaluate>=0.4.3 scikit_learn>=1.5.2 transformers>=4.43.3 wandb>=0.18.1 ```

🚀 Run this on FlexAI

Managed checkpoints mean you never lose a run to preemption. Jobs launch in under 60 seconds — no infra setup, built-in observability.

Get started → Talk to us
# Fine-Tune and Deploy an LLM on FlexAI with LlamaFactory Source: https://docs.flex.ai/blueprints/llama-factory Fine-tune and deploy any LLM with LlamaFactory on FlexAI. Managed checkpoints, train and serve in one platform, no lost runs from spot preemption. [LlamaFactory](https://github.com/hiyouga/LLaMA-Factory) is a unified framework for fine-tuning 100+ open-source LLMs from a single YAML config — SFT, PPO, DPO, and more. This blueprint shows how to run it end-to-end on FlexAI: configure a training run, launch with managed checkpoints, and deploy the resulting model as a production inference endpoint. The workflow is model- and dataset-agnostic. Throughout this guide we use `Qwen/Qwen2.5-7B` + the [`openhermes-fr`](https://huggingface.co/datasets/legmlai/openhermes-fr) dataset as a worked example — swap them for any LlamaFactory-supported model and any HuggingFace dataset by editing the YAML config. > **Note**: If you haven't already connected FlexAI to GitHub, run `flexai code-registry connect` to set up a code registry connection. This allows FlexAI to pull repositories directly using the repository URL in training commands. Register your dataset in LlamaFactory's dataset registry. The example below uses `openhermes-fr`; replace it with any HuggingFace dataset by editing the `hf_hub_url` and column mapping. Navigate to `experiments/code/llama-factory/data/dataset_info.json` and verify the dataset entry exists: ```json theme={null} { "openhermes-fr": { "hf_hub_url": "legmlai/openhermes-fr", "columns": { "prompt": "prompt", "response": "accepted_completion" } } } ``` For your own use case, replace the entry with your dataset and update the `columns` mapping to match your dataset's schema. The `qwen25-7B_sft.yaml` file contains the training configuration. Swap the `model_name_or_path` and `dataset` values for your own. Key settings shown in the example: * **Model**: `Qwen/Qwen2.5-7B` — example base model (replace `model_name_or_path` with any HuggingFace model ID) * **Stage**: `sft` (Supervised Fine-Tuning) — switch to `dpo`, `ppo`, or `reward` for other alignment methods LlamaFactory supports * **Dataset**: `openhermes-fr` — example dataset (replace with your own entry in `dataset_info.json`) * **Training**: Full fine-tuning with DeepSpeed ZeRO Stage 3 for memory-efficient distributed training ```yaml theme={null} --- ### model model_name_or_path: Qwen/Qwen2.5-7B trust_remote_code: true ### method stage: sft do_train: true finetuning_type: full deepspeed: code/llama-factory/ds_z3_config.json ### dataset dataset: openhermes-fr dataset_dir: code/llama-factory/data template: qwen cutoff_len: 2048 max_samples: 100000 overwrite_cache: true preprocessing_num_workers: 16 dataloader_num_workers: 4 ### output output_dir: /output-checkpoint logging_steps: 10 save_steps: 1000 plot_loss: true overwrite_output_dir: true save_only_model: false report_to: none ### train per_device_train_batch_size: 1 gradient_accumulation_steps: 2 learning_rate: 1.0e-5 num_train_epochs: 3.0 lr_scheduler_type: cosine warmup_ratio: 0.1 bf16: true ddp_timeout: 180000000 resume_from_checkpoint: ``` ## Create Secrets To access gated models and datasets on HuggingFace, you need a HuggingFace token. Use the [`flexai secret create` command](https://docs.flex.ai/cli/commands/secret/) to store your *HuggingFace Token* as a secret. Replace `` with your desired name for the secret: ```bash theme={null} flexai secret create ``` Then paste your *HuggingFace Token* API key value. ## \[Optional] Pre-fetch the Model To speed up training and avoid downloading large models at runtime, you can pre-fetch your HuggingFace model to FlexAI storage. For example, to pre-fetch the `Qwen/Qwen2.5-7B` model: 1. **Create a HuggingFace storage provider:** ```bash theme={null} flexai storage create HF-STORAGE --provider huggingface --hf-token-name ``` 2. **Push the model checkpoint to your storage:** ```bash theme={null} flexai checkpoint push qwen25-7b --storage-provider HF-STORAGE --source-path Qwen/Qwen2.5-7B ``` This pre-fetched checkpoint can then be used in your training command to reduce startup time. ## Training For a 7B model, we recommend using **1 node (8 × H100 GPUs)** to ensure reasonable training time and avoid out-of-memory issues. ### Standard Training (without prefetch) ```bash theme={null} flexai training run domain-specific-qwen25-7b-sft \ --accels 8 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --env FORCE_TORCHRUN=1 \ --secret HF_TOKEN= \ --requirements-path code/llama-factory/requirements.txt \ -- /layers/flexai_pip-install/packages/bin/llamafactory-cli train code/llama-factory/qwen25-7B_sft.yaml ``` ### Training with Model Prefetch To take advantage of model pre-fetching performed in the [Optional: Pre-fetch the Model](#optional-pre-fetch-the-model) section, use: ```bash theme={null} flexai training run domain-specific-qwen25-7b-sft-prefetched \ --accels 8 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --checkpoint qwen25-7b \ --env FORCE_TORCHRUN=1 \ --secret HF_TOKEN= \ --requirements-path code/llama-factory/requirements.txt \ -- /layers/flexai_pip-install/packages/bin/llamafactory-cli train code/llama-factory/qwen25-7B_sft.yaml ``` ## Monitoring Training Progress You can check the status and lifecycle events of your Training Job by running: ```bash theme={null} flexai training inspect domain-specific-qwen25-7b-sft ``` Additionally, you can view the logs of your Training Job by running: ```bash theme={null} flexai training logs domain-specific-qwen25-7b-sft ``` ### Training Observability with TensorBoard For advanced monitoring and visualization of training metrics, you can leverage TensorBoard integration. FlexAI supports TensorBoard logging for detailed insights into training progress, loss curves, and model performance. To enable TensorBoard logging, update your YAML configuration: ```yaml theme={null} report_to: tensorboard ``` Once enabled, you can access training metrics and visualizations through the FlexAI console. For more details on observability features, see the [FlexAI TensorBoard documentation](https://docs.flex.ai/platform/tensorboard/). ## Getting Training Checkpoints Once the Training Job completes successfully, you will be able to list all the produced checkpoints: ```bash theme={null} flexai training checkpoints domain-specific-qwen25-7b-sft ``` Look for checkpoints marked as `INFERENCE READY = true` - these are ready for serving. ## Serving the Trained Model Deploy your trained model directly from the checkpoint using FlexAI inference. Replace `` with the ID from an inference-ready checkpoint: ```bash theme={null} flexai -v inference serve domain-specific-endpoint --checkpoint ``` > **Note**: GPU specification for inference endpoints is currently managed automatically by FlexAI. Future versions will allow explicit GPU count specification for inference workloads to optimize cost and performance based on your specific requirements. You can monitor your inference endpoint status: ```bash theme={null} # List all inference endpoints flexai inference list # Get detailed endpoint information flexai inference inspect domain-specific-endpoint # Check endpoint logs flexai inference logs domain-specific-endpoint ``` ## Testing Your Fine-Tuned Model Once the endpoint is running, test it with prompts representative of your domain. A well-fine-tuned model should show measurable improvements over the base model on your target tasks — better in-domain vocabulary and terminology, more accurate factual recall, and more appropriate tone or structure. ### Example API Call ```bash theme={null} curl -X POST "https://your-endpoint-url/v1/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_API_KEY" \ -d '{ "prompt": "Your evaluation prompt here", "max_tokens": 200, "temperature": 0.7 }' ``` Compare base-model and fine-tuned responses side-by-side on a held-out prompt set to quantify improvement. ## Expected Results After fine-tuning on domain-specific data, your model should achieve: * **Domain Expertise**: Specialized knowledge and terminology understanding for your target domain * **Task-Specific Performance**: Enhanced capabilities for domain-relevant tasks and workflows * **Maintained General Capabilities**: Preserved reasoning, problem-solving, and general language skills ## Technical Details ### Training Configuration Breakdown * **DeepSpeed ZeRO Stage 3**: Enables training of 7B model on 1 node efficiently * **Mixed Precision (bf16)**: Accelerates training while maintaining numerical stability * **Gradient Accumulation**: Effective batch size of 4 (2 steps × 2 per device) * **Learning Rate Schedule**: Cosine decay with 10% warmup for stable convergence * **Context Length**: 2048 tokens, optimized for conversation tasks ### Resource Requirements **Recommended Configuration for Qwen2.5-7B:** * **Nodes**: 1 node (cost-effective for 7B models) * **Accelerators**: 8 × H100 GPUs per node * **Memory**: \~200GB+ GPU memory total * **Training Time**: \~2-4 hours for 3 epochs * **Storage**: \~30GB for checkpoints **Command Line Parameters Explained:** * `FORCE_TORCHRUN=1`: Ensures proper distributed training setup ### Scaling Options * For faster training: Increase to 2 nodes (16 × H100) * For larger datasets: Adjust `max_samples` parameter * For longer context: Increase `cutoff_len` (requires more memory) * For memory efficiency: Switch to `finetuning_type: lora` for QLoRA training ## Troubleshooting **Common Issues:** **Training Job Fails to Start:** ```bash theme={null} # Check FlexAI authentication flexai auth status # Verify repository access git clone https://github.com/flexaihq/blueprints ``` **Out of Memory Errors:** * Reduce `per_device_train_batch_size` from 1 to lower value * Increase `gradient_accumulation_steps` to maintain effective batch size * Consider using `finetuning_type: lora` for memory efficiency **Checkpoint Not Inference Ready:** * Wait for training to complete fully (check with `flexai training inspect`) * Ensure `save_only_model: false` in YAML configuration * Verify training completed successfully without errors **Endpoint Deployment Issues:** * Verify checkpoint shows `INFERENCE READY = true` status * Check FlexAI cluster availability with `flexai inference list` * Review detailed logs with `flexai inference logs ` ## Code ### `code/llama-factory/qwen25-7B_sft.yaml` ```yaml theme={null} --- ### model model_name_or_path: Qwen/Qwen2.5-7B trust_remote_code: true ### method stage: sft do_train: true finetuning_type: freeze deepspeed: code/llama-factory/ds_z3_config.json ### dataset dataset: openhermes-fr dataset_dir: code/llama-factory/data template: qwen cutoff_len: 2048 max_samples: 100000 overwrite_cache: true preprocessing_num_workers: 16 dataloader_num_workers: 4 ### output output_dir: /output-checkpoint logging_steps: 10 save_steps: 1000 plot_loss: true overwrite_output_dir: true save_only_model: false report_to: none # choices: [none, wandb, tensorboard, swanlab, mlflow] ### train per_device_train_batch_size: 1 gradient_accumulation_steps: 2 learning_rate: 1.0e-5 num_train_epochs: 3.0 lr_scheduler_type: cosine warmup_ratio: 0.1 bf16: true ddp_timeout: 180000000 resume_from_checkpoint: ### eval # eval_dataset: alpaca_en_demo # val_size: 0.1 # per_device_eval_batch_size: 1 # eval_strategy: steps # eval_steps: 500 ``` ### `code/llama-factory/ds_z3_config.json` ```json theme={null} { "train_batch_size": "auto", "train_micro_batch_size_per_gpu": "auto", "gradient_accumulation_steps": "auto", "gradient_clipping": "auto", "zero_allow_untested_optimizer": true, "fp16": { "enabled": "auto", "loss_scale": 0, "loss_scale_window": 1000, "initial_scale_power": 16, "hysteresis": 2, "min_loss_scale": 1 }, "bf16": { "enabled": "auto" }, "zero_optimization": { "stage": 3, "overlap_comm": false, "contiguous_gradients": true, "sub_group_size": 1e9, "reduce_bucket_size": "auto", "stage3_prefetch_bucket_size": "auto", "stage3_param_persistence_threshold": "auto", "stage3_max_live_parameters": 1e9, "stage3_max_reuse_distance": 1e9, "stage3_gather_16bit_weights_on_model_save": true } } ``` ### `code/llama-factory/data/dataset_info.json` ```json theme={null} { "identity": { "file_name": "identity.json" }, "alpaca_en_demo": { "file_name": "alpaca_en_demo.json" }, "wikitext": { "file_name": "/input/hf-wikitext/wikitext-2-raw-v1" }, "openhermes-fr": { "hf_hub_url": "legmlai/openhermes-fr", "columns": { "prompt": "prompt", "response": "accepted_completion" } }, "openhermes-fr-prefetched": { "hf_hub_url": "/input/openhermes-fr", "columns": { "prompt": "prompt", "response": "accepted_completion" } } } ``` ### `code/llama-factory/requirements.txt` ```text theme={null} llamafactory @ git+https://github.com/hiyouga/LLaMA-Factory.git@v0.9.3 deepspeed>=0.10.0,<=0.16.9 transformers==4.51.3 ```

🚀 Run this on FlexAI

Managed checkpoints mean you never lose a run to preemption. Jobs launch in under 60 seconds — no infra setup, built-in observability.

Get started → Talk to us
# Evaluate Any LLM Across 300+ Benchmarks with lm-evaluation-harness Source: https://docs.flex.ai/blueprints/lm-evaluation-harness Run comprehensive LLM evaluation using EleutherAI's lm-evaluation-harness on FlexAI. 300+ tasks and benchmarks, reproducible results, no infra setup required. This blueprint provides a step-by-step guide for evaluating language models on FlexAI using the LM-Evaluation-Harness framework. LM-Evaluation-Harness is a unified, extensible toolkit for few-shot evaluation of language models across hundreds of standardized NLP benchmarks. In this guide, you'll learn how to: * Run evaluations across multiple NLP tasks and benchmarks * Compare model performance consistently * Use FlexAI's managed compute environment for large-scale, reproducible model evaluation If you haven't already connected FlexAI to GitHub, run `flexai code-registry connect` to set up a code registry connection. This allows FlexAI to pull repositories directly using the repository URL in training commands. ## What is LM-Evaluation-Harness? LM-Evaluation-Harness is a comprehensive evaluation framework that provides: * **300+ tasks**: standardized implementations of popular NLP benchmarks * **Multiple Model Backends**: Support for HuggingFace, OpenAI, Anthropic, and more * **Reproducible Evaluation**: Consistent evaluation protocols across different models * **Flexible Configuration**: Easy customization of evaluation parameters * **Comprehensive Metrics**: Detailed performance metrics and statistical analysis Popular evaluation tasks include: * **HellaSwag**: commonsense reasoning * **MMLU**: multi-task language understanding (57 subjects) * **GSM8K**: grade school math word problems * **HumanEval**: code generation capabilities * **TruthfulQA**: model truthfulness and reliability * **ARC**: AI2 Reasoning Challenge * **WinoGrande**: Winograd schema challenge ## Quick Start Run a basic evaluation on HellaSwag with this single command: ```bash theme={null} flexai training run lm-eval-basic \ --accels 2 \ --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --requirements-path code/lm-evaluation-harness/requirements.txt \ --runtime nvidia-25.03 \ -- lm_eval \ --model hf \ --model_args pretrained=EleutherAI/gpt-j-6B \ --tasks hellaswag \ --device cuda \ --batch_size 8 ``` The LM-Evaluation-Harness uses command-line arguments to configure evaluations. Here are the key parameters: ### Model Configuration ```bash theme={null} --model hf # Use HuggingFace backend --model_args pretrained=MODEL_NAME # Specify model to evaluate --model_args pretrained=MODEL_NAME,dtype=bfloat16 # With precision control ``` ### Task Selection ```bash theme={null} --tasks hellaswag # Single task --tasks hellaswag,arc_easy,arc_challenge # Multiple tasks --tasks mmlu_* # All MMLU subtasks --tasks all # All available tasks (not recommended) ``` ### Evaluation Parameters ```bash theme={null} --batch_size 8 # Batch size for evaluation --max_batch_size 32 # Maximum batch size --device cuda # GPU device --num_fewshot 5 # Number of few-shot examples --limit 1000 # Limit number of samples per task ``` ### Output Configuration ```bash theme={null} --output_path /output-checkpoint/results.json # Save results JSON --log_samples # Log individual sample results --show_config # Display configuration ``` ### Comprehensive Model Evaluation For a thorough evaluation across multiple benchmarks: ```bash theme={null} flexai training run lm-eval-comprehensive \ --accels 4 \ --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --secret HF_TOKEN= \ --requirements-path code/lm-evaluation-harness/requirements.txt \ --runtime nvidia-25.03 \ -- lm_eval \ --model hf \ --model_args pretrained=microsoft/DialoGPT-medium \ --tasks hellaswag,arc_easy,arc_challenge,mmlu,gsm8k \ --device cuda \ --batch_size 16 \ --output_path /output-checkpoint/comprehensive_eval.json \ --log_samples ``` ### Large Model Evaluation For evaluating large models (7B+ parameters): ```bash theme={null} flexai training run lm-eval-large-model \ --accels 8 \ --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --secret HF_TOKEN= \ --requirements-path code/lm-evaluation-harness/requirements.txt \ --runtime nvidia-25.03 \ -- lm_eval \ --model hf \ --model_args pretrained=meta-llama/Meta-Llama-3.1-8B,dtype=bfloat16 \ --tasks mmlu,hellaswag,arc_challenge,truthfulqa_mc2 \ --device cuda \ --batch_size 4 \ --output_path /output-checkpoint/llama31_8b_eval.json ``` ### Code Generation Evaluation For evaluating code generation capabilities: ```bash theme={null} flexai training run lm-eval-code \ --accels 2 \ --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --secret HF_TOKEN= \ --requirements-path code/lm-evaluation-harness/requirements.txt \ --runtime nvidia-25.03 \ -- lm_eval \ --model hf \ --model_args pretrained=Salesforce/codegen-350M-mono \ --tasks humaneval \ --device cuda \ --batch_size 8 \ --output_path /output-checkpoint/code_eval.json ``` ### Few-Shot Learning Evaluation For testing few-shot learning capabilities: ```bash theme={null} flexai training run lm-eval-fewshot \ --accels 2 \ --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --secret HF_TOKEN= \ --requirements-path code/lm-evaluation-harness/requirements.txt \ --runtime nvidia-25.03 \ -- lm_eval \ --model hf \ --model_args pretrained=EleutherAI/gpt-neo-1.3B \ --tasks winogrande,piqa,openbookqa \ --num_fewshot 5 \ --device cuda \ --batch_size 16 \ --output_path /output-checkpoint/fewshot_eval.json ``` ## Create Secrets To access models from HuggingFace (especially gated models), you need a HuggingFace token. Use the [`flexai secret create` command](https://docs.flex.ai/cli/commands/secret/) to store your *HuggingFace Token* as a secret: ```bash theme={null} flexai secret create ``` Then paste your *HuggingFace Token* API key value. ## \[Optional] Pre-fetch Models To speed up evaluation and avoid downloading large models at runtime, you can pre-fetch your models to FlexAI storage: 1. **Create a HuggingFace storage provider:** ```bash theme={null} flexai storage create HF-STORAGE --provider huggingface --hf-token-name ``` 2. **Push the model checkpoint to your storage:** ```bash theme={null} flexai checkpoint push llama31-8b --storage-provider HF-STORAGE --source-path meta-llama/Meta-Llama-3.1-8B ``` 3. **Use the pre-fetched model in your evaluation:** ```bash theme={null} flexai training run lm-eval-prefetched \ --accels 8 \ --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --checkpoint llama31-8b \ --requirements-path code/lm-evaluation-harness/requirements.txt \ --runtime nvidia-25.03 \ -- lm_eval \ --model hf \ --model_args pretrained=/input-checkpoint/llama31-8b \ --tasks mmlu,hellaswag \ --device cuda \ --batch_size 4 \ --output_path /output-checkpoint/prefetched_eval.json ``` ## Monitoring Evaluation Progress You can check the status and progress of your evaluation job: ```bash theme={null} # Check job status flexai training inspect lm-eval-comprehensive # View evaluation logs flexai training logs lm-eval-comprehensive ``` ## Getting Evaluation Results Once the evaluation job completes, you can access the results: ```bash theme={null} # List all checkpoints/outputs flexai training checkpoints lm-eval-comprehensive # Download results JSON flexai checkpoint fetch --destination ./results/ ``` The results JSON will be saved with detailed metrics for each task. ## Understanding Evaluation Results ### Sample Results Structure ```json theme={null} { "results": { "hellaswag": { "acc": 0.6234, "acc_stderr": 0.0048, "acc_norm": 0.8012, "acc_norm_stderr": 0.0040 }, "mmlu": { "acc": 0.4567, "acc_stderr": 0.0031 } }, "config": { "model": "hf", "model_args": "pretrained=EleutherAI/gpt-j-6B", "batch_size": 8, "device": "cuda" }, "git_hash": "abc123", "date": 1698123456 } ``` ### Key Metrics Explained * **acc**: Raw accuracy score * **acc\_stderr**: Standard error of accuracy * **acc\_norm**: Length-normalized accuracy (for some tasks) * **pass\@k**: Pass rate for code generation tasks * **bleu**: BLEU score for generation tasks * **rouge**: ROUGE scores for summarization tasks ## Advanced Evaluation Scenarios ### Multi-Node Evaluation For very large models or extensive benchmark suites: ```bash theme={null} flexai training run lm-eval-multi-node \ --accels 8 \ --nodes 2 \ --repository-url https://github.com/flexaihq/blueprints \ --secret HF_TOKEN= \ --requirements-path code/lm-evaluation-harness/requirements.txt \ --runtime nvidia-25.03 \ -- lm_eval \ --model hf \ --model_args pretrained=meta-llama/Meta-Llama-3.1-70B,dtype=bfloat16,device_map=auto \ --tasks mmlu_*,hellaswag,arc_challenge,truthfulqa_mc2 \ --device cuda \ --batch_size 2 \ --output_path /output-checkpoint/multi_node_eval.json ``` ### Custom Task Evaluation For evaluating on custom tasks or datasets: ```bash theme={null} flexai training run lm-eval-custom \ --accels 4 \ --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --secret HF_TOKEN= \ --requirements-path code/lm-evaluation-harness/requirements.txt \ --runtime nvidia-25.03 \ -- lm_eval \ --model hf \ --model_args pretrained=microsoft/DialoGPT-medium \ --tasks_list /path/to/custom_tasks.yaml \ --device cuda \ --batch_size 8 \ --output_path /output-checkpoint/custom_eval.json ``` ### Evaluation with Reduced Precision For memory-efficient evaluation: ```bash theme={null} flexai training run lm-eval-fp16 \ --accels 4 \ --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --secret HF_TOKEN= \ --requirements-path code/lm-evaluation-harness/requirements.txt \ --runtime nvidia-25.03 \ -- lm_eval \ --model hf \ --model_args pretrained=EleutherAI/gpt-neox-20b,dtype=float16 \ --tasks hellaswag,mmlu \ --device cuda \ --batch_size 1 \ --output_path /output-checkpoint/fp16_eval.json ``` ## Expected Results and Benchmarks ### Typical Benchmark Performance **HellaSwag (Commonsense Reasoning):** * Random baseline: \~25% * GPT-3 (175B): \~78.9% * Human performance: \~95.6% **MMLU (Multi-task Language Understanding):** * Random baseline: \~25% * GPT-3 (175B): \~43.9% * Human expert performance: \~89.8% **GSM8K (Grade School Math):** * Random baseline: \~0% * GPT-3 (175B): \~17.9% * Human performance: \~85% **HumanEval (Code Generation):** * Random baseline: \~0% * GPT-3 (175B): \~14.2% * Human performance: \~90% ### Model Size vs Performance Generally, larger models perform better, but with diminishing returns: * **Small models (\< 1B)**: Basic language understanding * **Medium models (1B-7B)**: Reasonable performance on most tasks * **Large models (7B-70B)**: Strong performance across benchmarks * **Very large models (> 70B)**: State-of-the-art performance ## Technical Details ### Resource Requirements **Recommended Configurations:** **Small Models (\< 1B parameters):** * **Accelerators**: 1-2 GPUs * **Memory**: 8-16GB GPU memory * **Evaluation Time**: 30 minutes - 2 hours **Medium Models (1B-7B parameters):** * **Accelerators**: 2-4 GPUs * **Memory**: 16-32GB GPU memory * **Evaluation Time**: 1-6 hours **Large Models (7B-70B parameters):** * **Accelerators**: 4-8 GPUs * **Memory**: 40-80GB GPU memory * **Evaluation Time**: 2-12 hours **Very Large Models (> 70B parameters):** * **Accelerators**: 8+ GPUs (multi-node recommended) * **Memory**: 80GB+ GPU memory * **Evaluation Time**: 6-24 hours ### Optimization Tips **Memory Optimization:** * Use `dtype=bfloat16` or `dtype=float16` for reduced memory usage * Reduce `batch_size` for large models * Use `device_map=auto` for automatic device placement **Speed Optimization:** * Increase `batch_size` when memory allows * Use `max_batch_size` for adaptive batching * Set `limit` for quick testing with subset of data * Use multiple GPUs with `--device cuda` **Accuracy Optimization:** * Use appropriate `num_fewshot` for few-shot tasks * Enable `--log_samples` for detailed analysis * Run multiple times with different seeds for statistical significance ### Command Line Parameters Explained * `--model hf`: Use HuggingFace Transformers backend * `--model_args`: Model-specific arguments (path, dtype, etc.) * `--tasks`: Comma-separated list of evaluation tasks * `--device`: Device placement (cuda, cpu) * `--batch_size`: Batch size for evaluation * `--num_fewshot`: Number of examples for few-shot evaluation * `--output_path`: Path to save results JSON * `--log_samples`: Save individual sample predictions * `--limit`: Limit number of samples per task (for testing) ## Popular Task Collections ### Core Benchmarks ```bash theme={null} --tasks hellaswag,arc_easy,arc_challenge,winogrande,piqa ``` ### Academic Benchmarks ```bash theme={null} --tasks mmlu,truthfulqa_mc2,gsm8k,humaneval ``` ### Reasoning Tasks ```bash theme={null} --tasks arc_challenge,hellaswag,winogrande,piqa,openbookqa ``` ### Language Understanding ```bash theme={null} --tasks mmlu_*,truthfulqa_mc2,lambada_openai ``` ### Code Generation ```bash theme={null} --tasks humaneval,mbpp ``` ### Math and Logic ```bash theme={null} --tasks gsm8k,mathqa,aqua_rat ``` ## Troubleshooting **Common Issues:** **Out of Memory Errors:** * Reduce `batch_size` (try 1, 2, 4) * Use mixed precision: `dtype=bfloat16` or `dtype=float16` * Increase number of GPUs: `--accels 8` * Enable CPU offloading: `device_map=auto` **Model Loading Errors:** * Verify model name is correct on HuggingFace * Check that your HuggingFace token has permission to access gated models * Use `trust_remote_code=True` for custom models * Verify there's sufficient disk space for model downloads **Task Not Found Errors:** * List available tasks: `lm_eval --tasks list` * Check task name spelling and capitalization * Verify task is supported in your lm-eval version * Use task groups: `mmlu_*` instead of individual tasks **Slow Evaluation:** * Increase `batch_size` when memory allows * Use `--limit` for quick testing (e.g., `--limit 100`) * Consider fewer tasks for initial testing * Use faster/smaller models for development **Authentication Issues:** * Create HuggingFace token with appropriate permissions * Store token as FlexAI secret correctly * Verify token is not expired * Check model access permissions (especially for gated models) **Job Monitoring Commands:** ```bash theme={null} # Check job status flexai training inspect # View logs flexai training logs ``` **Results Access:** ```bash theme={null} # List outputs flexai training checkpoints # Download results JSON flexai checkpoint fetch --destination ./results/ ``` *** ## References * **LM-Evaluation-Harness GitHub**: [https://github.com/EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) * **Documentation**: [https://github.com/EleutherAI/lm-evaluation-harness/tree/master/docs](https://github.com/EleutherAI/lm-evaluation-harness/tree/master/docs) * **Paper**: "Language Model Evaluation Harness" - [https://arxiv.org/abs/2101.00027](https://arxiv.org/abs/2101.00027) * **Task List**: [https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/task\_table.md](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/task_table.md) * **FlexAI Documentation**: [https://docs.flex.ai](https://docs.flex.ai) * **HuggingFace Transformers**: [https://huggingface.co/docs/transformers](https://huggingface.co/docs/transformers) ## Code ### `requirements.txt` ```text theme={null} git+https://github.com/EleutherAI/lm-evaluation-harness.git@v0.4.9.1 ```

🔬 Run evaluations on FlexAI

Scale across 300+ benchmarks with reproducible results. Jobs launch in under 60 seconds — no infra setup, built-in observability.

Get started → Talk to us
# Fine-Tune Stable Diffusion XL with LoRA on FlexAI Source: https://docs.flex.ai/blueprints/lora-ft-on-a-diffusion-model Step-by-step guide to fine-tuning SDXL with LoRA on FlexAI. Launches in under 60s, managed checkpoints, built-in observability — no infra setup required. In this experiment, we will fine-tune Stable Diffusion XL on the Naruto dataset with LoRA using the `Diffusers` library. The goal is to specialize the model for generating content related to Naruto. Fine-tuning enables the model to better produce outputs tailored to the themes and characteristics of the dataset, which may include unique styles, characters, or captions associated with the Naruto universe. > **Note**: If you haven't already connected FlexAI to GitHub, you'll need to run `flexai code-registry connect` to set up a code registry connection. This allows FlexAI to pull repositories directly using the `-u` flag in training commands. ## Prepare the Dataset We will be using the `lambdalabs/naruto-blip-captions` dataset. You can download the pre-processed version of the dataset by running the following command: ```bash theme={null} DATASET_NAME=sdxl-tokenized-naruto && curl -L -o ${DATASET_NAME}.zip "https://bucket-docs-samples-99b3a05.s3.eu-west-1.amazonaws.com/${DATASET_NAME}.zip" && unzip ${DATASET_NAME}.zip && rm ${DATASET_NAME}.zip ``` > If you'd like to reproduce the pre-processing steps yourself to use a different dataset or simply to learn more about the process, you can refer to the [Manual Dataset Pre-processing](#manual-dataset-pre-processing) section below. Next, push the contents of the `sdxl-tokenized-naruto/` directory as a new FlexAI dataset: ```bash theme={null} flexai dataset push sdxl-tokenized-naruto --file sdxl-tokenized-naruto ``` ## Training To start the Training Job, run the following command: ```bash theme={null} flexai training run text-to-image-lora-SDXL-training-ddp --repository-url https://github.com/flexaihq/blueprints --dataset sdxl-tokenized-naruto --secret HF_TOKEN= --secret WANDB_API_KEY= --requirements-path code/diffuser/requirements.txt \ --nodes 1 --accels 2 \ -- code/diffuser/train_text_to_image_lora_sdxl.py \ --pretrained_model_name_or_path stabilityai/stable-diffusion-xl-base-1.0 \ --pretrained_vae_model_name_or_path madebyollin/sdxl-vae-fp16-fix \ --train_dataset_load_dir /input/sdxl-tokenized-naruto \ --caption_column text \ --resolution 1024 \ --random_flip \ --train_batch_size 1 \ --num_train_epochs 2 \ --checkpointing_steps 500 \ --learning_rate 1e-04 \ --lr_scheduler constant \ --lr_warmup_steps 0 \ --mixed_precision fp16 \ --seed 42 \ --output_dir /output-checkpoint \ --validation_prompt "'cute dragon creature'" ``` ## Optional Extra Steps ### Manual Dataset Pre-processing If you'd prefer to perform the dataset pre-processing step yourself, you can follow these instructions. You can run these in a FlexAI Interactive Session or in a local env (e.g. `pipenv install --python 3.10`), if you have hardware that's capable of doing inference. #### Clone this repository If you haven't already, clone this repository on your host machine: ```bash theme={null} git clone https://github.com/flexaihq/blueprints.git blueprints --depth 1 --branch main && cd blueprints ``` #### Install the dependencies Depending on your environment, you might need to install - if not already - the experiments' dependencies by running: ```bash theme={null} pip install -r code/diffuser/requirements.txt ``` #### Dataset preparation Prepare the dataset by running the following command: ```bash theme={null} python dataset/prepare_save_text_to_image.py \ --dataset_name lambdalabs/naruto-blip-captions \ --train_dataset_save_dir sdxl-tokenized-naruto ``` The prepared dataset will be saved to the `sdxl-tokenized-naruto/` directory. ## Code ### `code/diffuser/train_text_to_image_lora_sdxl.py` ```python theme={null} #!/usr/bin/env python # coding=utf-8 # Copyright 2024 The HuggingFace Inc. team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Adapted from: https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora_sdxl.py """Fine-tuning script for Stable Diffusion XL for text2image with support for LoRA.""" import argparse import logging import math import os import shutil import sys from contextlib import nullcontext from pathlib import Path import datasets import diffusers import numpy as np import torch import torch.nn.functional as F import torch.utils.checkpoint import transformers import wandb from accelerate import Accelerator from accelerate.logging import get_logger from accelerate.utils import ( DistributedDataParallelKwargs, DistributedType, ProjectConfiguration, set_seed, ) from diffusers import ( AutoencoderKL, DDPMScheduler, StableDiffusionXLPipeline, UNet2DConditionModel, ) from diffusers.loaders import StableDiffusionLoraLoaderMixin from diffusers.optimization import get_scheduler from diffusers.training_utils import ( _set_state_dict_into_text_encoder, cast_training_params, compute_snr, ) from diffusers.utils import ( convert_state_dict_to_diffusers, convert_unet_state_dict_to_peft, ) # ... (full script in repository) ``` ### `code/diffuser/requirements.txt` ```text theme={null} accelerate>=1.8.1 bitsandbytes>=0.43.2 datasets>=2.21.0 diffusers==0.30.3 peft>=0.15.0 transformers>=4.43.3 trl>=0.11.4 wandb>=0.18.1 ```

🚀 Run this on FlexAI

Managed checkpoints mean you never lose a run to preemption. Jobs launch in under 60 seconds — no infra setup, built-in observability.

Get started → Talk to us
# Deploy Multi-Agent LangGraph Systems on FlexAI Inference Endpoints Source: https://docs.flex.ai/blueprints/multi-agent Build and deploy a multi-agent LangGraph system on FlexAI inference endpoints. Full tutorial with working code, auto-scaling, cold start under 60 seconds. This experiment explores a Multi-Agent architecture where specialized AI agents work together under the guidance of a central supervisor. The supervisor acts as an intelligent coordinator, managing communication between agents and strategically delegating tasks based on each agent's expertise and the specific requirements of the problem at hand. In this experiment, you'll create a multi-agent system powered by LangGraph with two agents — a research and a math expert. The web search agent is using [Tavily](https://www.tavily.com/), visit their website to get an API key. ## Start the FlexAI endpoints Create the FlexAI secret that contains your HF token in order to access the inference models: ```bash theme={null} # Enter your HF token value when prompted flexai secret create hf-token ``` Export your Tavily API key: ```bash theme={null} export TAVILY_API_KEY= ``` Start the FlexAI endpoint of the LLM: ```bash theme={null} LLM_INFERENCE_NAME=qwen-llm export LLM_MODEL_NAME=Qwen/Qwen2.5-32B-Instruct flexai inference serve $LLM_INFERENCE_NAME --hf-token-secret hf-token -- --model=$LLM_MODEL_NAME --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 16384 ``` Store the returned Inference Endpoint API KEY and Endpoint URL: ```bash theme={null} export LLM_API_KEY= export LLM_URL=$(flexai inference inspect $LLM_INFERENCE_NAME -j | jq .config.endpointUrl -r) ``` > You'll notice the last `export` line uses the `jq` tool to extract the value of `endpointUrl` from the JSON output of the `inspect` command. > > If you don't have it already, you can get `jq` from its official website: [https://jqlang.org/](https://jqlang.org/) ## Setup ```bash theme={null} cd code/multi-agent/ ``` ```bash theme={null} pip install -r requirements.txt ``` ```bash theme={null} python main.py ``` When prompted, enter your question. The system will automatically route it to the appropriate agents (research and math experts). **Research + Math Question:** ``` In 2025, how old are Trump and Macron? Also sum their ages. ``` **Expected Output:** ``` In 2025, Donald Trump will be 79 years old and Emmanuel Macron will be 48 years old. The sum of their ages in 2025 will be 127 years. ``` ## Code ### `requirements.txt` ```text theme={null} langgraph>=0.6.6 langgraph-supervisor>=0.0.29 langchain-tavily>=0.2.11 langchain[openai]>=0.3.25 ``` ### `main.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import getpass import os from agents.supervisor import supervisor from langchain_core.messages import convert_to_messages def _set_if_undefined(var: str): if not os.environ.get(var): os.environ[var] = getpass.getpass(f"Please provide your {var}") _set_if_undefined("LLM_API_KEY") _set_if_undefined("LLM_URL") _set_if_undefined("LLM_MODEL_NAME") _set_if_undefined("TAVILY_API_KEY") def pretty_print_message(message, indent=False): pretty_message = message.pretty_repr(html=True) if not indent: print(pretty_message) return indented = "\n".join("\t" + c for c in pretty_message.split("\n")) print(indented) def pretty_print_messages(update, last_message=False): is_subgraph = False if isinstance(update, tuple): ns, update = update # skip parent graph updates in the printouts if len(ns) == 0: return graph_id = ns[-1].split(":")[0] print(f"Update from subgraph {graph_id}:") print("\n") is_subgraph = True for node_name, node_update in update.items(): update_label = f"Update from node {node_name}:" if is_subgraph: update_label = "\t" + update_label print(update_label) print("\n") messages = convert_to_messages(node_update["messages"]) if last_message: messages = messages[-1:] for m in messages: pretty_print_message(m, indent=is_subgraph) print("\n") while True: user_question = input("Please enter your question: ") for chunk in supervisor.stream( { "messages": [ { "role": "user", "content": user_question, } ] }, ): if not ("supervisor" in chunk and chunk["supervisor"] is None): pretty_print_messages(chunk, last_message=True) final_message_history = chunk["supervisor"]["messages"] ``` ### `agents/supervisor.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import os from langchain_openai import ChatOpenAI from langgraph_supervisor import create_supervisor from .math import math_agent from .web_search import research_agent llm = ChatOpenAI( model_name=os.environ.get("LLM_MODEL_NAME"), openai_api_key=os.environ.get("LLM_API_KEY"), openai_api_base=os.environ.get("LLM_URL") + "/v1", ) supervisor = create_supervisor( model=llm, agents=[research_agent, math_agent], prompt=( "You are a supervisor managing two agents:\n" "- a research agent. Assign research-related tasks to this agent\n" "- a math agent. Assign math-related tasks to this agent\n" "Assign work to one agent at a time, do not call agents in parallel.\n" "Do not do any work yourself." ), add_handoff_back_messages=True, output_mode="full_history", ).compile() ``` ### `agents/math.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import os from langchain_openai import ChatOpenAI from langgraph.prebuilt import create_react_agent llm = ChatOpenAI( model_name=os.environ.get("LLM_MODEL_NAME"), openai_api_key=os.environ.get("LLM_API_KEY"), openai_api_base=os.environ.get("LLM_URL") + "/v1", ) def add(a: float, b: float): """Add two numbers.""" return a + b def multiply(a: float, b: float): """Multiply two numbers.""" return a * b def divide(a: float, b: float): """Divide two numbers.""" return a / b math_agent = create_react_agent( model=llm, tools=[add, multiply, divide], prompt=( "You are a math agent.\n\n" "INSTRUCTIONS:\n" "- Assist ONLY with math-related tasks\n" "- After you're done with your tasks, respond to the supervisor directly\n" "- Respond ONLY with the results of your work, do NOT include ANY other text." ), name="math_agent", ) ``` ### `agents/web_search.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import os from langchain_openai import ChatOpenAI from langchain_tavily import TavilySearch from langgraph.prebuilt import create_react_agent web_search = TavilySearch(max_results=3) llm = ChatOpenAI( model_name=os.environ.get("LLM_MODEL_NAME"), openai_api_key=os.environ.get("LLM_API_KEY"), openai_api_base=os.environ.get("LLM_URL") + "/v1", ) research_agent = create_react_agent( model=llm, tools=[web_search], prompt=( "You are a research agent.\n\n" "INSTRUCTIONS:\n" "- Assist ONLY with research-related tasks, DO NOT do any math\n" "- After you're done with your tasks, respond to the supervisor directly\n" "- Respond ONLY with the results of your work, do NOT include ANY other text." ), name="research_agent", ) ```

⚡ Deploy this on FlexAI

Any model, NVIDIA or AMD, live in under 60 seconds. No code changes, no infra management, hardware-agnostic.

Get started → Talk to us
# Fine-Tune Llama 3.1 with QLoRA (4-bit) on FlexAI Source: https://docs.flex.ai/blueprints/qlora-ft-on-a-language-model Fine-tune Llama 3.1 using QLoRA 4-bit quantization on FlexAI. Cut memory requirements by 75%, launch in under 60s, managed checkpoints included. In this experiment, we will fine-tune a causal language model using QLoRA and the `SFTTrainer` from `trl`. We will use `Llama3.1` as our large language model (LLM) and train it on the `openassistant-guanaco` dataset but you can easily choose another model or dataset from the HuggingFace hub. If you haven't already connected FlexAI to GitHub, you'll need to set up a code registry connection: ```bash theme={null} flexai code-registry connect ``` This will allow FlexAI to pull repositories directly from GitHub using the `-u` flag in training commands. In this experiment, we will use a pre-processed version of the the `openassistant-guanaco` dataset that has been set up for the `Llama3.1` model. ```bash theme={null} DATASET_NAME=llama-tokenized-oag && curl -L -o ${DATASET_NAME}.zip "https://bucket-docs-samples-99b3a05.s3.eu-west-1.amazonaws.com/${DATASET_NAME}.zip" && unzip ${DATASET_NAME}.zip && rm ${DATASET_NAME}.zip ``` > If you'd like to reproduce the pre-processing steps yourself to use a different dataset or simply to learn more about the process, you can refer to the [Manual Dataset Pre-processing](#manual-dataset-pre-processing) section below. Next, push the contents of the `llama-tokenized-oag/` directory as a new FlexAI dataset: ```bash theme={null} flexai dataset push llama-tokenized-oag --file llama-tokenized-oag ``` ## Create Secrets To access the Llama-3.1-8B model, you need to [accept the license](https://huggingface.co/meta-llama/Llama-3.1-8B) with your HuggingFace account. To be authenticated within your code, you will use your *HuggingFace Token*. Use the [`flexai secret create` command](https://docs.flex.ai/cli/commands/secret/) to store your *HuggingFace Token* as a secret. Replace `` with your desired name for the secret: ```bash theme={null} flexai secret create ``` Then paste your *HuggingFace Token* API key value. ## Training To start the Training Job, run the following command: ```bash theme={null} flexai training run llama3-1-training-ddp --repository-url https://github.com/flexaihq/blueprints --requirements-path code/causal-language-modeling-qlora/requirements.txt \ --dataset llama-tokenized-oag \ --secret HF_TOKEN= --secret WANDB_API_KEY= --env WANDB_PROJECT= \ --nodes 1 --accels 2 \ -- code/causal-language-modeling-qlora/train.py \ --model_name_or_path meta-llama/Meta-Llama-3.1-8B \ --dataset_name timdettmers/openassistant-guanaco \ --tokenized_dataset_load_dir /input/llama-tokenized-oag \ --dataset_text_field text \ --load_in_4bit \ --use_peft \ --per_device_train_batch_size 4 \ --gradient_accumulation_steps 2 \ --output_dir /output-checkpoint \ --log_level info ``` ## Optional Extra Steps ### Manual Dataset Pre-processing If you'd prefer to perform the dataset pre-processing step yourself, you can follow these instructions. You can run these in a FlexAI Interactive Session or in a local env (e.g. `pipenv install --python 3.10`), if you have hardware that's capable of doing inference. #### Clone this repository If you haven't already, clone this repository on your host machine: ```bash theme={null} git clone https://github.com/flexaihq/blueprints.git blueprints --depth 1 --branch main && cd blueprints ``` #### Install the dependencies Depending on your environment, you might need to install - if not already - the experiments' dependencies by running: ```bash theme={null} pip install -r requirements.txt ``` #### Dataset preparation Prepare the dataset by running the following command: ```bash theme={null} python dataset/prepare_save_sft.py \ --model_name_or_path meta-llama/Meta-Llama-3.1-8B \ --dataset_name timdettmers/openassistant-guanaco \ --dataset_text_field text \ --log_level info \ --tokenized_dataset_save_dir llama-tokenized-oag \ --output_dir ./.sft.output # This argument is not used but is required to use the SFT argument parser. ``` The prepared dataset will be saved to the `llama-tokenized-oag/` directory. ## Code ### `code/causal-language-modeling-qlora/train.py` ```python theme={null} # Adapted from: https://github.com/huggingface/trl/blob/2cad48d511fab99ac0c4b327195523a575afcad3/examples/scripts/sft.py # flake8: noqa # Copyright 2023 The HuggingFace Inc. team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import os import sys from dataclasses import dataclass, field from typing import Optional import datasets from datasets import load_dataset from transformers import AutoTokenizer from trl import ( ModelConfig, SFTConfig, SFTTrainer, get_kbit_device_map, get_peft_config, get_quantization_config, ) from trl.commands.cli_utils import SFTScriptArguments, TrlParser SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) sys.path.append(os.path.dirname(SCRIPT_DIR)) from utils.experiment_tracking import set_wandb @dataclass class AdditionalArguments: """ Additional arguments that are not part of the TRL arguments. """ tokenized_dataset_load_dir: Optional[str] = field( default=None, metadata={ "help": ( "Directory to load the tokenized dataset using the " "`datasets.load_from_disk` method. No tokenization will be done" " the dataset will be loaded from this directory." ) }, ) if __name__ == "__main__": parser = TrlParser( (SFTScriptArguments, SFTConfig, ModelConfig, AdditionalArguments) ) args, training_args, model_config, additional_args = parser.parse_args_and_config() set_wandb(training_args) quantization_config = get_quantization_config(model_config) model_kwargs = dict( revision=model_config.model_revision, trust_remote_code=model_config.trust_remote_code, attn_implementation=model_config.attn_implementation, torch_dtype=model_config.torch_dtype, use_cache=False if training_args.gradient_checkpointing else True, device_map=get_kbit_device_map() if quantization_config is not None else None, quantization_config=quantization_config, ) training_args.model_init_kwargs = model_kwargs tokenizer = AutoTokenizer.from_pretrained( model_config.model_name_or_path, trust_remote_code=model_config.trust_remote_code, use_fast=True, ) tokenizer.pad_token = tokenizer.eos_token if additional_args.tokenized_dataset_load_dir: dataset = datasets.load_from_disk(additional_args.tokenized_dataset_load_dir) skip_prepare_dataset = True if training_args.dataset_kwargs is None: training_args.dataset_kwargs = {} training_args.dataset_kwargs["skip_prepare_dataset"] = True else: dataset = load_dataset(args.dataset_name) trainer = SFTTrainer( model=model_config.model_name_or_path, args=training_args, train_dataset=dataset[args.dataset_train_split], eval_dataset=dataset[args.dataset_test_split], tokenizer=tokenizer, peft_config=get_peft_config(model_config), ) trainer.train() trainer.save_model(training_args.output_dir) ``` ### `code/causal-language-modeling-qlora/requirements.txt` ```text theme={null} accelerate>=1.8.1 bitsandbytes>=0.43.2 datasets>=2.21.0 evaluate>=0.4.3 peft>=0.15.0 transformers>=4.43.3 trl==0.11.4 wandb>=0.18.1 ```

🚀 Run this on FlexAI

Managed checkpoints mean you never lose a run to preemption. Jobs launch in under 60 seconds — no infra setup, built-in observability.

Get started → Talk to us
# Build a RAG App with LangChain on FlexAI Inference Endpoints Source: https://docs.flex.ai/blueprints/rag-application Build a retrieval-augmented generation app using LangChain and FlexAI inference endpoints. Interactive Q&A from your documents, deploy in minutes. The RAG (Retrieval-Augmented Generation) application is designed to provide an interactive interface for users to ask questions based on provided documents. ## Start the FlexAI endpoints Create the FlexAI secret that contains your HF token in order to access the inference models: ```bash theme={null} # Enter your HF token value when prompted flexai secret create hf-token ``` Start the FlexAI endpoint of the LLM: ```bash theme={null} LLM_INFERENCE_NAME=qwen-llm export LLM_MODEL_NAME=Qwen/Qwen3-30B-A3B-Instruct-2507 flexai inference serve $LLM_INFERENCE_NAME --hf-token-secret hf-token -- --model=$LLM_MODEL_NAME --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 16384 # store the returned information export LLM_API_KEY= export LLM_URL=$(flexai inference inspect $LLM_INFERENCE_NAME -j | jq .config.endpointUrl -r) ``` Start the FlexAI endpoint of the embedder: ```bash theme={null} EMBED_INFERENCE_NAME=e5-embed export EMBEDDINGS_MODEL_NAME=intfloat/multilingual-e5-large flexai inference serve $EMBED_INFERENCE_NAME --runtime vllm-nvidia-0.9.1 --hf-token-secret hf-token -- --model=$EMBEDDINGS_MODEL_NAME --task=embed --trust-remote-code --dtype=float32 # store the returned information export EMBEDDINGS_API_KEY= export EMBEDDINGS_URL=$(flexai inference inspect $EMBED_INFERENCE_NAME -j | jq .config.endpointUrl -r) ``` ## LangSmith (Optional) ```bash theme={null} export LANGSMITH_TRACING="true" export LANGSMITH_API_KEY="..." export LANGSMITH_PROJECT=rag-with-flexai ``` ## Setup The code of this experiment is located at `code/rag` and the following commands should be run from this location. ### Using Docker ```bash theme={null} docker build -t rag-application . ``` ```bash theme={null} docker run -p 7860:7860 \ -e LLM_MODEL_NAME=$LLM_MODEL_NAME \ -e LLM_API_KEY=$LLM_API_KEY \ -e LLM_URL=$LLM_URL \ -e EMBEDDINGS_MODEL_NAME=$EMBEDDINGS_MODEL_NAME \ -e EMBEDDINGS_API_KEY=$EMBEDDINGS_API_KEY \ -e EMBEDDINGS_URL=$EMBEDDINGS_URL \ -e LANGSMITH_TRACING=$LANGSMITH_TRACING \ -e LANGSMITH_API_KEY=$LANGSMITH_API_KEY \ -e LANGSMITH_PROJECT=$LANGSMITH_PROJECT \ rag-application ``` ### Local Setup ```bash theme={null} pip install -r requirements.txt ``` ```bash theme={null} python ./run_rag.py ``` ## Usage Once the application is running, you can access the Gradio interface in your web browser at `http://localhost:7860`. You can upload documents and ask questions based on the content of those documents. For examples, you can upload documents located at `code/rag/data` and ask questions such as * What is this demo about? * For which workflows LLM agents are useful? * Where can I find bioluminescent fungis? ## Code ### `requirements.txt` ```text theme={null} beautifulsoup4>=4.13.4 docx2txt>=0.9 gradio==5.49.1 langchain-community==0.3.30 langchain-openai==0.3.35 langchain-text-splitters==0.3.11 langgraph==0.6.8 lxml>=5.4.0 pypdf>=5.4.0 transformers>=4.43.3 ``` ### `run_rag.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import logging import os from uuid import uuid4 import gradio as gr from src.rag_pipeline import RagPipeline current_dir = os.path.dirname(os.path.abspath(__file__)) logo_path = os.path.join(current_dir, "logo.png") rag_pipeline = RagPipeline(chunk_size=500, chunk_overlap=50, use_tools=False) def get_endpoint_config(): config = rag_pipeline.get_endpoint_config() return [ config.get("llm_model_name", ""), config.get("llm_api_key", ""), config.get("llm_url", ""), config.get("embeddings_model_name", ""), config.get("embeddings_api_key", ""), config.get("embeddings_url", ""), ] def set_endpoint_config( llm_model_name, llm_api_key, llm_url, embeddings_model_name, embeddings_api_key, embeddings_url, ): config = { "llm_model_name": llm_model_name, "llm_api_key": llm_api_key, "llm_url": llm_url, "embeddings_model_name": embeddings_model_name, "embeddings_api_key": embeddings_api_key, "embeddings_url": embeddings_url, } try: rag_pipeline.set_endpoint_config(config) error_msg = "" error_visible = False except Exception as e: error_msg = f"Error: {e}" error_visible = True config_values = get_endpoint_config() doc_list = clear_document_list() return ( *config_values, doc_list, gr.update(value=error_msg, visible=error_visible), ) def toggle_api_key_visibility(visible, value): type = "text" if visible else "password" return gr.Textbox(label="API Key", type=type, value=value) def clear_history(): new_id = uuid4() logging.info(f"New thread_id: {new_id}") return str(new_id), None def clear_document_list(): rag_pipeline.clear_vector_store() return [] def add_message(history, message, thread_id): if message: history.append({"role": "user", "content": message}) try: response = rag_pipeline.query(message, thread_id=thread_id) bot_message = response["messages"][-1].content except Exception as e: bot_message = ( f"Error: {e}" ) history.append({"role": "assistant", "content": bot_message}) return history, "" def on_files_uploaded(history, files): error_msgs = [] for f in files: try: rag_pipeline.add_documents([f]) except Exception as e: filename = f.split("/")[-1] error_msgs.append(f"File '{filename}': {e}") if error_msgs: error_msg = "
".join( f"Error: {msg}" for msg in error_msgs ) history.append({"role": "assistant", "content": error_msg}) return history def on_files_deleted(): rag_pipeline.clear_vector_store() css = """ :root { color-scheme: light dark; } footer { visibility: hidden; } #chatbot { background: light-dark(#f7f7fa, #000) !important; } #main-card { background: #fff; } .file-preview { min-height: 240px; } .user { background: light-dark(#fff, #000); border-color: light-dark(#fff, #000); } .bot { background: light-dark(#f7f7fa, #18181b); border-color: light-dark(#f7f7fa, #18181b); box-shadow: 0px 0px 0px 0px !important; } #chat-area { background: light-dark(#f7f7fa, #000); } #chat-input { background: #fff; } #send-btn { background: #fff; } #filetable { button { color: black; background-color: white; } } .tab-content { background: #fff; } """ with gr.Blocks( title="FlexBot: Ask me anything!", fill_height=True, css=css, theme=gr.themes.Default(primary_hue=gr.themes.colors.blue), ) as demo: uuid_state = gr.State(value=str(uuid4())) doc_list = gr.State(value=[]) title = f"""

Search RAG

FlexBot can answer questions based on the provided documents.

Logo
""" gr.HTML(title) with gr.Row(): with gr.Column(scale=3): with gr.Group(elem_id="main-card"): with gr.Group(elem_id="chat-area"): with gr.Row(): with gr.Column(scale=1): chatbot = gr.Chatbot( height="70dvh", show_label=False, elem_id="chatbot", type="messages", scale=1, ) chat_input = gr.Textbox( interactive=True, placeholder="Ask me anything!", show_label=False, submit_btn=True, ) with gr.Column(scale=2): with gr.Tabs(): with gr.TabItem("Documents"): with gr.Group(elem_id="tab-content"): files_new = gr.File( file_count="multiple", interactive=True, scale=1, file_types=[ ".txt", ".pdf", ".csv", ".doc", ".docx", ".html", ".htm", ], ) with gr.TabItem("Endpoint Config."): with gr.Group(elem_id="tab-content"): llm_model_name = gr.Textbox(label="LLM Model Name") llm_url = gr.Textbox(label="LLM URL") with gr.Row(): llm_api_key = gr.Textbox( placeholder="LLM API Key", type="password", scale=4, show_label=False, ) llm_api_key_visible = gr.Checkbox( label="Show LLM API Key", value=False, scale=1, ) embeddings_model_name = gr.Textbox( label="Embeddings Model Name" ) embeddings_url = gr.Textbox(label="Embeddings URL") with gr.Row(): embeddings_api_key = gr.Textbox( label="Embeddings API Key", type="password", scale=4, show_label=False, ) embeddings_api_key_visible = gr.Checkbox( label="Show Embeddings API Key", value=False, scale=1, ) save_btn = gr.Button("Save") config_error = gr.Markdown(value="", visible=False) # Prefill on load demo.load( get_endpoint_config, inputs=None, outputs=[ llm_model_name, llm_api_key, llm_url, embeddings_model_name, embeddings_api_key, embeddings_url, ], ) demo.load( clear_document_list, inputs=None, outputs=[doc_list], ) demo.load( clear_history, inputs=None, outputs=[uuid_state, chatbot], ) save_btn.click( set_endpoint_config, inputs=[ llm_model_name, llm_api_key, llm_url, embeddings_model_name, embeddings_api_key, embeddings_url, ], outputs=[ llm_model_name, llm_api_key, llm_url, embeddings_model_name, embeddings_api_key, embeddings_url, doc_list, config_error, ], ) llm_api_key_visible.change( toggle_api_key_visibility, inputs=[llm_api_key_visible, llm_api_key], outputs=llm_api_key, ) embeddings_api_key_visible.change( toggle_api_key_visibility, inputs=[embeddings_api_key_visible, embeddings_api_key], outputs=embeddings_api_key, ) chatbot.clear(clear_history, outputs=[uuid_state, chatbot]) chat_msg = chat_input.submit( add_message, [chatbot, chat_input, uuid_state], [chatbot, chat_input] ).then(lambda: gr.Textbox(interactive=True), None, [chat_input]) files_new.upload(on_files_uploaded, inputs=[chatbot, files_new], outputs=[chatbot]) files_new.delete(on_files_deleted, inputs=None, outputs=None) files_new.clear(on_files_deleted, inputs=None, outputs=None) if __name__ == "__main__": logging.basicConfig(level=logging.INFO) demo.launch(allowed_paths=[logo_path], favicon_path=logo_path) ``` ### `Dockerfile` ```text theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT FROM python:3.11-slim # Set the working directory WORKDIR /app # Copy the requirements file COPY requirements.txt . # Install the dependencies RUN pip install --no-cache-dir --upgrade pip && pip install --no-cache-dir -r requirements.txt # Copy the application code COPY src/ ./src/ COPY run_rag.py ./run_rag.py COPY logo.png ./logo.png # Expose the port the app runs on EXPOSE 7860 ENV GRADIO_SERVER_NAME="0.0.0.0" # Command to run the application CMD ["python", "run_rag.py"] ``` ### `src/rag_pipeline.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import logging import os import uuid from typing import List from langchain.schema import BaseMessage from langchain_core.documents import Document from langchain_core.messages import SystemMessage from langchain_core.prompts import PromptTemplate from langchain_core.tools import tool from langchain_core.vectorstores import InMemoryVectorStore from langchain_openai import ChatOpenAI, OpenAIEmbeddings from langchain_text_splitters import RecursiveCharacterTextSplitter from langgraph.checkpoint.memory import MemorySaver from langgraph.graph import END, START, MessagesState, StateGraph from langgraph.prebuilt import ToolNode, tools_condition from src.reader import MixedFileTypeLoader from transformers import AutoConfig class RagPipeline: def __init__( self, chunk_size: int = 500, chunk_overlap: int = 50, top_k: int = 5, use_tools: bool = False, ): """ Initializes the pipeline and sets up models and vector store. Args: chunk_size (int, optional): The size of each chunk for processing. Defaults to 500. chunk_overlap (int, optional): The overlap size between consecutive chunks. Defaults to 50. top_k (int, optional): The number of top scored documents to retrieve. Defaults to 5. use_tools (bool, optional): Flag to indicate whether to use LLM tool calling. If True, the LLM will decide whether it needs to use tools to retrieve information or directly respond to the query without invoking the document search. If False, document search will always be performed and the LLM will always receive the corresponding context to respond to the query. Defaults to False. """ self.top_k: int = top_k self.chunk_size: int = chunk_size self.chunk_overlap: int = chunk_overlap self.use_tools: bool = use_tools self.intro_prompt: str = ( "You are FlexBot, an assistant for question-answering tasks. " ) self.rag_prompt: str = ( "You are FlexBot, an assistant for question-answering tasks. " "Use the following pieces of retrieved context to answer " "the question. If you don't know the answer, say that you " "don't know. Use six sentences maximum and keep the " "answer concise." ) self.llm: ChatOpenAI self.embeddings: OpenAIEmbeddings self.vector_store: InMemoryVectorStore self.prompt: PromptTemplate self.graph: StateGraph self.llm_model_name: str self.llm_api_key: str self.llm_url: str self.embeddings_model_name: str self.embeddings_api_key: str self.embeddings_url: str self._set_endpoint_config() self._set_models() self._set_vector_store() self._set_graph() def add_documents(self, file_paths: List[str]) -> None: # parse documents into chunks loader = MixedFileTypeLoader(file_paths) docs = loader.load() for doc in docs: # only keep document filename doc.metadata["source"] = os.path.basename(doc.metadata["source"]) text_splitter = RecursiveCharacterTextSplitter( chunk_size=self.chunk_size, chunk_overlap=self.chunk_overlap ) all_splits = text_splitter.split_documents(docs) # index chunked documents _ = self.vector_store.add_documents(documents=all_splits) def _check_env(self) -> None: if "LLM_API_KEY" not in os.environ: raise ValueError("Please set the LLM_API_KEY environment variable.") if "LLM_URL" not in os.environ: raise ValueError("Please set the LLM_URL environment variable.") if "LLM_MODEL_NAME" not in os.environ: raise ValueError("Please set the LLM_MODEL_NAME environment variable.") if "EMBEDDINGS_API_KEY" not in os.environ: raise ValueError("Please set the EMBEDDINGS_API_KEY environment variable.") if "EMBEDDINGS_URL" not in os.environ: raise ValueError("Please set the EMBEDDINGS_URL environment variable.") if "EMBEDDINGS_MODEL_NAME" not in os.environ: raise ValueError( "Please set the EMBEDDINGS_MODEL_NAME environment variable." ) def get_endpoint_config(self) -> dict: return { "llm_model_name": self.llm_model_name, "llm_api_key": self.llm_api_key, "llm_url": self.llm_url, "embeddings_model_name": self.embeddings_model_name, "embeddings_api_key": self.embeddings_api_key, "embeddings_url": self.embeddings_url, } def set_endpoint_config(self, config: dict) -> None: for key, value in config.items(): if key not in [ "llm_model_name", "llm_api_key", "llm_url", "embeddings_model_name", "embeddings_api_key", "embeddings_url", ]: raise ValueError(f"Invalid config key: {key}") setattr(self, key, value) self._set_models() self._set_vector_store() def _set_endpoint_config( self, ) -> None: self._check_env() self.llm_model_name = os.getenv("LLM_MODEL_NAME") self.llm_api_key = os.getenv("LLM_API_KEY") self.llm_url = os.getenv("LLM_URL") self.embeddings_model_name = os.getenv("EMBEDDINGS_MODEL_NAME") self.embeddings_api_key = os.getenv("EMBEDDINGS_API_KEY") self.embeddings_url = os.getenv("EMBEDDINGS_URL") def _set_models(self) -> None: config = AutoConfig.from_pretrained(self.embeddings_model_name) assert self.chunk_size <= config.max_position_embeddings llm = ChatOpenAI( model_name=self.llm_model_name, openai_api_key=self.llm_api_key, openai_api_base=self.llm_url + "/v1", ) embeddings = OpenAIEmbeddings( model=self.embeddings_model_name, deployment=self.embeddings_model_name, openai_api_key=self.embeddings_api_key, openai_api_base=self.embeddings_url + "/v1", tiktoken_enabled=False, ) self.llm = llm self.embeddings = embeddings def _set_vector_store(self) -> None: self.vector_store = InMemoryVectorStore(self.embeddings) def _get_generate_prompt( self, state: MessagesState, docs_content: str ) -> List[BaseMessage]: # Format into prompt system_message_content = f"{self.rag_prompt}" "\n\n" f"{docs_content}" conversation_messages = [ message for message in state["messages"] if message.type in ("human", "system") or (message.type == "ai" and not message.tool_calls) ] prompt = [SystemMessage(system_message_content)] + conversation_messages return prompt def _serialize_docs(self, docs: List[Document]) -> str: """Serialize documents.""" serialized_docs = [] for doc in docs: if "source" in doc.metadata: # remove "source" key from metadata and display it first src_string = doc.metadata.pop("source") if doc.metadata: src_string += f" {doc.metadata}" else: src_string = str(doc.metadata) serialized_docs.append( f"Source: {src_string}\n" f"Content: {doc.page_content}" ) return "\n\n".join(serialized_docs) def _set_graph(self) -> None: if self.use_tools: self._set_graph_with_tool_calling() else: self._set_graph_without_tool_calling() def _set_graph_with_tool_calling(self) -> None: @tool(response_format="content_and_artifact") def retrieve(query: str): """Use this tool to retrieve information from documents stored in the knowledge base.""" retrieved_docs = self.vector_store.similarity_search(query, k=self.top_k) serialized = self._serialize_docs(retrieved_docs) # return content, artifact return serialized, retrieved_docs def query_or_respond(state: MessagesState): """Generate tool call for retrieval or respond.""" llm_with_tools = self.llm.bind_tools([retrieve]) prompt = [SystemMessage(self.intro_prompt)] + state["messages"] response = llm_with_tools.invoke(prompt) # MessagesState appends messages to state instead of overwriting return {"messages": [response]} def generate(state: MessagesState): """Generate answer.""" # Get context from generated ToolMessages recent_tool_messages = [] for message in reversed(state["messages"]): if message.type == "tool": recent_tool_messages.append(message) else: break tool_messages = recent_tool_messages[::-1] docs_content = "\n\n".join(doc.content for doc in tool_messages) prompt = self._get_generate_prompt(state, docs_content) # Run response = self.llm.invoke(prompt) return {"messages": [response]} graph_builder = StateGraph(MessagesState) memory = MemorySaver() tools = ToolNode([retrieve]) # Node 1: Generate an AIMessage that may include a tool-call to be sent. graph_builder.add_node(query_or_respond) # Node 2: Execute the retrieval tool. graph_builder.add_node(tools) # Node 3: Generate a response using the retrieved content. graph_builder.add_node(generate) graph_builder.set_entry_point("query_or_respond") graph_builder.add_conditional_edges( "query_or_respond", tools_condition, {END: END, "tools": "tools"}, ) graph_builder.add_edge("tools", "generate") graph_builder.add_edge("generate", END) self.graph = graph_builder.compile(checkpointer=memory) def _set_graph_without_tool_calling(self) -> None: class ContextState(MessagesState): context: List[Document] def retrieve(state: ContextState): # Get last HumanMessage, which is the question for message in reversed(state["messages"]): if message.type == "human": question = message.content break else: raise ValueError("No human message found in the state.") retrieved_docs = self.vector_store.similarity_search(question) return {"context": retrieved_docs} def generate(state: ContextState): """Generate answer.""" docs_content = self._serialize_docs(state["context"]) prompt = self._get_generate_prompt(state, docs_content) # Run response = self.llm.invoke(prompt) return {"messages": [response]} graph_builder = StateGraph(ContextState).add_sequence([retrieve, generate]) graph_builder.add_edge(START, "retrieve") memory = MemorySaver() self.graph = graph_builder.compile(checkpointer=memory) def query( self, input_message: str, thread_id: str = None ) -> dict[str, List[BaseMessage]]: if thread_id is None: logging.warning("thread_id is NONE") thread_id = str(uuid.uuid4()) config = {"configurable": {"thread_id": thread_id}} res = self.graph.invoke( {"messages": [{"role": "user", "content": input_message}]}, config=config, ) return res def clear_vector_store(self) -> None: """Clear the vector store.""" logging.info("Clear the vector store") self._set_vector_store() ``` ### `src/reader.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import os from langchain.document_loaders.base import BaseLoader from langchain_community.document_loaders import ( BSHTMLLoader, Docx2txtLoader, PyPDFLoader, WebBaseLoader, ) from langchain_community.document_loaders.csv_loader import CSVLoader from langchain_community.document_loaders.text import TextLoader class MixedFileTypeLoader(BaseLoader): def __init__(self, file_path): self.file_path = file_path def load(self): if isinstance(self.file_path, list): documents = [] for file in self.file_path: docs = self._load_file(file) if isinstance(docs, list): documents.extend(docs) else: documents.append(docs) return documents return self._load_file(self.file_path) def _load_file(self, file): if file.startswith("http://") or file.startswith("https://"): is_url = True else: is_url = False if not os.path.exists(file): raise FileNotFoundError(f"File not found: {file}") file_extension = os.path.splitext(file)[1].lower() if is_url: return self._load_web(file) elif file_extension == ".pdf": return self._load_pdf(file) elif file_extension in [".doc", ".docx"]: return self._load_word(file) elif file_extension == ".txt": return self._load_txt(file) elif file_extension in [".html", ".htm"]: return self._load_html(file) elif file_extension == ".csv": return self._load_csv(file) else: raise ValueError(f"Unsupported file type: {file_extension}") def _load_pdf(self, file_path): loader = PyPDFLoader(file_path) return loader.load() def _load_word(self, file_path): loader = Docx2txtLoader(file_path) return loader.load() def _load_txt(self, file_path): loader = TextLoader(file_path) return loader.load() def _load_html(self, file_path): loader = BSHTMLLoader(file_path) return loader.load() def _load_csv(self, file_path): loader = CSVLoader(file_path) return loader.load() def _load_web(self, file_path): loader = WebBaseLoader(file_path) return loader.load() ```

⚡ Deploy this on FlexAI

Any model, NVIDIA or AMD, live in under 60 seconds. No code changes, no infra management, hardware-agnostic.

Get started → Talk to us
# Run Your First Training Job on FlexAI — Quickstart Guide Source: https://docs.flex.ai/blueprints/running-a-simple-training-job Get your first training job running on FlexAI in minutes. Step-by-step quickstart — no infra setup, managed checkpoints, built-in observability. This experiment demonstrates how easy it is to leverage **FlexAI** to run a Training Job with a couple of commands. We will use a simple example of training a causal language model (LLM) on the `wikitext` dataset using the `GPT-2` model. You will see that this straightforward process only requires two components: a training script and a dataset. The training script is responsible for defining the model, setting up and applying hyperparameters, running the training loop, and applying its respective evaluation logic, while the dataset contains the information that will be used to train the model. > **Note**: If you haven't already connected FlexAI to GitHub, you'll need to run `flexai code-registry connect` to set up a code registry connection. This allows FlexAI to pull repositories directly using the `-u` flag in training commands. In this experiment, we will use a pre-processed version of the the `wikitext` dataset that has been set up for the `GPT-2` model. > If you'd like to reproduce the pre-processing steps yourself to use a different dataset or simply to learn more about the process, you can refer to the [Manual Dataset Pre-processing](#manual-dataset-pre-processing) section below. 1. Download the dataset: ```bash theme={null} DATASET_NAME=gpt2-tokenized-wikitext && curl -L -o ${DATASET_NAME}.zip "https://bucket-docs-samples-99b3a05.s3.eu-west-1.amazonaws.com/${DATASET_NAME}.zip" && unzip ${DATASET_NAME}.zip && rm ${DATASET_NAME}.zip ``` 2. Upload the dataset (located in `gpt2-tokenized-wikitext/`) to FlexAI: ```bash theme={null} flexai dataset push gpt2-tokenized-wikitext --file gpt2-tokenized-wikitext ``` Now, it's time to train your LLM on the dataset you just *pushed* in the previous step, `gpt2-tokenized-wikitext`. This experiment uses the `GPT-2` model, however, the training script we will use leverages the HuggingFace Transformers `Trainer` class, which makes it easy to replace `GPT-2` with another model from the HuggingFace Model Hub. To start the Training Job, run the following command: ```bash theme={null} flexai training run first-training-job --repository-url https://github.com/flexaihq/blueprints --dataset gpt2-tokenized-wikitext \ --requirements-path code/causal-language-modeling/requirements.txt \ -- code/causal-language-modeling/train.py \ --do_eval \ --do_train \ --dataset_name wikitext \ --tokenized_dataset_load_dir /input/gpt2-tokenized-wikitext \ --model_name_or_path openai-community/gpt2 \ --output_dir /output-checkpoint \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 8 \ --logging_steps 50 \ --save_steps 500 \ --eval_steps 500 \ --eval_strategy steps ``` The first line defines the 3 main components required to run a Training Job in FlexAI: 1. The Training Job's name (`first-training-job`). 2. The URL of the repository containing the training script (`https://github.com/flexaihq/blueprints`). 3. The name of the dataset to be used (`gpt2-tokenized-wikitext`). The second line defines the script that will be executed when the Training Job is started (`code/causal-language-modeling/train.py`). After the second line come the script's arguments, which are passed to the script when it is executed to adjust the Training Job hyperparameters or customize its behavior. For instance, `--max_train_samples` and `--max_eval_samples` can be used to tweak the sample size. You can check the status and life cycle events of your Training Job by running: ```bash theme={null} flexai training inspect first-training-job ``` Additionally, you can view the logs of your Training Job by running: ```bash theme={null} flexai training logs first-training-job ``` Once the Training Job completes successfully, you will be able to list all the produced checkpoints: ```bash theme={null} flexai training checkpoints first-training-job ``` They can be downloaded with: ```bash theme={null} flexai checkpoint fetch "" ``` You now have a trained model that you can use for inference or further fine-tuning! Check out the [Extra](#optional-extra-steps) section below for more information on how to run your fine-tuned model locally, or even better, how to run the training script directly on FlexAI using an Interactive Training Session. You can also learn how to manually pre-process the dataset if you're interested in understanding the process better. You can also have a look at other FlexAI experiments within this repository to explore more advanced use cases and techniques. ## Optional Extra Steps ### Try your fine-tuned model locally You can run your newly fine-tuned model in a FlexAI Interactive Session or in a local env (e.g. `pipenv install --python 3.11`), if you have hardware that's capable of doing inference. #### 1. Clone this repository If you haven't already, clone this repository on your host machine: ```bash theme={null} git clone https://github.com/flexaihq/blueprints.git blueprints --depth 1 --branch main && cd blueprints ``` #### 2. Install the dependencies Depending on your environment, you might need to install - if not already - the experiments' dependencies by running: ```bash theme={null} pip install -r code/causal-language-modeling/requirements.txt ``` #### 3. Extract the model artifacts First, list the available checkpoints from your training job: ```bash theme={null} flexai training checkpoints first-training-job ``` Then fetch the specific checkpoint you want to use (replace `` with the actual checkpoint ID from the list): ```bash theme={null} flexai checkpoint fetch "" --destination ./checkpoint ``` This will download the checkpoint to a local `checkpoint` directory. Make note of this location, as you will use it next. #### 4. Run the inference script Run the script made for inference on this model by running the command below, replacing `**PATH_TO_THE_CHECKPOINT_DIRECTORY**` with the path to the checkpoint directory you downloaded: ```bash theme={null} python code/causal-language-modeling/predict.py \ --model_name_or_path **PATH_TO_THE_CHECKPOINT_DIRECTORY** \ --input_str "Once upon a time, " \ --max_new_tokens 30 ``` ### Run the training script directly on FlexAI using an Interactive Training Session An Interactive Training Session allows you to connect to a Training Environment runtime on FlexAI and run your both training and prediction or inference scripts directly from this environment. This is a great way to test your scripts and experiment with different hyperparameters without having to create multiple Training Jobs per configuration change. You will find the guide on how to run an Interactive Training Session in the [FlexAI Documentation](https://docs.flex.ai/cli/guides/interactive-training/). You'll need to use the path for the `flexaihq/blueprints` repository as your `--repository-url` and pass the `gpt2-tokenized-wikitext` dataset you pushed earlier as `--dataset`, unless you want to leverage the Interactive Training Session's compute resources to manually pre-process the dataset. ### Manual Dataset Pre-processing To prepare and save the `wikitext` dataset for the `GPT-2` model run the following command: ```bash theme={null} python code/dataset/prepare_save_dataset.py \ --dataset_name wikitext \ --tokenized_dataset_save_dir gpt2-tokenized-wikitext \ --dataset_config_name wikitext-2-raw-v1 \ --tokenizer_model_name openai-community/gpt2 \ --dataset_group_text true ``` The generated dataset will be created in the directory set as the value of `--tokenized_dataset_save_dir`, in this case: `gpt2-tokenized-wikitext`. Keep in mind that you can use other combinations of datasets and models available on HuggingFace. ## Code ### `code/causal-language-modeling/train.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import math import os import sys from dataclasses import dataclass, field from typing import Optional import evaluate import numpy import torch import transformers from transformers import ( AutoConfig, AutoModelForCausalLM, AutoTokenizer, HfArgumentParser, Trainer, TrainingArguments, default_data_collator, ) SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) sys.path.append(os.path.dirname(SCRIPT_DIR)) from dataset.prepare_save_dataset import DatasetArguments, load_and_tokenize from utils.experiment_tracking import set_wandb transformers.logging.set_verbosity_info() @dataclass class ModelArguments: """ Model arguments. """ model_name_or_path: str = field( default=None, metadata={ "help": "Model name hosted inside a model repo on HF or a directory containing model weights." }, ) torch_dtype: Optional[str] = field( default=None, metadata={ "help": ( "Override the default `torch.dtype` and load the model under this dtype. If `auto` is passed, the " "dtype will be automatically derived from the model's weights." ), "choices": ["auto", "bfloat16", "float16", "float32"], }, ) attn_implementation: Optional[str] = field( default=None, metadata={ "help": ( "The attention implementation to use in the model (if relevant). " "E.g. `flash_attention_2`." ) }, ) @dataclass class AdditionalArguments: """ Additional arguments that are not part of the Hugging Face Trainer/Model/Dataset arguments. """ max_train_samples: Optional[int] = field( default=None, metadata={ "help": ( "For debugging purposes or quicker training, truncate the number of training examples to this " "value if set." ) }, ) max_eval_samples: Optional[int] = field( default=None, metadata={ "help": ( "For debugging purposes or quicker training, truncate the number of evaluation examples to this " "value if set." ) }, ) def parse_args(): parser = HfArgumentParser( (DatasetArguments, ModelArguments, TrainingArguments, AdditionalArguments) ) return parser.parse_args_into_dataclasses() def _load_model_and_tokenizer(model_args, print_model=False): # Load pretrained model and tokenizer config = AutoConfig.from_pretrained(model_args.model_name_or_path) tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path) tokenizer.pad_token = tokenizer.eos_token tokenizer.pad_token_id = tokenizer.eos_token_id torch_dtype = ( model_args.torch_dtype if model_args.torch_dtype in ["auto", None] else getattr(torch, model_args.torch_dtype) ) extra_model_args = {} if model_args.attn_implementation is not None: extra_model_args["attn_implementation"] = model_args.attn_implementation model = AutoModelForCausalLM.from_pretrained( model_args.model_name_or_path, config=config, torch_dtype=torch_dtype, **extra_model_args, ) if print_model: print(model) return model, tokenizer def train(dataset_args, model_args, training_args, additional_args): set_wandb(training_args) print(f"Training/evaluation parameters {training_args}") # Get dataset train_dataset, eval_dataset = load_and_tokenize( tokenizer_model_name=model_args.model_name_or_path, do_eval=training_args.do_eval, **vars(dataset_args), ) max_train_samples = float("inf") max_eval_samples = float("inf") if not dataset_args.dataset_streaming: max_train_samples = len(train_dataset) if training_args.do_eval: max_eval_samples = len(eval_dataset) if additional_args.max_train_samples is not None: max_train_samples = min(max_train_samples, additional_args.max_train_samples) train_dataset = train_dataset.take(max_train_samples) if additional_args.max_eval_samples is not None: assert training_args.do_eval, "Cannot set max_eval_samples without do_eval" max_eval_samples = min(max_eval_samples, additional_args.max_eval_samples) eval_dataset = eval_dataset.take(max_eval_samples) model, tokenizer = _load_model_and_tokenizer(model_args, print_model=True) # Setup evaluation metric def preprocess_logits_for_metrics(logits, labels): if isinstance(logits, tuple): logits = logits[0] return logits.argmax(dim=-1) metric = evaluate.load("accuracy") def compute_metrics(eval_preds): preds, labels = eval_preds mask = (labels != tokenizer.pad_token_id) & (labels != -100) labels = numpy.concatenate( [label[mask[i]][1:] for i, label in enumerate(labels)] ) preds = numpy.concatenate([pred[mask[i]][:-1] for i, pred in enumerate(preds)]) return metric.compute(predictions=preds, references=labels) # Initialize Trainer trainer = Trainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset, tokenizer=tokenizer, data_collator=default_data_collator, compute_metrics=compute_metrics, preprocess_logits_for_metrics=preprocess_logits_for_metrics, ) # Train train_result = trainer.train( resume_from_checkpoint=training_args.resume_from_checkpoint ) trainer.save_model() metrics = train_result.metrics metrics["train_samples"] = max_train_samples trainer.log_metrics("train", metrics) trainer.save_metrics("train", metrics) trainer.save_state() if training_args.do_eval: metrics = trainer.evaluate() metrics["eval_samples"] = max_eval_samples try: perplexity = math.exp(metrics["eval_loss"]) except OverflowError: perplexity = float("inf") metrics["perplexity"] = perplexity trainer.log_metrics("eval", metrics) trainer.save_metrics("eval", metrics) if __name__ == "__main__": train(*parse_args()) ``` ### `code/causal-language-modeling/predict.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import os import sys from dataclasses import dataclass, field from transformers import HfArgumentParser SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) sys.path.append(SCRIPT_DIR) from train import ModelArguments, _load_model_and_tokenizer @dataclass class PredictArguments: """ Arguments for prediction. """ input_str: str = field( metadata={"help": "Input string to generate predictions for."}, ) max_new_tokens: int = field( default=100, metadata={"help": "Maximum number of tokens to generate."}, ) def parse_args(): parser = HfArgumentParser((ModelArguments, PredictArguments)) return parser.parse_args_into_dataclasses() def predict(model_args, predict_args): model, tokenizer = _load_model_and_tokenizer(model_args, None) input_str = predict_args.input_str model_inputs = tokenizer(input_str, return_tensors="pt") output = model.generate(**model_inputs, max_new_tokens=predict_args.max_new_tokens) print("\nGenerated text:") print(tokenizer.decode(output[0], skip_special_tokens=True)) if __name__ == "__main__": predict(*parse_args()) ``` ### `code/causal-language-modeling/requirements.txt` ```text theme={null} accelerate>=1.8.1 datasets>=2.21.0 evaluate>=0.4.3 scikit_learn>=1.5.2 transformers>=4.43.3 wandb>=0.18.1 ```

🚀 Run this on FlexAI

Managed checkpoints mean you never lose a run to preemption. Jobs launch in under 60 seconds — no infra setup, built-in observability.

Get started → Talk to us
# Run Distributed Data Parallel (DDP) Training on FlexAI Source: https://docs.flex.ai/blueprints/running-a-simple-training-job-with-ddp Start a distributed DDP training job on FlexAI with just 2 flags. Multi-GPU training made simple — no SLURM, no infra config, launch in under 60 seconds. This experiment demonstrates how easy it is to leverage **FlexAI** to run a Training Job with a couple of commands. We will use a simple example of training a causal language model (LLM) on the `wikitext` dataset using the `GPT-2` model. You will see that this straightforward process only requires two components: a training script and a dataset. The training script is responsible for defining the model, setting up and applying hyperparameters, running the training loop, and applying its respective evaluation logic, while the dataset contains the information that will be used to train the model. If you haven't already connected FlexAI to GitHub, you'll need to set up a code registry connection: ```bash theme={null} flexai code-registry connect ``` This will allow FlexAI to pull repositories directly from GitHub using the `-u` flag in training commands. In this experiment, we will use a pre-processed version of the the `wikitext` dataset that has been set up for the `GPT-2` model. > If you'd like to reproduce the pre-processing steps yourself to use a different dataset or simply to learn more about the process, you can refer to the [Manual Dataset Pre-processing](#manual-dataset-pre-processing) section below. 1. Download the dataset: ```bash theme={null} DATASET_NAME=gpt2-tokenized-wikitext && curl -L -o ${DATASET_NAME}.zip "https://bucket-docs-samples-99b3a05.s3.eu-west-1.amazonaws.com/${DATASET_NAME}.zip" && unzip ${DATASET_NAME}.zip && rm ${DATASET_NAME}.zip ``` 2. Upload the dataset (located in `gpt2-tokenized-wikitext/`) to FlexAI: ```bash theme={null} flexai dataset push gpt2-tokenized-wikitext --file gpt2-tokenized-wikitext ``` Now, it's time to train your LLM on the dataset you just *pushed* in the previous step, `gpt2-tokenized-wikitext`. This experiment uses the `GPT-2` model, however, the training script we will use leverages the HuggingFace Transformers `Trainer` class, which makes it easy to replace `GPT-2` with another model from the HuggingFace Model Hub. To start the Training Job, run the following command: ```bash theme={null} flexai training run first-ddp-training-job --repository-url https://github.com/flexaihq/blueprints --dataset gpt2-tokenized-wikitext \ --nodes 2 --accels 8 --requirements-path code/causal-language-modeling/requirements.txt \ -- code/causal-language-modeling/train.py \ --do_eval \ --do_train \ --dataset_name wikitext \ --tokenized_dataset_load_dir /input/gpt2-tokenized-wikitext \ --model_name_or_path openai-community/gpt2 \ --output_dir /output-checkpoint \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 8 \ --logging_steps 50 \ --save_steps 500 \ --eval_steps 500 \ --eval_strategy steps ``` The first line defines the 3 main components required to run a Training Job in FlexAI: 1. The Training Job's name (`first-ddp-training-job`). 2. The URL of the repository containing the training script (`https://github.com/flexaihq/blueprints`). 3. The name of the dataset to be used (`gpt2-tokenized-wikitext`). The second line allows you to pick the number of nodes you want to use for the Training Job, as well as the number of accelerators per each node. In this case the Training Job will run on 16 GPUs distributed across 2 nodes. The third line defines the script that will be executed when the Training Job is started (`code/causal-language-modeling/train.py`). After the third line come the script's arguments, which are passed to the script when it is executed to adjust the Training Job hyperparameters or customize its behavior. For instance, `--max_train_samples` and `--max_eval_samples` can be used to tweak the sample size. You can check the status and life cycle events of your Training Job by running: ```bash theme={null} flexai training inspect first-ddp-training-job ``` Additionally, you can view the logs of your Training Job by running: ```bash theme={null} flexai training logs first-ddp-training-job ``` Once the Training Job completes successfully, you will be able to list all the produced checkpoints: ```bash theme={null} flexai training checkpoints first-ddp-training-job ``` They can be downloaded with: ```bash theme={null} flexai checkpoint fetch "" ``` You now have a trained model that you can use for inference or further fine-tuning! Check out the [Extra](#optional-extra-steps) section below for more information on how to run your fine-tuned model locally, or even better, how to run the training script directly on FlexAI using an Interactive Training Session. You can also learn how to manually pre-process the dataset if you're interested in understanding the process better. You can also have a look at other FlexAI experiments within this repository to explore more advanced use cases and techniques. ## Optional Extra Steps ### Try your fine-tuned model locally You can run your newly fine-tuned model in a FlexAI Interactive Session or in a local env (e.g. `pipenv install --python 3.10`), if you have hardware that's capable of doing inference. #### 1. Clone this repository If you haven't already, clone this repository on your host machine: ```bash theme={null} git clone https://github.com/flexaihq/blueprints.git blueprints --depth 1 --branch main && cd blueprints ``` #### 2. Install the dependencies Depending on your environment, you might need to install - if not already - the experiments' dependencies by running: ```bash theme={null} pip install -r code/causal-language-modeling/requirements.txt ``` #### 3. Extract the model artifacts First, list the available checkpoints from your training job: ```bash theme={null} flexai training checkpoints first-ddp-training-job ``` Then fetch the specific checkpoint you want to use (replace `` with the actual checkpoint ID from the list): ```bash theme={null} flexai checkpoint fetch "" --destination ./checkpoint ``` This will download the checkpoint to a local `checkpoint` directory. Make note of this location, as you will use it next. #### 4. Run the inference script Run the script made for inference on this model by running the command below, replacing `**PATH_TO_THE_CHECKPOINT_DIRECTORY**` with the path to the checkpoint directory you downloaded: ```bash theme={null} python code/causal-language-modeling/predict.py \ --model_name_or_path **PATH_TO_THE_CHECKPOINT_DIRECTORY** \ --input_str "Once upon a time, " \ --max_new_tokens 30 ``` ### Run the training script directly on FlexAI using an Interactive Training Session An Interactive Training Session allows you to connect to a Training Environment runtime on FlexAI and run your both training and prediction or inference scripts directly from this environment. This is a great way to test your scripts and experiment with different hyperparameters without having to create multiple Training Jobs per configuration change. You will find the guide on how to run an Interactive Training Session in the [FlexAI Documentation](https://docs.flex.ai/cli/guides/interactive-training/). You'll need to use the path for the `flexaihq/blueprints` repository as your `--repository-url` and pass the `gpt2-tokenized-wikitext` dataset you pushed earlier as `--dataset`, unless you want to leverage the Interactive Training Session's compute resources to manually pre-process the dataset. ### Manual Dataset Pre-processing To prepare and save the `wikitext` dataset for the `GPT-2` model run the following command: ```bash theme={null} python code/dataset/prepare_save_dataset.py \ --dataset_name wikitext \ --tokenized_dataset_save_dir gpt2-tokenized-wikitext \ --dataset_config_name wikitext-2-raw-v1 \ --tokenizer_model_name openai-community/gpt2 \ --dataset_group_text true ``` The generated dataset will be created in the directory set as the value of `--tokenized_dataset_save_dir`, in this case: `gpt2-tokenized-wikitext`. Keep in mind that you can use other combinations of datasets and models available on HuggingFace. ## Code ### `code/causal-language-modeling/train.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import math import os import sys from dataclasses import dataclass, field from typing import Optional import evaluate import numpy import torch import transformers from transformers import ( AutoConfig, AutoModelForCausalLM, AutoTokenizer, HfArgumentParser, Trainer, TrainingArguments, default_data_collator, ) SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) sys.path.append(os.path.dirname(SCRIPT_DIR)) from dataset.prepare_save_dataset import DatasetArguments, load_and_tokenize from utils.experiment_tracking import set_wandb transformers.logging.set_verbosity_info() @dataclass class ModelArguments: model_name_or_path: str = field(default=None) torch_dtype: Optional[str] = field(default=None) attn_implementation: Optional[str] = field(default=None) @dataclass class AdditionalArguments: max_train_samples: Optional[int] = field(default=None) max_eval_samples: Optional[int] = field(default=None) def parse_args(): parser = HfArgumentParser( (DatasetArguments, ModelArguments, TrainingArguments, AdditionalArguments) ) return parser.parse_args_into_dataclasses() def _load_model_and_tokenizer(model_args, print_model=False): config = AutoConfig.from_pretrained(model_args.model_name_or_path) tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path) tokenizer.pad_token = tokenizer.eos_token tokenizer.pad_token_id = tokenizer.eos_token_id torch_dtype = ( model_args.torch_dtype if model_args.torch_dtype in ["auto", None] else getattr(torch, model_args.torch_dtype) ) extra_model_args = {} if model_args.attn_implementation is not None: extra_model_args["attn_implementation"] = model_args.attn_implementation model = AutoModelForCausalLM.from_pretrained( model_args.model_name_or_path, config=config, torch_dtype=torch_dtype, **extra_model_args, ) if print_model: print(model) return model, tokenizer def train(dataset_args, model_args, training_args, additional_args): set_wandb(training_args) print(f"Training/evaluation parameters {training_args}") train_dataset, eval_dataset = load_and_tokenize( tokenizer_model_name=model_args.model_name_or_path, do_eval=training_args.do_eval, **vars(dataset_args), ) max_train_samples = float("inf") max_eval_samples = float("inf") if not dataset_args.dataset_streaming: max_train_samples = len(train_dataset) if training_args.do_eval: max_eval_samples = len(eval_dataset) if additional_args.max_train_samples is not None: max_train_samples = min(max_train_samples, additional_args.max_train_samples) train_dataset = train_dataset.take(max_train_samples) if additional_args.max_eval_samples is not None: assert training_args.do_eval, "Cannot set max_eval_samples without do_eval" max_eval_samples = min(max_eval_samples, additional_args.max_eval_samples) eval_dataset = eval_dataset.take(max_eval_samples) model, tokenizer = _load_model_and_tokenizer(model_args, print_model=True) metric = evaluate.load("accuracy") def preprocess_logits_for_metrics(logits, labels): if isinstance(logits, tuple): logits = logits[0] return logits.argmax(dim=-1) def compute_metrics(eval_preds): preds, labels = eval_preds mask = (labels != tokenizer.pad_token_id) & (labels != -100) labels = numpy.concatenate([label[mask[i]][1:] for i, label in enumerate(labels)]) preds = numpy.concatenate([pred[mask[i]][:-1] for i, pred in enumerate(preds)]) return metric.compute(predictions=preds, references=labels) trainer = Trainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset, tokenizer=tokenizer, data_collator=default_data_collator, compute_metrics=compute_metrics, preprocess_logits_for_metrics=preprocess_logits_for_metrics, ) train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint) trainer.save_model() metrics = train_result.metrics metrics["train_samples"] = max_train_samples trainer.log_metrics("train", metrics) trainer.save_metrics("train", metrics) trainer.save_state() if training_args.do_eval: metrics = trainer.evaluate() metrics["eval_samples"] = max_eval_samples try: perplexity = math.exp(metrics["eval_loss"]) except OverflowError: perplexity = float("inf") metrics["perplexity"] = perplexity trainer.log_metrics("eval", metrics) trainer.save_metrics("eval", metrics) if __name__ == "__main__": train(*parse_args()) ``` ### `code/causal-language-modeling/predict.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import os import sys from dataclasses import dataclass, field from transformers import HfArgumentParser SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) sys.path.append(SCRIPT_DIR) from train import ModelArguments, _load_model_and_tokenizer @dataclass class PredictArguments: input_str: str = field(metadata={"help": "Input string to generate predictions for."}) max_new_tokens: int = field(default=100, metadata={"help": "Maximum number of tokens to generate."}) def parse_args(): parser = HfArgumentParser((ModelArguments, PredictArguments)) return parser.parse_args_into_dataclasses() def predict(model_args, predict_args): model, tokenizer = _load_model_and_tokenizer(model_args, None) input_str = predict_args.input_str model_inputs = tokenizer(input_str, return_tensors="pt") output = model.generate(**model_inputs, max_new_tokens=predict_args.max_new_tokens) print("\nGenerated text:") print(tokenizer.decode(output[0], skip_special_tokens=True)) if __name__ == "__main__": predict(*parse_args()) ``` ### `code/causal-language-modeling/requirements.txt` ```text theme={null} accelerate>=1.8.1 datasets>=2.21.0 evaluate>=0.4.3 scikit_learn>=1.5.2 transformers>=4.43.3 wandb>=0.18.1 ```

🚀 Run this on FlexAI

Managed checkpoints mean you never lose a run to preemption. Jobs launch in under 60 seconds — no infra setup, built-in observability.

Get started → Talk to us
# Deploy Speech-to-Text with FlexAI Inference Endpoints Source: https://docs.flex.ai/blueprints/speech-to-text-inference Build a speech-to-text transcription app using FlexAI inference endpoints. Record audio and get transcriptions in real time, hardware-agnostic deployment. This Speech-to-Text application provides an interactive interface for users to record audio messages using their microphone and receive accurate transcriptions. ## Start the FlexAI endpoints Create the FlexAI secret that contains your HF token in order to access the inference models: ```bash theme={null} # Enter your HF token value when prompted flexai secret create hf-token ``` Start the FlexAI endpoint of the LLM: ```bash theme={null} LLM_INFERENCE_NAME=speech2text flexai inference serve $LLM_INFERENCE_NAME --hf-token-secret hf-token --runtime vllm-nvidia-0.10.1 -- --model=openai/whisper-large-v3 ``` Store the returned Inference Endpoint API KEY and Endpoint URL: ```bash theme={null} export LLM_API_KEY= export LLM_URL=$(flexai inference inspect $LLM_INFERENCE_NAME -j | jq .config.endpointUrl -r) ``` You'll notice the last `export` line uses the `jq` tool to extract the value of `endpointUrl` from the JSON output of the `inspect` command. If you don't have it already, you can get `jq` from its official website: [https://jqlang.org/](https://jqlang.org/) ## Setup ```bash theme={null} cd code/speech-to-text/ ``` ```bash theme={null} pip install -r requirements.txt ``` ```bash theme={null} python main.py ``` The application will start and display two URLs: * **Local URL**: For local access (e.g., `http://127.0.0.1:7860`) * **Public URL**: For sharing (e.g., `https://xxxxxxxxxx.gradio.live`) Open either URL in your browser to start transcribing audio. ## Usage 1. **Access the Interface**: Open the Gradio interface in your web browser. To avoid any microphone access permission errors, prefer to use the **public URL** rather than the local one. 2. **Record Audio**: * Click the **record icon** to start recording * Speak your message * Click **stop** when finished recording 3. **Get Transcription**: Click the **"Transcribe"** button to process your audio and receive the text transcription. 4. **View Results**: The transcribed text will appear in the results panel on the right side of the interface. ## Code ### main.py ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import os import gradio as gr from openai import OpenAI def check_env() -> None: if "LLM_API_KEY" not in os.environ: raise ValueError("Please set the LLM_API_KEY environment variable.") if "LLM_URL" not in os.environ: raise ValueError("Please set the LLM_URL environment variable.") def infer(audio_path: str, client: OpenAI): with open(audio_path, "rb") as f: transcription = client.audio.transcriptions.create( file=f, model="openai/whisper-large-v3", response_format="json", temperature=0.0, extra_body=dict( seed=42, repetition_penalty=1.3, ), ) return transcription.text def transcribe_audio(audio_file): if audio_file is None: return "No audio file provided" client = OpenAI( api_key=os.environ.get("LLM_API_KEY"), base_url=os.environ.get("LLM_URL") + "/v1", ) result = infer(audio_file, client) return result def create_gradio_interface(): with gr.Blocks(title="Speech-to-Text Transcription") as demo: gr.Markdown("# Speech-to-Text Transcription") gr.Markdown("Record audio using your microphone and get the transcription.") with gr.Row(): with gr.Column(): audio_input = gr.Audio( sources=["microphone"], type="filepath", label="Record Audio" ) transcribe_btn = gr.Button("Transcribe", variant="primary") with gr.Column(): output_text = gr.Textbox( label="Transcription Result", lines=10, max_lines=20, placeholder="Your transcription will appear here...", ) transcribe_btn.click( fn=transcribe_audio, inputs=[audio_input], outputs=[output_text] ) return demo if __name__ == "__main__": check_env() demo = create_gradio_interface() demo.launch(share=True) ``` ### requirements.txt ```text theme={null} gradio>=5.43.1 openai>=1.101.0 ```

⚡ Deploy this on FlexAI

Any model, NVIDIA or AMD, live in under 60 seconds. No code changes, no infra management, hardware-agnostic.

Get started → Talk to us
# Stream Large Datasets During Training — No Memory Limits Source: https://docs.flex.ai/blueprints/streaming-datasets Train on datasets that don't fit in memory using streaming on FlexAI. No preprocessing, no memory limits, works with any dataset size. Step-by-step walkthrough. In some cases you might want to use large datasets that would be too large to download or push to FlexAI and you'd prefer to use that data transfer time more efficiently. *Streaming* such datasets can be a useful technique in those cases. This experiment demonstrates how to stream a large dataset during a Training Job on FlexAI. We'll use the [HuggingFace Datasets library](https://huggingface.co/docs/datasets/en/stream)'s Streaming capabilities to achieve this. ## Connect to GitHub (if needed) If you haven't already connected FlexAI to GitHub, you'll need to set up a code registry connection: ```bash theme={null} flexai code-registry connect ``` This will allow FlexAI to pull repositories directly from GitHub using the `-u` flag in training commands. ## Running the Training Job streaming a dataset Here is an example using the `code/causal-language-modeling/train.py` script to stream the over 90 TB [Fineweb dataset](https://huggingface.co/datasets/HuggingFaceFW/fineweb): ```bash theme={null} flexai training run gpt2training-stream --repository-url https://github.com/flexaihq/blueprints --requirements-path code/causal-language-modeling/requirements.txt \ -- code/causal-language-modeling/train.py \ --dataset_streaming true \ --do_train \ --eval_strategy no \ --dataset_name HuggingFaceFW/fineweb \ --dataset_config_name CC-MAIN-2024-10 \ --dataset_group_text true \ --dataloader_num_workers 8 \ --max_steps 2500 \ --model_name_or_path openai-community/gpt2 \ --output_dir /output-checkpoint \ --per_device_train_batch_size 8 \ --logging_steps 50 \ --save_steps 1000 ``` The first line defines the 3 main components required to run a Training Job in FlexAI: 1. The Training Job's name (`gpt2training-stream`). 2. The URL of the repository containing the training script (`https://github.com/flexaihq/blueprints`). 3. The name of the dataset to be used (`empty-dataset` or any other dataset you have available). The second line defines the script that will be executed when the Training Job is started (`code/causal-language-modeling/train.py`). Below that, the first argument passed to the script is `--dataset_streaming true`, which value tells the script to use the Datasets library with streaming capabilities enabled. The next lines specify the arguments that will be passed to the training script during execution to adjust the Training Job's hyperparameters or customize its behavior. For instance, `--max_train_samples` and `--max_eval_samples` can be used to tweak the sample size. ## The code You will notice that the `train` function in the `code/causal-language-modeling/train.py` script makes a call to the `_load_model_and_tokenizer` function to load the model and tokenizer using the user-provided arguments: ```python theme={null} def train(dataset_args, model_args, training_args, additional_args): # <--- 1. This is the function that will be called by the `flexai training run` command set_wandb(training_args) print(f"Training/evaluation parameters {training_args}") # Get dataset train_dataset, eval_dataset = load_and_tokenize( # <--- 2. Here the script calls the `load_and_tokenize` helper function tokenizer_model_name=model_args.model_name_or_path, do_eval=training_args.do_eval, **vars(dataset_args), # <--- 3. These are the arguments passed to the script ) ``` The `load_and_tokenize` helper function from the `code/dataset/prepare_save_dataset.py` file is the one responsible for using the HuggingFace's Datasets library and enable its *streaming capabilities* by simply setting the `load_dataset`'s `streaming` argument to `True`: ```python theme={null} def load_and_tokenize( dataset_name: str, # ... dataset_streaming: bool, ) -> Dict[str, Dataset]: # ... loaded_datasets = load_dataset( # <--- 1. HuggingFace's Datasets library `load_dataset` function is called dataset_name, dataset_config_name, streaming=dataset_streaming, # <--- 2. The `streaming` argument is set to `True` ) ``` This is all that is needed to stream a dataset during a Training Job on FlexAI! You are no longer restricted by the challenges that come with large dataset transfer processes, and can now use them more efficiently. ## Code ### `code/causal-language-modeling/train.py` ```python theme={null} # Copyright (c) 2025 FlexAI # This file is part of the FlexAI Experiments repository. # SPDX-License-Identifier: MIT import math import os import sys from dataclasses import dataclass, field from typing import Optional import evaluate import numpy import torch import transformers from transformers import ( AutoConfig, AutoModelForCausalLM, AutoTokenizer, HfArgumentParser, Trainer, TrainingArguments, default_data_collator, ) SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) sys.path.append(os.path.dirname(SCRIPT_DIR)) from dataset.prepare_save_dataset import DatasetArguments, load_and_tokenize from utils.experiment_tracking import set_wandb transformers.logging.set_verbosity_info() @dataclass class ModelArguments: model_name_or_path: str = field(default=None) torch_dtype: Optional[str] = field(default=None) attn_implementation: Optional[str] = field(default=None) @dataclass class AdditionalArguments: max_train_samples: Optional[int] = field(default=None) max_eval_samples: Optional[int] = field(default=None) def parse_args(): parser = HfArgumentParser( (DatasetArguments, ModelArguments, TrainingArguments, AdditionalArguments) ) return parser.parse_args_into_dataclasses() def _load_model_and_tokenizer(model_args, print_model=False): config = AutoConfig.from_pretrained(model_args.model_name_or_path) tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path) tokenizer.pad_token = tokenizer.eos_token tokenizer.pad_token_id = tokenizer.eos_token_id torch_dtype = ( model_args.torch_dtype if model_args.torch_dtype in ["auto", None] else getattr(torch, model_args.torch_dtype) ) extra_model_args = {} if model_args.attn_implementation is not None: extra_model_args["attn_implementation"] = model_args.attn_implementation model = AutoModelForCausalLM.from_pretrained( model_args.model_name_or_path, config=config, torch_dtype=torch_dtype, **extra_model_args, ) if print_model: print(model) return model, tokenizer def train(dataset_args, model_args, training_args, additional_args): set_wandb(training_args) print(f"Training/evaluation parameters {training_args}") train_dataset, eval_dataset = load_and_tokenize( tokenizer_model_name=model_args.model_name_or_path, do_eval=training_args.do_eval, **vars(dataset_args), ) max_train_samples = float("inf") max_eval_samples = float("inf") if not dataset_args.dataset_streaming: max_train_samples = len(train_dataset) if training_args.do_eval: max_eval_samples = len(eval_dataset) if additional_args.max_train_samples is not None: max_train_samples = min(max_train_samples, additional_args.max_train_samples) train_dataset = train_dataset.take(max_train_samples) if additional_args.max_eval_samples is not None: assert training_args.do_eval, "Cannot set max_eval_samples without do_eval" max_eval_samples = min(max_eval_samples, additional_args.max_eval_samples) eval_dataset = eval_dataset.take(max_eval_samples) model, tokenizer = _load_model_and_tokenizer(model_args, print_model=True) metric = evaluate.load("accuracy") def preprocess_logits_for_metrics(logits, labels): if isinstance(logits, tuple): logits = logits[0] return logits.argmax(dim=-1) def compute_metrics(eval_preds): preds, labels = eval_preds mask = (labels != tokenizer.pad_token_id) & (labels != -100) labels = numpy.concatenate([label[mask[i]][1:] for i, label in enumerate(labels)]) preds = numpy.concatenate([pred[mask[i]][:-1] for i, pred in enumerate(preds)]) return metric.compute(predictions=preds, references=labels) trainer = Trainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset, tokenizer=tokenizer, data_collator=default_data_collator, compute_metrics=compute_metrics, preprocess_logits_for_metrics=preprocess_logits_for_metrics, ) train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint) trainer.save_model() metrics = train_result.metrics metrics["train_samples"] = max_train_samples trainer.log_metrics("train", metrics) trainer.save_metrics("train", metrics) trainer.save_state() if training_args.do_eval: metrics = trainer.evaluate() metrics["eval_samples"] = max_eval_samples try: perplexity = math.exp(metrics["eval_loss"]) except OverflowError: perplexity = float("inf") metrics["perplexity"] = perplexity trainer.log_metrics("eval", metrics) trainer.save_metrics("eval", metrics) if __name__ == "__main__": train(*parse_args()) ``` ### `code/causal-language-modeling/requirements.txt` ```text theme={null} accelerate>=1.8.1 datasets>=2.21.0 evaluate>=0.4.3 scikit_learn>=1.5.2 transformers>=4.43.3 wandb>=0.18.1 ```

🚀 Run this on FlexAI

Managed checkpoints mean you never lose a run to preemption. Jobs launch in under 60 seconds — no infra setup, built-in observability.

Get started → Talk to us
# Generate Audio with Stable Audio Open on FlexAI Endpoints Source: https://docs.flex.ai/blueprints/text-to-audio-inference Deploy Stable Audio Open 1.0 for audio generation via FlexAI inference endpoints. High-quality output, hardware-agnostic, scales automatically. This experiment demonstrates how to deploy and use a text-to-audio model (Stable Audio Open 1.0) using Flexai's inference serving capabilities. ## Prerequisites Before starting, make sure you have: * A Flexai account with access to the platform * A Hugging Face token with access to the `stabilityai/stable-audio-open-1.0` model * The `flexai` CLI installed and configured ## Setup FlexAI Secret for Hugging Face Token First, create a FlexAI secret that contains your Hugging Face token to access the inference model: ```bash theme={null} # Enter your HF token value when prompted flexai secret create MY_HF_TOKEN ``` Make sure your Hugging Face token has access to the `stabilityai/stable-audio-open-1.0` model. You may need to accept the model's license terms on Hugging Face first. ## Start the FlexAI Inference Endpoint Start the FlexAI endpoint for the Stable Audio Open 1.0 model: ```bash theme={null} INFERENCE_NAME=stable-audio-open flexai inference serve $INFERENCE_NAME --runtime flexserve --hf-token-secret MY_HF_TOKEN -- --task text-to-audio --model stabilityai/stable-audio-open-1.0 ``` This command will: * Create an inference endpoint named `stable-audio-open` * Use the `flexserve` runtime optimized for text-to-audio tasks * Load the Stable Audio Open 1.0 model from Hugging Face * Configure it for text-to-audio generation ## Get Endpoint Information Once the endpoint is deployed, you'll see the API key displayed in the output. Store it in an environment variable: ```bash theme={null} export INFERENCE_API_KEY= ``` Then retrieve the endpoint URL: ```bash theme={null} export INFERENCE_URL=$(flexai inference inspect $INFERENCE_NAME -j | jq .config.endpointUrl -r) ``` You'll notice these `export` lines use the `jq` tool to extract values from the JSON output of the `inspect` command. If you don't have it already, you can get `jq` from its official website: [https://jqlang.org/](https://jqlang.org/) ## Generate Audio Now you can generate audio by making HTTP POST requests to your endpoint. Here are some examples: ### Example 1: Relaxing Music ```bash theme={null} curl -X POST \ -H "Authorization: Bearer $INFERENCE_API_KEY" \ -H 'Content-Type: application/json' \ -d '{ "inputs": "Relaxing piano music with soft ambient sounds, calm and peaceful", "parameters": { "audio_end_in_s": 10.0, "num_inference_steps": 200, "guidance_scale": 7.0, "seed": 42 } }' \ -o relaxing_music.wav \ "$INFERENCE_URL/v1/audios/generations" ``` ### Example 2: Nature Soundscape ```bash theme={null} curl -X POST \ -H "Authorization: Bearer $INFERENCE_API_KEY" \ -H 'Content-Type: application/json' \ -d '{ "inputs": "Forest ambience with birds chirping and a gentle stream flowing", "parameters": { "audio_end_in_s": 15.0, "num_inference_steps": 200, "guidance_scale": 7.0, "negative_prompt": "distorted, low quality, muffled", "seed": 123 } }' \ -o nature_sounds.wav \ "$INFERENCE_URL/v1/audios/generations" ``` ### Example 3: Electronic Beat ```bash theme={null} curl -X POST \ -H "Authorization: Bearer $INFERENCE_API_KEY" \ -H 'Content-Type: application/json' \ -d '{ "inputs": "Upbeat electronic music with synthesizers and energetic drums, 128 bpm", "parameters": { "audio_end_in_s": 20.0, "num_inference_steps": 200, "guidance_scale": 7.0, "seed": 456 } }' \ -o electronic_beat.wav \ "$INFERENCE_URL/v1/audios/generations" ``` These will save the generated audio files in your current directory. ## Parameters Explanation The API accepts the following parameters: * **inputs**: The text prompt describing the audio you want to generate * **audio\_length\_in\_s**: Output audio duration in seconds (typically up to 47 seconds for Stable Audio Open) * **num\_inference\_steps**: Number of denoising steps (higher = better quality but slower, recommended: 100-200) * **guidance\_scale**: As you increase the value, the model tries harder to match your prompt. * **negative\_prompt**: Description of what you don't want in the audio (helps improve quality) * **seed**: Random seed for reproducible results

⚡ Deploy this on FlexAI

Any model, NVIDIA or AMD, live in under 60 seconds. No code changes, no infra management, hardware-agnostic.

Get started → Talk to us
# Text-to-Speech with Qwen3-TTS on FlexAI Inference Endpoints Source: https://docs.flex.ai/blueprints/text-to-audio-qwen3-tts Deploy Qwen3-TTS for high-quality text-to-speech synthesis via FlexAI inference endpoints. Interactive demo, low-latency, production-ready. This experiment demonstrates how to deploy and use a text-to-audio model (Qwen3-TTS) using Flexai's inference serving capabilities. ## Setup The demo code for this experiment is located at `code/qwen3-tts` but it is advisable to follow the steps below before jumping to the full demo. ## Prerequisites Before starting, make sure you have: * A Flexai account with access to the platform * The `flexai` CLI installed and configured ### Start the FlexAI Inference Endpoint for voice cloning Base model capable of 3-second rapid voice clone from user audio input; can be used for fine-tuning (FT) other models. Start the FlexAI endpoint for the Qwen3-TTS-12Hz-1.7B-Base model: ```bash theme={null} INFERENCE_NAME=Qwen3-base flexai inference serve $INFERENCE_NAME --runtime vllm-omni-0.14.0 -- Qwen/Qwen3-TTS-12Hz-1.7B-Base --stage-configs-path /workspace/vllm-omni/vllm_omni/model_executor/stage_configs/qwen3_tts.yaml --omni --trust-remote-code --enforce-eager ``` This command will: * Create an inference endpoint named `Qwen3-base` * Use the `vllm-omni` runtime * Load the Qwen3-TTS-12Hz-1.7B-Base model from Hugging Face ### Get Endpoint Information Once the endpoint is deployed, you'll see the API key displayed in the output. Store it in an environment variable: ```bash theme={null} export INFERENCE_API_KEY_BASE= ``` Then retrieve the endpoint URL: ```bash theme={null} export INFERENCE_URL_BASE=$(flexai inference inspect $INFERENCE_NAME -j | jq .config.endpointUrl -r) ``` You'll notice these `export` lines use the `jq` tool to extract values from the JSON output of the `inspect` command. If you don't have it already, you can get `jq` from its official website: [https://jqlang.org/](https://jqlang.org/) ### Generate Audio Now you can clone a voice by making HTTP POST requests to your endpoint. Here is an example: ```bash theme={null} curl -v -X POST -H "Authorization: Bearer $INFERENCE_API_KEY_BASE" \ -H "Content-Type: application/json" \ -d '{ "input": "Hello, how are you? I am so excited. For Sure!", "ref_text": "Exactly. And, you know, one of the things we have, we ve been wondering about why some of these companies in the last seven, eight years are in the graveyard. And one of the challenges was they all went after CUDA or NVIDIA silicon.", "ref_audio": "https://tmpfiles.org/26240870/sample_1.wav", "language": "Auto", "task_type": "Base" }' -o excited.wav \ $INFERENCE_URL_BASE/v1/audio/speech ``` ### Parameters Explanation The API accepts the following parameters: * **inputs**: The text prompt describing the text you want to generate * **ref\_text**: The transcription of the reference audio file (the voice you want to clone) * **ref\_audio**: A URL pointing to the reference audio file (must be in WAV format and less than 10 seconds long) * **language**: Language of the input text (set to "Auto" for automatic detection) * **task\_type**: Set to "Base" for voice cloning ### Start the FlexAI Inference Endpoint for using custom voice Provides style control over target timbres via user instructions; supports 9 premium timbres covering various combinations of gender, age, language, and dialect. Start the FlexAI endpoint for the Qwen3-TTS-12Hz-1.7B-CustomVoice model: ```bash theme={null} INFERENCE_NAME=Qwen3-custom-voice flexai inference serve $INFERENCE_NAME --runtime vllm-omni-0.14.0 -- Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --stage-configs-path /workspace/vllm-omni/vllm_omni/model_executor/stage_configs/qwen3_tts.yaml --omni --trust-remote-code --enforce-eager ``` This command will: * Create an inference endpoint named `Qwen3-custom-voice` * Use the `vllm-omni` runtime * Load the Qwen3-TTS-12Hz-1.7B-CustomVoice model from Hugging Face ### Get Endpoint Information Once the endpoint is deployed, you'll see the API key displayed in the output. Store it in an environment variable: ```bash theme={null} export INFERENCE_API_KEY_TTS= ``` Then retrieve the endpoint URL: ```bash theme={null} export INFERENCE_URL_TTS=$(flexai inference inspect $INFERENCE_NAME -j | jq .config.endpointUrl -r) ``` ### Generate Audio Now you can create audio with the provided custom voice by making HTTP POST requests to your endpoint. Here is an example: ```bash theme={null} curl -v -X POST \ -H "Authorization: Bearer $INFERENCE_API_KEY_TTS" \ -H "Content-Type: application/json" \ -d '{ "input": "Hello, how are you?", "voice": "vivian", "language": "English" }' \ -o output.wav \ "$INFERENCE_URL_TTS/v1/audio/speech" ``` ```bash theme={null} curl -X POST "$INFERENCE_URL_TTS/v1/audio/speech" \ -H "Content-Type: application/json" \ -d '{ "input": "I am so excited!", "voice": "vivian", "language": "English", "instructions": "Speak with great enthusiasm" }' --output excited.wav ``` ### Parameters Explanation The API accepts the following parameters: * **inputs**: The text prompt describing the text you want to generate * **voice**: The voice you want to use (you can list the available voices by making a GET request to the endpoint's `/v1/audio/voices` path) * **instructions**: How the model should speak the text (e.g., "Speak with great enthusiasm", "Speak like a news anchor", etc.) * **language**: Language of the input text (set to "Auto" for automatic detection) ## Demo App The demo app allows you to easily test the endpoints you just created. You can find it in the `code/qwen3-tts` directory. In the same directory you will find scripts to start the inference endpoints for both the base and custom voice models. Make sure to start both endpoints before running the demo app: ### Start Qwen3-TTS-12Hz-1.7B-Base model ```shell theme={null} source ./START/run-qwen3-tts-cloneVoice.sh ``` ### Start Qwen3-TTS-12Hz-1.7B-CustomVoice ```shell theme={null} source ./START/run-qwen3-tts-customVoice.sh ``` ### Start the demo app #### Using uv (recommended) ``` uv run app.py ``` #### Using pip ``` python3.13 -m venv venv source ./venv/bin/activate uv pip install -r requirements.txt ``` then you can start the demo with: ``` python app.py ``` ## Code ### app.py ```python theme={null} # coding=utf-8 # Qwen3-TTS Gradio Demo # Supports: Voice Clone (Base), TTS (CustomVoice) # based on https://huggingface.co/spaces/Qwen/Qwen3-TTS # Copyright 2026 The Alibaba Qwen team. # SPDX-License-Identifier: Apache-2.0 import argparse import base64 import io import os import gradio as gr import numpy as np import soundfile as sf from openai import OpenAI # Speaker and language choices for CustomVoice model SPEAKERS = [ "Aiden", "Dylan", "Eric", "Ono_anna", "Ryan", "Serena", "Sohee", "Uncle_fu", "Vivian", ] LANGUAGES = [ "Auto", "Chinese", "English", "Japanese", "Korean", "French", "German", "Spanish", "Portuguese", "Russian", ] ## ============================================================================ ## GLOBAL MODEL LOADING - Load all models at startup ## ============================================================================ # Check required environment variables required_env_vars = [ "INFERENCE_API_KEY_BASE", "INFERENCE_URL_BASE", "INFERENCE_API_KEY_TTS", "INFERENCE_URL_TTS", ] missing_vars = [var for var in required_env_vars if not os.getenv(var)] if missing_vars: raise EnvironmentError( f"Missing required environment variables: {', '.join(missing_vars)}" ) class EndpointConfig: def __init__( self, tts_api_key=None, tts_url=None, voice_api_key=None, voice_url=None ): self.tts_api_key = tts_api_key self.tts_url = tts_url self.voice_api_key = voice_api_key self.voice_url = voice_url def get_endpoint_config(self): # Return as a list, not a tuple, and append 'none' as the fifth element return [self.tts_api_key, self.tts_url, self.voice_api_key, self.voice_url] def set_endpoint_config(self, tts_api_key, tts_url, voice_api_key, voice_url): self.tts_api_key = tts_api_key self.tts_url = tts_url self.voice_api_key = voice_api_key self.voice_url = voice_url class Models: def __init__(self): self.base_model_1_7b = None self.base_model_1_7b_model = None self.custom_voice_model_1_7b = None self.custom_voice_model_1_7b_model = None endpoint_config = EndpointConfig() models_endpoint = Models() def update_config(tts_api_key, tts_url, voice_api_key, voice_url): endpoint_config.set_endpoint_config(tts_api_key, tts_url, voice_api_key, voice_url) outcome = "" try: models_endpoint.base_model_1_7b = OpenAI(api_key=tts_api_key, base_url=tts_url) models = models_endpoint.base_model_1_7b.models.list() if len(models.data) != 1: raise ValueError( f"Expected exactly one model in the endpoint {tts_url}, but got {len(models)}" ) models_endpoint.base_model_1_7b_model = models.data[0].id outcome = "Saving config successfully!" except Exception as e: outcome = f"Error: {type(e).__name__}: {e}" try: models_endpoint.custom_voice_model_1_7b = OpenAI( api_key=voice_api_key, base_url=voice_url, ) models = models_endpoint.custom_voice_model_1_7b.models.list() if len(models.data) != 1: raise ValueError( f"Expected exactly one model in the endpoint {voice_url} , but got {len(models)}" ) models_endpoint.custom_voice_model_1_7b_model = models.data[0].id outcome = "Saving config successfully!" except Exception as e: outcome = f"Error: {type(e).__name__}: {e}" config = endpoint_config.get_endpoint_config() config.append(outcome) return config update_config( tts_api_key=os.getenv("INFERENCE_API_KEY_BASE"), tts_url=f"{os.getenv('INFERENCE_URL_BASE')}/v1", voice_api_key=os.getenv("INFERENCE_API_KEY_TTS"), voice_url=f"{os.getenv('INFERENCE_URL_TTS')}/v1", ) ## ============================================================================ def _normalize_audio(wav, eps=1e-12, clip=True): """Normalize audio to float32 in [-1, 1] range.""" x = np.asarray(wav) if np.issubdtype(x.dtype, np.integer): info = np.iinfo(x.dtype) if info.min < 0: y = x.astype(np.float32) / max(abs(info.min), info.max) else: mid = (info.max + 1) / 2.0 y = (x.astype(np.float32) - mid) / mid elif np.issubdtype(x.dtype, np.floating): y = x.astype(np.float32) m = np.max(np.abs(y)) if y.size else 0.0 if m > 1.0 + 1e-6: y = y / (m + eps) else: raise TypeError(f"Unsupported dtype: {x.dtype}") if clip: y = np.clip(y, -1.0, 1.0) if y.ndim > 1: y = np.mean(y, axis=-1).astype(np.float32) return y def _audio_to_tuple(audio): """Convert Gradio audio input to (wav, sr) tuple.""" if audio is None: return None if isinstance(audio, tuple) and len(audio) == 2 and isinstance(audio[0], int): sr, wav = audio wav = _normalize_audio(wav) return wav, int(sr) if isinstance(audio, dict) and "sampling_rate" in audio and "data" in audio: sr = int(audio["sampling_rate"]) wav = _normalize_audio(audio["data"]) return wav, sr return None def encode_audio_to_base64(audio_tuple) -> str: """Encode a (wav, sr) tuple to base64 data URL (WAV format).""" wav, sr = audio_tuple buffer = io.BytesIO() sf.write(buffer, wav, sr, format="WAV") buffer.seek(0) audio_bytes = buffer.read() audio_b64 = base64.b64encode(audio_bytes).decode("utf-8") return f"data:audio/wav;base64,{audio_b64}" def decode_response_audio(content) -> tuple: """Decode audio bytes from API response to (wav, sr) tuple.""" audio_buffer = io.BytesIO(content) audio_np, sr = sf.read(audio_buffer) return sr, audio_np def generate_voice_clone( ref_audio, ref_text, target_text, language, use_xvector_only, progress=gr.Progress(track_tqdm=True), ): """Generate speech using Base (Voice Clone) model.""" if not target_text or not target_text.strip(): return None, "Error: Target text is required." audio_tuple = _audio_to_tuple(ref_audio) audio_b64 = encode_audio_to_base64(audio_tuple) if audio_tuple is None: return None, "Error: Reference audio is required." if not use_xvector_only and (not ref_text or not ref_text.strip()): return ( None, "Error: Reference text is required when 'Use x-vector only' is not enabled.", ) try: response = models_endpoint.base_model_1_7b.audio.speech.create( input=target_text.strip(), voice=None, model=models_endpoint.base_model_1_7b_model, extra_body={ "language": language, "ref_audio": audio_b64, "ref_text": ref_text.strip() if ref_text else None, "x_vector_only_mode": use_xvector_only, "task_type": "Base", }, ) return ( decode_response_audio(response.content), "Generation completed successfully!", ) except Exception as e: print(response.content) return None, f"Error: {type(e).__name__}: {e}" def generate_custom_voice( text, language, speaker, instruct, progress=gr.Progress(track_tqdm=True) ): """Generate speech using CustomVoice model.""" if not text or not text.strip(): return None, "Error: Text is required." if not speaker: return None, "Error: Speaker is required." try: response = models_endpoint.custom_voice_model_1_7b.audio.speech.create( model=models_endpoint.custom_voice_model_1_7b_model, voice=speaker.lower().replace(" ", "_"), input=text.strip(), instructions=instruct.strip() if instruct else None, extra_body={ "language": language, }, ) return ( decode_response_audio(response.content), "Generation completed successfully!", ) except Exception as e: print(response.content) return None, f"Error: {type(e).__name__}: {e}" # Build Gradio UI def build_ui(): theme = gr.themes.Soft( font=[gr.themes.GoogleFont("Source Sans Pro"), "Arial", "sans-serif"], ) css = """ .gradio-container {max-width: none !important;} .tab-content {padding: 20px;} """ with gr.Blocks(theme=theme, css=css, title="Qwen3-TTS Demo") as demo: gr.Markdown( """ # Qwen3-TTS This demo is using flex.ai inference API: - **TTS (CustomVoice)**: Generate speech with predefined speakers and optional style instructions - **Voice Clone (Base)**: Clone any voice from a reference audio """ ) def toggle_api_key_visibility(visible, value): type = "text" if visible else "password" return gr.Textbox(label="API Key", type=type, value=value) with gr.Tabs(): # Tab 1: TTS (CustomVoice) with gr.Tab("TTS (CustomVoice)"): gr.Markdown("### Text-to-Speech with Predefined Speakers") with gr.Row(): with gr.Column(scale=2): tts_text = gr.Textbox( label="Text to Synthesize", lines=4, placeholder="Enter the text you want to convert to speech...", value=""" Hello! Welcome to Flex dot AI. This demo is using Flex private cloud inference API to run Qwen3-TTS models in the cloud, so no GPU is required on your end! Just enter your text and settings, and let the flex compute do the heavy lifting to generate high-quality speech at scale for you, fast and scalable inference capabilities included! And do not forget that you can also finetune your own custom TTS model here and then deploy on flex dot ai in just one click! """, ) with gr.Row(): tts_language = gr.Dropdown( label="Language", choices=LANGUAGES, value="English", interactive=True, ) tts_speaker = gr.Dropdown( label="Speaker", choices=SPEAKERS, value="Aiden", interactive=True, ) with gr.Row(): tts_instruct = gr.Textbox( label="Style Instruction (Optional)", lines=2, placeholder="e.g., Speak in a cheerful and energetic tone", value="Speak in a cheerful and energetic tone", ) tts_btn = gr.Button("Generate Speech", variant="primary") with gr.Column(scale=2): tts_audio_out = gr.Audio(label="Generated Audio", type="numpy") tts_status = gr.Textbox( label="Status", lines=2, interactive=False ) tts_btn.click( generate_custom_voice, inputs=[tts_text, tts_language, tts_speaker, tts_instruct], outputs=[tts_audio_out, tts_status], ) # Tab 2: Voice Clone (Base) with gr.Tab("Voice Clone (Base)"): gr.Markdown("### Clone Voice from Reference Audio") with gr.Row(): with gr.Column(scale=2): clone_ref_audio = gr.Audio( label="Reference Audio (Upload a voice sample to clone)", type="numpy", ) clone_ref_text = gr.Textbox( label="Reference Text (Transcript of the reference audio)", lines=2, placeholder="Enter the exact text spoken in the reference audio...", ) clone_xvector = gr.Checkbox( label="Use x-vector only (No reference text needed, but lower quality)", value=False, ) with gr.Column(scale=2): clone_target_text = gr.Textbox( label="Target Text (Text to synthesize with cloned voice)", lines=4, placeholder="Enter the text you want the cloned voice to speak...", ) with gr.Row(): clone_language = gr.Dropdown( label="Language", choices=LANGUAGES, value="Auto", interactive=True, ) clone_btn = gr.Button("Clone & Generate", variant="primary") with gr.Row(): clone_audio_out = gr.Audio(label="Generated Audio", type="numpy") clone_status = gr.Textbox( label="Status", lines=2, interactive=False ) clone_btn.click( generate_voice_clone, inputs=[ clone_ref_audio, clone_ref_text, clone_target_text, clone_language, clone_xvector, ], outputs=[clone_audio_out, clone_status], ) # Tab 3: Endpoint Config with gr.Tab("Endpoint Config"): with gr.Group(elem_id="tab-content"): tts_url = gr.Textbox(label="TTS CustomVoice URL") with gr.Row(): tts_api_key = gr.Textbox( placeholder="TTS CustomVoice API Key", type="password", scale=4, show_label=False, ) tts_api_key_visible = gr.Checkbox( label="Show TTS CustomVoice API Key", value=False, scale=1, ) voice_url = gr.Textbox(label="Voice Clone URL") with gr.Row(): voice_api_key = gr.Textbox( label="Voice Clone API Key", type="password", scale=4, show_label=False, ) voice_api_key_visible = gr.Checkbox( label="Voice Clone API Key", value=False, scale=1, ) save_btn = gr.Button("Save") config_error = gr.Markdown(value="", visible=False) save_btn.click( update_config, inputs=[ tts_api_key, tts_url, voice_api_key, voice_url, ], outputs=[ tts_api_key, tts_url, voice_api_key, voice_url, config_error, ], ) tts_api_key_visible.change( toggle_api_key_visibility, inputs=[tts_api_key_visible, tts_api_key], outputs=tts_api_key, ) voice_api_key_visible.change( toggle_api_key_visibility, inputs=[voice_api_key_visible, voice_api_key], outputs=voice_api_key, ) demo.load( endpoint_config.get_endpoint_config, inputs=None, outputs=[ tts_api_key, tts_url, voice_api_key, voice_url, ], ) return demo if __name__ == "__main__": parser = argparse.ArgumentParser(description="Qwen3-TTS Gradio Demo") parser.add_argument( "--share", action="store_true", default=False, help="Enable public sharing of the Gradio app (default: False)", ) args = parser.parse_args() demo = build_ui() demo.launch(share=args.share) ``` ### requirements.txt ```text theme={null} # Qwen3-TTS Dependencies for flexAI gradio==6.9.0 numpy==2.4.3 openai==2.26.0 soundfile==0.13.1 sox==1.5.0 ``` ### START/run-qwen3-tts-cloneVoice.sh ```bash theme={null} #!/bin/bash tmp_log=$(mktemp) export MODEL_NAME=Qwen/Qwen3-TTS-12Hz-1.7B-Base export INFERENCE_NAME=${MODEL_NAME//\//-}-$(whoami)-$(uuidgen | cut -d '-' -f 1) export INFERENCE_NAME=Qwen3-Base-$(whoami)-$(uuidgen | cut -d '-' -f 1) export TENSOR_PARALLEL_SIZE=1 export RUNTIME=vllm-omni-0.14.0 flexai inference serve $INFERENCE_NAME \ --affinity "cluster=k8s-training-smc-001" \ --runtime ${RUNTIME} \ --accels ${TENSOR_PARALLEL_SIZE} \ -- $MODEL_NAME --stage-configs-path /workspace/vllm-omni/vllm_omni/model_executor/stage_configs/qwen3_tts.yaml --omni --trust-remote-code --enforce-eager \ 2>&1 | tee "${tmp_log}" export INFERENCE_API_KEY_BASE=$(grep 'API Key' ${tmp_log} | cut -d ':' -f 2 | tr -d ' ') sleep 10 # wait for the inference to be ready STATUS="enqueued" while [ ${STATUS} != "running" ]; do STATUS=$(flexai inference inspect $INFERENCE_NAME -j | jq -r .runtime.status) echo "${STATUS}" sleep 10 done export INFERENCE_URL_BASE=$(flexai inference inspect $INFERENCE_NAME -j | jq .config.endpointUrl -r) ``` ### START/run-qwen3-tts-customVoice.sh ```bash theme={null} #!/bin/bash tmp_log=$(mktemp) # lets use a working model export MODEL_NAME=Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice export INFERENCE_NAME=Qwen3-TTS-$(whoami)-$(uuidgen | cut -d '-' -f 1) export TENSOR_PARALLEL_SIZE=1 export RUNTIME=vllm-omni-0.14.0 flexai inference serve $INFERENCE_NAME \ --affinity "cluster=k8s-training-smc-001" \ --runtime ${RUNTIME} \ --accels ${TENSOR_PARALLEL_SIZE} \ -- $MODEL_NAME --stage-configs-path /workspace/vllm-omni/vllm_omni/model_executor/stage_configs/qwen3_tts.yaml --omni --trust-remote-code --enforce-eager \ 2>&1 | tee "${tmp_log}" export INFERENCE_API_KEY_TTS=$(grep 'API Key' ${tmp_log} | cut -d ':' -f 2 | tr -d ' ') sleep 10 # wait for the inference to be ready STATUS="enqueued" while [ ${STATUS} != "running" ]; do STATUS=$(flexai inference inspect $INFERENCE_NAME -j | jq -r .runtime.status) echo "${STATUS}" sleep 10 done export INFERENCE_URL_TTS=$(flexai inference inspect $INFERENCE_NAME -j | jq .config.endpointUrl -r) ```

⚡ Deploy this on FlexAI

Any model, NVIDIA or AMD, live in under 60 seconds. No code changes, no infra management, hardware-agnostic.

Get started → Talk to us
# Generate Images with Stable Diffusion 3.5 on FlexAI Endpoints Source: https://docs.flex.ai/blueprints/text-to-image-inference Deploy Stable Diffusion 3.5 Large for AI image generation via FlexAI inference endpoints. High-quality output, auto-scaling, no code changes needed. This experiment demonstrates how to deploy and use a text-to-image model (Stable Diffusion 3.5 Large) using Flexai's inference serving capabilities. ## Prerequisites Before starting, make sure you have: * A Flexai account with access to the platform * A Hugging Face token with access to the `stabilityai/stable-diffusion-3.5-large` model * The `flexai` CLI installed and configured ## Setup FlexAI Secret for Hugging Face Token First, create a FlexAI secret that contains your Hugging Face token to access the inference model: ```bash theme={null} # Enter your HF token value when prompted flexai secret create MY_HF_TOKEN ``` Make sure your Hugging Face token has access to the `stabilityai/stable-diffusion-3.5-large` model. You may need to accept the model's license terms on Hugging Face first. ## Start the FlexAI Inference Endpoint Start the FlexAI endpoint for the Stable Diffusion 3.5 Large model: ```bash theme={null} INFERENCE_NAME=stable-diffusion-35-large flexai inference serve $INFERENCE_NAME --runtime flexserve --hf-token-secret MY_HF_TOKEN -- --task text-to-image --model stabilityai/stable-diffusion-3.5-large ``` This command will: * Create an inference endpoint named `stable-diffusion-35-large` * Use the `flexserve` runtime optimized for text-to-image tasks * Load the Stable Diffusion 3.5 Large model from Hugging Face * Configure it for text-to-image generation ## Get Endpoint Information Once the endpoint is deployed, you'll see the API key displayed in the output. Store it in an environment variable: ```bash theme={null} export INFERENCE_API_KEY= ``` Then retrieve the endpoint URL: ```bash theme={null} export INFERENCE_URL=$(flexai inference inspect $INFERENCE_NAME -j | jq .config.endpointUrl -r) ``` You'll notice these `export` lines use the `jq` tool to extract values from the JSON output of the `inspect` command. If you don't have it already, you can get `jq` from its official website: [https://jqlang.org/](https://jqlang.org/) ## Generate Images Now you can generate images by making HTTP POST requests to your endpoint. Here's an example that generates a high-quality image of a golden retriever: ```bash theme={null} curl -X POST \ -H "Authorization: Bearer $INFERENCE_API_KEY" \ -H 'Content-Type: application/json' \ -d '{ "inputs": "A highly detailed, realistic photograph of a happy golden retriever sitting on a sofa, 8k, cinematic", "parameters": { "height": 1024, "width": 1024, "num_inference_steps": 20, "seed": 42, "negative_prompt": "blurry, low quality, distorted, deformed, mutated, extra limbs, cropped, out of frame, ugly, unrealistic, cartoon, drawing, painting, watermark, text, logo, nsfw" } }' \ -o dog.png \ "$INFERENCE_URL/v1/images/generations" ``` This will save the generated image as `dog.png` in your current directory. ## Parameters Explanation The API accepts the following parameters: * **inputs**: The text prompt describing the image you want to generate * **height**: Output image height in pixels * **width**: Output image width in pixels * **num\_inference\_steps**: Number of denoising steps (higher = better quality but slower) * **seed**: Random seed for reproducible results * **negative\_prompt**: Description of what you don't want in the image (helps improve quality)

⚡ Deploy this on FlexAI

Any model, NVIDIA or AMD, live in under 60 seconds. No code changes, no infra management, hardware-agnostic.

Get started → Talk to us
# Text-to-Speech with Kokoro on FlexAI Inference Endpoints Source: https://docs.flex.ai/blueprints/text-to-speech-inference Deploy the Kokoro TTS model for natural voice synthesis via FlexAI inference endpoints. Low-latency, hardware-agnostic, production-ready deployment. This experiment demonstrates how to deploy and use a text-to-speech model (Kokoro) using Flexai's inference serving capabilities. ## Prerequisites Before starting, make sure you have: * A Flexai account with access to the platform * The `flexai` CLI installed and configured ## Start the FlexAI Inference Endpoint Start the FlexAI endpoint for the Kokoro text-to-speech model: ```bash theme={null} INFERENCE_NAME=text-to-speech flexai inference serve $INFERENCE_NAME --runtime flexserve -- --model hexgrad/Kokoro-82M --task text-to-speech --lang-code a ``` This command will: * Create an inference endpoint named `text-to-speech` * Use the `flexserve` runtime optimized for text-to-speech tasks * Load the Kokoro model for natural voice synthesis * Configure it for English language text-to-speech generation ## Get Endpoint Information Once the endpoint is deployed, you'll see the API key displayed in the output. Store it in an environment variable: ```bash theme={null} export INFERENCE_API_KEY= ``` Then retrieve the endpoint URL: ```bash theme={null} export INFERENCE_URL=$(flexai inference inspect $INFERENCE_NAME -j | jq .config.endpointUrl -r) ``` You'll notice these `export` lines use the `jq` tool to extract values from the JSON output of the `inspect` command. If you don't have it already, you can get `jq` from its official website: [https://jqlang.org/](https://jqlang.org/) ## Generate Speech Now you can generate speech by making HTTP POST requests to your endpoint. Here is an example: ```bash theme={null} curl -X POST \ -H "Authorization: Bearer $INFERENCE_API_KEY" \ -H 'Content-Type: application/json' \ -d '{ "inputs": "Welcome to FlexAI! Produce lifelike speech and spin up production inference endpoints in seconds.", "parameters": { "voice": "af_heart" } }' \ -o welcome.wav \ "$INFERENCE_URL/v1/speeches/generations" ``` ## Parameters Explanation The API accepts the following parameters: * **inputs**: The text you want to convert to speech * **voice**: The voice model to use for synthesis ## Supported Languages You can configure different languages by changing the `--lang-code` parameter when starting the inference endpoint. The voice model to use is specified in the `voice` parameter of your request. For a complete list of supported languages and available voices, check the [Kokoro GitHub repository](https://github.com/hexgrad/kokoro)

⚡ Deploy this on FlexAI

Any model, NVIDIA or AMD, live in under 60 seconds. No code changes, no infra management, hardware-agnostic.

Get started → Talk to us
# Generate Video with Wan2.2 on FlexAI Inference Endpoints Source: https://docs.flex.ai/blueprints/text-to-video-inference Deploy Wan2.2-T2V for high-quality AI video generation via FlexAI inference endpoints. Hardware-agnostic, auto-scaling, cold start under 60 seconds. This experiment demonstrates how to deploy and use a text-to-video model (Wan2.2-TI2V-5B) using Flexai's inference serving capabilities. ## About Wan2.2 Wan2.2 is developed by Alibaba. It represents a major upgrade to visual generative models, now open-sourced with more powerful capabilities, better performance, and superior visual quality making it an excellent choice for text-to-video applications. ## Prerequisites Before starting, make sure you have: * A Flexai account with access to the platform * A Hugging Face token with access to the `Wan-AI/Wan2.2-TI2V-5B-Diffusers` model * The `flexai` CLI installed and configured ## Setup FlexAI Secret for Hugging Face Token First, create a FlexAI secret that contains your Hugging Face token to access the inference model: ```bash theme={null} # Enter your HF token value when prompted flexai secret create MY_HF_TOKEN ``` Make sure your Hugging Face token has access to the `Wan-AI/Wan2.2-TI2V-5B-Diffusers` model. You may need to accept the model's license terms on Hugging Face first. ## Start the FlexAI Inference Endpoint Start the FlexAI endpoint for the Wan2.2-TI2V-5B model: ```bash theme={null} INFERENCE_NAME=wan-text-to-video flexai inference serve $INFERENCE_NAME --runtime flexserve --hf-token-secret MY_HF_TOKEN -- --task text-to-video --model Wan-AI/Wan2.2-TI2V-5B-Diffusers --quantization-config bitsandbytes_4bit ``` This command will: * Create an inference endpoint named `wan-text-to-video` * Use the `flexserve` runtime optimized for text-to-video tasks * Load the Wan2.2-TI2V-5B model from Hugging Face * Configure it for text-to-video generation with 4-bit quantization for memory efficiency ## Get Endpoint Information Once the endpoint is deployed, you'll see the API key displayed in the output. Store it in an environment variable: ```bash theme={null} export INFERENCE_API_KEY= ``` Then retrieve the endpoint URL: ```bash theme={null} export INFERENCE_URL=$(flexai inference inspect $INFERENCE_NAME -j | jq .config.endpointUrl -r) ``` You'll notice these `export` lines use the `jq` tool to extract values from the JSON output of the `inspect` command. If you don't have it already, you can get `jq` from its official website: [https://jqlang.org/](https://jqlang.org/) ## Generate Videos Now you can generate videos by making HTTP POST requests to your endpoint. Here's an example that generates a high-quality video of anthropomorphic cats boxing: ```bash theme={null} curl -X POST \ -H "Authorization: Bearer $INFERENCE_API_KEY" \ -H 'Content-Type: application/json' \ -d '{ "inputs": "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage.", "parameters": { "num_frames": 81, "height": 704, "width": 1280, "num_inference_steps": 20, "guidance_scale": 4.0, "guidance_scale_2": 3.0 } }' \ -o boxing_cats.mp4 \ "$INFERENCE_URL/v1/videos/generations" ``` This will save the generated video as `boxing_cats.mp4` in your current directory. ## Parameters Explanation The API accepts the following parameters: * **inputs**: The text prompt describing the video you want to generate * **num\_frames**: Number of frames in the output video (affects duration) * **height**: Output video height in pixels * **width**: Output video width in pixels * **num\_inference\_steps**: Number of denoising steps (higher = better quality but slower) * **guidance\_scale** and **guidance\_scale\_2**: As you increase the value, the model tries harder to match your prompt.

⚡ Deploy this on FlexAI

Any model, NVIDIA or AMD, live in under 60 seconds. No code changes, no infra management, hardware-agnostic.

Get started → Talk to us
# Fine-Tune YOLO11 Object Detection Models on FlexAI Source: https://docs.flex.ai/blueprints/ultralytics Fine-tune and deploy YOLO11 for object detection, segmentation, and pose estimation on FlexAI using Ultralytics. From training to production endpoint. This blueprint demonstrates how to train, validate, and deploy computer vision models using Ultralytics YOLO11 on FlexAI. You'll learn how to fine-tune YOLO models for object detection, instance segmentation, and pose estimation tasks, then deploy them as production-ready inference endpoints. YOLO (You Only Look Once) is one of the most popular real-time object detection frameworks, and YOLO11 brings significant improvements in accuracy and efficiency across multiple computer vision tasks. If FlexAI is not yet connected to your GitHub account, run: ```bash theme={null} flexai code-registry connect ``` This enables FlexAI to automatically pull code from repositories referenced in `--repository-url`. ## Overview This blueprint covers: * **Object Detection Training**: Fine-tune YOLO11 on custom datasets for object detection * **Instance Segmentation**: Train models to detect and segment objects at the pixel level * **Pose Estimation**: Train models for human pose detection and keypoint tracking * **Model Validation**: Evaluate model performance with comprehensive metrics * **Model Export**: Export to optimized formats (ONNX, TensorRT) for deployment * **Inference Deployment**: Deploy trained models as FlexAI inference endpoints All YOLO tasks use the same CLI interface with different task modes (`detect`, `segment`, `pose`, `track`, etc.). This guide demonstrates the core workflows that apply across all computer vision tasks. YOLO models require datasets in a specific format. We'll use the COCO8 dataset (a small subset of COCO) for this example, but you can easily adapt this to your own custom dataset. ### Option A: Use COCO8 Dataset (Quick Start) The COCO8 dataset will be automatically downloaded by Ultralytics during training. No manual download is required. ### Option B: Prepare Your Own Custom Dataset For custom datasets, follow the YOLO format structure: ``` dataset/ ├── images/ │ ├── train/ │ │ ├── image1.jpg │ │ └── image2.jpg │ └── val/ │ ├── image3.jpg │ └── image4.jpg └── labels/ ├── train/ │ ├── image1.txt │ └── image2.txt └── val/ ├── image3.txt └── image4.txt ``` Each label file contains annotations in YOLO format (one object per line): ``` ``` All coordinates must be normalized to the range \[0, 1]. Create a data configuration file (`data.yaml`): ```yaml theme={null} path: /input train: /input/images/train val: /input/images/val names: 0: person 1: bicycle 2: car # ... add your classes ``` Both mapping and list formats are supported for class names. You can also use: `names: ["person", "bicycle", "car"]` Use spaces (not tabs) for indentation in `data.yaml`. Tab characters will cause silent parsing failures. YOLO accepts both relative and absolute paths, but using absolute paths (`/input/...`) reduces ambiguity inside FlexAI jobs. If your dataset uses a different annotation format (COCO JSON, Pascal VOC, etc.), convert it to YOLO format before uploading. Refer to the [Ultralytics Data Format documentation](https://docs.ultralytics.com/datasets/detect/) for conversion guidance. ### Upload Custom Dataset to FlexAI Once your dataset is prepared, upload it to FlexAI: ```bash theme={null} flexai dataset push yolo-custom-dataset --file path/to/your-dataset ``` When you use the dataset in a training job with `--dataset yolo-custom-dataset`, FlexAI will mount the dataset contents directly at `/input/` in your training environment. **All dataset contents are mounted under `/input`, preserving their original folder structure.** This means: * If your dataset structure is `dataset/images/train/...`, it will be accessible at `/input/images/train/...` * Your `data.yaml` should be at `/input/data.yaml` * The `path` field in your `data.yaml` should be `/input` as shown in the example above Ensure these paths are reflected in your `data.yaml` before starting a training job. Train a YOLO11 model for object detection. We'll start with the nano model (YOLO11n) which is fast and efficient. ### Training on COCO8 Dataset ```bash theme={null} flexai training run yolo11-detection-coco8 \ --accels 4 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --requirements-path code/ultralytics/requirements.txt \ -- yolo detect train \ data=coco8.yaml \ model=yolo11n.pt \ epochs=100 \ imgsz=640 \ batch=16 \ device=0,1,2,3 \ project=/output-checkpoint \ name=yolo11n-coco8 \ patience=50 \ save=True \ val=True ``` `--accels` specifies the number of GPUs to allocate (e.g., `--accels 4` = 4 GPUs of the chosen accelerator type). ### Training on Custom Dataset ```bash theme={null} flexai training run yolo11-detection-custom \ --accels 4 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --dataset yolo-custom-dataset \ --requirements-path code/ultralytics/requirements.txt \ -- yolo detect train \ data=/input/data.yaml \ model=yolo11n.pt \ epochs=100 \ imgsz=640 \ batch=16 \ device=0,1,2,3 \ project=/output-checkpoint \ name=yolo11n-custom \ save=True ``` ### Training with Larger Models For better accuracy, use larger YOLO variants. Adjust batch size based on model size: **YOLO11s (Small)**: ```bash theme={null} flexai training run yolo11s-detection \ --accels 4 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --requirements-path code/ultralytics/requirements.txt \ -- yolo detect train \ data=coco8.yaml \ model=yolo11s.pt \ epochs=100 \ imgsz=640 \ batch=16 \ device=0,1,2,3 \ project=/output-checkpoint ``` **YOLO11m (Medium)**: ```bash theme={null} flexai training run yolo11m-detection \ --accels 4 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --requirements-path code/ultralytics/requirements.txt \ -- yolo detect train \ data=coco8.yaml \ model=yolo11m.pt \ epochs=100 \ imgsz=640 \ batch=8 \ device=0,1,2,3 \ project=/output-checkpoint ``` **YOLO11l (Large)**: ```bash theme={null} flexai training run yolo11l-detection \ --accels 8 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --requirements-path code/ultralytics/requirements.txt \ -- yolo detect train \ data=coco8.yaml \ model=yolo11l.pt \ epochs=100 \ imgsz=640 \ batch=4 \ device=0,1,2,3,4,5,6,7 \ project=/output-checkpoint ``` Instance segmentation detects objects and generates pixel-level masks for each instance. ```bash theme={null} flexai training run yolo11-segmentation \ --accels 4 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --requirements-path code/ultralytics/requirements.txt \ -- yolo segment train \ data=coco8-seg.yaml \ model=yolo11n-seg.pt \ epochs=100 \ imgsz=640 \ batch=16 \ device=0,1,2,3 \ project=/output-checkpoint \ name=yolo11n-segmentation \ save=True ``` For custom segmentation datasets, ensure your labels include polygon annotations in YOLO segmentation format. For human pose estimation tasks with keypoint detection: ```bash theme={null} flexai training run yolo11-pose \ --accels 4 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --requirements-path code/ultralytics/requirements.txt \ -- yolo pose train \ data=coco8-pose.yaml \ model=yolo11n-pose.pt \ epochs=100 \ imgsz=640 \ batch=16 \ device=0,1,2,3 \ project=/output-checkpoint ``` After training, validate your model's performance on the validation dataset. ### List Training Checkpoints ```bash theme={null} flexai training checkpoints yolo11-detection-coco8 ``` ### Run Validation For validation with the COCO8 dataset (will be auto-downloaded): ```bash theme={null} flexai training run yolo11-validation \ --accels 1 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --checkpoint \ --requirements-path code/ultralytics/requirements.txt \ -- yolo detect val \ model=/checkpoint/weights/best.pt \ data=coco8.yaml \ imgsz=640 \ batch=16 \ project=/output-checkpoint/validation ``` For validation with a custom dataset: ```bash theme={null} flexai training run yolo11-validation-custom \ --accels 1 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --checkpoint \ --dataset yolo-custom-dataset \ --requirements-path code/ultralytics/requirements.txt \ -- yolo detect val \ model=/checkpoint/weights/best.pt \ data=/input/data.yaml \ imgsz=640 \ batch=16 \ project=/output-checkpoint/validation ``` ### Understanding Validation Metrics YOLO provides comprehensive metrics: * **mAP50**: Mean Average Precision at IoU threshold 0.5 * **mAP50-95**: Mean Average Precision averaged across IoU thresholds 0.5-0.95 * **Precision**: Ratio of true positive detections * **Recall**: Ratio of detected ground truth objects * **F1-Score**: Harmonic mean of precision and recall These metrics are saved to the results directory after each training run. Export your trained model to various formats for optimized deployment. ### Download Checkpoint Locally First, download the best checkpoint to your local machine: ```bash theme={null} flexai checkpoint fetch "" --destination ./yolo11-checkpoint ``` ### Export to ONNX ONNX format is widely supported and optimized for cross-platform inference: ```bash theme={null} flexai training run yolo11-export-onnx \ --accels 1 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --checkpoint \ --requirements-path code/ultralytics/requirements.txt \ -- yolo export \ model=/checkpoint/weights/best.pt \ format=onnx \ imgsz=640 \ simplify=True \ dynamic=False ``` ### Export to TensorRT For NVIDIA GPU deployment with maximum performance: ```bash theme={null} flexai training run yolo11-export-tensorrt \ --accels 1 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --checkpoint \ --requirements-path code/ultralytics/requirements.txt \ -- yolo export \ model=/checkpoint/weights/best.pt \ format=engine \ imgsz=640 \ half=True \ device=0 ``` ### Available Export Formats | Format | format Argument | Use Case | | - | - | - | | PyTorch | `torchscript` | General PyTorch deployment | | ONNX | `onnx` | Cross-platform inference | | TensorRT | `engine` | NVIDIA GPU optimization | | CoreML | `coreml` | Apple devices (iOS/macOS) | | TFLite | `tflite` | Mobile and embedded devices | | OpenVINO | `openvino` | Intel hardware acceleration | | NCNN | `ncnn` | Mobile deployment | See the [Ultralytics Export documentation](https://docs.ultralytics.com/modes/export/) for more formats and options. After training, you can run inference on images or videos using your trained model. ### Run Inference as a Training Job FlexAI's managed inference endpoints currently support vLLM only. For YOLO models, use a training job to execute predictions directly: ```bash theme={null} flexai training run yolo11-predict \ --accels 1 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --checkpoint \ --requirements-path code/ultralytics/requirements.txt \ -- yolo detect predict \ model=/checkpoint/weights/best.pt \ source=https://ultralytics.com/images/bus.jpg \ conf=0.25 \ iou=0.45 \ save=True \ project=/output-checkpoint/predictions ``` The predictions and annotated images will be saved in the job's output directory. ### Run Batch Predictions To run inference on a directory of images, first upload them as a dataset: ```bash theme={null} flexai dataset push test-images --file path/to/images/ ``` Then run prediction: ```bash theme={null} flexai training run yolo11-predict-batch \ --accels 1 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --checkpoint \ --dataset test-images \ --requirements-path code/ultralytics/requirements.txt \ -- yolo detect predict \ model=/checkpoint/weights/best.pt \ source=/input \ conf=0.25 \ save=True \ project=/output-checkpoint/predictions ``` ### Download Prediction Results After the prediction job completes, download the results: ```bash theme={null} flexai training checkpoints yolo11-predict flexai checkpoint fetch "" --destination ./predictions ``` ### Quick Local Testing Test your trained model locally before deploying to production. You can test your model locally after downloading the checkpoint: ```bash theme={null} # Download the checkpoint flexai checkpoint fetch "" --destination ./yolo11-checkpoint # Run inference locally yolo detect predict \ model=./yolo11-checkpoint/weights/best.pt \ source=path/to/image.jpg \ conf=0.25 \ save=True ``` Results will be saved to `runs/detect/predict/`. ## Monitoring Training Progress ### Check Training Status ```bash theme={null} flexai training inspect yolo11-detection-coco8 ``` ### View Training Logs ```bash theme={null} flexai training logs yolo11-detection-coco8 ``` ### Training Observability with TensorBoard Ultralytics automatically logs training metrics. Access FlexAI's hosted [TensorBoard](https://tensorboard.flex.ai/) instance to track: * Training and validation loss curves * mAP (mean Average Precision) metrics * Precision and recall curves * Learning rate schedules ### Weights & Biases Integration For advanced monitoring, integrate with Weights & Biases by adding environment variables: ```bash theme={null} flexai training run yolo11-detection-wandb \ --accels 4 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --secret WANDB_API_KEY= \ --env WANDB_PROJECT=yolo11-experiments \ --requirements-path code/ultralytics/requirements.txt \ -- yolo detect train \ data=coco8.yaml \ model=yolo11n.pt \ epochs=100 \ project=/output-checkpoint ``` ## Advanced Use Cases ### Object Tracking YOLO11 supports multi-object tracking in videos. First upload your video as a dataset: ```bash theme={null} flexai dataset push test-video --file path/to/video.mp4 ``` Then run tracking: ```bash theme={null} flexai training run yolo11-tracking \ --accels 1 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --checkpoint \ --dataset test-video \ --requirements-path code/ultralytics/requirements.txt \ -- yolo track \ model=/checkpoint/weights/best.pt \ source=/input/video.mp4 \ conf=0.25 \ iou=0.45 \ save=True \ project=/output-checkpoint/tracking ``` ### Model Benchmarking Compare model performance across different formats and hardware: ```bash theme={null} flexai training run yolo11-benchmark \ --accels 1 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --checkpoint \ --requirements-path code/ultralytics/requirements.txt \ -- yolo benchmark \ model=/checkpoint/weights/best.pt \ data=coco8.yaml \ imgsz=640 ``` This will benchmark: * PyTorch inference speed * ONNX inference speed * TensorRT inference speed (if available) * Model accuracy metrics ### Hyperparameter Tuning Use Ultralytics' built-in hyperparameter tuning: ```bash theme={null} flexai training run yolo11-tune \ --accels 4 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --requirements-path code/ultralytics/requirements.txt \ -- yolo detect tune \ data=coco8.yaml \ model=yolo11n.pt \ epochs=30 \ iterations=300 \ optimizer=AdamW ``` ## Expected Results ### Detection Performance (YOLO11n on COCO8) * **mAP50**: typically 0.45–0.55 * **mAP50-95**: typically 0.30–0.40 * **Inference Speed**: 1.5–2.5 ms/image (H100 TensorRT) * **Model Size**: \~3 MB ### Segmentation Performance (YOLO11n-seg on COCO8-seg) * **Box mAP50**: typically 0.45–0.55 * **Mask mAP50**: typically 0.40–0.50 * **Inference Speed**: 2–3 ms/image (H100 TensorRT) ### Training Time * **YOLO11n on COCO8**: \~5–10 minutes (4 × H100, 100 epochs). * **YOLO11s on COCO8**: \~10–15 minutes (4 × H100, 100 epochs). * **YOLO11m on full COCO**: \~4–6 hours (4 × H100, 100 epochs). ## Technical Details ### Recommended Resource Configuration | Model | GPUs | Batch Size | Memory | Training Time (100 epochs, COCO8) | | - | - | - | - | - | | YOLO11n | 1-4 × H100 | 16-32 | 8GB+ | 5-10 min | | YOLO11s | 2-4 × H100 | 8-16 | 12GB+ | 10-15 min | | YOLO11m | 4 × H100 | 4-8 | 16GB+ | 15-20 min | | YOLO11l | 4-8 × H100 | 2-4 | 24GB+ | 20-30 min | ### Key Training Parameters **Image Size (`imgsz`)**: * Standard: 640×640 * Small objects: 1280×1280 (slower but better detection) * Real-time applications: 320×320 or 416×416 (faster inference) **Batch Size (`batch`)**: * Larger batches generally lead to better convergence * Adjust based on GPU memory: YOLO11n (16-32), YOLO11s (8-16), YOLO11m (4-8) **Epochs**: * Small datasets (\< 1000 images): 100-150 epochs * Medium datasets (1000-10000): 50-100 epochs * Large datasets (> 10000): 30-50 epochs **Early Stopping (`patience`)**: * Stops training if no improvement for N epochs * Recommended: 50 epochs for COCO8, 30 epochs for larger datasets **Data Augmentation**: * Enabled by default with optimized settings * Includes mosaic, mixup, HSV augmentation, and geometric transforms ### Multi-GPU Training FlexAI automatically enables distributed training when multiple GPUs are requested: ```bash theme={null} flexai training run yolo11-multi-gpu \ --accels 8 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --requirements-path code/ultralytics/requirements.txt \ -- yolo detect train \ data=coco8.yaml \ model=yolo11n.pt \ epochs=100 \ batch=32 \ device=0,1,2,3,4,5,6,7 \ project=/output-checkpoint ``` ### Transfer Learning Continue training from a previous checkpoint: ```bash theme={null} flexai training run yolo11-transfer-learning \ --accels 4 --nodes 1 \ --repository-url https://github.com/flexaihq/blueprints \ --checkpoint \ --requirements-path code/ultralytics/requirements.txt \ -- yolo detect train \ data=coco8.yaml \ model=/checkpoint/weights/best.pt \ epochs=50 \ batch=16 \ project=/output-checkpoint ``` ## Troubleshooting ### Common Issues **Fixing Training Job Failures:** ```bash theme={null} # Check FlexAI authentication flexai auth status # Verify dataset upload flexai dataset list ``` **Fixing Out-of-Memory Errors:** * Reduce batch size: `batch=8` or `batch=4` * Use smaller image size: `imgsz=416` * Try a smaller model variant: YOLO11n instead of YOLO11s/m/l **Fixing Low mAP / Poor Performance:** * Train for more epochs: `epochs=200` or `epochs=300` * Increase image size: `imgsz=1280` * Use a larger model: YOLO11s/m instead of YOLO11n * Check dataset quality and annotation accuracy * Ensure balanced class distribution **Fixing Dataset Format Errors:** * Verify YOLO format: ` ` * Check that coordinates are normalized (0-1 range) * Ensure `data.yaml` paths are correct * Verify image-label pairs match (same filename, different extension) **Fixing Export Failures:** * Update Ultralytics: Add `ultralytics>=8.3.0` to requirements * Check CUDA/TensorRT compatibility * Try without optimization: `simplify=False` for ONNX * Ensure model path is correct **Fixing Prediction Job Failures:** * Verify checkpoint ID is correct * Check that model path `/checkpoint/weights/best.pt` exists * Ensure source path is correct (use `/input` for datasets) * Check logs: `flexai training logs ` ## Dataset Format Requirements If you have a dataset in another format (COCO JSON, Pascal VOC, etc.), you'll need to convert it to YOLO format before using it with this blueprint. ### YOLO Format Specification Each image should have a corresponding text file with the same name: ``` ``` Where: * `class_id`: Integer class ID (0-indexed) * `x_center`, `y_center`, `width`, `height`: Normalized coordinates (0-1 range) Example label file (`image001.txt`): ``` 0 0.5 0.5 0.3 0.4 1 0.2 0.3 0.1 0.15 ``` ### Converting Other Formats For converting from other formats: * **COCO JSON**: See [Ultralytics COCO format documentation](https://docs.ultralytics.com/datasets/detect/coco/) * **Pascal VOC**: See [Ultralytics VOC format documentation](https://docs.ultralytics.com/datasets/detect/voc/) * **Other formats**: Refer to [Ultralytics Data Formats guide](https://docs.ultralytics.com/datasets/detect/) After converting your dataset to YOLO format, ensure your `data.yaml` has `path: /input` before uploading to FlexAI. ## Best Practices ### Dataset Preparation * Use high-quality, diverse images * Ensure balanced class distribution * Include various lighting conditions, angles, and backgrounds * Minimum 1500 images per class recommended * Validate annotation accuracy before training ### Training * Start with pre-trained weights (transfer learning) * Use data augmentation (enabled by default) * Monitor validation metrics, not just training loss * Use early stopping to prevent overfitting * Save checkpoints regularly ### Deployment * Export to optimized formats (ONNX/TensorRT) for production * Test on representative images before deployment * Set appropriate confidence thresholds (0.25-0.5 typical range) * Benchmark inference speed on target hardware ## References * [Ultralytics YOLO Documentation](https://docs.ultralytics.com/) * [Ultralytics CLI Guide](https://docs.ultralytics.com/usage/cli/) * [YOLO11 Model Information](https://docs.ultralytics.com/models/yolo11/) * [FlexAI Documentation](https://docs.flex.ai/) ## Code ### requirements.txt ```text theme={null} git+https://github.com/flexaihq/ultralytics.git@b67379789fd0ec84bd0399f99f5aef7c3075916a ```

🚀 Run this on FlexAI

Managed checkpoints mean you never lose a run to preemption. Jobs launch in under 60 seconds — no infra setup, built-in observability.

Get started → Talk to us
# Fine-Tune Vision-Language Models with GRPO on FlexAI Source: https://docs.flex.ai/blueprints/vlm-grpo Train vision-language models using GRPO reinforcement learning with TRL, LoRA, vLLM, and DeepSpeed ZeRO-3 on FlexAI. Full tutorial with working code. This experiment demonstrates how to perform Group Relative Policy Optimization (GRPO) training on a Vision-Language Model (VLM) using the TRL library. GRPO is a reinforcement learning technique that optimizes the model based on relative rewards within groups of generated responses. We will use `Qwen2.5-VL-3B-Instruct` as our base VLM and apply LoRA (Low-Rank Adaptation) for efficient fine-tuning. The training leverages vLLM for fast inference during the policy optimization phase and DeepSpeed ZeRO-3 for distributed training. ## Prerequisites ### Connect to GitHub (if needed) If you haven't already connected FlexAI to GitHub, you'll need to set up a code registry connection: ```bash theme={null} flexai code-registry connect ``` This will allow FlexAI to pull repositories directly from GitHub using the `-u` flag in training commands. ### Create Secrets To access the Qwen2.5-VL-3B-Instruct model, you may need authentication with your HuggingFace account depending on the model's access requirements. Use the `flexai secret create` command to store your *HuggingFace Token* as a secret. Replace `` with your desired name for the secret: ```bash theme={null} flexai secret create ``` Then paste your *HuggingFace Token* value. ## Training To start the GRPO Training Job, run the following command: ```bash theme={null} flexai training run vlm-grpo-training --repository-url https://github.com/flexaihq/blueprints \ --requirements-path code/vlm-grpo/requirements.txt \ --secret HF_TOKEN= \ --runtime pytorch-28-vllm-0110-nvidia \ --nodes 1 --accels 2 \ -- accelerate launch \ --config_file=code/vlm-grpo/deepspeed_zero3.yaml \ code/vlm-grpo/grpo_vlm.py \ --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \ --output_dir /output-checkpoint/grpo-Qwen2.5-VL-3B-Instruct \ --learning_rate 1e-5 \ --gradient_checkpointing \ --dtype bfloat16 \ --max_prompt_length 2048 \ --max_completion_length 1024 \ --use_vllm \ --vllm_mode colocate \ --use_peft \ --lora_target_modules "q_proj" "v_proj" \ --log_completions \ --logging_steps 200 ``` ### Key Arguments Explained * `--model_name_or_path`: The base VLM model to fine-tune (Qwen2.5-VL-3B-Instruct) * `--output_dir`: Directory where checkpoints will be saved * `--learning_rate`: Learning rate for optimization (1e-5) * `--gradient_checkpointing`: Enables gradient checkpointing to reduce memory usage * `--dtype bfloat16`: Uses bfloat16 precision for training * `--max_prompt_length`: Maximum length of input prompts (2048 tokens) * `--max_completion_length`: Maximum length of generated completions (1024 tokens) * `--use_vllm`: Enables vLLM for fast inference during policy optimization * `--vllm_mode colocate`: Runs vLLM on the same GPUs as training * `--use_peft`: Enables LoRA for parameter-efficient fine-tuning * `--lora_target_modules`: Specifies which modules to apply LoRA (q\_proj, v\_proj) * `--log_completions`: Logs generated completions during training ### Configuration Files The experiment uses a DeepSpeed ZeRO-3 configuration file (`code/vlm-grpo/deepspeed_zero3.yaml`) that specifies: * ZeRO Stage 3 for memory-efficient distributed training * 2 processes (GPUs) for multi-GPU training * Mixed precision training with bfloat16 ## Monitoring the Training Job You can check the status and life cycle events of your Training Job by running: ```bash theme={null} flexai training inspect vlm-grpo-training ``` Additionally, you can view the logs of your Training Job by running: ```bash theme={null} flexai training logs vlm-grpo-training ``` ## Fetching the Trained Model Once the Training Job completes successfully, you can list all the produced checkpoints: ```bash theme={null} flexai training checkpoints vlm-grpo-training ``` Download a checkpoint with: ```bash theme={null} flexai checkpoint fetch "" ``` The checkpoint will contain the LoRA adapters that can be merged with the base model for inference. ## Additional Resources * [TRL Documentation](https://huggingface.co/docs/trl/index) * [GRPO Paper](https://arxiv.org/abs/2402.03300) * [Qwen2-VL Model Card](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) * [vLLM Documentation](https://docs.vllm.ai/) ## Code ### grpo\_vlm.py ```python theme={null} # Adapted from: https://github.com/huggingface/trl/blob/e622196097109080b73584d598d4162e64fc6bea/examples/scripts/grpo_vlm.py # Copyright 2020-2025 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # /// script # dependencies = [ # "trl", # "Pillow", # "peft", # "math-verify", # "latex2sympy2_extended", # "torchvision", # "trackio", # "kernels", # ] # /// """ pip install math_verify # For Qwen/Qwen2.5-VL-3B-Instruct accelerate launch \ --config_file examples/accelerate_configs/deepspeed_zero3.yaml \ examples/scripts/grpo_vlm.py \ --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \ --output_dir grpo-Qwen2.5-VL-3B-Instruct \ --learning_rate 1e-5 \ --gradient_checkpointing \ --dtype bfloat16 \ --max_prompt_length 2048 \ --max_completion_length 1024 \ --use_vllm \ --vllm_mode colocate \ --use_peft \ --lora_target_modules "q_proj", "v_proj" \ --log_completions # For HuggingFaceTB/SmolVLM2-2.2B-Instruct pip install num2words==0.5.14 accelerate launch \ --config_file examples/accelerate_configs/deepspeed_zero3.yaml \ examples/scripts/grpo_vlm.py \ --model_name_or_path HuggingFaceTB/SmolVLM2-2.2B-Instruct \ --output_dir grpo-SmolVLM2-2.2B-Instruct \ --learning_rate 1e-5 \ --dtype bfloat16 \ --max_prompt_length 2048 \ --max_completion_length 1024 \ --use_peft \ --lora_target_modules "q_proj", "v_proj" \ --log_completions \ --per_device_train_batch_size 1 \ --gradient_accumulation_steps 2 \ --num_generations 2 """ import os import torch from datasets import load_dataset from trl import ( GRPOConfig, GRPOTrainer, ModelConfig, ScriptArguments, TrlParser, get_kbit_device_map, get_peft_config, get_quantization_config, ) from trl.rewards import accuracy_reward, think_format_reward # Enable logging in a Hugging Face Space os.environ.setdefault("TRACKIO_SPACE_ID", "trl-trackio") if __name__ == "__main__": parser = TrlParser((ScriptArguments, GRPOConfig, ModelConfig)) script_args, training_args, model_args = parser.parse_args_and_config() ################ # Model ################ dtype = ( model_args.dtype if model_args.dtype in ["auto", None] else getattr(torch, model_args.dtype) ) training_args.model_init_kwargs = dict( revision=model_args.model_revision, attn_implementation=model_args.attn_implementation, dtype=dtype, ) quantization_config = get_quantization_config(model_args) if quantization_config is not None: # Passing None would not be treated the same as omitting the argument, so we include it only when valid. training_args.model_init_kwargs["device_map"] = get_kbit_device_map() training_args.model_init_kwargs["quantization_config"] = quantization_config ################ # Dataset ################ dataset = load_dataset("lmms-lab/multimodal-open-r1-8k-verified", split="train") dataset = dataset.train_test_split(test_size=100, seed=42) SYSTEM_PROMPT = ( "A conversation between user and assistant. The user asks a question, and the assistant solves it. The " "assistant first thinks about the reasoning process in the mind and then provides the user with the answer. " "The reasoning process and answer are enclosed within tags, i.e., \nThis is my " "reasoning.\n\nThis is my answer." ) def make_conversation(example): prompt = [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": example["problem"]}, ] return {"prompt": prompt} dataset = dataset.map(make_conversation) # Filter have big images def filter_big_images(example): image = example["image"] return image.size[0] < 512 and image.size[1] < 512 dataset = dataset.filter(filter_big_images) def convert_to_rgb(example): image = example["image"] if image.mode != "RGB": image = image.convert("RGB") example["image"] = image return example dataset = dataset.map(convert_to_rgb) train_dataset = dataset["train"] eval_dataset = dataset["test"] if training_args.eval_strategy != "no" else None ################ # Training ################ trainer = GRPOTrainer( model=model_args.model_name_or_path, args=training_args, reward_funcs=[think_format_reward, accuracy_reward], train_dataset=train_dataset, eval_dataset=eval_dataset, peft_config=get_peft_config(model_args), ) trainer.train() # Save and push to hub trainer.save_model(training_args.output_dir) if training_args.push_to_hub: trainer.push_to_hub(dataset_name=script_args.dataset_name) ``` ### requirements.txt ```text theme={null} deepspeed==0.18.2 math_verify==0.8.0 peft==0.18.0 trl==0.25.1 ``` ### deepspeed\_zero3.yaml ```yaml theme={null} --- compute_environment: LOCAL_MACHINE debug: false deepspeed_config: deepspeed_multinode_launcher: standard offload_optimizer_device: none offload_param_device: none zero3_init_flag: true zero3_save_16bit_model: true zero_stage: 3 distributed_type: DEEPSPEED downcast_bf16: no machine_rank: 0 main_training_function: main mixed_precision: bf16 num_machines: 1 num_processes: 2 rdzv_backend: static same_network: true tpu_env: [] tpu_use_cluster: false tpu_use_sudo: false use_cpu: false ```

🚀 Run this on FlexAI

Managed checkpoints mean you never lose a run to preemption. Jobs launch in under 60 seconds — no infra setup, built-in observability.

Get started → Talk to us
# Changelog Source: https://docs.flex.ai/changelog Model availability, pricing, and API changes for the FlexAI Token Factory. Inference # Video results are single-download, and expire after an hour * **A clip is deleted when you download it.** The first successful retrieval returns the video, then removes the clip and the prompt that made it. A second request returns `410 job_already_retrieved`, so save the bytes on the first download rather than re-fetching. * **An uncollected clip expires after one hour**, not 24. After that the job returns `410 job_expired`. Detail, including where this content lives for EU workspaces, is on the [Regions](/inference-api/reference/regions) page. Inference # Usage lookup by request ID works again Looking up a request's usage by the `id` returned in the response briefly returned `404`. It resolves again, and charges were always correct — nothing was billed twice or went unbilled. Requests made while it was broken stay unqueryable by id; their usage is still counted in your billing totals. Inference # reasoning\_effort now works on hybrid models, and tool calling on the Responses API * **`reasoning_effort` is honoured on eight models that used to drop it.** These models have a reasoning on/off switch but no depth dial, so a request carrying `reasoning_effort` — the parameter the OpenAI SDKs, coding agents and OpenRouter all send — produced no reasoning at all. An effort of `low`, `medium`, `high`, `xhigh` or `max` now turns reasoning on, and the response says so with a new header — `x-flexai-adjusted-params: reasoning_effort=on`, or `reasoning=on` if you sent it as OpenRouter's `reasoning.effort` or through `/v1/responses` — because the depth itself cannot be adjusted. `none` and `minimal` leave it off, which is what you asked for. Turning reasoning off explicitly still wins over a depth: send `thinking: {"type": "disabled"}`, or `enable_thinking: false` inside `chat_template_kwargs`, and the effort you set aside is reported in `x-flexai-dropped-params`. The models are `GLM-4.5-Air-FP8`, `NVIDIA-Nemotron-3.5-Lightning-30B-A3B`, `Qwen3-8B-FP8`, `Qwen3.5-9B`, `Qwen3.6-27B-FP8`, `Qwen3.6-35B-A3B-FP8`, `gemma-4-26B-A4B-it` and `gemma-4-31b-it`. * **Tool calls from `GLM-4.5-Air-FP8` are no longer buried in the reasoning output.** On a non-streaming `/v1/responses` call with thinking off, the whole reply — tool call included — could come back as a single reasoning item, so an agent saw no call to make. When the reply is nothing but that mislabelled call, it now returns the leading text as a message item followed by one `function_call` item per call. Anything else is left exactly as the engine sent it, so a forced `tool_choice` or a reply cut short at `max_output_tokens` is not repaired. Chat completions were already correct; the streaming Responses path got a separate fix in this release, for forced tool choice. * **When we override `parallel_tool_calls`, we now tell you.** Sending `tool_choice: "required"` to `GLM-5.2`, `MiniMax-M2.7` or `NVIDIA-Nemotron-3.5-Lightning-30B-A3B` forces `parallel_tool_calls` to `false`. If you sent `true`, that used to change silently; the response now carries `x-flexai-dropped-params: parallel_tool_calls`. Sending `false`, or omitting it, is not an override and adds no header. * **An unrecognised `/v1/responses` input item type is rejected with the index that caused it.** Previously it reached the engine and came back as a bare "Invalid request." with nothing naming the offender. You now get `input[i].type` pointing at the item. Types the Responses API defines — `reasoning`, `item_reference`, hosted-tool calls — pass through unchanged. * **Recommended sampling defaults, where a model has them.** A model can now carry a default for the sampling fields you leave unset. The first is `DeepSeek-V4-Flash-0731`, which may fill `top_p` 0.95 — the value its model card recommends for agentic use — when you do not send one. Any value you send always wins, **including `temperature: 0`**. When we fill a field, the response names it: `x-flexai-sampling-defaulted: top_p`. Platform # Billing emails follow your top-ups, and EU accounts bill in euros * **If you pay by top-up, low-balance emails now follow your last top-up rather than the calendar month.** Each top-up arms a fresh pair of alerts: one "running low" when the balance falls below a fifth of what it was right after that credit, and one "out of credit" at \$0. Your next top-up re-arms both. The old behaviour was a percentage of the month's spend, which moved every time you topped up, reset on the 1st, and fired every month for accounts on auto-recharge that had nothing wrong. * **With auto-recharge on, you get only the "out of credit" email.** The running-low warning is skipped, because the rule exists to top you up before it would fire. If you do reach \$0 with auto-recharge enabled, that email means a recharge did not go through — usually a declined card or a cap being reached — and is worth acting on. * **Postpaid accounts now get budget alerts at 80% and 100% of their cap**, where there used to also be one at 50%. * **The "card added" confirmation email is gone.** Adding a card is visible in the dashboard immediately. * **Accounts in the EU region are charged and invoiced in euros.** Prices, usage and your wallet balance stay in US dollars; a euro top-up is converted once, when the card is charged, at a reference rate fixed for the calendar month and recorded against that payment. Nothing is re-converted afterwards, so credit you have already bought does not move with the exchange rate. Full detail is on the Regions page in the Inference API reference. Inference # New model: Qwen3.8 Flash Next **[Qwen3.8 Flash Next](https://flex.ai/models/qwen3-8-flash-next)** has been serving on the Inference API since 28 September — \$0.15 per million input tokens, \$0.47 per million output, with a 262K-token context window. Call it by its id through the usual endpoint: ```bash theme={null} curl https://api.flex.ai/v1/chat/completions \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "Qwen3.8-Flash-Next", "messages": [{"role": "user", "content": "Hello"}]}' ``` Inference # Token Factory changelog Changes to the Token Factory are now published here — models arriving and leaving, pricing changes, and changes to the API. For what is serving right now, `GET /v1/models` remains the authoritative list and carries current per-token pricing for every model on it. Inference # New models: GLM 5.3 Flash and Qwen3.8 27B Both are serving now on the Inference API, and both read images as well as text. * **[GLM 5.3 Flash](https://flex.ai/models/glm-5-3-flash)** — \$0.075 per million input tokens, \$0.25 per million output, with a **1M-token context window**. * **[Qwen3.8 27B](https://flex.ai/models/qwen3-8-27b)** — \$0.42 per million input tokens, \$3.00 per million output, with a 262K-token context window. Call either by its id through the usual endpoint: ```bash theme={null} curl https://api.flex.ai/v1/chat/completions \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "GLM-5.3-Flash", "messages": [{"role": "user", "content": "Hello"}]}' ``` # FlexAI CLI Source: https://docs.flex.ai/cli Command-line interface for FlexAI - manage your AI workloads from the terminal The FlexAI Command Line Interface (CLI) allows you to manage your AI workloads directly from your terminal. Build, train, and deploy models with simple commands. ## Getting Started ### Installation & Authentication * [CLI Installation](/cli/install/) - Install the FlexAI CLI * [CLI Authentication](/cli/authenticate/) - Authenticate with your FlexAI account * [CLI Autocompletion](/cli/reference/completion/) - Enable shell *autocompletion* for the CLI ## Core Services ### Model Inference Deploy and manage Inference Endpoints for your models. * [Inference Commands](/cli/reference/inference/) - All inference-related commands * Quickstart guide: [Deploying a Public Model](/core-services/inference/quickstart/create-public/) * Quickstart guide: [Deploying a Private Model](/core-services/inference/quickstart/create-private/) ### Model Training Train and Fine-tune models. * [Training Commands](/cli/reference/training/) - All training-related commands * Quickstart guide: [Creating a Training Job](/core-services/training/quickstart/creating/) ### Platform Services Manage checkpoints, secrets, and storage. * [Checkpoint Manager](/platform-services/checkpoint-manager/) - Manage model checkpoints * [Code Registry Manager](/platform-services/code-registry-manager/) - Manage code repositories * [Dataset Manager](/platform-services/dataset-manager/) - Manage datasets * Guide: [Uploading Datasets](/platform-services/dataset-manager/) * [Observability Services](/platform-services/observability/) - Monitor logs and metrics for your workloads * [Remote Storage Connection Manager](/platform-services/remote-storage-connections-manager/) - Manage remote storage * [Secret Manager](/platform-services/secret-manager/) - Manage secrets and credentials ## Updates & Changes * [Changelog](/cli/changelog/) - Latest CLI updates and changes # FlexAI CLI Login Source: https://docs.flex.ai/cli/authenticate Authenticate the FlexAI CLI using your GitHub account After [installing the FlexAI CLI](/cli/install/) you'll need to authenticate using the GitHub account you've registered with FlexAI. You can start the authentication process by running: ```bash theme={null} flexai auth login ``` Depending on your system, your default browser will automatically open the "*Sign in*" page, or you will be prompted to visit the "*Sign in*" URL: ```text theme={null} Please visit the following URL in your browser to authenticate: `https://api.flex.ai:443/api/v1/auth/login?redirect_endpoint=%2Fcallback&redirect_host=localhost%3A34283&redirect_scheme=http` ``` Depending on your system, you can either use `Ctrl+Left Click` to open the link or copy and paste it directly into your browser to sign in with your GitHub account. If you are running the FlexAI CLI within a headless environment (such as a remote SSH session), you can use the [method described here](/best-practices/alternative-methods/logging-from-a-remote-env/) to authenticate. The browser will prompt you to sign in with your GitHub account (see image below). After signing in, you will be redirected to the FlexAI website. Feel free to close the browser window and return to your terminal. Screenshot of the 'Sign in with GitHub' page You can now go back to your terminal, where you will have a `You are logged in!` confirmation message. > If you are unable to authenticate, please reach out to our support team through the dedicated channels we've setup for your team. If you haven't already, we recommend following the [Training Quickstart Tutorial](/core-services/training/quickstart/), which will guide you through an example that shows you how to train a model (NanoGPT in this case) to learn more about the main functionalities FlexAI has to offer. # Changelog: 2024-02-21 Source: https://docs.flex.ai/cli/changelog/2024-02-21 *Initial release of the Flex CLI.* ## Added * Installation script. * `update` command. # Changelog: 2024-02-22 Source: https://docs.flex.ai/cli/changelog/2024-02-22 ## Added * Installation script improvements. # Changelog: 2024-03-07 Source: https://docs.flex.ai/cli/changelog/2024-03-07 ## Added * Added a doctor command to check system readiness: `flexai doctor`. * Added `torchrun` support. * Added an `--arch` flag. ## Changed * Renamed the cli to `flexai` # Changelog: 2024-03-08 Source: https://docs.flex.ai/cli/changelog/2024-03-08 ## Fixed * Fixed issues with the doctor command used to diagnose installation problems. # Changelog: 2024-03-19 Source: https://docs.flex.ai/cli/changelog/2024-03-19 ## Added * `train` command for training models. # Changelog: 2024-03-29 Source: https://docs.flex.ai/cli/changelog/2024-03-29 ## Added * Added new flags to the `train` command: `arch`, `model path`, `parallelism`. ## Changed * Add GPU node limits based on the selected architecture through the `arch` flag. # Changelog: 2024-05-07 Source: https://docs.flex.ai/cli/changelog/2024-05-07 ## Added * CLI update * Added a subcommand to list Training Jobs: `flexai training list`. ## Fixed * Fixed issues with the `flexai doctor` command. # Changelog: 2024-05-22 Source: https://docs.flex.ai/cli/changelog/2024-05-22 ## Added * Added a subcommand to stream logs of a Training Job: `flexai training logs `. # Changelog: 2024-06-25 Source: https://docs.flex.ai/cli/changelog/2024-06-25 ## Added * Added the `flexai dataset push ` subcommand to upload files to a dataset. * Added the ability to upload multiple files to a dataset by passing multiple instances of the `--file` flag to the `flexai dataset push` subcommand. * Added the `flexai dataset list` subcommand to get a table view of all the datasets. * Added a `--dataset` flag to the `flexai training run ` subcommand to specify the dataset to use for training. * Added flags to select a specific source when using `flexai training run `: `--repo-url` and `--revision`. * Added a local configuration file: `./flexai/config.yaml`. * Added an ASCII progress bar to follow the progress of `flexai dataset push` uploads. * Added a global flag: `-v, --verbose` to enable verbose output for all commands. ## Changed * Improved the readability of the `flexai training list` subcommand output. # Changelog: 2024-07-08 Source: https://docs.flex.ai/cli/changelog/2024-07-08 ## Added * oAuth 2.0 login flow using the `flexai auth login` command. * Added the `update` and `delete` subcommands to the `flexai source` command. * Automatically open a browser window using the oAuth 2.0 login URL. * Added a structured render data view to the `flexai training inspect ` command. * Added the ability to specify file paths within a dataset through the `--file` flag in the `flexai dataset push` command. * Check for updates on every CLI command execution to make sure it is always up to date. ## Changed * Cleanup order triggered by logging out with `flexai auth logout`. ## Removed * **Breaking**: Removed the `flexai training dashboard` command. # Changelog: 2024-07-22 Source: https://docs.flex.ai/cli/changelog/2024-07-22 ## Added * Added inspect command to retrieve additional metadata for datasets. * Checks for file name duplicates in dataset uploads. * Added user feedback messages to the `flexai training run ` command. * Login flow improvements. ## Changed * Dataset file upload now supports multipart. ## Fixed * Fixed authentication redirect URL. * Fixed `flexai training fetch ` download issues. # Changelog: 2024-07-24 Source: https://docs.flex.ai/cli/changelog/2024-07-24 ## Fixed * Login flow response issues. * Installation checks improved to avoid failing on errors. # Changelog: 2024-07-29 Source: https://docs.flex.ai/cli/changelog/2024-07-29 ## Fixed * Dataset naming restrictions relaxed. # Changelog: 2024-07-30 Source: https://docs.flex.ai/cli/changelog/2024-07-30 *No changes affecting end users in this version: internal observability.* # Changelog: 2024-08-14 Source: https://docs.flex.ai/cli/changelog/2024-08-14 ## Added * Added CLI a subcommand for dataset inspection: `flexai dataset inspect `. * Added support for directory-based dataset uploads through the same flag used for file uploads. * Included a "FILES COUNT" column in the `flexai dataset list` command. ## Removed * **Breaking**: Removed the `--dataset-target-dir` flag from the `flexai training run ` command. # Changelog: 2024-09-03 Source: https://docs.flex.ai/cli/changelog/2024-09-03 ## Changed * [`flexai dataset inspect `](/cli/reference/dataset/inspect) now displays **file details**. * **Breaking**: Renamed the `--revision` flag to `--source-revision` in the [`flexai training run `](/cli/reference/training/run#flags) command. ## Fixed * Fixed a bug in the `flexai dataset push` command that didn't allow for the creation of a dataset using a name of a previously deleted dataset. * `flexai dataset push` now properly reports the progress of the dataset upload. # Changelog: 2024-10-24 Source: https://docs.flex.ai/cli/changelog/2024-10-24 ## Highlights * *Secrets management: Securely store sensitive information such as API keys or tokens. `flexai training run ` now receives `--secret` flags to pass it on to the training environment as an Environment Variable. [Learn more here](/cli/reference/secret).* * *Training Environment Variables: `flexai training run ` now receives `--env` flags to pass non-sensitive values to the training environment as an Environment Variable. [More information here](/cli/reference/training/run/#arguments).* * *Changelog CLI versioning: Changelog entries will now version CLI releases using their date in the format `[YYYY-MM-DD]`.* ## Added * FCS Secret Manager: `flexai secret` command to manage secrets. [Learn more here](/cli/reference/secret). * CLI a subcommand for dataset deletion: [`flexai dataset delete `](/cli/reference/dataset/delete). ## Changed * Changed the internals of the CLI installation flow. * Changed the output of the CLI's version command to reflect its specific build. An SHA hash will be displayed instead of the previous SemVer version: ae9c1508a46146a6b071dddbbb374c76c1147bcd ## Removed * Removed unnecessary source checks in the `source list` command, improving performance. # Changelog: 2024-11-15 Source: https://docs.flex.ai/cli/changelog/2024-11-15 ## Highlights * *`training stop`*: Added the `flexai training stop ` command to stop a running Training Job. [Learn more here](/cli/reference/training/stop). ## Added * Added the ability to **stop** a training `flexai training stop ` command to stop a running Training Job. # Changelog: 2024-11-27 Source: https://docs.flex.ai/cli/changelog/2024-11-27 ## Highlights * *Remote Storage Source Connection*: Added the `flexai training storage` command manage connections to Remote Storage Sources, such as AWS, GCS, MinIO, and R2. * *Pushing a Dataset from a Remote Storage Source*: Added support for pushing Datasets from a Remote Storage Source Connection to `flexai dataset push`. ## Added * Added the `flexai training storage` command to manage connections to Remote Storage Sources, such as AWS, GCS, MinIO, and R2. To learn more visit: * The [Storage command reference](/cli/reference/storage) * The [Uploading Datasets from Remote Sources](/platform-services/dataset-manager/from-remote/) guide ## Changed * Added support for *Remote Storage Source Connections* to `flexai dataset push`: ```diff lang="flexi" theme={null} flexai dataset push \ + --source-path + --storage-provider ``` For more details: * Visit the `flexai dataset push` command reference [Dataset push command reference](/cli/reference/dataset/push). # Changelog: 2024-12-02 Source: https://docs.flex.ai/cli/changelog/2024-12-02 ## Highlights * \[Breaking] *`flexai dataset push --file`* flag value syntax inverted when passing a value pair. ## Added * Enhanced set of checks before initiating a Training Job to ensure the correct configuration has been set. ## Changed * The `-f, --file` flag of `flexai dataset push` can receive both a string and a *Key Value pair* separated by an equals sign `=`. The *Key Value pair* syntax **is now inverted**, with the local path to first, followed by the destination path in the Dataset: ```diff lang="flexi" theme={null} flexai dataset push \ - --file = + --file = ``` For more details: * Visit the [Uploading Datasets from your local machine](/platform-services/dataset-manager/from-local/) guide. # Changelog: 2024-12-13 Source: https://docs.flex.ai/cli/changelog/2024-12-13 ## Highlights * \[New feature] **Interactive Training**: Debug and optimize your Training Jobs faster. You can now *SSH into* a Training Runtime environment and interact with it in real-time, giving you more control. * \[New feature] **Infrastructure Metrics Dashboard**: Optimize your batch sizes and performance. Access your dashboard to gain real-time insights into GPU utilization, memory usage, temperature, and other infrastructure metrics. ## Added * **Interactive Training**: The `flexai training debug-ssh` command lets you SSH into a Training Runtime environment, providing real-time access to its file system. This feature can help you debug Training Jobs more efficiently, run detailed diagnostics, and perform unlimited tests to ensure everything is optimized before scaling up to large-scale production Training Jobs. For more details, visit the [Interactive Training](/interactive-development/) guide. * **Infrastructure Metrics Dashboard**: Access your dashboard to gain real-time insights into GPU utilization, memory usage, temperature, and other infrastructure metrics. Use this tool to optimize batch sizes, identify potential bottlenecks, and troubleshoot performance issues. Visit [dashboards.flex.ai](http://dashboards.flex.ai) * ***`flexai training run`*** checks: Before starting a Training Job, FCS performs these sanity checks: * The `--source-revision` in the repository specified through the `--source-name` flag exists. * A `requirements.txt` file exists at the root of the source's revision. * The `requirements.txt` file either: * Doesn't specify versions for the following packages `torch`, `torchaudio`, and `torchvision`. * Doesn't include the `torch`, `torchaudio`, and `torchvision` packages. * The [*Entry point script path*](/cli/reference/training/run/#arguments) exists. ## Changed * Enhanced CLI help text (`--help`) for `flexai storage create` & `flexai dataset push` commands to provide more detailed information on how to use them to push Datasets from a Remote Storage Connection to FCS. # Changelog: 2024-12-20 Source: https://docs.flex.ai/cli/changelog/2024-12-20 ## Added * **System metrics in Dashboard:** Monitor CPU utilization, memory usage, and network throughput metrics in the [Infrastructure Monitor](http://dashboards.flex.ai). Use these insights to diagnose bottlenecks, define scaling strategies, and refine preprocessing decisions. * **Delete Training Jobs**: Use `flexai training delete ` command to delete successful or failed Training Jobs to keep your workspace organized and easier to manage. ## Changed * **List command sorting**: All entity listing subcommands (e.g. `flexai training list`) sort objects by creation date in ascending order, making it easier to locate relevant objects. * **General UX**: Enhanced clarity and formatting of CLI messages for improved user experience. # Changelog: 2025-01-31 Source: https://docs.flex.ai/cli/changelog/2025-01-31 ## Highlights * \[New Feature] **Managed Checkpoints**: Simplify and accelerate your AI training workflows with secure, scalable, and reusable snapshots of model states. * \[New Feature] **Live Checkpoint Capture**: List checkpoints from an in-progress or completed Training Job and reuse it for evaluation or further fine-tuning without waiting for downloads or uploads. * \[Improved UX] **Training Logs and Output Handling**: Training logs now clearly indicate their start and end, and outputs retain directory structures when fetched. ## Added * **Managed Checkpoints:** Manage checkpoints using the [`flexai checkpoint`](/cli/reference/checkpoint) command checkpoints to list, upload, download, and export them. List checkpoints from an in-progress or completed Training Job and reuse it for evaluation or resume the training using the `--checkpoint ` flag without any manual data transfer. * **Fine-Tuning:** Bring pre-trained models into FCS from a local machine or remote storage (e.g. AWS) once and reuse them for any of your fine-tuning workloads, saving on egress fees and transfer times. * **Inspect checkpoints:** View detailed checkpoint information, including creation time and metadata, using `checkpoint inspect`. * **Check for updates**: Use the `doctor` command to check for CLI updates and stay up-to-date with the latest features. * **Multi-GPU Interactive Training**: Enhanced Interactive Training with support for multi-GPU single-node setups. Use `flexai training debug-ssh --accels <1 to 8>` for debug and optimization of distributed workloads. ## Changed * **Sorted Checkpoint List**: The `training checkpoints` or `checkpoint list` sorts checkpoints by creation time in ascending order, making it easier to locate relevant checkpoints. * **Directory Retention**: Outputs fetched from Training Jobs or checkpoints maintain their original directory structure, simplifying file management. * **Improved Help Text**: Help messages for `push` commands provide clearer guidance. * **Improved Training flow UX**: Improved the `training run` output message to suggest the command to run to monitor its log output. ```bash theme={null} flexai training run mistral_ft_34 ... Training job mistral_ft_34. Use 'flexai training logs mistral_ft_34' to follow the progress of your Training Job. ``` * **Clear Logs Start**: `training logs` clearly indicate the start point of the logs stream. ## Fixed * **Logs Termination UX**: The logs stream from `training logs` terminates at the end of the Training Job. * **Relative Path Uploads**: Fixed failures with path resolutions for dataset and checkpoint uploads. # Changelog: 2025-02-04 Source: https://docs.flex.ai/cli/changelog/2025-02-04 ## Highlights * **Enabled support for FlashAttention**: Training Jobs can now make use of the FlashAttention package without requiring any additional setup other than including it in the `requirements.txt` file of the repository. ## Added * **Checkpoint Details**: Use the `flexai checkpoint inspect` command to view the detailed contents and metadata of checkpoints uploaded via the `flexai checkpoint push`, including its files, source, and both creation and update times. * **Storage Connection Details:** Use the `storage inspect` command to review connections to storage providers (e.g. AWS S3) and associated metadata. * **Enabled support for FlashAttention**: The Training runtime now includes the `FLASH_ATTENTION_SKIP_CUDA_BUILD=1` environment variable to allow for the [flash-attn](https://github.com/Dao-AILab/flash-attention) to be used during Training Jobs. ## Changed * **Validation for Storage Connections**: When creating a Remote Storage Provider Connection using `flexai storage create`, an error will be returned if the specified Secret name cannot be found, providing immediate feedback instead of creating an invalid connection. * **Enhanced Training commands messages**: Improved the messages for `flexai training` subcommands to provide more context and better guidance during the different workflows. ## Fixed * **Last Checkpoint Availability:** Fixed an issue where the final checkpoint created by FCS-managed checkpoints for a Training Job was sometimes inaccessible. # Changelog: 2025-03-10 Source: https://docs.flex.ai/cli/changelog/2025-03-10 ## Changed * **No Dataset Size Limitation**: Training jobs can now process datasets of any size. Previously, FCS recommended that datasets be limited to 150GB for Training Jobs. * **Faster Training Job start times**: Dataset management optimizations have reduced the overall starting time of Training Jobs, by speeding up the time to load a dataset.Previously, Training Job start times could be much longer depending on the size of the dataset. * **Read-Only /input/ Directory**: The `/input/` directory ─ where Datasets are mounted ─ is now set as read-only, improving clarity on its intended usage. You can continue to use the `/output/` directory to save artifacts and results as well as any data processing operations you need to perform during training. * **Faster Training Fetch Availability**: The `training fetch ` command is now available 35% sooner after a Training Job completes due to an optimized save process. * UX improvement for `training fetch` to display a smoother data download progress. # Changelog: 2025-04-01 Source: https://docs.flex.ai/cli/changelog/2025-04-01 ## Highlights * **Training Jobs now support multiple Datasets**: You can now attach multiple datasets to a Training Job. * **`flexai source` command deprecated**: The `flexai source` command has been deprecated. You can now pass the `--repository-url` and `--repository-revision` flags directly to the `flexai training run` command. ## Added * **Support for multiple Datasets in Training Jobs**: You can now attach multiple datasets to a Training Job, allowing for more complex training scenarios and improved data management. * **`flexai code-registry` command**: The new `flexai code-registry` command allows you to grant access to your code repositories to the FlexAI. It replaces the functionality of the deprecated `flexai source` command. The `flexai code-registry` command provides a more streamlined and efficient way to manage your code repositories within the FlexAI ecosystem. * **`flexai training run` new flags**: You can now pass the `--repository-url` and `--repository-revision` flags to the `flexai training run` command, allowing you to pass any public GitHub repository URL or the url of a private GitHub repository that has been registered with FlexAI using the `flexai code-registry connect` command. ```diff lang="flexi" theme={null} flexai training run my_training_job42 \ - --source-name --source-revision \ + --repository-url --repository-revision \ // ... ``` ## Changed * **`flexai training run` command**: The `flexai training run` command has been updated to accept multiple instances of the `-D` / `--dataset` flag, which can be used to specify a Dataset name and also supports a value mapping to specify the name of the Dataset inside the `/input/` directory. For example: * Key-value pair: ```bash theme={null} flexai training run -D media_files_revisited_20250221=media_files //... ``` Will be mounted as: ```text theme={null} /input/media_files ``` * Dataset name: ```bash theme={null} flexai training run -D media_files_revisited_20250221 //... ``` Will be mounted as: ```text theme={null} /input/media_files_revisited_20250221 ``` ## Removed * **`flexai source` command deprecated**: The `flexai source` command has been deprecated in favor of the `flexai code-registry` command. # Changelog: 2025-05-15 Source: https://docs.flex.ai/cli/changelog/2025-05-15 ## Highlights * **FlexAI Inference** - *Preview Release*: You can now create and manage Inference Endpoints using the `flexai inference` command! * **Hugging Face Storage Connector**: Hugging Face is now part of the list of Remote Storage Providers you can leverage by creating a `storage` connection. * **TensorBoard Support**: Training Jobs can now be tracked using [TensorBoard](https://www.tensorflow.org/tensorboard), allowing for real-time visualization of training metrics and performance. ## Added * **Preview release**: Support for creating and managing Inference Endpoints via the [`flexai inference` command](/cli/reference/inference/serve), enabling users to serve, update, and monitor model inference endpoints directly from the CLI. You only need to specify pass a `--hf-token-secret` flag pointing to the name of the Secret containing your Hugging Face access token and the name of the [vLLM supported model](https://docs.vllm.ai/en/latest/models/supported_models.html) you'd like to serve. Refer to the [inference CLI command documentation](/cli/reference/inference) for more details. * Hugging Face as a supported [Remote Storage Provider](/cli/reference/storage/create). This addition enables you to easily push Datasets and model Checkpoints directly from the Hugging Face Hub — no need to download them locally to push them back again any more. * Support for TensorBoard in Training Jobs has been added to FlexAI's suite of observability tools. TensorBoard is enabled by default and you only have to make your code write the files required by TensorBoard to the path specified in the `TENSORBOARD_LOG_DIR` environment variable. Visit [https://tensorboard.flex.ai/](https://tensorboard.flex.ai/) and log in using the same credentials you use to access the FlexAI CLI. A screenshot of the TensorBoard interface Contrary to what the name might indicate, **TensorBoard** is not limited to TensorFlow projects; it can also be used with **PyTorch** and other frameworks. # Changelog: 2025-05-19 Source: https://docs.flex.ai/cli/changelog/2025-05-19 ## Changed * FlexAI Inference vLLM Arguments: The `flexai inference serve` command's [optional `vLLM_arguments`](/cli/reference/inference/serve) have switched from a "Supported argument list" schema to a "Unsupported argument list" schema. This enables you to safely pass most of [vLLM Engine Arguments](https://docs.vllm.ai/en/latest/serving/engine_args.html#engine-arguments) when creating an Inference Endpoint. Currently, the only *unsupported* argument is `--device`, which value is handled by FlexAI. # Changelog: 2025-07-15 Source: https://docs.flex.ai/cli/changelog/2025-07-15 ## Highlights * **Added support for multi-project repositories**: You can now use the `-q`/`--requirements-path` flag to specify the path of a `requirements.txt` file in a specific path within your repository. ## Added * **Support for multi-project repositories**: The `-q`/`--requirements-path` flag allows specifying a path to a `requirements.txt` file within a specific project in a multi-project repository. The `requirements.txt` file will be fetched from the root of the repository by default. * Available for both the [`flexai training run`](/cli/reference/training/run/#--requirements-path) and [`flexai training debug-ssh`](/cli/reference/training/debug-ssh/#--requirements-path) commands. ## Changed * **Training Job Checkpoint details**: The `flexai training checkpoint ` command now includes [more detailed information](/cli/reference/training/checkpoints/#example) about the FlexAI Managed Checkpoints, such as *Step*, *Training Loss*, and *Evaluation Loss*. # Changelog: 2025-07-25 Source: https://docs.flex.ai/cli/changelog/2025-07-25 > Version: `b28c9594f9e596118341be7be53dbaca616072dc` ## Added * **Checkpoint status metadata**: The `flexai checkpoint inspect ` command now includes [checkpoint status metadata](/cli/reference/checkpoint/inspect/#returned-information) such as *Step*, *Training Loss*, and *Evaluation Loss*. * **`json` output format to a Training Job's checkpoint list**: The `flexai training checkpoints ` command now supports [the `-j`/`--json` flag](/cli/reference/training/checkpoints/#--json), allowing you to view the checkpoint list in JSON format aside from the default table format. # Installing the FlexAI CLI Source: https://docs.flex.ai/cli/install Install and set up the FlexAI command-line interface ## System requirements The FlexAI CLI is available for the following operating systems and versions: * Ubuntu 20.04, 22.04, 24.04 * macOS 14 and above * Windows 10 and Windows 11 (through WSL2 in both cases) ## Installation Steps 1. Run the installation script ```bash theme={null} curl -fsSL https://cli.flex.ai/install.sh | sh ``` You can review the installation script before running it: You can view the installation script at [https://cli.flex.ai/install.sh](https://cli.flex.ai/install.sh) 2. Add it to your system's `$PATH` 3) Verify your installation As a first step, it's always a good idea to verify your system can connect to FlexAI and make sure CLI is properly installed. That's the purpose of the `doctor` command: ```bash theme={null} flexai doctor ``` You should get a message like this: ```text theme={null} Running doctor checks... [1/4] ✅ The FlexAI binary file is part of the system's $PATH variable [2/4] ✅ Successfully connected to the FlexAI API server [3/4] ✅ Successfully connected to the FlexAI update server [4/4] ✅ Using the latest FlexAI CLI version available Your system is healthy ``` 4) Logging in The next step is to [log in and authenticate your CLI](/cli/authenticate). *** ## Uninstalling the FlexAI CLI If you ever need to uninstall the FlexAI CLI, you can do so by removing both the `~/.flexai` directory and the reference to the *flexai binary* from your `$PATH` variable. # Command: checkpoint Source: https://docs.flex.ai/cli/reference/checkpoint Manage model checkpoints on FlexAI — push, export, inspect, and delete Checkpoints are a FlexAI entity that represents a snapshot of a model's state at a given point in time. Checkpoints capture a model's state at various stages of training. These snapshots include model weights, optimizer state, and other relevant training data. This allows you to **resume training** from a specific point, preventing data loss and enabling experimentation with different training paths while helping you avoid unnecessarily repeating training iterations. Visit the [FlexAI Checkpoint Manager Service section](/platform-services/checkpoint-manager/) to learn more about how FlexAI Managed Checkpoints work. Checkpoints can be [*pushed*](/cli/reference/checkpoint/push) to FlexAI directly from the host machine running the FlexAI CLI or from a [Remote Storage Provider Connection](/cli/reference/storage), such as Amazon S3, Cloudflare R2, or GCP Cloud Storage, among others. They can be individual files or entire directories. You will find more information about Managed Checkpoints and the benefits they bring to your AI training workflows in the [Managed Checkpoints](/platform-services/checkpoint-manager/) page. You can manage Checkpoints using the `flexai checkpoint` set of subcommands. ## Available subcommands * [`flexai checkpoint delete`](/cli/reference/checkpoint/delete) - Deletes a Checkpoint. * [`flexai checkpoint export`](/cli/reference/checkpoint/export) - Exports a Checkpoint to a [Storage Provider](/cli/reference/storage). * [`flexai checkpoint fetch`](/cli/reference/checkpoint/fetch) - Fetches a Checkpoint. * [`flexai checkpoint inspect`](/cli/reference/checkpoint/inspect) - Inspects a Checkpoint's metadata. * [`flexai checkpoint list`](/cli/reference/checkpoint/list) - Lists Checkpoints. * [`flexai checkpoint push`](/cli/reference/checkpoint/push) - Pushes a Checkpoint to FlexAI. # checkpoint delete Source: https://docs.flex.ai/cli/reference/checkpoint/delete Delete a checkpoint from FlexAI Deletes a checkpoint. ## Usage ```bash theme={null} flexai checkpoint delete [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `checkpoint_identifier` | string | Yes | The name or identifier of the Checkpoint to delete. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Provides more detailed output when deleting a Checkpoint. | # checkpoint export Source: https://docs.flex.ai/cli/reference/checkpoint/export Export a checkpoint to a remote storage provider Uploads a Checkpoint generated by a Training Job to a [Remote Storage Provider Connection](/cli/reference/storage), such as Amazon S3, Cloudflare R2, or GCP Cloud Storage. ## Usage ```bash theme={null} flexai checkpoint export --storage-provider --destination-path [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `checkpoint_identifier` | string | Yes | The name or UUID of the Checkpoint to export. | ## Flags | Flag | Short | Type | Required | Description | | - | - | - | - | - | | `--storage-provider` | `-s` | string | Yes | The name of the Remote Storage Provider Connection to upload to. | | `--destination-path` | `-d` | string | Yes | The destination path in the remote storage where the checkpoint will be uploaded. | | `--help` | `-h` | boolean | No | Displays this help page. | | `--verbose` | `-v` | boolean | No | Provides more detailed output when exporting a Checkpoint. | # checkpoint fetch Source: https://docs.flex.ai/cli/reference/checkpoint/fetch Download checkpoint files to your local machine Downloads a Checkpoint from FlexAI to the host machine running the FlexAI CLI. ## Usage ```bash theme={null} flexai checkpoint fetch [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `checkpoint_identifier` | string | Yes | The name of a Checkpoint pushed to the FlexAI Checkpoint Manager Service, or the UUID of a Checkpoint created during the execution of a FlexAI Training or Fine-tuning workload. See [`flexai checkpoint list`](/cli/reference/checkpoint/list) or [`flexai training checkpoints`](/cli/reference/training/checkpoints) for more details. | ## Flags | Flag | Short | Type | Default | Description | | - | - | - | - | - | | `--destination` | `-d` | string | `./` | Specifies the directory where the Checkpoint will be saved in the host machine's filesystem. | | `--help` | `-h` | boolean | | Displays this help page. | | `--verbose` | `-v` | boolean | | Provides more detailed output when fetching a Checkpoint. | # checkpoint inspect Source: https://docs.flex.ai/cli/reference/checkpoint/inspect View detailed information about a specific checkpoint Returns detailed information about a Checkpoint: its inception time, metadata, status, file content list, and more. It outputs the information in YAML format by default. ## Usage ```bash theme={null} flexai checkpoint inspect [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `checkpoint_identifier` | string | Yes | The name of the Checkpoint to inspect. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Displays checkpoint information in JSON format instead of the default ASCII table. | | `--verbose` | `-v` | boolean | Provides more detailed output when inspecting a Checkpoint. | ## Examples ### Inspecting a checkpoint Inspect a checkpoint named `my-checkpoint` ```bash theme={null} flexai checkpoint inspect my-checkpoint ``` Which will output: ```yaml title="flexai checkpoint inspect response (formatted)" theme={null} name: my-checkpoint uuid: 12345678-1234-1234-1234-123456789abc created_at: "2024-01-15T10:30:00Z" status: ready size_bytes: 1073741824 files: - path: model.safetensors size: 1073741824 ``` ```json title="flexai checkpoint inspect response" theme={null} { "name": "my-checkpoint", "uuid": "12345678-1234-1234-1234-123456789abc", "created_at": "2024-01-15T10:30:00Z", "status": "ready", "size_bytes": 1073741824, "files": [ { "path": "model.safetensors", "size": 1073741824 } ] } ``` # checkpoint list Source: https://docs.flex.ai/cli/reference/checkpoint/list List all checkpoints and their current status Lists all the available Checkpoints. ## Usage ```bash theme={null} flexai checkpoint list [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Displays checkpoint information in JSON format instead of the default ASCII table. | | `--verbose` | `-v` | boolean | Provides more detailed output when listing a Checkpoint. | ## Examples ### List all available checkpoints ```text theme={null} flexai checkpoint list ``` ### List checkpoints in JSON format ```text theme={null} flexai checkpoint list --json ``` # checkpoint push Source: https://docs.flex.ai/cli/reference/checkpoint/push Upload checkpoint files to FlexAI from local or remote storage Upload files to create a new checkpoint in FlexAI ## Usage ```bash theme={null} flexai checkpoint push [(--file ... | --file = ...) | (--storage-provider --source-path )] [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `checkpoint_name` | string | Yes | Pushes a new Checkpoint to FlexAI from either the host machine running the FlexAI CLI or from a [Remote Storage Provider Connection](/cli/reference/storage), such as Hugging Face, Amazon S3, Cloudflare R2 or GCP Cloud Storage. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--file` | `-f` | key-value | Pushes a new Checkpoint to FlexAI from either the host machine running the FlexAI CLI or from a [Remote Storage Provider Connection](/cli/reference/storage), such as Hugging Face, Amazon S3, Cloudflare R2 or GCP Cloud Storage. 'Key=Value' path pair: ``=`` | | `--help` | `-h` | boolean | Displays this help page. | | `--source-path` | | string | The path to the source checkpoint files. | | `--storage-provider` | | string | The name of the Remote Storage Provider Connection to use for the Checkpoint upload. | | `--verbose` | `-v` | boolean | Provides more detailed output when pushing a Checkpoint. | ## Examples ### Push checkpoint from local files flexai checkpoint push my-checkpoint --file /path/to/checkpoint/ ```bash theme={null} flexai checkpoint push my-checkpoint --file /path/to/checkpoint/ ``` ### Push checkpoint from storage provider flexai checkpoint push my-checkpoint --storage-provider my-s3-provider --source-path bucket-name/checkpoints/model/ ```bash theme={null} flexai checkpoint push my-checkpoint --storage-provider my-s3-provider --source-path bucket-name/checkpoints/model/ ``` # Command: cluster Source: https://docs.flex.ai/cli/reference/cluster Discover the compute clusters available to your organization on FlexAI Clusters are the compute backends that run your FlexAI workloads. Each cluster is provisioned with a specific accelerator type (for example, NVIDIA H100 or A100) and is made available to one or more organizations. By default, FlexAI selects a cluster automatically based on the `--device-arch` flag of the command you run. If you need to pin a workload to a specific cluster — for instance, to target a particular accelerator model — you can pass `--affinity cluster=` to [`flexai training run`](/cli/reference/training/run), [`flexai training debug-ssh`](/cli/reference/training/debug-ssh), or [`flexai inference serve`](/cli/reference/inference/serve). Use the `flexai cluster` set of subcommands to discover which clusters your organization can use. ## Available subcommands * [`flexai cluster list`](/cli/reference/cluster/list) - Lists the clusters available to your organization. # cluster list Source: https://docs.flex.ai/cli/reference/cluster/list List the compute clusters available to your organization Lists the compute clusters available to your organization. The `Name` column of the output is the value you pass to `--affinity cluster=` when starting a [training job](/cli/reference/training/run), an [interactive debug-ssh session](/cli/reference/training/debug-ssh), or an [inference endpoint](/cli/reference/inference/serve). ## Usage ```bash theme={null} flexai cluster list [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Displays cluster information in JSON format instead of the default ASCII table. | ## Output Each row in the output represents one cluster: | Column | Description | | - | - | | `Name` | The cluster's identifier. Use this value with `--affinity cluster=`. | | `Status` | The current operational status of the cluster. | | `Default` | `true` if this is the cluster FlexAI selects when no `--affinity` is provided and the requested `--device-arch` matches. | | `Shared` | `true` if the cluster is shared across multiple organizations; `false` if it is dedicated to your organization. | ## Examples ### List all available clusters ```text theme={null} flexai cluster list ``` ### List clusters in JSON format ```text theme={null} flexai cluster list --json ``` ### Use a listed cluster name with `--affinity` ```bash theme={null} flexai training run my-training-job \ --repository-url https://github.com/flexaihq/nanoGPT/ \ --affinity cluster= \ --accels 4 -- train.py ``` # Command: code-registry Source: https://docs.flex.ai/cli/reference/code-registry Manage GitHub repository connections via the FlexAI GitHub App The FlexAI code-registry command manages the connection between FlexAI and registries that host the source code of the models. Connections are handled by the FlexAI GitHub App, which can be installed at the organization level by an organization owner, as well as at the repository level by the repository owner. The FlexAI GitHub App only needs to be installed once per organization or repository. Once installed, organization members can use any of the allowed repositories to create workloads with FlexAI. ## Available subcommands * [connect](/cli/reference/code-registry/connect) * [list](/cli/reference/code-registry/list) # code-registry connect Source: https://docs.flex.ai/cli/reference/code-registry/connect Connect a GitHub repository to FlexAI Displays a unique URL that can be used to install the FlexAI GitHub App in a GitHub organization or for a sub-set of repositories. ## Usage ```bash theme={null} flexai code-registry connect [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Provides more detailed output when connecting a Code Registry. | # code-registry list Source: https://docs.flex.ai/cli/reference/code-registry/list List all connected GitHub repositories Lists all the connected code registries by providing their Name, Provider Name, Age, and Status. ## Usage ```bash theme={null} flexai code-registry list [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Displays code registry list information in JSON format instead of the default ASCII table. | | `--verbose` | `-v` | boolean | Provides more detailed output when listing a Code Registry. | # completion Source: https://docs.flex.ai/cli/reference/completion Generate shell autocompletion scripts for the FlexAI CLI Generate the autocompletion scripts for the flexai CLI for the specified shell. ## Usage ```bash theme={null} flexai completion [shell_name] ``` ## Supported Shells * [`flexai completion bash`](/cli/reference/completion/bash) * [`flexai completion fish`](/cli/reference/completion/fish) * [`flexai completion powershell`](/cli/reference/completion/powershell) * [`flexai completion zsh`](/cli/reference/completion/zsh) ## Usage 1. Identify your shell: ```bash theme={null} echo $0 ``` 2. Follow the instructions for your shell by visiting the corresponding command's documentation page to enable autocompletion. Alternatively, you can follow Cobra's [official documentation](https://cobra.dev/docs/how-to-guides/shell-completion/) for more detailed instructions on setting up shell completions. # completion bash Source: https://docs.flex.ai/cli/reference/completion/bash Generate bash autocompletion script for the FlexAI CLI Generate the autocompletion script for the bash shell. ## Usage ```bash theme={null} flexai completion bash [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--no-descriptions` | | boolean | Disable completion descriptions | | `--verbose` | `-v` | boolean | Enable verbose logging | ## Examples ### Generate a bash completion script ```bash theme={null} flexai completion bash > flexai-completion.bash && \ mkdir -p ~/.local/share/bash-completion/completions && \ mv flexai-completion.bash ~/.local/share/bash-completion/completions/flexai && \ source ~/.bashrc ``` # completion fish Source: https://docs.flex.ai/cli/reference/completion/fish Generate fish autocompletion script for the FlexAI CLI Generate the autocompletion script for the fish shell. ## Usage ```bash theme={null} flexai completion fish [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--no-descriptions` | | boolean | Disable completion descriptions | | `--verbose` | `-v` | boolean | Enable verbose logging | ## Examples ### Generate a fish completion script ```bash theme={null} flexai completion fish > ~/.config/fish/completions/flexai.fish && \ source ~/.config/fish/config.fish ``` # completion powershell Source: https://docs.flex.ai/cli/reference/completion/powershell Generate PowerShell autocompletion script for the FlexAI CLI Generate the autocompletion script for powershell. ## Usage ```bash theme={null} flexai completion powershell [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--no-descriptions` | | boolean | Disable completion descriptions | | `--verbose` | `-v` | boolean | Enable verbose logging | ## Examples ### Generate a powershell completion script ```bash theme={null} flexai completion powershell > flexai-completion.ps1 && \ mkdir -p (Split-Path $PROFILE) && \ Add-Content -Path $PROFILE -Value ``` # completion zsh Source: https://docs.flex.ai/cli/reference/completion/zsh Generate zsh autocompletion script for the FlexAI CLI Generate the autocompletion script for the zsh shell. ## Usage ```bash theme={null} flexai completion zsh [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--no-descriptions` | | boolean | Disable completion descriptions | | `--verbose` | `-v` | boolean | Enable verbose logging | ## Examples ### Generate a zsh completion script ```bash theme={null} flexai completion zsh > _flexai ``` ### Create the zsh site-functions directory if it doesn't exist ```bash theme={null} sudo mkdir -p /usr/local/share/zsh/site-functions ``` ### Move the completion script to the site-functions directory ```bash theme={null} sudo mv _flexai /usr/local/share/zsh/site-functions/ ``` ### Reload your zsh configuration ```bash theme={null} source ~/.zshrc ``` # Command: dataset Source: https://docs.flex.ai/cli/reference/dataset Manage datasets on FlexAI — upload, inspect, and delete datasets The `flexai dataset` command manages Datasets by allowing for their creation, listing, and deletion. ## Available subcommands * [`flexai dataset delete`](/cli/reference/dataset/delete) - Deletes a Dataset. * [`flexai dataset inspect`](/cli/reference/dataset/inspect) - Displays detailed information about a Dataset. * [`flexai dataset list`](/cli/reference/dataset/list) - Lists all the Datasets. * [`flexai dataset push`](/cli/reference/dataset/push) - Uploads a Dataset to the FlexAI. ## Dataset Statuses Datasets in FlexAI can have different statuses, which indicate their availability and readiness for use. The following statuses are used: | Status | Description | | - | - | | `available` | The dataset is ready for use | | `error` | The dataset is in an error state and cannot be used | | `pending` | The dataset is being processed by FlexAI | | `uploading` | The dataset is being uploaded to FlexAI | | `syncing` | The dataset is being synchronized with the Remote Storage Provider | # dataset delete Source: https://docs.flex.ai/cli/reference/dataset/delete Delete a dataset from FlexAI Deletes a dataset from FlexAI. ## Usage ```bash theme={null} flexai dataset delete [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `dataset_name` | string | Yes | The name of the Dataset to delete. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Provides more detailed output when deleting a Dataset. | # dataset inspect Source: https://docs.flex.ai/cli/reference/dataset/inspect View detailed information about a specific dataset Returns detailed information about a dataset: its inception time, metadata, status, file content list, and more. It outputs the information in YAML format by default. ## Usage ```bash theme={null} flexai dataset inspect [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `dataset_name` | string | Yes | The name of the Dataset to inspect. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Displays dataset information in JSON format instead of the default YAML. | | `--verbose` | `-v` | boolean | Provides more detailed output when inspecting a Dataset. | # dataset list Source: https://docs.flex.ai/cli/reference/dataset/list List all datasets and their current status Lists all the available datasets. ## Usage ```bash theme={null} flexai dataset list [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Displays dataset information in JSON format instead of the default ASCII table. | | `--verbose` | `-v` | boolean | Provides more detailed output when listing datasets. | ## Examples ### List all datasets flexai dataset list ```bash theme={null} flexai dataset list ``` ### List datasets in JSON format flexai dataset list --json ```bash theme={null} flexai dataset list --json ``` # dataset push Source: https://docs.flex.ai/cli/reference/dataset/push Upload dataset files to FlexAI from local or remote storage Pushes a new dataset to a FlexAI. Multiple files can be uploaded at once by using multiple instances of the --file flag, or by pointing the --file flag to a directory containing dataset files. The [Datasets Manager Service page](/platform-services/dataset-manager/) provides more information on how to upload (or *push*) both local files and files from a [Remote Storage Provider](/platform-services/remote-storage-connections-manager). ## Usage ```bash theme={null} flexai dataset push [(--file = ...) | (--storage-provider --source-path )] [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `dataset_name` | string | Yes | The name of the Dataset you want to create. Dataset names must be unique within a given FlexAI organization and must follow the [Resource Naming conventions](/best-practices/resource-naming-conventions/). | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--file` | `-f` | string/key-value | The local path to the dataset files on the host machine running the FlexAI CLI. The Dataset will be mounted and made available to your code under the `/input/` path. Can also be a mapping between a local path and the path where the dataset file will be stored in the Dataset. If no target nested path is provided, the file will be stored at the root of the Dataset (`/input//`). If a target directory path is provided, the file will be stored under that path (`/input///`). If the source path is a directory, the entire directory will be uploaded recursively to the Dataset. To learn more about Dataset Organization and their File Structure, visit the [Datasets Manager Service page](/platform-services/dataset-manager/#dataset-organization). | | `--help` | `-h` | boolean | Displays this help page. | | `--source-path` | | string | The path to the bucket and file or directory to be pushed. Syntax: `/`. Used in together with the [`--storage-provider`](#--storage-provider) flag. | | `--storage-provider` | | string | The name of the Remote Storage Provider Connection to use for the Dataset upload. You can list the available Remote Storage Provider Connections using the [`flexai storage list`](/cli/reference/storage/list) command. | | `--verbose` | `-v` | boolean | Provides more detailed output when pushing a Dataset. | ## Examples ### Push local files to a dataset flexai dataset push my-dataset --file /path/to/file1.csv --file /path/to/file2.txt ```bash theme={null} flexai dataset push my-dataset --file /path/to/file1.csv --file /path/to/file2.txt ``` ### Push files with custom paths flexai dataset push my-dataset --file /path/to/file1.csv=train/data/file1.csv --file /path/to/file2.txt=eval/data/file2.txt ```bash theme={null} flexai dataset push my-dataset --file /path/to/file1.csv=train/data/file1.csv --file /path/to/file2.txt=eval/data/file2.txt ``` ### Push from remote storage provider flexai dataset push my-dataset --storage-provider my-s3-provider --source-path bucket-name/path/to/data/ ```bash theme={null} flexai dataset push my-dataset --storage-provider my-s3-provider --source-path bucket-name/path/to/data/ ``` # General Usage Commands Source: https://docs.flex.ai/cli/reference/general General CLI commands for authentication, diagnostics, and updates A set of commands that manage the CLI tool and its connection to FlexAI: * [`flexai auth`](/cli/reference/general/auth) - Authenticates the CLI. * [`flexai auth login`](/cli/reference/general/auth##auth-login) - Logs the current instance of the FlexAI CLI in. * [`flexai auth logout`](/cli/reference/general/auth##auth-logout) - Logs the current account out of the CLI. * [`flexai auth user`](/cli/reference/general/user) - Displays details about the current user. * [`flexai auth user`](/cli/reference/general/user/info) - Displays detailed information about the current user. * [`flexai auth user`](/cli/reference/general/user/orgs) - Displays detailed information about the current user's organizations. * [`flexai auth user`](/cli/reference/general/user/set-default-org) - Sets the default organization for the current user. * [`flexai doctor`](/cli/reference/general/doctor) - Runs a set of tests to check if the CLI is working properly. * [`flexai update`](/cli/reference/general/update) - Updates the CLI to the latest version. * [`flexai version`](/cli/reference/general/version) - Prints out the CLI's version SHA hash. # general auth Source: https://docs.flex.ai/cli/reference/general/auth Manage FlexAI CLI authentication Manages the authentication state of the CLI. ## Usage ```bash theme={null} flexai auth [command] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Enable verbose logging. | # general auth login Source: https://docs.flex.ai/cli/reference/general/auth/login Authenticate the FlexAI CLI with your account Initiates the authentication flow of the CLI against the FlexAI authentication service. Depending on the host system's capabilities, the CLI will either open a browser window or display a URL to visit. ## Usage ```bash theme={null} flexai auth login ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Enable verbose logging. | ## Examples ### Log in to FlexAI ```bash theme={null} flexai auth login ``` # general auth logout Source: https://docs.flex.ai/cli/reference/general/auth/logout Sign out of the FlexAI CLI Logs the current user out of the CLI. This command will remove the authentication tokens from the host system's `~/.flexai/config.yaml` file. ## Usage ```bash theme={null} flexai auth logout ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Enable verbose logging. | ## Examples ### Log out from FlexAI ```bash theme={null} flexai auth logout ``` # general doctor Source: https://docs.flex.ai/cli/reference/general/doctor Run diagnostics to verify your FlexAI CLI installation Run test suites to check if your system is ready to run the CLI. ## Usage ```bash theme={null} flexai doctor [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Enable verbose logging | # general update Source: https://docs.flex.ai/cli/reference/general/update Update the FlexAI CLI to the latest version If a newer version is available, this tool is updated in place with it ## Usage ```bash theme={null} flexai update [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Enable verbose logging | # general user Source: https://docs.flex.ai/cli/reference/general/user View and manage your FlexAI user account Manage user account operations including viewing user info, listing organizations, and setting default organization. ## Usage ```bash theme={null} flexai user [command] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Enable verbose logging. | # general user info Source: https://docs.flex.ai/cli/reference/general/user/info Display your FlexAI account information Display information about the current user including their organizations and default organization. ## Usage ```bash theme={null} flexai user info [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Enable verbose logging. | # general user orgs Source: https://docs.flex.ai/cli/reference/general/user/orgs List organizations associated with your FlexAI account Display all organizations that the current user has access to. ## Usage ```bash theme={null} flexai user orgs [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Enable verbose logging. | # general user set-default-org Source: https://docs.flex.ai/cli/reference/general/user/set-default-org Set the default organization for FlexAI CLI commands Set the specified organization as the default organization for the current user. ## Usage ```bash theme={null} flexai user set-default-org [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `organization_identifier` | string | Yes | The name of the organization to set as the default for the current user. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Enable verbose logging. | # general version Source: https://docs.flex.ai/cli/reference/general/version Display the current FlexAI CLI version Prints out the CLI's version SHA hash. ## Usage ```bash theme={null} flexai version ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Enable verbose logging | # Command: inference Source: https://docs.flex.ai/cli/reference/inference Manage inference endpoints on FlexAI — deploy, scale, and monitor models The `flexai inference` command manages Inference Endpoints by allowing for their deployment and management. An Inference Endpoint is a hosted model that can be used for inference tasks, such as text generation, image classification, and more. Inference Endpoints are created from models hosted on Hugging Face, and they can be deployed to FlexAI's infrastructure for easy access and scalability. You can manage Inference Endpoints using the `flexai inference` set of subcommands. ## Available subcommands * [`flexai inference delete`](/cli/reference/inference/delete) - Deletes an Inference Endpoint. * [`flexai inference inspect`](/cli/reference/inference/inspect) - Displays detailed information about an Inference Endpoint. * [`flexai inference list`](/cli/reference/inference/list) - Lists all the Inference Endpoints. * [`flexai inference logs`](/cli/reference/inference/logs) - Displays a stream of logs from an Inference Endpoint. * [`flexai inference resume`](/cli/reference/inference/resume) - Resumes a paused Inference Endpoint. * [`flexai inference scale`](/cli/reference/inference/scale) - Allows for the definition of scaling policies for an Inference Endpoint. * [`flexai inference serve`](/cli/reference/inference/serve) - Creates an Inference Endpoint from a model hosted by Hugging Face. * [`flexai inference stop`](/cli/reference/inference/stop) - Stops an Inference Endpoint. # inference delete Source: https://docs.flex.ai/cli/reference/inference/delete Delete an inference endpoint Delete an existing Inference Endpoint by specifying its name. This action is irreversible and will permanently remove the endpoint ## Usage ```bash theme={null} flexai inference delete [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `inference_endpoint_name` | string | Yes | The name of the Inference Endpoint to delete. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Provides more detailed output when deleting an Inference Endpoint. | # inference inspect Source: https://docs.flex.ai/cli/reference/inference/inspect View detailed information about a specific inference endpoint Displays detailed information about an Inference Endpoint, including its name, ID, creator, owner, configuration, runtime status, creation date, endpoint URL, and its Playground URL (a hosted [Chainlit](https://chainlit.io/) instance that's already hooked up to your Inference Endpoint). ## Usage ```bash theme={null} flexai inference inspect [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `inference_endpoint_name` | string | Yes | The name of the Inference Endpoint to inspect. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Displays inference information in JSON format. | | `--verbose` | `-v` | boolean | Provides more detailed output when inspecting an Inference Endpoint. | # inference list Source: https://docs.flex.ai/cli/reference/inference/list List all inference endpoints and their current status Lists all the Inference Endpoints that have been created. The output includes the name of the deployment, its status, age, and endpoint. ## Usage ```bash theme={null} flexai inference list [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Displays inference information in JSON format instead of the default ASCII table. | | `--verbose` | `-v` | boolean | Provides more detailed output when listing a Inference. | ## Examples ### List all inference endpoints ```bash theme={null} flexai inference list ``` ### List inference endpoints in JSON format flexai inference list --json ```bash theme={null} flexai inference list --json ``` # inference logs Source: https://docs.flex.ai/cli/reference/inference/logs Stream or retrieve logs from an inference endpoint Displays a stream of logs from an Inference Endpoint. The logs include information about the deployment's status, the model being served, and the requests being processed. ## Usage ```bash theme={null} flexai inference logs [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `inference_endpoint_name` | string | Yes | The name of the Inference Endpoint to view logs for. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--no-color` | | boolean | Disables color formatting in the log output. | | `--verbose` | `-v` | boolean | Provides more detailed output when viewing logs. | # inference resume Source: https://docs.flex.ai/cli/reference/inference/resume Resume a stopped inference endpoint Resumes a stopped Inference Endpoint, making it available to serve requests again. ## Usage ```bash theme={null} flexai inference resume [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `inference_endpoint_name` | string | Yes | The name of the Inference Endpoint to resume. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Provides more detailed output when resuming an Inference Endpoint. | # inference runtime Source: https://docs.flex.ai/cli/reference/inference/runtime Manage inference endpoint runtimes Manage inference runtimes. ## Usage ```bash theme={null} flexai inference runtime [command] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Enable verbose logging. | # inference runtime list Source: https://docs.flex.ai/cli/reference/inference/runtime/list List available inference runtimes and their configurations List inference runtimes. ## Usage ```bash theme={null} flexai inference runtime list [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Output in JSON format (default ASCII table). | | `--verbose` | `-v` | boolean | Enable verbose logging. | # inference scale Source: https://docs.flex.ai/cli/reference/inference/scale Scale the number of replicas for an inference endpoint Allows for the definition of scaling policies for an Inference Endpoint. ## Usage ```bash theme={null} flexai inference scale [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `inference_endpoint_name` | string | Yes | The name of the Inference Endpoint to scale. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--max-replicas` | | integer | The maximum number of replicas to scale to. | | `--min-replicas` | | integer | The minimum number of replicas to maintain. | | `--verbose` | `-v` | boolean | Provides more detailed output when scaling an Inference Endpoint. | # inference serve Source: https://docs.flex.ai/cli/reference/inference/serve Create a new inference endpoint from a Hugging Face model Creates an Inference Endpoint from a model hosted by Hugging Face. The target model must be part of the [vLLM supported models list](https://docs.vllm.ai/en/latest/models/supported_models.html). ## Usage ```bash theme={null} flexai inference serve [ --accels --accel-sm-slices --affinity --api-key-secret --checkpoint --device-arch --hf-token-secret --max-replicas --min-replicas --no-queuing --runtime ] (-- --model= []) ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `inference_endpoint_name` | string | Yes | The name of the Inference Endpoint to create. It must be unique within the organization and must follow the [Resource Naming Conventions](/best-practices/resource-naming-conventions/). | | `model_name` | string | Yes | The name of the model to use for the Inference Endpoint. Visit the [vLLM supported models list](https://docs.vllm.ai/en/latest/models/supported_models.html) to see the list of supported models. | | `vllm_args` | option-list | No | [vLLM Engine Arguments](https://docs.vllm.ai/en/latest/models/supported_models.html) that can be passed after the End-of-options marker (`--`). Note: The `--device` argument is not supported: FlexAI handles the device selection tasks. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--accel-sm-slices` | `-s` | integer | Number of slices to divide each SM into. | | `--accels` | `-a` | integer | Number of accelerators/GPUs to use. Default: `1` | | `--affinity` | | key-value | Pins the Inference Endpoint to a specific cluster. Format: `cluster=`. Use [`flexai cluster list`](/cli/reference/cluster/list) to see clusters available to your organization. Useful when you need a specific accelerator type (for example, NVIDIA H100 vs A100) that lives on a particular cluster. The only recognized key today is `cluster`. When set, this overrides `--device-arch`: the cluster's hardware determines the accelerator architecture. Default: `[]` | | `--api-key-secret` | `-K` | string | The name of a [FlexAI Secret](/platform-services/secret-manager) containing the API key you want to set to protect the Inference Endpoint. If not provided: A FlexAI Secret will be created. Its value will be prompted ─ only once ─ after the creation of the Inference Endpoint. A new secret with the name `-api-key` containing the auto-generated API key will be created. | | `--checkpoint` | `-C` | string | A Checkpoint to serve the Inference Endpoint from. The name of a previously pushed Checkpoint (use [`flexai checkpoint list`](/cli/reference/checkpoint/list) to see available Checkpoints) or the UUID of an [*Inference Ready* Checkpoint](/platform-services/checkpoint-manager/inference-ready-checkpoints/) generated during the execution of a Training or Fine-tuning job (use [`flexai training checkpoints`](/cli/reference/training/checkpoints) to see available Checkpoints). | | `--device-arch` | `-d` | string | The architecture of the device to run the Inference Endpoint on. Default: `nvidia` | | `--help` | `-h` | boolean | Displays this help page. | | `--hf-token-secret` | `-T` | string | The name of the FlexAI Secret containing the Hugging Face token that will be used to access the model. | | `--max-replicas` | | integer | The maximum number of replicas to use for the Inference Endpoint. Visit the [FlexAI Inference Autoscaling](/core-services/inference/autoscaling) page to learn more about how autoscaling works and how to configure it. | | `--min-replicas` | | integer | The minimum number of replicas to use for the Inference Endpoint. Visit the [FlexAI Inference Autoscaling](/core-services/inference/autoscaling) page to learn more about how autoscaling works and how to configure it. | | `--no-queuing` | | boolean | Disable queuing for the Inference Endpoint. This means that if there are not enough resources available in the cluster, the request will be rejected immediately instead of being queued. | | `--runtime` | `-r` | string | The name of the runtime to use for the Inference Endpoint. If not provided, the default runtime set for the organization will be used. | | `--verbose` | `-v` | boolean | Provides more detailed output when initiating an Inference Serving operation. | ## Examples ### Gated Models Keep in mind that some models are "Gated", meaning that you need to go through a process of agreeing to their license agreement, privacy policy, or similar before you can use them. You can visit the model's page on the Hugging Face Hub to see if it is marked as "Gated". Gated models can be identified by a special indicator symbol. If the model is "Gated", you will find the necessary information on how to proceed. Mistral-7B-v0.1 model page on Hugging Face Hub: Before being granted access If you have already gone through the process, you will find a badge on the model's page indicating that you have access to the model. Mistral-7B-v0.1 model page on Hugging Face Hub: After being granted access Make sure you go through the process using **the same Hugging Face account** that owns the Access Token stored in the Secret passed to the `--hf-token-secret` flag. Learn more about deploying an Inference Endpoint from a Private or Gated model in the [Creating an Inference Endpoint: Private Model](/core-services/inference/quickstart/create-private/) quickstart guide. # inference stop Source: https://docs.flex.ai/cli/reference/inference/stop Stop a running inference endpoint Stops an Inference Endpoint. ## Usage ```bash theme={null} flexai inference stop [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `inference_endpoint_name` | string | Yes | The name of the Inference Endpoint to stop. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Provides more detailed output when stopping an Inference Endpoint. | # Command: secret Source: https://docs.flex.ai/cli/reference/secret Manage encrypted secrets for training jobs and inference endpoints The `flexai secret` command allows for the management of Secrets using the FlexAI Secret Manager. Secrets are encrypted at rest and are only made available to Workload Runtimes as system *environment variables*. Any number of Secrets can be passed on to a Training Job using the `-S`/`--secret` flag: ```bash {4,5} theme={null} flexai training run test-training-123 \ --dataset open_web \ --repository-url https://github.com/flexaihq/nanoGPT/ \ --secret HF_TOKEN=hf_token_dev \ --secret WANDB_API_KEY=wandb-key \ -- train.py ... ``` The same way Secrets can be passed to when creating an Inference Endpoint using the `--hf-token-secret` or `--api-key-secret` flags: ```bash {2,3} theme={null} flexai inference serve llm-text-inference-prod \ --hf-token-secrethf_token_prod \ --api-key-secret api-key-prod \ -- --model=mistralai/Mistral-7B-Instruct-v0.1 ... ``` Commands that leverage the FlexAI Secret Manager include: * [`flexai training run`](/cli/reference/training/run) - Starts a new Training Job. * [`flexai inference serve`](/cli/reference/inference/serve) - Creates an Inference Endpoint. * [`flexai storage create`](/cli/reference/storage/create) - Creates a new Remote Storage Provider Connection. ## Available subcommands * [`flexai secret create`](/cli/reference/secret/create) - Creates and stores a new secret in the FlexAI Secret Manager. * [`flexai secret delete`](/cli/reference/secret/delete) - Deletes an existing secret from the FlexAI Secret Manager. * [`flexai secret list`](/cli/reference/secret/list) - Lists all the secrets that have been created. # secret create Source: https://docs.flex.ai/cli/reference/secret/create Create a new secret in the FlexAI Secret Manager Creates and stores a new Secret using the FlexAI Secret Manager. ## Usage ```bash theme={null} flexai secret create [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `secret_name` | string | Yes | The name of the FlexAI Secret to create. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--value-stdin` | `-p` | boolean | Take the Secret's value directly from the Standard input (stdin) instead of the default interactive prompt. Especially useful for storing secrets from environment variables or files. | | `--verbose` | `-v` | boolean | Provides more detailed output when creating a Secret. | ## Examples ### Interactive prompt (Default) Writing in or pasting a value using the interactive prompt: ```bash theme={null} flexai secret create hf-token-dev ✔ Secret Value: █ ``` ### Reading from stdin Setting a Secret's value by *piping in* from the stdout of another program: ```bash theme={null} cat ~/Downloads/gcp-service-account.json | flexai secret create -p secret-from-sa-file ``` # secret delete Source: https://docs.flex.ai/cli/reference/secret/delete Delete a secret from the FlexAI Secret Manager Deletes an existing Secret from the FlexAI Secret Manager. ## Usage ```bash theme={null} flexai secret delete [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `secret_name` | string | Yes | The name of the FlexAI Secret to delete. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Provides more detailed output when deleting a Secret. | # secret list Source: https://docs.flex.ai/cli/reference/secret/list List all secrets stored in the FlexAI Secret Manager Lists all the Secrets available in the FlexAI Secret Manager. ## Usage ```bash theme={null} flexai secret list [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Displays secret information in JSON format instead of the default ASCII table. | | `--verbose` | `-v` | boolean | Provides more detailed output when listing Secrets. | # secret update Source: https://docs.flex.ai/cli/reference/secret/update Update the value of an existing secret Updates an existing Secret's value ## Usage ```bash theme={null} flexai secret update [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `secret_name` | string | Yes | The name of the FlexAI Secret to update. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--value-stdin` | `-p` | boolean | Take the secret value from stdin | | `--verbose` | `-v` | boolean | Enable verbose logging | # Command: storage Source: https://docs.flex.ai/cli/reference/storage Manage remote storage connections for artifact transfer The `flexai storage` command manages the connection between FlexAI and Remote Storage Providers. A Remote Storage Provider is a cloud storage service that allows you to store and retrieve data over the internet. Having the ability to move artifacts between a Remote Storage Provider and FlexAI is particularly useful when working with large datasets, model checkpoints, and other artifacts that may not fit on your local machine or need to be shared across different environments. ## Supported Remote Storage Providers * `s3`: Amazon S3 * `gcs`: Google Cloud Storage * `r2`: Cloudflare R2 * `minio`: MinIO * `huggingface`: Hugging Face Hub ## Available Commands * [`flexai storage create`](/cli/reference/storage/create) * [`flexai storage inspect`](/cli/reference/storage/inspect) * [`flexai storage list`](/cli/reference/storage/list) # storage create Source: https://docs.flex.ai/cli/reference/storage/create Create a new remote storage provider connection Creates a new Remote Storage Provider Connection. ## Usage ```bash theme={null} flexai storage create [( --provider --access-key-id --endpoint --secret-access-key-name )] [( --provider gcs --service-account-file-name )] [( --provider s3 --region )] [( --provider huggingface --hf-token-name )] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `storage_provider_connection_name` | string | Yes | The name that will be used to identify the Storage Provider Connection while performing operations on it. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--provider` | `-p` | string | Storage provider to use. One of: `s3`, `gcs`, `huggingface`, `minio`, `r2`. | | `--access-key-id` | | string | S3-compatible storage Access Key ID. | | `--endpoint` | | string | S3-compatible storage endpoint. Only required when using the `s3` provider. | | `--region` | | string | S3-compatible storage Region where the target Storage Bucket is located. Only required when using the `s3` provider. | | `--secret-access-key-name` | | string | Name of the FlexAI Secret containing the S3-compatible storage Secret Access Key. | | `--service-account-file-name` | | string | Name of the FlexAI Secret containing the GCP Service Account JSON file (for `gcs` provider). | | `--hf-token-name` | | string | Name of the FlexAI Secret containing the Hugging Face Access token (for `huggingface` provider). | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Provides more detailed output. | # storage inspect Source: https://docs.flex.ai/cli/reference/storage/inspect View detailed information about a remote storage connection Displays detailed information about a Remote Storage Provider Connection. ## Usage ```bash theme={null} flexai storage inspect [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `storage_provider_name` | string | Yes | The name of the Storage Provider Connection to inspect. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Output in JSON format instead of the default ASCII table. | | `--verbose` | `-v` | boolean | Provides more detailed output. | # storage list Source: https://docs.flex.ai/cli/reference/storage/list List all remote storage connections Lists all Remote Storage Provider Connections. ## Usage ```bash theme={null} flexai storage list [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Displays storage information in JSON format instead of the default ASCII table. | | `--verbose` | `-v` | boolean | Provides more detailed output when listing Remote Storage Provider Connections. | # Command: training Source: https://docs.flex.ai/cli/reference/training Manage training jobs on FlexAI — run, monitor, inspect, and delete jobs The `flexai training` command manages Training Jobs: from starting a new Training Job, retrieving its logs, and inspecting its execution, to downloading its output artifacts, among other tasks. ## Available subcommands * [`flexai training checkpoints`](/cli/reference/training/checkpoints) - Lists out the checkpoints that were generated by a Training Job. * [`flexai training debug-ssh`](/cli/reference/training/debug-ssh) - Establishes an SSH connection to a running Training Job. * [`flexai training delete`](/cli/reference/training/delete) - Deletes a Training Job. * [`flexai training fetch`](/cli/reference/training/fetch) - Fetches artifacts from a Training Job. * [`flexai training inspect`](/cli/reference/training/inspect) - Displays detailed information about a Training Job. * [`flexai training list`](/cli/reference/training/list) - Lists all the Training Jobs. * [`flexai training logs`](/cli/reference/training/logs) - Displays the logs from a Training Job. * [`flexai training run`](/cli/reference/training/run) - Starts a new Training Job. * [`flexai training stop`](/cli/reference/training/stop) - Stops a Training Job. ## Training Lifecycle Statuses A Training Job will go through a subset of the following statuses during its lifecycle: | Status | Description | Is Terminal | | - | - | - | | `pending` | The initial status of a Training Job. It means the Training Job was successfully received by FlexAI and is waiting to be processed | false | | `scheduling` | FlexAI is looking for a suitable Node to handle the training given its hardware requirements | false | | `rejected` | A Node that meets the requirements specified for the Training Job could not be found or is unavailable at the moment | true | | `building` | A Node suitable for the Training Job was found. The building process has started. FlexAI is gathering all the components required for the training, in particular, cloning the revision specified for the source repository and installing the required dependencies | false | | `in progress` | The building process completed successfully. The required FlexAI compute resources are being allocated | false | | `enqueued` | The required compute resources specified for the Training Job were not available at the time. The Training Job is temporarily put on hold and will be scheduled to start once the required resources are freed up | false | | `succeeded` | The Training Job completed successfully. The entry point training script terminated with exit code 0. Output artifacts can be downloaded using the `training fetch` command | true | | `failed` | A Training Job can fail because one of the following reasons:
  • The `--repository-revision` could not be found.
  • The `requirements.txt` file could not be found in the root directory of the repository.
  • The Training Job failed. The *Entry point* training script terminated with an exit code above 0.
  • The Training Job's duration exceeded the time limit (`BackoffLimitExceeded`).
| true | | `stop in progress` | The Training Job stopping process initiation was requested by the user and it is being performed | false | | `stopped` | The Training Job was successfully stopped. If the Training Job was stopped while is status was `scheduling`, `building`, or `enqueued`, then no GPU resources were allocated

If it was stopped after the training status changed to `building`, then the hardware resources were allocated and eventually released after the Training Job was successfully stopped | true | | `stop failed` | The process of stopping a the Training Job failed | true | A Training or Fine-tuning Job in a *"terminal"* status can be deleted using the [`flexai training delete`](/cli/reference/training/delete) command. # training checkpoints Source: https://docs.flex.ai/cli/reference/training/checkpoints List checkpoints generated by a training job Lists out the Checkpoints that have been generated for a Training Job. ## Usage ```bash theme={null} flexai training checkpoints [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `training_or_fine_tuning_job_name` | string | Yes | The name of the Training or Fine-tuning Job to list checkpoints for. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Outputs checkpoint list in JSON format instead of the default ASCII table. | | `--verbose` | `-v` | boolean | Provides more detailed output when listing checkpoints. | # training debug-ssh Source: https://docs.flex.ai/cli/reference/training/debug-ssh Start an interactive SSH session into a training job's runtime Starts an Interactive Training Job that allows connecting through SSH or VSCode to the Training Runtime, useful for fast test iterations. The `--vscode` flag is optional, but highly recommended to leverage the full potential of the Interactive Training Runtime. ## Usage ```bash theme={null} flexai training debug-ssh \ --repository-url \ [--dataset ...] \ [--repository-revision ] \ [--checkpoint ] \ [--env =...] \ [--secret =...] \ [--nodes ] \ [--accels ] \ [--device-arch ] \ [--git-author-name ] \ [--git-author-email ] \ [--dotfiles ] \ [--authorized-keys ] \ [--session-timeout ] \ [--vscode] ``` ## Flags | Flag | Short | Type | Default | Description | | - | - | - | - | - | | `--accels` | `-a` | integer | `1` | Number of accelerators/GPUs to use. | | `--affinity` | | key=value | | Pins the workload to a specific cluster. Format: `cluster=`. Use [`flexai cluster list`](/cli/reference/cluster/list) to see clusters available to your organization. Useful when you need a specific accelerator type (for example, NVIDIA H100 vs A100). The only recognized key today is `cluster`. When set, this takes precedence over the cluster FlexAI would have selected from `--device-arch`. | | `--authorized-keys` | | string | ssh-agent keys | List of SSH public keys to allow connecting to the interactive environment. If not provided, keys will be gathered from the local ssh-agent, if available. | | `--build-secret` | | key=value | | FlexAI Secrets to make available during the image build process. Format: `=` | | `--checkpoint` | `-C` | string | | A Checkpoint to mount on the runtime environment. Can be a name or UUID. | | `--dataset` | `-D` | string | | Dataset to mount on the runtime environment. Can be repeated for multiple datasets. Optionally specify mount path with `name=path` format. | | `--device-arch` | `-d` | string | `nvidia` | The architecture of the device to run on. | | `--dotfiles` | | string | | GitHub dotfiles repository URL that will be installed in the home directory of the interactive environment. | | `--env` | `-E` | key=value | | Environment variables to set in the interactive environment. Can be repeated. | | `--git-author-email` | | string | git config | The Git commit author email to use in the interactive training environment. | | `--git-author-name` | | string | git config | The Git commit author name to use in the interactive training environment. | | `--help` | `-h` | boolean | | Displays this help page. | | `--no-queuing` | | boolean | | Disables queuing for this workload: If no resources are available, the workload will fail immediately. | | `--nodes` | `-n` | integer | `1` | The number of nodes across which to distribute the workload. Selecting more than 1 node will set 8 accelerators per node. | | `--repository-revision` | `-b` | string | `main` | The branch name, commit SHA, or tag of the repository to use. | | `--repository-url` | `-u` | string | | Git repository URL containing code to mount on the workload environment. Will be mounted on `/workspace`. | | `--requirements-path` | `-q` | string | | Path to a pip requirements.txt file in the repository. | | `--runtime` | `-r` | string | | Name of the runtime to use. | | `--secret` | `-S` | key=value | | Environment variables set from FlexAI Secrets. Format: `=`. Can be repeated. | | `--session-timeout` | | integer | `600` | Timeout in seconds after which the interactive session will be stopped if no activity is detected. | | `--verbose` | `-v` | boolean | | Provides more detailed output when running a debug-ssh session. | | `--vscode` | | boolean | | Opens Visual Studio Code connected to the runtime environment via SSH. | # training delete Source: https://docs.flex.ai/cli/reference/training/delete Delete a training job and its associated resources Deletes a Training Job. ## Usage ```bash theme={null} flexai training delete [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Provides more detailed output when deleting a Training Job. | # training fetch Source: https://docs.flex.ai/cli/reference/training/fetch Download output files from a completed training job Fetches artifacts from a Training Job. The artifacts are stored in the `/output` directory of the Training Runtime environment and can be accessed even after the Training Job has completed. ## Usage ```bash theme={null} flexai training fetch [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `training_or_fine_tuning_job_name` | string | Yes | The name of the Training or Fine-tuning Job to fetch artifacts from. | ## Flags | Flag | Short | Type | Default | Description | | - | - | - | - | - | | `--destination` | `-d` | string | `./` | Destination directory to store fetched artifacts. | | `--help` | `-h` | boolean | | Displays this help page. | | `--verbose` | `-v` | boolean | | Provides more detailed output when fetching artifacts. | # training inspect Source: https://docs.flex.ai/cli/reference/training/inspect View detailed information about a specific training job Displays detailed information about a Training Job, including its name, ID, creator, owner, configuration, status, and execution details. ## Usage ```bash theme={null} flexai training inspect [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `training_or_fine_tuning_job_name` | string | Yes | The name of the Training or Fine-tuning Job to inspect. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Displays training information in JSON format. | | `--verbose` | `-v` | boolean | Provides more detailed output when inspecting a Training Job. | # training list Source: https://docs.flex.ai/cli/reference/training/list List all training jobs and their current status Lists all the Training Jobs that have been created. The output includes the name of the job, its status, age, and other relevant information. ## Usage ```bash theme={null} flexai training list [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Displays training information in JSON format instead of the default ASCII table. | | `--verbose` | `-v` | boolean | Provides more detailed output when listing a Training. | ## Examples ### List all training jobs flexai training list ```bash theme={null} flexai training list ``` ### List training jobs in JSON format flexai training list --json ```bash theme={null} flexai training list --json ``` # training logs Source: https://docs.flex.ai/cli/reference/training/logs Stream or retrieve logs from a running or completed training job Displays the logs from a Training Job. The logs include information about the job's execution, including any output from the training script. ## Usage ```bash theme={null} flexai training logs [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `training_or_fine_tuning_job_name` | string | Yes | The name of the Training Job to view logs for. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--no-color` | | boolean | Disables color formatting in the log output. | | `--verbose` | `-v` | boolean | Provides more detailed output when viewing logs. | # training run Source: https://docs.flex.ai/cli/reference/training/run Start a new training job on FlexAI with custom configuration Starts a new Training Job. This command allows you to specify the dataset, repository, hardware requirements, and other parameters for the job. ## Usage ```bash theme={null} flexai training run [flags] -- [script_args] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `training_or_fine_tuning_job_name` | string | Yes | A unique name for the Training Job. | | `entry_point_script_path` | string | Yes | The path to the entry point script for the Training or Fine-tuning Job. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--accels` | `-a` | integer | Number of accelerators/GPUs to use. Default: `1` | | `--affinity` | | key-value | Pins the workload to a specific cluster. Format: `cluster=`. Use [`flexai cluster list`](/cli/reference/cluster/list) to see clusters available to your organization. Useful when you need a specific accelerator type (for example, NVIDIA H100 vs A100) that lives on a particular cluster. The only recognized key today is `cluster`. When set, this takes precedence over the cluster FlexAI would have selected from `--device-arch`. | | `--build-secret` | | key-value | FlexAI Secrets to make available during the image build process. Format: ``=`` | | `--checkpoint` | `-C` | string | A Checkpoint to use as a starting point for a Fine-tuning Job. The name of a previously pushed Checkpoint (use [`flexai checkpoint list`](/cli/reference/checkpoint/list) to see available Checkpoints) or the UUID of an [*Inference Ready Checkpoint*](/platform-services/checkpoint-manager/inference-ready-checkpoints/) generated during the execution of a Training or Fine-tuning job (use [`flexai training checkpoints`](/cli/reference/training/checkpoints) to see available Checkpoints). | | `--dataset` | `-D` | string | Dataset to mount on the runtime environment. Can be specified as a simple dataset name, or as `=` to use a custom mount path under `/input`. | | `--device-arch` | `-d` | string | The architecture of the device to run the Inference Endpoint on. Default: `nvidia` | | `--env` | `-E` | key-value | Environment variables to set in the interactive environment. | | `--help` | `-h` | boolean | Displays this help page. | | `--no-queuing` | | boolean | Disables queuing for this workload: If no resources are available, the workload will fail immediately instead of waiting for resources to become available. | | `--nodes` | `-n` | integer | The number of nodes across which to distribute the workload. Selecting more than 1 node will overwrite the value provided in the `--accels` flag to 8 accelerator per node. Default: `1` | | `--repository-revision` | `-b` | string | The branch name of the repository (default: `main`), a commit SHA hash, or a tag name. | | `--repository-url` | `-u` | string | Git repository URL containing code to mount on the workload environment. Will be mounted on the `/workspace` directory. | | `--requirements-path` | `-q` | string | Path to a pip requirements.txt file in the repository. | | `--runtime` | `-r` | string | Name of the runtime to use | | `--secret` | `-S` | key-value | Environment variables that will be set in the Training Runtime. Secrets are sensitive values like API keys, tokens, or credentials that need to be accessed by your Training Job but should not be exposed in logs or command history. When using the --secret flag, the actual secret values are retrieved from the Secrets Storage and injected into the environment at runtime. Syntax: ``=`` where `` is the name of the environment variable to set, and `` is the name of the Secret containing the sensitive value. | | `--verbose` | `-v` | boolean | Enables verbose logging for detailed output during training job execution. | ## Examples ### Start a training job with dataset and repository ```bash theme={null} flexai training run gpt2training-1 --dataset my_data=wikitext-2-raw-v1 --repository-url https://github.com/flexaihq/nanoGPT/ --accels 4 --secret HF_TOKEN=hf-token-dev --env BATCH_SIZE=32 -- train.py --batch-size 32 --epochs 10 ``` ### Pin a training job to a specific cluster Use `--affinity cluster=` to target a specific cluster — for example, when you need a particular accelerator type. Run [`flexai cluster list`](/cli/reference/cluster/list) to discover the cluster names available to your organization. ```bash theme={null} flexai training run gpt2training-1 \ --repository-url https://github.com/flexaihq/nanoGPT/ \ --affinity cluster= \ --accels 4 -- train.py ``` # training runtime Source: https://docs.flex.ai/cli/reference/training/runtime Manage training job runtimes Manage training runtimes. ## Usage ```bash theme={null} flexai training runtime [command] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Enable verbose logging. | # training runtime list Source: https://docs.flex.ai/cli/reference/training/runtime/list List available training runtimes and their configurations List training runtimes. ## Usage ```bash theme={null} flexai training runtime list [flags] ``` ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--json` | `-j` | boolean | Output in JSON format (default ASCII table). | | `--verbose` | `-v` | boolean | Enable verbose logging. | # training stop Source: https://docs.flex.ai/cli/reference/training/stop Stop a running training job Stops a running Training Job. This will terminate the job and save any output that has been generated so far. ## Usage ```bash theme={null} flexai training stop [flags] ``` ## Arguments | Argument | Type | Required | Description | | - | - | - | - | | `training_or_fine_tuning_job_name` | string | Yes | The name of the Training or Fine-tuning Job to stop. | ## Flags | Flag | Short | Type | Description | | - | - | - | - | | `--help` | `-h` | boolean | Displays this help page. | | `--verbose` | `-v` | boolean | Provides more detailed output when stopping a Training Job. | # FlexAI Web Console Source: https://docs.flex.ai/console Manage training, fine-tuning, and inference workloads from the FlexAI Web Console The FlexAI Web Console is a powerful tool designed to streamline the management of your AI workloads through a web-based interface. It provides a user-friendly interface managing your training, fine-tuning, and inference workloads, as well as your FlexAI resources, all in a single place. With the Web Console, you can easily create and manage your AI workloads, monitor their progress, and access detailed logs and metrics. It allows you to configure your Training Jobs, set up Inference Endpoints, and manage your datasets and models without having to heavily rely on the command line. You can sign in to the FlexAI Web Console by visiting the [FlexAI Web Console's Sign In page](https://platform.flex.ai/login). ## Getting Started ### Console Access & Authentication * [Access the FlexAI Console](https://platform.flex.ai) - Sign in with your FlexAI account * [Account Sign Up](/console/getting-started/signup/) - Learn about authentication and account setup ## Core Services ### Model Inference Deploy and manage Inference Endpoints for your models. * [Deploying an Inference Endpoint ](https://platform.flex.ai/s/deployed-models/new-deployed-model) - Create a new Inference Endpoint for your model * [FlexAI Inference](/core-services/inference/) Overview * Quickstart guide: [Deploying a Public Model](/core-services/inference/quickstart/create-public/) * Quickstart guide: [Deploying a Private Model](/core-services/inference/quickstart/create-private/) ### Model Fine-tuning Fine-tune models using the FlexAI Console. * [Launching a Fine-tuning Job](https://platform.flex.ai/s/fine-tunings/new-fine-tuning) - Start a new Fine-tuning Job from a base model or FlexAI Checkpoint * [FlexAI Fine-tuning](/core-services/fine-tuning/) Overview * Quickstart guide: [Fine-tuning Quickstart](/core-services/fine-tuning/quickstart) ### Model Training Train models using the FlexAI Console. * [Launching a Training Job](https://platform.flex.ai/s/trainings/new-training) - Start a new Training Job using your Dataset and training scripts * [FlexAI Training](/core-services/training/) Overview * Quickstart guide: [Training Quickstart](/core-services/training/quickstart) ### Platform Services Manage checkpoints, secrets, datasets, and storage. * [Checkpoint Manager](/platform-services/checkpoint-manager/) - A data store for checkpoints produced in training runs * [Code Registry Manager](/platform-services/code-registry-manager/) - Manage code associated to each FlexAI tenant * [Dataset Manager](/platform-services/dataset-manager/) - Manage datasets * [Observability](/platform-services/observability/) - Monitor training runs * [Remote Storage Connection Manager](/platform-services/remote-storage-connections-manager/) - Add remote storage connections * [Secret Manager](/platform-services/secret-manager/) - Manage secrets and credentials # Sign In Source: https://docs.flex.ai/console/getting-started/signin Sign in to the FlexAI Web Console with email or GitHub You can sign in to the FlexAI Web Console by visiting the [FlexAI Web Console's Sign In page](https://platform.flex.ai/login). The authentication page will provide two options to sign in: 1. Sign in with Email and Password: If you prefer, you can sign in using an email address. You'll need to provide some information to get started. 2. Sign in with GitHub: If you have a GitHub account, you can sign in by authorizing FlexAI as an OAuth application. Screenshot of the 'Sign in with Console' page ## Sign In with Email and Password If you signed up to FlexAI with *email and password*, you'll need to provide the **Work Email** and **Password** combination you used during sign up. Once you enter your credentials, select the **Sign in with password** button. You'll be redirected to the [FlexAI Web Console](https://platform.flex.ai/). If you ever forget your password, you can reset it by [visiting this page](https://platform.flex.ai/recovery) or selecting the **Forgot password?** link on the [sign-in page](https://platform.flex.ai/login). You'll be prompted to enter your email address. You will then receive an email with instructions to reset your password. ## Sign In with GitHub If you had signed up to FlexAI with GitHub, then select the "Sign in with GitHub" button. ### If you are not logged into your GitHub account If you're not already logged in, you'll be redirected to the GitHub login page. Follow the regular GitHub login process to log in to your account. This may involve entering your username and password, or using a 2FA authentication method if you have enabled it. Once the FlexAI authentication request completes successfully, you'll be redirected to the [FlexAI Web Console](https://platform.flex.ai/). If you encounter any issues during the sign in process, do not hesitate to reach out to our support team for assistance at [support@flex.ai](mailto:support@flex.ai). # Sign Up for FlexAI Source: https://docs.flex.ai/console/getting-started/signup Create a new FlexAI account to get started You can sign up for FlexAI by visiting the [FlexAI Web Console's Sign Up page](https://platform.flex.ai/signup). Then follow the instructions below to create your account. Here you will get two options to sign up: 1. Sign up with Email and Password: If you prefer, you can sign up using an email address. You'll need to provide some information to get started. 2. Sign up with GitHub: If you have a GitHub account, you can sign up by authorizing FlexAI as an OAuth application. ## Sign Up with Email and Password If you prefer to sign up with email and password, you'll need to provide: * **Work Email**: Your work email address. Must be a valid email address using your company's domain. For example, `george@example-company.com` * **Password**: A strong password for your account. * **First Name**: *Given* name. * **Last Name**: *Family* name. * **Company Name**: The name of your company or organization. * **Phone Number**: An contact phone number. Optional. FlexAI Sign Up Page The email address you provide must be a valid work email address using your company's domain. Personal email addresses (e.g., @gmail.com, @aol.com, @wanadoo.fr, @uol.com.br, etc.) are not accepted. After filling out the form, check the *I accept the Terms of Service* box to agree to the terms, and finally select the **Sign up** button. Once you submit the form, you'll get a confirmation message indicating that a **verification code** has been sent to your email address. Check your inbox for the verification code. Enter the **verification code** in the provided field on the sign-up page or follow the link in the email. If you decided to manually enter the code, select the **Continue** button to complete the sign-up process. If you don't receive the verification code after a couple of minutes, you can request a new one by selecting the **Resend Code** button. Once you enter the verification code, you'll be redirected to the [FlexAI Web Console](https://platform.flex.ai/), where you can start managing your AI workloads. ## Sign Up with GitHub If you choose to sign up with GitHub but you're not already logged in, you'll be redirected to the GitHub login page. ### If you are not logged into your GitHub account Follow the regular GitHub login process to log in to your account. This may involve entering your username and password, or using a two-factor authentication method if you have it enabled. Once logged in, you'll be redirected back to the FlexAI authentication authorization page. ### FlexAI Authentication Authorization page Here, you can review the permissions that FlexAI is requesting. Select the **Authorize flexaihq** button. Depending on your GitHub account's settings, you may need to go through an additional GitHub account verification step. Once the FlexAI authentication request completes successfully, you'll be redirected to the [FlexAI Web Console](https://platform.flex.ai/). If you encounter any issues during the sign up process, do not hesitate to reach out to our support team for assistance at [support@flex.ai](mailto:support@flex.ai). # Fine-Tuning Source: https://docs.flex.ai/core-services/fine-tuning Fine-tune AI models with your own data using FlexAI Fine-tune pre-trained AI models with your own data to create custom models tailored to your specific use cases. You can fine-tune public and private models hosted on the Hugging Face Model Hub or models you've previously trained with FlexAI. ## Key Features Bring your own Custom Model tailored to your needs Launch fine-tuning jobs from an existing model or a FlexAI Checkpoint Automatic checkpoint management Real-time training progress tracking Support for various Datasets and any kind of file formats Pull *base models* and Datasets from third-party sources such as Hugging Face Pick from a single GPU to multiple GPUs running on different Nodes ## Anatomy of a Fine-tuning Job A Fine-tuning Job in FlexAI consists of the following components: 1. **Base Model Checkpoint**: The base model Checkpoint you want to use to fine-tune your new model. 2. **Fine-tuning Script**: The script that defines the fine-tuning process, including data loading, model training, evaluation, Checkpoint generation, and so on. 3. **Dataset**: The Dataset you will use for to Fine-tune your new model. * It can be an existing [FlexAI Dataset](/platform-services/dataset-manager/), or it can be pulled/streamed from an external source during runtime. 4. [**Secrets**](/platform-services/secret-manager/): Any sensitive information (e.g., API keys, passwords) required for the fine-tuning process. 5. **Hyperparameters**: Configuration settings that control the training process (e.g., learning rate, batch size). ### Base Model Checkpoint A Fine-tuning Job by definition takes a base model as its starting point. It can be a base model you pull from an external source such as Hugging Face, or it can be a [FlexAI Checkpoint](/platform-services/checkpoint-manager/) from a previous Training or Fine-tuning Job. A Fine-tuning Job can also generate multiple Checkpoints during its execution, which you can use to go back in time and pick the one that performs the best for your use case. ### Fine-tuning Script The fine-tuning script is a Python script that defines the fine-tuning process. It handles task such as: * Loading the base model Checkpoint from the `/input-checkpoint/` path, or downloading it from an external source. * Loading a Dataset mounted through the FlexAI Dataset Manager on the `/input/` path or from an external source. * Performing any necessary data processing steps. * Reading the values of any Secrets passed to the Fine-tuning Job's runtime environment. * Applying any hyperparameters configurations passed to the Fine-tuning Script. * Running the training loop and periodically saving Checkpoints to `/output-checkpoints/` path. ### Dataset A Fine-tuning Job generally leverages a Dataset to adjust a pre-trained model's weights using new data. The Dataset can be: * An existing [FlexAI Dataset](/platform-services/dataset-manager/). * Pulled/streamed from an external source during runtime. The Dataset can be in any format, and it can contain any kind of files. The Fine-tuning Script is responsible for loading and processing the data as needed. ### Secrets Secrets are used to securely pass sensitive information to the Fine-tuning Job's runtime environment. This can include API keys, passwords, or any other confidential data required for the fine-tuning process. Secrets are managed through the [FlexAI Secret Manager](/platform-services/secret-manager/) and are injected into the Fine-tuning Job's environment as environment variables. ### Hyperparameters Hyperparameters are configuration settings that control the behavior of the Fine-tuning Script. They can include parameters such as learning rate, batch size, number of epochs, where to load the dataset from, where to save the model checkpoints, how often to save checkpoints, and so on. ## Key Concepts ### Third-Party Integration Your FlexAI Fine-tuning Jobs can interact with third-party APIs and services by leveraging the [FlexAI Secret Manager](/platform-services/secret-manager/) to securely store sensitive information such as API keys and tokens. These secrets can then be injected into the Fine-tuning Job's runtime environment as environment variables, allowing your Fine-tuning scripts to access them securely and enabling flexible and powerful AI workflows. ### A Fine-tuning Job's Lifecycle Learn more about the different statuses a Fine-tuning Job can have in the [Fine-tuning Job Lifecycle](/core-services/fine-tuning/lifecycle/) page. ### CLI Reference The [`flexai training` command](/cli/reference/training) manages Training Jobs as well as Fine-tuning Jobs. These two types of workloads share the same command set. A Fine-tuning Job is a Training Job that begins its execution from an existing model Checkpoint. ## Getting Started FlexAI fine-tuning jobs can be launched in a few steps. The Quickstart guide will walk you through the process of preparing your data and starting your first Fine-tuning job. Here's a brief overview of the steps involved: 1. Preparing your dataset for fine-tuning. 2. Launching a fine-tuning job using FlexAI. 3. Monitoring training progress and managing checkpoints. The button below will lead you to the FlexAI Fine-Tuning Quickstart guide's overview, where you'll find more details. # Lifecycle Source: https://docs.flex.ai/core-services/fine-tuning/lifecycle Understand the status progression of a fine-tuning job from creation to completion A Fine-tuning Job progresses through several statuses: * **pending**: The workload has been registered and its creation process has begun. * **scheduling**: FlexAI is searching for the suitable hardware resources to run the workload. * **building**: The Runtime environment is being prepared, including cloning the source repository on the specified revision and installing the project's dependencies. * **in progress**: The workload is currently executing the code defined in the entry point file. * **enqueued**: The workload is waiting for resources to become available. * **succeeded**: The workload completed successfully and output artifacts are ready for download. * **failed**: The workload did not complete successfully. Common reasons include missing files, dependency errors, or exceeding time limits. * **stopped**: The workload was stopped by the user before completion. Terminal statuses (**succeeded**, **failed**, **stopped**) mean the job is finished and can be deleted if desired. ```mermaid theme={null} stateDiagram-v2 [*] --> pending pending --> scheduling scheduling --> building scheduling --> enqueued building --> in_progress: in progress building --> enqueued enqueued --> in_progress: in progress enqueued --> scheduling in_progress --> succeeded: succeeded [Terminal] in_progress --> failed: failed [Terminal] in_progress --> stopped: stopped [Terminal] succeeded --> [*] failed --> [*] stopped --> [*] ``` # Fine-tuning a model with FlexAI Source: https://docs.flex.ai/core-services/fine-tuning/quickstart Step-by-step guide to fine-tuning a model on FlexAI ## Overview This Quickstart tutorial will walk you through the steps needed to train a model on **FlexAI**. By the end of this tutorial, you will have successfully trained a version of [nanoGPT](https://github.com/karpathy/nanoGPT) that has been optimized to make it readily usable. This enhanced version is available at: [https://github.com/flexaihq/nanoGPT](https://github.com/flexaihq/nanoGPT). The process of Fine-tuning a model from a starting poiont (Base model or *Checkpoint*) a model consists of **4 main steps**: ### Prerequisites You should have a FlexAI account. If you don't have one, you can [sign up for a free account](/signup). ### 1. Loading a *Dataset* A **Dataset** is the collection of files that you want to use to train your model. You can upload files from your local machine or sync them in from a remote location when hosted by a third party Storage Provider (e.g., S3, GCS, R2, etc.). The Dataset files can be in any format, such as text, images, or audio. ### 2. Running a *Fine-tuning Job* **Fine-tuning Job** is the name assigned to the FlexAI component that represents the process of executing training code on the FlexAI platform. Creating a Fine-tuning Job requires the following 5 things: 1. **A Name** that describes your Fine-tuning Job 2. At least one **Dataset** that will be used to Fine-tune your model 3. A link to a GitHub **repository with the Fine-tuning code** 4. **A Checkpoint** name or ID, which to use to start the Fine-tuning Job from 5. The path to the *entry point script* (the file that contains the code that will be executed when the Fine-tuning Job begins) A number of optional flags allow for the customization of a Fine-tuning Job's execution, such as: * Specifying how many accelerators (GPUs) to use and across how many nodes. * Setting Environment Variables and Secrets that will then be passed to the Training Runtime so they can be used by the training scripts. * Specifying a previous *checkpoint* to resume execution from. * Setting a specific *revision* (branch, tag, or commit) of the code repository to use. Additionally, any number of **Hyperparameters** can be specified passed to the *entry point script*. These Hyperparameters can be used to control the behavior of the training code, such as the learning rate, batch size, or number of epochs. ### 3. Getting the Fine-tuning Job's details Once the Fine-tuning Job is running, you can monitor its progress, view logs, evaluate its performance and resource usage, all through the FlexAI CLI, FlexAI's hosted TensorBoard, and FlexAI's Dashboard UI ### 4. Fetching the Fine-tuning Job's output The output of a Fine-tuning Job is the result of the training process, which can include: * Checkpoints: Saved states of the model at different points during training. * Files written to the `/output` directory by the training scripts, such as binaries, logs, metrics, or any other files that you want to keep as a result of a successful Fine-tuning Job completion. # Checking the Fine-tuning Job's Details Source: https://docs.flex.ai/core-services/fine-tuning/quickstart/checking-details Inspect the details and status of your fine-tuning job ## Fine-tuning Summary ### General Details about your Training Jobs The "All Trainings" table in the FlexAI Console provides a summary of all your Training Jobs. You can select the gear icon ⚙️ (labeled as *Configure*) in the `Actions` field of the Training Jobs list page. This will open a "Details" panel. The **Details** tab will be selected by default, showing all the relevant information about your Training Job. | Field | Description | | - | - | | `Name` | The name you assigned to the Training Job. | | `Status` | The current status of the Training Job (e.g., `pending`, `scheduling`, `building`, `in progress`, `succeeded`, `failed`, `stopped`, etc.). | | `Created At` | Workload creation age. | You can learn more about the different Training Job statuses on the [Lifecycle](/core-services/training/lifecycle) page. *** ### General Details about your Training Jobs You can use the `list` command to get a table with general information about all the Training Jobs you have access to through your FlexAI account: ```bash theme={null} flexai training list ``` This provides an output similar to the following: ```text theme={null} NAME | DEVICE | NODE | ACCELERATOR | DATASET | REPOSITORY | STATUS | AGE ---------------------------+--------+------+-------------+-----------------+-------------------------------------+----------+------ quickstart-fine-tuning-job | nvidia | 1 | 1 | nanoGPT-dataset | https://github.com/flexaihq/nanogpt | building | 15s ``` *** ## Fine-tuning Configuration | Field | Description | | - | - | | `Dashboard URL` | The URL of the Training Job dashboard, where you can monitor the performance and resource usage of your Training Job. | | `Tensorboard Dashboard URL` | The URL of the FlexAI-hosted TensorBoard dashboard, where you can visualize the training process of your models. | | `Node Count` | The number of nodes allocated to the Training Job. | | `Accelerator Count` | The number of accelerators (GPUs) allocated to the Training Job. | | `Repository URL` | The URL of the Git repository containing your training code. | | `Repository Revision` | The specific commit or branch of the repository that was used to create the Training Job. | | `Repository Revision SHA` | The SHA hash of the specific commit or branch of the repository that was used to create the Training Job. | | `Entry Point` | The entry point script along with its arguments. | | `Datasets` | The datasets that were attached to the Training Job. | | `Environment` | The environment variables and secrets that were set for the Training Job. Displayed in a Key-Value pai format where the Key is the name of the environment value within the Training Runtime, and the value is either the raw value (for Environment Variables) or the name of the **FlexAI secret** containing the secret value. | | `Checkpoints` | The checkpoints that were created during the Training Job. These are stored in the FlexAI object storage and can be used to resume training or to create an Inference Endpoint (depending on the type of model). | *** You can have a deeper look at the Training Job status using the `flexai training inspect quickstart-fine-tuning-job` command. Especially useful for debugging purposes: ```bash theme={null} flexai training inspect quickstart-fine-tuning-job ``` Below you will find an example of the output you will get when running the `inspect` command: ```yaml theme={null} kind: Training metadata: name: quickstart-training-job id: 75179cc2-ec63-4f93-b4da-44e49ea86049 creatorUserID: 16e289cc-c81b-4a15-91d9-0e2aae00a317 ownerOrgId: 270a5476-b91a-442f-8a13-852ef7bb5b9c config: device: nvidia nodes: 1 accelerator: 1 entrypoint: - train.py - config/train_shakespeare_char.py - --out_dir=/output-checkpoint - --max_iters=1500 datasetsNames: - nanoGPT-dataset checkpointName: "" sourceName: "" repositoryURL: https://github.com/flexaihq/nanogpt repositoryRevision: main secrets: [] environment: [] runtime: status: succeeded queuePosition: 0 repositoryRevisionSha: 116799dbae7b0fe33caf1b90f73a72f84bc32adc selectedAgentId: k8s-training-sesterce-001-CLIENT-PROD-client-prod lifecycleEvents: - type: AgentSelection status: ResponseReceived message: |- Cluster Scheduling result{ Name: aws-cloud AgentID: k8s-training-aws-001-CLIENT-PROD-client-prod Response: NoAnswer Conditions: [NonSchedulable: NoAnswer] } raisedAt: "2025-06-30T11:41:54Z" - type: AgentSelection status: ResponseReceived message: |- Cluster Scheduling result{ Name: sesterce-h100-bm-01 AgentID: k8s-training-sesterce-001-CLIENT-PROD-client-prod Response: OK Conditions: [] } raisedAt: "2025-06-30T11:41:54Z" - type: AgentSelection status: ResponseReceived message: |- Cluster Scheduling result{ Name: sesterce-h200-bm-01 AgentID: k8s-training-sesterce-002-CLIENT-PROD-client-prod Response: NoAnswer Conditions: [NonSchedulable: NoAnswer] } raisedAt: "2025-06-30T11:41:54Z" - type: AgentSelection status: ResponseReceived message: |- Cluster Scheduling result{ Name: sesterce-l40s-bm-01 AgentID: k8s-training-sesterce-003-CLIENT-PROD-client-prod Response: NoAnswer Conditions: [NonSchedulable: NoAnswer] } raisedAt: "2025-06-30T11:41:54Z" - type: AgentSelection status: ResponseReceived message: |- Cluster Scheduling result{ Name: sesterce-a100-bm-01 AgentID: k8s-training-sesterce-004-CLIENT-PROD-client-prod Response: NoAnswer Conditions: [NonSchedulable: NoAnswer] } raisedAt: "2025-06-30T11:41:54Z" - type: AgentSelection status: ResponseReceived message: |- Cluster Scheduling result{ Name: k8s-training-smc-001 AgentID: k8s-training-smc-001-CLIENT-PROD-client-prod Response: NoAnswer Conditions: [NonSchedulable: NoAnswer, OrgNotAuthorized] } raisedAt: "2025-06-30T11:41:54Z" - type: AgentSelection status: Completed message: Selected agent k8s-training-sesterce-001-CLIENT-PROD-client-prod raisedAt: "2025-06-30T11:41:54Z" - type: BuildSubmission status: Succeeded message: Build request sent to flex-agent raisedAt: "2025-06-30T11:41:54Z" - type: BuildExecution status: Succeeded message: Build completed with image rg.fr-par.scw.cloud/paas-trainings-client-prod/9f9c379c-8d46-419b-8bf5-d0b0986a6dd9-arch_nvidia-1x1@sha256:0d854f75f698a549d2a8a0e024e930383b885bdac2863ee0cf74ebdc8a8f358c raisedAt: "2025-06-30T11:41:54Z" - type: TrainingPreparation status: Succeeded message: Training trainings-client-prod/training-75b79cc2-ec63-4f93-b4da-44e49a4a6049-zqg6d created raisedAt: "2025-06-30T11:41:54Z" - type: TrainingExecution status: InProgress message: Training in progress raisedAt: "2025-06-30T11:42:00Z" - type: TrainingExecution status: Succeeded message: Training complete, output available raisedAt: "2025-06-30T11:43:48Z" createdAt: "2025-06-30T11:41:54Z" lastUpdate: "2025-06-30T11:43:48Z" ``` ```json theme={null} { "kind": "Training", "metadata": { "name": "quickstart-training-job", "id": "75179cc2-ec63-4f93-b4da-44e49ea86049", "creatorUserID": "16e2894c-c81b-4a15-91d9-0e2aae00a317", "ownerOrgID": "108dddec-e922-49b8-a466-4d7ed5dcc746" }, "config": { "device": "nvidia", "nodes": 1, "accelerator": 1, "entrypoint": [ "train.py", "config/train_shakespeare_char.py", "--out_dir=/output-checkpoint", "--max_iters=1500" ], "datasetsNames": [ "nanoGPT-dataset" ], "checkpointName": "", "sourceName": "", "repositoryURL": "https://github.com/flexaihq/nanogpt", "repositoryRevision": "main", "secrets": [], "environment": [] }, "runtime": { "status": "succeeded", "queuePosition": 0, "repositoryRevisionSha": "116799dbae7b0fe33caf1b90f73a72f84bc32adc", "selectedAgentId": "k8s-training-sesterce-001-CLIENT-PROD-client-prod", "lifecycleEvents": [ { "type": "AgentSelection", "status": "ResponseReceived", "message": "Cluster Scheduling result{\n Name: aws-cloud\n AgentID: k8s-training-aws-001-CLIENT-PROD-client-prod\n Response: NoAnswer\n Conditions: [NonSchedulable: NoAnswer]\n}", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "AgentSelection", "status": "ResponseReceived", "message": "Cluster Scheduling result{\n Name: sesterce-h100-bm-01\n AgentID: k8s-training-sesterce-001-CLIENT-PROD-client-prod\n Response: OK\n Conditions: []\n}", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "AgentSelection", "status": "ResponseReceived", "message": "Cluster Scheduling result{\n Name: sesterce-h200-bm-01\n AgentID: k8s-training-sesterce-002-CLIENT-PROD-client-prod\n Response: NoAnswer\n Conditions: [NonSchedulable: NoAnswer]\n}", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "AgentSelection", "status": "ResponseReceived", "message": "Cluster Scheduling result{\n Name: sesterce-l40s-bm-01\n AgentID: k8s-training-sesterce-003-CLIENT-PROD-client-prod\n Response: NoAnswer\n Conditions: [NonSchedulable: NoAnswer]\n}", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "AgentSelection", "status": "ResponseReceived", "message": "Cluster Scheduling result{\n Name: sesterce-a100-bm-01\n AgentID: k8s-training-sesterce-004-CLIENT-PROD-client-prod\n Response: NoAnswer\n Conditions: [NonSchedulable: NoAnswer]\n}", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "AgentSelection", "status": "ResponseReceived", "message": "Cluster Scheduling result{\n Name: k8s-training-smc-001\n AgentID: k8s-training-smc-001-CLIENT-PROD-client-prod\n Response: NoAnswer\n Conditions: [NonSchedulable: NoAnswer, OrgNotAuthorized]\n}", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "AgentSelection", "status": "Completed", "message": "Selected agent k8s-training-sesterce-001-CLIENT-PROD-client-prod", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "BuildSubmission", "status": "Succeeded", "message": "Build request sent to flex-agent", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "BuildExecution", "status": "Succeeded", "message": "Build completed with image rg.fr-par.scw.cloud/paas-trainings-client-prod/9f9c379c-8d46-419b-8bf5-d0b0986a6dd9-arch_nvidia-1x1@sha256:0d854f75f698a549d2a8a0e024e930383b885bdac2863ee0cf74ebdc8a8f358c", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "TrainingPreparation", "status": "Succeeded", "message": "Training trainings-client-prod/training-75b79cc2-ec63-4f93-b4da-44e49a4a6049-zqg6d created", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "TrainingExecution", "status": "InProgress", "message": "Training in progress", "raisedAt": "2025-06-30T11:42:00Z" }, { "type": "TrainingExecution", "status": "Succeeded", "message": "Training complete, output available", "raisedAt": "2025-06-30T11:43:48Z" } ], "createdAt": "2025-06-30T11:41:54Z", "lastUpdate": "2025-06-30T11:43:48Z" } } ``` ## Next Steps Once a Fine-tuning Job is running, you can monitor its progress by checking its logs and leveraging the FlexAI Observability Services. You'll learn more about this in the next step of the Quickstart Tutorial: [Monitoring a Fine-tuning Job's Progress](/core-services/training/quickstart/monitoring-progress/). # Creating a Fine-tuning Job Source: https://docs.flex.ai/core-services/fine-tuning/quickstart/creating Create and configure a new fine-tuning job on FlexAI With a Dataset available on your FlexAI account, you can now create a Fine-tuning Job that will use it. ## The Model's repository For this tutorial we will use the [**FlexAI fork of the nanoGPT repository**](https://github.com/flexaihq/nanogpt), originally created by [Andrej Karpathy](https://github.com/karpathy). A Fine-tuning Job requires at least a **Name**, a link to a **GitHub repository** where its code resides, and the ***path to the entry point script*** that will initiate the Workload. In addition, the *entry point script* can be followed by any arguments required, such as configuration files or Hyperparameters. ### Entry Point script arguments The entry point script path for this quickstart tutorial is [`./train.py`](https://github.com/flexaihq/nanoGPT/blob/main/train.py), and it expects the following arguments: * `config/train_shakespeare_char.py`: A configuration file, which contains the default Workload Parameters. * `--dataset_dir`: The path within the `/input` directory of the Workload Runtime where the Dataset files are located. * `--out_dir`: The output directory, which will be mounted into the Workload Runtime as `/output-checkpoint`. * `--max_iters`: The maximum number of iterations to run the Workload script for (optional). These include any **Environment Settings** and **Hyperparameters** the entry point script may require. For this tutorial: | Parameter | Type | Description | | - | - | - | | `config/train_shakespeare_char.py` | Environment Setting | A positional argument pointing to a configuration file used by nanoGPT's `train.py` script to set the default Workload Parameters | | `--out_dir=/output-checkpoint` | Environment Setting | The output directory where the Workload script will write checkpoint files. In order to take advantage of FlexAI's Managed Checkpoints feature, this **`should always be /output-checkpoint`** | | `--max_iters=1500` | Hyperparameter | The maximum number of iterations to run the Workload script for. This is an optional hyperparameter that can be used to tweak the Workload execution | ## Starting a new Fine-tuning Job The **Start a new Fine-tuning job** form consists of a set of required and optional fields that you can use to customize your deployment. ### To open the **Start a new Fine-tuning job** form Either: * Follow the direct link to the "Start a new Fine-tuning Job" page. Or Navigate to the **Fine-tuning** section from either the navigation bar or the card on the home page. Select the **New** button to display the creation form. A drawer menu with the creation form will be displayed. ### Required Fields * **Name**: Your Fine-tuning Job name. Should follow the [resource naming conventions](/best-practices/resource-naming-conventions/). * **Repository URL**: The URL of the Git repository containing your Fine-tuning code.\`. * **Entry Point**: The path to the entry point script in your repository that will initiate the Fine-tuning Job. * The *entry point script* can be followed by any arguments you want to pass to it, such as configurations and Hyperparameters. **Value**: `train.py config/train_shakespeare_char.py --dataset_dir=my_dataset --out_dir=/output-checkpoint --max_iters=1500`. ### Other fields * **Repository Revision**: The Git revision (branch, tag, or commit) you want to use for this Fine-tuning Job. The `main` branch will be used by default. * **Node Count**: The number of nodes you want to use for this Fine-tuning Job. Defaults to `1`. * This will determine the amount of Accelerators you will have available for your Fine-tuning Job: * 1 node will allow you to use up to 8 Accelerators. * Using more than 1 node will make all 8 Accelerators per Node available to your Fine-tuning Job. * **Accelerator Count**: The number of Accelerators you want to use for this Fine-tuning Job. Must follow the logic described above. Defaults to `1`. * **Datasets**: Can be selected from a dropdown list of the datasets you want to use for this Fine-tuning Job. You can add multiple datasets as well as specify the mount path within the Fine-tuning Runtime (they will be mounted under `/input`). You can read more about this in the [Pushing a Dataset guide](/core-services/training/quickstart/uploading-a-dataset/). After picking a Dataset, select the "Add" button — otherwise it won't be added to the Training Job. * **Environment Variables & Secrets**: Add any environment variables you want to set for this Fine-tuning Job. These will be available to your Fine-tuning code as environment variables within the Training Runtime. * You can also reference **Secrets**, which will be securely injected into the Fine-tuning Job's Runtime. * **Cluster**: The cluster where the Fine-tuning workload will run on. It can be selected from a dropdown list of available clusters in your FlexAI account. A default cluster will be automatically selected for you if none is specified. ### Form Values | Field Name | Value | | - | - | | **Name** | `nanoGPT-flexai-console` | | **Repository URL** | `https://github.com/flexaihq/nanogpt` | | **Repository Revision** | `main` | | **Node Count** | `1` | | **Accelerator Count** | `1` | | **Entry Point** | `train.py config/train_shakespeare_char.py --dataset_dir=my_dataset --out_dir=/output-checkpoint --max_iters=1500` | | **Datasets** | Dataset: `nanoGPT-dataset` (from the CLI quickstart),
Mount Directory: `my_dataset` | | **Cluster** | *Your organization's designated cluster* | ### Entry Point script arguments The entry point script for this Fine-tuning Job is `train.py`, and it expects the following arguments: * `config/train_shakespeare_char.py`:A configuration file, which contains the default Fine-tuning Parameters. * `--dataset_dir`: The path within the `/input` directory of the Fine-tuning Runtime where the Dataset files are located. * `--out_dir`: The output directory, which will be mounted into the Fine-tuning Runtime as `/output-checkpoint`. * `--max_iters`: The maximum number of iterations to run the Fine-tuning script for (optional). These include any **Environment Settings** and **Hyperparameters** the Fine-tuning script may require. For this tutorial: | Parameter | Type | Description | | - | - | - | | `config/train_shakespeare_char.py` | Environment Setting | A positional argument pointing to a configuration file used by nanoGPT's `train.py` script to set the default Fine-tuning Parameters | | `--out_dir=/output-checkpoint` | Environment Setting | The output directory where the Fine-tuning script will write checkpoint files. In order to take advantage of FlexAI's Managed Checkpoints feature, this **`should always be /output-checkpoint`** | | `--max_iters=1500` | Hyperparameter | The maximum number of iterations to run the Fine-tuning script for. This is an optional hyperparameter that can be used to tweak the Fine-tuning Job execution | *** After filling out the form, select the **Submit** button to start the Fine-tuning Job. You should get a confirmation message indicating that the Fine-tuning Job creation process has been initiated successfully. The **Start a new training job** form will close and you will be redirected to the Fine-tuning Jobs list page, where you can see your newly created Fine-tuning Job in the list. *** Considering the minimum required elements for the creation of a Fine-tuning Job, the following command will initiate its creation and start it running immediately: ```bash theme={null} flexai training run quickstart-fine-tuning-job \ --dataset nanoGPT-dataset=my_dataset \ --repository-url https://github.com/flexaihq/nanogpt \ --checkpoint a1b18a7f-9b85-4c74-91a9-6aca526e8ce4 \ -- train.py config/train_shakespeare_char.py --dataset_dir=my_dataset --out_dir=/output-checkpoint --max_iters=1500 ``` #### Arguments | FlexAI command Argument | Value | Description | | - | - | - | | **Fine-tuning Job Name** | `quickstart-fine-tuning-job` | The name of the Fine-tuning Job | #### Flags | Flag | Value | Description | | - | - | - | | **Dataset** | `my_dataset=nanoGPT-dataset` | The mount path followed by the Dataset name. The dataset will be accessible at `/input/my_dataset` | | **Repository URL** | `https://github.com/flexaihq/nanogpt` | The URL of the GitHub repository containing the workload's code | | **Entry Point Script** | `train.py` | The path of entry point Fine-tuning script as defined by the repository | #### Entry Point script arguments These include any **Environment Settings** and **Hyperparameters** the entry point script may require. Keep in mind that these are specific to the code you're running: | Entry point script argument | Type | Description | | - | - | - | | `config/train_shakespeare_char.py` | Environment Setting | A positional argument pointing to a configuration file used by nanoGPT's `train.py` script to set the default runtime Parameters | | `--out_dir=/output-checkpoint` | Environment Setting | The output directory where the script will write checkpoint files. In order to take advantage of FlexAI's Managed Checkpoints feature, this **`should always be /output-checkpoint`** | | `--max_iters=1500` | Hyperparameter | The maximum number of iterations to run. This is an optional hyperparameter that can be used to tweak the Workload execution | ## Up next Next you'll learn how to get a Fine-tuning Job's details and monitor its progress. # Getting a Fine-tuning Job's Output Source: https://docs.flex.ai/core-services/fine-tuning/quickstart/getting-output Retrieve output files and results from a completed fine-tuning job ## FlexAI Managed Checkpoints [FlexAI's Managed Checkpoints](/platform-services/checkpoint-manager/) feature enables you to get the final result of your Training Job after it completes, as well as being able to get intermediate checkpoints generated by your Training script. The only thing you need to do is to make sure your Training script calls the `torch.save()` function and writes its output to the path specified by the `FLEXAI_OUTPUT_CHECKPOINT_DIR` environment variable. FlexAI's Managed Checkpoints will handle the rest. *** Once the Training Job is running, every time its code calls the `torch.save()` function, FlexAI's Managed Checkpoints feature will automatically capture a Checkpoint and store it in the `/output-checkpoint` directory. Each Checkpoint will be assigned a unique ID and its creation time will be recorded. This means that you can go to a specific point in time and retrieve the state of the model at that moment, allowing you to resume training from that point or evaluate the model's performance on a validation dataset. After a Training Job completes, the last Checkpoint will be the one with the most recent creation timestamp. You can retrieve checkpoints generated by *FlexAI's Managed Checkpoints* at any point, which allows you to go back to a previous point in the past to resume training, to test your model, or to use it for inference. ### Listing Checkpoints You can list all available checkpoints for a specific Training Job by running the [`flexai training checkpoints`](/cli/reference/training/checkpoints) command: ```bash theme={null} flexai training checkpoints quickstart-training-job ``` This will return a table with a list of Checkpoint IDs and their corresponding creation timestamps, similar to the following: ```text theme={null} ID │ TIMESTAMP ──────────────────────────────────────┼──────────────────────────────────── 50e5ec69-32b6-e483-9c49-38a73cc34294 │ 2025-06-30 12:42:55.214 +0100 WEST 82d21263-8ba8-dd73-9c61-732d3b7b0adc │ 2025-06-30 12:43:01.77 +0100 WEST 32d07a60-61cc-4598-b4f6-2073a4f8d0af │ 2025-06-30 12:43:14.734 +0100 WEST ``` ## Getting Checkpoints You can select the gear icon ⚙️ (labeled as *Configure*) in the `Actions` field of the Training Jobs list page. This will open a "Details" panel. The **Details** tab will be selected by default, showing all the relevant information about your Training Job. Navigate to the **Checkpoints** tab to view the list of checkpoints created during the Training Job. Each checkpoint entry includes details such as: * Actions: * Download * Deploy: If the Checkpoint is an "Inference-ready Checkpoint", you can deploy it directly as an Inference Endpoint * Created: * Creation date and age * Training Loss: The reported training loss at the time the checkpoint was created * Evaluation Loss: The reported evaluation loss at the time the checkpoint was created * Status: The status of the checkpoint (`available` or `processing`) *** Once you have the desired Checkpoint ID, you can download it to your host machine using the [`flexai checkpoint fetch`](/cli/reference/checkpoint/fetch) command: ```bash theme={null} flexai checkpoint fetch 32d07a60-61cc-4598-b4f6-2073a4f8d0af ``` ```text theme={null} Writing in: /home/diego/ckpt.pt Progress: 0.4% (1.31 MB / 343.79 MB) // ... Progress: 100% (343.79 MB / 343.79 MB) ``` You can use this checkpoint file to resume training from the exact point it was saved, or to evaluate the model's performance on a validation dataset. Use the [`flexai checkpoint export`](/cli/reference/checkpoint/export) command to export a Checkpoint to a remote location by using a previously registered Remote Storage Connection, such as S3, GCS, MinIO or R2. This allows you to store your checkpoints in a more permanent location for later use. ## Workload Raw Outputs Fetching raw outputs from the `/output` directory is done through the FlexAI CLI — see the **Using the FlexAI CLI** tab. Any data written to the `/output` directory will be compressed into a zip file and made available to you via the [`flexai training fetch`](/cli/reference/training/fetch) command: ```bash theme={null} flexai training fetch quickstart-training-job ``` This will download a `.zip` file to the current working directory on your host machine. Once extracted you'll get a local directory named `output` it will contain any files written to the `/output` directory by the training scripts. For this quickstart example we configured the training code to write to the checkpoints directory `/output-checkpoint` directory when we set the value of `--out_dir`, so the `/output` directory won't be used. However, you can run another Training Job that instead writes out to the `/output` directory. In this case, only the last checkpoint will be saved in the `/output` directory as a result of the training process. Note that your code can use both locations (`/output` and `/output-checkpoint`) simultaneously if needed. # Monitoring a Fine-tuning Job's Progress Source: https://docs.flex.ai/core-services/fine-tuning/quickstart/monitoring-progress Monitor your fine-tuning job's progress with logs and metrics ## Logs You can select the gear icon ⚙️ (labeled as *Configure*) in the `Actions` field of the Training or Fine-tuning Jobs list page. This will open a "Details" panel. Select the **Logs** tab to view the messages emitted to the standard output (`stdout`) created by the runtime environment build process and your own code. You can use the **Search bar** input field to filter the logs by a specific keyword. This is useful to quickly find relevant information in the logs. Once a Training Job begins, logs emitted to the standard output (`stdout`) created by the runtime environment build process and your own code can be retrieved by running the [`flexai training logs`](/cli/reference/training/logs) command: ```bash theme={null} flexai training logs quickstart-training-job ``` This will output a stream of logs including both the FlexAI runtime execution logs and your code's messages. ## Infrastructure Metrics You can monitor the infrastructure metrics of your Fine-tuning Job using the FlexAI Infrastructure Monitor. This will give you insights into the resource usage of your Fine-tuning Job, such as CPU and memory usage, disk I/O, and network traffic. Access FlexAI's Infrastructure Monitor by visiting [https://dashboards.flex.ai/](https://dashboards.flex.ai/). Visit the [FlexAI Infrastructure Monitor](/platform-services/observability/infrastructure-monitor/) page to learn more. ## TensorBoard You can also use FlexAI's hosted TensorBoard to visualize the fine-tuning process of your model. TensorBoard provides a suite of tools for inspecting and understanding your Fine-tuning Job's evolution. Visit [https://dashboards.flex.ai/tensorboard](https://dashboards.flex.ai/tensorboard) and log in using your credentials. Learn more at the [FlexAI TensorBoard](/platform-services/observability/tensorboard/) page. ## Next Steps After a few minutes, your Fine-tuning Job should have completed successfully! The next step of this Quickstart Tutorial will guide you through the process of getting its outputs or *results*. # Uploading a Dataset Source: https://docs.flex.ai/core-services/fine-tuning/quickstart/uploading-a-dataset Upload and prepare a dataset for your fine-tuning job ## Download The Sample Dataset Files You can upload files or directories from your local machine to create a new Dataset. For this quickstart tutorial, we will use two pre-generated files: `train.bin` and `val.bin` that you can download to your machine by using the links below: * [train.bin](https://github.com/flexaihq/nanoGPT/raw/refs/heads/prepare_dataset/data/shakespeare_char/dataset/train.bin): Training dataset * [val.bin](https://github.com/flexaihq/nanoGPT/raw/refs/heads/prepare_dataset/data/shakespeare_char/dataset/val.bin): Validation dataset for model evaluation *** Download the pre-generated training dataset files for this quickstart tutorial: ```bash theme={null} curl -L --remote-name-all https://github.com/flexaihq/nanoGPT/raw/refs/heads/prepare_dataset/data/shakespeare_char/dataset/{train.bin,val.bin} ``` This downloads two binary files: * `train.bin`: Training dataset (Shakespeare character-level data) * `val.bin`: Validation dataset for model evaluation *** ## Uploading files to the FlexAI Dataset Manager Service Navigate to the Dataset Manager in the FlexAI Console: Visit the ["Add Dataset" section of the FlexAI Console](https://platform.flex.ai/datasets/new) Enter a name for your dataset: `nanoGPT-dataset` * Must follow the FlexAI [Resource Naming conventions](/best-practices/resource-naming-conventions/) Select the "Local" option for "Upload Origin" Select the `+ Upload Item` button to open the "Upload Items" dialog Use the "Select file" option to open a file browser dialog: Select both `train.bin` and `val.bin` files from your local machine * Depending on your system and browser, you might need to hold down the Ctrl/Cmd key while selecting multiple files * When selecting multiple files, your browser might prompt you with a confirmation message asking you to allow multiple file selection. * You can also select files individually if you prefer In the "Destination Path" field, enter `shakespeare_char` Select the `Add` button to confirm the file selection and destination mapping The "Upload Items" dialog will close and you will get back to the "Add a Dataset" form, where you will see the files you just added listed under the "Upload Items" section. Here you will see a list of files similar to the one below: ``` shakespeare_char/ ├── train.bin (1.91MB) └── val.bin (217.85KB) ``` Below the file list you will find an `+ Add items` button that will open up the "Upload Items" dialog again, in case you want to add more files or directories to the Dataset. Finally, select the `Add Dataset` button to start the upload process. *** Use the `flexai dataset push` command to create and upload your Dataset: ```bash theme={null} flexai dataset push nanoGPT-dataset \ --file train.bin=shakespeare_char/train.bin \ --file val.bin=shakespeare_char/val.bin ``` The command above has the following components: | Component | Value | Description | | - | - | - | | **Dataset Name** | `nanoGPT-dataset` | Name for the Dataset in FlexAI. Must follow the FlexAI [Resource Naming conventions](/best-practices/resource-naming-conventions/). | | **File Mapping** | `train.bin=shakespeare_char/train.bin` | Maps local file to dataset path | | **File Mapping** | `val.bin=shakespeare_char/val.bin` | Maps local file to dataset path | *** ## Dataset structure Once upload is complete, you can select the gear icon ⚙️ (labeled as *Configure*) in the `Actions` field of the Dataset list page. This will open the Dataset "Details" panel where you will be able to see the Dataset's name, status, creation date, and a list of files that were uploaded as part of the Dataset. ### Summary * **Name**: nanoGPT-dataset * **Status**: Ready * **Creation Time**: 8/5/2025, 2:13:07 PM ### Details ``` shakespeare_char/ ├── train.bin (1.191MB) └── val.bin (217.85KB) ``` To learn more about the ways your workloads can access Datasets, check out the [Runtime Access](/platform-services/dataset-manager/#runtime-access) section of the Dataset Manager overview page. *** Check that your dataset was uploaded successfully: ```bash theme={null} flexai dataset list ``` Expected output: ```text theme={null} NAME │ FILES COUNT │ TOTAL SIZE │ STATUS │ CREATED AT ─────────────────────┼─────────────┼────────────┼───────────┼──────────────────────────── nanoGPT-dataset │ 2 │ 2.13 MB │ available │ 2025-08-05 13:13:07 (1h) ``` The status should show `available` when the upload is complete and the dataset is ready for training. You can run the `flexai dataset inspect ` command to get more detailed information about your Dataset: ```bash theme={null} flexai dataset inspect nanoGPT-dataset ``` Which will output something like: ```yaml theme={null} kind: Dataset metadata: name: nanoGPT-dataset id: 1b541f62-2faf-4e32-8fd1-a6bc27e26b58 creatorUserID: 16e289cc-c81b-4a15-91d9-0e2aae00a317 ownerOrgId: 270a5476-b91a-442f-8a13-852ef7bb5b9c spec: fromLocalFiles: - train.bin - val.bin storageProvider: "" sourcePath: "" status: status: available storageProviderID: 00000000-0000-0000-0000-000000000000 size: 2.13 MB files: - path: shakespeare_char/train.bin size: 1.91 MB - path: shakespeare_char/val.bin size: 217.85 KB createdAt: 2025-08-05 14:13:07 (57d) updatedAt: 2025-08-05 14:13:08 (57d) dataSyncs: [] ``` ```json theme={null} { "kind": "Dataset", "metadata": { "name": "nanoGPT-dataset", "id": "1b541f62-2faf-4e32-8fd1-a6bc27e26b58", "creatorUserID": "15e2894c-c81b-4a15-91d5-0e2aae00a317", "ownerOrgID": "270a5476-b91a-442f-8a13-852ef7bb5b9c" }, "spec": { "fromLocalFiles": [ "train.bin", "val.bin" ], "storageProvider": "", "sourcePath": "" }, "status": { "status": "available", "storageProviderID": "00000000-0000-0000-0000-000000000000", "size": 2230788, "files": [ { "path": "shakespeare_char/train.bin", "size": 2007708 }, { "path": "shakespeare_char/val.bin", "size": 223080 } ], "createdAt": "2025-08-05T13:13:07Z", "updatedAt": "2025-08-05T13:13:08Z", "dataSyncs": [] } } ``` *** # FlexAI Inference Endpoints Source: https://docs.flex.ai/core-services/inference Deploy and manage AI models for inference with FlexAI Deploy and manage AI models as HTTP API endpoints that you can securely query from your applications. You can deploy public and private models hosted on the Hugging Face Model Hub or deploy models you've fine-tuned using FlexAI. ## Key Features Get models running in minutes Automatic scaling based on demand Support for multiple vLLM supported models Pay only for what you use Built-in metrics and logging ## Anatomy of an Inference Endpoint A FlexAI Inference Endpoint consists of the following components: 1. **Model to Serve**: The model you want to serve through a FlexAI Inference Endpoint. * A Checkpoint generated by a Training or Fine-tuning Job ran with FlexAI. 2. **Authentication Key**: The secret key required for querying the Inference Endpoint. 3. ***Hugging Face Token*'s Secret Name**: The name of the Secret stored in the FlexAI Secret Manager that holds a Hugging Face Token with access to the model you want to serve. 4. **Endpoint Configuration**: Settings that define how the endpoint behaves, such as scaling policies, and vLLM-specific arguments. ### Model to Serve The model served by your Inference Endpoint can be: * A public or private model from the Hugging Face Model Hub * A Checkpoint you manually pushed to the FlexAI Checkpoint Manager * A Checkpoint generated by a FlexAI Training or Fine-tuning Job This flexibility allows you to deploy both open-source and custom models as scalable, production-ready endpoints. ### Authentication Key Each Inference Endpoint is protected by an authentication key (secret key). This key must be provided with every request to the endpoint, ensuring that only authorized users can access your deployed models. You can create your own authentication key by assigning it to a ***Secret*** in the FlexAI Secret Manager. If an authentication key is not provided, FlexAI will create one for your Inference Endpoint and associate it with a ***Secret*** with the name of your Inference Endpoint: `-api-key`. This enables you to update the Secret in case you want to refresh the authentication key. ### Hugging Face Token's Secret Name If your model is gated or private on Hugging Face, you will need to provide a Hugging Face Access Token. Store this token as a Secret in the FlexAI Secret Manager, and reference its name in your endpoint configuration. FlexAI will inject this token securely at runtime, allowing the endpoint to pull the model as needed. ### Endpoint Configuration Endpoint configuration defines how your Inference Endpoint behaves. This includes: * Scaling policies (auto-scaling, min/max replicas) * [vLLM-specific arguments](https://docs.vllm.ai/en/latest/serving/engine_args.html) (e.g., batch size, max tokens) These settings allow you to optimize performance, cost, and reliability for your specific use case. ## Key Concepts ### Endpoint Authentication FlexAI Inference Endpoints are protected by an authentication token that you can provide during setup. This ensures that only authorized users can access your models. ### Auto-scaling FlexAI Inference Endpoints can automatically scale based on demand. This means that during periods of high traffic, additional resources will be allocated to handle requests, ensuring consistent performance, and during low traffic periods, resources will be scaled back down to optimize efficient resource utilization. ## vLLM FlexAI Inference Endpoints leverage the vLLM library for efficient model serving. vLLM is designed to optimize the performance of large language models, making it easier to deploy and scale them in production environments. You can learn more about vLLM and how to take advantage of its features in the [vLLM documentation](https://docs.vllm.ai/en/stable/usage/index.html). ## CLI Reference The FlexAI CLI's [`inference` family of commands](/cli/reference/inference) allows you to manage your inference endpoints directly from the command line. ## Getting Started FlexAI Inference Endpoints can be deployed in a few steps. The Quickstart guide will walk you through the process of setting up your first endpoint. Here's a brief overview of the steps involved: 1. Deploying a FlexAI Inference Endpoint from a model hosted in the Hugging Face Hub. 2. Deploying a FlexAI Inference Endpoint from a model fine-tuned with FlexAI. 3. Querying a FlexAI Inference Endpoint. The button below will lead you to the FlexAI Inference Endpoints Quickstart guide's overview, where you'll find more details. # FlexAI Inference Autoscaling Source: https://docs.flex.ai/core-services/inference/autoscaling Set up and manage autoscaling rules for FlexAI Inference Endpoints ## Overview FlexAI Inference Endpoints support automatic scaling to handle varying traffic loads efficiently. Autoscaling allows you to define minimum and maximum accel counts, ensuring your endpoints can scale up during high demand and scale down during low usage to optimize costs. Autoscaling in FlexAI provides several key benefits: * **Cost Optimization**: Scale down to zero accels when there's no traffic * **Performance**: Automatically scale up to handle increased demand * **Resource Management**: Set upper bounds to control maximum resource usage * **Fixed Scaling**: Lock accels to a specific count by setting min and max to the same value FlexAI Inference Endpoints support automatic scaling to handle varying traffic loads efficiently. Autoscaling allows you to define minimum and maximum replica counts, ensuring your endpoints can scale up during high demand and scale down during low usage to optimize costs. Autoscaling in FlexAI provides several key benefits: * **Cost Optimization**: Scale down to zero replicas when there's no traffic * **Performance**: Automatically scale up to handle increased demand * **Resource Management**: Set upper bounds to control maximum resource usage * **Fixed Scaling**: Lock replicas to a specific count by setting min and max to the same value ## Configuration Options #### Minimum Accels The minimum number of accels for your Inference Endpoint. This value determines the baseline capacity that will always be available. * **Setting to 0**: Allows the endpoint to scale down completely when there are no requests, optimizing costs * **Setting above 0**: Ensures a baseline level of availability and faster response times for initial requests #### Maximum Accels The maximum number of accels that can be created for your Inference Endpoint. This acts as an upper bound to control resource usage and costs. * **Set this at or above the minimum accels** so the endpoint can scale up when demand rises * **Mind the ceiling**: a higher maximum gives more headroom for spikes, but also raises your potential cost #### Minimum Replicas The minimum number of replicas for your Inference Endpoint. This value determines the baseline capacity that will always be available. * **Setting to 0**: Allows the endpoint to scale down completely when there are no requests, optimizing costs * **Setting above 0**: Ensures a baseline level of availability and faster response times for initial requests #### Maximum Replicas The maximum number of replicas that can be created for your Inference Endpoint. This acts as an upper bound to control resource usage and costs. * **Set this at or above the minimum replicas** so the endpoint can scale up when demand rises * **Mind the ceiling**: a higher maximum gives more headroom for spikes, but also raises your potential cost ## Managing Autoscaling ### Setting Scaling Policies Navigate to the **Inference** section in the FlexAI Console and select the "+ New" button or open the form directly Fill out the form with the required values Select the "Enable Auto-scaling" toggle element to enable this section of the form Adjust the **Min Accels** and **Max Accels** values Use the `flexai inference scale` command to configure autoscaling for your Inference Endpoints: ```bash theme={null} flexai inference scale --min-replicas --max-replicas ``` ##### Flags | Flag | Type | Description | | - | - | - | | `--max-replicas` | integer | The maximum number of replicas for the Inference Endpoint. | | `--min-replicas` | integer | The minimum number of replicas for the Inference Endpoint. | ## Common Scaling Scenarios ### Cost-Optimized Scaling For development or low-traffic endpoints where cost optimization is prioritized: | Auto-scaling Policy | Value | | :- | -: | | Min Accels | 0 | | Max Accels | 2 | This configuration allows the endpoint to: * Scale down to zero when there's no traffic (no costs) * Scale up to a maximum of 2 replicas during usage ### Performance-Oriented Scaling For production endpoints where consistent performance is crucial: | Auto-scaling Policy | Value | | :- | -: | | Min Accels | 2 | | Max Accels | 10 | This configuration ensures: * Always maintains 2 replicas for immediate response * Can scale up to 10 replicas during high demand ### Fixed Scaling For endpoints requiring consistent resources or predictable performance: | Auto-scaling Policy | Value | | :- | -: | | Min Accels | 4 | | Max Accels | 4 | This configuration: * Disables autoscaling by setting min and max to the same value * Always maintains exactly 4 replicas *** ## Other Example Scenarios ### Development Environment Testing and development workloads | Auto-scaling Policy | Value | | :- | -: | | Min Accels | 0 | | Max Accels | 1 | Scales to zero when not in use, single replica maximum for cost control. ### Production API High-availability production service | Auto-scaling Policy | Value | | :- | -: | | Min Accels | 3 | | Max Accels | 15 | Maintains 3 replicas for consistent availability, scales up to 15 replicas during peak traffic. ### Batch Processing Predictable batch workloads | Auto-scaling Policy | Value | | :- | -: | | Min Accels | 5 | | Max Accels | 5 | Fixed scaling for consistent processing capacity, prevents resource contention. ## Best Practices ### Choosing Minimum Replicas * **Set to 0** for development, testing, or infrequently used endpoints * **Set to 1-2** for production endpoints that need quick response times * **Set higher** for high-availability services or when you need guaranteed capacity ### Choosing Maximum Replicas * Consider your infrastructure limits and budget constraints * Monitor typical traffic patterns to set appropriate upper bounds * Leave room for traffic spikes (set 20-50% above normal peak usage) ### Monitoring and Optimization Monitor your endpoint [logs and life cycles](/core-services/inference/quickstart/create-public/#checking-the-status-of-your-inference-endpoint) as well as the [Infrastructure Monitor](/platform-services/observability/infrastructure-monitor/) to understand usage patterns and optimize your scaling configuration over time. * Track response times and error rates during scaling events * Monitor costs and adjust min/max values based on usage patterns * Use gradual changes when adjusting scaling policies in production ## Troubleshooting ### Common Issues **Slow Response Times**: If your endpoint takes too long to respond after being idle: * Increase `Min Accels` to maintain warm replicas * Consider the cold-start time of your model **High Costs**: If your endpoint is more expensive than expected: * Reduce `Max Accels` to limit peak resource usage * Set `Min Accels` to 0 for non-critical endpoints **Capacity Issues**: If your endpoint can't handle traffic spikes: * Increase `Max Accels` to allow for more scaling headroom * Monitor scaling metrics to understand demand patterns ### Cost-Optimized Scaling For development or low-traffic endpoints where cost optimization is prioritized: ```bash theme={null} flexai inference scale my_dev_endpoint --min-replicas 0 --max-replicas 2 ``` This configuration allows the endpoint to: * Scale down to zero when there's no traffic (no costs) * Scale up to a maximum of 2 replicas during usage ### Performance-Oriented Scaling For production endpoints where consistent performance is crucial: ```bash theme={null} flexai inference scale my_prod_endpoint --min-replicas 2 --max-replicas 10 ``` This configuration ensures: * Always maintains 2 replicas for immediate response * Can scale up to 10 replicas during high demand ### Fixed Scaling For endpoints requiring consistent resources or predictable performance: ```bash theme={null} flexai inference scale my_stable_endpoint --min-replicas 4 --max-replicas 4 ``` This configuration: * Disables autoscaling by setting min and max to the same value * Always maintains exactly 4 replicas *** ## Other Example Scenarios ### Development Environment Testing and development workloads ```bash theme={null} flexai inference scale mistral7b-dev --min-replicas 0 --max-replicas 1 ``` Scales to zero when not in use, single replica maximum for cost control. ### Production API High-availability production service ```bash theme={null} flexai inference scale my_prod_inference_endpoint --min-replicas 3 --max-replicas 15 ``` Maintains 3 replicas for consistent availability, scales up to 15 replicas during peak traffic. ### Batch Processing Predictable batch workloads ```bash theme={null} flexai inference scale my_batch_inference_endpoint --min-replicas 5 --max-replicas 5 ``` Fixed scaling for consistent processing capacity, prevents resource contention. ## Best Practices ### Choosing Minimum Replicas * **Set to 0** for development, testing, or infrequently used endpoints * **Set to 1-2** for production endpoints that need quick response times * **Set higher** for high-availability services or when you need guaranteed capacity ### Choosing Maximum Replicas * Consider your infrastructure limits and budget constraints * Monitor typical traffic patterns to set appropriate upper bounds * Leave room for traffic spikes (set 20-50% above normal peak usage) ### Monitoring and Optimization Monitor your endpoint [logs and life cycles](/core-services/inference/quickstart/create-public/#checking-the-status-of-your-inference-endpoint) as well as the [Infrastructure Monitor](/platform-services/observability/infrastructure-monitor/) to understand usage patterns and optimize your scaling configuration over time. * Track response times and error rates during scaling events * Monitor costs and adjust min/max values based on usage patterns * Use gradual changes when adjusting scaling policies in production ## Troubleshooting ### Common Issues **Slow Response Times**: If your endpoint takes too long to respond after being idle: * Increase `--min-replicas` to maintain warm replicas * Consider the cold-start time of your model **High Costs**: If your endpoint is more expensive than expected: * Reduce `--max-replicas` to limit peak resource usage * Set `--min-replicas` to 0 for non-critical endpoints **Capacity Issues**: If your endpoint can't handle traffic spikes: * Increase `--max-replicas` to allow for more scaling headroom * Monitor scaling metrics to understand demand patterns # Quickstart: FlexAI Inference Endpoints Source: https://docs.flex.ai/core-services/inference/quickstart Step-by-step guide to deploying inference endpoints on FlexAI In this guide, we will walk you through the steps required to deploy a FlexAI Inference Endpoint, how to query it, and how to manage it. ## Prerequisites * A FlexAI account. If you don't have one, you can [sign up for free](https://platform.flex.ai/signup). * We will be exploring how to leverage the Hugging Face Model Hub to source models, and depending on the model, a Hugging Face Access Token may be required. You can create an access token by following the instructions in the [Hugging Face documentation](https://huggingface.co/docs/hub/security-tokens). If you don't yet have a Hugging Face Account, your can create one for free [by signing up here](https://huggingface.co/join)! ## Content 1. Deploying a FlexAI Inference Endpoint from a model hosted in the Hugging Face Hub. * **Creating a FlexAI Secret**: to securely store your Hugging Face Access Token. 2. Deploying a FlexAI Inference Endpoint from a model fine-tuned with FlexAI. * **Pushing a checkpoint** to the FlexAI Checkpoint Manager to create an Inference Endpoint from a custom model. 3. Querying a FlexAI Inference Endpoint. * **Making an HTTP request**: to the Inference Endpoint with common tools and libraries. # Creating an Inference Endpoint: Private Model Source: https://docs.flex.ai/core-services/inference/quickstart/create-private Deploy an inference endpoint from a private or gated Hugging Face model We'll use a model that requires authentication on the Hugging Face Hub. This could be a private model you have access to, or a "Gated model" that requires agreeing to the creator's license agreement, privacy policy, or similar before you can use them. In this guide, we'll be using [https://huggingface.co/mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1), a "Gated model" that requires agreeing to the creator's terms before you can use it. ## Gated Models Gated models can be identified by a special symbol. If the model is "Gated", you will find the necessary information on how to proceed. Example: Mistral-7B-v0.1 model page on Hugging Face Hub: Before being granted access If you have already gone through the process, you will find a badge on the model's page indicating that you have access to the model. Example: Mistral-7B-v0.1 model page on Hugging Face Hub: After being granted access Once you've been granted access to the model, you can proceed with the process of creating a Hugging Face Access Token to authenticate requests towards the Hugging Face Hub. ## Creating a Hugging Face Access Token To create a Hugging Face Access Token, follow these steps: 1. Go to your [Hugging Face account settings](https://huggingface.co/settings/tokens). Confirm your identity if requested by entering your credentials. 2. Select the "+ Create new token" button. 3. Set the "Token type" to Read. 4. Enter a name for your token in the "Token name" field. 5. Select the "Create token" button. 6. A "Save your Access Token" prompt will appear. Copy the generated token. We recommend you keep the "Save your Access Token" prompt open until you've securely stored your token as a FlexAI Secret, as you won't be able to view it again. ## Storing a Hugging Face Access Token as a FlexAI Secret To store your Hugging Face Access Token as a FlexAI Secret, follow these steps: 1. Navigate to the **Secrets** section from the navigation bar. 2. Select the **+ New** button to open the creation form panel. 3. Set a **Secret Name** that clearly describes the purpose of the Secret. It must follow the FlexAI Resource Naming Conventions. In this example we'll use `hf_token`. 4. Enter the **Secret Value** either manually or by pasting it from your clipboard. 5. Select the **Submit** button to save your new Secret. 1. Create a Secret by using the `flexai secret create` command, which will receive the name of the secret as its only argument. In this case we will use `hf_token` to store a Hugging Face Access Token. ```bash theme={null} flexai secret create hf_token ``` 2. You will be prompted to enter the value for the secret. You can either type in the value directly or paste it. In any case, the value will not be displayed in the terminal. ```bash theme={null} flexai secret create hf_token Secret Value: █ ``` ## Creating a FlexAI Inference Endpoint 1. Navigate to the **Inference** section from either the navigation bar or the card on the home page. 2. Select the "+ New" button to display the "Launch Inference" panel. 3. Fill out the *Launch Inference* form according to the instructions below. ### The *Launch Inference* form The *Launch Inference* form consists of a set of required and optional fields that you can use to customize your deployment. #### Required Fields * **Name**: A unique name for your inference endpoint. This will be used to identify your endpoint in the FlexAI console. It must follow the FlexAI [resource naming conventions](/best-practices/resource-naming-conventions/). * **Hugging Face Model**: The name of the model to deploy. * **Hugging Face Token**: The name of the Secret you used to store your Hugging Face Access Token. * **Cluster**: The cluster where the Training workload will run. It can be selected from a dropdown list of available clusters in your FlexAI account. #### Form Values | Field | Value | | - | - | | **Name** | `quickstart-inference-mistral` | | **Hugging Face Model** | `mistralai/Mistral-7B-v0.1` | | **Hugging Face Token** | `hf_token` | | **Cluster** | *Your organization's designated cluster* | #### Other fields There are a few optional fields that you can use to customize your deployment: * **API Key**: A secret key that will be used to authenticate requests to your Inference endpoint. If left empty, a random API Key will be generated and displayed to you after you initiate the deployment process. Make sure to copy it and store it in a safe place, as you will not be able to see it again. * **vLLM Parameters**: A [set of arguments that will be passed to vLLM](https://docs.vllm.ai/en/latest/serving/engine_args.html). You can use this to customize the behavior of the vLLM server, such as setting the maximum number of tokens to generate, the temperature, and other parameters. FlexAI handles device selection for you, so the `--device` vLLM parameter isn't supported here — just leave it out. ### Starting the Inference Endpoint After filling out the form, select the **Submit** button to start the Inference Endpoint deployment. You should get a confirmation window displaying the details of your Inference Endpoint, including the API Key that needs to be used to authenticate requests towards the endpoint. Make sure to copy the API Key and **safely store it**. You can always delete the Endpoint and create a new one if you lose it, but better to save yourself the hassle. 1. Create an Inference endpoint for the model you want to deploy and pass in name of the secret where the Hugging Face Access Token is stored to the `--hf-token-secret` flag. ```bash theme={null} flexai inference serve mistral-7b-instruct \ --hf-token-secret hf_token \ -- --model=mistralai/Mistral-7B-Instruct-v0.1 ``` 2. Store the API Key you're prompted with. You will need it to authenticate your requests to the Inference Endpoint. ```text title="flexai inference serve mistral-7b-instruct..." theme={null} Inference "mistral-7b-instruct" started successfully Here is your API Key. Please store it safely: 0ad394db-8dba-4395-84d9-c0db1b1be0a8 . You can reference it with the name: mistral-7b-instruct-api-key ``` 3. Run [`flexai inference list`](/cli/reference/inference/list) to follow the status of your Inference Endpoint and get its URL. ```bash theme={null} flexai inference list ``` ```text title="'flexai inference list' response" wrap=false theme={null} NAME │ STATUS │ AGE │ ENDPOINT ────────────────────┼──────────────────┼─────┼────────────────────────────────────────────────────────────────────────────────── mistral-7b-instruct │ starting │ 69s │ https://inference-55178dae-2bea-4756-a918-8a40e20b711a.platform.flex.ai ``` ## Checking the status of your Inference Endpoint After a few minutes, your Inference Endpoint should be up and running. **Inference Endpoint Details** You can select the gear icon ⚙️ (labeled as *Configure*) in the `Actions` field of the Inference Endpoint list row of your newly created Endpoint to open the *details panel* of the Inference Endpoint deployment. The Details tab will be opened by default, showing you all the relevant information about your Inference Endpoint. This tab provides you with detailed information about your Inference Endpoint, including: **The Summary tab** | Field | Description | | - | - | | `ID` | The unique identifier of the Inference Endpoint. | | `Name` | The name you assigned to the Inference Endpoint. | | `Status` | The current status of the Inference Endpoint (e.g., `Running`, `Stopped`, etc.). | | `URL` | The base URL of the Inference Endpoint, which you can use to query the model. | | `Playground URL` | The URL of the Inference Playground, a user-friendly interface to interact with your deployed model. | | `Dashboard URL` | The URL of the Inference Endpoint dashboard, where you can monitor the performance and usage of your model. | **Configuration** | Field | Description | | - | - | | `Device Architecture` | The architecture of the device where the Inference Endpoint is running (e.g., `nvidia`). | | `Runtime Args` | The vLLM runtime arguments that were used to deploy the Inference Endpoint. These can be customized when creating or updating the Inference Endpoint. | | `HF Token Secret Name` | The name of the FlexAI Secret that contains the Hugging Face Access Token, if applicable. This is only shown if the Inference Endpoint requires a Hugging Face Access Token to access the model. | | `API Key Secret Name` | The name of the FlexAI Secret that contains the API Key used to authenticate requests to the Inference Endpoint. | **The Activity tab** The Activity tab provides you with a timeline of events related to your Inference Endpoint, including deployment status changes, scaling events, and more. *** **The Logs tab** The Logs tab provides you with real-time logs from your Inference Endpoint, allowing you to monitor its activity and troubleshoot any issues that may arise. You can use the **Search bar** input field to filter the logs by a specific keyword. This is useful to quickly find relevant information in the logs. *** ##### `flexai inference list` The `flexai inference list` command offers a quick overview of the current **status** and **HTTP endpoint** of your Inference Endpoints. ```bash theme={null} flexai inference list ``` Output: ```text title="'flexai inference list' response" wrap=false theme={null} NAME │ STATUS │ AGE │ ENDPOINT ────────────────────┼──────────────────┼─────┼────────────────────────────────────────────────────────────────────────────────── mistral-7b-instruct │ starting │ 69s │ https://inference-55178dae-2bea-4756-a918-8a40e20b711a.platform.flex.ai ``` #### `flexai inference inspect` The [`flexai inference inspect`](/cli/reference/inference/inspect) command provides detailed information about a specific Inference Endpoint, including its current status, configuration, and lifecycle. ```bash theme={null} flexai inference inspect mistral-7b-instruct ``` This will display your endpoint's full configuration, runtime status, and connection details. ##### `flexai inference logs` ```bash theme={null} flexai inference logs mistral-7b-instruct ``` Output: ```text title="flexai inference logs response" wrap=false theme={null} [...] [INFO] Starting inference for mistral-7b-instruct [INFO] Inference completed successfully [...] ``` ## Managing your Inference Endpoint The Inference Endpoints table's Actions column provides a set of actions that you can use to manage your Inference Endpoint: * **Configure**: To access its Details panel. * **Pause**: To temporarily stop the Inference Endpoint without deleting it. * **Delete**: To permanently remove the Inference Endpoint. * **Resume**: To restart a paused Inference Endpoint. *** The FlexAI CLI provides a set of commands that you can use to manage your Inference Endpoints: * [`flexai inference delete`](/cli/reference/inference/delete) - Deletes an Inference Endpoint. * [`flexai inference scale`](/cli/reference/inference/scale) - Allows for the definition of scaling policies for an Inference Endpoint. * [`flexai inference stop`](/cli/reference/inference/stop) - Stops an Inference Endpoint. *** # FlexAI Inference - Public Model Source: https://docs.flex.ai/core-services/inference/quickstart/create-public Deploy an inference endpoint from a public Hugging Face model We'll get started by creating an Inference Endpoint for a public model hosted in the Hugging Face Hub that does not require us to provide a Hugging Face Access Token —This will allow us to focus on the deployment process itself for now. We'll be using [https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct). ## Creating an Inference Endpoint 1. Navigate to the **Inference** section from either the navigation bar or the card on the home page. 2. Select the "+ New" button to display the "Launch Inference" panel. 3. Fill out the *Launch Inference* form according to the instructions below. ### The *Launch Inference* form The *Launch Inference* form consists of a set of required and optional fields that you can use to customize your deployment. #### Required Fields * **Name**: A unique name for your inference endpoint. This will be used to identify your endpoint in the FlexAI console. It must follow the FlexAI [resource naming conventions](/best-practices/resource-naming-conventions/). * **Hugging Face Model**: The name of the model to deploy. * **Cluster**: The cluster where the Training workload will run. It can be selected from a dropdown list of available clusters in your FlexAI account. #### Form Values | Field | Value | | - | - | | **Name** | `quickstart-inference-qwen` | | **Hugging Face Model** | `Qwen/Qwen2.5-1.5B-Instruct` | | **Cluster** | *Your organization's designated cluster* | #### Other fields There are a few optional fields that you can use to customize your deployment: * **Hugging Face Token**: Needed only for private or gated models — you can skip it for a public model like this one. * **API Key**: A secret key that will be used to authenticate requests to your Inference endpoint. If left empty, a random API Key will be generated and displayed to you after you initiate the deployment process. Make sure to copy it and store it in a safe place, as you will not be able to see it again. * **vLLM Parameters**: A [set of arguments that will be passed to vLLM](https://docs.vllm.ai/en/latest/serving/engine_args.html). You can use this to customize the behavior of the vLLM server, such as setting the maximum number of tokens to generate, the temperature, and other parameters. FlexAI handles device selection for you, so the `--device` vLLM parameter isn't supported here — just leave it out. ### Starting the Inference Endpoint After filling out the form, select the **Submit** button to start the Inference Endpoint deployment. You should get a confirmation window displaying the details of your Inference Endpoint, including the API Key that needs to be used to authenticate requests towards the endpoint. Make sure to copy the API Key and **safely store it**. You can always delete the Endpoint and create a new one if you lose it, but better to save yourself the hassle. 1. Create a new Inference endpoint. ```bash theme={null} flexai inference serve qwen-small \ --hf-token-secret hf_token \ -- --model=Qwen/Qwen2.5-1.5B-Instruct ``` 2. Store the API Key you're prompted with. You will need it to authenticate your requests to the Inference Endpoint. ```text title="flexai inference serve qwen-small..." theme={null} Inference endpoint "qwen-small" started successfully Here is your API Key. Please store it safely: 0ad394db-8dba-4395-84d9-c0db1b1be0a8 You can reference it with the secret name: qwen-small-api-key ``` 3. Run [`flexai inference list`](/cli/reference/inference/list) to follow the status of your Inference Endpoint and get its URL. ```bash theme={null} flexai inference list ``` ```text title="flexai inference list" wrap=false theme={null} NAME │ STATUS │ AGE │ ENDPOINT ───────────┼──────────────────┼─────┼────────────────────────────────────────────────────────────────────────────────── qwen-small │ starting │ 32s │ https://inference-e0ed242b-dbd3-423e-b525-76beac222d44-d41b1b44.platform.flex.ai ``` ## Checking the status of your Inference Endpoint After a few minutes, your Inference Endpoint should be up and running. **Inference Endpoint Details** You can select the gear icon ⚙️ (labeled as *Configure*) in the `Actions` field of the Inference Endpoint list row of your newly created Endpoint to open the *details panel* of the Inference Endpoint deployment. The Details tab will be opened by default, showing you all the relevant information about your Inference Endpoint. This tab provides you with detailed information about your Inference Endpoint, including: **The Summary tab** | Field | Description | | - | - | | `ID` | The unique identifier of the Inference Endpoint. | | `Name` | The name you assigned to the Inference Endpoint. | | `Status` | The current status of the Inference Endpoint (e.g., `Running`, `Stopped`, etc.). | | `URL` | The base URL of the Inference Endpoint, which you can use to query the model. | | `Playground URL` | The URL of the Inference Playground, a user-friendly interface to interact with your deployed model. | | `Dashboard URL` | The URL of the Inference Endpoint dashboard, where you can monitor the performance and usage of your model. | **Configuration** | Field | Description | | - | - | | `Device Architecture` | The architecture of the device where the Inference Endpoint is running (e.g., `nvidia`). | | `Runtime Args` | The vLLM runtime arguments that were used to deploy the Inference Endpoint. These can be customized when creating or updating the Inference Endpoint. | | `HF Token Secret Name` | The name of the FlexAI Secret that contains the Hugging Face Access Token, if applicable. This is only shown if the Inference Endpoint requires a Hugging Face Access Token to access the model. | | `API Key Secret Name` | The name of the FlexAI Secret that contains the API Key used to authenticate requests to the Inference Endpoint. | **The Activity tab** The Activity tab provides you with a timeline of events related to your Inference Endpoint, including deployment status changes, scaling events, and more. *** **The Logs tab** The Logs tab provides you with real-time logs from your Inference Endpoint, allowing you to monitor its activity and troubleshoot any issues that may arise. You can use the **Search bar** input field to filter the logs by a specific keyword. This is useful to quickly find relevant information in the logs. *** ##### `flexai inference list` The `flexai inference list` command offers a quick overview of the current **status** and **HTTP endpoint** of your Inference Endpoints. ```bash theme={null} flexai inference list ``` Output: ```text title="'flexai inference list' response" wrap=false theme={null} NAME │ STATUS │ AGE │ ENDPOINT ───────────┼──────────────────┼─────┼────────────────────────────────────────────────────────────────────────────────── qwen-small │ starting │ 32s │ https://inference-e0ed242b-dbd3-423e-b525-76beac222d44-d41b1b44.platform.flex.ai ``` #### `flexai inference inspect` The [`flexai inference inspect`](/cli/reference/inference/inspect) command provides detailed information about a specific Inference Endpoint, including its current status, configuration, and lifecycle. ```bash theme={null} flexai inference inspect qwen-small ``` This will display your endpoint's full configuration, runtime status, and connection details. ##### `flexai inference logs` ```bash theme={null} flexai inference logs qwen-small ``` Output: ```text title="flexai inference logs response" wrap=false theme={null} [...] [INFO] Starting inference for qwen-small [INFO] Inference completed successfully [...] ``` ## Managing your Inference Endpoint The Inference Endpoints table's Actions column provides a set of actions that you can use to manage your Inference Endpoint: * **Configure**: To access its Details panel. * **Pause**: To temporarily stop the Inference Endpoint without deleting it. * **Delete**: To permanently remove the Inference Endpoint. * **Resume**: To restart a paused Inference Endpoint. *** The FlexAI CLI provides a set of commands that you can use to manage your Inference Endpoints: * [`flexai inference delete`](/cli/reference/inference/delete) - Deletes an Inference Endpoint. * [`flexai inference scale`](/cli/reference/inference/scale) - Allows for the definition of scaling policies for an Inference Endpoint. * [`flexai inference stop`](/cli/reference/inference/stop) - Stops an Inference Endpoint. *** # Querying an Inference Endpoint Source: https://docs.flex.ai/core-services/inference/quickstart/query Query a deployed inference endpoint using the playground or HTTP requests ## The Playground The FlexAI Inference Playground is a convenient way to interact with your deployed models without needing to write any code —or cURL requests. It allows you to test your Inference Endpoint using a user-friendly interface. It is a FlexAI-hosted instance of [Chainlit](https://chainlit.io/), a UI tool for interacting with multimodal AI models. Screenshot of the FlexAI Inference Playground You can get the Inference **Playground URL** by opening the Inference Endpoint's **Details** drawer menu. You will find the URL under the default **Details** tab's Summary section. ## Querying the Inference Endpoint through code As long as you have the correct **Endpoint URL**, **Access key**, **model name**, and the right request body format, you can use any HTTP client to interact with your Inference Endpoint. Request: ```bash "/v1/completions" {3} {5} title="curl Request" wrap=false theme={null} curl -X POST https://inference-e0ed242b-dbd3-423e-b525-76beac222d44-7de0c0f4.platform.flex.ai/v1/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer 4279eecf-bb20-4c58-9745-f18fd1e764bb" \ -d '{ "model": "Qwen/Qwen2.5-1.5B-Instruct", "prompt": "What is love?", "max_tokens": 256 }' ``` Response: ```json {12} title="curl response (rendered as JSON by using jq)" theme={null} { "id": "chatcmpl-e6c2d199-b57d-4904-a1cb-89d604b146d9", "object": "text_completion", "created": 1748238665, "model": "Qwen/Qwen2.5-1.5B-Instruct", "choices": [ { "index": 0, "message": { "role": "assistant", "reasoning_content": null, "content": "Love is a powerful and meaningful emotion that brings people close together. It is a connection between two individuals that transcends time and space, supporting and nurturing a deep and deepening relationship. Love is the foundation that holds relationships together, reflecting in the happiness, relationships, and harmony surrounding us in every aspect of life. To understand love and its complexities, one would need to delve deep into the arena of psychology, philosophy, anthropology, literature, and social sciences. It is an elusive but natural human reaction, serving as an enabling force for the bonding of our species, and it is an integral aspect of human nature.", "tool_calls": [] }, "logprobs": null, "finish_reason": "stop", "stop_reason": null } ], "usage": { "prompt_tokens": 20, "total_tokens": 162, "completion_tokens": 142, "prompt_tokens_details": null }, "prompt_logprobs": null, "kv_transfer_params": null } ``` Request: ```javascript {5} {8} title="Node.js Fetch Request" wrap=false theme={null} const response = await fetch('https://inference-e0ed242b-dbd3-423e-b525-76beac222d44-d41b1b44.platform.flex.ai/v1/completions', { method: 'POST', headers: { 'Content-Type': 'application/json', 'Authorization': 'Bearer 0ad394db-8dba-4395-84d9-c0db1b1be0a8' }, body: JSON.stringify({ model: 'Qwen/Qwen2.5-1.5B-Instruct', prompt: 'What is love?', max_tokens: 256 }) }); const data = await response.json(); console.log(data); ``` Response: ```json {12} title="Node.js Fetch response" theme={null} { "id": "chatcmpl-e6c2d199-b57d-4904-a1cb-89d604b146d9", "object": "text_completion", "created": 1748238665, "model": "Qwen/Qwen2.5-1.5B-Instruct", "choices": [ { "index": 0, "message": { "role": "assistant", "reasoning_content": null, "content": "Love is a powerful and meaningful emotion that brings people close together. It is a connection between two individuals that transcends time and space, supporting and nurturing a deep and deepening relationship. Love is the foundation that holds relationships together, reflecting in the happiness, relationships, and harmony surrounding us in every aspect of life. To understand love and its complexities, one would need to delve deep into the arena of psychology, philosophy, anthropology, literature, and social sciences. It is an elusive but natural human reaction, serving as an enabling force for the bonding of our species, and it is an integral aspect of human nature.", "tool_calls": [] }, "logprobs": null, "finish_reason": "stop", "stop_reason": null } ], "usage": { "prompt_tokens": 20, "total_tokens": 162, "completion_tokens": 142, "prompt_tokens_details": null }, "prompt_logprobs": null, "kv_transfer_params": null } ``` Request: ```python {7} {10} title="Python Request" wrap=false theme={null} import requests import json url = "https://inference-e0ed242b-dbd3-423e-b525-76beac222d44-d41b1b44.platform.flex.ai/v1/completions" headers = { "Content-Type": "application/json", "Authorization": "Bearer 0ad394db-8dba-4395-84d9-c0db1b1be0a8" } data = { "model": "Qwen/Qwen2.5-1.5B-Instruct", "prompt": "What is love?", "max_tokens": 256 } response = requests.post(url, headers=headers, json=data) formatted_json = json.dumps(response.json(), indent=2) print(formatted_json) ``` Response: ```json {12} title="Python Request response" theme={null} { "id": "chatcmpl-e6c2d199-b57d-4904-a1cb-89d604b146d9", "object": "text_completion", "created": 1748238665, "model": "Qwen/Qwen2.5-1.5B-Instruct", "choices": [ { "index": 0, "message": { "role": "assistant", "reasoning_content": null, "content": "Love is a powerful and meaningful emotion that brings people close together. It is a connection between two individuals that transcends time and space, supporting and nurturing a deep and deepening relationship. Love is the foundation that holds relationships together, reflecting in the happiness, relationships, and harmony surrounding us in every aspect of life. To understand love and its complexities, one would need to delve deep into the arena of psychology, philosophy, anthropology, literature, and social sciences. It is an elusive but natural human reaction, serving as an enabling force for the bonding of our species, and it is an integral aspect of human nature.", "tool_calls": [] }, "logprobs": null, "finish_reason": "stop", "stop_reason": null } ], "usage": { "prompt_tokens": 20, "total_tokens": 162, "completion_tokens": 142, "prompt_tokens_details": null }, "prompt_logprobs": null, "kv_transfer_params": null } ``` # Training Source: https://docs.flex.ai/core-services/training Train AI models from scratch or continue training existing models with FlexAI Train AI models from scratch or continue training existing models using FlexAI's powerful training infrastructure. You can train public and private models hosted on the Hugging Face Model Hub or models you've previously fine-tuned with FlexAI. ## Key Features Scale across multiple GPUs and nodes Automatic saving and resuming Debug and experiment interactively Track GPU usage and performance ## Anatomy of a Training Job A Training Job in FlexAI consists of the following components: 1. **Training Script**: The script that defines the training process, including data loading, model training, evaluation, Checkpoint generation, and so on. 2. **Dataset**: The Dataset you will use to train your model. * It can be an existing [FlexAI Dataset](/platform-services/dataset-manager/), or it can be pulled/streamed from an external source during runtime. 3. [**Secrets**](/platform-services/secret-manager/): Any sensitive information (e.g., API keys, passwords) required for the training process. 4. **Hyperparameters**: Configuration settings that control the training process (e.g., learning rate, batch size). ### Training Script The training script is a Python script that defines the training process. It handles tasks such as: * Loading the base model Checkpoint from the `/input-checkpoint/` path, or downloading it from an external source. * Loading a Dataset mounted through the FlexAI Dataset Manager on the `/input/` path or from an external source. * Performing any necessary data processing steps. * Reading the values of any Secrets passed to the Training Job's runtime environment. * Applying any hyperparameters configurations passed to the Training Script. * Running the training loop and periodically saving Checkpoints to the `/output-checkpoints/` path. ### Dataset A Training Job generally leverages a Dataset to adjust a model's weights using new data. The Dataset can be: * An existing [FlexAI Dataset](/platform-services/dataset-manager/). * Pulled/streamed from an external source during runtime. The Dataset can be in any format, and it can contain any kind of files. The Training Script is responsible for loading and processing the data as needed. ### Secrets Secrets are used to securely pass sensitive information to the Training Job's runtime environment. This can include API keys, passwords, or any other confidential data required for the training process. Secrets are managed through the [FlexAI Secret Manager](/platform-services/secret-manager/) and are injected into the Training Job's environment as environment variables. ### Hyperparameters Hyperparameters are configuration settings that control the behavior of the Training Script. They can include parameters such as learning rate, batch size, number of epochs, where to load the dataset from, where to save the model checkpoints, how often to save checkpoints, and so on. ## Key Concepts ### Checkpoints FlexAI automatically manages Checkpoints during training, allowing you to resume, analyze, fetch, export or deploy model Checkpoints as [Inference Endpoints](/core-services/inference/). Every time your code calls the `torch.save()` function to save a model Checkpoint to the `/output-checkpoints/` directory, the [FlexAI Checkpoint Manager](/platform-services/checkpoint-manager/) will be triggered and manage the saved Checkpoint, making it available for later use. ### A Training Job's Lifecycle Learn more about the different statuses a Training Job can have in the [Training Job Lifecycle](/core-services/training/lifecycle/) page. ### Third-Party Integration Your FlexAI Training Jobs can interact with third-party APIs and services by leveraging the [FlexAI Secret Manager](/platform-services/secret-manager/) to securely store sensitive information such as API keys and tokens. These secrets can then be injected into the Training Job's runtime environment as environment variables, allowing your Training scripts to access them securely and enabling flexible and powerful AI workflows. ## CLI Reference The [`flexai training` command](/cli/reference/training) manages Training Jobs: you can start new jobs, monitor their progress, inspect details, fetch output artifacts, and manage their lifecycle from the command line. ## A Training Job's Lifecycle Learn more about the different statuses a Training Job can have in the [Training Job Lifecycle](/core-services/training/lifecycle/) page. ## Getting Started FlexAI training jobs can be launched in a few steps. The Quickstart guide will walk you through the process of preparing your data and starting your first training job. Here's a brief overview of the steps involved: 1. Preparing your dataset for training. 2. Launching a training job using FlexAI. 3. Monitoring training progress and managing checkpoints. The button below will lead you to the FlexAI Training Quickstart guide's overview, where you'll find more details. # Lifecycle Source: https://docs.flex.ai/core-services/training/lifecycle Understand the status progression of a training job from creation to completion A Training Job progresses through several statuses: * **pending**: The workload has been registered and its creation process has begun. * **scheduling**: FlexAI is searching for the suitable hardware resources to run the workload. * **building**: The Runtime environment is being prepared, including cloning the source repository on the specified revision and installing the project's dependencies. * **in progress**: The workload is currently executing the code defined in the entry point file. * **enqueued**: The workload is waiting for resources to become available. * **succeeded**: The workload completed successfully and output artifacts are ready for download. * **failed**: The workload did not complete successfully. Common reasons include missing files, dependency errors, or exceeding time limits. * **stopped**: The workload was stopped by the user before completion. Terminal statuses (**succeeded**, **failed**, **stopped**) mean the job is finished and can be deleted if desired. ```mermaid theme={null} stateDiagram-v2 [*] --> pending pending --> scheduling scheduling --> building scheduling --> enqueued building --> in_progress: in progress building --> enqueued enqueued --> in_progress: in progress enqueued --> scheduling in_progress --> succeeded: succeeded [Terminal] in_progress --> failed: failed [Terminal] in_progress --> stopped: stopped [Terminal] succeeded --> [*] failed --> [*] stopped --> [*] ``` # Training a model with FlexAI Source: https://docs.flex.ai/core-services/training/quickstart Step-by-step guide to training a model on FlexAI ## Overview This Quickstart tutorial will walk you through the steps needed to train a model on **FlexAI**. By the end of this tutorial, you will have successfully trained a version of [nanoGPT](https://github.com/karpathy/nanoGPT) that has been optimized to make it readily usable. This enhanced version is available at: [https://github.com/flexaihq/nanoGPT](https://github.com/flexaihq/nanoGPT). The process of training a model consists of **4 main steps**: ### Prerequisites You should have a FlexAI account. If you don't have one, you can [sign up for a free account](/signup). ### 1. Loading a *Dataset* A **Dataset** is the collection of files that you want to use to train your model. You can upload files from your local machine or sync them in from a remote location when hosted by a third party Storage Provider (e.g., S3, GCS, R2, etc.). The Dataset files can be in any format, such as text, images, or audio. ### 2. Running a *Training Job* **Training Job** is the name assigned to the FlexAI component that represents the process of executing training code on the FlexAI platform. Creating a Training Job requires a name, at least one Dataset, a link to a GitHub repository with the training code, and the path to the *entry point script* (the file that contains the code that will be executed when the Training Job begins execution). A number of optional flags allow for the customization of a Training Job's execution, such as: * Specifying how many accelerators (GPUs) to use and across how many nodes. * Setting Environment Variables and Secrets that will then be passed to the Training Runtime so they can be used by the training scripts. * Specifying a previous *checkpoint* to resume execution from. * Setting a specific *revision* (branch, tag, or commit) of the code repository to use. Additionally, any number of **Hyperparameters** can be specified passed to the *entry point script*. These Hyperparameters can be used to control the behavior of the training code, such as the learning rate, batch size, or number of epochs. ### 3. Getting the Training Job's details Once the Training Job is running, you can monitor its progress, view logs, evaluate its performance and resource usage, all through the FlexAI CLI, FlexAI's hosted TensorBoard, and FlexAI's Dashboard UI ### 4. Fetching the Training Job's output The output of a Training Job is the result of the training process, which can include: * Checkpoints: Saved states of the model at different points during training. * Files written to the `/output` directory by the training scripts, such as binaries, logs, metrics, or any other files that you want to keep as a result of a successful Training Job completion. # Checking the Training Job's Details Source: https://docs.flex.ai/core-services/training/quickstart/checking-details Inspect the details and status of your training job ## Training Summary ### General Details about your Training Jobs The "All Trainings" table in the FlexAI Console provides a summary of all your Training Jobs. You can select the gear icon ⚙️ (labeled as *Configure*) in the `Actions` field of the Training Jobs list page. This will open a "Details" panel. The **Details** tab will be selected by default, showing all the relevant information about your Training Job. | Field | Description | | - | - | | `Name` | The name you assigned to the Training Job. | | `Status` | The current status of the Training Job (e.g., `pending`, `scheduling`, `building`, `in progress`, `succeeded`, `failed`, `stopped`, etc.). | | `Created At` | Workload creation age. | You can learn more about the different Training Job statuses on the [Lifecycle](/core-services/training/lifecycle) page. ### General Details about your Training Jobs You can use the `list` command to get a table with general information about all the Training Jobs you have access to through your FlexAI account: ```bash theme={null} flexai training list ``` This provides an output similar to the following: ```text theme={null} NAME | DEVICE | NODE | ACCELERATOR | DATASET | REPOSITORY | STATUS | AGE ------------------------+--------+------+-------------+-----------------+-------------------------------------+----------+------ quickstart-training-job | nvidia | 1 | 1 | nanoGPT-dataset | https://github.com/flexaihq/nanogpt | building | 15s ``` ## Training Configuration | Field | Description | | - | - | | `Dashboard URL` | The URL of the Training Job dashboard, where you can monitor the performance and resource usage of your Training Job. | | `Tensorboard Dashboard URL` | The URL of the FlexAI-hosted TensorBoard dashboard, where you can visualize the training process of your models. | | `Node Count` | The number of nodes allocated to the Training Job. | | `Accelerator Count` | The number of accelerators (GPUs) allocated to the Training Job. | | `Repository URL` | The URL of the Git repository containing your training code. | | `Repository Revision` | The specific commit or branch of the repository that was used to create the Training Job. | | `Repository Revision SHA` | The SHA hash of the specific commit or branch of the repository that was used to create the Training Job. | | `Entry Point` | The entry point script along with its arguments. | | `Datasets` | The datasets that were attached to the Training Job. | | `Environment` | The environment variables and secrets that were set for the Training Job. Displayed in a Key-Value pai format where the Key is the name of the environment value within the Training Runtime, and the value is either the raw value (for Environment Variables) or the name of the **FlexAI secret** containing the secret value. | | `Checkpoints` | The checkpoints that were created during the Training Job. These are stored in the FlexAI object storage and can be used to resume training or to create an Inference Endpoint (depending on the type of model). | You can have a deeper look at the Training Job status using the [flexai training inspect](/cli/reference/training/inspect) command. Especially useful for debugging purposes: ```bash theme={null} flexai training inspect quickstart-training-job ``` Below you will find an example of the output you will get when running the `inspect` command: ```yaml theme={null} kind: Training metadata: name: quickstart-training-job id: 75179cc2-ec63-4f93-b4da-44e49ea86049 creatorUserID: 16e289cc-c81b-4a15-91d9-0e2aae00a317 ownerOrgId: 270a5476-b91a-442f-8a13-852ef7bb5b9c config: device: nvidia nodes: 1 accelerator: 1 entrypoint: - train.py - config/train_shakespeare_char.py - --out_dir=/output-checkpoint - --max_iters=1500 datasetsNames: - nanoGPT-dataset checkpointName: "" sourceName: "" repositoryURL: https://github.com/flexaihq/nanogpt repositoryRevision: main secrets: [] environment: [] runtime: status: succeeded queuePosition: 0 repositoryRevisionSha: 116799dbae7b0fe33caf1b90f73a72f84bc32adc selectedAgentId: k8s-training-sesterce-001-CLIENT-PROD-client-prod lifecycleEvents: - type: AgentSelection status: ResponseReceived message: |- Cluster Scheduling result{ Name: aws-cloud AgentID: k8s-training-aws-001-CLIENT-PROD-client-prod Response: NoAnswer Conditions: [NonSchedulable: NoAnswer] } raisedAt: "2025-06-30T11:41:54Z" - type: AgentSelection status: ResponseReceived message: |- Cluster Scheduling result{ Name: sesterce-h100-bm-01 AgentID: k8s-training-sesterce-001-CLIENT-PROD-client-prod Response: OK Conditions: [] } raisedAt: "2025-06-30T11:41:54Z" - type: AgentSelection status: ResponseReceived message: |- Cluster Scheduling result{ Name: sesterce-h200-bm-01 AgentID: k8s-training-sesterce-002-CLIENT-PROD-client-prod Response: NoAnswer Conditions: [NonSchedulable: NoAnswer] } raisedAt: "2025-06-30T11:41:54Z" - type: AgentSelection status: ResponseReceived message: |- Cluster Scheduling result{ Name: sesterce-l40s-bm-01 AgentID: k8s-training-sesterce-003-CLIENT-PROD-client-prod Response: NoAnswer Conditions: [NonSchedulable: NoAnswer] } raisedAt: "2025-06-30T11:41:54Z" - type: AgentSelection status: ResponseReceived message: |- Cluster Scheduling result{ Name: sesterce-a100-bm-01 AgentID: k8s-training-sesterce-004-CLIENT-PROD-client-prod Response: NoAnswer Conditions: [NonSchedulable: NoAnswer] } raisedAt: "2025-06-30T11:41:54Z" - type: AgentSelection status: ResponseReceived message: |- Cluster Scheduling result{ Name: k8s-training-smc-001 AgentID: k8s-training-smc-001-CLIENT-PROD-client-prod Response: NoAnswer Conditions: [NonSchedulable: NoAnswer, OrgNotAuthorized] } raisedAt: "2025-06-30T11:41:54Z" - type: AgentSelection status: Completed message: Selected agent k8s-training-sesterce-001-CLIENT-PROD-client-prod raisedAt: "2025-06-30T11:41:54Z" - type: BuildSubmission status: Succeeded message: Build request sent to flex-agent raisedAt: "2025-06-30T11:41:54Z" - type: BuildExecution status: Succeeded message: Build completed with image rg.fr-par.scw.cloud/paas-trainings-client-prod/9f9c379c-8d46-419b-8bf5-d0b0986a6dd9-arch_nvidia-1x1@sha256:0d854f75f698a549d2a8a0e024e930383b885bdac2863ee0cf74ebdc8a8f358c raisedAt: "2025-06-30T11:41:54Z" - type: TrainingPreparation status: Succeeded message: Training trainings-client-prod/training-75b79cc2-ec63-4f93-b4da-44e49a4a6049-zqg6d created raisedAt: "2025-06-30T11:41:54Z" - type: TrainingExecution status: InProgress message: Training in progress raisedAt: "2025-06-30T11:42:00Z" - type: TrainingExecution status: Succeeded message: Training complete, output available raisedAt: "2025-06-30T11:43:48Z" createdAt: "2025-06-30T11:41:54Z" lastUpdate: "2025-06-30T11:43:48Z" ``` ```json theme={null} { "kind": "Training", "metadata": { "name": "quickstart-training-job", "id": "75179cc2-ec63-4f93-b4da-44e49ea86049", "creatorUserID": "16e289cc-c81b-4a15-91d9-0e2aae00a317", "ownerOrgId": "270a5476-b91a-442f-8a13-852ef7bb5b9c" }, "config": { "device": "nvidia", "nodes": 1, "accelerator": 1, "entrypoint": [ "train.py", "config/train_shakespeare_char.py", "--out_dir=/output-checkpoint", "--max_iters=1500" ], "datasetsNames": ["nanoGPT-dataset"], "checkpointName": "", "sourceName": "", "repositoryURL": "https://github.com/flexaihq/nanogpt", "repositoryRevision": "main", "secrets": [], "environment": [] }, "runtime": { "status": "succeeded", "queuePosition": 0, "repositoryRevisionSha": "116799dbae7b0fe33caf1b90f73a72f84bc32adc", "selectedAgentId": "k8s-training-sesterce-001-CLIENT-PROD-client-prod", "lifecycleEvents": [ { "type": "AgentSelection", "status": "ResponseReceived", "message": "Cluster Scheduling result{\n Name: aws-cloud\n AgentID: k8s-training-aws-001-CLIENT-PROD-client-prod\n Response: NoAnswer\n Conditions: [NonSchedulable: NoAnswer]\n}\n", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "AgentSelection", "status": "ResponseReceived", "message": "Cluster Scheduling result{\n Name: sesterce-h100-bm-01\n AgentID: k8s-training-sesterce-001-CLIENT-PROD-client-prod\n Response: OK\n Conditions: []\n}\n", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "AgentSelection", "status": "ResponseReceived", "message": "Cluster Scheduling result{\n Name: sesterce-h200-bm-01\n AgentID: k8s-training-sesterce-002-CLIENT-PROD-client-prod\n Response: NoAnswer\n Conditions: [NonSchedulable: NoAnswer]\n}\n", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "AgentSelection", "status": "ResponseReceived", "message": "Cluster Scheduling result{\n Name: sesterce-l40s-bm-01\n AgentID: k8s-training-sesterce-003-CLIENT-PROD-client-prod\n Response: NoAnswer\n Conditions: [NonSchedulable: NoAnswer]\n}\n", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "AgentSelection", "status": "ResponseReceived", "message": "Cluster Scheduling result{\n Name: sesterce-a100-bm-01\n AgentID: k8s-training-sesterce-004-CLIENT-PROD-client-prod\n Response: NoAnswer\n Conditions: [NonSchedulable: NoAnswer]\n}\n", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "AgentSelection", "status": "ResponseReceived", "message": "Cluster Scheduling result{\n Name: k8s-training-smc-001\n AgentID: k8s-training-smc-001-CLIENT-PROD-client-prod\n Response: NoAnswer\n Conditions: [NonSchedulable: NoAnswer, OrgNotAuthorized]\n}\n", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "AgentSelection", "status": "Completed", "message": "Selected agent k8s-training-sesterce-001-CLIENT-PROD-client-prod", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "BuildSubmission", "status": "Succeeded", "message": "Build request sent to flex-agent", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "BuildExecution", "status": "Succeeded", "message": "Build completed with image rg.fr-par.scw.cloud/paas-trainings-client-prod/9f9c379c-8d46-419b-8bf5-d0b0986a6dd9-arch_nvidia-1x1@sha256:0d854f75f698a549d2a8a0e024e930383b885bdac2863ee0cf74ebdc8a8f358c", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "TrainingPreparation", "status": "Succeeded", "message": "Training trainings-client-prod/training-75b79cc2-ec63-4f93-b4da-44e49a4a6049-zqg6d created", "raisedAt": "2025-06-30T11:41:54Z" }, { "type": "TrainingExecution", "status": "InProgress", "message": "Training in progress", "raisedAt": "2025-06-30T11:42:00Z" }, { "type": "TrainingExecution", "status": "Succeeded", "message": "Training complete, output available", "raisedAt": "2025-06-30T11:43:48Z" } ], "createdAt": "2025-06-30T11:41:54Z", "lastUpdate": "2025-06-30T11:43:48Z" } } ``` ## Next Steps Once a Training Job is running, you can monitor its progress by checking its logs and leveraging the FlexAI Observability Services. You'll learn more about this in the next step of the Quickstart Tutorial: [Monitoring a Training Job's Progress](/core-services/training/quickstart/monitoring-progress/). # Creating a Training Job Source: https://docs.flex.ai/core-services/training/quickstart/creating Create and configure a new training job on FlexAI With a Dataset available on your FlexAI account, you can now create a Training Job that will use it. ## The Model's repository For this tutorial we will use the [**FlexAI fork of the nanoGPT repository**](https://github.com/flexaihq/nanogpt), originally created by [Andrej Karpathy](https://github.com/karpathy). A Training Job requires at least a **Name**, a link to a **GitHub repository** where its code resides, and the ***path to the entry point script*** that will initiate the Workload. In addition, the *entry point script* can be followed by any arguments required, such as configuration files or Hyperparameters. ### Entry Point script arguments The entry point script path for this quickstart tutorial is [`./train.py`](https://github.com/flexaihq/nanoGPT/blob/main/train.py), and it expects the following arguments: * `config/train_shakespeare_char.py`: A configuration file, which contains the default Workload Parameters. * `--dataset_dir`: The path within the `/input` directory of the Workload Runtime where the Dataset files are located. * `--out_dir`: The output directory, which will be mounted into the Workload Runtime as `/output-checkpoint`. * `--max_iters`: The maximum number of iterations to run the Workload script for (optional). These include any **Environment Settings** and **Hyperparameters** the entry point script may require. For this tutorial: | Parameter | Type | Description | | - | - | - | | `config/train_shakespeare_char.py` | Environment Setting | A positional argument pointing to a configuration file used by nanoGPT's `train.py` script to set the default Workload Parameters | | `--out_dir=/output-checkpoint` | Environment Setting | The output directory where the Workload script will write checkpoint files. In order to take advantage of FlexAI's Managed Checkpoints feature, this **`should always be /output-checkpoint`** | | `--max_iters=1500` | Hyperparameter | The maximum number of iterations to run the Workload script for. This is an optional hyperparameter that can be used to tweak the Workload execution | ## Starting a new Training Job The **Start a new Training job** form consists of a set of required and optional fields that you can use to customize your deployment. ### To open the **Start a new Training job** form Either: * Follow the direct link to the [Start a new Training Job](https://platform.flex.ai/training/create) page. Or Navigate to the **Training** section from either the navigation bar or the card on the home page. Select the **New** button to display the creation form. A drawer menu with the creation form will be displayed. ### Required Fields * **Name**: Your Training Job name. Should follow the [resource naming conventions](/best-practices/resource-naming-conventions/). * **Repository URL**: The URL of the Git repository containing your Training code. * **Entry Point**: The path to the entry point script in your repository that will initiate the Training Job. * The *entry point script* can be followed by any arguments you want to pass to it, such as configurations and Hyperparameters. **Value**: `train.py config/train_shakespeare_char.py --dataset_dir=my_dataset --out_dir=/output-checkpoint --max_iters=1500`. ### Other fields * **Repository Revision**: The Git revision (branch, tag, or commit) you want to use for this Training Job. The `main` branch will be used by default. * **Node Count**: The number of nodes you want to use for this Training Job. Defaults to `1`. * This will determine the amount of Accelerators you will have available for your Training Job: * 1 node will allow you to use up to 8 Accelerators. * Using more than 1 node will make all 8 Accelerators per Node available to your Training Job. * **Accelerator Count**: The number of Accelerators you want to use for this Training Job. Must follow the logic described above. Defaults to `1`. * **Datasets**: Can be selected from a dropdown list of the datasets you want to use for this Training Job. You can add multiple datasets as well as specify the mount path within the Training Runtime (they will be mounted under `/input`). You can read more about this in the [Pushing a Dataset guide](/core-services/training/quickstart/uploading-a-dataset/). After picking a Dataset, select the "Add" button — otherwise it won't be added to the Training Job. * **Environment Variables & Secrets**: Add any environment variables you want to set for this Training Job. These will be available to your Training code as environment variables within the Training Runtime. * You can also reference **Secrets**, which will be securely injected into the Training Job's Runtime. * **Cluster**: The cluster where the Training workload will run on. It can be selected from a dropdown list of available clusters in your FlexAI account. A default cluster will be automatically selected for you if none is specified. ### Form Values | Field Name | Value | | - | - | | **Name** | `nanoGPT-flexai-console` | | **Repository URL** | `https://github.com/flexaihq/nanogpt` | | **Repository Revision** | `main` | | **Node Count** | `1` | | **Accelerator Count** | `1` | | **Entry Point** | `train.py config/train_shakespeare_char.py --dataset_dir=my_dataset --out_dir=/output-checkpoint --max_iters=1500` | | **Datasets** | Dataset: `nanoGPT-dataset` (from the CLI quickstart), Mount Directory: `my_dataset` | | **Cluster** | *Your organization's designated cluster* | After filling out the form, select the **Submit** button to start the Training Job. You should get a confirmation message indicating that the Training Job creation process has been initiated successfully. The **Start a new training job** form will close and you will be redirected to the Training Jobs list page, where you can see your newly created Training Job in the list. Considering the minimum required elements for the creation of a Training Job, the following command will initiate its creation and start it running immediately: ```bash theme={null} flexai training run quickstart-training-job \ --dataset nanoGPT-dataset=my_dataset \ --repository-url https://github.com/flexaihq/nanogpt \ -- train.py config/train_shakespeare_char.py --dataset_dir=my_dataset --out_dir=/output-checkpoint --max_iters=1500 ``` #### Arguments | FlexAI command Argument | Value | Description | | - | - | - | | **Training Job Name** | `quickstart-training-job` | The name of the Training Job | #### Flags | Flag | Value | Description | | - | - | - | | **Dataset** | `my_dataset=nanoGPT-dataset` | The mount path followed by the Dataset name. The dataset will be accessible at `/input/my_dataset` | | **Repository URL** | `https://github.com/flexaihq/nanogpt` | The URL of the GitHub repository containing the workload's code | | **Entry Point Script** | `train.py` | The path of entry point Training script as defined by the repository | #### Entry Point script arguments These include any **Environment Settings** and **Hyperparameters** the entry point script may require. Keep in mind that these are specific to the code you're running: | Entry point script argument | Type | Description | | - | - | - | | `config/train_shakespeare_char.py` | Environment Setting | A positional argument pointing to a configuration file used by nanoGPT's `train.py` script to set the default runtime Parameters | | `--out_dir=/output-checkpoint` | Environment Setting | The output directory where the script will write checkpoint files. In order to take advantage of FlexAI's Managed Checkpoints feature, this **`should always be /output-checkpoint`** | | `--max_iters=1500` | Hyperparameter | The maximum number of iterations to run. This is an optional hyperparameter that can be used to tweak the Workload execution | ## Up next Next you'll learn how to get a Training Job's details and monitor its progress. # Getting a Training Job's Output Source: https://docs.flex.ai/core-services/training/quickstart/getting-output Retrieve output files and results from a completed training job ## FlexAI Managed Checkpoints [FlexAI's Managed Checkpoints](/platform-services/checkpoint-manager/) feature enables you to get the final result of your Training Job after it completes, as well as being able to get intermediate checkpoints generated by your Training script. The only thing you need to do is to make sure your Training script calls the `torch.save()` function and writes its output to the path specified by the `FLEXAI_OUTPUT_CHECKPOINT_DIR` environment variable. FlexAI's Managed Checkpoints will handle the rest. Once the Training Job is running, every time its code calls the `torch.save()` function, FlexAI's Managed Checkpoints feature will automatically capture a Checkpoint and store it in the `/output-checkpoint` directory. Each Checkpoint will be assigned a unique ID and its creation time will be recorded. This means that you can go to a specific point in time and retrieve the state of the model at that moment, allowing you to resume training from that point or evaluate the model's performance on a validation dataset. After a Training Job completes, the last Checkpoint will be the one with the most recent creation timestamp. You can retrieve checkpoints generated by *FlexAI's Managed Checkpoints* at any point, which allows you to go back to a previous point in the past to resume training, to test your model, or to use it for inference. ### Listing Checkpoints You can list all available checkpoints for a specific Training Job by running the [`flexai training checkpoints`](/cli/reference/training/checkpoints) command: ```bash theme={null} flexai training checkpoints quickstart-training-job ``` This will return a table with a list of Checkpoint IDs and their corresponding creation timestamps, similar to the following: ```text theme={null} ID │ TIMESTAMP ──────────────────────────────────────┼──────────────────────────────────── 50e5ec69-32b6-e483-9c49-38a73cc34294 │ 2025-06-30 12:42:55.214 +0100 WEST 82d21263-8ba8-dd73-9c61-732d3b7b0adc │ 2025-06-30 12:43:01.77 +0100 WEST 32d07a60-61cc-4598-b4f6-2073a4f8d0af │ 2025-06-30 12:43:14.734 +0100 WEST ``` ## Getting Checkpoints You can select the gear icon ⚙️ (labeled as *Configure*) in the `Actions` field of the Training Jobs list page. This will open a "Details" panel. The **Details** tab will be selected by default, showing all the relevant information about your Training Job. Navigate to the **Checkpoints** tab to view the list of checkpoints created during the Training Job. Each checkpoint entry includes details such as: * Actions: * Download * Deploy: If the Checkpoint is an "Inference-ready Checkpoint", you can deploy it directly as an Inference Endpoint * Created: * Creation date and age * Training Loss: The reported training loss at the time the checkpoint was created * Evaluation Loss: The reported evaluation loss at the time the checkpoint was created * Status: The status of the checkpoint (`available` or `processing`) Once you have the desired Checkpoint ID, you can download it to your host machine using the [`flexai checkpoint fetch`](/cli/reference/checkpoint/fetch) command: ```bash theme={null} flexai checkpoint fetch 32d07a60-61cc-4598-b4f6-2073a4f8d0af ``` ```text theme={null} Writing in: /home/diego/ckpt.pt Progress: 0.4% (1.31 MB / 343.79 MB) // ... Progress: 100% (343.79 MB / 343.79 MB) ``` You can use this checkpoint file to resume training from the exact point it was saved, or to evaluate the model's performance on a validation dataset. **Exporting a Checkpoint** Use the [`flexai checkpoint export`](/cli/reference/checkpoint/export) command to export a Checkpoint to a remote location by using a previously registered Remote Storage Connection, such as S3, GCS, MinIO or R2. This allows you to store your checkpoints in a more permanent location for later use. ## Workload Raw Outputs Fetching raw outputs from the `/output` directory is done through the FlexAI CLI — see the **Using the FlexAI CLI** tab. Any data written to the `/output` directory will be compressed into a zip file and made available to you via the [`flexai training fetch`](/cli/reference/training/fetch) command: ```bash theme={null} flexai training fetch quickstart-training-job ``` This will download a `.zip` file to the current working directory on your host machine. Once extracted you'll get a local directory named `output` it will contain any files written to the `/output` directory by the training scripts. For this quickstart example we configured the training code to write to the checkpoints directory `/output-checkpoint` directory when we set the value of `--out_dir`, so the `/output` directory won't be used. However, you can run another Training Job that instead writes out to the `/output` directory. In this case, only the last checkpoint will be saved in the `/output` directory as a result of the training process. Note that your code can use both locations (`/output` and `/output-checkpoint`) simultaneously if needed. # Monitoring a Training Job's Progress Source: https://docs.flex.ai/core-services/training/quickstart/monitoring-progress Monitor your training job's progress with logs and metrics ## Logs You can select the gear icon ⚙️ (labeled as *Configure*) in the `Actions` field of the Training or Fine-tuning Jobs list page. This will open a "Details" panel. Select the **Logs** tab to view the messages emitted to the standard output (`stdout`) created by the runtime environment build process and your own code. You can use the **Search bar** input field to filter the logs by a specific keyword. This is useful to quickly find relevant information in the logs. Once a Training Job begins, logs emitted to the standard output (`stdout`) created by the runtime environment build process and your own code can be retrieved by running the [`flexai training logs`](/cli/reference/training/logs) command: ```bash theme={null} flexai training logs quickstart-training-job ``` This will output a stream of logs including both the FlexAI runtime execution logs and your code's messages. ## Infrastructure Metrics You can monitor the infrastructure metrics of your Training Job using the FlexAI Infrastructure Monitor. This will give you insights into the resource usage of your Training Job, such as CPU and memory usage, disk I/O, and network traffic. Access FlexAI's Infrastructure Monitor by visiting [https://dashboards.flex.ai/](https://dashboards.flex.ai/). Visit the [FlexAI Infrastructure Monitor](/platform-services/observability/infrastructure-monitor/) page to learn more. ## TensorBoard You can also use FlexAI's hosted TensorBoard to visualize the training process of your model. TensorBoard provides a suite of tools for inspecting and understanding your Training Job's evolution. Visit [https://dashboards.flex.ai/tensorboard](https://dashboards.flex.ai/tensorboard) and log in using your credentials. Learn more at the [FlexAI TensorBoard](/platform-services/observability/tensorboard/) page. ## Next Steps After a few minutes, your Training Job should have completed successfully! The next step of this Quickstart Tutorial will guide you through the process of getting its outputs or *results*. # Uploading a Dataset Source: https://docs.flex.ai/core-services/training/quickstart/uploading-a-dataset Upload and prepare a dataset for your training job ## Download The Sample Dataset Files You can upload files or directories from your local machine to create a new Dataset. For this quickstart tutorial, we will use two pre-generated files: `train.bin` and `val.bin` that you can download to your machine by using the links below: * [train.bin](https://github.com/flexaihq/nanoGPT/raw/refs/heads/prepare_dataset/data/shakespeare_char/dataset/train.bin): Training dataset * [val.bin](https://github.com/flexaihq/nanoGPT/raw/refs/heads/prepare_dataset/data/shakespeare_char/dataset/val.bin): Validation dataset for model evaluation Download the pre-generated training dataset files for this quickstart tutorial: ```bash theme={null} curl -L --remote-name-all https://github.com/flexaihq/nanoGPT/raw/refs/heads/prepare_dataset/data/shakespeare_char/dataset/{train.bin,val.bin} ``` This downloads two binary files: * `train.bin`: Training dataset (Shakespeare character-level data) * `val.bin`: Validation dataset for model evaluation ## Uploading files to the FlexAI Dataset Manager Service Navigate to the Dataset Manager in the FlexAI Console: Visit the ["Add Dataset" section of the FlexAI Console](https://platform.flex.ai/datasets/new) Enter a name for your dataset: `nanoGPT-dataset` Must follow the FlexAI [Resource Naming conventions](/best-practices/resource-naming-conventions/) Select the "Local" option for "Upload Origin" Select the `+ Upload Item` button to open the "Upload Items" dialog Use the "Select file" option to open a file browser dialog: Select both `train.bin` and `val.bin` files from your local machine * Depending on your system and browser, you might need to hold down the Ctrl/Cmd key while selecting multiple files * When selecting multiple files, your browser might prompt you with a confirmation message asking you to allow multiple file selection. * You can also select files individually if you prefer In the "Destination Path" field, enter `shakespeare_char` Select the `Add` button to confirm the file selection and destination mapping The "Upload Items" dialog will close and you will get back to the "Add a Dataset" form, where you will see the files you just added listed under the "Upload Items" section. Here you will see a list of files similar to the one below: ``` shakespeare_char/ ├── train.bin (1.91MB) └── val.bin (217.85KB) ``` Below the file list you will find an `+ Add items` button that will open up the "Upload Items" dialog again, in case you want to add more files or directories to the Dataset. Finally, select the `Add Dataset` button to start the upload process. Use the `flexai dataset push` command to create and upload your Dataset: ```bash theme={null} flexai dataset push nanoGPT-dataset \ --file train.bin=shakespeare_char/train.bin \ --file val.bin=shakespeare_char/val.bin ``` The command above has the following components: | Component | Value | Description | | - | - | - | | **Dataset Name** | `nanoGPT-dataset` | Name for the Dataset in FlexAI. Must follow the FlexAI [Resource Naming conventions](/best-practices/resource-naming-conventions/). | | **File Mapping** | `train.bin=shakespeare_char/train.bin` | Maps local file to dataset path | | **File Mapping** | `val.bin=shakespeare_char/val.bin` | Maps local file to dataset path | ## Dataset structure Once upload is complete, you can select the gear icon ⚙️ (labeled as *Configure*) in the `Actions` field of the Dataset list page. This will open the Dataset "Details" panel where you will be able to see the Dataset's name, status, creation date, and a list of files that were uploaded as part of the Dataset. ### Summary * **Name**: nanoGPT-dataset * **Status**: Ready * **Creation Time**: 8/5/2025, 2:13:07 PM ### Details ``` shakespeare_char/ ├── train.bin (1.191MB) └── val.bin (217.85KB) ``` To learn more about the ways your workloads can access Datasets, check out the [Runtime Access](/platform-services/dataset-manager/#runtime-access) section of the Dataset Manager overview page. Check that your dataset was uploaded successfully: ```bash theme={null} flexai dataset list ``` Expected output: ```text theme={null} NAME │ FILES COUNT │ TOTAL SIZE │ STATUS │ CREATED AT ─────────────────────┼─────────────┼────────────┼───────────┼──────────────────────────── nanoGPT-dataset │ 2 │ 2.13 MB │ available │ 2025-08-05 13:13:07 (1h) ``` The status should show `available` when the upload is complete and the dataset is ready for training. You can run the `flexai dataset inspect ` command to get more detailed information about your Dataset: ```bash theme={null} flexai dataset inspect nanoGPT-dataset ``` Which will output something like: ```yaml theme={null} kind: Dataset metadata: name: nanoGPT-dataset id: 1b541f62-2faf-4e32-8fd1-a6bc27e26b58 creatorUserID: 16e289cc-c81b-4a15-91d9-0e2aae00a317 ownerOrgId: 270a5476-b91a-442f-8a13-852ef7bb5b9c spec: fromLocalFiles: - train.bin - val.bin storageProvider: "" sourcePath: "" status: status: available storageProviderID: 00000000-0000-0000-0000-000000000000 size: 2.13 MB files: - path: shakespeare_char/train.bin size: 1.91 MB - path: shakespeare_char/val.bin size: 217.85 KB createdAt: 2025-08-05 14:13:07 (57d) updatedAt: 2025-08-05 14:13:08 (57d) dataSyncs: [] ``` ```json theme={null} { "kind": "Dataset", "metadata": { "name": "nanoGPT-dataset", "id": "1b541f62-2faf-4e32-8fd1-a6bc27e26b58", "creatorUserID": "16e289cc-c81b-4a15-91d9-0e2aae00a317", "ownerOrgId": "270a5476-b91a-442f-8a13-852ef7bb5b9c" }, "spec": { "fromLocalFiles": ["train.bin", "val.bin"], "storageProvider": "", "sourcePath": "" }, "status": { "status": "available", "storageProviderID": "00000000-0000-0000-0000-000000000000", "size": "2.13 MB", "files": [ { "path": "shakespeare_char/train.bin", "size": "1.91 MB" }, { "path": "shakespeare_char/val.bin", "size": "217.85 KB" } ], "createdAt": "2025-08-05 14:13:07 (57d)", "updatedAt": "2025-08-05 14:13:08 (57d)", "dataSyncs": [] } } ``` # Frequently Asked Questions: FlexAI Source: https://docs.flex.ai/faq Answers to common questions about the FlexAI platform ## The stack & requirements ### What is the supported stack? * Python: 3.11, 3.12 * PyTorch: 2.7, 2.7.1, 2.8 * CUDA: 12.8, 12.9 * ROCm: 6.3, 6.4 * Hardware available: Nvidia, AMD Exact versions for your environment can be found by running `flexai training runtime list` or `flexai inference runtime list`. ### Which Git providers are supported? * FlexAI supports both private and public **GitHub** repositories. ## Checkpoints ### Does FlexAI offer managed Checkpoints? Yes, FlexAI offers Managed checkpoints out of the box, so you don't need to set anything up. Every time your code calls `torch.save`, FlexAI Cloud training Services stores its output so you can resume your Training Job from a previous point in case of a failure or if you need to go back to a specific state. Checkpoints should be saved to the `/output-checkpoint` directory, which is automatically mounted to your Training Job's container. You can refer to it by using the `FLEXAI_OUTPUT_CHECKPOINT_DIR` environment variable. ### How do I download a Managed Checkpoint? Managed checkpoints can be downloaded even if the Training Job is still running. You can download a checkpoint by using the `flexai checkpoint fetch ` command. ### What is the maximum size Managed Checkpoints currently support? Managed Checkpoints will work with checkpoints of up to 5GB in size. If your checkpoints are larger than 5GB, the Managed Checkpoints feature will not be enabled. However, you can continue to write checkpoints to the `/output/` (also available through the `FLEXAI_TRAINING_OUTPUT_PATH` environment variable) directory without a problem, you'll only have to wait until the Training Job completes before being able to fetch the contents of the `/output/` directory using the `flexai training fetch ` command. ### Do Managed Checkpoints work when running a Training Job across multiple nodes? Yes, you can use Managed Checkpoints when running a Training Job across multiple nodes. You should make sure only one node is in charge of calling `torch.save` to avoid conflicts. ### How do I resume a Training Job? You can resume a Training Job from a Checkpoint by creating a new Training Job using `flexai training run` with the `--checkpoint` flag and passing the name or ID of the Checkpoint you want to resume from. Your code should look for the checkpoint file in the `/input-checkpoint/` directory, which is also available through the `FLEXAI_INPUT_CHECKPOINT_DIR` environment variable. ## Datasets ### What is the maximum size of a Dataset that I can use with FlexAI? We don't set a limit on the size of the Dataset you can use with FlexAI. ### Dataset and output Artifact management * Dataset will be made available for your training scripts in the `/input/` directory, which can also be referenced using the `FLEXAI_TRAINING_INPUT_PATH` environment variable. * Dataset names are used as the name of the directory in which the Dataset is stored: * For example, if you have a Dataset called `my_dataset`, it will be available in the `/input/my_dataset/` directory. * The `/input/` directory and its contents are read-only. * Your training scripts should write any output artifacts or the results of data processing operations to the `/output/` directory. * Data written to any other directory will be erased after the Training Job is completed (successfully or otherwise). * Once a Training Job is completed, the contents of the `/output/` directory will be made available for download by using the `flexai training fetch `. ### What file types are supported for Datasets? We have no restrictions on the file types that can be uploaded to FlexAI. You can upload any file type that is required for your Training Job, whether it be images, text, audio, or video files, tokenized data, or any other file type. ## Observability & Monitoring ### How can I monitor Infrastructure Metrics for my Training and Fine-tuning Jobs? You can use the [FlexAI Infrastructure Monitor](https://dashboards.flex.ai) to monitor the performance of your Training and Fine-tuning Jobs. It provides real-time system and GPU metrics that will enable you to make optimizations to your training scripts so you can take full advantage of FlexAI compute resources. The data retention period is 30 days. ### How can I monitor my Training and Fine-tuning Jobs? You can use the [FlexAI TensorBoard](https://tensorboard.flex.ai) to monitor your Training and Fine-tuning Jobs. It provides real-time visualizations of your training metrics, such as loss and accuracy, and allows you to compare different runs. The data retention period is 30 days. ## The FlexAI platform ### How are my code dependencies managed? FlexAI's training runtime handles the installation of dependencies via a `requirements.txt` file. The FlexAI Runtime environment will look for it in the root of your code repository and install the dependencies listed there. You can also specify a custom location for the `requirements.txt` file using [the `-q`/`--requirements-path` flag](/cli/reference/training/run/#--requirements-path) when running a Training or Fine-tuning Job. ### How do I specify the version of my PyTorch-associated dependencies? You should not include references to `torch`, `torchvision` or `torchaudio` in the `requirements.txt` file. FlexAI will install the correct version of these libraries based on the Runtime environment. ### Which types of clients can I use to interact with FlexAI? * [FlexAI Console](https://platform.flex.ai): a web-based interface. * [FlexAI CLI](/cli/install/): which is available for Linux, macOS, and Windows (through WSL2). ### Where is the CLI executed? The FlexAI CLI is executed on your host machine (non-graphical environments, such as GitHub Codespaces or a remote shell, require a [few additional steps](/best-practices/alternative-methods/logging-from-a-remote-env/)) and communicates with FlexAI via the internet. ### Is it possible to access more compute/GPU capacity? Yes, it is possible to have more capacity available. Please reach out to us to discuss your needs. ### Does FlexAI offer shared or dedicated environments? We have a common pool of compute resources with strict isolation between user environments. If you're interested in getting a dedicated environment, please let us know. ## FlexAI and 3rd party Compute Providers ### Can I use FlexAI on my current infrastructure? Yes, we can support this through our Compute Provider partnerships. Please reach out to us for more. ### As a Compute Provider, can I offer FlexAI as a solution to my users by leveraging my compute power? Yes, FlexAI can be offered by Compute Providers to their users. Please reach out to us for a deep dive. # Frequently Asked Questions: FlexAI Credits Source: https://docs.flex.ai/faq/credits How FlexAI credits work, where they come from, and how your usage is billed ## How billing works ### How do I pay for FlexAI? Pay as you go. You're billed for what you actually use — per token for inference, per GPU-hour for training and fine-tuning — at the rates published on [flex.ai/pricing](https://www.flex.ai/pricing). There's no subscription, and no commitment beyond your usage. ### Do I need a card to get started? No. You can [create an account](https://platform.flex.ai/signup), create an API key, browse the model catalog, and run the in-browser demos on [flex.ai](https://www.flex.ai) without entering any payment details. What a card unlocks is spending. Inference draws against your account balance, so until the account is funded — by a top-up or by credit — requests come back `402 Payment Required` rather than running. That lets you wire a key into your code and your deploy pipeline before any money is involved. When you're ready to send real traffic, add a billing address and card from the [billing page](https://platform.flex.ai/billing) and top up by any amount from \$5. *** ## Where credits come from ### How do I get FlexAI credits? Credits arrive with a program or an agreement rather than automatically at signup: * **Verified first top-up match** — verify your identity and we match your first top-up, dollar for dollar, up to \$100. US-dollar accounts only, and claimed by emailing us rather than applied automatically. See [how the match works](#how-does-the-first-top-up-match-work) below. * **FlexAI Startup Program** — accepted startups receive \$1,000 in Token Factory credit, then 50% off their first three months and 30% off the next three. [Apply on flex.ai/startups](https://www.flex.ai/startups). * **Negotiated and invited offers** — for an evaluation, a proof of concept, or a committed-use agreement, talk to [commercial@flex.ai](mailto:commercial@flex.ai?subject=Request%20about%20FlexAI%20credits). Credit never arrives automatically at signup — each of the above is something you apply for, qualify for, or claim. If you are looking for the monthly free-credit offer advertised previously, it was retired on 2026-08-20. ### How does the first top-up match work? Verify your identity, then make your first top-up, and we add the same amount again as credit — up to a \$100 match. A \$25 top-up becomes \$50 of balance; a \$100 top-up becomes \$200. Above \$100 the match stays at \$100. There is no minimum beyond the platform's own \$5 top-up floor, so a \$10 first top-up earns the full \$10. Two conditions, both deliberate: * **Verification comes first.** The match is tied to a verified identity rather than to an email address, which is what lets us offer it at this size. * **It is once per account, and once per verified person.** It applies to the first top-up you make after verifying — so topping up before you verify does not use it up — and verifying again from a second account does not earn a second match. **The match is on US-dollar accounts only.** Accounts in the EU region bill in euros (see the changelog entry on EU billing), and the offer does not apply to them. Both individuals and companies are eligible. To claim it, email [support@flex.ai](mailto:support@flex.ai?subject=Verified%20first%20top-up%20match) and we'll walk you through verification and apply the match — we're handling these by hand while the self-serve flow is being finished, so it won't appear as a button in the dashboard yet. ### Are existing users eligible for credits? Yes — credits aren't limited to new accounts. Reach out to [commercial@flex.ai](mailto:commercial@flex.ai?subject=Request%20about%20FlexAI%20credits) with your workload and we'll tell you what applies. *** ## Using credits ### How is credit usage calculated? Credits draw down against your usage at the same published rates you'd otherwise pay: per million input and output tokens for inference, per GPU-hour for training and fine-tuning. Per-model rates are in [the model catalog](https://www.flex.ai/models), and the full picture is on [flex.ai/pricing](https://www.flex.ai/pricing). Your balance draws down before your card is charged, so a credited account bills nothing until the credit is used. ### Do credits expire? Each grant carries its own terms, including any expiry, and those terms are stated when the credit is applied to your account. Check the [billing page](https://platform.flex.ai/billing) for your current balance, or ask [commercial@flex.ai](mailto:commercial@flex.ai?subject=Question%20about%20my%20FlexAI%20credit%20terms) if you're unsure which terms apply to you. ### What happens when my credits run out? For the Inference API, requests return `402 Payment Required` and start succeeding again as soon as you top up — no key regeneration needed. See [Billing & Quotas](/inference-api/reference/billing) for the exact response and how to add funds. For training and fine-tuning workloads, [talk to us](mailto:commercial@flex.ai?subject=FlexAI%20credit%20balance) before you run out and we'll help you plan the transition. ### How do I track my credit usage? Your balance and spend are on the [billing page](https://platform.flex.ai/billing), updated within a few minutes of each request. ### Will I be notified if credits are running low? Yes. You'll get an email when your balance falls below a fifth of what it was after your last top-up, and another if it reaches zero. If you have auto-recharge switched on we send only the second one — the rule exists to top you up before the first would fire, so a zero balance with auto-recharge on means a recharge did not go through. We also reach out directly on larger accounts to discuss next steps before anything interrupts. *** ## Rules and restrictions ### Can I transfer credits to someone else? No. Credits are tied to the account they're granted to. They aren't transferable, and credits from separate accounts can't be pooled. ### Can I withdraw credits as cash? No. Credits are usable on the FlexAI platform only. ### Can I combine credits with other offers? Credits generally don't stack with other promotional offers. Where a program says otherwise — the Startup Program's credit and its staged discounts, for instance — the program's own terms apply. *** ## Need more help? Email [support@flex.ai](mailto:support@flex.ai?subject=Question%20about%20FlexAI%20credits) or join our [Slack community](https://flexaicommunity.slack.com/ssb/redirect). For anything commercial — credits, committed use, or a quote — [commercial@flex.ai](mailto:commercial@flex.ai?subject=Request%20about%20FlexAI%20credits) is the faster route. # Getting Started Source: https://docs.flex.ai/getting-started Get started with FlexAI platform You can sign up for FlexAI by visiting the [FlexAI Web Console's Sign Up page](https://platform.flex.ai/signup). Then follow the [Signing Up instructions](/signup/) to create your account. FlexAI empowers AI builders everywhere to develop groundbreaking AI solutions with effortless access to universal AI compute. FlexAI provides a unified platform that handles the complexities of allocating and setting up the infrastructure needed to run your AI workloads: From managing hardware, software, networking, and storage layers, FlexAI orchestrates the entirety of the process of securely and efficiently running AI workloads. ## How it works The FlexAI platform offers a seamless experience for AI developers to run their workloads using the client of their choice: * [**FlexAI Web Console**](/console/): A web-based interface for managing your AI workloads ([https://platform.flex.ai/](https://platform.flex.ai/)). * [**FlexAI CLI**](/cli/install/): The command line interface for interacting with the FlexAI platform. Both the FlexAI CLI and the FlexAI Web Console offer similar features, however, the CLI exposes a more advanced set of functionalities at the moment. The Web Console is continuously being improved to match the CLI's capabilities, so you can expect more features to be continuously added. ## FlexAI Core Services FlexAI provides three Core Services: 1. [**Inference**](/core-services/inference/): Deploy your models for real-time inference and integrate them into your applications. 2. [**Fine-Tuning**](/core-services/fine-tuning/): Easily fine-tune pre-trained models on your specific datasets to adapt them to your needs. 3. [**Training**](/core-services/training/): Train your models on FlexAI's powerful infrastructure with just a few clicks or commands. ## FlexAI Platform Services The FlexAI Core Services are supported by a set of Platform Services that help you efficiently manage your AI workloads and resources: 1. [**Checkpoint Manager**](/platform-services/checkpoint-manager/): Manage your model checkpoints, including uploading, downloading, and organizing them. 2. [**Code Registry Manager**](/platform-services/code-registry-manager/): Manage your code repositories, allowing you to version control and share your training and fine-tuning scripts. 3. [**Dataset Manager**](/platform-services/dataset-manager/): Manage your datasets, including uploading, organizing, and accessing them for your training and fine-tuning jobs. 4. [**Secret Manager**](/platform-services/secret-manager/): Securely store and manage sensitive information such as API keys, passwords, and tokens. 5. [**Remote Storage Connection Manager**](/platform-services/remote-storage-connections-manager/): Connect and manage your remote storage solutions, such as AWS S3, Google Cloud Storage, and Azure Blob Storage. 6. [**Observability Services**](/platform-services/observability/): Monitor and analyze the performance of your AI workloads with built-in logging and metrics. # FlexAI Token Factory Source: https://docs.flex.ai/index An OpenAI-compatible inference API for open models — one key, every model, billed per token.

Every model you and your agent need — one OpenAI-compatible key.

FlexAI Token Factory is an OpenAI-compatible inference API for open text, code, reasoning, vision, and embedding models. Point any OpenAI SDK at api.flex.ai and your existing code works unchanged.

Create your API key with no card, then add one when you're ready to spend — pay per token, no packs, no subscriptions. See billing.

Get an API key Read the quickstart
## Drop-in OpenAI compatibility Already have code that calls OpenAI? Change the base URL and key — nothing else. ```bash cURL theme={null} curl https://api.flex.ai/v1/chat/completions \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "Meta-Llama-3.1-8B-Instruct-FP8", "messages": [{"role": "user", "content": "Say hello in one line."}] }' ``` ```python Python theme={null} # pip install openai import os from openai import OpenAI client = OpenAI( base_url="https://api.flex.ai/v1", api_key=os.environ["FLEXAI_API_KEY"], ) resp = client.chat.completions.create( model="Meta-Llama-3.1-8B-Instruct-FP8", messages=[{"role": "user", "content": "Say hello in one line."}], ) print(resp.choices[0].message.content) ``` ```typescript TypeScript theme={null} // npm install openai import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.flex.ai/v1", apiKey: process.env.FLEXAI_API_KEY, }); const resp = await client.chat.completions.create({ model: "Meta-Llama-3.1-8B-Instruct-FP8", messages: [{ role: "user", content: "Say hello in one line." }], }); console.log(resp.choices[0].message.content); ``` Browse the live catalog at [flex.ai/models](https://flex.ai/models), then follow the full [Quickstart](/inference-api/quickstart) to get a key and make your first request. *** ## Why Token Factory A drop-in for the OpenAI SDK. Swap the base URL and model id — keep your code, prompts, and tools. Tool calls, streaming, structured output, and vision — all native. See the [streaming](/inference-api/guides/streaming) and [vision](/inference-api/guides/vision) guides. Text, code, reasoning, vision, and embedding models behind one key. Filter the live catalog from code with [model discovery](/inference-api/guides/model-discovery). Dollar-denominated, per-token billing — no token packs, no subscriptions. Account-level budgets and rate limits. *** ## Start building What the API is, what you get, and where everything lives. Get a key and make your first request in under two minutes. Streaming, tool use, vision, embeddings, model discovery, batching, and concurrency. Endpoints, the compatibility matrix, errors, and billing. *** ## Scale beyond serverless The serverless API is the fastest way to start. When you need more, the FlexAI platform also offers dedicated inference endpoints, fine-tuning, training, and private deployments. Dedicated endpoints, fine-tuning, training, and platform services. Pricing, the full product story, and scaling to private AI cloud. ***

Ready to build?

Get an API key and make your first request in minutes.

Get an API key Talk to us
Need help? Email [support@flex.ai](mailto:support@flex.ai), join our [Slack community](https://join.slack.com/t/flexaicommunity/shared_invite/zt-3fqfcq9hj-Bv_Ehtyip0Y6fjS7gG5hHg), or check [status.flex.ai](https://status.flex.ai). # LangChain & LangGraph Source: https://docs.flex.ai/inference-api/agents/langchain Point a LangChain chat model or a LangGraph agent at FlexAI with four lines of config, and pick a model that has been measured to hold up in a tool-calling loop. LangChain and LangGraph work against FlexAI's OpenAI-compatible API today: `ChatOpenAI` takes a `base_url`, so you set it to `https://api.flex.ai/v1`, pass your FlexAI key, name a served model, and every LangChain surface — chat, streaming, tool calling, agents, structured output — runs unchanged. There is no package to install beyond LangChain's own. The headline for agent builders: every model in the [table below](#which-models-to-use) completed a two-tool LangGraph agent loop against production as written here, and a model that failed any check does not appear on this page. New to FlexAI? [Create an account](https://platform.flex.ai/signup?utm_source=docs\&utm_medium=referral\&utm_campaign=langchain), create a workspace, and mint an `sk-…` key from **Workspace home** — no card needed for any of that. You will need a funded balance before requests are served; the [quickstart](/inference-api/quickstart) walks through both. Export the key as `FLEXAI_API_KEY` for the snippets on this page. ## Connect ```bash theme={null} pip install -U langchain langchain-openai langgraph ``` ```python theme={null} import os from langchain_openai import ChatOpenAI llm = ChatOpenAI( model="DeepSeek-V4-Flash-0731", base_url="https://api.flex.ai/v1", api_key=os.environ["FLEXAI_API_KEY"], max_tokens=4096, ) print(llm.invoke("In one sentence, what is a GPU?").content) ``` That is the whole integration. `model` is an id that `GET /v1/models` returns right now — the serving subset of the [catalog](https://flex.ai/models); a model offered for dedicated deployment or not currently serving is absent from that list and fails immediately. Ids are case sensitive. `max_tokens=4096` matters as soon as tools are involved; [what to plan for](#what-to-plan-for) explains why. If you would rather not pass the URL and key in code, LangChain reads the OpenAI SDK's environment variables: ```bash theme={null} export OPENAI_BASE_URL=https://api.flex.ai/v1 export OPENAI_API_KEY=$FLEXAI_API_KEY ``` and `ChatOpenAI(model="DeepSeek-V4-Flash-0731", max_tokens=4096)` connects the same way. ## Stream Streaming works with `.stream()` and `.astream()`. Set `stream_usage=True` so token usage arrives at the end of the stream — that is the `stream_options.include_usage` flag the [streaming guide](/inference-api/guides/streaming) asks for, and it keeps your own accounting in step with your bill. Usage rides on its own chunk after the last piece of text (LangChain then emits one empty closing chunk), so add the chunks together and read `usage_metadata` off the total rather than off the last object: ```python theme={null} llm = ChatOpenAI( model="DeepSeek-V4-Flash-0731", base_url="https://api.flex.ai/v1", api_key=os.environ["FLEXAI_API_KEY"], max_tokens=4096, stream_usage=True, ) full = None for chunk in llm.stream("Write two short sentences about GPUs."): print(chunk.content, end="", flush=True) full = chunk if full is None else full + chunk print(full.usage_metadata) # {'input_tokens': ..., 'output_tokens': ..., 'total_tokens': ...} ``` The *Streaming* column in the table passes only when this snippet streamed text in more than one chunk and usage arrived after the last text chunk. ## Call tools Define tools with `@tool`, bind them, and feed results back as `ToolMessage`s. The round trip below is the pattern LangGraph automates in the next section. ```python theme={null} import json from langchain_core.messages import HumanMessage, ToolMessage from langchain_core.tools import tool @tool def get_weather(city: str) -> dict: """Return current weather for a city.""" return {"city": city, "temperature_c": 21, "conditions": "clear"} llm_with_tools = llm.bind_tools([get_weather]) messages = [HumanMessage("What's the weather in Paris?")] ai = llm_with_tools.invoke(messages) # the model asks for get_weather messages.append(ai) for call in ai.tool_calls: result = get_weather.invoke(call["args"]) messages.append(ToolMessage(content=json.dumps(result), tool_call_id=call["id"])) final = llm_with_tools.invoke(messages) # the model answers with the result print(final.content) ``` Two details of the gateway make this loop simpler than on some providers. An assistant turn that is a pure tool call may carry `content` as `null`, omitted, or `""` — the gateway accepts all three and normalizes the empty string, so whatever your LangChain version serializes is fine. A `user` message directly after a `tool` message is the other one, and here the honest answer is narrower: the gateway does bridge that gap, but only for the specific models whose chat template is known to insist on strict role alternation, and that list is short. LangGraph's own loop does not produce the shape — it sends the tool result and lets the model answer — so most agents never hit it. If you are building a multi-turn chat where the user types again straight after a tool result and you see a `400` about message ordering, insert an assistant turn between them. This page's probe does not measure that case, so the table says nothing about it. To force a tool call, bind with `tool_choice="any"` (LangChain's name for OpenAI's `"required"`). Every model in the table honored it in the last run, with one thing worth knowing about how that was measured: the probe binds a single tool, and for a single-tool request the gateway can rewrite a forced call into a named one, which the serving stack enforces more reliably. On `gpt-oss-120b` that rewrite is what the column is reporting — with two or more tools bound, generic `required` is not enforced at the engine, and a turn where the model answers in prose instead of calling a tool comes back as a `4xx`. If you need a forced call across several bound tools on that model, name the tool you want rather than relying on `required`. Leave `tool_choice` at its default when you want the model to decide. ## Run a LangGraph agent `create_agent` (LangChain 1.x, built on LangGraph) gives you the ReAct loop — model turn, tool calls, tool results, repeat until the model answers. With two tools the model has to chain them: ```python theme={null} from langchain.agents import create_agent from langchain_core.tools import tool @tool def get_weather(city: str) -> dict: """Return current weather for a city.""" return {"city": city, "temperature_c": 21, "conditions": "clear"} @tool def convert_c_to_f(celsius: float) -> float: """Convert a Celsius temperature to Fahrenheit.""" return round(celsius * 9 / 5 + 32, 1) agent = create_agent( llm, tools=[get_weather, convert_c_to_f], system_prompt="You are a concise assistant. Use the tools when they help.", ) state = agent.invoke({"messages": [{"role": "user", "content": "What's the weather in Paris, in Fahrenheit?"}]}) print(state["messages"][-1].content) ``` `state["messages"]` holds the full trace — the two tool calls, their results and the final answer — so you can log or replay it. The *Agent loop* column in the table passes only when that trace shows the chain in order: one turn calling `get_weather`, its result, a later turn calling `convert_c_to_f` with the temperature the weather tool returned, its result, and a final answer that carries the converted value. Both calls in one turn, the converter first, or an answer that ignores the conversion do not pass. If you have graphs built on the older `langgraph.prebuilt.create_react_agent`, they run against FlexAI with the same `llm`; LangGraph 1.x marks that entry point deprecated in favour of `create_agent`, but nothing about the provider changes either way. ## Get structured output `with_structured_output` returns a Pydantic object instead of text. Which method to use depends on what the model advertises in `supported_parameters` on `GET /v1/models` — see [model discovery](/inference-api/guides/model-discovery): * `structured_outputs` listed → strict JSON Schema, `method="json_schema"` (LangChain's default for `ChatOpenAI`). The gateway enforces your schema, so a plain instruction is enough. * only `response_format` listed → JSON mode, `method="json_mode"`. The model promises valid JSON, not your field names, so spell the keys out in the prompt. ```python theme={null} from pydantic import BaseModel, Field class Weather(BaseModel): city: str temperature_c: float = Field(description="Temperature in Celsius") conditions: str # Models advertising `structured_outputs`: the schema is enforced for you. structured = llm.with_structured_output(Weather, method="json_schema") print(structured.invoke("It's 21°C and clear in Paris. Extract the weather.")) # Models advertising only `response_format`: name the keys, then parse. json_mode = llm.with_structured_output(Weather, method="json_mode") print(json_mode.invoke( "It's 21°C and clear in Paris. Extract the weather as JSON with exactly these keys: " "city (string), temperature_c (number), conditions (string)." )) ``` The *Structured output* column in the table records which method each model passed with. The json\_mode rows were measured with the keyed prompt above — without it, every json\_mode model returned valid JSON under its own field names and the Pydantic parse failed, which is the behaviour JSON mode promises — and then with a second prompt that never asks for JSON at all under `response_format: json_object`, which only an enforced grammar can turn into JSON. A model whose runtime accepts JSON mode without enforcing it fails that second check and is left off the table. A model that lists neither parameter returns `400` with `param: "response_format"` rather than guessing — the [compatibility matrix](/inference-api/reference/openai-compatibility) has the exact contract. ## Which models to use Every row below comes from running this page's snippets against production with `scripts/agents_langchain_probe.py` in the token-service repository: one chat call, a stream, the tool round trip above, the two-tool `create_agent` loop above, structured output with the method the model advertises, and a forced tool call. A model appears here only if it passed every one of those checks; the roster is FlexAI's priced production catalog, and models that were not available on the serverless API at run time, or that failed a check, are left off and counted in the footer. **yes** means it worked as written; the *Structured output* column also names the method that passed. Models are listed by id. | Model | Chat | Streaming | Tool calling | Agent loop | Structured output | `tool_choice="required"` | First call | | - | - | - | - | - | - | - | - | | `DeepSeek-V4-Flash-0731` | yes | yes | yes | yes | yes (json\_schema) | yes | 0.5 s | | `GLM-5.2` | yes | yes | yes | yes | yes (json\_mode) | yes | 4.9 s | | `GLM-5.3-Flash` | yes | yes | yes | yes | yes (json\_mode) | yes | 2.6 s | | `MiniMax-M2.7` | yes | yes | yes | yes | yes (json\_mode) | yes | 2.1 s | | `Qwen3-30B-A3B-Thinking-2507-FP8` | yes | yes | yes | yes | yes (json\_mode) | yes | 3.2 s | | `Qwen3-Coder-30B-A3B-Instruct-FP8` | yes | yes | yes | yes | yes (json\_mode) | yes | 0.6 s | | `Qwen3.6-27B-FP8` | yes | yes | yes | yes | yes (json\_mode) | yes | 10.9 s | | `Qwen3.6-35B-A3B-FP8` | yes | yes | yes | yes | yes (json\_schema) | yes | 3.3 s | | `Qwen3.8-27B` | yes | yes | yes | yes | yes (json\_mode) | yes | 1.7 s | | `Step-3.7-Flash` | yes | yes | yes | yes | yes (json\_mode) | yes | 7.0 s | | `gemma-4-26B-A4B-it` | yes | yes | yes | yes | yes (json\_schema) | yes | 0.5 s | | `gemma-4-31b-it` | yes | yes | yes | yes | yes (json\_schema) | yes | 0.8 s | | `gpt-oss-120b` | yes | yes | yes | yes | yes (json\_schema) | yes | 0.7 s | Measured on 2026-09-17 against `https://api.flex.ai/v1` with `scripts/agents_langchain_probe.py` in the token-service repository, whose full run record — every model probed, including those this table omits — is committed beside it under `scripts/agents_langchain_probe_runs/`. "First call" is one cold request from a laptop and is indicative only, not a benchmark. 8 served models that failed a check in this run are left off the table and kept in the run record. 22 further rows in the priced catalog could not be measured because they were not answering on the serverless API at run time, so they are left off without a verdict. For the models serving right now, call `GET /v1/models`. There is no context column here on purpose. A model's window is catalog metadata rather than something this page measures, and it moves on its own schedule, so any number printed here would start drifting the day it was written. Read `context_length` from `GET /v1/models` and size prompts against that. One caveat worth knowing: the advertised window can sit above what the engine currently accepts, so treat a `400` on a very long prompt as the engine's limit rather than a bug in your client. Prices are on [flex.ai/pricing](https://flex.ai/pricing); this page does not repeat them so they cannot go stale here. ## What to plan for * **Give tool turns output headroom.** Several models write tool calls in a verbose native markup that the engine parses into `tool_calls`. If `max_tokens` cuts it off you get `finish_reason: "length"` and no `tool_calls` — to an agent that looks like the model narrated an action and stopped. `max_tokens=4096` on the `ChatOpenAI` constructor is the floor for tool turns; treat `length` without `tool_calls` as a truncation to retry with more budget. * **Stream long agent turns.** Non-streaming requests are bounded by a roughly 95-second generation budget, and `llm.stream(...)` lifts it. `agent.stream(...)` on its own does not: LangGraph's default stream mode streams graph state, and the agent node still calls the model without streaming, so the same budget applies to each turn. Pass `stream_mode="messages"` (or use `astream_events`) to make LangChain issue a streaming request to the model. * **Read the window from the API.** Prompts over a model's `context_length` return `400`. Size against the served number, not the model card. * **`tool_choice="required"` is per model.** The table says which served models honor a forced call. Default `tool_choice` works everywhere tool calling does. * **The agent-loop column is about chaining, not parallelism.** It passes only when a model calls the two tools in order, feeding the first tool's result into the second. A model that emits both calls in one turn fails it and is left off this table — which is a statement about dependent chains, not a defect: emitting several tool calls at once is a capability, and most tool-capable models we host can do it (the [tool-use guide](/inference-api/guides/tool-use) covers that case). Separately, a few models are served with single-call tool parsing and return `400` with `This model only supports single tool-calls at once!` when asked for two at once. So if your agent needs parallel tool calls, this table is the wrong filter — use it when your tools depend on each other. * **Reasoning defaults differ by model, and reasoning tokens bill as output.** Models fall into three groups, and the group a given model is in can change as the fleet moves, so confirm with one plain call rather than trusting a list. *Off unless asked:* `DeepSeek-V4-Flash-0731` turns on with `ChatOpenAI(reasoning_effort="high")`, and the hybrid-thinking models — `gemma-4-26B-A4B-it`, `gemma-4-31b-it`, `Qwen3.6-35B-A3B-FP8` and `GLM-5.2` among them — with `ChatOpenAI(extra_body={"thinking": {"type": "enabled"}})` (the raw engine switch, `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`, works too). *Its own dial, no off:* `gpt-oss-120b` takes `reasoning_effort` at `low`, `medium` or `high`; it cannot be switched off, so a disable is the same `400` as the thinking-only group below, and `low` is the least it will reason. *Thinking-only:* some models reason unconditionally and reject an attempt to disable it with a `400` saying the model is thinking-only — a name containing `Thinking` is the usual hint, and for those the only lever is `max_tokens`. To place a model yourself, send one plain call and look for `reasoning_content` in `response_metadata`, then try the disable switch once and see whether it is accepted. Whenever reasoning is on, raise `max_tokens` to cover it. Use `extra_body` for these switches, not `model_kwargs`: the OpenAI SDK rejects unknown top-level parameters, and `extra_body` is how they reach the request body. The [tool-use guide](/inference-api/guides/tool-use#agent-loops-on-deepseek-v4-flash-0731) covers the DeepSeek switches in detail. ## FAQ Yes. LangGraph agents talk to the model through LangChain's `ChatOpenAI`, and FlexAI serves the OpenAI chat-completions API at `https://api.flex.ai/v1`. Set `base_url` and `api_key` on `ChatOpenAI`, name a model from the table, and `create_agent` (or an existing `create_react_agent` graph) runs unchanged. Any model in the table completed the two-tool agent loop against production; `DeepSeek-V4-Flash-0731` is the model most of FlexAI's coding-agent traffic runs on. The table is measured, so trust it over a model card. No. LangChain's `ChatOpenAI` with `base_url="https://api.flex.ai/v1"` is the integration; `pip install langchain-openai` is the only provider dependency. The model ran out of output budget while writing the call. Set `max_tokens=4096` or more on `ChatOpenAI` for tool turns and retry; see [what to plan for](#what-to-plan-for). The raw OpenAI-SDK round trip, parallel calls, and the agent-loop notes for DeepSeek-V4-Flash-0731. The exact fields and endpoints the gateway implements. # Audio (TTS & transcription) Source: https://docs.flex.ai/inference-api/guides/audio Generate speech with `POST /v1/audio/speech` and transcribe audio with `POST /v1/audio/transcriptions`. Two audio endpoints, both OpenAI-shaped: * **Text-to-speech (TTS)** — `POST /v1/audio/speech` returns audio bytes for a text input. Find models whose `output_modalities` includes `audio`. * **Transcription (ASR)** — `POST /v1/audio/transcriptions` returns text for an audio file upload. Find models whose `input_modalities` includes `audio`. See model discovery for the filter snippets — [text-to-speech](/inference-api/guides/model-discovery#text-to-speech) and [speech-to-text](/inference-api/guides/model-discovery#speech-to-text). ## Text-to-speech The example below uses `Kokoro-82M`. The response body is raw audio bytes (WAV by default) — write them straight to disk. ```bash cURL theme={null} curl https://api.flex.ai/v1/audio/speech \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "Kokoro-82M", "input": "Hello from FlexAI.", "voice": "neutral_female" }' \ --output out.wav ``` ```python Python theme={null} import os from openai import OpenAI client = OpenAI(base_url="https://api.flex.ai/v1", api_key=os.environ["FLEXAI_API_KEY"]) resp = client.audio.speech.create( model="Kokoro-82M", input="Hello from FlexAI.", voice="neutral_female", ) resp.stream_to_file("out.wav") ``` ```typescript TypeScript theme={null} import fs from "node:fs"; import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.flex.ai/v1", apiKey: process.env.FLEXAI_API_KEY, }); const resp = await client.audio.speech.create({ model: "Kokoro-82M", input: "Hello from FlexAI.", voice: "neutral_female", }); fs.writeFileSync("out.wav", Buffer.from(await resp.arrayBuffer())); ``` ### Voices Voice names are model-specific — there's no shared "Alloy/Echo/Fable/…" namespace. Check the [model catalog](https://flex.ai/models) for the supported set, or fall back to the model's upstream documentation. ### Billing TTS bills per input character. Read the active rate from `GET /v1/pricing` — find the row for your model, where `pricing_type` is `per_character`, and take `unit_price_usd`. A model that is temporarily not serving has no row until it is back. See also [billing](/inference-api/reference/billing). The same rate is on the model's own entry in `GET /v1/models`, under `media_pricing`, if you are already fetching that and would rather not make a second call. There the `billing_unit` reads `character`. ## Transcription `/v1/audio/transcriptions` is Whisper-compatible. Send the audio file as `multipart/form-data`, get back a JSON `{ "text": "..." }`. The current model is `whisper-large-v3-turbo`. ```bash cURL theme={null} curl https://api.flex.ai/v1/audio/transcriptions \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ -F "file=@input.wav" \ -F "model=whisper-large-v3-turbo" \ -F "language=en" ``` ```python Python theme={null} import os from openai import OpenAI client = OpenAI(base_url="https://api.flex.ai/v1", api_key=os.environ["FLEXAI_API_KEY"]) with open("input.wav", "rb") as f: resp = client.audio.transcriptions.create( model="whisper-large-v3-turbo", file=f, language="en", ) print(resp.text) ``` ```typescript TypeScript theme={null} import fs from "node:fs"; import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.flex.ai/v1", apiKey: process.env.FLEXAI_API_KEY, }); const resp = await client.audio.transcriptions.create({ model: "whisper-large-v3-turbo", file: fs.createReadStream("input.wav"), language: "en", }); console.log(resp.text); ``` ### Request fields | Field | Required | Notes | | - | - | - | | `file` | Yes | `multipart/form-data` upload. WAV, MP3, M4A, FLAC, OGG. | | `model` | Yes | Must be a speech-to-text model — on `/v1/models`, `category` is `audio` and `input_modalities` includes `audio`. | | `language` | No | ISO-639-1 code (`en`, `fr`, …). Auto-detected when omitted. | ### Billing ASR bills per minute of input audio. Read the active rate from `GET /v1/pricing` — find the row for your model, where `pricing_type` is `per_minute`, and take `unit_price_usd`. A model that is temporarily not serving has no row until it is back. As with the other per-unit models, `GET /v1/models` carries the same rate under `media_pricing` on the model's entry, so one call can do both discovery and pricing. The `billing_unit` there is `minute_audio`, not `minute` — the raw unit, rather than the `pricing_type` stem. ## See also * [Model discovery](/inference-api/guides/model-discovery) — how to tell text-to-speech and speech-to-text models apart on `/v1/models`. * [OpenAI compatibility](/inference-api/reference/openai-compatibility) — `/v1/audio/translations` is not supported. # Batching Source: https://docs.flex.ai/inference-api/guides/batch Batch processing on the Inference API is coming soon — reach out if you have a batch workload to run. Batch processing is coming soon to the Inference API. If you have a batch workload you'd like to run, [reach out to us](mailto:support@flex.ai) — we'd like to hear about your use case. # Concurrency Source: https://docs.flex.ai/inference-api/guides/concurrency How concurrent requests on a single API key behave, what the per-key rate limit is, and how to back off cleanly when you hit it. Multiple in-flight requests on the same API key run **independently and in parallel**. There is no head-of-line blocking, no per-key concurrency cap, and no implicit queuing — each request is dispatched to a backend as soon as it arrives. The only bound on a single key is the per-minute request ceiling described below. Inside that ceiling, fan out as wide as your workload needs. ## Per-key rate limit Each key carries a requests-per-minute (RPM) tier. When you exceed it the API returns `429 Too Many Requests` with a `Retry-After` header — see [Rate limits](/inference-api/reference/billing#rate-limits) for the full headers. | Tier | Requests per minute | | - | - | | Free (default on signup) | 10 | | Paid | 100 | | Custom | [Contact FlexAI](mailto:support@flex.ai) for higher limits | These are per-key, per-minute. They do not aggregate across keys on the same account today. The tier is stored on the key — request an upgrade from [support@flex.ai](mailto:support@flex.ai) if your steady-state load needs more. We may introduce account-level concurrency or aggregate RPM caps as the platform scales. We will notify customers before tightening any existing per-key limit; we will not silently lower it. ## Backpressure pattern Bound your in-flight count to roughly `RPM / 60` if you want to stay in a steady-state window, and retry on `429` with exponential backoff that respects `Retry-After`: ```python Python theme={null} import asyncio, random from openai import AsyncOpenAI, RateLimitError client = AsyncOpenAI(base_url="https://api.flex.ai/v1") # Bound concurrency to your tier: free=10/60≈1, elevated=60/60=1, paid=300/60=5. sem = asyncio.Semaphore(5) async def one(prompt: str, *, attempt: int = 0) -> str: async with sem: try: r = await client.chat.completions.create( model="Llama-3.3-70B-Instruct-FP8", messages=[{"role": "user", "content": prompt}], ) return r.choices[0].message.content except RateLimitError as e: retry_after = int(e.response.headers.get("retry-after", 1)) jitter = random.uniform(0, 1) await asyncio.sleep(retry_after + jitter) if attempt >= 5: raise return await one(prompt, attempt=attempt + 1) ``` ```typescript TypeScript theme={null} import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.flex.ai/v1" }); async function withBackoff(fn: () => Promise, max = 5): Promise { for (let attempt = 0; ; attempt++) { try { return await fn(); } catch (err: any) { if (err?.status !== 429 || attempt >= max) throw err; const retryAfter = Number(err.headers?.["retry-after"] ?? 1); await new Promise((r) => setTimeout(r, (retryAfter + Math.random()) * 1000)); } } } ``` If you're sustaining 429s after backoff, you're above your tier's steady-state capacity — request an upgrade rather than retrying harder. Tight retry loops don't move you through the limit faster; they just burn your quota on rejected requests. ## Ordering Concurrent requests on one key are independent — completion order is not guaranteed to match submission order. If you need to correlate responses back to inputs, carry your own correlation id in the prompt or response metadata; don't rely on arrival order. The `id` we return (`chatcmpl-…`) is unique per request and safe to use as a join key in your logs. ## Related * [Batching](/inference-api/guides/batch) — batch processing is coming soon. * [Billing & quotas](/inference-api/reference/billing) — tier limits and rate-limit response headers. * [Errors](/inference-api/reference/errors) — the full 429 body. # Embeddings Source: https://docs.flex.ai/inference-api/guides/embeddings Generate dense vector representations of text via `POST /v1/embeddings`. Embeddings turn text into fixed-length float vectors you can use for similarity search, clustering, and retrieval. The endpoint is OpenAI-shaped — point an OpenAI SDK at it and it works unchanged. The current embedding model is `bge-m3` — multilingual, 1024-dim, 8K context. `encoding_format` is optional and defaults to `"float"`, matching OpenAI. Pass `"base64"` if you want the compact wire format. The one thing to avoid is sending an explicit `null` — pass a string or omit the field entirely. ## Example ```bash cURL theme={null} curl https://api.flex.ai/v1/embeddings \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "bge-m3", "input": "the quick brown fox", "encoding_format": "float" }' ``` ```python Python theme={null} import os from openai import OpenAI client = OpenAI(base_url="https://api.flex.ai/v1", api_key=os.environ["FLEXAI_API_KEY"]) resp = client.embeddings.create( model="bge-m3", input="the quick brown fox", encoding_format="float", # optional; "float" is the default ) vector = resp.data[0].embedding # list[float], len 1024 ``` ```typescript TypeScript theme={null} import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.flex.ai/v1", apiKey: process.env.FLEXAI_API_KEY, }); const resp = await client.embeddings.create({ model: "bge-m3", input: "the quick brown fox", encoding_format: "float", // optional; "float" is the default }); const vector = resp.data[0].embedding; // number[], len 1024 ``` ## Batch inputs Pass an array of strings to embed several at once. The response `data[]` order matches the input order. ```python Python theme={null} resp = client.embeddings.create( model="bge-m3", input=["the quick brown fox", "jumps over the lazy dog"], encoding_format="float", ) vectors = [d.embedding for d in resp.data] ``` ## Response ```json theme={null} { "object": "list", "model": "bge-m3", "data": [ { "object": "embedding", "index": 0, "embedding": [0.0123, -0.456, ...] } ], "usage": { "prompt_tokens": 5, "total_tokens": 5 } } ``` With `encoding_format: "base64"`, each `embedding` field is a base64-encoded byte string of little-endian float32 values instead of a JSON array of numbers. Decode with your language's `base64` + struct/buffer helpers. ## Billing Embeddings bill per input token only. On `GET /v1/models` the model's `pricing.prompt` is the per-token input rate and `pricing.completion` is `"0"` — an embedding call returns a vector, not generated tokens, so there is nothing to charge on the way out. See [billing](/inference-api/reference/billing) for how that becomes a charge. ## See also * [Model discovery](/inference-api/guides/model-discovery#embeddings) — finding embedding models programmatically (`category` is `embedding`). * [OpenAI compatibility](/inference-api/reference/openai-compatibility) — the full list of supported endpoints and deviations. # Image generation Source: https://docs.flex.ai/inference-api/guides/images Generate images from text prompts via `POST /v1/images/generations`. Image generation is synchronous and OpenAI-shaped: send a prompt, get back base64-encoded PNGs in the response body. Find image-capable models by their `category` of `image` — see [model discovery](/inference-api/guides/model-discovery#image-generation). The example below uses `FLUX.1-schnell`. ## Example ```bash cURL theme={null} curl https://api.flex.ai/v1/images/generations \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "FLUX.1-schnell", "prompt": "a watercolor of a fox in a snowdrift", "size": "1024x1024" }' \ | jq -r '.data[0].b64_json' | base64 -d > out.png ``` ```python Python theme={null} import base64, os from openai import OpenAI client = OpenAI(base_url="https://api.flex.ai/v1", api_key=os.environ["FLEXAI_API_KEY"]) resp = client.images.generate( model="FLUX.1-schnell", prompt="a watercolor of a fox in a snowdrift", size="1024x1024", ) with open("out.png", "wb") as f: f.write(base64.b64decode(resp.data[0].b64_json)) ``` ```typescript TypeScript theme={null} import fs from "node:fs"; import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.flex.ai/v1", apiKey: process.env.FLEXAI_API_KEY, }); const resp = await client.images.generate({ model: "FLUX.1-schnell", prompt: "a watercolor of a fox in a snowdrift", size: "1024x1024", }); fs.writeFileSync("out.png", Buffer.from(resp.data[0].b64_json!, "base64")); ``` ## Request fields | Field | Required | Notes | | - | - | - | | `model` | Yes | Must be an image-generation model — `category` is `image` on `/v1/models`. | | `prompt` | Yes | — | | `n` | No | Must be `1` (the default). Multiple images per request are not currently supported. | | `size` | No | One of `512x512`, `768x768`, `1024x1024`, `1024x1792`, `1792x1024`. Default `1024x1024`. | | `response_format` | No | **`b64_json` only.** OpenAI's `url` format is not supported — the field is accepted for SDK compatibility but anything other than `b64_json` is a 400. | ## Response ```json theme={null} { "created": 1748902000, "data": [ { "b64_json": "iVBORw0KGgoAAAANSUhEUgAA..." } ] } ``` Decode `b64_json` with your language's base64 helper and write the bytes as a `.png`. ## Billing Image models bill per output image, not per token. Read the active rate from `GET /v1/pricing` — find the row for your model, where `pricing_type` is `per_image`, and take `unit_price_usd` (rates can vary by resolution tier). A model that is temporarily not serving has no row until it is back. The [billing reference](/inference-api/reference/billing) explains how that becomes a charge. If you are already calling `GET /v1/models` to discover the model, its `media_pricing` array carries the same rates on the same entry, which saves the second round trip. One difference to watch: `media_pricing` reports the raw `billing_unit`, so an image model reads `image` or `megapixel` there where `/v1/pricing` reports both as the single `pricing_type` of `per_image`. Both surfaces quote the same `unit_price_usd`. # Model discovery Source: https://docs.flex.ai/inference-api/guides/model-discovery How to list the live model catalog programmatically, find the right model for your task, and call the matching endpoint. `GET /v1/models` is the programmatic way to enumerate the catalog. It returns every model that is **currently serving**, across all modalities — text, code, reasoning, vision, embeddings, image, audio, and video — in OpenAI's `Model` shape, with each entry enriched so you can tell what the model accepts, what it costs, and which endpoint to call it through. It needs a bearer key, but nothing more: discovery is exempt from budget checks, so the call keeps working even when your balance has run out. That is deliberate — deciding what to top up for is exactly the moment you need to read the catalog. `/v1/models` is the catalog endpoint, and like every `/v1/*` route it needs a key. If you don't have one yet, start with the [quickstart](/inference-api/quickstart). For a human-readable roster of everything hosted, including models that are not currently serving, see the [model catalog](https://flex.ai/models). ## Calling `/v1/models` ```bash cURL theme={null} curl https://api.flex.ai/v1/models \ -H "Authorization: Bearer $FLEXAI_API_KEY" | jq '.data[0]' ``` ```python Python theme={null} import os from openai import OpenAI client = OpenAI(base_url="https://api.flex.ai/v1", api_key=os.environ["FLEXAI_API_KEY"]) models = client.models.list() print(models.data[0]) ``` ```typescript TypeScript theme={null} import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.flex.ai/v1", apiKey: process.env.FLEXAI_API_KEY, }); const models = await client.models.list(); console.log(models.data[0]); ``` The response is an OpenAI-shaped list — `{"object": "list", "data": [...]}` — so the entries are under `.data`, not at the top level. Each entry looks like this, abridged to the fields worth explaining — a live entry carries a few more, and key order is not part of the contract: ```json theme={null} { "id": "Qwen3-Coder-30B-A3B-Instruct-FP8", "object": "model", "created": 1753920000, "owned_by": "flexai", "is_free": false, "is_ready": true, "name": "Qwen3 Coder 30B A3B", "hugging_face_id": "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8", "aliases": ["coder-30b"], "category": "code", "input_modalities": ["text"], "output_modalities": ["text"], "quantization": "fp8", "context_length": 262144, "max_output_length": 262144, "pricing": { "currency": "USD", "input_per_mtok": 0.07, "output_per_mtok": 0.26, "cached_input_per_mtok": 0.0105, "prompt": "0.00000007", "completion": "0.00000026", "input_cache_read": "0.0000000105", "request": "0", "image": "0", "web_search": "0", "internal_reasoning": "0" }, "media_pricing": null, "supported_features": ["tools"], "supported_parameters": [ "temperature", "top_p", "top_k", "min_p", "frequency_penalty", "presence_penalty", "repetition_penalty", "stop", "seed", "max_tokens", "logit_bias", "logprobs", "top_logprobs", "response_format", "tools", "tool_choice", "structured_outputs" ] } ``` The fields you'll reach for most often: * **`id`** — the value to pass as `model` in subsequent calls. * **`aliases[]`** — stable, human-friendly handles for the same model (e.g. `coder-30b`). Any alias works in place of `id`. The field is omitted for models that have no alias. Unlike `id`, an alias does not change when the underlying checkpoint or quantization does, so **pin config to an alias** if you want it to survive model updates. * **`category`** — the model's modality/family. This is what tells you which endpoint to call; see [Category → endpoint](#category--endpoint). * **`supported_parameters[]`** — the request parameters this specific model actually honors. This is the field to gate on: check it before sending a parameter rather than learning the answer from a runtime error. * **`supported_sampling_parameters[]`** — a legacy field, kept so existing integrations keep working. Gate on `supported_parameters` instead. It is a blanket list rather than a per-model one: every chat model carries the same twelve entries, and every non-chat model carries an empty array. Because it is fixed, it is wrong in both directions on a given model — it advertises `top_a`, which this gateway does not read, validate or forward on any model, and it omits `tools`, `tool_choice`, `logprobs`, `top_logprobs`, `response_format`, `structured_outputs` and `reasoning_effort`, which are exactly the capabilities worth checking before you send a request. * **`input_modalities[]`** — what the model accepts. Vision support shows up here as `image`, *not* in `supported_parameters`, because image input is a message shape rather than a request parameter. * **`context_length` / `max_output_length`** — the context window and the maximum completion length, in tokens. On chat-family models the two are equal: the engine has no separate output ceiling, so a completion may use however much of the window your prompt leaves free (`context_length − prompt tokens`). Budget the two together — a longer prompt means less room for the completion, and `max_tokens` is how you choose the split. A model where `max_output_length` is *smaller* than `context_length` genuinely caps completions at that value. Both keys are on every entry; `null` means we haven't published the value yet (or, for `max_output_length`, that the model doesn't produce tokens) — never that the model is unlimited. * **`pricing`** — per-token rates on models billed per token: `input_per_mtok` / `output_per_mtok` in USD per million tokens, plus the same rates per single token as decimal strings (`prompt` / `completion`). On models with prompt caching enabled, `cached_input_per_mtok` (and `input_cache_read`) is the discounted rate for prompt tokens served from cache; when it is absent the model has no cache discount and the whole prompt bills at the input rate. `pricing` is `null` on per-unit models (image, audio, video), whose rates are in `media_pricing` instead, and on a model whose price isn't published yet — `null` never means free (see [Reading prices](#reading-prices) below). * **`media_pricing`** — on image, audio and video models, a list of per-unit USD rates: each entry has a `billing_unit` (`image`, `megapixel`, `second_video`, `character` or `minute_audio`) and a `unit_price_usd`, plus `resolution_tier` / `quality_tier` / `preset` when the model has more than one price point. * **`is_free` / `is_ready`** — two routing switches for third-party catalogs that ingest this endpoint, stated on every entry so the payload says what it means rather than leaning on an ingester's defaults. `is_free` is always `false`: every model bills against the calling key's balance, including the ones with no `pricing` object here, so never read a missing price as a free one. `is_ready` says whether an ingester may route to the model — `false` means its price cannot be expressed in the per-token `pricing` shape (per-unit media billing) or has not been published yet. **`is_ready` is not a health or availability signal.** Every model this endpoint lists is live, and an `is_ready: false` model is fully callable with a direct request to its endpoint; the flag only governs third-party routing. * **`supported_features[]`** — what the model *can do*, as opposed to what you may *send*. The two fields answer different questions, and the difference is worth holding onto. `supported_parameters` is the request contract: every entry has honored request behaviour behind it, which makes it the field to gate a request on. Most entries are literal keys you send; `structured_outputs` is the exception, marking support for a *mode* of another key (`response_format: {"type": "json_schema"}` — see [Structured outputs](#structured-outputs)) rather than something you send by that name. `supported_features` is a capability description, so it can name an ability with no request key behind it at all. Today it carries `reasoning` on models that produce an extended reasoning trace, `tools` on tool-calling models, and `json_mode` (which no model in the catalog currently declares). `reasoning` appears only here because reasoning is not uniformly controllable: some models reason unconditionally and offer no switch, while the depth dial we do advertise is `reasoning_effort`, which appears in `supported_parameters` on the models that accept it. So to find the reasoning models, read this field. Controlling how much they reason is a separate question, and `supported_parameters` is where it is answered: a model listing `reasoning_effort` accepts that dial, and if `reasoning_effort` is absent then `/v1/models` advertises no reasoning-control parameter for that model. Gate on what is advertised — as everywhere else on this endpoint, a parameter you don't see listed is one we don't undertake to honor. ## Retrieve one model `GET /v1/models/{model}` returns a single entry — the same object the list returns for that model — so you can check one model's limits and price without fetching the whole catalog. `model` may be the `id` or any alias; the response always carries the canonical `id`. ```bash theme={null} curl -s https://api.flex.ai/v1/models/coder-30b \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ | jq '{id, context_length, max_output_length, pricing}' ``` A model that isn't in your live catalog right now returns `404` with `"code": "model_not_found"`. With the OpenAI SDKs this is `client.models.retrieve("coder-30b")`. ## Category → endpoint A model's `category` tells you which endpoint it's called through: | `category` | Call with | | - | - | | `text`, `code`, `reasoning`, `vision`, `multimodal` (deprecated) | `POST /v1/chat/completions` | | `embedding` | `POST /v1/embeddings` | | `image` | `POST /v1/images/generations` | | `audio` (text-to-speech) | `POST /v1/audio/speech` | | `audio` (speech-to-text) | `POST /v1/audio/transcriptions` | | `video` | `POST /v1/videos/generations` | For `audio`, `input_modalities` disambiguates the direction: a model whose `input_modalities` includes `audio` is speech-to-text; otherwise it's text-to-speech. `category` also answers "which chat models can read an image": every chat model that accepts image input is categorized `vision`. The `multimodal` value is **deprecated** — those models are being folded into `vision`, and the value disappears from responses once that rollout completes. Until then, treat the two identically; the filters below include both so they return the full set on either side of the rollout. ## Per-capability discovery flow Each filter below finds the models, and the endpoint that follows is what you call once you have an `id`. ### Chat & text ```bash theme={null} curl -s https://api.flex.ai/v1/models \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ | jq '[.data[] | select(.category | IN("text", "code", "reasoning", "vision", "multimodal")) | select(.supported_parameters | index("max_tokens")) | .id]' ``` Call with `POST /v1/chat/completions`. See [streaming](/inference-api/guides/streaming) and [tool use](/inference-api/guides/tool-use) for the common patterns. The `max_tokens` check is the general form of "is this model callable through the chat endpoint": every chat model advertises the sampling surface, and no other endpoint's parameter list contains it. Category alone is a weaker signal — it describes what a model *is*, not which endpoint accepts it — so the two together are what you want whenever you intend to POST the result to `/v1/chat/completions`. ### Tool calling Gate on `supported_parameters` containing `tools`. If it's absent, tool calling does not work on that model, whatever your SDK lets you send. ```bash theme={null} curl -s https://api.flex.ai/v1/models \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ | jq '[.data[] | select(.supported_parameters | index("tools")) | .id]' ``` ### Structured outputs Strict schema enforcement — `response_format: {"type": "json_schema"}` — requires `structured_outputs`. Sending a `json_schema` request to a model without it is rejected with a `400`. ```bash theme={null} curl -s https://api.flex.ai/v1/models \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ | jq '[.data[] | select(.supported_parameters | index("structured_outputs")) | .id]' ``` ### Vision (image input on chat) ```bash theme={null} curl -s https://api.flex.ai/v1/models \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ | jq '[.data[] | select(.input_modalities | index("image")) | select(.supported_parameters | index("max_tokens")) | .id]' ``` The second `select` is doing real work: `input_modalities` says the model *accepts* an image, which is also true of an image-editing model that you call through `/v1/images/generations`. Requiring `max_tokens` narrows it to models that implement the chat sampling surface, so you get only the ones this endpoint is right for. These are the models categorized `vision` (or the deprecated `multimodal`, until its fold into `vision` completes — see [Category → endpoint](#category--endpoint)), so once the fold is live, `select(.category | IN("vision", "multimodal"))` is the shorter spelling of the same filter. Call with `POST /v1/chat/completions` and pass image parts in the `content` array. The full pattern lives in the [vision guide](/inference-api/guides/vision). ### Embeddings ```bash theme={null} curl -s https://api.flex.ai/v1/models \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ | jq '[.data[] | select(.category == "embedding") | .id]' ``` Call with `POST /v1/embeddings`. See the [embeddings guide](/inference-api/guides/embeddings). ### Image generation ```bash theme={null} curl -s https://api.flex.ai/v1/models \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ | jq '[.data[] | select(.category == "image") | .id]' ``` Call with `POST /v1/images/generations`. See the [image generation guide](/inference-api/guides/images). ### Text-to-speech Both directions of audio share `category: "audio"`, so the modality arrays are what separate them. A text-to-speech model **produces** audio: ```bash theme={null} curl -s https://api.flex.ai/v1/models \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ | jq '[.data[] | select(.output_modalities | index("audio")) | .id]' ``` Call with `POST /v1/audio/speech`. See the [audio guide](/inference-api/guides/audio). ### Speech-to-text A transcription model **accepts** audio: ```bash theme={null} curl -s https://api.flex.ai/v1/models \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ | jq '[.data[] | select(.input_modalities | index("audio")) | .id]' ``` Call with `POST /v1/audio/transcriptions`. See the [audio guide](/inference-api/guides/audio). ## Reading prices `pricing.input_per_mtok` and `pricing.output_per_mtok` are USD per million tokens — the same numbers `GET /v1/pricing` reports. `pricing.prompt` and `pricing.completion` carry the same rates **per single token**, as decimal strings so they survive a JSON round-trip without floating-point loss; use those if you need exact arithmetic. Every price object states its `currency`. ```bash theme={null} curl -s https://api.flex.ai/v1/models \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ | jq '[.data[] | select(.pricing) | { id, input_per_mtok: .pricing.input_per_mtok, output_per_mtok: .pricing.output_per_mtok, cached_input_per_mtok: .pricing.cached_input_per_mtok }]' ``` On models with prompt caching enabled, `pricing.cached_input_per_mtok` is the rate charged for prompt tokens served from the cache instead of the full input rate (`input_cache_read` is the same rate per token). A model without it has no cache discount, so its whole prompt bills at `input_per_mtok`; absence never means cache reads are free. Models billed **per unit rather than per token** have `pricing: null` — image generation bills per image, text-to-speech per character, transcription per minute of audio, and video per second, and none of those units has a key in the per-token shape above. Their rates are on the same entry, in `media_pricing`, quoted in USD per `billing_unit`: ```bash theme={null} curl -s https://api.flex.ai/v1/models \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ | jq '[.data[] | select(.media_pricing) | {id, category, media_pricing}]' ``` A model with more than one price point — several resolutions or quality levels — has one `media_pricing` entry per point, told apart by `resolution_tier`, `quality_tier` and `preset`. `GET /v1/pricing` carries the same rows (with a `pricing_type` such as `per_image` or `per_second`) if you want every price in one flat list. Both endpoints list only the models currently being served, so a model that is scaled down or restarting drops out until it is back; keep your own copy of a rate rather than reading an absent entry as a price change. A `null` `pricing` or `media_pricing` never means free — every model in the catalog bills against your balance. ## Retired model ids and the `suggested_model` hint **Dated ids** (a model id ending in a date-style suffix, like `DeepSeek-V4-Flash-0731`) are pinned to the exact checkpoint they name: a dated id is never re-pointed at different weights, so the model behind it can't change under you. If you need reproducible behavior, pin a dated id where one exists. An **undated id** is the model's family name and may move to a newer checkpoint when one replaces the old — when that happens we announce it, and going forward the replacement ships under a new dated id while the outgoing id is retired rather than left pointing at weights it no longer describes. A request that names a retired — or temporarily unavailable — id fails with `404`, and when a direct replacement exists the error body names it in a stable machine-readable field: ```json theme={null} { "error": "unknown_model", "model": "DeepSeek-V4-Flash", "message": "Model id not recognized. ...", "did_you_mean": ["DeepSeek-V4-Flash-0731"], "suggested_model": "DeepSeek-V4-Flash-0731" } ``` `suggested_model` is a declared successor — a contract you can act on programmatically (update your configured model id and retry), unlike `did_you_mean`, which is fuzzy spelling help. Prices may differ between a retired id and its successor; check `/v1/models` or the [model catalog](https://flex.ai/models) before switching. **Current cases:** `DeepSeek-V4-Flash` (the family's 0422 checkpoint) is retired from the serverless API — requests fail as above with `suggested_model: "DeepSeek-V4-Flash-0731"`, the serverless offering. The 0422 checkpoint itself is not gone: it may be available as a dedicated deployment depending on capacity — [contact us](mailto:support@flex.ai) to discuss your requirements. The two are distinct checkpoints with distinct pricing, so treat the switch as a model change, not a rename. Note that the undated id `DeepSeek-V4-Flash` may later be re-pointed to the newest checkpoint in the family — so if it reappears on `/v1/models`, that does not mean the 0422 checkpoint returned; check which checkpoint the id names before switching back. If you need the 0422 checkpoint specifically, use the contact route above rather than watching `/v1/models`. The dated id `deepseek-ai/DeepSeek-V4-Flash-0731` announced at the 2026-08 cutover keeps working unchanged. ## See also * [Model catalog](https://flex.ai/models) — human-readable table of every hosted model, with capabilities and pricing. * [OpenAI compatibility](/inference-api/reference/openai-compatibility) — what's in and out of `/v1/*`, including the scope of `/v1/models`. * [Billing](/inference-api/reference/billing) — how per-token pricing maps to charges. # Streaming Source: https://docs.flex.ai/inference-api/guides/streaming Stream tokens as they are generated and collect the final usage block for accurate spend tracking. Streaming lets you render tokens as the model produces them instead of waiting for the whole response. Set `stream: true` on any `/v1/chat/completions` request. **Always set `stream_options.include_usage: true` when streaming.** Without it you cannot bill the request correctly on your side, and per-key spend tracking on ours silently loses information. Every server-sent event framework we recommend below does this by default. **Non-streaming requests run under a fixed time budget; stream for long outputs.** Your `max_tokens` / `max_completion_tokens` is honored in full on both paths — output is never silently truncated to a fixed ceiling. The difference is time: a request without `stream: true` returns nothing until the whole generation is finished, so it must complete within a fixed server-side time budget (roughly 95 seconds). A generation that can't finish in time returns a **timeout error**, not partial content. Streaming has no such budget — each token keeps the connection alive and resets the clock — so set `stream: true` for any response that may be long (a large `max_tokens`, a reasoning model, or long structured output). ## Example ```bash cURL theme={null} curl https://api.flex.ai/v1/chat/completions \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ -H "Content-Type: application/json" \ -N \ -d '{ "model": "Llama-3.3-70B-Instruct-FP8", "messages": [{"role":"user","content":"Count to five."}], "stream": true, "stream_options": {"include_usage": true} }' ``` ```python Python theme={null} import os from openai import OpenAI client = OpenAI(base_url="https://api.flex.ai/v1", api_key=os.environ["FLEXAI_API_KEY"]) stream = client.chat.completions.create( model="Llama-3.3-70B-Instruct-FP8", messages=[{"role": "user", "content": "Count to five."}], stream=True, stream_options={"include_usage": True}, ) for chunk in stream: if chunk.choices and chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) if chunk.usage: print(f"\n\nprompt={chunk.usage.prompt_tokens} completion={chunk.usage.completion_tokens}") ``` ```typescript TypeScript theme={null} import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.flex.ai/v1", apiKey: process.env.FLEXAI_API_KEY, }); const stream = await client.chat.completions.create({ model: "Llama-3.3-70B-Instruct-FP8", messages: [{ role: "user", content: "Count to five." }], stream: true, stream_options: { include_usage: true }, }); for await (const chunk of stream) { if (chunk.choices[0]?.delta?.content) { process.stdout.write(chunk.choices[0].delta.content); } if (chunk.usage) { console.log(`\n\nprompt=${chunk.usage.prompt_tokens} completion=${chunk.usage.completion_tokens}`); } } ``` ## Anatomy of the stream The response is `text/event-stream`. Each event is a line beginning with `data: ` followed by a JSON object, then a blank line: ``` data: {"id":"chatcmpl-abc","choices":[{"delta":{"content":"One"},"index":0,"finish_reason":null}]} data: {"id":"chatcmpl-abc","choices":[{"delta":{"content":", "},"index":0,"finish_reason":null}]} ... data: {"id":"chatcmpl-abc","choices":[{"delta":{},"index":0,"finish_reason":"stop"}]} data: {"id":"chatcmpl-abc","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":8,"total_tokens":20}} data: [DONE] ``` Three things worth knowing: 1. **`delta.content` is the incremental piece** — concatenate them to reconstruct the full response. 2. **The penultimate chunk carries `finish_reason`** (`stop`, `length`, `tool_calls`, or `content_filter`). Its `delta` is typically empty. 3. **The final-before-`[DONE]` chunk carries `usage`** (only when you set `stream_options.include_usage: true`). This is the one source of truth for token counts; do not try to count tokens client-side. ## Runaway streams A model can get stuck repeating itself — the same phrase, over and over, until it hits its context window. Left alone, that stream would run for many minutes, hold a slot on the model, and bill every repeated token. The gateway watches the streamed text for this. When the last few kilobytes stop containing anything new for several checks in a row, it ends the stream: you receive a normal terminal chunk with `finish_reason: "length"`, and that chunk (and the usage chunk after it) carries `flexai_stopped: "degeneration"` so you can tell it apart from a stream that simply reached `max_tokens`. On `/v1/responses` the terminal event is `response.incomplete` with `incomplete_details.reason: "max_output_tokens"`, the same `flexai_stopped` field on the response object, and the text streamed so far collected into one assistant message in `output` (so a client that reads the final response object rather than the deltas still gets what it was billed for). A second, looser bound cuts a stream that has grown past roughly half the model's context window while also repeating itself (`flexai_stopped: "ceiling"`); a long stream that is still saying new things is never cut by it. Treat `flexai_stopped` as a signal to change something — usually the sampling parameters (a `temperature` of 0 with a long open-ended prompt is the common trigger), sometimes the prompt — rather than to retry as-is. The tokens you received are billed; the tail the model would have gone on to produce is not generated. ``` data: {"id":"chatcmpl-abc","choices":[{"delta":{},"index":0,"finish_reason":"length"}],"flexai_stopped":"degeneration"} data: {"id":"chatcmpl-abc","choices":[],"usage":{"prompt_tokens":120,"completion_tokens":2210,"total_tokens":2330},"flexai_stopped":"degeneration"} data: [DONE] ``` ## Tool calls while streaming Tool call arguments arrive as `delta.tool_calls[].function.arguments` fragments that you accumulate the same way as `delta.content`. See the [tool use guide](/inference-api/guides/tool-use) for a full example. ## Handling disconnects ### No resume protocol There is no `Last-Event-ID`-style resume. A dropped TCP connection ends the stream — reconnecting issues a brand-new request with a fresh `chatcmpl-…` id, not a continuation of the old one. Tokens emitted before the disconnect are not retransmitted. If you need durability across flaky networks, persist the partial output your client has already received and re-prompt with the conversation so far when you reconnect; the model will not pick up exactly where it left off, but it will continue the conversation. ### Cancellation billing When the client closes the connection mid-generation, the server propagates the cancellation to the model backend and **bills only the tokens actually emitted up to the disconnect** — not the requested `max_tokens`. The model stops generating when you stop reading. This means it is safe to disconnect early to cap latency or cost. The matching warehouse row will reflect the tokens you received, not the budget you requested. ### Retrying A retry is a fresh, independently-billed request — there is no server-side idempotency key, so we cannot deduplicate. Pragmatic guidance: * **Short prompts:** retry directly; the duplicate cost is bounded. * **Long generations:** surface the partial output to the user and let them decide whether to continue. Echo what you already have back as context if they choose to retry. See [Concurrency](/inference-api/guides/concurrency) for the per-key rate limit headers that govern how aggressively you can retry. # Tool Use Source: https://docs.flex.ai/inference-api/guides/tool-use Let the model call functions you define, then feed their output back in. Tool use (a.k.a. "function calling") lets the model decide when to call a function you've defined, returns the arguments it wants to pass, and lets you feed the result back in for a final answer. Models that support tool use are labeled `tool_use` in the [catalog](https://flex.ai/models) — for this guide we'll use `Llama-3.3-70B-Instruct-FP8`. **Tool-calling reliability scales with model size.** The smallest models — for example `Meta-Llama-3.1-8B-Instruct-FP8` — handle a single, simple tool call well, but they get less dependable as the schema tightens or the conversation runs over several turns. Two things to watch for: arguments can come back loosely typed (a number as the string `"5"`, or an array collapsed into one comma-separated string), so validate and coerce on your side rather than trusting the raw JSON; and in a multi-step loop the model may keep emitting `tool_calls` instead of settling on a final answer, so cap the number of tool round trips your code will follow. For multi-step agents, or whenever you depend on exact argument types, prefer a larger tool-capable model such as the `Llama-3.3-70B-Instruct-FP8` we use below. ## Full round trip ```python Python theme={null} import json import os from openai import OpenAI client = OpenAI(base_url="https://api.flex.ai/v1", api_key=os.environ["FLEXAI_API_KEY"]) def get_weather(city: str) -> dict: # In a real app this would hit a weather API. return {"city": city, "temperature_c": 21, "conditions": "clear"} tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Return current weather for a city.", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }] messages = [{"role": "user", "content": "What's the weather in Paris?"}] # First turn: model decides to call the tool. first = client.chat.completions.create( model="Llama-3.3-70B-Instruct-FP8", messages=messages, tools=tools, ) tool_call = first.choices[0].message.tool_calls[0] args = json.loads(tool_call.function.arguments) result = get_weather(**args) # Second turn: feed the tool result back in. messages.extend([ first.choices[0].message, { "role": "tool", "tool_call_id": tool_call.id, "content": json.dumps(result), }, ]) second = client.chat.completions.create( model="Llama-3.3-70B-Instruct-FP8", messages=messages, tools=tools, ) print(second.choices[0].message.content) ``` ```typescript TypeScript theme={null} import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.flex.ai/v1", apiKey: process.env.FLEXAI_API_KEY, }); const getWeather = (city: string) => ({ city, temperature_c: 21, conditions: "clear" }); const tools = [ { type: "function" as const, function: { name: "get_weather", description: "Return current weather for a city.", parameters: { type: "object", properties: { city: { type: "string" } }, required: ["city"], }, }, }, ]; const messages: any[] = [{ role: "user", content: "What's the weather in Paris?" }]; const first = await client.chat.completions.create({ model: "Llama-3.3-70B-Instruct-FP8", messages, tools, }); const toolCall = first.choices[0].message.tool_calls![0]; const args = JSON.parse(toolCall.function.arguments); const result = getWeather(args.city); messages.push(first.choices[0].message, { role: "tool", tool_call_id: toolCall.id, content: JSON.stringify(result), }); const second = await client.chat.completions.create({ model: "Llama-3.3-70B-Instruct-FP8", messages, tools, }); console.log(second.choices[0].message.content); ``` ## Forcing a specific tool Set `tool_choice` to force the model's hand: ```json theme={null} { "tool_choice": { "type": "function", "function": { "name": "get_weather" } } } ``` Or `"tool_choice": "required"` to force it to call *some* tool, or `"none"` to forbid tool calls entirely. For `GLM-5.2`, `MiniMax-M2.7` and `NVIDIA-Nemotron-3.5-Lightning-30B-A3B`, `"required"` is bounded to exactly one tool call when every tool you send is a function tool. Without the bound these models repeat the same call until the token limit. The gateway forces `parallel_tool_calls: false` (if you sent `true`, the response carries `x-flexai-dropped-params: parallel_tool_calls`) and reserves one of the four `stop` entries for the model's closing tool tag, so a `"required"` request that already sends four `stop` entries, none of them the model's closing tool tag, is a `400` naming `stop`. A reply that is not exactly one finished call is rejected with `tool_choice_not_honored`; when the reply stopped at the token limit, the message says so and names the budget to raise. Streamed, the gateway holds the tool-call deltas until the end and replaces them with the error event; that hold is best-effort above the hold cap described below, so treat the streamed check as a backstop and the stop bound as the fix. Thinking may be on or off on this path. `"auto"` and named tool choices are unchanged. **A forced choice is enforced whether or not you stream.** Not every serving engine can make a model call a tool, so when you pin `tool_choice` to `"required"` or to a named function and the model answers in prose instead, the gateway rejects the answer rather than hand your agent text it would mistake for a result. Non-streamed that is a `400` with `code: "tool_choice_not_honored"`. The same `400` covers a non-streamed forced reply that stops at the token limit (`finish_reason: "length"`) even when it carries tool calls: a call list cut off by the budget is duplicate or partial execution waiting to happen, so raise `max_tokens` or use `"auto"`. On every other model that rule is non-streamed only: a stream releases tool-call deltas as they arrive and cannot take them back, so check `finish_reason` before dispatching calls from a stream. Streamed, the status is already `200` when the answer is known, so the rejection is the stream's last event: reasoning deltas stream as usual, any text the model wrote before deciding to call a tool is released together with the first `tool_calls` delta, and a stream that ends with no call at all ends with one `error` event (same `type`, `param` and `code` as the 400) instead of the prose. The gateway holds back at most a few thousand characters of that prose; a model that keeps writing past that is streamed from there on, and the same `error` event still ends the stream. A named function is satisfied only by calls to that function: a reply that calls a different tool, instead of it or as well as it, is rejected the same way (on a named stream the call is delivered at the end, once the whole answer is known). Either way the tokens the model generated are billed, since the answer was produced before it could be judged; on a chat stream the usage chunk arrives ahead of the `error` event so you can see that spend. Either way, retry with `tool_choice: "auto"` or pick a model that supports forced tool calls; see [errors](/inference-api/reference/errors) for the exact shapes. ## Agent loops on DeepSeek-V4-Flash-0731 `DeepSeek-V4-Flash-0731` is the model most of our coding-agent traffic runs on, and a few of its defaults are worth knowing before you put it in a multi-step loop. **Reasoning is off by default.** A plain request gets a direct answer with no `reasoning_content`. To turn reasoning on, send `reasoning_effort`: `low` and `minimal` keep it off, `medium` and `high` select the model's high effort level, and `max` selects its maximum. `thinking: {"type": "enabled" | "disabled"}` is the on/off switch and wins over `reasoning_effort`: `disabled` turns reasoning off even if a `reasoning_effort` is also sent (on a plain request it is redundant), and `enabled` on its own turns reasoning on at the model's default depth — pair it with a `reasoning_effort` to pick the depth. Reasoning tokens are returned in `reasoning_content` (and streamed as `delta.reasoning_content`) and are billed as output tokens, so raise `max_tokens` accordingly. **Give tool calls output headroom.** The model writes its tool calls in a verbose native markup that the serving engine parses into `tool_calls`. If `max_tokens` cuts that markup off, the response ends with `finish_reason: "length"` and **no** `tool_calls` — from your agent's point of view the model narrated an action and then stopped. Set `max_tokens` to 4096 or more on tool turns (an explicit value is honored as-is; only non-streaming requests that set none default to 2048, and only on non-reasoning models — a reasoning model gets no default, since it would spend it on the reasoning), and treat `finish_reason: "length"` without `tool_calls` as a truncation to retry with a larger budget, not as a final answer. The gateway marks that case for you: a non-streamed response cut off with nothing visible carries `x-flexai-empty-at-length: true`, and one that ran under the default carries `x-flexai-max-tokens-defaulted: 2048`. Non-streaming requests are also bounded by a \~95 s generation budget, so [stream](/inference-api/guides/streaming) long agent turns. **Read the window from the API, don't hard-code it.** The model's `context_length` on `GET /v1/models` is the prompt limit (see [model discovery](/inference-api/guides/model-discovery)); prompts over it return a `400`. For streaming requests, output is bounded only by the remaining window. **If you are building the loop with a framework**, rather than by hand, see [LangChain and LangGraph on FlexAI](/inference-api/agents/langchain). It covers the same ground with the framework's own abstractions, and carries a table of which models were measured to complete a real two-tool agent loop — which is a stricter bar than tool calling working at all. ## Parallel tool calls Most tool-capable models we host can emit multiple tool calls per turn (`message.tool_calls` will have length > 1). Handle each one, then add one `role: "tool"` message per call — each with the matching `tool_call_id` — before the next completion call. # Vision Source: https://docs.flex.ai/inference-api/guides/vision Send images to multimodal models using the OpenAI `image_url` content shape. Vision-capable models accept images alongside text in the same message. Pass a list of content parts — one `text` part, one or more `image_url` parts — instead of a plain string. The current vision-capable model is `phi-4-multimodal-instruct`. Any model with `vision` in its [capabilities list](https://flex.ai/models) accepts the same request shape. ## Example ```bash cURL theme={null} curl https://api.flex.ai/v1/chat/completions \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "phi-4-multimodal-instruct", "messages": [{ "role": "user", "content": [ {"type": "text", "text": "What is in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}} ] }] }' ``` ```python Python theme={null} import os from openai import OpenAI client = OpenAI(base_url="https://api.flex.ai/v1", api_key=os.environ["FLEXAI_API_KEY"]) response = client.chat.completions.create( model="phi-4-multimodal-instruct", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What is in this image?"}, {"type": "image_url", "image_url": { "url": "https://example.com/photo.jpg", }}, ], }], ) print(response.choices[0].message.content) ``` ```typescript TypeScript theme={null} import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.flex.ai/v1", apiKey: process.env.FLEXAI_API_KEY, }); const response = await client.chat.completions.create({ model: "phi-4-multimodal-instruct", messages: [{ role: "user", content: [ { type: "text", text: "What is in this image?" }, { type: "image_url", image_url: { url: "https://example.com/photo.jpg" } }, ], }], }); console.log(response.choices[0].message.content); ``` ## Image sources The `image_url.url` field accepts either a public HTTPS URL or a base64 data URL. Data URLs are the right choice when the image isn't already hosted somewhere — no pre-upload step required. ```python Python theme={null} import base64 with open("photo.jpg", "rb") as f: b64 = base64.b64encode(f.read()).decode() data_url = f"data:image/jpeg;base64,{b64}" # pass data_url as image_url.url in the request above ``` Supported formats: JPEG, PNG, WebP, non-animated GIF. ## Multiple images Pass more than one `image_url` part in the same `content` list to ask the model to reason across several images at once. The order is preserved in the model's context. ## Usage accounting Vision tokens count against your key's budget the same way text tokens do — the `usage` block in the response includes any vision-encoded tokens in `prompt_tokens`. There's no separate "per image" charge for vision input (unlike image *generation*, which is [priced per output image](/inference-api/reference/billing)). # Overview Source: https://docs.flex.ai/inference-api/overview OpenAI-compatible inference API for text, code, reasoning, vision, and embedding models. One API key, billed per token. The FlexAI Inference API is an OpenAI-compatible HTTP gateway for text, code, reasoning, vision, and embedding models. Point any OpenAI SDK at `https://api.flex.ai/v1` and your existing code works unchanged. Get a key and make your first request in under two minutes. Browse every model we host, with context windows and pricing. Filter the live catalog from code and find the right endpoint per modality. Stream tokens token-by-token with usage tracking. Send images to multimodal models in the OpenAI `image_url` shape. Call functions from model responses. ## What you get * **One API key, every model.** Chat, completions, vision, and embeddings all authenticate with the same `sk-…` bearer token. * **Account-level credit, per-key rate limits.** Your balance is shared across every key on the account; each key has its own requests-per-minute limit. Exceed either and you get an unambiguous 402 or 429 with headers explaining why. * **No card to start.** Sign up and mint a key without entering payment details. Add a card when you're ready to spend; usage is then metered at per-model rates, with no token packs and no subscriptions. Prices and usage are in dollars everywhere; EU accounts are charged and invoiced in euros (see [Regions](/inference-api/reference/regions#billing-currency)). * **Pay as you go.** No commitment beyond your usage, so you can start evaluating at whatever scale suits you and stop whenever you like. ## OpenAI compatibility The surface is intentionally identical to OpenAI's for the endpoints we support. If your code runs against `api.openai.com`, swap the `base_url` and the model id — that's the whole migration. See the [compatibility matrix](/inference-api/reference/openai-compatibility) for what's supported and what isn't. ## Where things live | Surface | URL | | - | - | | API base URL (US) | `https://api.flex.ai/v1` | | API base URL (EU) | `https://eu.api.flex.ai/v1` | | Dashboard (keys, billing, usage) | `https://platform.flex.ai/dashboard` | | Status | `https://status.flex.ai` | | Docs (this site) | `https://docs.flex.ai/inference-api` | Which API host you use follows the region your workspace was created in, and a key is only valid against its own region's host. If you haven't chosen a region, you're in `us` and `https://api.flex.ai/v1` is yours. See [Regions](/inference-api/reference/regions). Use `https://api.flex.ai/v1` for anything new in the US region, and `https://eu.api.flex.ai/v1` if your workspace is in the EU. Those are two different stacks serving different workspaces, not two names for the same one — a key is valid against one of them, not both. The dashboard is a browser app, reached at `platform.flex.ai` and never on an API host, so the two kinds of link are not interchangeable. If you integrated with us early and your code calls a FlexAI host that is not listed above, it keeps working. Nothing here deprecates it and there is no migration to do — these are the hosts to use for new work, not a replacement you need to move to. If you are unsure which one your key belongs to, ask us at [support@flex.ai](mailto:support@flex.ai) rather than guessing: a key sent to the wrong host returns `401`, which looks like a bad key. # Quickstart Source: https://docs.flex.ai/inference-api/quickstart Get an API key and make your first inference request in under two minutes. ## 1. Sign up Visit [platform.flex.ai](https://platform.flex.ai) and create an account. Sign-up is self-serve — no approval step and no waiting list. The one exception is a company that has claimed its own email domain and made it invitation-only. If that is yours, sign-up is refused and you need an invite from someone who already administers your organization. ## 2. Create a workspace Signing up gives you an organization; a **workspace** is what holds API keys, usage and logs, and you create the first one yourself from the dashboard. You'll be asked which region it lives in while you do. That choice is permanent, so if you need your inference to stay in the EU, read [Regions](/inference-api/reference/regions) before you pick. ## 3. Create an API key No card is needed for this step. On **Workspace home** in the dashboard, find **Create a new key**, name it, click **Create Key**, and copy the `sk-…` value — it's shown once. What the key does not carry yet is money. Inference draws against your account balance, and a new account's balance is zero, so your first request comes back [`402 Payment Required`](/inference-api/reference/errors) until you fund it. That is deliberate: you can wire the key into your code, your CI and your secret store before any money is involved, and add billing when you're ready to send real traffic. ## 4. Add funds From the dashboard's **Billing** page, add a billing address and a card. We can currently bill addresses in the European Union, the United States and India — if yours is elsewhere, the card cannot be added yet, and [support@flex.ai](mailto:support@flex.ai) can tell you where that stands. Then top up by any amount between \$5 and \$500 (€5 to €500 on an EU account). The amount you enter is the credit you get; sales tax or VAT is added on top of it at checkout. Your balance is credited once the payment settles — normally a few seconds, and the dashboard shows the top-up as pending until it does. From then on requests simply start succeeding, with no key regeneration and no restart. Adding a card also moves the key off the free rate-limit tier. See [Billing & Quotas](/inference-api/reference/billing) for the per-minute limits, how spend is metered, and what the `402` body looks like. You're billed only for what you use, at per-model rates. If your account carries credit — from a program, a negotiated offer, or the verified first top-up match on a US-dollar account — it draws down before your card is charged. See the [credits FAQ](/faq/credits). ## 5. Make a request Pick a text model from [the catalog](https://flex.ai/models). The examples below use `Llama-3.3-70B-Instruct-FP8`. ```bash cURL theme={null} curl https://api.flex.ai/v1/chat/completions \ -H "Authorization: Bearer $FLEXAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "Llama-3.3-70B-Instruct-FP8", "messages": [ {"role": "user", "content": "In one sentence, what is FlexAI?"} ] }' ``` ```python Python theme={null} # pip install openai import os from openai import OpenAI client = OpenAI( base_url="https://api.flex.ai/v1", api_key=os.environ["FLEXAI_API_KEY"], ) response = client.chat.completions.create( model="Llama-3.3-70B-Instruct-FP8", messages=[{"role": "user", "content": "In one sentence, what is FlexAI?"}], ) print(response.choices[0].message.content) ``` ```typescript TypeScript theme={null} // npm install openai import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.flex.ai/v1", apiKey: process.env.FLEXAI_API_KEY, }); const response = await client.chat.completions.create({ model: "Llama-3.3-70B-Instruct-FP8", messages: [{ role: "user", content: "In one sentence, what is FlexAI?" }], }); console.log(response.choices[0].message.content); ``` You should see a single-line description come back. That's it — your key works. ## Next steps Deliver tokens as they're generated instead of waiting for the full response. Let the model invoke your functions and work with their output. Filter the live catalog from code to find the right text, vision, or embedding model. Rate limits, credit exhaustion behavior, and the 402 response. Every 4xx/5xx we return, with example bodies and how to recover. # Billing & Quotas Source: https://docs.flex.ai/inference-api/reference/billing How we price requests, rate limits, credits, and what happens when your balance runs out. ## Getting started Creating an API key needs no card. Sign up, create a workspace (see [quickstart step 2](/inference-api/quickstart) — signup does not make one for you, and keys live inside it), then mint a key from **Workspace home** in the dashboard. It exists, authenticates, and appears in your key list immediately. Spending is what needs funding. Every request is checked against your balance before it reaches a model, and a new account's balance is zero, so until you add funds your requests come back `402 Payment Required` (the exact body is [below](#when-credit-runs-out)). Having no card, and having no balance, are both fine at key-creation time — so you can distribute the key, commit the integration and run your deploy pipeline before any payment detail exists. (Key creation is still refused in a few other cases: your role in the workspace, a workspace you are not an active member of, a workspace that has been suspended or archived, the per-workspace limit on how many keys may be live at once, and an account frozen over a payment dispute. Organization-level ownership is not a substitute for workspace membership — if you administer the organization but are not a member of the workspace, you cannot mint a key in it.) There are two ways to get a balance. Add a card and top up — see [Adding a card](#adding-a-card) — or carry credit — from the [FlexAI Startup Program](https://www.flex.ai/startups), a negotiated offer, or the verified first top-up match, which doubles your first top-up up to \$100 on US-dollar accounts once you have verified your identity. See the [credits FAQ](/faq/credits#how-does-the-first-top-up-match-work) for the terms and how to claim it. Credit draws down before your card is charged, so a credited account bills nothing until the credit is spent. New accounts do not come with an automatic credit grant; see the [credits FAQ](/faq/credits) for where credits come from. Funds are **workspace-level** — one wallet per workspace, topped up once, and every key in that workspace draws on the whole balance. There is no per-key allowance. Three things can still bound what a key spends, all set by someone else: a per-member spend cap set by a workspace admin, a monthly ceiling on the organization, and a cap we apply directly in rare cases. Rate limits, by contrast, really are per-key, and run out independently of money. ## How we bill Dollar-denominated, per-request. We don't sell token bundles, we don't subscribe you — we meter what you use: | Model type | Unit | | - | - | | Text / chat / completions / reasoning | Per million input tokens, per million output tokens. | | Embeddings | Per million input tokens. | Per-model pricing is in [the catalog](https://flex.ai/models). Spend appears in the dashboard within a few minutes of each request. Streaming a chat completion and stopping early still bills for the tokens that were generated before you closed the connection — the model did the work. Set `max_tokens` to cap worst-case spend. ### Per-request cost in the response Chat completion responses carry what the request cost you, in the `usage` block: | Field | Meaning | | - | - | | `usage.cost` | Our charge for this request, in USD. It equals the two fields below added together. | | `usage.cost_details.upstream_inference_prompt_cost` | The part of that charge for input tokens, at the model's input rate (and its cached-input rate for tokens served from the prompt cache). | | `usage.cost_details.upstream_inference_completions_cost` | The part of that charge for output tokens, including reasoning tokens. | These are **FlexAI's charge to you**, not an upstream provider's cost to us. The `upstream_inference_*` names follow OpenRouter's usage-accounting fields, so tools that already read them work unchanged. Where they appear: * **Non-streaming:** on the response's `usage`. * **Streaming:** on the `usage` of the final chunk. Send `stream_options: { "include_usage": true }` to get that chunk. Values are USD numbers rounded to 8 decimal places. The fields are absent, not `0`, when the response cannot state the charge yet. That happens when the gateway has no per-token price for the model, or when a stream's final chunk reports zero tokens and the count is corrected after the stream ends. The request is still billed. To read the recorded charge for one request, call `GET /v1/flex/usage/request/{request_id}` with the `x-litellm-call-id` response header. `GET /v1/pricing` is the published price list. These charges are computed from it. ## Rate limits Per-API-key, per-minute. Limits are set by tier: | Tier | Requests per minute | | - | - | | Free (default on signup) | 10 | | Paid | 100 | | Custom | [Contact FlexAI](mailto:support@flex.ai) for higher limits | A key minted on an account with no card on file starts on the free tier. Adding a card moves the account's keys to the paid tier — it is having the card, not having spent anything, that makes the difference, and it is the only tier change you can make yourself. It is not instantaneous: the promotion is applied by a reconciler shortly after the card is added, so allow a minute before judging the limit from the headers. Keys an admin has given an explicit per-member rate limit keep that limit, and so do disabled keys and suspended members. Beyond the paid tier, limits are set by hand: email [support@flex.ai](mailto:support@flex.ai) and tell us the rate you need. Every response from the API carries rate-limit headers: | Header | Meaning | | - | - | | `x-ratelimit-limit-requests` | RPM ceiling for this key. | | `x-ratelimit-remaining-requests` | Requests left in the current window. | | `x-ratelimit-reset-requests` | Unix timestamp when the window resets. | When you exceed the limit you get `429 Too Many Requests` with a `Retry-After` header (seconds to wait). Don't retry in a tight loop — back off until the header's deadline. ## When credit runs out Our API returns `402 Payment Required`: ```json theme={null} { "error": { "message": "You exceeded your budget. Please add credits to continue.", "type": "insufficient_quota", "code": "insufficient_quota" } } ``` Chat and completions answer a little differently — that path checks the request's worst-case cost against your balance before generating, so the code is `insufficient_balance` and the useful numbers arrive under `error.provider_specific_fields`. [Errors](/inference-api/reference/errors#402--payment-required) has both shapes in full; render the fields rather than matching on the sentence. Once you add credit, subsequent requests succeed immediately — no key regeneration needed. ## Adding a card From the [billing page](https://platform.flex.ai/billing): 1. Click **Add payment method**. 2. Enter card details (processed by Stripe; we never see the number). 3. Top up by any amount between \$5 and \$500 — or €5 to €500 if your account bills in euros, which is a native range rather than a converted one. The amount you enter is the credit you receive. **Any sales tax or VAT is added on top of it**, so a €100 top-up credits €100 of balance and charges your card €100 plus the VAT for your country. The exact total is shown before you confirm. Your balance is credited once the payment settles, which is usually seconds but is not instant — the dashboard shows the top-up as pending until then. You can also set an **auto-recharge** rule on the same page: pick a balance threshold and a top-up amount, and we charge the card on file when the balance falls below it, which makes a long-running job much less likely to stop at `402`. With no rule set, nothing is charged automatically. It is not a guarantee, and it is worth knowing why. The rule is evaluated when something moves your balance rather than on a timer, so an account that is already below the threshold and has just come off a short cooldown waits for its next balance-moving event instead of being swept up. Attempts are bounded too: by default no more than three in any rolling 24 hours, with a short cooldown between them — or, if you set a monthly spending cap on the rule, by that cap instead. And the card still has to clear. If your balance reaches zero with a rule in place, one of those is why. The emails differ depending on which you are on. **With no auto-recharge rule**, we warn you once your balance drops under a fifth of what it was straight after the credit that started the current run — usually a top-up, but a promotional bonus or an adjustment we apply counts too — so a run starting at \$50 warns below \$10. We write again when the balance reaches zero. **With auto-recharge on**, we send only the second one, because a balance that reaches zero with a rule in place means a recharge did not go through — usually a declined card or a cap being hit. Accounts in the EU region are charged and invoiced in euros rather than dollars; see [Regions](/inference-api/reference/regions#billing-currency). # Errors Source: https://docs.flex.ai/inference-api/reference/errors Every 4xx and 5xx status we return, with an example body and what to do about it. **Branch on the HTTP status code, not the body.** The status is what the OpenAI SDKs map to exception classes, and it's stable across every layer of our stack. Where a human-readable explanation exists it's safe to surface in your UI, but don't assume it's there. It's `error.message` on the envelope (including chat gate rejections, which also keep their structured payload under `error.provider_specific_fields`), top-level `message` on the flat shape, `detail` on the bare shape and on one flat body — and a few carry no explanation at all. Fall back to the status plus the `error` string code. Which shape you get depends on **which layer turned you away**, so don't assume `error` is an object — or that it's present at all. Endpoints we serve directly (`/v1/images`, `/v1/videos`, `/v1/audio`, and parts of `/v1/flex/*`) answer in either the OpenAI envelope or a flatter shape where `error` is a string code with the context beside it. Chat and completions run through our model router, which always answers in the envelope. And the 502s on `GET /v1/flex/key` and `GET /v1/models` return a bare `detail` string with no `error` key: ```json OpenAI envelope theme={null} { "error": { "message": "…", "type": "invalid_request_error", "param": "size", "code": null } } ``` ```json Flat shape theme={null} { "error": "model_not_supported", "message": "Model 'NoSuchModel-XYZ' is not available on this endpoint.", "supported_models": ["FLUX.1-schnell", "FLUX.2-dev", "stable-diffusion-3.5-large"] } ``` ```json Gate rejection (chat) theme={null} { "error": { "message": "Model id not recognized. …", "type": "not_found_error", "param": null, "code": "unknown_model", "provider_specific_fields": { "error": "unknown_model", "model": "Llama-3.3-70B-Instruc", "message": "Model id not recognized. …", "did_you_mean": ["Llama-3.3-70B-Instruct"] } } } ``` ```json Bare detail theme={null} { "detail": "upstream gateway unavailable" } ``` When a chat request is turned away by one of our pre-flight gates — unknown model, insufficient balance, region not served — the envelope carries the gate's sentence in `error.message` and its code in `error.code` (`unknown_model`, `insufficient_balance`, `region_not_served`), with `type` and `param` as proper JSON values. The gate's structured payload (`did_you_mean`, `suggested_model`, `max_usage_cost_usd`, `key_remaining_usd`, `region`, …) stays under `error.provider_specific_fields`, where earlier releases put everything, so code written against that keeps working. A gate with no sentence (the balance gate) repeats its code as `error.message`; render its payload fields instead. On the model-router endpoints (chat, completions, embeddings, responses and videos), error messages never carry our internals: no exception class names, file paths, engine names or hostnames. When a model backend's own message can't be shown without them, you get a generic message for the status instead. A 5xx, or an error inside a stream, carries its own message only when its `code` is one we document on this page; any other server-side failure gets the generic message. This check fails closed: text that merely looks internal is treated as internal. A message that mentions a name ending in `.local`, `.internal` or a similar private suffix (for example `file.local`), or a private, loopback, link-local or unique-local IP address (for example `10.0.0.5`, `fd00::2` or a bare `::`), is replaced by the generic message too, even when it is harmless. We're converging these shapes. Until then, treat the status as the contract and every field in the body as optional — read `error` defensively (object, string, or absent), and when it's an object, check `provider_specific_fields` before concluding there's no detail. ## What we return The `Where` column matters: `code` is **not** consistent across endpoint families, so switch on the status and treat `code` as a debugging aid, not a contract. | Status | `type` | `code` | Where | When | | - | - | - | - | - | | 400 | `invalid_request_error` | `null`, or a stable code where one exists | chat | Malformed body, bad field value, prompt over the context window, unfetchable `image_url` | | 400 | `invalid_request_error` | `string_too_long` | audio | Text-to-speech `input` over the model's character limit | | 400 | `invalid_request_error` | `invalid_request_body` | media | Body isn't valid JSON, or the wrong content type | | 400 | *flat* `invalid_request` | — | video | Async job rejected at submission (e.g. unsupported dimensions) | | 400 | *flat* `model_not_supported` | — | media | Model isn't valid for the endpoint you called | | 400 | *flat* `unsupported_audio` | — | audio | Uploaded audio couldn't be decoded | | 401 | `invalid_request_error` | `missing_api_key` | media | No `Authorization` header | | 401 | `invalid_request_error` | `invalid_api_key` | media | Key malformed, unknown, revoked, or expired | | 401 | `invalid_request_error` | `missing_authorization` | flex | No `Authorization` header | | 401 | `invalid_request_error` | `malformed_authorization` | flex | Header present but not `Bearer sk-…` | | 401 | `authentication_error` | `invalid_api_key` | flex | Key unknown, revoked, or expired | | 401 | `auth` | *none* | `/v1/models` | Missing or invalid key | | 402 | `insufficient_quota` | `insufficient_quota` | media | Out of credit, monthly spend cap reached, or account on hold | | 402 | `insufficient_quota` | `insufficient_balance` | chat | Worst-case request cost exceeds the remaining balance — payload under `provider_specific_fields` | | 403 | `invalid_request_error` | `region_not_served` | media | Your org's data region isn't served by this endpoint | | 403 | `permission_error` | `region_not_served` | chat | The same region gate on the chat plane — payload under `provider_specific_fields` | | 403 | `auth` | *none* | `/v1/models` | Key is blocked | | 404 | `invalid_request_error` | `model_not_found` | media | Media model not currently available | | 404 | — | `model_deactivated` | chat | Model not currently available (stopped, scaled to zero, or its cluster is down) | | 404 | *flat* `pricing_unavailable` | — | media | We couldn't price the request, so we declined to run it | | 404 | `not_found_error` | `unknown_model` | chat | Model id we don't recognize — payload under `provider_specific_fields` | | 404 | `invalid_request_error` | `request_not_found` | flex | Unknown request id in a usage lookup | | 410 | *flat* `job_already_retrieved` | — | video | This completed clip was already downloaded; video results are one-shot | | 410 | *flat* `job_expired` | — | video | The one-hour pickup window elapsed, or the job is no longer available | | 413 | *flat* `payload_too_large` | — | audio | Speech-to-text upload over the size limit | | 422 | `invalid_request_error` | `invalid_request_body` | media | A field is missing or semantically invalid — `param` names it | | 422 | *flat* `invalid_request` | — | audio | Endpoint needs `multipart/form-data`; the explanation is in `detail`, not `message` | | 429 | `rate_limit_error` | `rate_limit_exceeded` | chat, media | Over your key's requests-per-minute ceiling | | 429 | `rate_limit` | *none* | `/v1/models` | Over the discovery endpoint's own throttle | | 429 | `capacity_error` | `gateway_capacity_exhausted` | chat | Your account is moving between serving planes, or no engine arm can take the request right now; retry after `Retry-After` | | 500 | `api_error` on chat, *(varies)* elsewhere | `null` | all | Unexpected failure in our gateway or a model backend | | 502 | *bare* `detail` | — | `/v1/flex/key`, `/v1/models` | An upstream call inside our gateway was unreachable, rejected, or returned an unusable body | | 503 | `api_error` | `service_unavailable` | media | Key verification temporarily unavailable | | 503 | `api_error` | `auth_service_unavailable` | flex | Key verification temporarily unavailable | | 503 | `service_unavailable_error` | `balance_unavailable` | chat | Your balance couldn't be checked right now (not a sign you're out of credit); retry after `Retry-After` | | 503 | *flat* `backend_unavailable` | — | media | A media backend is temporarily unreachable | | 503 | *flat* `transcode_overloaded` | — | audio | Audio processing is saturated; retry shortly | | 503 | *nested* `global_plane_unavailable` | — | portal | An internal control-plane dependency was briefly unreachable; retry shortly | | 503 | *nested* `stripe_tenant_not_configured` | — | portal | Billing is not fully configured for your account's region; we refuse rather than charge the wrong account | | 504 | `gateway_timeout` | `usage_timeout` | flex | A usage query took too long | `Where` reads as: **chat** = `/v1/chat/completions`, `/v1/completions`; **media** = `/v1/images`, `/v1/videos`, `/v1/audio`, with **audio** and **video** called out where a code belongs to only one of them; **flex** = `/v1/flex/*`; **portal** = the dashboard's `/api/*` management routes, rather than the Inference API. A `code` of `null` matches OpenAI, which reserves non-null codes for a narrow set. The 500 row is the exception: we don't author those bodies, so treat its `type` and `code` as unstable. The chat 402 and 403 rows share their envelope builder with the chat 404, which is verified live; provoking them needs an org that is out of credit or pinned to an unserved region. ## 400 — Bad request Malformed body, or a field value we can't serve. `param` names the offending field when we can identify one. ```json theme={null} { "error": { "message": "'size' must be a WxH string with each dimension in [64, 2048] (e.g. '1024x1024').", "type": "invalid_request_error", "param": "size", "code": null } } ``` A 400 is deterministic — retrying the same request fails the same way. Change it or fail the call. ### Prompt longer than the context window Refused before any tokens are generated, so you aren't billed. Every such refusal is a 400 with `code: "context_length_exceeded"`, so you can branch on the code rather than on the wording. `param` names the field that carries the prompt: `"messages"` on chat completions, `"input"` on `/v1/responses` and `/v1/embeddings`, and `"prompt"` on `/v1/completions`: ```json theme={null} { "error": { "message": "This model's maximum context length is 131072 tokens. Reduce the length of the messages or the requested output tokens.", "type": "invalid_request_error", "param": "messages", "code": "context_length_exceeded" } } ``` The refusal can come from two places, and both use this shape. When the model's serving engine refuses the prompt, the message usually names the model's maximum context length, as above; on the few engines that do not report it, the message names no token counts. When a model is served on more than one kind of hardware, the gateway can refuse an oversized request itself, before it reaches an engine; that message names no token counts, because the gateway sizes the request with an estimate rather than the engine's tokenizer. Treat both the same way: shorten the prompt or lower the output cap, which is `max_tokens` or `max_completion_tokens` on chat completions and `max_output_tokens` on `/v1/responses`. A non-streamed request that sets no output cap is sized with its default of 2048 output tokens (none on reasoning models), so setting a smaller cap explicitly can also bring it under the limit. A streamed request is refused the same way, with the same 400 before any event is sent. Read the limit from `context_length` on `GET /v1/models` ([model discovery](/inference-api/guides/model-discovery)) rather than hard-coding it — a model's window changes when we move it to different hardware. The window covers the prompt **and** the completion. A `max_tokens` on top of a prompt that nearly fills the window can still be refused. ### Unfetchable or undecodable `image_url` A URL we can't fetch, or bytes we can't decode, is your fault rather than an outage — so it's a 400, not a 500: ```json theme={null} { "error": { "message": "The provided image_url could not be fetched or decoded: 404, message='Not Found'", "type": "invalid_request_error", "code": null } } ``` Check the URL is publicly reachable, returns image bytes, and uses an `http(s)` or `data` scheme. Genuine gateway failures on an image request still return 500, so treat this 400 as "fix the URL". ### Forced `tool_choice` the model didn't honor Pin `tool_choice` to `required` or a named function and get no tool call back, and we fail the request rather than return a toolless answer your agent loop would mistake for a result: ```json theme={null} { "error": { "message": "The model 'gpt-oss-120b' returned no tool call, but tool_choice requested required. …", "type": "invalid_request_error", "param": "tool_choice", "code": "tool_choice_not_honored" } } ``` Retry with `tool_choice="auto"`, or use a model that supports forced tool selection. It's a 400 rather than a 5xx so a retrying agent changes the request instead of looping. See [tool use](/inference-api/guides/tool-use). The same rule holds when you stream. A stream has already started with a 200 by the time the model's answer is known, so the rejection arrives as the stream's last event instead of a status: the reasoning deltas you have already received stand, the prose the model produced instead of a call is withheld (up to a few thousand characters; a model that keeps writing past that is streamed from there on, and the verdict still ends the stream), and the stream ends with one `error` event carrying the same `type`, `param` and `code` as above, followed by `data: [DONE]`. When `stream_options.include_usage` is set, the usage chunk arrives ahead of the `error` event: the tokens the model generated are billed, as the answer was produced before it could be judged. On `/v1/responses` it is the flat `error` event (`"type": "error"` with `code`, `message` and `param` beside it, and a `sequence_number` continuing the stream's). A named function is honoured only by calls to that function; a reply that calls a different tool, instead of or as well as the one you named, is rejected with the same code and a message naming the function you asked for. Treat that event exactly as you would the 400: change the request rather than retry it as-is. Here is how it looks on chat completions: ```text theme={null} data: {"id":"chatcmpl-…","choices":[{"delta":{"reasoning_content":"…"}}],…} data: {"id":"chatcmpl-…","choices":[],"usage":{"prompt_tokens":52,"completion_tokens":31,"total_tokens":83}} data: {"error":{"message":"The model 'Muse-Glimmer-30B' returned no tool call, but tool_choice requested required. …","type":"invalid_request_error","param":"tool_choice","code":"tool_choice_not_honored"}} data: [DONE] ``` Non-streamed only, the same `code` also covers a forced reply that stopped at the output token limit before its tool call was complete (`finish_reason: "length"` on chat completions; `status: "incomplete"` with `incomplete_details.reason: "max_output_tokens"` on `/v1/responses`): the calls it carries are not one complete turn, so the reply is rejected rather than handed to your agent. Raise `max_tokens` (`max_output_tokens` on `/v1/responses`), or use `tool_choice="auto"`. The exception is `"required"` on the models the gateway bounds to one call (`GLM-5.2`, `MiniMax-M2.7`, `NVIDIA-Nemotron-3.5-Lightning-30B-A3B`; see the [tool use guide](/inference-api/guides/tool-use)): there, a reply with several calls at the limit means the model kept repeating its call, and a larger budget would only let it run longer. That reply is rejected as an unsafe required-tool response instead, and the message does not mention the limit: use `tool_choice="auto"` or a named tool choice. On those models this check applies to streams too. A stream releases tool-call deltas as they arrive and cannot take them back, so a streamed reply cut off at the limit ends with its `finish_reason` and no `error` event; check `finish_reason` yourself before dispatching the calls. ## 401 — Unauthorized Check the `Authorization: Bearer sk-…` header. If the key looks right it may be revoked or expired — issue a new one from the [dashboard](https://platform.flex.ai/dashboard/keys). ```json theme={null} { "error": { "message": "Invalid API key provided", "type": "invalid_request_error", "code": "invalid_api_key" } } ``` **Don't branch on `type` or `code` for a 401.** Three endpoint families report the same rejection differently today — the media plane uses `missing_api_key` / `invalid_api_key`, `/v1/flex/*` uses `missing_authorization` / `malformed_authorization` and switches `type` to `authentication_error` for a bad key, and `/v1/models` returns `type: "auth"` with **no `code` field at all**. Branch on the 401 status. We're unifying these onto one shape, so anything you match on the body today is liable to change. The OpenAI SDKs are unaffected — they map 401 to their authentication-error class by status code, not by body — so this only matters if you're calling the API over raw HTTP. Note the `/v1/models` case in particular: reading `error.code` there gets you `None`, not a string. Some 401s also carry a `doc_url` pointing at the dashboard page that fixes them. ## 402 — Payment required Four causes share this status; `message` tells them apart — credit exhausted, monthly spend cap reached, account on hold, or not provisioned for billing. ```json theme={null} { "error": { "message": "You exceeded your budget. Please add credits to continue.", "type": "insufficient_quota", "code": "insufficient_quota" } } ``` Chat and completions have one more: before generating, we check the request's worst-case cost against your remaining balance. That one is a gate rejection: `error.code` is `insufficient_balance`, and the actionable content is under `error.provider_specific_fields`, as `max_usage_cost_usd` (what the request could cost) and `key_remaining_usd` (what's left). Render those rather than looking for a sentence. Top up on the [billing page](https://platform.flex.ai/billing), or ask an org admin to raise the cap. The same key resumes working — nothing needs reissuing. See [billing & quotas](/inference-api/reference/billing). ## 403 — Forbidden Two causes, told apart by `code`. ### Region not served Your org is pinned to a data region this endpoint doesn't serve, so we decline rather than move your data out of it. On the media endpoints that's a plain envelope: ```json theme={null} { "error": { "message": "This organization's data region is not served by this endpoint.", "type": "invalid_request_error", "code": "region_not_served" } } ``` On chat and completions the same gate answers with `error.code` `region_not_served`, and its payload (including `region`) under `error.provider_specific_fields`. This is not the error you get for calling the wrong host. The two regions keep separate key stores, so a key presented to the endpoint it does not belong to fails authentication first and comes back [`401`](#401--unauthorized) — see [Regions & Data Residency](/inference-api/reference/regions). This `403` means the workspace and the stack serving it disagree, which should not arise in normal use: report it to [support@flex.ai](mailto:support@flex.ai) rather than retrying against the other host. A region is permanent once set, and that is enforced in the database — it cannot be changed from the dashboard, and we cannot change it for you either. If a workspace ended up in the wrong region, support will help you set a new one up in the right region instead. ### Blocked key On `/v1/models`, a key an administrator has blocked returns 403 with `type: "auth"` and no `code`. Contact [support@flex.ai](mailto:support@flex.ai) — a blocked key can't be unblocked from the dashboard. ## 404 — Not found On chat and completions, an unrecognized model id is a gate rejection, so the `did_you_mean` list arrives nested under `error.provider_specific_fields`: ```json theme={null} { "error": { "message": "Model id not recognized. See /v1/models for the full list.", "type": "not_found_error", "param": null, "code": "unknown_model", "provider_specific_fields": { "error": "unknown_model", "model": "Llama-3.3-70B-Instruc", "message": "Model id not recognized. See /v1/models for the full list.", "did_you_mean": ["Llama-3.3-70B-Instruct"] } } } ``` When a checkpoint we no longer serve has a declared replacement, that inner object also carries `suggested_model` — a deterministic pointer, not a spelling guess. A media model that exists but isn't currently available uses the envelope with `code: "model_not_found"`. For models, use the canonical id from `GET /v1/models` — we require the bare model name, not a provider-prefixed path. ### Model in the catalog but not currently running A model can be in our catalog and still not be running right now — we may have stopped it deliberately, it may have scaled down to zero after a quiet period, or the cluster hosting it may be having a problem. Requests for it come back 404 with `code: "model_deactivated"`: ```json theme={null} { "error": { "code": "model_deactivated", "message": "The model 'Qwen3-Coder-Next' is not available. See GET /v1/models for the models you can call right now." } } ``` It's a 404 rather than a 503 because we don't spin a stopped model up to serve the request that found it stopped. Retrying the same call won't get you a different answer on any timescale you'd want to wait for, so treat this as "pick a different model", not "back off and try again". `GET /v1/models` is the list to pick from. It's built from the same signal this check uses, so the two tell the same story — though they're each cached for a few seconds, so right after a model stops or starts you may briefly see one before the other catches up. If you get this error for a model the listing still shows, wait a few seconds and re-read the listing. ### Model we can't price Rare, and a bug on our side rather than yours: `pricing_unavailable` means we couldn't resolve a price for a media request, so we declined to run it rather than serve work we can't bill. You were not charged. ```json theme={null} { "error": "pricing_unavailable", "message": "The model 'some-model' is not available. The request was not run and you were not charged. Use a different model from GET /v1/models, or contact support@flex.ai." } ``` It's a 404 for the same reason as the case above: it takes a configuration fix on our side, so no amount of retrying clears it. Use a different model and tell us at [support@flex.ai](mailto:support@flex.ai) — we'd like to know. Both of these moved on 2026-09-20: `model_deactivated` from 503 to 404, and `pricing_unavailable` from 500 to 404. If you special-cased either as a retryable 5xx, that branch is now dead — the `error` codes are unchanged, so matching on those still works. ## 410 — Gone Video results have a one-hour, one-download pickup window: ```json theme={null} { "error": "job_already_retrieved", "message": "This video was already retrieved. Each completed clip can be downloaded once." } ``` ```json theme={null} { "error": "job_expired", "message": "The video is no longer available. Clips expire one hour after completion." } ``` Download a completed video promptly and persist it on your side if you need it later. A job belonging to another key uses the same unavailable response as an expired job, so one key cannot probe for another key's work. ## 413 — Payload too large A speech-to-text upload over the limit. `max_size_mb` always reports the limit, so read it from there rather than hard-coding a number: ```json theme={null} { "error": "payload_too_large", "message": "Audio file exceeds 25 MB limit.", "max_size_mb": 25 } ``` We check twice — once against `Content-Length` before reading the upload, and once after, in case the header understated it. The second check omits `message`, so this is one of the bodies where you need the `error` code and `max_size_mb` rather than a sentence. Split or downsample the audio. See [audio](/inference-api/guides/audio). ## 422 — Unprocessable content The body parsed but a field is wrong. `param` names it, and `code` is `invalid_request_body`: ```json theme={null} { "error": { "message": "Missing required parameter: 'model'.", "type": "invalid_request_error", "param": "model", "code": "invalid_request_body" } } ``` Speech-to-text has its own 422 for the wrong content type, and it's the one body on the whole API that puts its explanation in `detail` rather than `message`: ```json theme={null} { "error": "invalid_request", "detail": "This endpoint requires multipart/form-data. Send the audio file as form field 'file' and the model name as form field 'model'." } ``` A 400 on the same endpoints means the body didn't parse at all — wrong content type or malformed JSON — whereas a 422 means it parsed and a field failed validation. ## 429 — Too many requests Over the per-key requests-per-minute ceiling. | Header | Meaning | | - | - | | `Retry-After` | Whole seconds to wait before retrying. | | `x-ratelimit-limit-requests` | RPM ceiling for this key. | | `x-ratelimit-remaining-requests` | Remaining in the window — always `0` on a 429. | | `x-ratelimit-reset-requests` | Time until the window resets, as a duration string such as `34s`. | `x-ratelimit-reset-requests` is a **duration**, not a Unix timestamp. Parse it as seconds-with-a-trailing-`s`, or use `Retry-After`, which carries the same number as a bare integer. ```json theme={null} { "error": { "message": "Rate limit exceeded: Limit 100 RPM, Used 100. Please try again in 34s.", "type": "rate_limit_error", "code": "rate_limit_exceeded" } } ``` Back off until `Retry-After`. If you hit the ceiling consistently, request a tier upgrade from [support@flex.ai](mailto:support@flex.ai). See [concurrency & rate limits](/inference-api/guides/concurrency). You don't have to wait for a 429 to see your headroom. A successful non-streaming response from a key with an RPM limit carries two more headers, with a different prefix from the ones above: `x-ratelimit-api_key-limit-requests` (the key's RPM ceiling) and `x-ratelimit-api_key-remaining-requests` (what's left in the current window, after this request). Streaming responses don't carry them. A 429 with `code: "gateway_capacity_exhausted"` (`type: "capacity_error"`) is a different thing: not your key's ceiling, but no capacity to take the request at that moment. It also carries `Retry-After`; retry after it. `GET /v1/models` throttles separately, on its own per-IP and global counters rather than your key's RPM, and reports it as `type: "rate_limit"` with no `code`. Cache the catalog instead of polling it — it changes rarely. ### The other 429: a flood limit at the edge The shape above is the **per-key** limit, and it is the one you will meet in normal use. There is a second, much higher ceiling applied per **source IP address** at our edge proxy, before a request reaches the API at all. It exists to absorb floods and is set far above any legitimate client's traffic — you should never see it. Because the proxy enforces it rather than the API, **its 429 carries different headers**, and the names are close enough to the ones above to be mistaken for them: | | Per-key 429 (above) | Edge 429 | | - | - | - | | `Retry-After` | Yes | **Absent** | | Limit | `x-ratelimit-limit-requests` | `x-ratelimit-limit` | | Remaining | `x-ratelimit-remaining-requests` | `x-ratelimit-remaining` | | Reset | `x-ratelimit-reset-requests`, a duration such as `34s` | `x-ratelimit-reset`, a **bare integer** of seconds | | Body | JSON `error` object | `text/plain`, the literal string `local_rate_limited` | An edge 429 looks like this in full: ```http theme={null} HTTP/2 429 content-type: text/plain x-ratelimit-limit: 100000 x-ratelimit-remaining: 0 x-ratelimit-reset: 16 local_rate_limited ``` Match rate-limit headers **exactly**, not by prefix. `x-ratelimit-reset` and `x-ratelimit-reset-requests` are different headers with different value formats — one a bare integer, the other a duration string — so a loose match will parse one as the other. And because the edge 429 has no `Retry-After`, treat that header's absence as "back off using `x-ratelimit-reset`, or your own exponential backoff" — never as "retry now". If you are seeing edge 429s, retrying harder will not help and backing off will: either you are sending genuinely extreme volume from one address, or many of your clients share a single egress IP (an office NAT or VPN gateway) and are counted as one caller. Get in touch at [support@flex.ai](mailto:support@flex.ai) rather than tuning your retry loop. ## 500 — Internal server error An unexpected failure in our gateway or a model backend. We don't shape these bodies, so read `message` and treat `type`/`code` as unstable. Retry with backoff. We don't support idempotency keys, so only retry requests you're willing to have execute twice. Persistent 500s mean an incident — check [status.flex.ai](https://status.flex.ai). ## 502 — Bad gateway Only `GET /v1/flex/key` and `GET /v1/models`. A component inside our gateway that these two endpoints depend on was unreachable, rejected our own credentials, or answered with a body we couldn't use. We fail closed on all three rather than serve a result we can't stand behind — an unverified key, or a credit figure we're not sure of. This is the one status that returns a bare `detail` string with no `error` key, so a client that reaches for `error.message` gets `undefined` here: ```json theme={null} { "detail": "upstream gateway unavailable" } ``` The `detail` values are `upstream gateway unavailable`, `upstream key verification unavailable`, and `upstream gateway returned an unusable response`. They're diagnostic, not a contract — don't branch on them. Retry with backoff. Both endpoints that return a 502 are read-only lookups, so retrying is always safe. Persistent 502s mean an incident — check [status.flex.ai](https://status.flex.ai). ## 503 — Service unavailable `code` says which dependency: `backend_unavailable` for a media backend we can't reach, `transcode_overloaded` when audio processing is saturated, `service_unavailable` / `auth_service_unavailable` when key verification itself is down (we fail closed rather than serve an unverified request), `balance_unavailable` on chat when your balance couldn't be checked (we refuse rather than run an unbilled request; it says nothing about your credit), and `global_plane_unavailable` when one of our own internal control-plane services was momentarily unreachable — typically a restart during a deploy, lasting seconds. `stripe_tenant_not_configured` means billing for your account's region is not fully set up on our side. We refuse the operation rather than complete it against the wrong payment processor account, so nothing is charged. It needs an operator's attention rather than a retry — contact support if you see it. Both of these arrive nested, under `detail`, rather than flat: ```json theme={null} { "detail": { "code": "global_plane_unavailable" } } ``` ```json theme={null} { "error": "backend_unavailable", "message": "Image generation failed: upstream unreachable." } ``` Retry with backoff. Every 503 here is a dependency we expect back. ## 504 — Gateway timeout Only `/v1/flex/*`. A usage query ran longer than the gateway allows, reported as `type: "gateway_timeout"` with `code: "usage_timeout"`. ```json theme={null} { "error": { "message": "Usage query timed out.", "type": "gateway_timeout", "code": "usage_timeout" } } ``` Narrow the time range and retry. This is a read-only query, so retrying is always safe. # OpenAI Compatibility Source: https://docs.flex.ai/inference-api/reference/openai-compatibility What is and isn't supported at launch when you point an OpenAI SDK at api.flex.ai. Point an OpenAI SDK at `https://api.flex.ai/v1` and, for everything marked Supported below, code written against `api.openai.com/v1` works by changing only `base_url` and `model`. This page is the exact compatibility matrix — the endpoints and request fields the gateway implements at launch. That is also why agent frameworks built on the OpenAI client work here without a FlexAI-specific package. If you are using LangChain or LangGraph, [their integration page](/inference-api/agents/langchain) is the shortest path: the same `base_url` change, plus a per-model table of what has actually been measured to work for tool calling, streaming, structured output and a full agent loop. ## Spec version we track The chat-completions request validator at our gateway is authored against **`openai-python` 2.33.0** — specifically the request shape `openai.types.chat.completion_create_params.CompletionCreateParamsBase`. That same version is asserted in our gateway image, so the SDK you call against and the shape we validate can't drift apart silently. We validate a focused subset of the SDK's surface — the fields where validation gaps were causing real problems (off-spec inputs returning `200` with garbage, or `500` with a stack trace). Unknown or unsupported fields are forwarded to the engine and accepted or rejected there. The matrix below is the exact subset. Bumps are deliberate: when we upgrade the pin, a snapshot test fails until we audit the SDK diff, and we update this page in the same PR. ## Endpoints | Endpoint | Supported | Notes | | - | - | - | | `POST /v1/chat/completions` | Yes | Streaming, tools, vision (on vision models), seed, response\_format. | | `POST /v1/completions` | Yes | Legacy; prefer chat completions for new integrations. No tool calling: a request carrying `tools` or `tool_choice` returns a `400` with `param` naming the field. | | `GET /v1/models` | Yes | Lists every **live** model across all modalities — text, code, reasoning, vision, embeddings, image, audio, video — in OpenAI's `Model` shape, each annotated with a `category` field so you can pick the matching endpoint. Only models currently serving appear. See [model discovery](/inference-api/guides/model-discovery) for the category→endpoint mapping, or the [catalog](https://flex.ai/models) for the full roster with pricing. | | `POST /v1/embeddings` | Yes | `bge-m3` (multilingual, 1024-dim, 8K context). `encoding_format` is optional and defaults to `"float"`; `"base64"` is also accepted. (Pass `"float"` or omit the field — don't send an explicit `null`.) See the [model catalog](https://flex.ai/models). | | `POST /v1/fine-tuning` | No | Use [FlexAI Fine-Tuning](/core-services/fine-tuning) instead. | | `POST /v1/assistants`, `/v1/threads`, `/v1/runs` | No | Not planned. | | `POST /v1/batches` | No | Coming soon — [reach out](mailto:support@flex.ai) if you need it. See [Batching](/inference-api/guides/batch). | | `GET /v1/files`, `/v1/files/*` | No | Use inline request payloads. | ## Request fields | Field on `POST /v1/chat/completions` | Supported | Notes | | - | - | - | | `model`, `messages`, `temperature`, `top_p`, `stop`, `seed`, `user` | Yes | A sampling field you omit may be filled with the model's recommended default (for example `top_p: 0.95` on `DeepSeek-V4-Flash-0731`, the value its model card recommends for agentic use). Only omitted fields are filled: an explicit value, including `temperature: 0`, is always sent as-is. The response names what was filled in `x-flexai-sampling-defaulted` (for example `top_p`); no header means nothing was filled. The same applies on `/v1/responses`. | | `max_tokens` | Yes | An explicit value is honored as-is (`max_completion_tokens` is honored too, though not yet listed in `supported_parameters`). Non-streaming requests that set neither are bounded by a \~95 s generation budget — [stream](/inference-api/guides/streaming) for longer outputs — and, on non-reasoning models only, default to 2048 output tokens; the response then carries `x-flexai-max-tokens-defaulted: 2048`. Reasoning models (always-thinking models such as `GLM-5.3-Flash`, `gpt-oss-120b`, `Step-3.7-Flash`, and the hybrids) get no default, because the reasoning would spend it and return an empty answer. A non-streamed response that ends with `finish_reason: "length"` and no visible content or tool call carries `x-flexai-empty-at-length: true` — the model was cut off, not incapable; retry with a larger `max_tokens` or stream. On tool turns set 4096 or more: a tool call cut off by `max_tokens` returns `finish_reason: "length"` with no `tool_calls`. | | `stream`, `stream_options.include_usage` | Yes | Set `include_usage: true` when streaming for correct billing. A stream the gateway ends because the model was repeating itself carries `finish_reason: "length"` and `flexai_stopped: "degeneration"` on its terminal chunks (`"ceiling"` when it was cut at the token ceiling while repeating) — see [Runaway streams](/inference-api/guides/streaming#runaway-streams). | | `tools`, `tool_choice` | Yes | On models labeled `tool_use` in the catalog. | | `response_format: { type: "json_object" }` | Partial | On models listing `response_format` in their `/v1/models` `supported_parameters`; other models return a `400` with `param: "response_format"`. | | `response_format: { type: "json_schema", ... }` | Partial | On models listing `structured_outputs` in their `/v1/models` `supported_parameters`; other models return a `400` with `param: "response_format"` (there is no automatic fallback to `json_object`). | | `logprobs`, `top_logprobs` | No | Not at launch. | | `n` (multiple completions per request) | No | Gateway returns a single choice; sending `n > 1` returns a `400` with `param: "n"`. | | `presence_penalty`, `frequency_penalty` | Yes | — | | `reasoning_effort` | Yes | Accepted on every chat model; never a 400. Levels: `minimal`, `low`, `medium`, `high`, `max` (case-insensitive), plus `none` and `xhigh` on the switch-only models below. Each model's chat template understands its own words, so the gateway translates per model: `gpt-oss-120b` / `gpt-oss-20b` take `low`, `medium`, `high` natively; `DeepSeek-V4-Flash-0731` — `low`/`minimal` keep reasoning off, `medium`/`high` select its high level, `max` its maximum; `GLM-5.2` — `minimal`/`low` off, `medium`/`high` → high, `max` → max; `Qwen3.8-27B` — `minimal` off, `low`/`medium` as sent, `high`/`max` → xhigh; `GLM-5.3-Flash` (always-thinking) — `minimal`/`low` → low, `medium`/`high` → high, `max` → max; `Muse-Glimmer-30B` (always-thinking) — `minimal`/`low` → low, `medium` → medium, `high`/`max` → high. These maps are the gateway's and apply whether or not `supported_parameters` lists `reasoning_effort` for the model. On a model with a reasoning on/off switch but no depth dial (`gemma-4-31b-it`, `gemma-4-26B-A4B-it`, `Qwen3.5-9B`, `Qwen3.6-27B-FP8`, `Qwen3.6-35B-A3B-FP8`, `GLM-4.5-Air-FP8`, `Qwen3-8B-FP8`, `NVIDIA-Nemotron-3.5-Lightning-30B-A3B`) the nearest supported level is the switch itself: `low`, `medium`, `high`, `xhigh` and `max` turn reasoning on at the model's own depth and the response carries `x-flexai-adjusted-params: reasoning_effort=on`, while `none` and `minimal` leave it off (`none` and `xhigh` are accepted on these models only; elsewhere they are a `400`). An explicit `thinking` or `chat_template_kwargs.enable_thinking` switch wins over a contradicting effort, which is then reported in `x-flexai-dropped-params`. On a model with no reasoning control at all (including `Step-3.7-Flash`, whose template does not act on the word) the parameter is removed and the response carries `x-flexai-dropped-params: reasoning_effort`, so a silent drop is never mistaken for a setting that took. Reasoning is **off by default** on every model with an on/off switch; any `reasoning_effort` from `low` up turns it on on the switch-only models above, an effort above `low` turns it on where the model has a dial, and `thinking` (below) turns it on everywhere. Reasoning text is returned in `message.reasoning_content` (`delta.reasoning_content` when streaming) and counted in `usage.completion_tokens_details.reasoning_tokens`. That count is the serving engine's own token count for the reasoning (on streams served by the new gateway it replaces an approximate recount); it is informational, since reasoning tokens are already billed inside `completion_tokens`. | | `thinking` | Yes | `{"type": "enabled"}` or `{"type": "disabled"}` — the reasoning on/off switch on reasoning-capable models: `DeepSeek-V4-Flash-0731` and the hybrid-thinking models (`gemma-4-31b-it`, `gemma-4-26B-A4B-it`, `Qwen3.5-9B`, `Qwen3.6-35B-A3B-FP8`, `GLM-5.2`, `Qwen3.8-27B`, `GLM-4.5-Air-FP8`, `Qwen3-8B-FP8`, `NVIDIA-Nemotron-3.5-Lightning-30B-A3B`, which are off by default; where the model also has a `reasoning_effort` dial — `GLM-5.2`, `Qwen3.8-27B` — an effort above `low` turns it on too; on the others any effort from `low` up turns it on). It takes precedence over `reasoning_effort`: `disabled` turns reasoning off even if `reasoning_effort` is also sent; `enabled` alone turns it on at the model's default depth, and combined with a `reasoning_effort` uses that depth (`low`/`minimal` floor to high). On a model that cannot stop reasoning — any model whose catalog row declares it always reasons (a name containing `Thinking` is the usual hint; the set moves with the fleet, so test it rather than keep a list), and `gpt-oss-120b` / `gpt-oss-20b`, whose `reasoning_effort` dial bottoms out at `low` — `{"type": "disabled"}` is a `400` naming `thinking`; the gpt-oss message points at `reasoning_effort: "low"` as the least reasoning it can do. Omit the parameter to accept the model's default. On a model with no reasoning switch (`Llama-3.3-70B-Instruct-FP8`, `Meta-Llama-3.1-8B-Instruct-FP8`, `Mistral-Nemo-Instruct-2407-FP8`, `Qwen3-Coder-30B-A3B-Instruct-FP8`) a well-formed `thinking` is removed and the response carries `x-flexai-dropped-params: thinking` (`reasoning` when sent in the OpenRouter dialect); a malformed value is a `400` on every model. The raw `chat_template_kwargs.enable_thinking` / `chat_template_kwargs.thinking` switches must be a boolean (or `null`, meaning unset) on every model, or the request is a `400` naming the field. Not yet listed in `supported_parameters`. See [tool use](/inference-api/guides/tool-use#agent-loops-on-deepseek-v4-flash-0731). | ### Multi-turn tool calls When a model turn is a pure tool call, the assistant message has no text to carry. OpenAI-compatible clients disagree on how to serialize that: some omit `content`, some send `content: null`, and some — notably the Vercel AI SDK — send `content: ""`. Because you resend that assistant turn verbatim on the next request so the model can see its own call, whichever shape your client produces is the shape we receive. The gateway accepts all three on an assistant message that carries `tool_calls`, and normalizes an empty string to `null` before the request reaches the model. You don't need a client-side wrapper to rewrite `content` — send the message your tool-calling library produces. That empty-string acceptance is scoped to exactly this case. On any message that is *not* an assistant message carrying `tool_calls`, an empty `content` string is still rejected with a `400` (`param: "messages[N].content"`), since an empty prompt otherwise reaches the model as nothing but chat-template tokens. ## Response fields | Field | Supported | Notes | | - | - | - | | `id`, `created`, `model`, `choices`, `usage` | Yes | — | | `usage.cache_read_input_tokens`, `usage.cache_creation_input_tokens` | Yes | Zero on models without prompt caching; populated for cache-capable models. | | `usage.cost` | Yes (FlexAI extension) | **Our charge** for this request, in USD: what we bill you, not an upstream provider's cost. Equals `cost_details.upstream_inference_prompt_cost + cost_details.upstream_inference_completions_cost`. See [per-request cost](/inference-api/reference/billing#per-request-cost-in-the-response). | | `usage.cost_details.upstream_inference_prompt_cost`, `usage.cost_details.upstream_inference_completions_cost` | Yes (FlexAI extension) | The prompt and completion parts of that same charge. The names follow OpenRouter's usage-accounting fields for compatibility; the values are still our charge to you. | | `system_fingerprint` | No | Not exposed. | ## Tool-result message ordering A normal agent loop ends a turn with tool results and then a `user` message — "here are the results, now summarize." OpenAI accepts a `user` message placed immediately after a `tool` message, and so does this gateway, on every served model. A few models ship a chat template that enforces strict role alternation and would otherwise reject that sequence; for those the gateway normalizes the request for you before it reaches the model, so the behavior matches OpenAI uniformly. You do not need to restructure your history or insert a bridge turn yourself — send the message sequence your tool-calling library produces. ## Authentication & headers * Bearer tokens only (`Authorization: Bearer sk-…`). No OAuth, no organization header. * Our rate limit response includes `Retry-After` and `x-ratelimit-*` headers — see [errors](/inference-api/reference/errors). * CORS is restricted; the API is intended for server-to-server calls, not browsers. ## Error shape Errors use OpenAI's envelope — `{ "error": { "message", "type", … } }` — in two flavors. **Request-validation failures** (`400`) come back verbatim in OpenAI's shape, with `param` set to the offending field path (or `null` when it isn't field-specific) and `code` set to `null` — we reserve non-null `code` strings the way OpenAI does, for a narrow documented set like `context_length_exceeded`: ```json theme={null} { "error": { "message": "…", "type": "invalid_request_error", "param": "messages[0].role", "code": null } } ``` **Auth and quota failures** add a FlexAI-only `doc_url` extension so your first failure points you at the dashboard: ```json theme={null} { "error": { "message": "Invalid API key. Create one from the dashboard.", "type": "authentication_error", "code": "invalid_api_key", "doc_url": "https://platform.flex.ai/dashboard/keys" } } ``` See [the error reference](/inference-api/reference/errors) for the full list of codes. # Regions & Data Residency Source: https://docs.flex.ai/inference-api/reference/regions The US and EU API endpoints, how a workspace is pinned to one, and what that changes about models, keys, and billing. FlexAI serves the Inference API from two regions. Which one you use is decided once, when your workspace is created, and everything else on this page follows from that single choice. | Region | API base URL | What it is | | - | - | - | | `us` (default) | `https://api.flex.ai/v1` | The global endpoint. Served from our worldwide capacity. | | `eu` | `https://eu.api.flex.ai/v1` | The EU endpoint. Your prompts, completions, keys and usage records stay inside the EU. | If you have never been asked about a region, you are in `us` and `https://api.flex.ai/v1` is your endpoint. ## What a region actually is A region is a physical fact about where a workspace runs, not a label on its rows. Each region is a **separate stack** with its own database, gateway and GPU fleet, and a workspace's API keys, request content and usage records live in exactly one of them. The two differ in how far their serving reaches: * **The EU endpoint is strict.** It routes only to GPU capacity inside the EU, so an EU workspace's prompts and completions are processed in the EU. That is enforced at the routing layer — the stack will not list capacity from anywhere else — rather than checked per request. * **The US endpoint is global.** Inference is served from our capacity wherever it sits, which includes EU hardware. It is the right default for performance and model coverage, and it is **not** a commitment that a given request is processed in any particular country. ### What the EU guarantee covers, and what it does not The boundary is a real one, and it is better learned here than in a procurement questionnaire. **Inside the EU stack:** your API keys, the content of your requests and responses, the authoritative records of your usage and spend, and the inference itself. For chat, completions and embeddings, request and response bodies are not retained once the request is served. Video generation is asynchronous, so content exists briefly inside the EU stack while the request is processed: the prompt is cleared when the job finishes, and the generated clip is deleted on its first retrieval. An uncollected clip stops being available after one hour and is physically deleted within about 65 minutes at the declared cleanup cadence. **Held centrally, in the US:** everything in the account layer above the workspace. That layer is a single global control plane rather than something duplicated per region, so treat the rule as the whole layer rather than a list to check against — it covers, among other things, your sign-in identity and sessions, your organization and its membership list, the billing account with its payment methods and ledger, and the records of any identity verification you complete, which includes government-ID checks run by our verification provider. It is also why a single dashboard manages your workspaces in either region, reachable at [platform.flex.ai](https://platform.flex.ai). The EU endpoint carries the API and the dashboard's own calls against it, nothing else; opening `eu.api.flex.ai` in a browser returns 404, which is correct rather than broken. **Operational telemetry is a separate question.** The metrics, traces and product analytics we use to run and monitor the service are not part of the regional boundary described above, and some of them carry usage-derived figures. They do not carry your prompts or completions. If your requirement covers that category too, raise it with us explicitly rather than reading it off this page. So the EU guarantee is about where your **inference and its data** are processed and stored, and that is the part most residency requirements are written about. If your requirement extends to account and billing metadata as well, talk to us at [support@flex.ai](mailto:support@flex.ai) before you build — do not infer it from this page. ## Choosing a region You pick the region when you create a workspace, and **the choice is permanent**. There is no migration path and no setting that changes it afterwards — moving a workspace between regions would mean moving the data the region exists to contain, which is the one thing the design rules out. An organization can hold **one workspace per region**, so if you need both, create a second workspace rather than trying to move the first. The two are independent: separate keys, separate balances, separate usage. If a workspace ended up in the wrong region, it cannot be moved — not from the dashboard and not by us. Contact [support@flex.ai](mailto:support@flex.ai) and we will help you set up a new workspace in the right one. ## Keys are region-scoped An API key belongs to the workspace that minted it, and therefore to that workspace's region. The two stacks hold separate key stores, so an EU key does not exist as far as `https://api.flex.ai/v1` is concerned, and a US key does not exist to `https://eu.api.flex.ai/v1`. That means **a key presented to the wrong host fails authentication** — `401`, the same answer a revoked or mistyped key gets — rather than returning anything that mentions regions. If a key that works in one place is suddenly unauthorized in another, check the host before you suspect the key. The fix is always to call the host for your own region, never to re-mint the key. (There is a separate `403` carrying the code `region_not_served`, which the [errors reference](/inference-api/reference/errors#region-not-served) documents. You are very unlikely to meet it: it guards against a stack somehow holding a workspace from a region it does not serve, which is an operational fault on our side rather than something a wrong base URL produces.) ## What each region serves The EU endpoint serves less than the US one, for two independent reasons: * **Capacity.** The EU endpoint routes only to EU hardware, so a model with no EU capacity behind it is not served there. The reverse is possible too — the US endpoint can only route to another region's hardware when the connection to it resolves — so treat the two catalogs as *different*, not as one containing the other. * **Licensing.** Some models carry license or provenance terms we are not prepared to serve into the EU, and those are withheld there regardless of capacity. Neither of these is a list you can read ahead of time, and both move as the fleet and the legal picture change. So treat `GET /v1/models` **against the host you are calling** as the authoritative answer, exactly as the [model discovery guide](/inference-api/guides/model-discovery) describes: ```bash theme={null} curl https://eu.api.flex.ai/v1/models \ -H "Authorization: Bearer $FLEXAI_API_KEY" ``` A model your region does not serve is absent from that list, and calling it anyway returns a `404` — but which `404` depends on why, and the difference is worth knowing when you are debugging: * A model with **no capacity** in your region answers `404` with the code `model_deactivated`. The id is one we recognize; there is nothing behind it here right now. * A model **not cleared for your region** answers the ordinary unknown-model `404` instead, with no separate code saying so. Usually that means the licensing review for it has not been completed yet rather than that a problem was found. We do not publish a per-region list, so treat absence from `/v1/models` as the answer. If you are porting code from a US workspace, re-check your model ids against the EU list rather than assuming they carry over. ## Billing currency A workspace's billing follows its region. US accounts are charged and invoiced in **US dollars**; EU accounts are charged and invoiced in **euros**, with EU VAT handled on the invoice. Underneath that, prices stay in dollars everywhere. The model catalog, your usage figures and your wallet balance are all USD, and a euro top-up is converted to dollars exactly once — at the moment the card is charged, at the European Central Bank reference rate fixed for that calendar month, recorded against the payment. Nothing is re-converted afterwards, so the dollar value of credit you have already bought never moves with the exchange rate. The practical upshot: you type euros, Stripe charges euros, your invoice reads euros, and everything you see about *usage* reads dollars. Top-up minimums and presets are set natively per currency rather than converted, so the euro figures are round numbers too. See [Billing & Quotas](/inference-api/reference/billing) for how metering, per-key rate limits and the `402` response work. Those behave the same in both regions; the edge's own flood limit and some billing mechanics differ, so read a limit off the response headers rather than assuming it carries across. # Usage Policy Source: https://docs.flex.ai/inference-api/reference/usage-policy Fair-use expectations, the abuse patterns we watch for, and what happens when a key trips them. The Inference API is a shared, multi-tenant platform. The capacity behind it — GPUs running real models — is finite, so a single key that consumes a disproportionate share degrades latency and availability for everyone else. This page describes what we consider fair use, the patterns we monitor for, and what we do when a key crosses a line. The goal is transparency: you should know the triggers before you hit them. ## Fair use Normal application traffic is always welcome — interactive chat, batch jobs, evaluation runs, production workloads. You do not need to ask permission to ramp up legitimate usage. What we ask is simply that your traffic reflect real work rather than patterns that exist only to consume capacity. Each API key carries explicit **rate limits** (requests per minute and tokens per minute) tied to your tier. Those limits are the hard ceiling and are enforced inline — requests over the limit receive `429 Too Many Requests`. The policy described below sits *above* those limits: it is about sustained patterns that, while they may stay under the per-request limits, still indicate abuse or a runaway client. ## What we monitor We continuously review per-key activity for a small set of patterns. On day one these are **observed and reviewed, not automatically blocked** — a flagged key triggers a human review, not an instant cutoff. The patterns are: * **Sustained excessive request rate.** A single key sending requests far above what its tier and normal application behavior would produce, held over a sustained window. * **Sustained excessive token consumption.** A single key generating tokens at a rate well beyond normal usage, sustained over time — the clearest signal of one key burning a disproportionate share of GPU budget. * **Runaway or hung requests.** A key accumulating many very long-running requests. This usually points at a misconfigured client (for example, an unbounded `max_tokens` on a non-streaming call that runs until it is cut off) rather than deliberate abuse, but the effect on shared capacity is the same. For long-running non-streaming requests specifically, the most reliable fix is on your side: set a sensible `max_tokens`, or stream the response. Streaming keeps the connection alive token-by-token and avoids the timeouts that long non-streamed generations hit. ## What happens when a key is flagged When a key trips one of these patterns, the typical sequence is: 1. **Review.** We look at the account and the traffic to distinguish a legitimate ramp (a batch job, a load test, real growth) from abuse or a runaway client. A legitimate ramp needs no action from you — if anything, it is a signal to talk about a higher tier. 2. **Contact.** If the pattern looks like abuse or a misconfigured client that is degrading the service for others, we reach out. 3. **Throttle or disable.** If a key is actively degrading availability for other customers and the situation cannot wait, we may temporarily disable it. A disabled key receives `402 Payment Required` on its next call; in-flight work is allowed to finish, but no new requests are accepted until the key is re-enabled. We re-enable as soon as the issue is resolved. If your application has a legitimate need for sustained high throughput, get in touch *before* you ramp — we would much rather provision for you than flag you. You can manage your keys and see your usage from the [dashboard](https://platform.flex.ai/dashboard), and reach support from the same place. # Overview Source: https://docs.flex.ai/interactive-development Develop and debug your AI workloads interactively with VSCode and SSH access FlexAI's Interactive Development features are designed to optimize your development workflows and enhance productivity. You can leverage these tools to streamline your coding-testing-debugging cycles, all within the FlexAI ecosystem. Connect directly to your Interactive Development environment from Visual Studio Code to leverage all of its supported features, like language servers, integrated terminal for system-wide command execution, debugging tools, and a large set of extensions. Access your an Interactive Development environment via SSH from any terminal or tool that can leverage it, giving you complete control over your development environment. You can use the Interactive Training mode to create a dynamic FlexAI Training runtime environment that you can *interact with* directly via SSH from your terminal or through a code editor —in this case, [Visual Studio Code (VSCode)](https://code.visualstudio.com/). ## Your AI development workflow within the FlexAI ecosystem The most straightforward way to integrate FlexAI into your development workflow is to use the Interactive Training mode to create a dynamic training environment that you can connect to from VSCode. Within the Interactive Training runtime environment you will have access to its file system —including your code base, Datasets, generated outputs and Checkpoints—, as well as the ability to run system commands to query logs and monitor performance. Starting an Interactive Training session using the `flexai training debug-ssh` command with the required `--repository-url` and `--authorized-keys` flags, as well as the optional, but highly recommended `--vscode` flag. ```bash title="Starting an Interactive Training session with VSCode support" theme={null} flexai training debug-ssh \ --repository-url https://github.com/flexaihq/nanoGPT \ --authorized-keys ~/.ssh/id_ed25519.pub \ --vscode \ ``` Once you're happy with the results after testing and iterating on your code, you can save your work by pushing changes to the source Git repository. This can be done because the CLI will attempt to get your Git Author name and Email from your local Git configuration and set them in the Interactive Training Runtime environment. ### Integrating more FlexAI services into your development workflow You can go deeper by integrating more FlexAI services, here's an example workflow: 1. You can deploy an Inference Endpoint of a coding-assistant LLM, such as [Qwen3-Coder](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct) or [Qwen2.5-Coder](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct), either as a base model, or a [Fine-tuned one with your own data](/core-services/fine-tuning/quickstart/creating/). 2. Start an Interactive Training session making sure to use the `--vscode` flag. 3. Once the Interactive Training session is running and you have connected to it from VSCode, you can install an extension such as [CodeGPT](https://marketplace.visualstudio.com/items?itemName=DanielSanMedium.codegpt) or [Continue](https://marketplace.visualstudio.com/items?itemName=Continue.continue) and configure it to use your deployed Inference Endpoint and API Key. ### Want to go deeper? How about testing your changes by deploying an Inference Endpoint from an "Inference-ready" Checkpoint you created during your Interactive Training session? You can learn how to do this and more in the [Inference-ready Checkpoints](/platform-services/checkpoint-manager/inference-ready-checkpoints/) documentation page. ## Prefer not to use Visual Studio Code? If you prefer not to use Visual Studio Code, you can still connect to the Interactive Training Runtime using any SSH client. Simply skip the `--vscode` flag when starting the Interactive Training session, then use the provided SSH command to connect to the Interactive Training Runtime from your terminal or SSH client of choice. # Interactive Training Session Source: https://docs.flex.ai/interactive-development/interactive-training Start an interactive SSH session for real-time training development and debugging The `flexai training debug-ssh` command allows you to start an Interactive Training Job on a Training Runtime similar to that of regular Training Jobs. This command will allocate the required resources and then set up an Interactive Training Runtime you can connect to via SSH. Having access to this *debug* Interactive Training Runtime will allow you to iterate quickly by enabling you to edit scripts, modify files, review logs and outputs, push your changes to your GitHub repository, and in general, have an inside out look at your Training results as they happen, before you commit to running a Training Job. ## Pre-requisites The environment in which the `flexai training debug-ssh` command is run should either: * Have an `ssh-agent` running and SSH keys loaded into it, or * Have an SSH key pair available ### Using an ssh-agent If an `ssh-agent` is not running and you would like to use one to be able to push your change from the Interactive Training Runtime to GitHub, run `eval $(ssh-agent)` in your terminal, and load your keys to it by running `ssh-add `. You can then confirm the keys have been loaded using `ssh-add -l` and start your Interactive Training Job - it will automatically use your `ssh-agent`. If SSH key pairs authenticating you to GitHub are loaded into your `ssh-agent`, you will be able to push your changes to GitHub from the Interactive Training Runtime by enabling the `ForwardAgent` option in your SSH configuration file (`~/.ssh/config`): ```text title="~/.ssh/config" theme={null} Host debug-gw.flex.ai ForwardAgent yes ``` To verify if an `ssh-agent` is running in your local environment and has your private keys loaded, run: ```bash theme={null} ssh-add -L ``` You can load more keys into your `ssh-agent` with: ```bash theme={null} ssh-add path/to/private/key ``` `ssh-add` relies on the environment variable `SSH_AUTH_SOCK` being correctly configured. If you are overriding it through the `IdentityAgent` ssh config option in `.ssh/config`, make sure to set the `SSH_AUTH_SOCK` environment variable to the same socket path when running the aforementioned `ssh-add` commands. Failing to do, `ssh-add` will load keys into the wrong `ssh-agent`. ### Using SSH key pairs When not using the `ssh-agent`, you *must* set the path to the public key you will use through the `--authorized-keys` flag, e.g.: ```bash theme={null} flexai training debug-ssh \ --repository-url https://github.com/flexaihq/nanoGPT \ --authorized-keys ~/.ssh/id_ed25519.pub \ --vscode ``` If the SSH key pair is located in a non-standard path (i.e. not in `~/.ssh/` with names like `id_rsa`, `id_ecdsa`, `id_ed25519`, etc.), you will need to pass an extra `-i ` flag to the `ssh` command when connecting to the training environment. ## Starting an Interactive Training Job In its simplest form, starting an Interactive Training Job will only require running the `debug-ssh` subcommand with the `--repository-url` flag, e.g.: ```bash title="Starting an Interactive Training Job" theme={null} flexai training debug-ssh \ --repository-url https://github.com/flexaihq/nanoGPT \ --vscode ``` Information on the various stages of the Interactive Training Job will be displayed in the terminal. Once the job is ready, you will be provided with the SSH command to connect to the Interactive Training Runtime, example output: ```text title="Output: Starting an Interactive Training Job" theme={null} flexai training debug-ssh \ --repository-url https://github.com/flexaihq/nanoGPT \ --vscode Interactive training interactive-training-61a01afb-66bc-411c-a23b-d9fc566c1f9c launching... ✅ Looking for an interactive training builder ✅ Starting interactive training environment [Node 0] To connect using ssh: ssh -o ForwardAgent=yes -p 36663 flexai@debug-gw.flex.ai [Node 0] To open VSCode, click the following URL (or open in your browser if not clickable): "vscode://vscode-remote/ssh-remote+flexai@debug-gw.flex.ai:36663/workspace?windowId=_blank" ✅ Automatically configuring ~/.ssh/known_hosts ``` If the --`vscode` flag was used, Visual Studio Code will automatically open. ## Attaching through Visual Studio Code You can also attach to the Interactive Training Runtime using VSCode. To do so, you will need to install the [Remote - SSH](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh) extension. Once installed, you can connect to the Interactive Training Runtime by simply clicking on the VSCode URL provided in the terminal output. This will open a new VSCode window with the SSH connection to the Interactive Training Runtime already established. The `--vscode` flag can be used to automatically open a VSCode window into the Interactive Training Runtime, useful if the VSCode URL is not clickable in your terminal. ## Lifetime of an Interactive Training Job ### Session timeout Interactive Training Jobs are automatically stopped after the specified session timeout (can be set with `--session-timeout`. Defaults to 600 seconds, which is 10 minutes) if there are no active SSH session into the Interactive Training Runtime. ### Stopping a running Interactive Training Job Interactive Training Jobs can be manually stopped like regular Training Jobs by using the `flexai training stop` command: ```bash theme={null} flexai training stop ``` ## Troubleshooting ### The Interactive Training Job fails to start You can check the Interactive Training session logs, just like a regular Training Job: ```bash theme={null} flexai training logs ``` The same applies to *inspecting* a Training Session to get more information on its [Lifecycle](/cli/reference/training/#training-lifecycle-statuses): ```bash theme={null} flexai training inspect ``` ### SSH connection issues #### When using an `ssh-agent` Make sure that the `ssh-agent` is running and has the correct keys loaded. You can verify this by running `ssh-add -L` and checking that the keys you expect are listed. A common issue is that the `ssh-agent` in your environment is not the same as the one used by the Interactive Training Job. This can happen if you have multiple `ssh-agent` instances running and the `SSH_AUTH_SOCK` environment variable points to a different agent than the one configured inside your ssh configuration file (`~/.ssh/config`). When loading keys into your agent with `ssh-add /path/to/your-key` or verifying which keys are loaded with `ssh-add -L`, make sure that the `SSH_AUTH_SOCK` environment variable is set to the same socket path as the one used by the Interactive Training Job, which you can find by running: ```bash theme={null} ssh -G debug-gw.flex.ai | awk '/^identityagent/ {print $2}' ``` ### Visual Studio Code connection issues If Visual Studio Code fails to open automatically: * Make sure you have the [Remote - SSH](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh) extension installed. * Copy the `vscode://` URL provided in the terminal output and: * In a Visual Studio instance open the command palette (You can use `Ctrl+Shift+P` / `Cmd+Shift+P`), then paste the `vscode://` URL. * Open a terminal window, then use the `open` or `xdg-open` command (depends on your system) followed by the `vscode://` URL you just copied: ```bash theme={null} open vscode://vscode-remote/ssh-remote+flexai@debug-gw.flex.ai:36663/workspace?windowId=_blank ``` ```bash theme={null} xdg-open vscode://vscode-remote/ssh-remote+flexai@debug-gw.flex.ai:36663/workspace?windowId=_blank ``` Note that in order to establish the connection, you may be prompted to unlock your SSH key if it is passphrase-protected. # Platform services Source: https://docs.flex.ai/platform FlexAI platform services and tools FlexAI Platform Services provide the essential building blocks for managing data, code, secrets, and storage across your AI workflows. Each service is designed to be secure, flexible, and deeply integrated with the FlexAI ecosystem. ## Key Platform Services Manage, version, and deploy model checkpoints from Training and Fine-tuning jobs Upload, manage, and organize datasets for your AI workloads Securely store and manage API keys, credentials, and secrets Connect to cloud storage providers for seamless data access and transfer Connect and manage GitHub repositories for training and inference Monitor, log, and analyze your FlexAI workloads and infrastructure ### [Checkpoint Manager](/platform-services/checkpoint-manager/) Capture, store, version, and deploy model checkpoints. Resume training, export checkpoints, and deploy models with confidence. ### [Dataset Manager](/platform-services/dataset-manager/) Upload datasets from local or cloud sources, organize data flexibly, and manage multiple datasets for complex training scenarios. ### [Secret Manager](/platform-services/secret-manager/) Secure, encrypted storage for API keys, tokens, and configuration files. Reference secrets in workloads without exposing sensitive data. ### [Remote Storage Connection Manager](/platform-services/remote-storage-connections-manager/) Integrate with AWS S3, Google Cloud Storage, and more. Move data directly between providers and FlexAI, and manage credentials securely. ### [Code Registry Manager](/platform-services/code-registry-manager/) Connect public or private GitHub repositories, reference specific commits or branches, and streamline your development-to-deployment workflow. ### [Observability Services](/platform-services/observability/) Monitor infrastructure, visualize metrics, and access logs for all your FlexAI workloads. # Checkpoint Manager Source: https://docs.flex.ai/platform-services/checkpoint-manager Manage, version, and deploy model checkpoints from Training and Fine-tuning jobs The FlexAI Checkpoint Manager automatically captures, stores, parses, and versions the Checkpoints your Training and Fine-tuning Jobs generate. You can use the FlexAI Checkpoint Manager to manually push existing Checkpoints to FlexAI, to fetch Checkpoints to a local machine, or export them to external supported Cloud Storage Providers. ## Checkpoints FlexAI Managed Checkpoints are a powerful feature that automatically saves and manages checkpoints during a Training Job's execution. This feature has been designed to simplify the process of saving and retrieving the state of a model during training, allowing you to easily resume training from a specific point in time to evaluate the model's performance, go back in time to re-start a Training Job from a Checkpoint before an interruption, or even roll back to a previous state if needed. ## Key Features Automatic checkpoint capture and secure storage without manual intervention required Access checkpoints from running Training and Fine-tuning Jobs for real-time evaluation and analysis Export checkpoints to supported Cloud Storage Providers or download them locally Resume Training and Fine-tuning Jobs from a checkpoint with full state preservation ## Training Integration ### Resume Training Continue training from any checkpoint with full state preservation: ```bash theme={null} flexai training run continued-training \ --checkpoint 3784735b-d7c6-4978-bd76-c6e9158d2ccc \ --dataset fineweb \ --repository-url https://github.com/flexaihq/flexbasemodel_t2t \ -- train.py --continue_from=3784735b-d7c6-4978-bd76-c6e9158d2ccc --checkpoint_path=/input-checkpoint/ ``` > The `3784735b-d7c6-4978-bd76-c6e9158d2ccc` UUID corresponds to the unique identifier the FlexAI Checkpoint Manager assigned to the Checkpoint when it was created. You can also use a Checkpoint name if you manually pushed it. ### Fine-Tuning Workflows Use checkpoints as the starting point of a Fine-tuning Job: ```bash theme={null} flexai training run fine-tune-from-checkpoint \ --checkpoint mistral_base_checkpoint \ --dataset fineweb \ --repository-url https://github.com/flexaihq/mistral-ft \ -- fine-tune.py --checkpoint_name=mistral_base_checkpoint --checkpoint_path=/input-checkpoint/ ``` > The `mistral_base_checkpoint` name corresponds to the name manually assigned to the Checkpoint when it was pushed it to FlexAI. You can also use a Checkpoint UUID from a Checkpoint captured by the FlexAI Checkpoint Manager. As you can see, the workflow is exactly the same in both cases. The only difference lies in the script you pass as entry point. This script will be responsible for loading the Checkpoint from the `/input-checkpoint/` path and either continuing a previous training Job or initiating a model Fine-tuning task. # FlexAI Checkpoints: In Practice Source: https://docs.flex.ai/platform-services/checkpoint-manager/in-practice Practical guide to managing checkpoints in training and fine-tuning workflows FlexAI uses a **directory-based** approach for managing Checkpoints, which simplifies the process of saving and loading checkpoint files across a large set of libraries. ## Environment Variables Environment variables are exposed in the Runtime environments to help you identify where to save and load checkpoints from. | Variable | Path | Description | | - | - | - | | `FLEXAI_INPUT_CHECKPOINT_DIR` | `/input-checkpoint/` | Directory where the selected Checkpoint is mounted | | `FLEXAI_OUTPUT_CHECKPOINT_DIR` | `/output-checkpoint/` | Directory where your code should write Checkpoints | ## Checkpoint Creation & Management The FlexAI Checkpoint Manager expects your code to write checkpoints during execution to the output directory the `FLEXAI_OUTPUT_CHECKPOINT_DIR` environment variable points to. Once there, the FlexAI Checkpoint Manager take care of them: ```text theme={null} /output-checkpoint/ ├── / │ └── ... └── / └── ... ``` Each sub-directory inside `/output-checkpoint/` is treated as a **complete** and **versioned checkpoint**. This folder-based approach ensures compatibility with a wide range of libraries, especially in the Hugging Face ecosystem. It makes it easy to resume training or launch inference jobs. ### Supported Libraries FlexAI automatically detects and tracks checkpoints emitted through: * Hugging Face [`safetensors`](https://huggingface.co/docs/safetensors/en/index) * Hugging Face [`accelerate`](https://huggingface.co/docs/accelerate/en/usage_guides/checkpoint) * Hugging Face [`transformers`](https://huggingface.co/docs/transformers/en/index) (e.g., [`Trainer.save_model`](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.Trainer.save_model)) * Any other libraries built on top of the Hugging Face `transformers` ecosystem, such as: * [`PEFT`](https://huggingface.co/docs/peft/en/developer_guides/checkpoint) * [`TRL`](https://huggingface.co/docs/trl/main/en/index) * [`Diffusers`](https://huggingface.co/docs/diffusers/en/index) As long as your code writes to a sub-directory of `/output-checkpoint/`, the FlexAI Checkpoint Manager will handle the rest. #### Using Hugging Face Transformers If you're saving checkpoints at different steps, you should create a new sub-directory for each one: ```python title="save_checkpoint_at_different_steps.py" theme={null} import os from pathlib import Path from transformers import Trainer, TrainingArguments checkpoint_dir = os.environ["FLEXAI_OUTPUT_CHECKPOINT_DIR"] training_args = TrainingArguments( output_dir=checkpoint_dir, # highlight-line ) trainer = Trainer( model=custom_model, args=training_args, ) trainer.train() trainer.save_model() ``` The Transformers library's `Trainer` class is designed so you only need to set the `TrainingArguments` `output_dir` argument to `/output-checkpoint/`. This way, the main checkpoint will be saved to the root of `/output-checkpoint/`, and subsequent checkpoints will be saved as individual sub-directories. ##### File Structure ```text theme={null} /output-checkpoint/ ├── config.json ├── pytorch_model.bin ├── tokenizer.json └── step-2000/ ├── config.json ├── pytorch_model.bin └── tokenizer.json ``` > Note that it is not uncommon for a Transformers-based framework to write a *special* Checkpoint in the root of the destination path, this being either what it considers the "latest" or "best" Checkpoint. *** #### Using PyTorch's `torch.save` function The FlexAI Checkpoint Manager also supports two common patterns when using PyTorch's `torch.save` function to save model checkpoints: The **legacy** flat file mode and the **recommended** directory-based mode. ##### Directory-based mode (recommended) Create a sub-directory inside `/output-checkpoint` and save your files there. Everything inside the sub-directory is grouped and tracked as a single checkpoint: ```python title="torch_save__directory_based.py" theme={null} import os from pathlib import Path import torch output_dir = Path(os.environ["FLEXAI_OUTPUT_CHECKPOINT_DIR"]) ckpt_path = output_dir / "step-1000" ckpt_path.mkdir(parents=True, exist_ok=True) torch.save(model.state_dict(), ckpt_path / "model.pt") ``` ##### Flat file mode (legacy) You can also write files directly into `/output-checkpoint`. In this case, each file is treated as its own checkpoint. This works, but may lead to ambiguity if other folder-based checkpoints are present, so it is discouraged. ```python title="torch_save__flat_file.py" theme={null} import os from pathlib import Path import torch output_dir = Path(os.environ["FLEXAI_OUTPUT_CHECKPOINT_DIR"]) ckpt_path = output_dir / "step-1000" torch.save(model.state_dict(), ckpt_path) ``` *** ## Checkpoint Loading ### Input Checkpoint While building the Training or Fine-tuning workload runtime environments, FlexAI mounts the **Checkpoint** specified during the resource creation process under the path specified by the `FLEXAI_INPUT_CHECKPOINT_DIR` environment variable: *** In the [Start a new Fine-tuning Job](https://platform.flex.ai/s/fine-tunings/new-fine-tuning) form of the FlexAI Console, you can select a Checkpoint from the **Checkpoint** menu. When creating a Fine-tuning workload using the FlexAI CLI, you can specify a Checkpoint by passing its UUID or name to the `-C/--checkpoint` flag: ```bash theme={null} flexai training run \ --checkpoint \ ... ``` Example: ```bash title="Creating a Fine-tuning Job with a Checkpoint" theme={null} flexai training run ft_mistral7b_01 \ --dataset FreedomIntelligence__medical-o1-reasoning-SFT=medical \ --repository-url https://github.com/funnierinspanish/Mistral-7b_01-med-ft \ --repository-revision new_med_settings \ --checkpoint=mistral7b_01_base \ --secret HF_TOKEN=hf_token \ --accels 4 \ -- fine-tune.py mistral_config --dataset=medical ``` Assuming you picked a Checkpoint named `mistral7b_01_base`, the directory structure would look like this: ```text theme={null} /input-checkpoint/ └── mistral7b_01_base/ ├── config.json ├── pytorch_model.bin └── tokenizer.json ``` ### Loading a Checkpoints Your code should load the Checkpoint directly from `FLEXAI_INPUT_CHECKPOINT_DIR/`: ```python title="load_checkpoint.py" theme={null} import os from transformers import AutoModelForCausalLM ckpt_input = os.environ["FLEXAI_INPUT_CHECKPOINT_DIR"] ckpt_input_dir = os.path.join(ckpt_input, "mistral7b_01_base") model = AutoModelForCausalLM.from_pretrained(ckpt_input_dir) ``` Now you can continue your Training Job from the loaded Checkpoint or Fine-tune it further. # Inference-ready Checkpoints Source: https://docs.flex.ai/platform-services/checkpoint-manager/inference-ready-checkpoints Mark and manage checkpoints ready for inference deployment FlexAI Checkpoints can be used not only to resume Training and Fine-tuning jobs but also to deploy Inference Endpoints. When a Checkpoint is marked as **Inference Ready**, it means that the Checkpoint contains all the necessary files and metadata required to deploy an Inference Endpoint directly from it. ## Checkpoint Metadata Extraction Process Currently, the FlexAI Checkpoint Manager automatically extracts metadata from Checkpoints created using the Hugging Face Transformers library. This metadata is used to determine if a Checkpoint contains the necessary information to be marked as **Inference Ready** and be deployed as an Inference Endpoint: ### Hugging Face Transformers Checkpoints Currently, the FlexAI runtime supports Hugging Face Transformers checkpoints, which include the `trainer_state.json` and `config.json` files that contain metadata about the training process and model configuration: * `STEP`, `TRAIN LOSS` & `EVAL LOSS`: Extracted from `trainer_state.json`'s `log_history` field (last entry). * `MODEL`: Determined from `config.json`'s `architectures` field. * `VERSION`: Retrieved from `config.json`'s `transformers_version` field. * `INFERENCE READY`: Set to `true` if the `architectures` field is present in `config.json`. ## Deploying an Inference-ready Checkpoint To deploy an Inference Endpoint from an Inference-ready Checkpoint, follow these steps: Available soon. For now, please use the FlexAI CLI. 1. List the Checkpoints associated with a Training or Fine-tuning job. ```bash title="Listing Checkpoints for a Training Job" theme={null} flexai training checkpoints ``` Which will return an output similar to: ```text title="List of Checkpoints for a Training Job" {5} theme={null} ID │ NAME │ NODE │ STEP │ TRAIN LOSS │ EVAL LOSS │ MODEL │ VERSION │ INFERENCE READY │ TIMESTAMP ─────────────────────────────────────┼───────────────────┼──────┼──────┼────────────┼───────────┼────────────────────┼─────────┼─────────────────┼────────────────────────── a494f07f-e183-4a53-a6e6-e7116ca177fd │ checkpoint-250 │ 0 │ 250 │ 0.8438 │ │ │ │ false │ 2025-09-25 02:39:42 (1d) 3784735b-d7b6-4978-bd76-c6e9158d2ecc │ checkpoint-300 │ 0 │ 300 │ 0.7895 │ │ │ │ false │ 2025-09-25 02:41:02 (1d) 7f7fe96a-c649-4c94-bfc7-218e17d392ba │ hf_checkpoint │ 0 │ 300 │ 0.7895 │ │ MistralForCausalLM │ 4.44.2 │ true │ 2025-09-25 02:41:02 (1d) ``` 2. Create an Inference Endpoint using the `flexai inference serve` command, specifying the Checkpoint's name or UUID with the `--checkpoint` flag. Example: ```bash title="Creating an Inference Endpoint from a Checkpoint" {2} theme={null} flexai inference serve \ --checkpoint \ [ ...] \ -- [ ...] ``` Which can look like: ```bash title="Creating an Inference Endpoint from a Checkpoint" {2} theme={null} flexai inference serve mistral7b-inference \ --checkpoint 7f7fe96a-c649-4c94-bfc7-218e17d392ba \ --model-type mistral \ --accels 2 ``` Check out the FlexAI Inference Endpoints documentation for more details. # Code Registry Manager Source: https://docs.flex.ai/platform-services/code-registry-manager Connect and manage GitHub repositories for training and inference workloads The FlexAI Code Registry Manager connects your organization's GitHub repositories directly to the FlexAI platform, so you can use your own code in Training and Fine-tuning Jobs without extra setup. ## Key Features Native integration with GitHub repositories through the FlexAI GitHub App Use private repositories in Training and Fine-tuning Jobs Define whether to provide access to the org or specific repositories ## Installing the FlexAI GitHub App To install the FlexAI GitHub App, follow these steps: 1. Visit the FlexAI GitHub App installation page: * [https://github.com/apps/flexai-cloud-services/installations/new](https://github.com/apps/flexai-cloud-services/installations/new) 2. Log into your GitHub account if you are not already logged in 3. Select the account or organization where you want to install the FlexAI GitHub App 4. Choose whether to install the app for all repositories or select specific repositories 5. Select the "Install & Authorize" button to complete the installation ## Configuring the FlexAI GitHub App If you ever want to change the repositories that the FlexAI GitHub App has access to, you can do so by following these steps: 1. Visit your GitHub account settings' Installations page: * [https://github.com/settings/installations](https://github.com/settings/installations) 2. Locate the "FlexAI Cloud Services" app from the list of installed applications and select the "Configure" button 3. In the "Repository access" section, update the repository access settings as needed, either by selecting specific repositories or picking the "All repositories" option 4. Save your changes to update the app's access permissions ## Suspending or Uninstalling the FlexAI GitHub App If you ever want to uninstall the FlexAI GitHub App, you can do so by following the same steps as above, but instead of selecting the repositories on the "Repository Access" section, you should scroll down to the "Danger Zone" section and select either the "Suspend" or "Uninstall" button. The full steps are: 1. Visit your GitHub account settings' Installations page: * [https://github.com/settings/installations](https://github.com/settings/installations) 2. Locate the "FlexAI Cloud Services" app from the list of installed applications and select the "Configure" button 3. In the "Danger Zone" section, select either the "Suspend" or "Uninstall" button 4. Confirm your choice in the dialog that appears to complete the process # Dataset Manager Source: https://docs.flex.ai/platform-services/dataset-manager Upload, manage, and organize Datasets for Training and Fine-tuning your AI models The FlexAI Dataset Manager provides a comprehensive solution for uploading, organizing, and managing Datasets used in your AI training and Fine-tuning workloads. Whether your data is stored locally or in the cloud, the Dataset Manager streamlines the process of making it available to your AI Workloads. ## Overview The FlexAI Dataset Manager enables you to: * **Upload Datasets** from a local machine or Cloud Storage providers * **Organize data** with flexible directory structures * **Manage multiple Datasets** for complex training scenarios ## Key Features Upload from a local machine, AWS S3, Google Cloud Storage, the Hugging Face Hub, and more Support for flat or hierarchical directory structures with custom file organization Datasets are immutable once created, ensuring reproducible training results Attach multiple Datasets to a single Training or Fine-tuning Job for complex scenarios. ## Dataset Sources ### Local Machine Upload Datasets directly from your local filesystem: * **Individual files** with custom destination paths * **Directory contents** preserving nested structure * **Multiple files** with batch upload capabilities * **Flexible mapping** between source and destination paths ### Remote Storage Providers Seamlessly upload Datasets from cloud storage without local downloads: * **Amazon S3**: Direct integration with S3 buckets and object keys * **Google Cloud Storage**: Native support for GCS buckets * **Cloudflare R2**: Compatible with R2 storage endpoints * **MinIO**: Self-hosted S3-compatible storage integration * **Hugging Face Hub**: Direct Dataset downloads from public and private repositories Visit the [Remote Storage Connection Manager](/platform-services/remote-storage-connections-manager) section for detailed instructions on setting up and using remote storage connections. ## Dataset Organization ### File Structure FlexAI supports completely flexible Dataset organization. #### A single Dataset with a flat structure Here's an example of a Dataset with a flat structure. It was named `my_flat_dataset` at the time of its attachment to a Training or Fine-tuning Job: ```text theme={null} /input/ # The contents of /input/ are read-only └── my_flat_dataset/ └── data.bin ``` #### A single Dataset with a hierarchical structure Here's an example of a Dataset with a large set of nested directories and files inside. It was named `my-dataset` at the time of its attachment to a Training or Fine-tuning Job: ```text theme={null} /input/ # The contents of /input/ are read-only └── my-dataset/ ├── train/ │ ├── images/ │ │ ├── image001.jpg │ │ ├── image002.jpg │ │ └── ... │ └── labels.csv ├── validation/ │ └── ... └── test/ └── ... ``` #### Multiple Datasets This example shows how multiple Datasets are organized when attached to a single Training or Fine-tuning Job. Each Dataset gets its own sub-directory under `/input/`. Here we have three Datasets attached to the same Job: * `fineweb` * `ylecun_mnist` * `OpenAssistant-oasst1` ```text theme={null} /input/ # The contents of /input/ are read-only ├── fineweb/ │ └── ... ├── ylecun_mnist/ │ └── ... └── OpenAssistant-oasst1/ └── ... ``` ## Runtime Access Datasets are mounted as **read-only resources** in your Runtime Environment: * **Location**: `/input//` * **Multiple Datasets**: Each Dataset gets its own sub-directory * **Preserved structure**: The original file structure at creation time is maintained * **Direct access**: Your code can access files immediately ## File Format Support The Dataset Manager supports **any file format** your training code can process: * **Images**: JPEG, PNG, TIFF, and other image formats * **Text**: Plain text, JSON, CSV, Parquet files * **Audio**: WAV, MP3, FLAC audio files * **Video**: MP4, AVI, MOV video formats * **Archives**: TAR, ZIP compressed archives * **Binary**: Model files, custom binary formats * **Encrypted**: Password-protected or encrypted files * **Other**: Any other file formats not listed ## Performance Optimization ### Batch Operations Optimize upload performance with efficient batch operations: * **Multiple file flags** - Upload many files in a single command * **Directory uploads** - Transfer entire directories efficiently * **Remote transfers** - Server-to-server transfers for cloud data * **Parallel processing** - Concurrent upload streams for faster transfers ### Storage Efficiency Maximize storage efficiency and minimize costs: * **Immutable Datasets** - Reuse existing Datasets across multiple Training or Fine-tuning Jobs * **Deduplication** - Whenever possible, try to avoid uploading duplicate files * **Versioning** - Create new Datasets for updated data while retaining previous versions * **Dataset management** - Archive or delete unused Datasets Notes on Compression: Even though you can upload compressed files (e.g., ZIP, TAR), you need to consider the fact that your training code will need to decompress them at runtime. This process can be time-consuming and impact the overall performance of your training job by leaving your compute resources idle while waiting for the decompression to complete. Additionally, the scratch storage available on your compute instances may not be sufficient to hold both the compressed and decompressed files, leading to potential *insufficient storage* issues. ## Getting Started Ready to start managing your Datasets? Explore these resources: Comprehensive guide to uploading Datasets from your local host Learn how to upload data from a Cloud Storage provider Learn how the Remote Storage Connection Manager works and how to set a connection up # Uploading Datasets from your local machine Source: https://docs.flex.ai/platform-services/dataset-manager/from-local Upload dataset files from your local machine to FlexAI ## Uploading files Navigate to the Dataset Manager in the FlexAI Console: Visit the ["Add Dataset" section of the FlexAI Console](https://platform.flex.ai/datasets/new) Enter a name for your dataset: `nanoGPT-dataset` Select the "Local" option for "Upload Origin" Select the `+ Upload Item` button to open the "Upload Items" dialog If your Datasets are stored on your machine or local network, you can upload them to the FlexAI Dataset Manager by using the FlexAI CLI's `dataset push` subcommand: ```bash theme={null} flexai dataset push (--file = ... | --file ...) ``` ### The `--file` flag The `dataset push` command's `-f`/`--file` [flag](/cli/reference/dataset/push/#flags) offers a flexible way to upload files to a Dataset as it can be used 3 different methods. ## One file at a time Uploading files individually is useful when you need to specify source files that may have different locations on your source machine and/or you want to set a specific destination path for each of them: Let's assume the following file structure on your local machine: ``` ~/ openwebtext_mini/ - urlsf_subset00.tar - urlsf_subset01.tar - test.tar sf-wikitext/ - test-00000-of-00001.parquet - train-00000-of-00001.parquet ``` You can upload specific files from the `openwebtext_mini` and `sf-wikitext` directories to a FlexAI Dataset named `text-records-dataset-1`, while also specifying a custom destination path for each of them—including a different file name for each—, as seen below: ``` text-records-dataset-1/ owt/ - urlsf_subset00.tar - urlsf_subset01.tar test/ - test.tar - test-00000-of-00001.parquet wikitext/ - train-00000-of-00001.parquet ``` You can achieve this by going through the following steps iteratively for each file: Use the "Select file" option to open a file browser dialog Select the file you want to upload from your local machine In the "Destination Path" field, enter the desired destination path within the Dataset Select the `Add` button to confirm the file selection and destination mapping Below the file list named "Upload Items" you will find an `+ Add items` button that will open up the "Upload Items" dialog again. Repeat the steps above for each file you want to upload Finally, select the `Add Dataset` button to start the upload process. Currently, the FlexAI Console does not offer the ability to set a custom destination file name. If you require that feature, please refer to the 'Using the FlexAI CLI' instructions instead. Uploading files individually is useful when you need to specify source files that may have different locations on your source machine and/or you want to set a specific destination path for each of them: Let's assume the following file structure on your local machine: ``` ~/ openwebtext_mini/ - urlsf_subset00.tar - urlsf_subset01.tar - test.tar sf-wikitext/ - test-00000-of-00001.parquet - train-00000-of-00001.parquet ``` You can upload specific files from the `openwebtext_mini` and `sf-wikitext` directories to a FlexAI Dataset named `text-records-dataset-1`, while also specifying a custom destination path for each of them—including a different file name for each—, by running the following command: ```bash theme={null} flexai dataset push text-records-dataset-1 \ --file openwebtext_mini/urlsf_subset00.tar=owt/0.tar \ --file openwebtext_mini/urlsf_subset01.tar=owt/1.tar \ --file openwebtext_mini/test.tar=test/owt.tar \ --file sf-wikitext/test-00000-of-00001.parquet=test/wikitext.parquet \ --file sf-wikitext/train-00000-of-00001.parquet=wikitext/train.parquet ``` Here you have the ability to pick individual files from your local machine and set the specific destination path for each of them: some files come from the local `openwebtext_mini` directory, others from `sf-wikitext`, and in both cases, the test files will be uploaded to the Dataset's `test` directory, as shown below: ``` ~/ text-records-dataset-1/ owt/ - 0.tar - 1.tar test/ - owt.tar - wikitext.parquet wikitext/ - train.parquet ``` Pushing files this way can begin to become a cumbersome task as the list of files continues to grow, fortunately you can automate the process by using a script or a loop to upload multiple files following the patterns that fit your needs. However, if having a specific Dataset structure is not a requirement, you can upload files without specifying a destination path. ## Multiple files, no defined destination path Considering the example above, you could decide to simply upload the files without specifying a destination path. This would result in the files being moved up to the root directory of the FlexAI Dataset: Since none of the files names are the same, they won't overwrite each other, ending up in a FlexAI Dataset structure that looks as follows: ``` text-records-dataset-2/ - urlsf_subset00.tar - urlsf_subset01.tar - test.tar - test-00000-of-00001.parquet - train-00000-of-00001.parquet ``` All intermediate directories are disregarded when uploading files without specifying a destination path. However, when picking and choosing files to upload is not required for your data workflow, then you can use the third method: a bulk upload of files in a directory. Considering the example above, you could decide to simply upload the files without specifying a destination path. This would result in the files being moved up to the root directory of the FlexAI Dataset: ```bash theme={null} flexai dataset push text-records-dataset-2 \ --file openwebtext_mini/urlsf_subset00.tar \ --file openwebtext_mini/urlsf_subset01.tar \ --file openwebtext_mini/test.tar \ --file sf-wikitext/test-00000-of-00001.parquet \ --file sf-wikitext/train-00000-of-00001.parquet ``` Since none of the files names are the same, they won't overwrite each other, ending up in a FlexAI Dataset structure that looks as follows: ``` text-records-dataset-2/ - urlsf_subset00.tar - urlsf_subset01.tar - test.tar - test-00000-of-00001.parquet - train-00000-of-00001.parquet ``` All intermediate directories are disregarded when uploading files without specifying a destination path. However, when picking and choosing files to upload is not required for your data workflow, then you can use the third method: a bulk upload of files in a directory. ## Entire directory contents The `--file` flag also allows you to push the contents of a directory into a Dataset. This is particularly useful when you already have a directory containing the multiple files that make up your dataset. Let's assume the following file structure on your local machine: ``` ~/ my-dataset/ train/ - t_1.txt - t_2.txt - t_3.txt deep-text/ - t_1.txt - t_2.txt test/ - test_1.txt - test_2.txt deep-text_test - t_1.txt ``` > Yes, file names have been deliberately kept similar to show how pushing a entire directory with nested sub-directories is handled (no overwrite risk!). Uploading the contents of the `my-dataset` directory to a Dataset named `text-records-dataset-3` would follow the same pattern as before: Resulting in the following FlexAI Dataset structure: ``` text-records-dataset-3/ train/ - t_1.txt - t_2.txt - t_3.txt deep-text/ - t_1.txt - t_2.txt test/ - test_1.txt - test_2.txt deep-text_test/ - t_1.txt ``` Notice that the source `my-dataset` directory is not included in the FlexAI Dataset structure, only its contents are uploaded, the file structure is preserved. The `--file` flag also allows you to push the contents of a directory into a Dataset. This is particularly useful when you already have a directory containing the multiple files that make up your dataset. Let's assume the following file structure on your local machine: ``` ~/ my-dataset/ train/ - t_1.txt - t_2.txt - t_3.txt deep-text/ - t_1.txt - t_2.txt test/ - test_1.txt - test_2.txt deep-text_test - t_1.txt ``` > Yes, file names have been deliberately kept similar to show how pushing a entire directory with nested sub-directories is handled (no overwrite risk!). Uploading the contents of the `my-dataset` directory to a Dataset named `text-records-dataset-3` would follow the same pattern as before: ```bash theme={null} flexai dataset push text-records-dataset-3 --file my-dataset ``` Resulting in the following FlexAI Dataset structure: ``` text-records-dataset-3/ train/ - t_1.txt - t_2.txt - t_3.txt deep-text/ - t_1.txt - t_2.txt test/ - test_1.txt - test_2.txt deep-text_test/ - t_1.txt ``` Notice that the source `my-dataset` directory is not included in the FlexAI Dataset structure, only its contents are uploaded, the file structure is preserved. ## Troubleshooting ### Pushing large files Some times you may encounter issues when trying to upload large files directly from your computer. This kind of problem is usually related to network issues. There are a few things you can try to solve this issue: * Switch to a wired connection if possible. * Split the file into smaller parts or chunks that you can then join back together at runtime. * If the file is stored in a Cloud Storage Service such as Amazon S3 or Google Cloud Storage, you can upload directly to the FlexAI Dataset Manager by creating a [Remote Storage Provider Connection](/platform-services/dataset-manager/from-remote/). #### The process fails when uploading files from a remote machine you're connected to via SSH If you are trying to upload files from a machine that you're connected to via SSH, the process may fail due to the SSH connection being closed. To avoid this, you can use the a *terminal multiplexer*, like [`screen`](https://www.gnu.org/software/screen/) or [`tmux`](https://github.com/tmux/tmux/wiki), to keep the process running even after you close the remote session. # Uploading Datasets from Remote Sources Source: https://docs.flex.ai/platform-services/dataset-manager/from-remote Upload datasets from a remote storage provider to FlexAI If you rely on a Cloud Storage Service to store your Datasets, you can upload them directly to the FlexAI Dataset Manager without having to download them to your local machine first, streamlining the process by leveraging a "Server to Server" connection and thus, saving you time. ## The logic behind the process The process of uploading Datasets from remote sources to FlexAI is split into three steps: 1. *storing* the Storage Provider access credentials into the FlexAI Secret Manager 2. Creating a Storage Provider Connection 3. And finally, uploading the Datasets to FlexAI ### Storing the required credentials Credentials are safely kept using FlexAI's Secret Manager, which encrypts and stores them in a secure manner. You can store strings and even entire files as Secrets, allowing for maximum flexibility when it comes to managing your credentials. ### Creating a Storage Provider Connection Once access credentials are stored in a FlexAI Secret, you can then move on to creating a Storage Provider Connection. A Storage Provider Connection is a FlexAI entity that holds the necessary information to connect to a Cloud Storage Service. This includes the type of Storage Provider, the Secret containing the access credentials, and any additional configuration required to establish the connection which may vary depending on the Storage Provider. ### Uploading the Datasets to FlexAI Uploading Datasets from a remote source to FlexAI is quite similar to uploading them from your local machine. In fact you use the same command: `flexai dataset push`, but instead of specifying a local file path, and a destination path in the FlexAI Dataset using the `--file` flag, you pass in the name of the Storage Provider Connection (`--storage-provider`) and the path to the file or directory in the Remote Storage Service (`--source-path`): ```bash theme={null} flexai dataset push --storage-provider --source-path ``` An asynchronous process will start, and you will be able to monitor the progress of the upload using the `flexai dataset inspect ` command. You can push multiple Datasets using the same Storage Provider Connection using different values for `--source-path` or create additional Storage Provider Connections for other providers you have access to. The contents of the `` you push will be placed into the root of the Dataset. Learn more about a Dataset's structure in the [Dataset Manager](/platform-services/dataset-manager/) section. ## Supported Cloud Storage Providers The FlexAI Remote Storage Connection Manager Service currently supports the following Cloud Storage Providers: * `s3`: Amazon S3 * `gcs`: Google Cloud Storage * `r2`: Cloudflare R2 * `minio`: MinIO * `huggingface`: Hugging Face Hub You can learn more about how to create a Storage Provider Connection for each of these providers in the [Remote Storage Connection Manager](/platform-services/remote-storage-connections-manager/) guide. ### Working with other Cloud Storage Providers Some Cloud Storage Providers are not directly supported by FlexAI, but if they provide the option of mounting their storage to a *file system path* (either on your host machine or a Virtual Machine), you can use the `flexai dataset push` command to upload your Datasets to FlexAI. ### Microsoft Azure Files stored in Azure Blob Storage can be mounted to a file system path via [`BlobFuse2`](https://github.com/Azure/azure-storage-fuse/). Once the data is available as a file system path, you can then use the `flexai dataset push` command to upload your dataset files to make them available to the FlexAI Dataset Manager Service, as described in the [Uploading Datasets from your local machine](/platform-services/dataset-manager/from-local/) guide. Note that if you decide to use a Virtual Machine without a graphical environment, you can follow the instructions in the [CLI on a Virtualized Environment](/best-practices/alternative-methods/cli-on-a-virtualized-env/) page to authenticate your FlexAI CLI. Visit Microsoft Learn's guide on [How to mount an Azure Blob Storage container on Linux with BlobFuse2](https://learn.microsoft.com/en-us/azure/storage/blobs/blobfuse2-how-to-deploy) for more details. # Step by step: Uploading a Dataset From a Remote Storage Provider Source: https://docs.flex.ai/platform-services/dataset-manager/from-remote/steps Step-by-step instructions for uploading datasets from remote storage Here you will find instructions on how to upload datasets from some of the Remote Storage Providers supported by FlexAI. To upload Datasets from Amazon S3 to FlexAI, you first need to create a Storage Provider Connection that holds the necessary information to connect to your Amazon S3 bucket. You will find this entry from the AWS Security Blog useful: [How to quickly find and update your access keys \[...\]](https://aws.amazon.com/blogs/security/how-to-find-update-access-keys-password-mfa-aws-management-console/). You will need the following: * Your Amazon S3 Secret Access Key * Your Amazon S3 Access Key ID * The Amazon S3 region * The endpoint URL associated with your Amazon S3 region 1. Store Your Credentials using the FlexAI Secret Manager Store your Amazon S3 Secret Access Key To store your Amazon S3 Secret Access Key as an FlexAI Secret, run the following command: ```bash theme={null} flexai secret create s3-secret-access-key ``` You will be prompted to enter your Amazon S3 Secret Access Key (of course you can paste it in!). Once you have entered it, hit Enter, and the Secret `s3-secret-access-key` will be created. 2. Create the Storage Provider Connection Create the Storage Provider Connection With the Amazon S3 Secret Access Key stored as an FlexAI Secret, you can now create a Storage Provider Connection for Amazon S3 using the `flexai storage` command by following the example shown by the command string below: ```bash theme={null} flexai storage create \ --provider s3 \ --region \ --endpoint \ --access-key-id \ --secret-access-key-name ``` > Note that the value of `--endpoint` will depend on the *region* where your Amazon S3 bucket is located. You can find the official [list of Amazon S3 endpoints here](https://docs.aws.amazon.com/general/latest/gr/s3.html). A Remote Storage Connection for an Amazon S3 bucket located in the `eu-west-1` region with the endpoint `s3.eu-west-1.amazonaws.com` and an Access Key ID `AKIAIOSFODIN7AAF89GU` would look like this: ```bash theme={null} flexai storage create aws-storage-conn-eu \ --provider s3 \ --region eu-west-1 \ --endpoint s3.eu-west-1.amazonaws.com \ --access-key-id AKIAIOSFODIN7AAF89GU \ --secret-access-key-name s3-secret-access-key ``` 3. Upload Datasets from Amazon S3 to FlexAI Upload Datasets from Amazon S3 to FlexAI Now you can use your newly created `aws-storage-conn-eu` Storage Provider Connection to upload Datasets from an Amazon S3 bucket directly to FlexAI by using the `flexai dataset push` command as shown by the command string below: ```bash theme={null} flexai dataset push \ --storage-provider aws-storage-conn-eu \ --source-path / ``` For instance, creating an FlexAI Dataset named `s3-dataset-audio` from an Amazon S3 bucket named `data-sets` with the object key `files/wav-files` using the `aws-storage-conn-eu` Storage Provider Connection would look like this: ```bash theme={null} flexai dataset push s3-dataset-audio \ --storage-provider aws-storage-conn-eu \ --source-path data-sets/files/wav-files ``` 4. Monitor the Dataset Upload Progress Monitor the Dataset Upload Progress The progress of the Dataset upload can be monitored by using the `inspect` subcommand from `flexai dataset`: ```bash theme={null} flexai dataset inspect ``` Which for our example would look like this: ```bash theme={null} flexai dataset inspect s3-dataset-audio ``` To upload Datasets from Google Cloud Storage (GCS) to FlexAI, you first need to create a Storage Provider Connection that holds the necessary information to connect to your GCS bucket. Setting up a Storage Provider Connection for GCS leverages Google Cloud Platform's Service Account JSON key file. You can find more information on how to create a Service Account JSON key file in the [Google Cloud documentation](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console) (Pick the Console tab for instructions on how to obtain it using the GCP web console). You will need the following: * A Google Cloud Service Account JSON key file with sufficient permissions to access the target GCS bucket. 1. Store Your Credentials using the FlexAI Secret Manager Store Your Google Cloud Service Account Key To store your Google Cloud Service Account JSON key file as an FlexAI Secret, run the following command: ```bash theme={null} cat <service_account_key.json> | flexai secret create <secret_name> --value-stdin ``` For example, if your Google Cloud Service Account JSON key file is named `gcp-service-account.json`, you can store it as an FlexAI Secret named `gcp-sa` by running the following command: ```bash theme={null} cat gcp-service-account.json | flexai secret create gcp-sa --value-stdin ``` This command reads the contents of your Google Cloud Service Account JSON key file and securely stores its entirety as an FlexAI Secret named `gcp-sa`. 2. Create the Storage Provider Connection Create the Storage Provider Connection With the Google Cloud Service Account JSON key file stored as an FlexAI Secret, you can now create a Storage Provider Connection for GCS using the `flexai storage` command as shown by the example below: ```bash theme={null} flexai storage create \ --provider gcs \ --service-account-file-name ``` For example, creating a Storage Provider Connection named `gcs-conn` that has the Service Account Key JSON details stored in the `gcp-sa` FlexAI Secret would look like this: ```bash theme={null} flexai storage create gcs-conn \ --provider gcs \ --service-account-file-name gcp-sa ``` After running the command, the Storage Provider Connection `gcs-conn` will be created. 3. Upload Datasets from Google Cloud Storage to FlexAI Upload Datasets from Google Cloud Storage to FlexAI Now you can use the `gcs-conn` Storage Provider Connection to upload Datasets from a GCS bucket to FlexAI by using the `flexai dataset push` command as shown by the command string below: ```bash theme={null} flexai dataset push \ --storage-provider \ --source-path / ``` For instance, creating an FlexAI Dataset named `gcs-dataset-audio` from a GCS bucket named `data-sets` with the object key `files/wav-files` using the `gcs-conn` Storage Provider Connection would look like this: ```bash theme={null} flexai dataset push gcs-dataset-audio \ --storage-provider gcs-conn \ --source-path data-sets/files/wav-files ``` After running the command, the dataset `gcs-dataset-audio` will begin to be synced by asynchronously copying the contents of the GCS bucket resource `data-sets/files/wav-files` into the root of the Dataset. 4. Monitor the Dataset Upload Progress Monitor the Dataset Upload Progress The progress of the Dataset upload can be monitored by using the `inspect` subcommand from `flexai dataset`: ```bash theme={null} flexai dataset inspect ``` Which for our example would look like this: ```bash theme={null} flexai dataset inspect gcs-dataset-audio ``` To upload Datasets from Hugging Face Hub to FlexAI, you first need to create a Storage Provider Connection that holds the necessary information to connect to your Hugging Face Hub account. Setting up a Storage Provider Connection for Hugging Face Hub requires a [Hugging Face Access Token](https://huggingface.co/settings/tokens). You can find more information on how to create a Hugging Face Access Token in the [Hugging Face documentation](https://huggingface.co/docs/huggingface_hub/how-to-authenticate). 1. Store Your Hugging Face Access Token using the FlexAI Secret Manager To store your Hugging Face Access Token as an FlexAI Secret, run the following command: ```bash theme={null} flexai secret create hf_token ``` You will be prompted to enter your Hugging Face Access Token. Once you have entered it, hit Enter, and the Secret `hf_token` will be created. 2. Create the Storage Provider Connection With the Hugging Face Access Token stored as an FlexAI Secret, you can now create a Storage Provider Connection for Hugging Face Hub using the `flexai storage` command as shown below: ```bash theme={null} flexai storage create hf-conn \ --provider huggingface \ --hf-token-name hf_token ``` After running the command, the Storage Provider Connection `hf-conn` will be created. 3. Upload Datasets from Hugging Face Hub to FlexAI Now you can use the `hf-conn` Storage Provider Connection to upload Datasets from Hugging Face Hub to FlexAI by using the `flexai dataset push` command as shown below: ```bash theme={null} flexai dataset push plant-disease-dataset \ --storage-provider hf-conn \ --source-path Diginsa/Plant-Disease-Detection-Project/dataset ``` This example creates an FlexAI Dataset named `plant-disease-dataset` from the Hugging Face repository `Diginsa/Plant-Disease-Detection-Project` using the `hf-conn` Storage Provider Connection, pulling the contents from the `dataset` directory in the repo. After running the command, the dataset `plant-disease-dataset` will begin to be synced by asynchronously copying the contents from Hugging Face Hub into the root of the Dataset. 4. Monitor the Dataset Upload Progress The progress of the Dataset upload can be monitored by using the `inspect` subcommand from `flexai dataset`: ```bash theme={null} flexai dataset inspect plant-disease-dataset ``` # Dataset Lifecycle Source: https://docs.flex.ai/platform-services/dataset-manager/lifecycle Understand the status progression of a dataset from upload to availability ## Dataset Lifecycle ### Creation and Upload 1. **Prepare data** - Organize files locally or identify cloud storage locations 2. **Upload Dataset** - Use either the FlexAI Console or CLI to upload the Dataset 3. **Verify completion** - Check upload status progress until it reaches the `available` status 4. **Ready for training** - Dataset becomes available for Training or Fine jobs ### Statuses Monitor Dataset lifecycle with clear status indicators: | Status | Source | Description | | - | - | - | | `uploading` | `Host` | Dataset files are being transferred from a host machine | | `syncing` | `Remote Storage Provider` | Dataset files are being synchronized from a Remote Storage Provider | | `pending` | `Host` or `Remote Storage Provider` | The Dataset is being processed by FlexAI | | `available` | `Host` or `Remote Storage Provider` | The Dataset is ready for use in Training or Fine-tuning Jobs | | `error` | `Host` or `Remote Storage Provider` | The upload process failed. Inspecting its details can provide more information: `flexai dataset inspect ` | ### Dataset lifecycle You will notice that the Dataset lifecycle follows a clear progression through its statuses, with the only difference being the source of the upload: ```d2 theme={null} "uploading (Host)" -> pending {style.animated: true} "uploading (Host)" -> error "syncing (Remote Storage Provider)" -> pending {style.animated: true} "syncing (Remote Storage Provider)" -> error pending -> available {style.animated: true} pending -> error ``` # Overview Source: https://docs.flex.ai/platform-services/observability Monitor and analyze infrastructure metrics, training performance, and system health FlexAI's Observability platform provides comprehensive monitoring and analytics capabilities for your AI workloads, enabling you to track infrastructure metrics, monitor training performance, and optimize resource utilization across your machine learning operations. The Observability platform enables you to: * **Monitor infrastructure metrics** in real-time during training and inference * **Track training performance** with detailed visualizations and analytics * **Optimize resource utilization** to maximize compute efficiency * **Debug performance issues** with comprehensive system insights * **Analyze cost patterns** and resource consumption trends ## Key Features Real-time GPU utilization, memory usage, temperature, and system metrics Native TensorBoard support for training metrics visualization and comparison Detailed analysis of compute efficiency, bottlenecks, and optimization opportunities 30-day data retention for trend analysis and performance comparison ## Infrastructure Monitoring ### FlexAI Infrastructure Monitor Access real-time infrastructure metrics at [dashboards.flex.ai](https://dashboards.flex.ai): * **GPU Metrics** - Utilization percentage, memory consumption, temperature monitoring * **System Resources** - CPU usage, RAM consumption, disk I/O patterns * **Network Activity** - Data transfer rates, network latency, bandwidth utilization * **Power Consumption** - Energy usage tracking and efficiency metrics ### Real-Time Dashboards Monitor your workloads with comprehensive dashboard views: * **Live updates** - Real-time metric updates during active training jobs * **Historical trends** - View performance patterns over time * **Comparative analysis** - Compare metrics across different training runs * **Custom filtering** - Focus on specific time ranges, jobs, or resource types ### Performance Optimization Use infrastructure metrics to optimize your workloads: * **Batch size tuning** - Identify optimal batch sizes for maximum GPU utilization * **Memory optimization** - Prevent OOM errors and maximize memory efficiency * **Bottleneck identification** - Locate performance bottlenecks in your training pipeline * **Resource scaling** - Determine when to scale up or down compute resources ## Training Analytics ### TensorBoard Integration Native TensorBoard support at [tensorboard.flex.ai](https://tensorboard.flex.ai): * **Training metrics** - Loss curves, accuracy trends, validation scores * **Model visualization** - Network architecture and computational graphs * **Hyperparameter tracking** - Compare different hyperparameter configurations * **Blueprint comparison** - Side-by-side analysis of multiple training runs ### Training Performance Metrics Comprehensive tracking of training progression: * **Loss functions** - Training and validation loss over time * **Learning curves** - Model performance improvement tracking * **Convergence analysis** - Identify when models reach optimal performance * **Overfitting detection** - Early warning signs of model overfitting ### Model Quality Assessment Evaluate model quality with detailed analytics: * **Accuracy metrics** - Precision, recall, F1-score, and custom metrics * **Validation performance** - Generalization capability assessment * **Statistical analysis** - Distribution analysis and statistical significance * **Quality degradation** - Monitor for model performance degradation over time ## Resource Utilization Analytics ### Compute Efficiency Maximize the value of your compute resources: * **GPU utilization rates** - Track how effectively GPUs are being used * **Memory efficiency** - Monitor memory allocation and usage patterns * **Idle time analysis** - Identify periods of resource underutilization * **Cost per training step** - Calculate efficiency metrics for cost optimization ### Capacity Planning Plan future resource needs with historical data: * **Usage patterns** - Understand peak and average resource consumption * **Growth projections** - Forecast future compute requirements * **Resource allocation** - Optimize resource distribution across workloads * **Budget planning** - Predict and plan for compute costs ### Multi-GPU Analysis Specialized monitoring for distributed training: * **GPU synchronization** - Monitor communication between GPUs * **Load balancing** - Ensure even distribution across available GPUs * **Scaling efficiency** - Measure performance improvements with additional GPUs * **Communication overhead** - Track inter-GPU communication costs ## Debugging and Troubleshooting ### Performance Debugging Identify and resolve performance issues: * **Slow training detection** - Automatic alerts for unusually slow training * **Memory leak identification** - Detect gradual memory consumption increases * **I/O bottlenecks** - Identify data loading and storage performance issues * **Network latency** - Monitor distributed training communication delays ### System Health Monitoring Comprehensive system health tracking: * **Hardware status** - Monitor GPU health, temperature warnings, and errors * **Service availability** - Track uptime and service reliability * **Error logging** - Centralized collection and analysis of system errors * **Alert systems** - Proactive notifications for system issues ### Root Cause Analysis Tools for deep performance investigation: * **Timeline analysis** - Detailed execution timeline for performance events * **Resource correlation** - Correlate performance issues with resource usage * **Comparative debugging** - Compare problematic runs with successful ones * **Historical context** - Understand how current issues relate to past performance ### Performance Optimizations * **Underutilized resources** - Track opportunities to reduce resource allocation * **Optimal instance types** - Identify the most cost-effective hardware * **Batch optimization** - Find optimal batch sizes for enhanced cost-performance ratio ## Data Retention and Access ### Data Storage Comprehensive data retention policies: * **30-day retention** - All metrics and logs retained for 30 days * **Export capabilities** - Download metrics data for external analysis * **API access** - Programmatic access to observability data * **Integration support** - Connect with external monitoring systems ### Data Privacy and Security Secure handling of observability data: * **Encrypted storage** - All metrics data encrypted at rest * **Access controls** - Role-based access to observability dashboards * **Audit trails** - Track access to sensitive performance data * **Compliance support** - Meet regulatory requirements for data handling ## Integration and APIs ### Third-Party Integrations Connect with external monitoring tools: * **Prometheus** - Export metrics to Prometheus for custom dashboards * **Grafana** - Visualize FlexAI metrics in Grafana dashboards * **DataDog** - Stream metrics to DataDog for unified monitoring * **Custom integrations** - Build custom integrations using API calls or webhooks in conjunction with FlexAI Secrets ## Getting Started Ready to start monitoring your AI workloads? Explore these resources: Access real-time infrastructure metrics and performance data Visualize training metrics and compare blueprints Debug performance issues with SSH access to training environments Common questions about monitoring and observability # Infrastructure Monitor Source: https://docs.flex.ai/platform-services/observability/infrastructure-monitor Monitor real-time GPU and system metrics for your FlexAI workloads FlexAI's **Infrastructure Monitor** provides real-time system and GPU metrics that will enable you to make any optimizations to your workloads' code and settings so you can take full advantage of FlexAI compute resources. Use the FlexAI Infrastructure Monitor data to plan your scaling strategies, make batch size or data preprocessing decisions, and in general, make informed decisions based on the data of any current or past trainings during the last 30 days. A screenshot of the FlexAI dashboards Visit [https://dashboards.flex.ai](https://dashboards.flex.ai) and log in using your FlexAI account to start using FlexAI's Infrastructure Monitor. # TensorBoard Source: https://docs.flex.ai/platform-services/observability/tensorboard Visualize training metrics in real time with FlexAI's hosted TensorBoard FlexAI's hosted [TensorBoard](https://tensorboard.flex.ai/) instance provides organization-wide real-time insights into your AI Workloads' progress. TensorBoard is enabled by default for every organization, allowing you to track metrics, visualize model graphs, and analyze performance without any additional setup other than making your code write the required files to the path specified in the `TENSORBOARD_LOG_DIR` environment variable. A screenshot of the TensorBoard interface Visit [https://tensorboard.flex.ai/](https://tensorboard.flex.ai/) and log in using your FlexAI account credentials to access the TensorBoard interface. You can learn more about TensorBoard itself by visiting the [TensorBoard documentation](https://www.tensorflow.org/tensorboard/get_started). # Overview Source: https://docs.flex.ai/platform-services/remote-storage-connections-manager Remote Storage Connection Manager The FlexAI Remote Storage Connection Manager enables seamless integration with cloud storage providers, facilitating efficient data transfer and access without the need for local downloads and re-uploads. Create secure connections to your cloud storage and move data directly between their servers and the FlexAI platform. ## Overview The Remote Storage Connection Manager enables you to: * **Connect to cloud storage** providers like AWS S3, Google Cloud Storage, and Cloudflare R2 * **Transfer data efficiently** with server-to-server connections * **Manage credentials securely** using the FlexAI Secret Manager * **Upload datasets directly** from a third party Cloud Storage Provider to FlexAI * **Export checkpoints** To your preferred storage destinations ## Supported Storage Providers Native support for AWS S3 buckets with regional endpoint configuration Direct integration with GCS buckets using Service Account authentication S3-compatible interface for Cloudflare R2 storage Direct access to datasets and models from Hugging Face repositories Self-hosted S3-compatible storage for on-premises deployments ## Common Data Transfer Operations ### Dataset Upload Workflows Efficiently move Datasets from a Cloud Storage Provider to FlexAI: 1. **Direct transfer** - Server-to-server transfer without local downloads 2. **Resource optimization** - FlexAI handles these data transfers without using compute resources 3. **Dataset structure preservation** - Your file and directory structure is kept intact ### Checkpoint Export Workflows Upload model Checkpoints to a Cloud Storage Provider: 1. **Flexible destinations** - Export to any\* connected storage provider 2. **Custom paths** - Specify exact destination locations 3. **Metadata preservation** - Maintain checkpoint metadata and versioning > Note: Exporting to the Hugging Face Hub is not currently supported. ### Model Checkpoint Import Workflows Upload model Checkpoints from a Cloud Storage Provider: 1. **Flexible sources** - Import base models or custom checkpoints of your Remote Storage Providers, like the Hugging Face Hub 2. **Metadata recognition** - FlexAI recognizes checkpoint metadata and versioning 3. **Resume Training** - Continue Training or Fine-tuning your imported Checkpoints 4. **Serve Models** - Deploy Inference Endpoints from imported Checkpoints ## Cost Optimization Minimize data transfer costs with smart routing: * **Regional optimization** - Use storage regions close to FlexAI infrastructure * **Direct transfers** - Avoid egress fees from intermediate downloads * **Efficient data transfers** - Avoid fetching Datasets during runtime ## Compliance * **Data residency** - Choose storage regions that meet compliance requirements * **Encryption at rest** - Leverage cloud provider encryption capabilities * **Access logging** - Maintain detailed logs for compliance auditing * **Data lifecycle** - Implement appropriate data retention and deletion policies ## Getting Started Ready to connect your cloud storage? Explore these resources: Complete CLI reference for storage connections Upload datasets from cloud storage Export checkpoints to cloud storage Manage storage credentials securely # Create a Remote Storage Connection Source: https://docs.flex.ai/platform-services/remote-storage-connections-manager/create Set up a new remote storage connection with credentials and provider configuration Creating a Remote Storage Connection involves two main steps: 1. **Store your access credentials** securely using the FlexAI Secret Manager. 2. **Create a Remote Storage Connection** that uses the stored credentials. ## Storing your Access Credentials Visit the [Create Secret](https://platform.flex.ai/s/secrets/new-secret) page in the FlexAI Console to create a new secret. > Note that currently, the FlexAI Console only supports creating secrets with text values. If you need to create a secret with a file value (e.g., a JSON key file for Google Cloud Storage), please use the FlexAI CLI instead. To upload Datasets from Amazon S3 to FlexAI, you first need to create a Storage Provider Connection that holds the necessary information to connect to your Amazon S3 bucket. You will find this entry from the AWS Security Blog useful: [How to quickly find and update your access keys \[...\]](https://aws.amazon.com/blogs/security/how-to-find-update-access-keys-password-mfa-aws-management-console/). You will need the following: * Your Amazon S3 Secret Access Key * Your Amazon S3 Access Key ID * The Amazon S3 region * The endpoint URL associated with your Amazon S3 region To store your Amazon S3 Secret Access Key as an FlexAI Secret, run the following command: ```bash theme={null} flexai secret create s3-secret-access-key ``` You will be prompted to enter your Amazon S3 Secret Access Key (of course you can paste it in!). Once you have entered it, hit Enter, and the Secret `s3-secret-access-key` will be created. To upload Datasets from Google Cloud Storage (GCS) to FlexAI, you first need to create a Storage Provider Connection that holds the necessary information to connect to your GCS bucket. Setting up a Storage Provider Connection for GCS leverages Google Cloud Platform's Service Account JSON key file. You can find more information on how to create a Service Account JSON key file in the [Google Cloud documentation](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console) (Pick the Console tab for instructions on how to obtain it using the GCP web console). You will need the following: * A Google Cloud Service Account JSON key file with sufficient permissions to access the target GCS bucket. To store your Google Cloud Service Account JSON key file as an FlexAI Secret, run the following command: ```bash theme={null} cat | flexai secret create --value-stdin ``` For example, if your Google Cloud Service Account JSON key file is named `gcp-service-account.json`, you can store it as an FlexAI Secret named `gcp-sa` by running the following command: ```bash theme={null} cat gcp-service-account.json | flexai secret create gcp-sa --value-stdin ``` This command reads the contents of your Google Cloud Service Account JSON key file and securely stores its entirety as an FlexAI Secret named `gcp-sa`. To upload Datasets from Hugging Face Hub to FlexAI, you first need to create a Storage Provider Connection that holds the necessary information to connect to your Hugging Face Hub account. Setting up a Storage Provider Connection for Hugging Face Hub requires a [Hugging Face Access Token](https://huggingface.co/settings/tokens). You can find more information on how to create a Hugging Face Access Token in the [Hugging Face documentation](https://huggingface.co/docs/huggingface_hub/how-to-authenticate). To store your Hugging Face Access Token as an FlexAI Secret, run the following command: ```bash theme={null} flexai secret create hf_token ``` You will be prompted to enter your Hugging Face Access Token. Once you have entered it, hit Enter, and the Secret `hf_token` will be created. ## Creating a Remote Storage Connection Currently, creating Remote Storage Connections via the FlexAI Console is not supported. Please use the FlexAI CLI instead. With the Amazon S3 Secret Access Key stored as an FlexAI Secret, you can now create a Storage Provider Connection for Amazon S3 using the `flexai storage` command by following the example shown by the command string below: ```bash theme={null} flexai storage create \ --provider s3 \ --region \ --endpoint \ --access-key-id \ --secret-access-key-name ``` > Note that the value of `--endpoint` will depend on the *region* where your Amazon S3 bucket is located. You can find the official [list of Amazon S3 endpoints here](https://docs.aws.amazon.com/general/latest/gr/s3.html). A Remote Storage Connection for an Amazon S3 bucket located in the `eu-west-1` region with the endpoint `s3.eu-west-1.amazonaws.com` and an Access Key ID `AKIAIOSFODIN7AAF89GU` would look like this: ```bash theme={null} flexai storage create aws-storage-conn-eu \ --provider s3 \ --region eu-west-1 \ --endpoint s3.eu-west-1.amazonaws.com \ --access-key-id AKIAIOSFODIN7AAF89GU \ --secret-access-key-name s3-secret-access-key ``` Now you can use your newly created `aws-storage-conn-eu` Storage Provider Connection to upload Datasets from an Amazon S3 bucket directly to FlexAI by using the `flexai dataset push` command as shown by the command string below: ```bash theme={null} flexai dataset push \ --storage-provider aws-storage-conn-eu \ --source-path / ``` For instance, creating an FlexAI Dataset named `s3-dataset-audio` from an Amazon S3 bucket named `data-sets` with the object key `files/wav-files` using the `aws-storage-conn-eu` Storage Provider Connection would look like this: ```bash theme={null} flexai dataset push s3-dataset-audio \ --storage-provider aws-storage-conn-eu \ --source-path data-sets/files/wav-files ``` The progress of the Dataset upload can be monitored by using the `inspect` subcommand from `flexai dataset`: ```bash theme={null} flexai dataset inspect ``` Which for our example would look like this: ```bash theme={null} flexai dataset inspect s3-dataset-audio ``` With the Google Cloud Service Account JSON key file stored as an FlexAI Secret, you can now create a Storage Provider Connection for GCS using the `flexai storage` command as shown by the example below: ```bash theme={null} flexai storage create \ --provider gcs \ --service-account-file-name ``` For example, creating a Storage Provider Connection named `gcs-conn` that has the Service Account Key JSON details stored in the `gcp-sa` FlexAI Secret would look like this: ```bash theme={null} flexai storage create gcs-conn \ --provider gcs \ --service-account-file-name gcp-sa ``` After running the command, the Storage Provider Connection `gcs-conn` will be created. Now you can use the `gcs-conn` Storage Provider Connection to upload Datasets from a GCS bucket to FlexAI by using the `flexai dataset push` command as shown by the command string below: ```bash theme={null} flexai dataset push \ --storage-provider \ --source-path / ``` For instance, creating an FlexAI Dataset named `gcs-dataset-audio` from a GCS bucket named `data-sets` with the object key `files/wav-files` using the `gcs-conn` Storage Provider Connection would look like this: ```bash theme={null} flexai dataset push gcs-dataset-audio \ --storage-provider gcs-conn \ --source-path data-sets/files/wav-files ``` After running the command, the dataset `gcs-dataset-audio` will begin to be synced by asynchronously copying the contents of the GCS bucket resource `data-sets/files/wav-files` into the root of the Dataset. The progress of the Dataset upload can be monitored by using the `inspect` subcommand from `flexai dataset`: ```bash theme={null} flexai dataset inspect ``` Which for our example would look like this: ```bash theme={null} flexai dataset inspect gcs-dataset-audio ``` With the Hugging Face Access Token stored as an FlexAI Secret, you can now create a Storage Provider Connection for Hugging Face Hub using the `flexai storage` command as shown below: ```bash theme={null} flexai storage create hf-conn \ --provider huggingface \ --hf-token-name hf_token ``` After running the command, the Storage Provider Connection `hf-conn` will be created. Now you can use the `hf-conn` Storage Provider Connection to push a Hugging Face Hub Dataset to to the FlexAI DatasetManager by using the `flexai dataset push` command as shown below: ```bash theme={null} flexai dataset push hf-finepdfs \ --storage-provider hf-conn \ --source-path HuggingFaceFW/finepdfs ``` This example creates an FlexAI Dataset named `hf-finepdfs` from the Hugging Face repository `HuggingFaceFW/finepdfs` using the `hf-conn` Storage Provider Connection, pulling the contents from the `dataset` directory in the repo. After running the command, the dataset `hf-finepdfs` will begin to be synced by asynchronously copying the contents from Hugging Face Hub into the root of the Dataset. The progress of the Dataset upload can be monitored by using the `inspect` subcommand from `flexai dataset`: ```bash theme={null} flexai dataset inspect hf-finepdfs ``` ## Next Steps Monitoring the push process of Datasets and Checkpoints: # Secret Manager Source: https://docs.flex.ai/platform-services/secret-manager Securely store and manage API keys, credentials, and secrets for your FlexAI workloads The FlexAI Secret Manager provides a secure, encrypted storage solution for managing sensitive information such as API keys, access tokens, credentials, and configuration data. All secrets are encrypted at rest and can be safely referenced in your training and inference workloads. The Secret Manager enables you to: * **Securely store credentials** for cloud storage providers (AWS S3, Google Cloud Storage, etc.) * **Manage API tokens** for services like Hugging Face Hub * **Store configuration files** as secrets for easy deployment * **Reference secrets in workloads** without exposing sensitive data * **Inject environment variables** automatically during training and inference ## Key Features All secrets are encrypted at rest using industry-standard encryption protocols Store strings, tokens, or entire files as secrets for maximum flexibility Automatically inject secrets as environment variables in Training and Inference jobs Secure access patterns ensure secrets are only available to authorized workloads ## Common Use Cases ### Cloud Storage Authentication Store access credentials for [Remote Storage Connections](/platform-services/remote-storage-connections-manager/) to enable seamless Dataset uploads and Checkpoint exports: * **AWS S3**: Store Secret Access Keys for S3 bucket access * **Google Cloud Storage**: Store Service Account JSON files * **Azure Blob Storage**: Store connection strings and access keys ### Model Repository Access Securely authenticate to **Hugging Face** to pull Models and Datasets. ### Training Configuration Store sensitive configuration data and credentials: * **Database connections**: Store connection strings and credentials * **External APIs**: Store API keys for third-party services * **Certificates**: Store SSL/TLS certificates and private keys * **Experiment Tracking**: Store API keys for experiment tracking services, such as Weights & Biases, Neptune AI, or MLflow ## Security Best Practices * **Never include secrets in code** - Always use the Secret Manager instead of hardcoding credentials * **Use descriptive names** - Name secrets clearly to identify their purpose and scope * **Rotate credentials regularly** - FlexAI Secret Manager allows you to update Secrets periodically and remove unused ones ## Getting Started Visit the [Secret Manager section](https://platform.flex.ai/s/secrets) of the FlexAI Console to create, update, or delete Secrets through an intuitive web interface. ### Creating a Secret Create a Secret by using the `flexai secret create` command, which will receive the name of the secret as its only argument. In this case we will use `hf_token` to store a Hugging Face Access Token. ```bash theme={null} flexai secret create hf_token ``` You will be prompted to enter the value for the secret. You can either type in the value directly or paste it. In any case, the value will not be displayed in the terminal. ```bash theme={null} flexai secret create hf_token Secret Value: █ ``` ### Passing a value from Standard input (stdin) You can pass the value of a Secret directly from standard input (stdin) by piping in the value. This is particularly useful for automated scripts, when you want to avoid interactive prompts, or when you want to store the contents of a file as a Secret—like a Google Cloud Service Account File. Piping in the value and using the `-p`/`--value-stdin` flag: ```bash theme={null} cat gcp-service-account.json | flexai secret create --value-stdin secret-from-sa-file ``` ### Updating a Secret An existing Secret can be updated by using the `flexai secret update` command, which will receive the name of the secret as its only argument. Let's update `hf_token` to modify the Hugging Face Access Token's value. ```bash theme={null} flexai secret update hf_token ``` You will be prompted to enter the new value for the Secret, just like when creating it. ```bash theme={null} flexai secret update hf_token Secret Value: █ ``` ### Using Secrets Secrets can be passed to Workloads in different ways: some commands flags receive *Secret names* as their value, and others explicitly require the name of a Secret and a name to associate it with, such as the `-S`/`--secret` flag of the `flexai training run` and `flexai training debug-ssh` commands. #### Training or Fine-tuning Workloads ```bash title="Creating a Training Job" theme={null} flexai training run test-123 \ --dataset open_web \ --repository-url https://github.com/flexaihq/nanoGPT/ \ --secret HF_TOKEN=hf_token \ --secret WANDB_API_KEY=wandb-key \ -- train.py ... ``` #### debug-ssh Sessions ```bash title="Starting a debug-ssh session" theme={null} flexai training debug-ssh \ --secret HF_TOKEN=hf_token_dev \ --secret SA_FILE=secret-from-sa-file \ -S DB_API_KEY=db-api-key-staging \ --repository-url https://github.com/flexaihq/nanoGPT/ \ --vscode ``` Note that Training and Fine-tuning Workloads, as well as a debug-ssh sessions can receive multiple instances of the `-S`/`--secret` flag. #### Inference Endpoints The `flexai inference serve` command's `--hf-token-secret` and `--api-key-secret` flags expect the name of a Secret: ```bash title="Creating an Inference Endpoint" theme={null} flexai inference serve llm-text-inference-prod \ --hf-token-secret hf_token_prod \ --api-key-secret api-key-prod \ -- --model=mistralai/Mistral-7B-Instruct-v0.1 ... ``` # Sign In Source: https://docs.flex.ai/signin Sign in to FlexAI with your email or GitHub account You can sign in to the FlexAI Web Console by visiting the [FlexAI Web Console's Sign In page](https://platform.flex.ai/login). The authentication page will provide two options to sign in: 1. Sign in with Email and Password: If you prefer, you can sign in using an email address. You'll need to provide some information to get started. 2. Sign in with GitHub: If you have a GitHub account, you can sign in by authorizing FlexAI as an OAuth application. Screenshot of the 'Sign in with Console' page ## Sign In with Email and Password If you signed up to FlexAI with *email and password*, you'll need to provide the **Work Email** and **Password** combination you used during sign up. Once you enter your credentials, select the **Sign in with password** button. You'll be redirected to the [FlexAI Web Console](https://platform.flex.ai/). If you ever forget your password, you can reset it by [visiting this page](https://platform.flex.ai/recovery) or selecting the **Forgot password?** link on the [sign-in page](https://platform.flex.ai/login). You'll be prompted to enter your email address. You will then receive an email with instructions to reset your password. ## Sign In with GitHub If you had signed up to FlexAI with GitHub, then select the "Sign in with GitHub" button. ### If you are not logged into your GitHub account If you're not already logged in, you'll be redirected to the GitHub login page. Follow the regular GitHub login process to log in to your account. This may involve entering your username and password, or using a 2FA authentication method if you have enabled it. Once the FlexAI authentication request completes successfully, you'll be redirected to the [FlexAI Web Console](https://platform.flex.ai/). If you encounter any issues during the sign in process, do not hesitate to reach out to our support team for assistance at [support@flex.ai](mailto:support@flex.ai). Run the `flexai auth login` command in your terminal. ```bash theme={null} flexai auth login ``` Depending on your system's capabilities, a browser tab pointing to the FlexAI sign in page will open automatically or you will be prompted with an authentication link to manually open in your browser. ```text title="flexai auth login response" theme={null} Please visit the following URL in your browser to authenticate: https://api.flex.ai/api/v1/auth/login?redirect_endpoint=%2Fcallback&redirect_host=localhost%3A40066&redirect_scheme=http ``` The authentication page will provide two options to sign in: 1. Sign in with Email and Password: If you prefer, you can sign in using an email address. You'll need to provide some information to get started. 2. Sign in with GitHub: If you have a GitHub account, you can sign in by authorizing FlexAI as an OAuth application. Screenshot of the 'Sign in with CLI' page ## Sign In with Email and Password If you signed up to FlexAI with *email and password*, you'll need to provide the **Work Email** and **Password** combination you used during sign up. Once you enter your credentials, select the **Sign in with password** button. You'll be redirected to the [FlexAI Web Console](https://platform.flex.ai/). If you ever forget your password, you can reset it by [visiting this page](https://platform.flex.ai/recovery) or selecting the **Forgot password?** link on the [sign-in page](https://platform.flex.ai/login). You'll be prompted to enter your email address. You will then receive an email with instructions to reset your password. ## Sign In with GitHub If you had signed up to FlexAI with GitHub, then select the "Sign in with GitHub" button. ### If you are not logged into your GitHub account If you're not already logged in, you'll be redirected to the GitHub login page. Follow the regular GitHub login process to log in to your account. This may involve entering your username and password, or using a 2FA authentication method if you have enabled it. Once the FlexAI authentication request completes successfully, you'll be redirected to the [FlexAI Web Console](https://platform.flex.ai/). If you encounter any issues during the sign in process, do not hesitate to reach out to our support team for assistance at [support@flex.ai](mailto:support@flex.ai). # Sign Up Source: https://docs.flex.ai/signup Create a FlexAI account with email or GitHub You can sign up for FlexAI by visiting the [FlexAI Web Console's Sign Up page](https://platform.flex.ai/signup). Then follow the instructions below to create your account. Here you will get two options to sign up: 1. Sign up with Email and Password: If you prefer, you can sign up using an email address. You'll need to provide some information to get started. 2. Sign up with GitHub: If you have a GitHub account, you can sign up by authorizing FlexAI as an OAuth application. ## Sign Up with Email and Password If you prefer to sign up with email and password, you'll need to provide: * **Work Email**: Your work email address. Must be a valid email address using your company's domain. For example, `george@example-company.com` * **Password**: A strong password for your account. * **First Name**: *Given* name. * **Last Name**: *Family* name. * **Company Name**: The name of your company or organization. * **Phone Number**: An contact phone number. Optional. FlexAI Sign Up Page The email address you provide must be a valid work email address using your company's domain. Personal email addresses (e.g., @gmail.com, @aol.com, @wanadoo.fr, @uol.com.br, etc.) are not accepted. After filling out the form, check the *I accept the Terms of Service* box to agree to the terms, and finally select the **Sign up** button. Once you submit the form, you'll get a confirmation message indicating that a **verification code** has been sent to your email address. Check your inbox for the verification code. Enter the **verification code** in the provided field on the sign-up page or follow the link in the email. If you decided to manually enter the code, select the **Continue** button to complete the sign-up process. If you don't receive the verification code after a couple of minutes, you can request a new one by selecting the **Resend Code** button. Once you enter the verification code, you'll be redirected to the [FlexAI Web Console](https://platform.flex.ai/), where you can start managing your AI workloads. ## Sign Up with GitHub If you choose to sign up with GitHub but you're not already logged in, you'll be redirected to the GitHub login page. ### If you are not logged into your GitHub account Follow the regular GitHub login process to log in to your account. This may involve entering your username and password, or using a two-factor authentication method if you have it enabled. Once logged in, you'll be redirected back to the FlexAI authentication authorization page. ### FlexAI Authentication Authorization page Here, you can review the permissions that FlexAI is requesting. Select the **Authorize flexaihq** button. Depending on your GitHub account's settings, you may need to go through an additional GitHub account verification step. Once the FlexAI authentication request completes successfully, you'll be redirected to the [FlexAI Web Console](https://platform.flex.ai/). If you encounter any issues during the sign up process, do not hesitate to reach out to our support team for assistance at [support@flex.ai](mailto:support@flex.ai).