Skip to content
FlexAI Logo FlexAI Logo FlexAI Docs

inference serve

Creates an Inference Endpoint from a model hosted by Hugging Face. The target model must be part of the vLLM supported models list found here 🔗 .

Usage

Section titled “Usage”
flexai inference serve <inference_endpoint_name> [
    --accels <number_of_accelerators>
    --accel-sm-slices <number_of_slices>
    --affinity <key1=value1,key2=value2,...>
    --api-key-secret <flexai_secret_name>
    --checkpoint <checkpoint_name_or_uuid>
    --device-arch <device_architecture>
    --hf-token-secret <flexai_secret_name>
    --max-replicas <max_replicas>
    --min-replicas <min_replicas>
    --no-queuing
    --runtime <runtime_name>
  ] (-- --model=<model_name> [<VLLM_Arguments...>])

Arguments

Section titled “Arguments”
inference_endpoint_name
<string>
Required
Resource Name

The name of the Inference Endpoint to create.

It must be unique within the organization and must follow the Resource Naming Conventions.

Examples
  • dev-llama-endpoint
  • prod-mixtral-endpoint
  • vision-gemma
model_name
<string>
Required
vllm_args
<option_list>
Optional
Flag

vLLM Engine Arguments 🔗 that can be passed after the End-of-options marker (—).

Note: The —device argument is not supported: FlexAI handles the device selection tasks.

Examples
  • --task
  • --enable-lora
  • --seed

Flags

Section titled “Flags”
-s , --accel-sm-slices
<integer>
Optional
Integer

Number of slices to divide each SM into.

-a , --accels
<integer>
Optional
Default Value: 1
Integer

Number of accelerators/GPUs to use.

Examples
  • --accels 2
  • --accels 5
  • --accels 8
--affinity
<key=value>
Optional
Default Value: []
-K , --api-key-secret
<string>
Optional
Resource Name

The name of a FlexAI Secret containing the API key you want to set to protect the Inference Endpoint.

If not provided:

  • A FlexAI Secret will be created. Its value will be prompted ─ only once ─ after the creation of the Inference Endpoint.
  • A new secret with the name <inference_endpoint_name>-api-key containing the auto-generated API key will be created.
Examples
  • --api-key-secret api-key-diego-test
  • --api-key-secret prod-llama-endpoint-key
-C , --checkpoint
<string>
Optional
Resource Name

A Checkpoint to serve the Inference Endpoint from.

The name of a previously pushed Checkpoint. Use flexai checkpoint list to see available Checkpoints.

Examples
  • --checkpoint Mixtral-8x7B-v0_1
  • --checkpoint gemma-3n-E4B-it
-d , --device-arch
<string>
Optional
Default Value: nvidia
Option list

The architecture of the device to run the Inference Endpoint on.

One of:

  • nvidia
  • amd
  • tt
Examples
  • --device-arch nvidia
-h , --help
<boolean>
Optional
Flag

Displays this help page.

Examples
  • --help
  • -h
-T , --hf-token-secret
<boolean>
Optional
Flag

The name of the FlexAI Secret containing the Hugging Face token that will be used to access the model.

How to create a Hugging Face token

To create a Hugging Face Access Token, follow these steps:

  1. Go to your Hugging Face account settings. Confirm your identity if requested by entering your credentials.
  2. Select the "+ Create new token" button.
  3. Set the "Token type" to Read.
  4. Enter a name for your token in the "Token name" field.
  5. Select the "Create token" button.
  6. A "Save your Access Token" prompt will appear. Copy the generated token.

We recommend you keep the "Save your Access Token" prompt open until you've securely stored your token as a FlexAI Secret, as you won't be able to view it again.

Examples
  • --hf-token-secret diego_hf_token
  • --hf-token-secret prod-hf-token
--max-replicas
<integer>
Optional
Integer

The maximum number of replicas to use for the Inference Endpoint.

Visit the FlexAI Inference Autoscaling page to learn more about how autoscaling works and how to configure it.

Examples
  • --max-replicas 1
  • --max-replicas 10
  • --max-replicas 5
--min-replicas
<integer>
Optional
Integer

The minimum number of replicas to use for the Inference Endpoint.

Visit the FlexAI Inference Autoscaling page to learn more about how autoscaling works and how to configure it.

Examples
  • --min-replicas 0
  • --min-replicas 5
--no-queuing
<boolean>
Optional
Flag

Disable queuing for the Inference Endpoint.

This means that if there are not enough resources available in the cluster, the request will be rejected immediately instead of being queued.

-r , --runtime
<string>
Optional
Option list

The name of the runtime to use for the Inference Endpoint.

If not provided, the default runtime set for the organization will be used.

-v , --verbose
<boolean>
Optional
Flag

Provides more detailed output when initiating an Inference Serving operation.

Examples
  • --verbose

Examples

Section titled “Examples”

Gated Models

Section titled “Gated Models”

Keep in mind that some models are “Gated”, meaning that you need to go through a process of agreeing to their license agreement, privacy policy, or similar before you can use them.

You can visit the model’s page on the Hugging Face Hub to see if it is marked as “Gated”. Gated models can be identified by this symbol: Hugging Face's 'Gated model' indicator symbol .

If the model is “Gated”, you will find the necessary information on how to proceed.

Example:

Mistral-7B-v0.1 model page on Hugging Face Hub: Before being granted access Mistral-7B-v0.1 model page on Hugging Face Hub: Before being granted access

If you have already gone through the process, you will find a badge on the model’s page indicating that you have access to the model. Example:

Mistral-7B-v0.1 model page on Hugging Face Hub: After being granted access Mistral-7B-v0.1 model page on Hugging Face Hub: After being granted access

Learn more about deploying an Inference Endpoint from a Private or Gated model in the Creating an Inference Endpoint: Private Model quickstart guide.