storage create

Creates a new Remote Storage Provider Connection.

flexai storage create <storage_provider_connection_name>
  [( --provider --access-key-id --endpoint --secret-access-key-name )]
  [( --provider gcs --service-account-file-name )]
  [( --provider s3 --region <region> )]
  [( --provider huggingface --hf-token-name <hf_token_name> )]

Arguments

storage_provider_connection_name: The name that will be used to identify the Storage Provider Connection while performing operations on it.

Flags

--access-key-id: S3-compatible storage Access Key ID.
--hf-token-name: Name of the FlexAI Secret (See flexai secret) containing the Hugging Face Access token 🔗.
-p , --provider: One of: gcs , s3 , minio , r2 , huggingface.
--endpoint: S3-compatible storage endpoint.
Only required when using the s3 provider.
--region: S3-compatible storage Region where the target Storage Bucket is located.
Only required when using the s3 provider.
--secret-access-key-name: Name of the FlexAI Secret (See flexai secret) containing the S3-compatible storage Secret Access Key.
--service-account-file-name: Name of the FlexAI Secret (See flexai secret) where the contents of the GCP Service Account JSON file have been stored (see flexai secret create --value-stdin).

Connection Details

The authentication details required by each Storage Provider are different in each case and are listed below:

Sensitive information such as secrets, access keys or tokens should be stored in the FlexAI Secret Store using the flexai secret create command.

Examples

A connection to a Google Cloud Storage Resource

Since a connection to Google Cloud Storage (GCS) requires a Service Account JSON file that is stored in the FCS Secret Manager, no additional details are required.

The GCP Service Account JSON file can be stored into the FCS Secret manager by running the following command:

cat <path_to_service_account_json_file> | flexai secret create gcp-sa-dev -p

For instance:

cat ~/Downloads/my-project-name-4273e0-fad0b650e9ad.json | flexai secret create gcp-sa-dev -p

Creating the Remote Storage Provider Connection to GCS only requires passing the connection name, the provider name, and the name of the Secret created above:

flexai storage create gcs-conn-dev \
  --provider gcs \
  --service-account-file-name gcp-sa-dev

A connection to an AWS S3 Storage Resource

This example Storage Resource has the following properties:

Endpoint: s3.eu-west-1.amazonaws.com
Region: eu-west-1
Access Key ID: AKIAIOSFODNN7AAF89GU
Secret Access Key: wJalrXUtnFEMI/K7MRENG/onXpfiCziZYzc6dpiE

The Secret Access Key needs to first be stored in the FCS Secret manager. For this example, it will be stored using the Secret name “aws-secret-accss-ky-eu” (flexai secret create aws-secret-accss-ky-eu).

flexai storage create aws-storage-conn-eu \
  --provider s3 \
  --endpoint s3.eu-west-1.amazonaws.com \
  --region eu-west-1 \
  --access-key-id AKIAIOSFODNN7AAF89GU \
  --secret-access-key-name aws-secret-accss-ky-eu

A connection to a Cloudflare R2 Storage Resource

This example Storage Resource has the following properties:

Endpoint: a9c5d8b45e9f1c2e4b8a3d7f6e0a9b1c.r2.cloudflarestorage.com
Access Key ID: f4e7b3a6d1c9e8f5a2b0d3c4e6f1a8b7
Secret Access Key: b1e2a3c4d5f6e7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2

This example will assume the Secret Access Key has been stored under the name “r2-sak-dev” as an FCS Secret Manager’s Secret (flexai secret create r2-sak-dev).

flexai storage create r2-storage-conn \
 --provider r2 \
 --endpoint a9c5d8b45e9f1c2e4b8a3d7f6e0a9b1c.r2.cloudflarestorage.com \
 --access-key-id f4e7b3a6d1c9e8f5a2b0d3c4e6f1a8b7 \
 --secret-access-key-name r2-sak-dev