Command: storage
The flexai storage command allows for the management of connections with Remote Storage Providers.
Remote Storage Provider connections allow for the transfer of datasets between a Remote Storage Provider and FCS Data Store, removing the need to manually download datasets to a host to then push them to FCS.
storage create
Creates a new Remote Storage Provider connection.
flexai storage create <storage_provider_connection_name> [( --provider --access-key-id --endpoint --secret-access-key-name )] [( --provider gcs --service-account-file-name )] [( --provider s3 --region <region> )]
Arguments
| Argument | Description | Example |
|---|---|---|
storage_provider_connection_name | Resource name. Must follow the FCS resource naming conventions | aws-storage-conn-eu |
Flags
| Flag | Type | Optional / Required | Definition | Example |
|---|---|---|---|---|
--access-key-id | String | Optional | S3-compatible storage Access Key ID | AKIAIOSFODNN7AAF89GU |
--endpoint | String | Optional | S3-compatible storage endpoint, e.g. "s3.eu-west-1.amazonaws.com", "s3.us-east-2.amazonaws.com" | s3.eu-west-1.amazonaws.com |
-p, --provider | String | Required | Storage Provider (gcs|s3|minio|r2) | s3 |
--region | String | Optional | S3-compatible storage Region where the target Storage Bucket is located, e.g. "eu-west-1", "us-east-2". Only required for the s3 provider | eu-west-1 |
--secret-access-key-name | String | Optional | Name of the 'FCS Secret' containing the S3-compatible storage Secret Access Key (visit secret create for more information on how to create a Secret) | aws-secret-accss-ky-eu |
--service-account-file-name | String | Optional | Name of the 'FCS Secret' containing the GCP Service Account JSON file created using the --value-stdin flag (visit secret create for more information) | gcp-sa-dev |
Storage Providers
Currently available Storage Providers are:
- Google Cloud Storage:
gcs - AWS S3:
s3 - MinIO:
minio - Cloudflare R2:
r2
Authentication details for the Storage Providers are required to establish a connection. Sensitive information such as Secret Access Keys need to be stored in the FCS Secret manager.
gcs: https://cloud.google.com/iam/docs/service-account-overviews3: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.htmlminio: https://min.io/docs/minio/container/administration/identity-access-management/minio-user-management.html#minio-id-access-keysr2: https://developers.cloudflare.com/r2/api/s3/tokens/
Examples
A connection to a Google Cloud Storage Resource
Since a connection to Google Cloud Storage (GCS) requires a Service Account JSON file that is stored in the FCS Secret Manager, no additional details are required.
The GCP Service Account JSON file can be stored into the FCS Secret manager by running the following command:
cat <path_to_service_account_json_file> | flexai secret create gcp-sa-dev -p
For instance:
cat ~/Downloads/my-project-name-4273e0-fad0b650e9ad.json | flexai secret create gcp-sa-dev -p
Creating the Remote Storage Provider connection to GCS only requires passing the connection name, the provider name, and the name of the Secret created above:
flexai storage create gcs-conn-dev \
--provider gcs \
--service-account-file-name gcp-sa-dev
A connection to an AWS S3 Storage Resource
This example Storage Resource has the following properties:
- Endpoint: s3.eu-west-1.amazonaws.com
- Region: eu-west-1
- Access Key ID: AKIAIOSFODNN7AAF89GU
- Secret Access Key: wJalrXUtnFEMI/K7MRENG/onXpfiCziZYzc6dpiE
The Secret Access Key needs to first be stored in the FCS Secret manager. For this example, it will be stored using the Secret name "aws-secret-accss-ky-eu" (flexai secret create aws-secret-accss-ky-eu).
flexai storage create aws-storage-conn-eu \
--provider s3 \
--endpoint s3.eu-west-1.amazonaws.com \
--region eu-west-1 \
--access-key-id AKIAIOSFODNN7AAF89GU \
--secret-access-key-name aws-secret-accss-ky-eu
A connection to a Cloudflare R2 Storage Resource
This example Storage Resource has the following properties:
- Endpoint: a9c5d8b45e9f1c2e4b8a3d7f6e0a9b1c.r2.cloudflarestorage.com
- Access Key ID: f4e7b3a6d1c9e8f5a2b0d3c4e6f1a8b7
- Secret Access Key: b1e2a3c4d5f6e7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2
This example will assume the Secret Access Key has been stored under the name "r2-sak-dev" as an FCS Secret Manager's Secret (flexai secret create r2-sak-dev).
flexai storage create r2-storage-conn \
--provider r2 \
--endpoint a9c5d8b45e9f1c2e4b8a3d7f6e0a9b1c.r2.cloudflarestorage.com \
--access-key-id f4e7b3a6d1c9e8f5a2b0d3c4e6f1a8b7 \
--secret-access-key-name r2-sak-dev
storage inspect
Displays detailed information about a Remote Storage Provider connection.
flexai storage inspect <storage_provider_connection_name>
Arguments
| Argument | Description | Example |
|---|---|---|
storage_provider_connection_name | Resource name. Must follow the FCS resource naming conventions | aws-storage-conn-eu |
Flags
| Flag | Type | Optional / Required | Description | Example |
|---|---|---|---|---|
--json | Flag | Optional | Output the information in JSON format | --json |
Example
flexai storage inspect aws-storage-conn-eu
Details on the returned information
Returned information
| Field | Description | Data Type |
|---|---|---|
kind | Resource type | String |
metadata | Metadata information | Object |
metadata.name | Resource name | String |
metadata.id | Resource ID | String (UUID) |
metadata.creatorUserID | User ID of the creator of the resource | String (UUID) |
metadata.ownerOrgID | Organization ID of the owner of the resource | String (UUID) |
spec | Specification information | Object |
spec.providerName | Storage Provider name | String |
spec.gcsConfig | Google Cloud Storage configuration information | Object |
spec.gcsConfig.serviceAccountRef | Name of the FCS Secret containing the GCP Service Account JSON file | String |
spec.s3Config | S3-compatible configuration information | Object |
spec.s3Config.endpoint | S3-compatible storage endpoint | URL |
spec.s3Config.accessKeyId | S3-compatible storage Access Key ID | String |
spec.s3Config.region | S3-compatible storage Region | String |
status | Status information | Object |
status.createdAt | Date and time the resource was created | String (ISO 8601) |
Example
kind: StorageProvider
metadata:
name: aws-storage-conn-eu
id: 42235d4b-5832-45ec-af5f-4d1d4773860e
creatorUserID: bd67af19-2599-4d57-832e-a1ac042f48be
ownerOrgID: 270a5476-b91a-412f-8d13-852ef7bb5b94
spec:
providerName: s3
s3Config:
endpoint: s3.eu-west-2.amazonaws.com
accessKeyId: AKIAIOSFODIN7AAF89GU
region: eu-west-2
status:
createdAt: "2024-12-16T17:58:05.545998Z"
storage list
Lists all Remote Storage Provider connections.
flexai storage list
Example
flexai storage list
NAME | PROVIDER | AGE
-----------------------------+----------+------
gcs-conn-dev | GCS | 42d
r2-storage-conn | R2 | 33d
aws-storage-conn-eu | S3 | 6h