Storage Commands
Complete CLI reference for storage connections
Overview
The FlexAI Remote Storage Connection Manager enables seamless integration with cloud storage providers, facilitating efficient data transfer and access without the need for local downloads and re-uploads.
Create secure connections to your cloud storage and move data directly between their servers and the FlexAI platform.
Overview
Section titled “Overview”The Remote Storage Connection Manager enables you to:
- Connect to cloud storage providers like AWS S3, Google Cloud Storage, and Cloudflare R2
- Transfer data efficiently with server-to-server connections
- Manage credentials securely using the FlexAI Secret Manager
- Upload datasets directly from a third party Cloud Storage Provider to FlexAI
- Export checkpoints To your preferred storage destinations
Supported Storage Providers
Section titled “Supported Storage Providers”Native support for AWS S3 buckets with regional endpoint configuration
Direct integration with GCS buckets using Service Account authentication
S3-compatible interface for Cloudflare R2 storage
Direct access to datasets and models from Hugging Face repositories
Self-hosted S3-compatible storage for on-premises deployments
Common Data Transfer Operations
Section titled “Common Data Transfer Operations”Dataset Upload Workflows
Section titled “Dataset Upload Workflows”Efficiently move Datasets from a Cloud Storage Provider to FlexAI:
- Direct transfer - Server-to-server transfer without local downloads
- Resource optimization - FlexAI handles these data transfers without using compute resources
- Dataset structure preservation - Your file and directory structure is kept intact
Checkpoint Export Workflows
Section titled “Checkpoint Export Workflows”Upload model Checkpoints to a Cloud Storage Provider:
- Flexible destinations - Export to any* connected storage provider
- Custom paths - Specify exact destination locations
- Metadata preservation - Maintain checkpoint metadata and versioning
Note: Exporting to the Hugging Face Hub is not currently supported.
Model Checkpoint Import Workflows
Section titled “Model Checkpoint Import Workflows”Upload model Checkpoints from a Cloud Storage Provider:
- Flexible sources - Import base models or custom checkpoints of your Remote Storage Providers, like the Hugging Face Hub
- Metadata recognition - FlexAI recognizes checkpoint metadata and versioning
- Resume Training - Continue Training or Fine-tuning your imported Checkpoints
- Serve Models - Deploy Inference Endpoints from imported Checkpoints
Cost Optimization
Section titled “Cost Optimization”Minimize data transfer costs with smart routing:
- Regional optimization - Use storage regions close to FlexAI infrastructure
- Direct transfers - Avoid egress fees from intermediate downloads
- Efficient data transfers - Avoid fetching Datasets during runtime
Compliance
Section titled “Compliance”- Data residency - Choose storage regions that meet compliance requirements
- Encryption at rest - Leverage cloud provider encryption capabilities
- Access logging - Maintain detailed logs for compliance auditing
- Data lifecycle - Implement appropriate data retention and deletion policies
Getting Started
Section titled “Getting Started”Ready to connect your cloud storage? Explore these resources:
Dataset Upload
Upload datasets from cloud storage
Checkpoint Export
Export checkpoints to cloud storage
Secret Management
Manage storage credentials securely