Skip to main content
The FlexAI Remote Storage Connection Manager enables seamless integration with cloud storage providers, facilitating efficient data transfer and access without the need for local downloads and re-uploads. Create secure connections to your cloud storage and move data directly between their servers and the FlexAI platform.

​
Overview

The Remote Storage Connection Manager enables you to:
  • Connect to cloud storage providers like AWS S3, Google Cloud Storage, and Cloudflare R2
  • Transfer data efficiently with server-to-server connections
  • Manage credentials securely using the FlexAI Secret Manager
  • Upload datasets directly from a third party Cloud Storage Provider to FlexAI
  • Export checkpoints To your preferred storage destinations

​
Supported Storage Providers

Amazon S3

Native support for AWS S3 buckets with regional endpoint configuration

Google Cloud Storage

Direct integration with GCS buckets using Service Account authentication

Cloudflare R2

S3-compatible interface for Cloudflare R2 storage

Hugging Face Hub

Direct access to datasets and models from Hugging Face repositories

MinIO

Self-hosted S3-compatible storage for on-premises deployments

​
Common Data Transfer Operations

​
Dataset Upload Workflows

Efficiently move Datasets from a Cloud Storage Provider to FlexAI:
  1. Direct transfer - Server-to-server transfer without local downloads
  2. Resource optimization - FlexAI handles these data transfers without using compute resources
  3. Dataset structure preservation - Your file and directory structure is kept intact

​
Checkpoint Export Workflows

Upload model Checkpoints to a Cloud Storage Provider:
  1. Flexible destinations - Export to any* connected storage provider
  2. Custom paths - Specify exact destination locations
  3. Metadata preservation - Maintain checkpoint metadata and versioning
Note: Exporting to the Hugging Face Hub is not currently supported.

​
Model Checkpoint Import Workflows

Upload model Checkpoints from a Cloud Storage Provider:
  1. Flexible sources - Import base models or custom checkpoints of your Remote Storage Providers, like the Hugging Face Hub
  2. Metadata recognition - FlexAI recognizes checkpoint metadata and versioning
  3. Resume Training - Continue Training or Fine-tuning your imported Checkpoints
  4. Serve Models - Deploy Inference Endpoints from imported Checkpoints

​
Cost Optimization

Minimize data transfer costs with smart routing:
  • Regional optimization - Use storage regions close to FlexAI infrastructure
  • Direct transfers - Avoid egress fees from intermediate downloads
  • Efficient data transfers - Avoid fetching Datasets during runtime

​
Compliance

  • Data residency - Choose storage regions that meet compliance requirements
  • Encryption at rest - Leverage cloud provider encryption capabilities
  • Access logging - Maintain detailed logs for compliance auditing
  • Data lifecycle - Implement appropriate data retention and deletion policies

​
Getting Started

Ready to connect your cloud storage? Explore these resources:

Storage Commands

Complete CLI reference for storage connections

Dataset Upload

Upload datasets from cloud storage

Checkpoint Export

Export checkpoints to cloud storage

Secret Management

Manage storage credentials securely