The name of the Dataset you want to create. Dataset names must be unique within a given FlexAI organization and must follow the Resource Naming conventions.
<string>
dataset push
Pushes a new dataset to a FlexAI. Multiple files can be uploaded at once by using multiple instances of the —file flag, or by pointing the —file flag to a directory containing dataset files.
The Datasets Manager Service page provides more information on how to upload (or push) both local files and files from a Remote Storage Provider.
flexai dataset push <dataset_name> [(--file <path_on_filesystem>=<fcs_dataset_path> ...) | (--storage-provider <storage_provider_name> --source-path <source_path>)] [flags]Arguments
Section titled “Arguments”- dataset_name
Required
- -f , --file
-
The local path to the dataset files on the host machine running the FlexAI CLI.
The Dataset will be mounted and made available to your code under the
/input/<dataset_name>path. -
--file /home/diego/datasets/successful-ai-scaleups.csv -
--file ~/datasets/flexai.txt -
A mapping between a local path on the host machine running the FlexAI CLI and the path where the dataset file will be stored in the Dataset.
If no target nested path is provided, the file will be stored at the root of the Dataset.
-
/input/<dataset_name>/<file_name>
If a target directory path is provided, the file will be stored under that path in the Dataset.
-
/input/<dataset_name>/<target_nested_path>/<file_name>
If the source path is a directory, the entire directory will be uploaded recursively to the Dataset. The directory original file structure will be preserved.
-
/input/<dataset_name>/<source_directory_name>/<file_name>
If a target directory path is provided, the source directory will be uploaded under that path in the Dataset.
-
/input/<dataset_name>/<target_nested_path>/<source_directory_name>/<file_name>
Examples:
-
—file /home/diego/datasets/successful-ai-scaleups.csv=eval/csv/ -
—file ~/datasets/flexai.txt=train/text/flexai.txt
To learn more about Dataset Organization and their File Structure, visit the Datasets Manager Service page.
-
-
--file /home/diego/datasets/successful-ai-scaleups.csv=train/csv/ -
--file ~/datasets/flexai.txt=train/text/flexai.txt -
--file ~/all-the-stuff/ -
--file /home/diego/datasets/=train/all-files/ - -h , --help
-
Displays this help page.
- --source-path
-
The path to the bucket and file or directory to be pushed.
Syntax: <bucket_name>/<path>.
Used in together with the
—storage-providerflag. -
--source-path train/img/128/dataset-1 -
--source-path test/eval/text - --storage-provider
-
The name of the Remote Storage Provider Connection to use for the Dataset upload.
You can list the available Remote Storage Provider Connections using the
flexai storage listcommand. -
--storage-provider s3-connection-dev -
--storage-provider gcs-bucket-prod -
--storage-provider test-diego-minio - -v , --verbose
-
Provides more detailed output when pushing a Dataset.
<string>
Optional
Path
Examples
Key Value Path Mapping
Examples
<boolean>
Optional
Flag
<string>
Required
Path
Examples
<string>
Optional
Examples
<boolean>
Optional
Flag
Examples
Section titled “Examples”Push local files to a dataset
Section titled “Push local files to a dataset”flexai dataset push my-dataset —file /path/to/file1.csv —file /path/to/file2.txt
flexai dataset push my-dataset --file /path/to/file1.csv --file /path/to/file2.txtPush files with custom paths
Section titled “Push files with custom paths”flexai dataset push my-dataset —file /path/to/file1.csv=train/data/file1.csv —file /path/to/file2.txt=eval/data/file2.txt
flexai dataset push my-dataset --file /path/to/file1.csv=train/data/file1.csv --file /path/to/file2.txt=eval/data/file2.txtPush from remote storage provider
Section titled “Push from remote storage provider”flexai dataset push my-dataset —storage-provider my-s3-provider —source-path bucket-name/path/to/data/
flexai dataset push my-dataset --storage-provider my-s3-provider --source-path bucket-name/path/to/data/