Skip to main content

​
Uploading files

Navigate to the Dataset Manager in the FlexAI Console:
1

Visit the Add Dataset section

Visit the “Add Dataset” section of the FlexAI Console
2

Enter a name for your dataset

Enter a name for your dataset: nanoGPT-dataset
3

Select the Local option

Select the “Local” option for “Upload Origin”
4

Select the Upload Item button

Select the + Upload Item button to open the “Upload Items” dialog

​
One file at a time

Uploading files individually is useful when you need to specify source files that may have different locations on your source machine and/or you want to set a specific destination path for each of them:Let’s assume the following file structure on your local machine:
~/
  openwebtext_mini/
    - urlsf_subset00.tar
    - urlsf_subset01.tar
    - test.tar
  sf-wikitext/
    - test-00000-of-00001.parquet
    - train-00000-of-00001.parquet
You can upload specific files from the openwebtext_mini and sf-wikitext directories to a FlexAI Dataset named text-records-dataset-1, while also specifying a custom destination path for each of them—including a different file name for each—, as seen below:
text-records-dataset-1/
  owt/
    - urlsf_subset00.tar
    - urlsf_subset01.tar
  test/
    - test.tar
    - test-00000-of-00001.parquet
  wikitext/
    - train-00000-of-00001.parquet
You can achieve this by going through the following steps iteratively for each file:
1

Select the file

Use the “Select file” option to open a file browser dialog
2

Choose the file

Select the file you want to upload from your local machine
3

Enter the destination path

In the “Destination Path” field, enter the desired destination path within the Dataset
4

Add the file

Select the Add button to confirm the file selection and destination mapping
5

Repeat for additional files

Below the file list named “Upload Items” you will find an + Add items button that will open up the “Upload Items” dialog again.
6

Continue adding files

Repeat the steps above for each file you want to upload
7

Complete the upload

Finally, select the Add Dataset button to start the upload process.
Currently, the FlexAI Console does not offer the ability to set a custom destination file name. If you require that feature, please refer to the ‘Using the FlexAI CLI’ instructions instead.

​
Multiple files, no defined destination path

Considering the example above, you could decide to simply upload the files without specifying a destination path. This would result in the files being moved up to the root directory of the FlexAI Dataset:Since none of the files names are the same, they won’t overwrite each other, ending up in a FlexAI Dataset structure that looks as follows:
text-records-dataset-2/
  - urlsf_subset00.tar
  - urlsf_subset01.tar
  - test.tar
  - test-00000-of-00001.parquet
  - train-00000-of-00001.parquet
All intermediate directories are disregarded when uploading files without specifying a destination path.
However, when picking and choosing files to upload is not required for your data workflow, then you can use the third method: a bulk upload of files in a directory.

​
Entire directory contents

The --file flag also allows you to push the contents of a directory into a Dataset. This is particularly useful when you already have a directory containing the multiple files that make up your dataset.Let’s assume the following file structure on your local machine:
~/
  my-dataset/
    train/
      - t_1.txt
      - t_2.txt
      - t_3.txt
      deep-text/
        - t_1.txt
        - t_2.txt
    test/
      - test_1.txt
      - test_2.txt
      deep-text_test
        - t_1.txt
Yes, file names have been deliberately kept similar to show how pushing a entire directory with nested sub-directories is handled (no overwrite risk!).
Uploading the contents of the my-dataset directory to a Dataset named text-records-dataset-3 would follow the same pattern as before:Resulting in the following FlexAI Dataset structure:
text-records-dataset-3/
  train/
    - t_1.txt
    - t_2.txt
    - t_3.txt
    deep-text/
      - t_1.txt
      - t_2.txt
  test/
    - test_1.txt
    - test_2.txt
    deep-text_test/
      - t_1.txt
Notice that the source my-dataset directory is not included in the FlexAI Dataset structure, only its contents are uploaded, the file structure is preserved.

​
Troubleshooting

​
Pushing large files

Some times you may encounter issues when trying to upload large files directly from your computer. This kind of problem is usually related to network issues. There are a few things you can try to solve this issue:
  • Switch to a wired connection if possible.
  • Split the file into smaller parts or chunks that you can then join back together at runtime.
  • If the file is stored in a Cloud Storage Service such as Amazon S3 or Google Cloud Storage, you can upload directly to the FlexAI Dataset Manager by creating a Remote Storage Provider Connection.

​
The process fails when uploading files from a remote machine you’re connected to via SSH

If you are trying to upload files from a machine that you’re connected to via SSH, the process may fail due to the SSH connection being closed. To avoid this, you can use the a terminal multiplexer, like screen or tmux, to keep the process running even after you close the remote session.