Creating a Training Job

The Model’s repository

For this tutorial we will use the FlexAI fork of the nanoGPT repository originally created by Andrej Karpathy: https://github.com/flexaihq/nanogpt 🔗.

nanoGPT’s entry point script path is train.py, and it expects a path to a configuration file as well as a few Hyperparameters to run the Training Job.

1. Log into https://console.flex.ai 🔗 using your FlexAI account credentials.

2. Navigate to the Training section from either the navigation bar or the card on the home page.

3. A drawer menu with the creation form will appear automatically. You can also select the New button to display the creation form.

The Start a new training job form

The Start a new training job form consists of a set of required and optional fields that you can use to customize your deployment.

Required Fields

• Name: Your Training Job name. Should follow the resource naming conventions.
• Repository URL: The URL of the Git repository containing your training code.`.
• Entry Point: The path to the entry point script in your repository that will initiate the Training Job.
  • The entry point script can be followed by any arguments you want to pass to it, such as configurations and Hyperparameters. Value: train.py config/train_shakespeare_char.py --dataset_dir=my_dataset --out_dir=/output-checkpoint --max_iters=1500.

Other fields

• Repository Revision: The Git revision (branch, tag, or commit) you want to use for this Training Job. The main branch will be used by default.

• Node Count: The number of nodes you want to use for this Training Job. Defaults to 1.

  • This will determine the amount of Accelerators you will have available for your Training Job:
    • 1 node will allow you to use up to 8 Accelerators.
    • Using more than 1 node will make all 8 Accelerators per Node available to your Training Job.

• Accelerator Count: The number of Accelerators you want to use for this Training Job. Must follow the logic described above. Defaults to 1.

• Datasets: Can be selected from a dropdown list of the datasets you want to use for this Training Job. You can add multiple datasets as well as specify the mount path within the Training Runtime (they will be mounted under /input). You can read more about this in the Uploading Datasets guide.

  Note

  Don’t forget to select the “Add” button after picking a Dataset, otherwise it won’t be added to the Training Job.

• Environment Variables & Secrets: Add any environment variables you want to set for this Training Job. These will be available to your training code as environment variables within the Training Runtime.

  • You can also reference Secrets, which will be securely injected into the Training Job’s Runtime.

• Cluster: The cluster where the Training workload will run on. It can be selected from a dropdown list of available clusters in your FlexAI account. A default cluster will be automatically selected for you if none is specified.

Form Values

Field Name	Value
Name	nanoGPT-flexai-console
Repository URL	https://github.com/flexaihq/nanogpt
Repository Revision	main
Node Count	1
Accelerator Count	1
Entry Point	train.py config/train_shakespeare_char.py --dataset_dir=my_dataset --out_dir=/output-checkpoint --max_iters=1500
Datasets	Dataset: nanoGPT-dataset (from the CLI quickstart), Mount Directory: my_dataset
Cluster	Your organization’s designated cluster

Entry Point script arguments

The entry point script for this Training Job is train.py, and it expects the following arguments:

• config/train_shakespeare_char.py:A configuration file, which contains the default Training Parameters.
• --dataset_dir: The path within the /input directory of the Training Runtime where the Dataset files are located.
• --out_dir: The output directory, which will be mounted into the Training Runtime as /output-checkpoint.
• --max_iters: The maximum number of iterations to run the training script for (optional).

Entry Point script arguments details

These include any Environment Settings and Hyperparameters the training script may require. For this tutorial:

Parameter	Type	Description
config/train_shakespeare_char.py	Environment Setting	A positional argument pointing to a configuration file used by nanoGPT’s train.py script to set the default Training Parameters
--out_dir=/output-checkpoint	Environment Setting	The output directory where the training script will write checkpoint files. In order to take advantage of FlexAI’s Managed Checkpoints feature, this should always be /output-checkpoint
--max_iters=1500	Hyperparameter	The maximum number of iterations to run the training script for. This is an optional hyperparameter that can be used to tweak the Training Job execution

Starting the Training Job

After filling out the form, select the Submit button to start the Training Job. You should get a confirmation message indicating that the Training Job creation process has been initiated successfully.

The Start a new training job form will close and you will be redirected to the Training Jobs list page, where you can see your newly created Training Job in the list.

Next Steps

Now that you have created a Training Job, you can monitor its progress. The next step of this Quickstart Tutorial will guide you through the process of monitoring your Training Job’s progress.