> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flex.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Interactive Training Session

> Start an interactive SSH session for real-time training development and debugging

The `flexai training debug-ssh` command allows you to start an Interactive Training Job on a Training Runtime similar to that of regular Training Jobs. This command will allocate the required resources and then set up an Interactive Training Runtime you can connect to via SSH.

Having access to this *debug* Interactive Training Runtime will allow you to iterate quickly by enabling you to edit scripts, modify files, review logs and outputs, push your changes to your GitHub repository, and in general, have an inside out look at your Training results as they happen, before you commit to running a Training Job.

## Pre-requisites

The environment in which the `flexai training debug-ssh` command is run should either:

* Have an `ssh-agent` running and SSH keys loaded into it, or
* Have an SSH key pair available

### Using an ssh-agent

If an `ssh-agent` is not running and you would like to use one to be able to push your change from the Interactive Training Runtime to GitHub, run `eval $(ssh-agent)` in your terminal, and load your keys to it by running `ssh-add <path_to_private_key>`.  You can then confirm the keys have been loaded using `ssh-add -l` and start your Interactive Training Job - it will automatically use your `ssh-agent`.

If SSH key pairs authenticating you to GitHub are loaded into your `ssh-agent`, you will be able to push your changes to GitHub from the Interactive Training Runtime by enabling the `ForwardAgent` option in your SSH configuration file (`~/.ssh/config`):

```text title="~/.ssh/config" theme={null}
Host debug-gw.flex.ai
  ForwardAgent yes
```

To verify if an `ssh-agent` is running in your local environment and has your private keys loaded, run:

```bash theme={null}
ssh-add -L
```

You can load more keys into your `ssh-agent` with:

```bash theme={null}
ssh-add path/to/private/key
```

<Note title="SSH_AUTH_SOCK and IdentityAgent">
  `ssh-add` relies on the environment variable `SSH_AUTH_SOCK` being correctly configured. If you are overriding it through the `IdentityAgent` ssh config option in `.ssh/config`, make sure to set the `SSH_AUTH_SOCK` environment variable to the same socket path when running the aforementioned `ssh-add` commands. Failing to do, `ssh-add` will load keys into the wrong `ssh-agent`.
</Note>

### Using SSH key pairs

When not using the `ssh-agent`, you *must* set the path to the public key you will use through the `--authorized-keys` flag, e.g.:

```bash theme={null}
flexai training debug-ssh \
  --repository-url https://github.com/flexaihq/nanoGPT \
  --authorized-keys ~/.ssh/id_ed25519.pub \
  --vscode
```

<Tip title="Non-standard SSH key paths">
  If the SSH key pair is located in a non-standard path (i.e. not in `~/.ssh/` with names like `id_rsa`, `id_ecdsa`, `id_ed25519`, etc.), you will need to pass an extra `-i <path_to_corresponding_private_key>` flag to the `ssh` command when connecting to the training environment.
</Tip>

## Starting an Interactive Training Job

In its simplest form, starting an Interactive Training Job will only require running the `debug-ssh` subcommand with the `--repository-url` flag, e.g.:

```bash title="Starting an Interactive Training Job" theme={null}
flexai training debug-ssh \
  --repository-url https://github.com/flexaihq/nanoGPT \
  --vscode
```

Information on the various stages of the Interactive Training Job will be displayed in the terminal. Once the job is ready, you will be provided with the SSH command to connect to the Interactive Training Runtime, example output:

```text title="Output: Starting an Interactive Training Job" theme={null}
flexai training debug-ssh \
  --repository-url https://github.com/flexaihq/nanoGPT \
  --vscode
Interactive training interactive-training-61a01afb-66bc-411c-a23b-d9fc566c1f9c launching...
✅ Looking for an interactive training builder
✅ Starting interactive training environment

[Node 0] To connect using ssh:
    ssh -o ForwardAgent=yes -p 36663 flexai@debug-gw.flex.ai
[Node 0] To open VSCode, click the following URL (or open in your browser if not clickable):
    "vscode://vscode-remote/ssh-remote+flexai@debug-gw.flex.ai:36663/workspace?windowId=_blank"
✅ Automatically configuring ~/.ssh/known_hosts
```

If the --`vscode` flag was used, Visual Studio Code will automatically open.

## Attaching through Visual Studio Code

You can also attach to the Interactive Training Runtime using VSCode. To do so, you will need to install the [Remote - SSH](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh) extension.

Once installed, you can connect to the Interactive Training Runtime by simply clicking on the VSCode URL provided in the terminal output. This will open a new VSCode window with the SSH connection to the Interactive Training Runtime already established.

The `--vscode` flag can be used to automatically open a VSCode window into the Interactive Training Runtime, useful if the VSCode URL is not clickable in your terminal.

## Lifetime of an Interactive Training Job

### Session timeout

Interactive Training Jobs are automatically stopped after the specified session timeout (can be set with `--session-timeout`. Defaults to 600 seconds, which is 10 minutes) if there are no active SSH session into the Interactive Training Runtime.

### Stopping a running Interactive Training Job

Interactive Training Jobs can be manually stopped like regular Training Jobs by using the `flexai training stop` command:

```bash theme={null}
flexai training stop <interactive_training_job_name>

```

## Troubleshooting

### The Interactive Training Job fails to start

You can check the Interactive Training session logs, just like a regular Training Job:

```bash theme={null}
flexai training logs <training_job_name>
```

The same applies to *inspecting* a Training Session to get more information on its [Lifecycle](/cli/reference/training/#training-lifecycle-statuses):

```bash theme={null}
flexai training inspect <training_job_name>
```

### SSH connection issues

#### When using an `ssh-agent`

Make sure that the `ssh-agent` is running and has the correct keys loaded. You can verify this by running `ssh-add -L` and checking that the keys you expect are listed.

A common issue is that the `ssh-agent` in your environment is not the same as the one used by the Interactive Training Job. This can happen if you have multiple `ssh-agent` instances running and the `SSH_AUTH_SOCK` environment variable points to a different agent than the one configured inside your ssh configuration file (`~/.ssh/config`).  When loading keys into your agent with `ssh-add /path/to/your-key` or verifying which keys are loaded with `ssh-add -L`, make sure that the `SSH_AUTH_SOCK` environment variable is set to the same socket path as the one used by the Interactive Training Job, which you can find by running:

```bash theme={null}
ssh -G debug-gw.flex.ai | awk '/^identityagent/ {print $2}'
```

### Visual Studio Code connection issues

If Visual Studio Code fails to open automatically:

* Make sure you have the [Remote - SSH](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh) extension installed.
* Copy the `vscode://` URL provided in the terminal output and:
  * In a Visual Studio instance open the command palette (You can use `Ctrl+Shift+P` / `Cmd+Shift+P`), then paste the `vscode://` URL.
  * Open a terminal window, then use the `open` or `xdg-open` command (depends on your system) followed by the `vscode://` URL you just copied:
    ```bash theme={null}
    open vscode://vscode-remote/ssh-remote+flexai@debug-gw.flex.ai:36663/workspace?windowId=_blank
    ```
    ```bash theme={null}
    xdg-open vscode://vscode-remote/ssh-remote+flexai@debug-gw.flex.ai:36663/workspace?windowId=_blank
    ```

Note that in order to establish the connection, you may be prompted to unlock your SSH key if it is passphrase-protected.