How AI models are trained

Evolution · 2.2

From data to a trained model on one page, then the five tools a training run uses, one at a time.

AI model training: top-down view

From data to a trained model, and the tools and infrastructure that make it work.

Training workflow

1Prepare dataCollect, clean and prepare the examples (e.g. documents, network logs, incident tickets).
2Train the modelRun a training script, written in Python with PyTorch.
3Use GPUsPyTorch uses NVIDIA CUDA to run the calculations on many GPUs in parallel.
4Learn & improvePredict, measure the error, adjust the weights, and repeat many times.
5Trained modelReady to use for inference, e.g. by an AI agent or an application.

Tool stack(software)

Infrastructure(hardware)

GPU servers
(e.g. Dell PowerEdge, HPE)
NVIDIA GPUs
(e.g. H100 / B200)
High-speed networking
(InfiniBand / Ethernet)
High-performance storage
(for large training datasets)

Key takeaway

Training uses data, specialised software (e.g. PyTorch + CUDA) and powerful GPU infrastructure to teach an AI model, improve it over time and create a trained model for real use.

How the tools are used

One training run, five tools

Five tools, five jobs. Python describes the training, Slurm finds it GPUs, PyTorch does the learning, CUDA runs it on the GPUs, and MLflow keeps the record.

Step 1 of 5

Python

Programming language

Before anything runs

An engineer writes the training script in Python: which model, which data, and settings such as how many passes over the data to make.

Example

A 200-line script that loads two years of resolved incident tickets and a pretrained model to fine-tune.

What it does not do

Python does not do the heavy maths. It tells libraries (PyTorch, CUDA) what to do, and they do it.

Step 2 of 5

Slurm

Job scheduler

When the job is submitted

The script is submitted to the GPU cluster as a job. Slurm queues it, waits for the GPUs it asked for, then starts it on those servers.

Example

“Give this job 8 GPUs on one server for up to 12 hours.” Other teams’ jobs wait their turn or run on other servers.

What it does not do

Slurm schedules; it does not train. Once the job starts, none of the calculations pass through it.

Step 3 of 5

PyTorch

Machine-learning framework

While training runs

PyTorch runs the training loop: the model makes predictions, the error is measured, PyTorch works out how each weight should change and updates it. Batch after batch.

Example

The model guesses “power supply fault” for an alarm pattern the ticket says was a fibre cut; PyTorch nudges the weights so the next guess is better.

What it does not do

PyTorch does not talk to the GPU hardware itself. On NVIDIA GPUs it hands each calculation to CUDA.

Step 4 of 5

NVIDIA CUDA

GPU computing platform

Inside every training step

CUDA (Compute Unified Device Architecture) and its libraries execute PyTorch’s calculations on the GPUs, thousands of them at once, and pass results between GPUs.

Example

One training step is millions of multiply-and-add operations. CUDA spreads them across the GPU’s cores so they finish together.

What it does not do

CUDA does not decide what the model learns, and it is NVIDIA-only: AMD GPUs use ROCm, Google’s TPUs (Tensor Processing Units) use XLA.

Step 5 of 5

MLflow

Experiment and model tracking

During and after the run

MLflow records each run’s settings, its results over time and the saved model files, so runs can be compared and the best model found, versioned and reproduced.

Example

Run 14 scored 91% on held-back tickets, run 15 only 88%: keep run 14’s model, version 1.3, and deploy it.

What it does not do

MLflow records; it does not train, schedule or run anything. It is a logbook and a model registry.