Skip to main content
BSS/OSS Academy
← Architecture · back to the map

Cluster software platform

Firmware, drivers, CUDA, NCCL, Kubernetes, Slurm and the operators between them — and why compatibility matrices and upgrade sequencing decide cluster uptime.

Hardware becomes a cluster through software, and the layer where that most often goes wrong is not any individual component — it is the relationships between them. Firmware, drivers, communication libraries, container runtime and scheduler have to be a validated set, not a collection of individually current versions.

The stack

From hardware to workload

  1. Workloads

    Training and inference jobs submitted by users or services.

  2. Scheduler — Kubernetes and/or Slurm

    Places work onto capacity, enforces queues, quotas and tenancy boundaries.

  3. Operators and device plugins

    GPU Operator and Network Operator automate device, driver and network configuration; device plugins advertise accelerators as schedulable resources.

  4. Container runtime

    Packages the job with its dependencies and exposes devices into the container.

  5. CUDA, NCCL and communication libraries

    The programming model, and the collective operations that keep accelerators in step. NCCL is where most cluster-wide performance behaviour actually lives.

  6. Drivers

    Kernel-level interface to accelerators and network adapters.

  7. Firmware

    Accelerators, adapters, switches, BMCs and power components. The layer most often left inconsistent.

  8. Hardware

    Compute trays, fabric, storage.

Read the dependencies sideways, not just downwards
Each layer depends on the one below, which is obvious. The harder property is that several layers must agree with each other across nodes: NCCL on node A negotiates with NCCL on node B, and both depend on driver and firmware versions matching closely enough. A single node out of step does not fail — it degrades or intermittently errors, and the symptom appears in whichever job happened to land on it.

Provisioning

Nodes are built from a defined image rather than configured by hand, so every node in the cluster starts identical.

Architecture
  • Bare-metal provisioning — network boot, image write, firmware baseline, and registration with the cluster.
  • Cluster images — the versioned artefact that defines what a node is. Changing the cluster means changing the image, not the running nodes.
  • Firmware baselining at provision time, so a node cannot enter the pool out of step.
Operational implications
The repair path is where consistency breaks
Initial provisioning is usually automated and consistent. The problem is the node replaced at 2am after a hardware failure: it arrives with whatever firmware the vendor shipped, gets put back into service because the cluster needs capacity, and becomes a source of intermittent faults nobody connects to the repair weeks later. Automated baseline enforcement on every node entering the pool is the control — not a documented procedure.

Kubernetes and Slurm

Estates frequently run both, which surprises people who expect one to have won. They suit different workload styles.

SlurmKubernetes
OriginHPC batch schedulingContainer orchestration for services
Natural fitLarge multi-node batch training jobsInference services, pipelines, mixed workloads
Gang schedulingNative — a job either gets all its nodes or waitsRequires additional components; not the default model
Topology awarenessLong-establishedAvailable, but needs deliberate configuration
EcosystemDeep in research computingVast, and where most tooling now lands

A conceptual orientation, not a recommendation. Which fits depends on workload mix and on what the team already operates well.

Inference does not reach the GPUs the way a training job does. A model serving users runs as a long-lived service, usually on Kubernetes, with its weights already loaded in GPU memory. A chat message never waits in a scheduler queue for GPUs to be allocated. Inside the service, an inference engine such as vLLM, SGLang or NVIDIA TensorRT-LLM batches requests from many users together and runs prefill and decode. In front of it, a KV-aware router such as NVIDIA Dynamo or llm-d sends each conversation back to the GPU that holds its cache. The scheduler places the service; the serving layer places the requests. Slurm stays the usual home for training jobs, which start, run and finish. See Key Fundamentals for prefill, decode and the KV cache.

Gang scheduling is the property that matters most
A distributed training job needs all its nodes simultaneously. Scheduling half of them and letting the job wait for the rest wastes exactly the capacity you were trying to use efficiently — the allocated accelerators sit idle holding a reservation. Any scheduler for training work needs all-or-nothing placement, whatever it is called in that ecosystem.

Scheduling, quotas and tenancy

  • Queues separate workload classes so a flood of small jobs cannot starve a large training run, and vice versa.
  • Quotas bound what a team or tenant may consume, which is a commercial control as much as a technical one.
  • Workload isolation keeps tenants from observing or affecting each other — enforced at the container, device and network layers, not only in the scheduler.
  • Capacity allocation connects scheduling to commercial commitments: reserved capacity someone has paid for is not the same as spare capacity available to anyone.
  • Topology-aware placement keeps a job's participants close in the fabric, which matters more here than in general-purpose scheduling.
Multi-tenancy claimed commercially but not enforced technically
It is common to sell isolated capacity and then implement isolation only in the scheduler — which prevents accidental interference, not deliberate. Genuine isolation needs enforcement at the device, network and storage layers too. This is worth confirming explicitly rather than assuming, because the gap is invisible until it matters.

Compatibility management

A compatibility matrix records which firmware, driver, library, runtime and scheduler versions are known to work together. Deviating from it is how clusters break.

Architecture

The matrix is a set, not a list of latest versions. Upgrading one component because a newer release exists — without checking what else must move with it — is the most common way a working cluster stops working. The matrix is also what makes an incident diagnosable: "which nodes are off-matrix" is a question with an answer.

Engineering deep dive

Upgrade sequencing

  1. Define the target setCluster engineers

    Every component version that will change, together, validated as a combination rather than individually.

  2. Canary poolCluster engineers

    Apply to a small set of nodes and run representative work — including a multi-node collective, because single-node testing will not surface communication problems.

  3. Validate against baselineSRE

    Compare collective performance to the recorded baseline, not just to "it ran". A regression that does not error is still a regression.

  4. Staged rolloutCluster engineers

    Batch by batch, draining work from each batch first. Halt on the first batch that misbehaves rather than continuing.

  5. Exercise rollbackSRE

    The rollback path must have been executed, not merely written down. A rollback attempted for the first time during an incident is not a rollback plan.

Roll back, do not fix forward, under pressure
When an upgrade breaks accelerator communication across a cluster, the instinct is to find and fix the incompatibility. Returning to the last validated set is almost always faster, and it restores a known state from which the problem can be investigated calmly. Fixing forward during an incident tends to produce a cluster in a state nobody has ever tested.
Operational implications
  • Version inventory across every node, continuously, compared against the matrix.
  • Off-matrix nodes treated as an incident, not as a backlog item.
  • Collective performance regression testing as part of the upgrade gate, since the failure mode is often slowness rather than error.

Key Takeaways

  • Components must be a validated set, not individually current versions.
  • Layers must agree across nodes as well as within one — NCCL negotiates with its peers.
  • A single off-matrix node degrades or errors intermittently rather than failing cleanly.
  • The repair path, not initial provisioning, is where consistency usually breaks.
  • Gang scheduling matters most: partial placement wastes exactly the capacity you were optimising.
  • Kubernetes and Slurm coexist in many estates because they suit different workload styles.
  • Multi-tenancy sold commercially is often enforced only in the scheduler — confirm the other layers.
  • Canary testing must include a multi-node collective; single-node testing cannot surface communication faults.
  • A rollback path that has never been executed is not a rollback path.
  • Under pressure, return to the last validated set rather than fixing forward into an untested state.