Skip to main content
BSS/OSS Academy

Architecture · Cluster · Simulation

Kubernetes and GPUs: giving each workload its own

Kubernetes runs software in pods and places them on machines. Once the vendor’s driver and device plugin are installed, it can give a pod dedicated GPUs. Dedicating GPUs to a whole cluster is different: it means dedicating the machines they sit in.

Try it: 8 GPUs, two teams

One cluster · one control plane · one scheduler

Node 1

  • GPU 1Free
  • GPU 2Free
  • GPU 3Free
  • GPU 4Free

Node 2

  • GPU 5Free
  • GPU 6Free
  • GPU 7Free
  • GPU 8Free
PendingNone waiting

Try it

One shared cluster. Add pods for either team and watch which GPUs they take.

Each pod gets its own GPUs, in one shared cluster

The pod asks for GPUs, and the scheduler places it on a node with enough free. For smaller work, MIG (Multi-Instance GPU) can split one GPU into as many as seven hardware-separated slices, depending on the GPU.

What it takes

One line in the pod’s specification.

resources:
  limits:
    nvidia.com/gpu: 1

How a pod reaches a GPU

Kubernetes does not talk to GPUs itself. Vendor software on each node makes them something it can schedule. This is the device plugin route; newer clusters can also use Dynamic Resource Allocation.

  1. 1

    NVIDIA GPU Operator

    Installs the driver, container toolkit and device plugin on each GPU node.

  2. 2

    Device plugin

    Tells Kubernetes how many GPUs the node has, as a resource called nvidia.com/gpu.

  3. 3

    Your pod

    Asks for a number of GPUs in its specification.

  4. 4

    Scheduler

    Places the pod on a node with that many GPUs free.

  5. 5

    Container

    Starts with those GPUs assigned to it. NVIDIA warns that a container requesting no GPUs may still see every GPU on the node.

Which level to use

LevelSeparationWhat it costsFits when
Per podWhole GPUs or MIG slices per pod. Nodes, network and control plane are shared.None beyond the setup. Teams compete for the same GPUs.One team, or teams that trust each other.
Per teamA GPU limit and access rules per namespace. Still one cluster.Quotas to maintain. Reserved nodes can sit idle.Several internal teams sharing a pool.
Per clusterSeparate control plane and machines. The strongest of the three.More clusters to run, and spare GPUs cannot move between them.Security, compliance or regional rules require it.

One shared cluster keeps GPUs available to everyone, so it is a sensible place to start. Add clusters when a specific security, compliance or regional rule requires it, because that decision is hard to undo and every extra cluster leaves some GPUs out of reach of the others.

For the same choices in more depth, including MIG profiles and running jobs across clusters, read Cluster software platform.

Three things to watch

Sharing is not the same as dedicating

Time-slicing lets pods take turns on one GPU, with no memory or fault isolation between them. MPS (Multi-Process Service) enforces limits on memory and compute, but a fatal fault still affects every client sharing that GPU, and its support in the device plugin is experimental. Only MIG divides a GPU in hardware. With sharing, or with MIG’s single strategy, nvidia.com/gpu: 1 may not mean a whole GPU.

Namespaces and quotas are not a strong security boundary

They organise access and limit use. Between tenants who do not trust each other, Kubernetes’ own guidance points to stronger measures such as sandboxing or a dedicated cluster.

A quota does not guarantee GPUs

It caps what a team may use. It does not set capacity aside, so a team under its quota can still wait if others have taken the GPUs. The simulation does not show this, because its two quotas add up to exactly the 8 GPUs.

Sources