3.9 Monitoring and observability

Monitoring and observability

What is actually watched across facility, compute, network, storage and workload layers — and which signals predict an incident rather than report one.

Most monitoring is built to answer "has something broken?". In an AI cluster the expensive question is different: "is something getting worse?" The failures that cost the most — a degrading optic, a slow accelerator, a lengthening checkpoint, a silently lost power feed — break no threshold until long after they started costing money.

Five layers of signal

Facility

Utility supply state, UPS state and mode, generator readiness and fuel.
Rack power draw, per-feed balance, and redundancy state.
CDU state, pump state, coolant supply and return temperature.
Flow rate, differential pressure, leak detection.
Environmental alarms: hall temperature, humidity, door state.

Redundancy state is the signal facility monitoring most often omits

Power draw is monitored everywhere. Whether the rack is still running on two feeds rather than one frequently is not — and that is the signal that tells you the rack is one failure from an outage while nothing appears wrong.

Compute

Accelerator utilisation and memory use, temperature, power draw.
Throttling events — the difference between "running" and "running at the speed you paid for".
ECC error counters, correctable as well as uncorrectable.
CPU utilisation, memory pressure and I/O wait on the host.
Firmware and driver versions, compared continuously against the compatibility matrix.
Scale-up link health and reported topology versus intended topology.

Network

Link state and flap history.
Packet loss and error counters.
Optical transmit and receive levels, and pre-FEC error rate — the leading indicators for silent degradation.
PFC pause counters and ECN marking rates — the fabric telling you it is under stress.
Buffer occupancy, latency, and route or topology state.

Storage

Read and write throughput, IOPS.
Metadata latency — the signal that predicts small-file workload problems.
Checkpoint duration, tracked as a trend.
Capacity, failed devices, and rebuild activity.

Workload

Queue time, job duration, failed jobs.
Accelerator occupancy — utilisation says a device is busy; occupancy says whether it is doing useful work.
Stragglers — per-participant timing within a job.
Training progress, inference latency, and tokens per second where the workload makes that meaningful.

Utilisation and occupancy are not the same thing

A GPU spinning in a synchronisation barrier waiting for a slower peer can report high utilisation while producing nothing. Utilisation is the most quoted AI infrastructure metric and one of the least informative on its own. Occupancy, step time and straggler distribution are what tell you whether the cluster is earning its keep.

Leading versus lagging signals

Fault	Lagging alarm (too late)	Leading indicator (useful)
Optical degradation	Link down	Pre-FEC error rate and receive power drifting from baseline
Accelerator failure	Uncorrectable ECC, job aborts	Correctable ECC error rate rising
Cooling problem	Thermal shutdown	Coolant delta-T trend, flow rate drift, differential pressure change
Storage saturation	Checkpoint write fails	Checkpoint duration growing week over week
Power exposure	Rack outage	Loss of feed redundancy
Fabric congestion	Job timeout	PFC pause and ECN marking counters rising
Version drift	Cluster-wide intermittent errors	Node inventory diverging from the compatibility matrix

The left column is what most monitoring is configured to catch. The right column is where the money is.

Cross-layer correlation

The facility team watches cooling. The platform team watches jobs. When a cooling trend causes a job slowdown, neither sees the connection — because nobody is looking at both.

Architecture

This is an organisational problem expressed as a tooling problem. Facility telemetry usually lives in a BMS or DCIM platform owned by one team; compute and workload telemetry live in Prometheus and Grafana owned by another. The two frequently share no timeline, no naming, and no on-call rota.

A correlation that only works if someone joined the data

Coolant supply temperature drifts upFacility (BMS)
Within design limits. No facility alarm. Visible only as a trend.
Accelerators in the affected racks begin throttlingCompute (DCGM)
Intermittently, under sustained load. Devices report healthy.
One job runs slower than its historical baselineWorkload (scheduler)
Its participants in those racks become stragglers, pacing everyone else.
The user reports "the cluster feels slow"Everyone
Three teams investigate three unconnected symptoms. The cause was visible in step 1, days earlier.

Operational implications

What makes correlation actually possible

Shared timestamps and a shared identifier scheme — rack, tray, node, job — across facility and platform telemetry. It is unglamorous work that nobody owns by default, and it is the difference between diagnosing that scenario in an afternoon and diagnosing it in a fortnight.

Alert design

Alert on deviation from a recorded baseline, not on absolute thresholds, for anything that degrades gradually.
Record the baseline at commissioning — optical levels, per-node throughput, collective performance, checkpoint duration. Without it, "deviation" has no meaning.
Alert on loss of redundancy, not only on loss of service.
Route by layer, not by severity. A cooling trend belongs with facilities; a straggler belongs with the platform team; both may need each other.
Treat alert volume as a design defect. An on-call engineer who filters alerts by habit has effectively turned monitoring off.

Representative tooling

Tool	Layer	Typically provides
NVIDIA DCGM	Compute	Accelerator health, utilisation, ECC, throttling, power
Prometheus	All	Metric collection and storage
Grafana	All	Dashboards and visual correlation
NVIDIA NetQ	Network	Fabric state, topology validation, telemetry
Redfish	Compute / facility	Standardised out-of-band hardware management
DCIM	Facility	Power, space, capacity and asset tracking
BMS	Facility	Cooling plant, electrical plant, environmental
ITSM platform	Process	Incidents, changes, problem management

Representative examples, not a recommended stack. The important property is not which tools are used but whether their data can be correlated.

Key Takeaways

The expensive AI failures degrade rather than break, so trends matter more than thresholds.
Baselines recorded at commissioning are what make "deviation" a meaningful alert.
Utilisation says a device is busy; occupancy says whether it is doing useful work.
Loss of redundancy produces no service impact and is therefore under-monitored and under-prioritised.
Correctable error rates, pre-FEC errors, coolant delta-T and checkpoint duration are leading indicators; link-down and thermal shutdown are lagging ones.
Facility and platform telemetry usually live in different systems owned by different teams — correlation is an organisational problem before it is a tooling one.
Shared timestamps and a shared rack/tray/node/job identifier scheme are what make cross-layer diagnosis possible.
Alert volume high enough that engineers filter by habit has effectively turned monitoring off.