BSS/OSS Academy

Glossary

Vendor-neutral definitions for the terminology used in this module. Search matches aliases and common misspellings — searching “RoCI” will find RoCE and say why.

Search terms, acronyms and aliases

Category

53 of 53 terms

2N redundancyFacility: Two complete independent systems, either of which can carry the full load. Contrast N+1, which provides one spare unit within a single system.
AI factoryCommercial & industry: A facility purpose-built to produce AI outputs at scale, where compute, network, storage, power and cooling are designed together as one system rather than assembled independently.
Baseboard Management ControllerBMCPlatform & software: An embedded controller providing out-of-band monitoring, console access and power control, independent of the host operating system.
BatchingPlatform & software: Running requests from many users through the GPU together, so that one read of the model’s weights serves all of them. Serving engines add and remove requests as each one finishes (continuous batching). Larger batches raise total throughput but can slow the answer each user sees.
BusbarFacility: The conductor distributing power vertically within a rack, feeding trays from shared power shelves.
BuswayFacility: Overhead or under-floor power distribution running through the data hall, from which racks tap their supply.
CheckpointingStorage: Periodically writing training state to storage so a job can resume after a failure. A synchronised, bursty, whole-cluster write — which is why storage sized to average throughput stalls the cluster.
Collective operationPlatform & software: A communication pattern in which all participating processes exchange data, such as all-reduce. In synchronous training, the slowest participant paces every other.
ColocationCommercial & industry: Renting space, power and cooling in someone else's facility while owning the equipment inside it.
Compatibility matrixPlatform & software: The validated set of firmware, driver, library and scheduler versions known to work together. Deviating from it is a leading cause of cluster-wide incidents.
Compute trayCompute: The physical serviceable unit holding processors, adapters and cold plates, installed into a rack. It defines the granularity of maintenance.
Configuration driftPlatform & software: Divergence between the intended configuration and what is actually running. In a fabric it produces faults that appear intermittent and resist reproduction.
Coolant Distribution UnitCDUFacility: The unit separating the facility water loop from the technology cooling loop that reaches the IT equipment, with pumps, heat exchange and monitoring.
Data Center Quantized Congestion NotificationDCQCNNetworking: A congestion-control approach combining ECN marking with rate control at the endpoint, intended to reduce reliance on PFC pausing in large RDMA deployments.
Data Processing UnitDPUCompute: A device that runs infrastructure services — networking, storage processing, security, telemetry — on its own processors, separate from the host CPU.
DecodePlatform & software: The second phase: the answer is generated one token at a time, and every token reads the model’s weights and the KV cache from GPU memory again. Limited by memory bandwidth more than compute, which is why HBM matters so much for serving.
Direct-to-chip coolingFacility: Liquid cooling in which coolant flows through cold plates mounted directly on the hottest components, rather than cooling the air around them.
Equal-Cost Multi-PathECMPNetworking: Distributing traffic across multiple equal-cost paths. Hashing decides which flow takes which path, so a small number of large flows can distribute unevenly.
Explicit Congestion NotificationECNNetworking: A mechanism allowing a router to signal congestion by marking packets rather than dropping them, so endpoints can slow down before loss occurs.
GPUDirect RDMANetworking: RDMA transfers that land directly in accelerator memory without staging through host memory.
High Bandwidth MemoryHBMCompute: Stacked memory placed close to the processor to provide far higher bandwidth than conventional attached memory. Usually the limiting resource for large models.
Immersion coolingFacility: Submerging equipment in a dielectric fluid so heat transfers directly into the fluid. Effective thermally, with significant implications for serviceability and facility design.
InfiniBandNetworking: An established high-performance interconnect architecture with native RDMA, its own fabric management and its own congestion mechanisms. Long used in HPC before its adoption in AI clusters.
KV cachePlatform & software: A model’s working memory of a conversation, held in GPU memory: the keys and values computed for every token so far, kept so they are not recomputed for each new token. It grows with every token and every concurrent user, so memory, more than compute, often caps how many users one GPU can serve.
NeocloudCommercial & industry: Informal term for a specialist GPU-cloud provider focused on accelerated compute rather than general-purpose cloud services. Related to but not synonymous with "hyperscaler" or "AI cloud".
Non-blockingNetworking: A fabric whose northbound capacity meets or exceeds what its endpoints can offer. A bandwidth property only — it says nothing about latency, buffering or hashing fairness.
NVLinkNetworking: A high-bandwidth, low-latency interconnect used for accelerator-to-accelerator communication within a scale-up domain. Not a substitute for, and not comparable in role to, Ethernet or InfiniBand.
NVMe over FabricsNVMe-oFStorage: Carrying the NVMe storage protocol across a network so remote devices behave much like local ones.
Out-of-band managementOOBPlatform & software: A management path independent of the data network, so equipment stays reachable when the data fabric fails. If it shares the data fabric it is useless in exactly the incident where it is needed.
OversubscriptionNetworking: The ratio between the bandwidth endpoints can offer and the bandwidth the fabric can carry upward. Greater than 1:1 means the fabric cannot carry all endpoints at line rate simultaneously.
Parallel file systemStorage: A shared file system that stripes data across many servers and devices to deliver aggregate throughput far beyond a single server.
PodCompute: Multiple racks operating together as one larger system, bounded by the fabric and by the facility power and cooling envelope.
Power shelfFacility: Shared power conversion within a rack, feeding the busbar instead of each server carrying its own supplies.
Power Usage EffectivenessPUEFacility: Total facility power divided by IT power. It measures facility overhead, and says nothing about whether the IT load is doing useful work — so it cannot describe AI cluster efficiency on its own.
Powered shellCommercial & industry: A building delivered with power and structure in place but without internal fit-out, which the tenant completes to their own design.
PrefillPlatform & software: The first phase of a language model answering a request: the whole prompt is read in one parallel pass, and what the model works out is stored in the KV cache. Limited by compute, and the main part of the wait before the first token appears.
Priority Flow ControlPFCNetworking: An Ethernet mechanism that pauses traffic for a specific priority class to prevent loss. Pausing propagates backwards through the fabric, which can cause head-of-line blocking and spread congestion far from its source.
Rack-scale systemCompute: A design in which compute, scale-up fabric, power and cooling are engineered as one coordinated rack rather than as independent servers. The rack becomes the deployable unit.
Rail-optimised networkingNetworking: A design in which each accelerator connects to a specific fabric plane ("rail"), so peer accelerators communicate within the same rail and reduce cross-fabric traffic.
RDMA over Converged EthernetRoCENetworking: RDMA carried over an Ethernet network. RoCEv2 encapsulates it in UDP/IP, which makes it routable across IP subnets.
Remote Direct Memory AccessRDMANetworking: A technique that lets a network adapter read and write application memory directly, bypassing the kernel networking stack and avoiding per-packet CPU copies.
Scale-outNetworking: Communication between servers, racks and pods across a routed fabric, typically Ethernet or InfiniBand, carrying east-west collective traffic.
Scale-upNetworking: Communication inside a tightly coupled compute domain — accelerator to accelerator, at interconnect bandwidth. The size of this domain bounds how large a tightly coupled model partition can be.
Sovereign AICommercial & industry: AI infrastructure operated within a jurisdiction to satisfy data-residency, regulatory or national-capability requirements.
Spine-leafNetworking: A two-tier Clos topology in which every leaf switch connects to every spine switch, giving predictable path counts and equal-cost multipath between any two endpoints.
StragglerPlatform & software: A participant slower than its peers, which in a synchronous job delays every step for everyone. Often caused by a degraded link, optic or device rather than an outright failure.
Stranded powerFacility: Contracted electrical capacity that cannot be used because of cooling, floor-space or distribution limits. Paid for, unusable.
SuperchipCompute: A vendor term for a package combining a CPU and one or more accelerators with a tightly coupled interconnect between them, presented as a single unit.
Technology cooling loopTCSFacility: The closed loop between the CDU and the IT equipment, kept separate from facility water so facility-side water quality never reaches the hardware.
Time to first tokenTTFTPlatform & software: How long a user waits between sending a request and seeing the first token of the answer. Set mostly by queueing and prefill; how fast the rest of the answer arrives is measured separately, as tokens per second.
Tokens per secondPlatform & software: The rate at which a model produces output. Quoted two ways that are easily confused: per user (how fast one answer appears) and per GPU or per system (total output across every request in the batch). Batching raises the second at some cost to the first.
Water Usage EffectivenessWUEFacility: Water consumed per unit of IT energy. Frequently trades against PUE: evaporative cooling improves one and worsens the other.
Zero-touch provisioningZTPPlatform & software: Automatically configuring a device on first boot from a central source, rather than by manual console configuration.