When an ordinary web application receives a request, the server may spend much of its time moving through familiar layers: CPU instructions, system memory, storage, network interfaces, and software services. Many workloads consist of relatively small bursts of computation, followed by waiting for data.
AI workloads change that balance. Training a large model can keep thousands of accelerator cores busy while repeatedly moving enormous amounts of data through high-bandwidth memory and between accelerators. Inference can create a different pressure: a model may need to keep large amounts of model state and attention-related data available while generating tokens for many users simultaneously.
That difference is changing what engineers optimize when they design a server. The accelerator is no longer simply an add-in device attached to a conventional computer. Memory bandwidth, accelerator-to-accelerator communication, networking, power delivery, cooling, and even rack architecture increasingly have to be designed as one system.
Current AI infrastructure illustrates how far this has moved. NVIDIA’s 2026 reference architecture, for example, separates front-end networking, management networking, GPU cluster networking and high-bandwidth NVLink connections inside an AI rack. (NVIDIA Docs) The result looks less like a collection of independent servers and more like a distributed computer assembled from many tightly connected processing devices.
Why Traditional Server Design Starts to Struggle
The central problem is not simply that AI calculations require more processing power. It is that the calculations have a different relationship with memory and communication.
A neural-network operation may perform large matrix multiplications involving many values at once. Modern AI accelerators therefore contain specialized matrix-processing hardware alongside conventional programmable compute units. These units can perform operations on formats such as FP16, BF16, FP8 or other reduced-precision representations, depending on the architecture and workload.
But fast arithmetic is useful only when the processor can receive data quickly enough.
This creates a familiar systems problem called the memory wall. Processor arithmetic capability can increase faster than the system’s ability to supply data. An accelerator with enormous theoretical compute throughput can therefore spend part of its time waiting for operands, weights, activations or intermediate results.
That is why AI accelerators increasingly pair specialized compute hardware with high-bandwidth memory (HBM). HBM places memory close to the accelerator package and provides a very wide interface. For example, AMD’s MI300X provides 192 GB of HBM3 and a specified peak memory bandwidth of 5.3 TB/s, while newer MI325X systems increase HBM capacity and bandwidth further. (AMD)
The important point is not the individual specification. It is the architectural relationship: more compute creates pressure for more memory bandwidth, which in turn changes packaging, power, cooling and server design.
The Accelerator Changes the Shape of the Server
In a conventional server, the CPU is usually the primary compute device and system RAM is the main working memory. PCIe-connected devices such as GPUs, storage controllers and network adapters operate around that central system.
AI servers reverse much of that emphasis.
The accelerator becomes a major compute engine, while the CPU increasingly coordinates workloads, handles operating-system functions, manages input and output, and supplies data to the accelerator. The exact division varies by architecture and software stack, but the hardware trend is clear.
AMD’s MI300 family demonstrates another approach: the MI300A combines Zen 4 CPU cores and CDNA 3 GPU compute resources with shared HBM3 memory. (AMD) Instead of treating CPU memory and accelerator memory as completely separate resources, this type of architecture reduces some of the data movement between processor domains.
That matters because moving data has a cost. A workload can be slowed not because an accelerator lacks arithmetic capacity, but because information must repeatedly cross an interface between different memory or processing domains.
This is one reason specifications such as memory capacity and bandwidth have become important when examining AI hardware. A useful way to understand those numbers is to separate two questions:
| Hardware characteristic | What it determines |
|---|---|
| Compute throughput | How much mathematical work the processor can potentially perform |
| HBM capacity | How much frequently accessed model/workload data can remain in high-speed local memory |
| HBM bandwidth | How quickly data can move between accelerator and HBM |
| CPU memory capacity | How much conventional host-side data can remain available |
| Accelerator interconnect | How quickly multiple accelerators can exchange data |
| Network bandwidth | How efficiently work can move between servers or racks |
| Storage bandwidth | How quickly large datasets and checkpoints can be read or written |
None of these numbers independently determines application performance. The workload has to use the particular resource efficiently.
One Accelerator Is Often Not Enough
A large model may exceed the memory capacity of a single accelerator. Training also frequently requires distributing computation across many devices.
That creates a second major hardware problem: communication between accelerators.
Suppose a model is divided across several accelerators. During computation, those devices may need to exchange activations, gradients, parameters or other intermediate information. If those transfers use a relatively slow path, the accelerators can spend valuable time waiting for communication.
This is why AI systems increasingly include dedicated high-bandwidth interconnects rather than relying exclusively on conventional peripheral connections.
NVIDIA’s NVLink is one example. NVIDIA describes its current NVLink 6 architecture as a scale-up fabric designed to connect accelerators inside large AI systems, with its 2026 platform material specifying 3.6 TB/s of bidirectional bandwidth per GPU and a 72-GPU NVL72 domain. (NVIDIA Developer)
These technologies solve a different problem from HBM.
HBM moves data quickly between an accelerator and its local memory. An accelerator fabric moves data quickly between accelerators.
That distinction is fundamental. Increasing HBM bandwidth cannot by itself eliminate a bottleneck caused by slow communication between separate GPUs.
Scale-Up and Scale-Out Are Different Problems
Once several accelerators connect inside one system or rack, another layer emerges: communication between separate servers.
This creates two related but distinct architectures.
- Scale-up connects processing devices inside a tightly coupled system. The goal is to make multiple accelerators communicate as efficiently as possible.
- Scale-out connects many such systems across a cluster. The goal is to allow a workload to extend beyond the resources available in one server or rack.
Modern AI infrastructure therefore has multiple networking layers. NVIDIA’s published data-center architecture describes a tenant-access network, a secure management network, a cluster interconnect network and an NVLink scale-up network. The cluster network can use Ethernet or InfiniBand, while NVLink serves the high-bandwidth scale-up domain. (NVIDIA Docs)
This separation exists because not every packet has the same job.
A user’s request might enter through conventional data-center networking. Large distributed training operations, meanwhile, may generate enormous volumes of accelerator-to-accelerator traffic that require completely unique latency and bandwidth characteristics.
The server consequently becomes part of a larger communication topology rather than an isolated machine.
Memory Is Becoming More Flexible
HBM is extremely valuable, but it is not the only memory technology involved in AI infrastructure.
Large AI systems may also need conventional DRAM, storage-class resources, local NVMe storage and other memory tiers. The challenge is deciding which data belongs at which level.
Frequently accessed computation data benefits from being close to the accelerator. Less frequently accessed information can remain in a larger but slower memory tier or storage system.
This is one area where Compute Express Link (CXL) is relevant. CXL provides standardized mechanisms for attaching and managing memory and accelerators over a high-speed interconnect. The CXL Consortium’s 4.0 specification, released in 2026, increases the signaling rate from 64 GT/s to 128 GT/s and adds capabilities including bundled ports and enhanced memory RAS features. (Compute Express Link –)
CXL does not magically turn external memory into HBM. Different memory technologies retain different latency, bandwidth and capacity characteristics. Instead, the technology provides another standardized building block for constructing systems where memory resources can be expanded, pooled or connected in ways that conventional server architectures do not easily provide.
That becomes increasingly relevant when workloads vary significantly in memory demand.
Networking Becomes Part of Compute Performance
AI clusters also expose an important misconception about networking: network bandwidth is not merely an input/output specification.
For distributed AI, communication can directly affect how much useful computation the accelerators perform.
Imagine eight accelerators calculating parts of the same training operation. If each device completes its local work quickly but then has to wait for the others to exchange results, the overall system can become communication-bound.
Consequently, AI networking increasingly includes technologies and software designed around collective operations such as all-reduce, broadcast and synchronization. NVIDIA’s current NVLink architecture, for example, includes networking features intended to accelerate collective communication within accelerator domains. (NVIDIA Developer)
This is also why AI clusters may use very high-speed Ethernet or InfiniBand connections between nodes. The exact choice depends on the system architecture, software stack and operator requirements; it should not be assumed that one interconnect is universally appropriate.
For readers who want to understand the conventional networking layers involved around these systems, TechTalksX’s Internet Network Ports Reference provides useful background, although ports themselves are only a small part of modern AI-cluster networking.
Power Stops Being a Supporting Detail
More accelerators, faster memory and higher-speed interconnects create another unavoidable consequence: more electrical power must reach the hardware, and the resulting heat must be removed.
This is where AI server design increasingly diverges from ordinary enterprise servers.
The International Energy Agency reported in 2026 that global data-center electricity consumption grew by 17% in 2025, while electricity consumption from AI-focused data centers grew considerably faster. Its analysis also identifies increasing accelerator adoption and higher power density as major factors shaping data-center infrastructure. (IEA)
At the rack level, the problem becomes physical. Electrical power eventually becomes heat. Removing that heat requires moving thermal energy away from chips and into the facility’s cooling system.
Air cooling becomes increasingly difficult as power density rises. Open Compute Project material from 2026 describes AI infrastructure work involving high-density racks, direct-to-chip liquid cooling and systems capable of handling accelerator devices with multi-kilowatt power levels. (Open Compute Project)
Liquid cooling therefore is not simply an efficiency feature added to an otherwise unchanged server. It can affect cold plates, tubing, manifolds, coolant distribution units, rack design, facility plumbing and maintenance procedures.
OCP and ASHRAE have explicitly been working together on standards and practices around liquid cooling because the boundary between IT equipment and facility infrastructure is becoming increasingly important. (Open Compute Project)
The Rack Is Becoming the Unit Engineers Design
The combined effect of these changes is significant.
With ordinary servers, replacing one machine with another can often be treated as a relatively contained hardware decision. AI infrastructure increasingly requires engineers to consider the server, accelerator interconnect, network fabric, power distribution and cooling system together.
NVIDIA’s current reference architecture provides a concrete example. Its compute-node documentation describes systems containing multiple accelerators, large HBM pools, NVMe storage and high-speed network connections, while the rack architecture separates different communication networks according to their roles. (NVIDIA Docs)
Other vendors use different architectures. AMD’s Instinct platforms, for example, use Infinity Fabric to connect accelerator modules and provide platform-level configurations with multiple accelerators. (AMD)
The general engineering principle is more important than any one vendor’s implementation: the bottleneck can move from the processor to memory, from memory to interconnect, from interconnect to networking, or from electronics to power and cooling.
Designing an AI server therefore becomes an exercise in balancing the entire data path.
What Happens to a Single AI Request?
Consider a simplified inference request.
A user sends text to an AI service. The request enters the data center through conventional networking and is routed to an application service. Software determines which model-serving resources should process it.
The model’s parameters already reside in some combination of accelerator memory and other memory tiers. The serving system prepares the input and schedules computation on an accelerator. Matrix operations execute on specialized hardware, while intermediate values move between compute units and HBM.
As generation continues, the system must repeatedly process additional tokens and maintain the state required by the model’s attention mechanism. When many users are being served concurrently, the system must manage memory capacity and bandwidth while keeping the accelerators sufficiently utilized.
If one accelerator cannot contain the required workload, additional devices may participate. Data can then move over an accelerator fabric or, for larger distributed systems, across a cluster network.
The final response travels back through the service and network layers to the user.
The visible experience is simply a stream of generated text. Behind it is a pipeline involving computation, memory hierarchy, accelerator interconnects, network fabrics, scheduling, storage and thermal infrastructure.
Why New AI Workloads Keep Changing Hardware Priorities
AI workloads are not static. Training, ordinary inference, long-context inference, multimodal generation, video generation and agentic systems can stress different parts of the infrastructure.
The IEA’s 2026 analysis highlights this changing workload profile: simple AI tasks can become much more energy-efficient while newer uses such as reasoning, video generation and agentic workloads can require substantially more computation. (IEA)
That creates an unusual hardware design problem. Improving efficiency does not necessarily reduce total infrastructure demand if applications simultaneously become more capable and more heavily used.
It also explains why the industry is moving beyond simply building a faster GPU. Current designs increasingly address memory capacity, memory bandwidth, scale-up fabrics, scale-out networking, power delivery and cooling as coordinated problems.
Some future architectures will use proprietary technologies; others will rely on open standards such as CXL and emerging open AI infrastructure specifications. The exact implementations will continue to differ between vendors, and many internal details of commercial AI systems are not publicly disclosed.
What can already be observed, however, is the direction of the engineering problem: AI performance increasingly depends on how quickly the entire system can move information, not merely on how quickly one chip can calculate.
Frequently Asked Questions
Why can’t AI workloads simply use faster CPUs?
CPUs are highly flexible and remain essential for operating systems, orchestration, preprocessing and many general-purpose tasks. AI matrix operations, however, can be executed much more efficiently on specialized parallel hardware. The resulting architecture usually combines CPUs with accelerators rather than replacing CPUs entirely.
Why does HBM matter so much for AI?
HBM provides very high memory bandwidth close to an accelerator. This helps feed compute hardware with the large volumes of data required by many AI operations. Capacity also matters because data that fits into local high-bandwidth memory does not need to be fetched from a slower memory tier as frequently.
Is more GPU memory always faster?
No. Capacity and bandwidth solve different problems. Additional memory can allow a larger model or workload to fit, but it does not automatically increase computation speed. Actual performance depends on whether compute throughput, memory bandwidth, communication, software efficiency, or another resource limits the workload.
Why are AI data centers using liquid cooling?
Higher accelerator power densities create more heat in a smaller physical area. Liquid cooling can transport heat away from high-power components more effectively than relying entirely on air. The precise cooling architecture varies, and liquid cooling introduces its requirements for coolant distribution, component compatibility, monitoring and maintenance.
Why do AI servers need special networking?
When many accelerators cooperate on one workload, they must exchange data and synchronize. Slow communication can leave otherwise powerful accelerators waiting. High-bandwidth, low-latency interconnects therefore become part of application performance rather than merely a way to connect servers to the outside network.
The Hardware Lesson Behind AI Infrastructure
The most useful way to look at an AI server is not as a faster version of a conventional server. It is a tightly coupled system designed around a particular movement of data.
Compute units need operands. HBM supplies them at high bandwidth. Multiple accelerators exchange intermediate data through specialized interconnects. Larger groups of machines communicate through high-speed cluster networks. Storage supplies datasets and checkpoints. Power systems deliver the electrical load, while cooling systems remove the resulting heat.
When one part becomes faster, another part can become the limiting factor.
That is why AI workloads are changing server hardware at every level—from chip packaging and memory interfaces to network fabrics, rack power and liquid cooling. Understanding those connections makes hardware specifications easier to interpret: a large accelerator is only one component of the system, and the performance users experience ultimately depends on whether the rest of the infrastructure can keep that accelerator supplied, connected and cool.