Marvell’s custom high-bandwidth memory (HBM) layout moves the memory controller off an accelerator’s compute die and into a custom base die beneath each DRAM stack. Marvell estimates that the change could free compute area and lower memory-interface power, while making heat beneath the stack a central packaging challenge. Marvell describes the architecture and its estimates.
The design’s payoff depends on two questions: where the memory logic sits, and whether the workload is waiting on data or computation.
What changes in custom HBM?
An XPU is the accelerator’s compute die. In the standard HBM arrangement Marvell describes, the memory controller and HBM interface sit on that die. In its custom layout, each DRAM stack sits above a base die that contains the controller; a narrower die-to-die (D2D) physical interface connects the base die to the XPU.
| Arrangement | Memory-controller location | Interface at the XPU edge |
| Standard HBM | On the XPU compute die | Wide HBM physical interface |
| Marvell’s custom-HBM layout | In a custom base die beneath each DRAM stack | Narrower D2D physical interface |
The change shifts memory-control logic away from the XPU’s edge and compute area. Marvell estimates that, compared with standard HBM, the architecture could provide up to 25% more compute area, room for up to 33% more HBM stacks, and up to 70% lower memory-interface power, depending on the XPU design.
Those figures describe potential design gains, not a guaranteed increase in AI workload speed. More compute area helps most when computation is the constraint; a workload waiting on data may instead be limited by memory bandwidth.
Why does the layout make thermal design more important?
Putting a base die and its D2D interface beneath a DRAM stack concentrates more design attention on heat removal from that region. Marvell identifies the D2D PHY in the base die as the hottest spot and argues that the stack, base die, package and cooling need to be designed together.
Marvell cites a Samsung thermal study comparing components in a 2.5D package: internal thermal resistance accounted for 57% of the HBM stack’s total and 13% of the adjacent ASIC’s total. Thermal resistance describes opposition to heat flow; those percentages refer to each component’s total, not to its temperature.
For next-generation HBM running above 28 Gbps, Marvell expects heat flux to exceed 2.0 W/mm². The company discusses several ways to address heat: silicon heat paths near the D2D PHY, copper-to-copper hybrid bonding and coolant channels etched into silicon. Each approach makes package-level thermal engineering part of the memory design, rather than a separate finishing step.
Marvell also describes adaptive power management for the D2D interface. It says that feature can reduce interface power by up to 75% under normal workloads and up to 42% at peak traffic, compared with an interface without adaptive power management. These are separate comparisons from the broader estimate of up to 70% lower memory-interface power versus standard HBM.
Why can more compute fail to make an AI workload faster?
Some workloads are limited by memory bandwidth rather than computing capacity. Arithmetic intensity measures floating-point operations per byte of memory accessed; the roofline model uses it to distinguish work constrained by memory bandwidth from work constrained by compute. When data movement is the bottleneck, adding compute alone may not speed up the task.
IBM Technology presenter Kaoutar El Maghraoui explains the memory wall, arithmetic intensity and the roofline model as general ways to understand that distinction. The concepts help put Marvell’s potential bandwidth and power gains in context: their practical value depends partly on what limits a workload.
Marvell relays a projection that internal temperatures in 16-high HBM stacks will exceed the thermal budget by 2028.