Researchers are developing “side-stacked” memory architectures to resolve heat and bandwidth bottlenecks in AI chips, with two new designs, V-Die and MOSAIC, promising significant increases in speed and capacity, according to presentations at the IEEE VLSI Symposium.
Current AI accelerators, such as the Nvidia B300, rely on HBM stacks—typically 12 dies high—placed beside the GPU on a shared substrate. While this setup allows for high data transmission rates, the vertical stacking creates a thermal trap. The materials filling the gaps between dies are roughly 100 times more thermally resistant than the silicon substrate, which prevents heat from reaching the heat sink, according to research presented at the IEEE VLSI Symposium.
As AI models grow, the demand for memory capacity and bandwidth has created a “massive bottleneck,” stated Heesoo Yang, a doctoral student at the Ulsan National Institute of Science and Technology (UNIST) in South Korea. Yang noted that increasing stack height forces a trade-off: more through-silicon vias (TSVs) are required to move data, which consumes the silicon area otherwise used for memory storage.
V-Die: Boosting Speed via Microfluidic Cooling
A research team led by Jimin Kwon at UNIST, in collaboration with Seongju Kim at Hanbat National University in Daejon, proposed a solution called V-Die. This architecture stacks DRAM vertically but then turns the assembly on its side to connect to the substrate. To manage heat, V-Die incorporates microfluidic cooling channels between the dies, which researchers say can maintain temperatures at 45°C, significantly lower than the typical 80°C-plus peaks seen in standard HBM.
V-Die eliminates the need for TSVs and a traditional base die, as each die features its own I/O systems along the bottom edge. These systems connect to the silicon substrate, on which the GPU sits, via links every 20 micrometers. According to the UNIST team, this allows for four times as many connections as HBM4 and reduces memory read time by 37 percent.
In simulations using a workload based on a GPT-3 sized large-language model and Nvidia H100 GPUs, the V-Die system delivered 540 tokens per second, compared to 296 for HBM4 with the same memory capacity. The team also calculated a 32 percent reduction in latency, or approximately 24 milliseconds, for the delivery of the first token. A prototype device is currently being developed to validate these electrical and thermal characteristics.
MOSAIC: Expanding Capacity with Inductive Coupling
A separate group from the University of Tokyo, Tohoku University, and the Riken national research institute developed the MOSAIC architecture. To solve the problem of precise physical alignment—where a few micrometers of difference in die thickness can cause a connection failure—the Japanese team implemented an inductive coupling transceiver system.
This system uses oblong-shaped inductive coils (approximately 80 by 240 micrometers) on the memory die and a corresponding set at a right angle on the substrate. Because the coils use magnetic fields to transmit data signals, they do not require exact physical overlap, providing more leeway during assembly. Power connections are located on the sides of the memory cube, according to University of Tokyo doctoral student Yuki Mitarai.
The MOSAIC design can fit 98 dies per cube, providing 294 GB of memory. While it lacks microfluidic cooling, the fact that heat can rise through the silicon fins themselves should keep such a structure at 81.3°C. Mitarai stated that if DRAM dies were thinned by two-thirds to 100 micrometers, a single MOSAIC cube could integrate 294 dies, reaching a capacity of 882 GB.
Comparing Volumetric DRAM Approaches
The shift toward volumetric DRAM represents a fundamental change in how memory interacts with processors. While traditional HBM4 focuses on verticality, the V-Die and MOSAIC systems prioritize thermal management and connection density.
| Feature | V-Die (South Korea) | MOSAIC (Japan) |
|---|---|---|
| Primary Goal | Speed and Latency Reduction | Capacity and Thermal Stability |
| Cooling Method | Microfluidic channels (45°C) | Silicon fin dissipation (81.3°C) |
| Connection Type | Direct substrate links (20μm) | Inductive coupling coils |
| Key Metric | 82% speed boost over HBM4 | Up to 882 GB capacity |
Despite these gains, integration remains a hurdle. James Myers, a program director at the Imec microelectronics research center in Belgium, warned that getting the thickness of the DRAM dies exactly right is critical. According to Myers, irregular dies in a stack can lead to missing the bond pads during the connection process to the substrate.
The development of these prototypes marks a transition from theoretical modeling to physical validation. The next phase for these research groups involves the completion of prototype devices to verify the thermal and electrical performance of side-stacked memory in real-world AI workloads.
Share your thoughts on the future of AI hardware in the comments below or share this analysis with your network.