The growing memory challenge in AI
AI workloads require vast amounts of data processing in real time, whether for energy-efficient edge AI applications or for high-performance data center AI training. However, conventional memory technologies, such as SRAM, low-power double-data-rate (LPDDR)-DRAM-based, and high-bandwidth memory (HBM)-DRAM-based, present major limitations:
- SRAM is fast but has high leakage power and poor scalability for large gigabyte discrete memory chips.
- LPDDR-DRAM offers more capacity but suffers from latency and power inefficiencies.
- HBM-DRAM provides high bandwidth but consumes significant power, impacting overall system efficiency.
The hidden cost of DRAM power consumption
One of the most pressing challenges in AI memory is the power consumption of DRAM, which remains a dominant memory technology in data centers. With DRAM consuming up to over 30% of a data center’s total power, improving memory efficiency is crucial for sustainable AI computing. Several factors contribute to this high power usage:
- Significant energy drain: As AI workloads demand larger memory capacities, DRAM power consumption increases accordingly.
- Background power consumption: A large portion of DRAM power usage comes from “background power,” including refresh cycles required to maintain data integrity.
- Workload-dependent energy usage: The actual power draw of DRAM fluctuates based on workload intensity, with heavier memory access leading to higher power consumption.
This growing energy demand poses a significant challenge for sustainable AI computing, making it essential to explore new memory solutions that can reduce power while maintaining high performance.
Memory challenges for large AI models
As AI models, especially large language models (LLMs), continue to scale in size, the demands on memory for both training and inferencing become more extreme. Ideal memory for AI should have:
- Faster read/write latency—matching or exceeding SRAM speeds for real-time AI processing
- Higher bandwidth than HBM—to keep up with the vast amounts of data AI workloads require
- Super-low power consumption—preferably nonvolatile, reducing the energy burden in both edge and data center AI
- Scalability and manufacturability—ensuring higher density and cost-effective production at scale
- Cost efficiency—For new technologies, cost structure is always a challenge. We need a strong total cost of ownership (TCO) story and a continuous effort to reduce silicon wafer costs, either through long-term scalability via memory-cell size reduction or by developing memory-cell stacking technology.
Emerging memory solutions: the future of AI computing
To break free from these limitations, new memory architectures must deliver high-speed, high-bandwidth, and energy-efficient solutions. Several emerging technologies are leading this transformation:
1. Magnetoresistive RAM (MRAM)
Why it matters: MRAM offers fast read speeds, nonvolatility, and significantly lower power consumption compared with DRAM and SRAM.
Advancements: New STT-MRAM is improving write endurance, bandwidth, and scalability, making it viable for AI accelerators and edge devices.
Impact: MRAM reduces standby power, enables in-memory computing, and lowers the TCO for AI systems.
2. Resistive RAM (RRAM)
Why it matters: RRAM is an ultra-low-power nonvolatile memory with high density and fast switching speeds.
Advancements: Improvements in endurance and retention are making RRAM a candidate for AI inference workloads and neuromorphic computing.
Impact: RRAM enables energy-efficient AI model storage and edge AI applications.
3. 3D DRAM and HBM evolution
Why it matters: Traditional DRAM scaling is slowing, but 3D DRAM stacking and next-gen HBM (such as HBM4 and beyond) are improving performance.
Advancements: Future HBM iterations aim for lower power and higher bandwidth per watt, addressing some AI bottlenecks.
Impact: This evolution enhances training and inference for large-scale AI models but still faces power constraints.
4. Compute-in-memory (CIM) and processing-in-memory (PIM)
Why it matters: AI inference is bottlenecked by memory movement, making CIM/PIM crucial for accelerating AI performance.
Advancements: MRAM, RRAM, phase-change memory (PCM), and DRAM are being adapted for in-memory computing architectures.
Impact: These approaches reduce data-transfer latency, improve AI accelerator efficiency, and support real-time AI workloads.
Beyond memory technology: ecosystem and infrastructure alignment
Besides the expectation of innovative memory technologies, other key factors play a crucial role in AI performance advancements:
- Ecosystem alignment—Memory technology must evolve alongside industry standards, including HBM and emerging interconnect technologies such as Universal Chiplet Interconnect Express (UCIe), ensuring seamless integration with AI accelerators.
- Higher die-stacking technology—To meet growing AI memory capacity demands, advancements in high-density die stacking are critical for improving scalability and efficiency.
- Compute-in-memory for AI efficiency—Reducing the interaction between the AI chip and memory through in-memory computation helps decrease processing load, improve power efficiency, and shorten AI processing times.
- SoC capability for optimized chip layout—To achieve the best efficiency across various AI components, including memory, SoC design must be optimized for seamless integration, reducing bottlenecks, and enhancing overall system performance.
Memory challenges in edge AI
For edge AI applications—including wearables, battery-powered devices such as smartwatches, electric vehicles, and smart cameras—the key challenge is extending battery life while maintaining high performance. Current memory architectures often rely on a combination of NOR flash for code storage and LPDDR for fast data access. However, this approach increases system complexity, power consumption, and board space.
A unified memory solution is needed to streamline architecture, reducing power and space while improving efficiency. Emerging nonvolatile memory technologies that combine fast read/write speeds with ultra-low power consumption can significantly enhance edge AI devices, enabling longer battery life without sacrificing performance.
AI computing is reaching a crossroads where traditional memory technologies are no longer sufficient to meet power and performance demands. As LLMs grow larger, memory must evolve to meet the need for SRAM-like speeds, HBM-level bandwidth, ultra-low power consumption, nonvolatility, and scalability.
By integrating next-generation memory solutions, including MRAM, RRAM, and in-memory computing architectures, which may overcome current memory bottlenecks and unlock new levels of efficiency in AI systems. The next wave of memory innovation will be pivotal in unlocking AI’s full potential, driving new breakthroughs from edge AI to hyperscale data centers. As the industry continues to innovate, rethinking memory design will be crucial in shaping the next frontier of AI performance.
###