SAN FRANCISCO—The artificial intelligence revolution is running headlong into a wall of zeros. As large language models (LLMs) balloon to trillions of parameters—Meta’s latest Llama 4 boasts 2 trillion—their energy demands and carbon footprints are spiraling out of control. Training a single LLM can emit as much CO₂ as five cars over their entire lifetimes, according to a 2019 study. Yet most of those parameters are zeros, or so close to zero they might as well be. This “sparsity” is the AI industry’s best-kept secret—and its biggest opportunity for efficiency.
For years, hardware designers have treated those zeros as computational dead weight. CPUs and GPUs, optimized for dense matrix operations, waste cycles and energy multiplying by zero. But a new generation of chips is flipping the script. Stanford University’s Onyx accelerator, unveiled in 2024, is the first programmable hardware to exploit sparsity at scale. By skipping zero-valued computations entirely, Onyx slashes energy employ by up to 98% while speeding up calculations eightfold. The breakthrough could finally square the circle: keeping AI’s performance gains while taming its environmental and financial costs.
This isn’t just about faster chips. Sparsity-aware hardware could unlock entirely new AI architectures, from ultra-efficient edge devices to models that learn with far less data. “We’re at the beginning of a fundamental shift,” says Dr. Priyanka Raina, the Stanford professor leading the Onyx project. “Hardware that understands zeros isn’t just an optimization—it’s a paradigm change.”
Why AI Is Drowning in Zeros
Neural networks, the engines behind modern AI, are essentially giant spreadsheets of numbers. These numbers—called weights and activations—are stored in matrices (2D arrays) or tensors (higher-dimensional arrays). In a dense model, every cell in these arrays holds a meaningful value. But in reality, most cells are zero or near-zero. This phenomenon, known as sparsity, is baked into the math of AI.
Take a social network graph: if you represent friendships as a matrix where each row and column is a user, the matrix will be overwhelmingly empty. Most people aren’t friends with most other people, so most cells are zero. This “natural sparsity” appears in recommendation systems (Netflix’s movie preferences), search engines (Google’s page rankings), and even language models (where many word combinations never occur).
Sparsity can also be induced. In 2023, AI chipmaker Cerebras demonstrated that up to 80% of parameters in Meta’s Llama 7B model could be zeroed out without losing accuracy. The trick? Pruning unimportant connections during training, then fine-tuning the remaining weights. The result: a model that’s 70% smaller and three times faster, with identical performance.
But here’s the catch: zeros are only useful if hardware can exploit them. Today’s GPUs and CPUs are designed for dense computations, where every cell is treated equally. When they encounter a sparse matrix, they still perform all the multiplications—even the ones that result in zero. “It’s like a factory that keeps running its assembly line even when there’s nothing on the conveyor belt,” explains Raina. “You’re burning energy for no reason.”
How Sparsity Can Slash AI’s Energy Bill
Sparse computation’s power comes from two simple properties of zero:
- Multiplication by zero is always zero. No need to perform the operation.
- Adding zero changes nothing. Skip the addition entirely.
In a dense matrix-vector multiplication (a core AI operation), a 4×4 matrix and a 4-element vector require 16 multiplications and 16 additions. But if the matrix is 75% sparse (three out of four cells are zero), only 4 multiplications and 4 additions are needed. The rest can be skipped.
Sparsity also enables compression. A dense 4×4 matrix occupies 16 memory slots. A sparse version stores only the nonzero values and their coordinates, reducing memory usage to as few as 3 slots (plus metadata). For a trillion-parameter model, that’s the difference between petabytes and terabytes of storage—and a corresponding drop in energy needed to move data.
The savings compound at scale. Training a single LLM can consume 1,287 MWh of electricity, equivalent to the annual usage of 120 U.S. Households. If sparsity can cut that by 70%, as Cerebras demonstrated, the environmental and financial benefits are staggering. “This isn’t just about efficiency,” says Raina. “It’s about making AI sustainable.”
Why GPUs and CPUs Fail at Sparsity
Despite its promise, sparsity has remained a niche tool due to the fact that today’s hardware isn’t built for it. GPUs, the workhorses of AI, excel at dense parallel computations but struggle with sparse data’s irregularity. Nvidia’s Ampere GPUs support structured sparsity—where zeros follow a predictable pattern (e.g., every other cell)—but most AI models benefit more from unstructured sparsity, where zeros appear randomly.
CPUs fare better but still hit bottlenecks. Apple’s M1 and M2 chips include prefetchers optimized for sparse data, but general-purpose CPUs waste cycles on indirect memory lookups. “It’s like trying to navigate a city with a map that only shows highways,” says Raina. “You’ll get there eventually, but it’s not efficient.”
Other companies are racing to fill the gap. Cerebras’s Wafer Scale Engine, a chip the size of a dinner plate, achieves 70% sparsity in LLMs but only for weights, not activations. Meta’s MTIA v2 accelerator claims a 7x speedup for sparse computations, but its sparsity support is limited to matrix multiplications. “The industry is stuck in halfway solutions,” says Raina. “We need hardware that handles sparsity end-to-end.”
Onyx: The First Chip Built for Zeros
Enter Onyx, the Stanford accelerator that treats zeros as a feature, not a bug. Built on a coarse-grained reconfigurable array (CGRA), Onyx bridges the gap between the flexibility of FPGAs and the efficiency of CPUs. Its secret? Programmable tiles that adapt to sparse or dense computations on the fly.
Here’s how it works:
- Memory tiles store compressed sparse matrices, eliminating zero storage.
- Processing element (PE) tiles perform computations only on nonzero values, skipping zeros entirely.
- The Onyx compiler translates software instructions into hardware configurations, optimizing for sparsity patterns.
In benchmarks against a 12-core Intel Xeon CPU, Onyx achieved a 565x improvement in energy-delay product (a measure of efficiency). For sparse workloads, it used 1/70th the energy of a CPU and ran 8x faster. For dense workloads, it reconfigured itself to mimic GPU-like parallelism. “It’s the first chip that doesn’t force a trade-off,” says Raina.
Onyx’s programmability is its superpower. Unlike fixed-function accelerators, it can handle a wide range of operations, from matrix multiplications to nonlinear layers like softmax. This flexibility could make it the Swiss Army knife of AI hardware, adapting to everything from LLMs to computer vision models.
What’s Next: A Sparser Future
The Onyx team is already working on next-gen chips that push sparsity further. Key challenges include:
- Full-model sparsity: Extending support beyond matrix multiplications to all AI operations (e.g., normalization, attention layers).
- Hybrid architectures: Seamlessly switching between sparse and dense computations within the same model.
- Multi-chip scaling: Distributing sparse computations across multiple accelerators to handle larger models.
Long-term, sparsity could redefine AI’s trajectory. Today’s models rely on brute-force scaling—more data, more parameters, more energy. Sparsity-aware hardware flips that script, enabling models that are smaller, faster, and greener without sacrificing performance. “This is how we break the scaling laws,” says Raina. “Not by making models bigger, but by making them smarter.”
For now, the industry is watching closely. If Onyx and similar accelerators gain traction, they could democratize AI, putting powerful models in the hands of researchers, startups, and even edge devices. The zeros, once a computational nuisance, may yet grow AI’s unsung heroes.
Key Takeaways
- Sparsity is everywhere: Up to 80% of parameters in AI models are zeros or near-zeros, offering huge efficiency gains.
- Hardware is the bottleneck: GPUs and CPUs waste energy processing zeros, but new accelerators like Onyx skip them entirely.
- Onyx’s breakthrough: The first programmable chip to exploit sparsity, achieving 565x efficiency gains over CPUs.
- Environmental impact: Sparsity could cut AI’s energy use by 70%, making models more sustainable.
- Future potential: Sparsity-aware hardware could enable smaller, faster, and greener AI models across industries.
What do you think? Could sparsity be the key to sustainable AI, or is it just another optimization fad? Share your thoughts in the comments and join the conversation on X.