📌 HBM Shortage Forces Inference Chips to Recalculate the Memory Equation
Competition in AI inference chips is extending from “whose compute power is greater” to “who can store model weights more efficiently.” This warrants attention because high-bandwidth memory (HBM) is no longer merely a supporting component for chips; whether its supply can keep pace may already affect product configurations, costs, and delivery schedules.
Reportedly, Nvidia is testing Rubin Ultra equipped with 192GB to 256GB of HBM, below the originally planned 1TB, against a backdrop of tightening memory supply. If this configuration is ultimately adopted, it means manufacturers will need to reassess trade-offs among performance targets, available memory capacity, and product costs. For HBM suppliers, tight supply will raise their position in the AI buildout cycle; but whether chipmakers can turn this constraint into a sellable product mix will still depend on final specifications and supply conditions.
Another path is to use less external memory. Relevant analysis says Nvidia is incorporating Groq’s SRAM LPX into its technology stack; AMD, meanwhile, is exploring through Taalas the direct hardwiring of model weights into logic gates or Mask ROM. The former brings more storage capability closer to compute units, while the latter seeks to eliminate weight loading. The shared goal of both approaches is to reduce reliance on external HBM during inference.
Therefore, the next stage of verification is not only whether HBM remains persistently tight, but also whether these architectures can balance performance and deployment flexibility in real-world models. Memory efficiency may become a more direct variable in inference-chip revenue and order allocation.
▌ Sources