3D Stacked DRAM for Inference: d-Matrix's Sudeep Bhoja

Listen through the publisher, then discuss the episode here or arrange an in-person Roundtable. Your city determines where to meet.
Roundtables are designed for groups of four, with a shared subject and room for everyone to speak. Come for discussion, with no dating expectation.
The fundamental physics of data movement are forcing a shift to new memory architectures for AI inference. Sudeep Bhoja, co-founder and CTO of d-Matrix, joins Vik to detail the company's journey from a non-consensus bet on memory-bound inference to a full-rack 3D-stacked DRAM solution. Sudeep explains the pivot from analog to digital in-memory compute, the system-level disaggregation strategies for their Corsair card, and how the next-generation Raptor chip integrates with Nvidia's ecosystem. "Essentially, we got 16 times the capacity of SRAM, while we retained SRAM bandwidth." — Sudeep Bhoja, d-Matrix When we sat down with Crusoe’s product team, Austin asked: what will you do with my data? Their answer: They don’t touch or see it. Your training data is encrypted so even Crusoe’s own admins can’t see it, and the fine-tuned model is yours. Run a few versions, compare them, and download the one that works best. Accounts are free, and get $5 in credit when you sign up. Try Serverless Fine-Tuning: https://semidoped.com/crusoe Key Takeaways: - The core problem in AI inference is a 1000x energy gap. Compute costs femtojoules per operation, but moving data from memory costs picojoules per bit. - d-Matrix's first analog in-memory compute chip failed because it couldn't maintain 32-bit accumulation precision, forcing a pivot to a digital architecture. - The Corsair card's digital in-memory compute delivers 150 TB/s of on-chip SRAM bandwidth — over 8x more than an Nvidia Blackwell GPU's 18 TB/s HBM. - The next-gen Raptor chip stacks a logic die on a DRAM die, delivering 16x the capacity (32 GB) of the prior SRAM design while retaining ~100 TB/s bandwidth. - This 3D stacking is over 10x more energy-efficient than HBM4, consuming just 0.37 pJ/bit versus an estimated 5 pJ/bit for an HBM4-plus-GPU system. - Choosing the stacking order is a 'pick your poison' trade-off. d-Matrix put logic-on-top for better thermals, accepting a harder power delivery challenge through the DRAM. - The go-to-market product is a full rack with 144 Raptor packages, aggregating 2.3 TB of high-bandwidth memory designed to drop into Nvidia's VR rack infrastructure. This episode is supported by Meter. You have the GPUs and the power. Chapters: 0:00 Intro: The Cost of a Token 3:28 The 2019 Bet on Inference 7:43 Nighthawk: Analog In-Memory Compute 12:08 Why Analog Failed 19:02 Corsair: Digital In-Memory Compute 26:04 System Strategy: Speculative Decoding 37:05 Raptor: 3D Stacked DRAM 39:40 The 'Free Lunch' of 3D Stacking 45:00 Trade-off: Logic-on-Top vs DRAM-on-Top 54:14 Go-to-Market: The Nvidia VR Rack 1:03:17 The Future: Lightning & 4-High Stacks 1:12:45 The End State of Inference Follow Chipstrat: Newsletter: https://www.chipstrat.com X: https://x.com/chipstrat Follow Vik: Newsletter: https://www.viksnewsletter.com/ X: https://x.com/vikramskr Follow Semi Doped:
Episode information is provided by the publisher. Listening takes place on the publisher’s service.
Listen at the source ↗








