English-language podcasts

Semi Doped

3D Stacked DRAM for Inference: d-Matrix's Sudeep Bhoja

76 min

Listen through the publisher, then discuss the episode here or arrange an in-person Roundtable. Your city determines where to meet.

Roundtables are designed for groups of four, with a shared subject and room for everyone to speak. Come for discussion, with no dating expectation.

The fundamental physics of data movement are forcing a shift to new memory architectures for AI inference. Sudeep Bhoja, co-founder and CTO of d-Matrix, joins Vik to detail the company's journey from a non-consensus bet on memory-bound inference to a full-rack 3D-stacked DRAM solution. Sudeep explains the pivot from analog to digital in-memory compute, the system-level disaggregation strategies for their Corsair card, and how the next-generation Raptor chip integrates with Nvidia's ecosystem. "Essentially, we got 16 times the capacity of SRAM, while we retained SRAM bandwidth." — Sudeep Bhoja, d-Matrix When we sat down with Crusoe’s product team, Austin asked: what will you do with my data? Their answer: They don’t touch or see it. Your training data is encrypted so even Crusoe’s own admins can’t see it, and the fine-tuned model is yours. Run a few versions, compare them, and download the one that works best. Accounts are free, and get $5 in credit when you sign up. Try Serverless Fine-Tuning: https://semidoped.com/crusoe Key Takeaways: - The core problem in AI inference is a 1000x energy gap. Compute costs femtojoules per operation, but moving data from memory costs picojoules per bit. - d-Matrix's first analog in-memory compute chip failed because it couldn't maintain 32-bit accumulation precision, forcing a pivot to a digital architecture. - The Corsair card's digital in-memory compute delivers 150 TB/s of on-chip SRAM bandwidth — over 8x more than an Nvidia Blackwell GPU's 18 TB/s HBM. - The next-gen Raptor chip stacks a logic die on a DRAM die, delivering 16x the capacity (32 GB) of the prior SRAM design while retaining ~100 TB/s bandwidth. - This 3D stacking is over 10x more energy-efficient than HBM4, consuming just 0.37 pJ/bit versus an estimated 5 pJ/bit for an HBM4-plus-GPU system. - Choosing the stacking order is a 'pick your poison' trade-off. d-Matrix put logic-on-top for better thermals, accepting a harder power delivery challenge through the DRAM. - The go-to-market product is a full rack with 144 Raptor packages, aggregating 2.3 TB of high-bandwidth memory designed to drop into Nvidia's VR rack infrastructure. This episode is supported by Meter. You have the GPUs and the power. Chapters: 0:00 Intro: The Cost of a Token 3:28 The 2019 Bet on Inference 7:43 Nighthawk: Analog In-Memory Compute 12:08 Why Analog Failed 19:02 Corsair: Digital In-Memory Compute 26:04 System Strategy: Speculative Decoding 37:05 Raptor: 3D Stacked DRAM 39:40 The 'Free Lunch' of 3D Stacking 45:00 Trade-off: Logic-on-Top vs DRAM-on-Top 54:14 Go-to-Market: The Nvidia VR Rack 1:03:17 The Future: Lightning & 4-High Stacks 1:12:45 The End State of Inference Follow Chipstrat: Newsletter: https://www.chipstrat.com X: https://x.com/chipstrat Follow Vik: Newsletter: https://www.viksnewsletter.com/ X: https://x.com/vikramskr Follow Semi Doped:

Episode information is provided by the publisher. Listening takes place on the publisher’s service.

Listen at the source ↗

Featured places to meet in Palo Alto, CA

Explore these places and arrange a conversation through ThinkMatch.

City guide for Palo Alto, CA

More from Semi Doped

Semi Doped

Google Built Two New TPUs Without New Silicon: The 8i Max and 8TX

54 min

Austin Lyons and Vik Sekar break down Google's two new TPU systems, the 8i Max and 8TX. These are not new silicon but new system configurations of the existing V8 chip, designed for specific workloads. The conversation covers the 8i Max's use of advanced packaging to increase density and the 8TX's swap to an AMD CPU to enable confidential computing for multi-tenant cloud customers. Accounts are free, and get $5 in credit when you sign up. Try Serverless Fine-Tuning: https://semidoped.com/crusoe Key Takeaways: - Google's new TPUs are system-level innovations, not new silicon. The 8i Max and 8TX are both built around the existing V8 chip, signaling a strategy of diversifying offerings around a single generation. - The 8i Max is a packaging play. It places two 8i chips on one module to double on-chip SRAM to ~768 MB, creating a single logical processor for long-context and agentic workloads. - The 8TX system is built for security, not performance. It combines 8T training silicon with an i

Discuss this episode

Semi Doped

Jensen Huang Resigns

32 min

Jensen Huang resigned as Nvidia CEO this morning, and Austin and Vik recorded two hours after the news broke. They argue that a board must prioritize corporate integrity over short-term shareholder value, even for a legendary founder. The discussion then turns to Nvidia's critical lack of a succession plan and the long-term risk of the company's unique, founder-led culture decaying into bureaucracy. Accounts are free, and get $5 in credit when you sign up. Try Serverless Fine-Tuning: https://semidoped.com/crusoe Key Takeaways: - A board's first duty is governance, not shareholder value. Even a founder CEO must be forced out for breaking company rules, because keeping them sends a message that lying and rule-breaking are acceptable. - The hardest job demands the highest standard. The immense pressure of being Nvidia's CEO is the reason for strict conduct rules, not an excuse for leniency. - Nvidia has no succession plan — Ian Buck is the only plausible internal CEO candidate. Unlike com

Discuss this episode

Semi Doped

Intel's Earnings Disaster: Bad Luck, Bad Planning, or Both?

58 min

Intel just had its worst day on the stock market in 55 years, with shares falling nearly 30%. But here's the strange part: Intel actually reported a strong quarter. Revenue grew, its data center business surged, and the company beat expectations. So what went wrong? In this episode of Semi Doped, Vik Sekar and Austin Lyons dig into Intel's Q3 2026 earnings and the problems behind its disappointing outlook. Three issues are at the center of Intel's troubles: a weakening PC market driven by rising memory prices, a shortage of advanced packaging substrates that limits server CPU production, and a tragic factory accident in Israel that disrupted chiplet supply. But how much of this was bad luck, and how much was poor planning? We debate whether Intel should have anticipated the surge in demand for server CPUs from agentic AI, whether its manufacturing strategy can respond quickly enough, and what these problems mean for its ambitions to become a leading-edge foundry alternative to TSMC. We

Discuss this episode

Community Poll

Public information should be withheld only when there is a clear reason for confidentiality.

Share your view, then see how readers voted. No sign-in required. This poll is shared across stories.

Do you agree or disagree?

Choose one answer

Common Quandary

A Common Quandary to discuss

Another conversation to explore, separate from this news story.

Your family wants an AI to settle an old argument. Would you hand over the group chat?

A recurring family dispute has reached the point where nobody agrees on what was actually said. Your cousin proposes uploading five years of the family group chat to an AI that can reconstruct events, flag contradictions, and recommend a fair resolution. Nearly everyone agrees—but one relative refuses.

Can a tool make a family argument fair without turning intimacy into evidence?

Explore this quandary