Karpathy Thinks 1B Will Be Enough. TwIL-8B, Held Up as Proof, Checked on a Mac Studio

By Sofia Reyes A viral post quotes Andrej Karpathy saying today’s AI models waste a ton of capacity remembering things that don’t matter, and that a one-billion-parameter model can be much more effective than most of them. Most of that is his words: on the No Priors podcast, published 5 September 2024, he said current models are “wasting a ton of capacity remembering stuff that doesn’t matter” and “I think even a billion suffices. We’ll probably get to that point.” Novel Cognition found that the post’s “90%” and its comparison with today’s models are not in that clip, and that Karpathy framed the idea as a prediction resting on distillation from much bigger models. The post held up webAI’s new open-weights TwIL-8B as the idea made real. Novel Cognition checked its numbers, read its licence and ran it on a Mac Studio. The full analysis is at cogcore.novcog.us.com. What others said Stout’s post passed 640,000 views in two days.

This excerpt comes from the named source. Read the original for the full reporting.