Craig Smith sits down with Ilya Sutskever - Co-Founder and Chief Scientist at Safe Superintelligence Inc. & former chief scientist at OpenAI and one of the primary minds behind GPT-3, GPT-4, and the deep learning revolution that preceded it - for a conversation that covers the intellectual history of deep learning and the hardest current questions simultaneously. Ilya's most important argument is one that most public discourse gets wrong: the critique that LLMs "just learn statistical patterns" misses what prediction at scale actually achieves. To predict text well - to truly compress it - a model must develop understanding of the true underlying processes that produced it. As models improve, he argues, that understanding will reach a "shocking degree," and the Bing/Sydney episode, where the system became combative when a user expressed preference for Google, is his evidence that psychological language is already the right frame for what these systems have internalized.
On hallucinations, Ilya is precise and unusually optimistic. He draws a clean distinction: the pre-training objective produces rich world knowledge but doesn't optimize for accurate outputs, it optimizes for statistically plausible ones. Reinforcement learning from human feedback addresses this directly, teaching the model to update away from outputs that users flag as wrong. His assessment: "quite a high chance" of solving hallucinations completely. The conversation also covers the transformer moment (switching from recurrent networks the day after the paper dropped), why Rich Sutton's "bitter lesson" overstates the case for pure scaling, and a genuinely surprising vision of AI's role in democracy, where citizens might use high-bandwidth interactions with AI to specify their values directly, enabling a more granular form of political participation than voting alone can provide.
Subscribe to Eye on A.I. for weekly conversations with the people building and deploying the future of AI.