Had so much fun chatting with my good friends Trenton Bricken and Sholto Douglas on the podcast.
No way to summarize it, except:
This is the best context dump out there on how LLMs are trained, what capabilities they're likely to soon have, and what exactly is going on inside them.
You would be shocked how much of what I know about this field, I've learned just from talking with them.
To the extent that you've enjoyed my other AI interviews, now you know why.
So excited to put this out. Enjoy! I certainly did :)
Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform.
There's a transcript with links to all the papers the boys were throwing down - may help you follow along.
Timestamps
(00:00:00) - Long contexts
(00:16:12) - Intelligence is just associations
(00:32:35) - Intelligence explosion & great researchers
(01:06:52) - Superposition & secret communication
(01:22:34) - Agents & true reasoning
(01:34:40) - How Sholto & Trenton got into AI research
(02:07:16) - Are feature spaces the wrong way to think about intelligence?
(02:21:12) - Will interp actually work on superhuman models
(02:45:05) - Sholto’s technical challenge for the audience
(03:03:57) - Rapid fire