In this episode of Gradient Dissent, Lukas Biewald talks with the CEO & founder of Surge AI, the billion-dollar company quietly powering the next generation of frontier LLMs. They discuss Surge's origin story, why traditional data labeling is broken, and how their research-focused approach is reshaping how models are trained.
You’ll hear why inter-annotator agreement fails in high-complexity tasks like poetry and math, why synthetic data is often overrated, and how Surge builds rich RL environments to stress-test agentic reasoning. They also go deep on what kinds of data will be critical to future progress in AI—from scientific discovery to multimodal reasoning and personalized alignment.
It’s a rare, behind-the-scenes look into the world of high-quality data generation at scale—straight from the team most frontier labs trust to get it right.
Timestamps:
00:00 – Intro: Who is Edwin Chen?
03:40 – The problem with early data labeling systems
06:20 – Search ranking, clickbait, and product principles
10:05 – Why Surge focused on high-skill, high-quality labeling
13:50 – From Craigslist workers to a billion-dollar business
16:40 – Scaling without funding and avoiding Silicon Valley status games
21:15 – Why most human data platforms lack real tech
25:05 – Detecting cheaters, liars, and low-quality labelers
28:30 – Why inter-annotator agreement is a flawed metric
32:15 – What makes a great poem? Not checkboxes
36:40 – Measuring subjective quality rigorously
40:00 – What types of data are becoming more important
44:15 – Scientific collaboration and frontier research data
47:00 – Multimodal data, Argentinian coding, and hyper-specificity
50:10 – What's wrong with LMSYS and benchmark hacking
53:20 – Personalization and taste in model behavior
56:00 – Synthetic data vs. high-quality human data
Follow Weights & Biases:
https://twitter.com/weights_biases
https://www.linkedin.com/company/wandb