These engagement failures, and how to fix them, map directly onto the Octalysis Core Drives. Get the free Core Drives in the Wild guide: professorgame.com/WildCD
Episode Summary
Rob breaks down why Amazon shut down KiroRank, the internal leaderboard that scored staff on raw AI usage on its Kiro developer platform. He shows how stacking Core Drive 2 (Development & Accomplishment) and Core Drive 5 (Social Influence & Relatedness) produced flawless compliance toward the wrong target, a textbook case of Goodhart's law: once a measure becomes a target, it stops being a good measure. Drawing on the Octalysis Strategy Dashboard and Toyota's Five Whys, he lays out the one question to ask before you measure anything. Listeners learn to measure outcomes instead of activity, and how to keep a proxy metric from quietly getting gamed.
About the Host
Rob Alvarez is Head of Engagement Strategy, Europe at The Octalysis Group (TOG), a leading gamification and behavioral design consultancy. A globally recognized gamification strategist and TEDx speaker, he founded and hosts Professor Game, the #1 gamification podcast, and has interviewed hundreds of global experts. He designs evidence-based engagement systems that drive motivation, loyalty, and results, and teaches LEGO® SERIOUS PLAY® and gamification at top institutions including IE Business School, EFMD, and EBS University across Europe, the Americas, and Asia.
Key Takeaways
- Amazon shut down KiroRank, its internal leaderboard scoring staff on AI usage on the Kiro developer platform, after employees set autonomous AI agents on needless tasks just to climb the ranks and inflated the company's compute costs.
- Goodhart's law explains the failure: when a measure becomes a target, it stops being a good measure. You get what you measure, not what you want, so raw AI usage climbed while productivity went unmeasured.
- KiroRank stacked Core Drive 2 (Development & Accomplishment) through a progress bar and ranking, and Core Drive 5 (Social Influence & Relatedness) through public status, producing flawless compliance toward the wrong outcome.
- The more powerful and expensive the tool being measured, the more a gamed metric costs you, which is why Amazon paid in real compute money rather than a rounding error.
- The Octalysis Strategy Dashboard starts with business metrics by asking what outcome you actually want, using Toyota's Five Whys to move from "increase AI usage" to a result worth hitting, like productivity per employee.
- Engagement is the value created for users and the business, not click counts or usage volume, which is why most dashboards measure activity when they should measure the outcome.
Topics Covered
- 0:00 - The $200 billion AI paradox
- 0:27 - Goodhart's law and gamed metrics
- 1:49 - The two Core Drives Amazon stacked
- 2:39 - Flawless compliance, the wrong target
- 3:38 - Amazon's KiroRank AI leaderboard
- 5:11 - Measure the right thing, not usage
- 5:38 - The Octalysis Strategy Dashboard
- 6:12 - Toyota's Five Whys for metrics
- 7:21 - When proxy metrics are unavoidable
- 7:58 - Measure the outcome, not the activity
- 8:33 - Get the Core Drives in the Wild guide
Mentioned in This Episode
Free Resources and Get in Touch
