episode-header-image

Yesterday

1h 18m

How AWS S3 is built

About this episode

Up next

The history of servers, the cloud, and what’s next – with Oxide

Brought to You By:•⁠ Statsig ⁠ — ⁠ The unified platform for flags, analytics, experiments, and more.•⁠ Linear ⁠ — ⁠ The system for modern product development.—How have servers and the cloud evolved in the last 30 years, and what might be next? Bryan Cantrill was a distinguished e ... Show More

Being a founding engineer at an AI startup

Brought to You By:•⁠ Statsig ⁠ — ⁠ The unified platform for flags, analytics, experiments, and more. •⁠ Linear ⁠ — ⁠ The system for modern product development. —Michelle Lim joined Warp as engineer number one and is now building her own startup, Flint. She brings a strong product ... Show More

Code security for software engineers

<p><strong>Brought to You By:</strong></p><p>•⁠ <a target="_blank" href="http://statsig.com/pragmatic"><strong>Statsig</strong></a> ⁠ — ⁠ The unified platform for flags, analytics, experiments, and more. Statsig are helping make the first-ever Pragmatic Summit a reality. Join me ... Show More

Recommended Episodes

Inside Devin: The world’s first autonomous AI engineer that's set to write 50% of its company’s code by end of year | Scott Wu (CEO and co-founder of Cognition)

Scott Wu is the co-founder and CEO of Cognition, the company behind Devin—the world’s first autonomous AI software engineer. Unlike other AI coding tools, Devin works like an autonomous engineer that you can interact with through Slack, Linear, and GitHub, just like with a remote ... Show More

How Do Agility and Software Architecture Fit Together?

<p><b>In this episode of the Scrum.org Community Podcast, Kurt Bittner guest hosts and has Professional Scrum Trainers Peter Goetz and Thomas Schissler on to talk about the relationship between software architecture and Agile Teams. They discuss common misconceptions Agile teams ... Show More

Reinventing the Developer Terminal with Warp Co-Founder and CEO Zach Lloyd

For decades, the developer terminal has remained largely unchanged. But for Warp CEO and co-founder Zach Lloyd, reinventing this core tool is the key to unlocking AI agents for coding, debugging, and automating the entire development process. Zach joins Elad Gil to discuss how se ... Show More

SE Radio 615: Kent Beck on "Tidy First?"

<p><strong>Kent Beck</strong>, Chief Scientist at Mechanical Orchard, and inventor of Extreme Programming and Test-Driven Development, joins SE Radio host Giovanni Asproni for a conversation on software design based on his latest book "Tidy First?". The episode starts with explor ... Show More

This AI Tool Can Build Any SaaS App in Minutes

Episode 60: Can you really build an $8 billion SaaS startup by yourself using AI agents? Nathan Lands (https://x.com/NathanLands) sits down with Matan Grinberg (https://x.com/matansf), a physicist, AI founder, and creator of Factory AI—one of Silicon Valley’s best-kept secrets. M ... Show More

Behind the product: Replit | Amjad Masad (co-founder and CEO)

<p><strong>Amjad Masad</strong> is the co-founder and CEO of Replit, a browser-based coding environment that allows anyone to write and deploy code. Replit has 34 million users globally and is one of the fastest-growing developer communities in the world. Prior to Replit, Amjad w ... Show More

Brought to You By:

• Statsig — ⁠ The unified platform for flags, analytics, experiments, and more.

• Sonar – The makers of SonarQube, the industry standard for automated code review

• WorkOS – Everything you need to make your app enterprise ready.

—

Amazon S3 is one of the largest distributed systems ever built, storing and serving data for a significant portion of the internet. Behind its simple interfaces hides an enormous amount of engineering work, careful tradeoffs, and long-term thinking.

In this episode, I sit down with Mai-Lan Tomsen Bukovec, VP of Data and Analytics at AWS, who has been running Amazon S3 for more than a decade. Mai-Lan shares how S3 operates at extreme scale, what it takes to design for durability and availability across millions of servers, and why building for failure is a core principle.

We also go deep into how AWS approaches correctness using formal methods, how storage tiers and limits shape system design, and why simplicity remains one of the hardest and most important goals at S3’s scale.

—

Timestamps

(00:00) Intro

(01:03) S3’s scale

(03:58) How S3 started

(07:25) Parquet, Iceberg, and S3 tables

(09:46) S3 for developers

(13:37) Why AWS keeps S3 prices low

(17:10) AWS pricing tiers

(19:38) Availability and durability

(26:21) The cost of S3's consistency

(31:22) Automated reasoning and proof of correctness

(35:14) Durability at AWS scale

(39:58) Correlated failure and crash consistency

(43:22) Failure allowances

(46:04) Two opposing principles in S3 design

(49:09) S3’s evolution

(52:21) S3 Vectors

(1:01:16) The 50 TB limit on AWS

(1:07:54) The simplicity principle

(1:10:10) Types of engineers working on S3

(1:14:15) Closing recommendations

—

The Pragmatic Engineer deepdives relevant for this episode:

• Inside Amazon’s engineering culture

• How AWS deals with a major outage

• A Day in the Life of a Senior Manager at Amazon

• What is a Principal Engineer at Amazon? – with Steve Huynh

• Working at Amazon as a software engineer – with Dave Anderson

Amazon papers recommended by Mai-Lan:

• Using lightweight formal methods to validate a key-value storage node in Amazon S3

• Formally verified cloud-scale authorization

• Analyzing metastable failures

• Amazon’s engineering tenets

—

Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.

Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe