logo
episode-header-image
Mar 2023
1h 9m

How Discord Stores Trillions of Messages...

Hussein Nasser
About this episode

Discord engineering goes into details of how they migrated from Cassandra to ScyllaDB, improved the performance of their reads and writes and rearchitected their backend to support the new load. It is an interesting episode lets get into it

0:00 Intro

1:50 Relational vs Distributed

7:00 The Cassandra Troubles

11:00 SnowFlake vs UUID

14:30 B+Tree

19:20 B+Tree and SSDs

25:30 LSM Trees

31:00 Hot partitions

36:00 Cassandra Garbage Collector Pauses

40:00 Changing the Architecture

45:00 The Data Services

55:00 The Migration

1:02:00 Zoned Named Spaces

1:04:00 Summary

Article here How Discord Stores Trillions of Messages

https://discord.com/blog/how-discord-stores-trillions-of-messages

Up next
Jul 10
How a query optimization gave birth to infinite scroll
The infinite scroll in social media as we know it today is despised by all. But it always start as a bad idea. It originated as a result of UI change to relief the backend database from inefficient queries. I explore this in this episode of the backend engineering. 0:00 Intro1:00 ... Show More
21m 20s
Jun 17
Postgres is half as fast in Linux 7.0
An aws engineer discovered a 50% regression in postgres throughput while testing the new Linux 7.0 kernel. The cause turns out to be massive TLB and page faults exacerbated by Postgres process-based design. In this backend engineering show episode I dive deep into how this was di ... Show More
34m 8s
Jun 8
Don't let AI rob you
A discussion about why many engineers still love the struggle, the mistakes, and the process of figuring things out themselves. This is how we grow and get better and stronger. Letting AI do everything (even though it can’t) robs us this feeling.. 
31m 53s
Recommended Episodes
Feb 2021
A Discussion on Discord
Discord started out as a useful tool for gamers, but today lots of people who have never played a video game rely on it to stay in touch. We look at the history and evolution of Discord. Learn more about your ad-choices at https://www.iheartpodcastnetwork.comSee omnystudio.com/li ... Show More
47m 16s
Aug 2024
#681: Amazon DynamoDB Deep Dive
Simon is joined by Jason Hunter, AWS Principal Specialist Solutions Architect, do dive super-deep into how to make the most of DynamoDB. Whether you are new to DynamoDB, or have been using it for years - there is something in this episode for everyone! Shownotes: Jason's Blog Pos ... Show More
48m 56s
Aug 2021
#467: [INTRODUCING] Amazon MemoryDB for Redis
Amazon MemoryDB for Redis is the newest fully managed database service from AWS. Today, Nikki is joined by Zach Gardner, Specialist Solutions Architect at AWS, to introduce this new Redis-compatible, durable, in-memory database service. Learn why we built MemoryDB and dive into b ... Show More
29m 36s
Dec 2023
708: How We Made Syntax.fm Faster
In this episode of Syntax, Wes and Scott talk through the ways they improved performance on the Syntax.fm website, how they knew it was slow to begin with, and the various changes they made to caching, and loading transcripts to improve the speed of the site. Show Notes 00:25 ... Show More
27m 36s
May 2022
From Kubernetes to PaaS - now what? (Ship It! #51)
Today we talk to Mark Ericksen about all the things that we could be doing on the new platform - this is a follow-up to episode 50. Mark specialises in Elixir, he hosts the Thinking Elixir podcast, and he also helps make Fly.io the best place to run Phoenix apps, such as chang ... Show More
58m 10s
Dec 2024
#491: DuckDB and Python: Ducks and Snakes living together
See the full show notes for this episode on the website at <a href="https://talkpython.fm/491">talkpython.fm/491</a> 
1h 2m
Mar 2022
ONE MORE thing every dev should know (Changelog Interviews #483)
The incomparable Jessica Kerr is back with another grab-bag of amazing topics. We talk about her journey to Honeycomb, devs getting satisfaction from the code they write, why step one for her is "get that new project into production" and step two is observe it, her angst for the ... Show More
1h 14m
Apr 2025
The AI 2027 scenario (Changelog News #139)
Daniel Kokotajlo and the AI Futures Project lays out a potential scenario of superhuman AI's impact, Liam ERD generates beautiful, interactive ER diagrams from your database, Mozilla takes on Gmail with "Thundermail", algernon explains why grepping remains terrible & Vitor M. de ... Show More
9m 4s