logo
episode-header-image
Apr 2024
33m 52s

How Apache Kafka got faster by switching...

Hussein Nasser
About this episode

https://oscourse.win


Allegro improved their Kafka produce tail latency by over 80% when they switched from ext4 to xfs. What I enjoyed most about this article is the detailed analysis and tweaking the team made to ext4 before considering switching to xfs. This is a classic case of how a good tech blog looks like in my opinion. 


0:00 Intro

0:30 Summary

2:35 How Kafka Works?

5:00 Producers Writes are Slow

7:10 Tracing Kafka Protocol 

12:00 Tracing Kernel System Calls

16:00 Journaled File Systems

21:00 Improving ext4

26:00 Switching to XFS



Blog

https://blog.allegro.tech/2024/03/kafka-performance-analysis.html



Up next
Jun 13
kTLS - Kernel level TLS
Fundamentals of Operating Systems Course https://oscourse.winktls is brilliant.TLS encryption/decryption often happens in userland. While TCP lives in the kernel. With ktls, userland can hand the keys to the kernel and the kernel does crypto. When calling write, the kernel encryp ... Show More
22m 55s
May 9
The beauty of the CPU
If you are bored of contemporary topics of AI and need a breather, I invite you to join me to explore a mundane, fundamental and earthy topic.The CPU.A reading of my substack article https://hnasr.substack.com/p/the-beauty-of-the-cpu 
9m 38s
Apr 2025
Sequential Scans in Postgres just got faster
This new PostgreSQL 17 feature is game changer. They know can combine IOs when performing sequential scan. Grab my database coursehttps://courses.husseinnasser.com 
27m 36s
Recommended Episodes
Mar 2020
Easier Stream Processing On Kafka With ksqlDB
Summary Building applications on top of unbounded event streams is a complex endeavor, requiring careful integration of multiple disparate systems that were engineered in isolation. The ksqlDB project was created to address this state of affairs by building a unified layer on top ... Show More
43m 36s
Aug 2023
Streaming alternatives to Kafka
Yaniv Ben Hemo (@yanivbh1, Founder/CEO at @memphis_Dev) talks about Memphis Cloud, an alternative architecture to delivering streaming data for applications.  SHOW: 747CLOUD NEWS OF THE WEEK - http://bit.ly/cloudcast-cnotwNEW TO CLOUD? CHECK OUT - "CLOUDCAST BASICS"SHOW SPONSORS: ... Show More
36m 53s
Dec 2023
708: How We Made Syntax.fm Faster
In this episode of Syntax, Wes and Scott talk through the ways they improved performance on the Syntax.fm website, how they knew it was slow to begin with, and the various changes they made to caching, and loading transcripts to improve the speed of the site. Show Notes 00:25 Wel ... Show More
27m 36s
Mar 2024
747: Middleware Explained
Join Wes and Scott for a hasty episode of Syntax as they unpack the power of middleware for developers, covering caching, authentication, A/B testing, error handling, and user redirection in a quick, punchy rundown to supercharge your backend skills! Show Notes 00:00 Welcome to S ... Show More
27m 45s
May 2022
From Kubernetes to PaaS - now what? (Ship It! #51)
Today we talk to Mark Ericksen about all the things that we could be doing on the new platform - this is a follow-up to episode 50. Mark specialises in Elixir, he hosts the Thinking Elixir podcast, and he also helps make Fly.io the best place to run Phoenix apps, such as changelo ... Show More
58m 10s
Aug 2022
Tech job interview support group
Struggling through the tech job interview process? We feel you! On this episode, Amal, Nick & Amelia get together to discuss the various ways the interview process disappoints, share their own interview stories, and suggest ways we can improve the process for everyone. Discuss on ... Show More
1h 4m
Mar 2024
Speed will win the AI computing battle with Tuhin Srivastava from Baseten
At a time when users are being asked to wait unthinkable seconds for AI products to generate art and answers, speed is what will win the battle heating up in AI computing. At least according to today’s guest, Tuhin Srivastava, the CEO and co-founder of Baseten which gives custome ... Show More
38m 32s
Oct 2023
Pushing ntfy to the next level (Changelog Interviews #562)
This week Jerod goes solo with Philipp Heckel, creator of ntfy, to discuss this simple HTTP-based service that lets you send notifications to your phone or desktop via scripts from any computer. They discuss why he built it, how he built it, and what his plans are for the future ... Show More
58m 21s
Sep 2023
Modernizing packages to ESM
Mark Erikson (web dev professor/historian, OSS Maintainer & engineer at Replay) joins us to talk about the shift from CommonJS to ESM. We discuss the history of module patterns in JS and the grueling effort to push the world’s biggest developer ecosystem forward. Get ready to go ... Show More
1h 5m
Aug 2023
Building a Reliable Kafka Data Processing Pipeline With Lily Mara
In today's episode Thomas Betts talks to Lily Mara, Engineering Manager at OneSignal in San Mateo, California. She manages the infrastructure services team, which is responsible for in-house services used by other OneSignal engineering teams. They will discuss how to build a reli ... Show More
34m 28s