logo
episode-header-image
Mar 2024
58m 25s

#454: Data Pipelines with Dagster

MICHAEL KENNEDY
About this episode
Do you have data that you pull from external sources or is generated and appears at your digital doorstep? I bet that data needs processed, filtered, transformed, distributed, and much more. One of the biggest tools to create these data pipelines with Python is Dagster. And we are fortunate to have Pedram Navid on the show this episode. Pedram is the Head of ... Show More
Up next
Yesterday
#541: Monty - Python in Rust for AI
When LLMs write code to accomplish a task, that code has to actually run somewhere. And right now, the options aren't great. Spin up a sandboxed container and you're paying a full second of cold start overhead plus the complexity of another service. Let the LLM loose on your actu ... Show More
1h 5m
Mar 13
#540: Modern Python monorepo with uv and prek
Monorepos -- you've heard the talks, you've read the blog posts, maybe you've seen a few tantalizing glimpses into how Google or Meta organize their massive codebases. But it's often in the abstract and behind closed doors. What if you could crack open a real, production monorepo ... Show More
1h 2m
Mar 6
#539: Catching up with the Python Typing Council
You're adding type hints to your Python code, your editor is happy, autocomplete is working great. But then you switch tools and suddenly there are red squiggles everywhere. Who decides what a float annotation actually means? Or whether passing None where an int is expected shoul ... Show More
1h 1m
Recommended Episodes
Sep 2021
Massively Parallel Data Processing In Python Without The Effort Using Bodo
<div class="wp-block-jetpack-markdown"><h2>Summary</h2> <p>Python has beome the de facto language for working with data. That has brought with it a number of challenges having to do with the speed and scalability of working with large volumes of information.There have been many ... Show More
1h 4m
Feb 2023
Shorten the distance between production data and insight
<p>Modern networked applications generate a lot of data, and every business wants to make the most of that data. Most of the time, that means moving production data through some transformation process to get it ready for the analytics process. But what if you could have in-app an ... Show More
20m 27s
Mar 2024
Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+
<h2>Summary</h2> <p>A core differentiator of Dagster in the ecosystem of data orchestration is their focus on software defined assets as a means of building declarative workflows. With their launch of Dagster+ as the redesigned commercial companion to the open source project t ... Show More
55m 40s
Apr 2024
Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer
<h2>Summary</h2> <p>Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the organization. In order to enable this use case, while mainta ... Show More
56m 23s
Oct 2023
#628: Data on EKS
Organizations use their data to make better decisions and build innovative experiences for their customers. With the exponential growth in data, and the rapid pace of innovation in machine learning (ML), there is a growing need to build modern data applications that are agile and ... Show More
20m 56s
Feb 2024
G: The World's Smartest Animal
<p>This episode begins with a rant. This rant, in particular, comes from Dan Engber - a science writer who loves animals but despises animal intelligence research. Dan told us that so much of the way we study animals involves tests that we think show a human is smart ... not the ... Show More
50m 20s
Jan 2024
Introducing On This Day in Working Class History: A new daily podcast from WCH
Introducing a brand-new daily podcast from the team at WCH. On This Day in Working Class History will be a brief reminder each morning of our collective struggles for a better world which have taken place on this date in history.<br />Launching on 1 February on a trial basis, eac ... Show More
2m 32s
Feb 2024
Episode 108 - Diving into Amazon Q Builder with Clare Liguori
🚀 Dive into the world of AI with Morgan Willis, Principal Cloud Technologist for AWS, as she interviews Clare Liguori, a Senior Principal Software Engineer at AWS and one of the visionaries behind Amazon Q. Discover the secrets behind this groundbreaking Generative AI conversati ... Show More
48m 6s