About this episode
Feb 8
From Legacy to AI-Ready: How MongoDB AMP Accelerates Modernization
SummaryIn this episode, Shilpa Kolhar, SVP of Product and Engineering at MongoDB, discusses using MongoDB as a unified foundation for AI-driven and agentic applications. She explains how the Application Modernization Platform (AMP) accelerates the transition from legacy relationa ... Show More
46m 45s
Feb 1
Branches, Diffs, and SQL: How Dolt Powers Agentic Workflows
Summary In this episode Tim Sehn, founder and CEO of DoltHub, talks about Dolt - the world’s first version‑controlled SQL database - and why Git‑style semantics belong at the heart of data systems and AI workflows. Tim explains how Dolt combines a MySQL/Postgres‑compatible interf ... Show More
56m 53s
Jan 25
Logical First, Physical Second: A Pragmatic Path to Trusted Data
Summary In this episode of the Data Engineering Podcast Jamie Knowles, Product Director for ER/Studio, talks about data architecture and its importance in driving business meaning. He discusses how data architecture should start with business meaning, not just physical schemas, a ... Show More
40m 50s
Mar 2022
Bayesian Machine Learning with Ravin Kumar (Ep. 191)
<p>This is one episode where passion for math, statistics and computers are merged.
I have a very interesting conversation with Ravin, data scientist at Google where he uses data to inform decisions.</p>
<p>He has previously worked at Sweetgreen, designing systems that would b ... Show More
31m 12s
Mar 2022
Mining the Golden Age of Data with Tableau’s CEO & President Mark Nelson
<p><a href="https://www.linkedin.com/in/markthomasnelson/">Mark Nelson</a> is the President and CEO of <a href="https://www.tableau.com/">Tableau</a>, a company dedicated to democratizing analytics and putting data back in the hands of consumers. But while this digital pioneer ma ... Show More
36m 32s
Sep 2021
From Different Leadership Vantage Points: Data Drives Value but is Driven by Values
<p>One way to think about data is that it is like rain, and it is pouring outside. Imagine c-suite executives running around in a parking lot with huge buckets trying to capture as much as they can. Afterward, they return to the office, analyze the data, and then decide what to d ... Show More
51m 50s
Nov 2021
Time Plus Data Equals Efficiency with Paul Dix, the Founder and CTO of InfluxData and the Creator of InfluxDB
<p>If the topic of databases is brought up to certain people, their eyes may gloss over. But if that happened, that would be because they just don’t know the awesome power of databases. Data can be valuable but only if it is contextualized, and time is an extremely relevant aspec ... Show More
36m 4s
Feb 2023
Shorten the distance between production data and insight
<p>Modern networked applications generate a lot of data, and every business wants to make the most of that data. Most of the time, that means moving production data through some transformation process to get it ready for the analytics process. But what if you could have in-app an ... Show More
20m 27s
Dec 2020
The Algorithms that Bring you Style with Stitch Fix’s Director of Data Science, Tatsiana Maskalevich
<p>The old saying, “look good, feel good,'' fits Stitch Fix perfectly. The direct-to-consumer, online personal styling service has boomed due to its ability to not only match consumers with trendy and comfortable clothes, but to make it a personalized experience for each buyer.</ ... Show More
52m 39s
Oct 2023
#628: Data on EKS
Organizations use their data to make better decisions and build innovative experiences for their customers. With the exponential growth in data, and the rapid pace of innovation in machine learning (ML), there is a growing need to build modern data applications that are agile and ... Show More
20m 56s
Jun 2024
Making ETL pipelines a thing of the past
<p>RelationalAI’s first <a href="https://relational.ai/resources/introducing-first-ai-coprocessor" target="_blank">big partner is Snowflake</a>, meaning customers can now start using their data with GenAI without worrying about the privacy, security, and governance hassle that wo ... Show More
26m 13s
Summary
The interfaces and design cues that a tool offers can have a massive impact on who is able to use it and the tasks that they are able to perform. With an eye to making data workflows more accessible to everyone in an organization Raj Bains and his team at Prophecy designed a powerful and extensible low-code platform that lets technical and non-technical users scale data flows without forcing everyone into the same layers of abstraction. In this episode he explores the tension between code-first and no-code utilities and how he is working to balance the strengths without falling prey to their shortcomings.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
- So now your modern data stack is set up. How is everyone going to find the data they need, and understand it? Select Star is a data discovery platform that automatically analyzes & documents your data. For every table in Select Star, you can find out where the data originated, which dashboards are built on top of it, who’s using it in the company, and how they’re using it, all the way down to the SQL queries. Best of all, it’s simple to set up, and easy for both engineering and operations teams to use. With Select Star’s data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets. Try it out for free and double the length of your free trial today at dataengineeringpodcast.com/selectstar. You’ll also get a swag package when you continue on a paid plan.
- Your host is Tobias Macey and today I’m interviewing Raj Bains about how improving the user experience for data tools can make your work as a data engineer better and easier
Interview
- Introduction
- How did you get involved in the area of data management?
- What are the broad categories of data tool designs that are available currently and how does that impact what is possible with them?
- What are the points of friction that are introduced by the tools?
- Can you share some of the types of workarounds or wasted effort that are made necessary by those design elements?
- What are the core design principles that you have built into Prophecy to address these shortcomings?
- How do those user experience changes improve the quality and speed of work for data engineers?
- How has the Prophecy platform changed since we last spoke almost a year ago?
- What are the tradeoffs of low code systems for productivity vs. flexibility and creativity?
- What are the most interesting, innovative, or unexpected approaches to developer experience that you have seen for data tools?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on user experience optimization for data tooling at Prophecy?
- When is it more important to optimize for computational efficiency over developer productivity?
- What do you have planned for the future of Prophecy?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
