About this episode
Mar 2
From Models to Momentum: Uniting Architects and Engineers with ER/Studio
Summary In this episode of the Data Engineering Podcast, Jamie Knowles (Product Director) and Ryan Hirsch (Product Marketing Manager) discuss the importance of enterprise data modeling with ER/Studio. They highlight how clear, shared semantic models are a foundational discipline ... Show More
45m 2s
Feb 22
From Data Models to Mind Models: Designing AI Memory at Scale
Summary In this episode of the Data Engineering Podcast, Vasilije "Vas" Markovich, founder of Cognee, discusses building agentic memory, a crucial aspect of artificial intelligence that enables systems to learn, adapt, and retain knowledge over time. He explains the concept of ag ... Show More
57m 47s
Feb 15
Prompt Management, Tracing, and Evals: The New Table Stakes for GenAI Ops
Summary In this episode of the Data Engineering Podcast, Aman Agarwal, creator of OpenLit, discusses the operational groundwork required to run LLM-powered applications reliably and cost-effectively. He highlights common blind spots that teams face, including opaque model behavio ... Show More
50m 43s
Feb 2023
Shorten the distance between production data and insight
<p>Modern networked applications generate a lot of data, and every business wants to make the most of that data. Most of the time, that means moving production data through some transformation process to get it ready for the analytics process. But what if you could have in-app an ... Show More
20m 27s
Nov 2021
Time Plus Data Equals Efficiency with Paul Dix, the Founder and CTO of InfluxData and the Creator of InfluxDB
<p>If the topic of databases is brought up to certain people, their eyes may gloss over. But if that happened, that would be because they just don’t know the awesome power of databases. Data can be valuable but only if it is contextualized, and time is an extremely relevant aspec ... Show More
36m 4s
Aug 2020
Introduction to GraphQL
<p>Tanmai Gopal (@tanmaigo, CEO Hasura) and Rajoshi Ghosh (@rajoshighosh, COO Hasura) talk about the evolution of GraphQL as an efficient way to engage with APIs and data models, and how Hasura Cloud helps simplify GraphQL for developers.</p><p><b>SHOW: </b>462<br/><br/><b>SHOW S ... Show More
40m 40s
May 2020
How Important are algorithm and data structures in backend engineering?
<p>Algorithms & Data Structures are critical to Backend Engineering however it really depends on what kind of application and infrastructure you are building. In this video I want to go through the following 1 Backend Engineers are two types - Integrating Existing ... Show More
13m 29s
Oct 2021
On Graph Databases | The Backend Engineering Show
<p>I get a lot of emails asking me to talk about graph databases, so I want to start researching them, but I wanted to give you guys the framework of how I think about any databases to defuse any “magic” that might be there.</p>
<p>In this video, I discuss what constrains a datab ... Show More
22m 27s
Mar 2024
#454: Data Pipelines with Dagster
See the full show notes for this episode on the website at <a href="https://talkpython.fm/454">talkpython.fm/454</a>
58m 25s
Oct 2023
#628: Data on EKS
Organizations use their data to make better decisions and build innovative experiences for their customers. With the exponential growth in data, and the rapid pace of innovation in machine learning (ML), there is a growing need to build modern data applications that are agile and ... Show More
20m 56s
Mar 2022
Bayesian Machine Learning with Ravin Kumar (Ep. 191)
<p>This is one episode where passion for math, statistics and computers are merged.
I have a very interesting conversation with Ravin, data scientist at Google where he uses data to inform decisions.</p>
<p>He has previously worked at Sweetgreen, designing systems that would b ... Show More
31m 12s
Mar 2022
Mining the Golden Age of Data with Tableau’s CEO & President Mark Nelson
<p><a href="https://www.linkedin.com/in/markthomasnelson/">Mark Nelson</a> is the President and CEO of <a href="https://www.tableau.com/">Tableau</a>, a company dedicated to democratizing analytics and putting data back in the hands of consumers. But while this digital pioneer ma ... Show More
36m 32s
Summary
While the overall concept of timeseries data is uniform, its usage and applications are far from it. One of the most demanding applications of timeseries data is for application and server monitoring due to the problem of high cardinality. In his quest to build a generalized platform for managing timeseries Paul Dix keeps getting pulled back into the monitoring arena. In this episode he shares the history of the InfluxDB project, the business that he has helped to build around it, and the architectural aspects of the engine that allow for its flexibility in managing various forms of timeseries data. This is a fascinating exploration of the technical and organizational evolution of the Influx Data platform, with some promising glimpses of where they are headed in the near future.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. With RudderStack you can use all of your customer data to answer more difficult questions and then send those insights to your whole customer data stack. Sign up free at dataengineeringpodcast.com/rudder today.
- We’ve all been asked to help with an ad-hoc request for data by the sales and marketing team. Then it becomes a critical report that they need updated every week or every day. Then what do you do? Send a CSV via email? Write some Python scripts to automate it? But what about incremental sync, API quotas, error handling, and all of the other details that eat up your time? Today, there is a better way. With Census, just write SQL or plug in your dbt models and start syncing your cloud warehouse to SaaS applications like Salesforce, Marketo, Hubspot, and many more. Go to dataengineeringpodcast.com/census today to get a free 14-day trial.
- Your host is Tobias Macey and today I’m interviewing Paul Dix about Influx Data and the different facets of the market for timeseries databases
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what you are building at Influx Data and the story behind it?
- Timeseries data is a fairly broad category with many variations in terms of storage volume, frequency, processing requirements, etc. This has led to an explosion of database engines and related tools to address these different needs. How do you think about your position and role in the ecosystem?
- Who are your target customers and how does that focus inform your product and feature priorities?
- What are the use cases that Influx is best suited for?
- Can you give an overview of the different projects, tools, and services that comprise your platform?
- How is InfluxDB architected?
- How have the design and implementation of the DB engine changed or evolved since you first began working on it?
- What are you optimizing for on the consistency vs. availability spectrum of CAP?
- What is your approach to clustering/data distribution beyond a single node?
- For the interface to your database engine you developed a custom query language. What was your process for deciding what syntax to use and how to structure the programmatic interface?
- How do you handle the lifecycle of data in an Influx deployment? (e.g. aging out old data, periodic compaction/rollups, etc.)
- With your strong focus on monitoring use cases, how do you handle the challenge of high cardinality in the data being stored?
- What are some of the data modeling considerations that users should be aware of as they are designing a deployment of Influx?
- What is the role of open source in your product strategy?
- What are the most interesting, innovative, or unexpected ways that you have seen the Influx platform used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Influx?
- When is Influx DB and/or the associated tools the wrong choice?
- What do you have planned for the future of Influx Data?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
- Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Links
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
