About this episode
Jan 25
Logical First, Physical Second: A Pragmatic Path to Trusted Data
Summary In this episode of the Data Engineering Podcast Jamie Knowles, Product Director for ER/Studio, talks about data architecture and its importance in driving business meaning. He discusses how data architecture should start with business meaning, not just physical schemas, a ... Show More
40m 50s
Jan 18
Your Data, Your Lake: How Observe Uses Iceberg and Streaming ETL for Observability
Summary In this episode Jacob Leverich, cofounder and CTO of Observe, talks about applying lakehouse architectures to observability workloads. Jacob discusses Observe’s decision to leverage cloud-native warehousing and open table formats for scale and cost efficiency. He digs int ... Show More
1h 12m
Jan 12
Semantic Operators Meet Dataframes: Building Context for Agents with FENIC
Summary In this episode Kostas Pardalis talks about Fenic - an open-source, PySpark-inspired dataframe engine designed to bring LLM-powered semantics into reliable data engineering workflows. Kostas shares why today’s data infrastructure assumptions (BI-first, expert-operated, CP ... Show More
56m 42s
Mar 2022
Bayesian Machine Learning with Ravin Kumar (Ep. 191)
<p>This is one episode where passion for math, statistics and computers are merged.
I have a very interesting conversation with Ravin, data scientist at Google where he uses data to inform decisions.</p>
<p>He has previously worked at Sweetgreen, designing systems that would b ... Show More
31m 12s
Mar 2022
Mining the Golden Age of Data with Tableau’s CEO & President Mark Nelson
<p><a href="https://www.linkedin.com/in/markthomasnelson/">Mark Nelson</a> is the President and CEO of <a href="https://www.tableau.com/">Tableau</a>, a company dedicated to democratizing analytics and putting data back in the hands of consumers. But while this digital pioneer ma ... Show More
36m 32s
Jan 2022
Academics and Data Science Innovation with Dr. David Bader, Distinguished Professor and Director, Institute for Data Science, New Jersey Institute of Technology
<p>The data science field is expanding because so many businesses and other institutions require skilled workers who can manage data as well as provide insights. Companies and students are clamoring for more academic programs. There is great need, but academic institutions are st ... Show More
39m 32s
Jan 2022
#498: [African Stories #2] EXPLORE Data Science Academy
The EXPLORE Data Science Academy (EDSA) is a premier data science academy in South Africa that offers students an intensive one-year training course and provides a fundamental understanding of software development and cloud computing technologies. In this episode, Simon speaks wi ... Show More
14m 33s
Nov 2021
Time Plus Data Equals Efficiency with Paul Dix, the Founder and CTO of InfluxData and the Creator of InfluxDB
<p>If the topic of databases is brought up to certain people, their eyes may gloss over. But if that happened, that would be because they just don’t know the awesome power of databases. Data can be valuable but only if it is contextualized, and time is an extremely relevant aspec ... Show More
36m 4s
Oct 2023
#628: Data on EKS
Organizations use their data to make better decisions and build innovative experiences for their customers. With the exponential growth in data, and the rapid pace of innovation in machine learning (ML), there is a growing need to build modern data applications that are agile and ... Show More
20m 56s
Jun 2018
SSR, SEO, Tips for Students, music, security and GraphQL
It’s a Potluck episode - you bring the questions and we turn them into tasty treats.
Freshbooks - Sponsor Get a 30 day free trial of Freshbooks at freshbooks.com/syntax and put SYNTAX in the “How did you hear about us?” section.
Coffeecup’s CSS Grid Builder Tool Check out Coffe ... Show More
1 h
Mar 2024
LLM Security and Privacy
<p>Sean Falconer (@seanfalconer, Head of Dev Relations @SkyflowAPI, Host @software_daily) talks about security and privacy of LLMs and how to prevent PII (personally identifiable information) from leaking out</p><p><b>SHOW: 807<br/><br/>CLOUD NEWS OF THE WEEK - </b><a href='http: ... Show More
26m 9s
Jul 2020
Nora Jones on Resilience Engineering, Mental Models, and Learning from Incidents
In this podcast, Nora Jones, Co-Founder and CEO at Jeli and co-author of O’Reilly’s “Chaos Engineering: System Resiliency in Practice”, sat down with InfoQ podcast co-host Daniel Bryant. Topics discussed included: chaos engineering and resilience engineering, planning and running ... Show More
36 m
Summary
Data Engineering is a broad and constantly evolving topic, which makes it difficult to teach in a concise and effective manner. Despite that, Daniel Molnar and Peter Fabian started the Pipeline Academy to do exactly that. In this episode they reflect on the lessons that they learned while teaching the first cohort of their bootcamp how to be effective data engineers. By focusing on the fundamentals, and making everyone write code, they were able to build confidence and impart the importance of context for their students.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at dataengineeringpodcast.com/hightouch.
- Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
- Your host is Tobias Macey and today I’m interviewing Daniel Molnar and Peter Fabian about the lessons that they learned from their first cohort at the Pipeline data engineering academy
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you start by sharing the curriculum and learning goals for the students?
- How did you set a common baseline for all of the students to build from throughout the program?
- What was your process for determining the structure of the tasks and the tooling used?
- What were some of the topics/tools that the students had the most difficulty with?
- What topics/tools were the easiest to grasp?
- What are some difficulties that you encountered while trying to teach different concepts?
- How did you deal with the tension of teaching the fundamentals while tying them to toolchains that hiring managers are looking for?
- What are the successes that you had with this cohort and what changes are you making to your approach/curriculum to build on them?
- What are some of the failures that you encountered and what lessons have you taken from them?
- How did the pandemic impact your overall plan and execution of the initial cohort?
- What were the skills that you focused on for interview preparation?
- What level of ongoing support/engagement do you have with students once they complete the curriculum?
- What are the most interesting, innovative, or unexpected solutions that you saw from your students?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working with your first cohort?
- When is a bootcamp the wrong approach for skill development?
- What do you have planned for the future of the Pipeline Academy?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
