Building Tools And Platforms For Data An...

Summary

Data engineers are responsible for building tools and platforms to power the workflows of other members of the business. Each group of users has their own set of requirements for the way that they access and interact with those platforms depending on the insights they are trying to gather. Benn Stancil is the chief analyst at Mode Analytics and in this episode he explains the set of considerations and requirements that data analysts need in their tools and. He also explains useful patterns for collaboration between data engineers and data analysts, and what they can learn from each other.

Announcements

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
• You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, and Data Counsil. Upcoming events include the O’Reilly AI conference, the Strata Data conference, the combined events of the Data Architecture Summit and Graphorum, and Data Council in Barcelona. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.
• Your host is Tobias Macey and today I’m interviewing Benn Stancil, chief analyst at Mode Analytics, about what data engineers need to know when building tools for analysts

Interview

• Introduction
• How did you get involved in the area of data management?
• Can you start by describing some of the main features that you are looking for in the tools that you use?
• What are some of the common shortcomings that you have found in out-of-the-box tools that organizations use to build their data stack?
• What should data engineers be considering as they design and implement the foundational data platforms that higher order systems are built on, which are ultimately used by analysts and data scientists?
  • In terms of mindset, what are the ways that data engineers and analysts can align and where are the points of conflict?
• In terms of team and organizational structure, what have you found to be useful patterns for reducing friction in the product lifecycle for data tools (internal or external)?
• What are some anti-patterns that data engineers can guard against as they are designing their pipelines?
• In your experience as an analyst, what have been the characteristics of the most seamless projects that you have been involved with?
• How much understanding of analytics are necessary for data engineers to be successful in their projects and careers?
  • Conversely, how much understanding of data management should analysts have?
• What are the industry trends that you are most excited by as an analyst?

Contact Info

• LinkedIn
• @bennstancil on Twitter
• Website

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

• Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
• To help other people find the show please leave a review on iTunes and tell your friends and co-workers
• Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat

Links

• Mode Analytics
• Data Council Presentation
• Yammer
• StitchFix Blog Post
• SnowflakeDB
• Re:Dash
• Superset
• Marquez
• Amundsen
  • Podcast Episode
• Elementl
• Dagster
  • Data Council Presentation
• DBT
  • Podcast Episode
• Great Expectations
  • Podcast.__init__ Episode
• Delta Lake
  • Podcast Episode
• Stitch
• Fivetran
  • Podcast Episode

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA