MLA 021 Databricks: Cloud Analytics and ...

Raybeam and Databricks

• Raybeam is a data science and analytics company, recently acquired by Dept Agency.
• While Raybeam focuses on data analytics, its acquisition has expanded its expertise into ML Ops and AI.
• The company recommends tools based on client requirements, frequently utilizing Databricks for its comprehensive nature.

Understanding Databricks

• Databricks is not merely an analytics platform; it is a competitor in the ML Ops space alongside tools like SageMaker and Kubeflow.
• It provides interactive notebooks, Python code execution, and runs on a hosted Apache Spark cluster.
• Databricks includes Delta Lake, which acts as a storage and data management layer.

Choosing the Right MLOps Tool

• Raybeam evaluates each client’s needs, existing expertise, and infrastructure before recommending a platform.
• Databricks, SageMaker, Kubeflow, and Snowflake are common alternatives, with the final selection dependent on current pipelines and operational challenges.
• Maintaining existing workflows is prioritized unless scalability or feature limitations necessitate migration.

Databricks Features

• Databricks is accessible via a web interface similar to Jupyter Hub and can be integrated with local IDEs (e.g., VS Code, PyCharm) using Databricks Connect.
• Notebooks on Databricks can be version-controlled with Git repositories, enhancing collaboration and preventing data loss.
• The platform supports configuration of computing resources to match model size and complexity.
• Databricks clusters are hosted on AWS, Azure, or GCP, with users selecting the underlying cloud provider at sign-up.

Parquet and Delta Lake

• Parquet files store data in a columnar format, which improves efficiency for aggregation and analytics tasks.
• Delta Lake provides transactional operations on top of Parquet files by maintaining a version history, enabling row edits and deletions.
• This approach offers a database-like experience for handling large datasets, simplifying both analytics and machine learning workflows.

Pricing and Usage

• Pricing for Databricks depends on the chosen cloud provider (AWS, Azure, or GCP) with an additional fee for Databricks’ services.
• The added cost is described as relatively small, and the platform is accessible to both individual developers and large enterprises.
• Databricks is recommended for newcomers to data science and ML for its breadth of features and straightforward setup.

Databricks, MLflow, and Other Integrations

• Databricks provides a hosted MLflow solution, offering experiment tracking and model management.
• The platform can access data stored in services like S3, Snowflake, and other cloud provider storage options.
• Integration with tools such as PyArrow is supported, facilitating efficient data access and manipulation.

Example Use Cases and Decision Process

• Migration to Databricks is recommended when a client’s existing infrastructure (e.g., on-premises Spark clusters) cannot scale effectively.
• The selection process involves an in-depth exploration of a client’s operational challenges and goals.
• Databricks is chosen for clients lacking feature-specific needs but requiring a unified data analytics and ML platform.

Personal Projects by Ming Chang

• Ming Chang has explored automated stock trading using APIs such as Alpaca, focusing on downloading and analyzing market data.
• He has also developed drone-related projects with Raspberry Pi, emphasizing real-world applications of programming and physical computing.

Additional Resources

• Databricks Homepage
• Delta Lake on Databricks
• Parquet Format
• Raybeam Overview
• MLFlow Documentation