MLA 003 Storage: HDF, Pickle, Postgres

Data Ingestion and Preprocessing

• Data Sources and Formats:

  • Datasets commonly originate as CSV (comma-separated values), TSV (tab-separated values), fixed-width files (FWF), JSON from APIs, or directly from databases.
  • Typical applications include structured data (e.g., real estate features) or unstructured data (e.g., natural language corpora for sentiment analysis).

• Pandas as the Core Ingestion Tool:

  • Pandas provides versatile functions such as read_csv, read_json, and others to load various file formats with robust options for handling edge cases (e.g., file encodings, missing values).
  • After loading, data cleaning is performed using pandas: dropping or imputing missing values, converting booleans and categorical columns to numeric form.

• Data Encoding for Machine Learning:

  • All features must be numerical before being supplied to machine learning models like TensorFlow or Keras.
  • Categorical data is one-hot encoded using pandas.get_dummies, converting strings to binary indicator columns.
  • The underlying NumPy array of a DataFrame is accessed via df.values for direct integration with modeling libraries.

Numerical Data Storage Options

• HDF5 for Storing Processed Arrays:

  • HDF5 (Hierarchical Data Format version 5) enables efficient storage of large multidimensional NumPy arrays.
  • Libraries like h5py and built-in pandas functions (to_hdf) allow seamless saving and retrieval of arrays or DataFrames.
  • TensorFlow and Keras use HDF5 by default to store neural network weights as multi-dimensional arrays for model checkpointing and early stopping, accommodating robust recovery and rollback.

• Pickle for Python Objects:

  • Python's pickle protocol serializes arbitrary objects, including machine learning models and arrays, into files for later retrieval.
  • While convenient for quick iterations or heterogeneous data, pickle is less efficient with NDarrays compared to HDF5, lacks significant compression, and poses security risks if not properly safeguarded.

• SQL Databases and Spreadsheets:

  • For mixed or heterogeneous data, or when producing results for sharing and collaboration, relational databases like PostgreSQL or spreadsheets such as CSVs are used.
  • Databases serve as the endpoint for production systems, where model outputs—such as generated recommendations or reports—are published for downstream use.

Storage Workflow in Machine Learning Pipelines

• Typical Process:

  • Data is initially loaded and processed with pandas, then converted to numerical arrays suitable for model training.
  • Intermediate states and model weights are saved using HDF5 during model development and training, ensuring recovery from interruptions and facilitating early stopping.
  • Final outputs, especially those requiring sharing or production use, are published to SQL databases or shared as spreadsheet files.

• Best Practices and Progression:

  • Quick project starts may involve pickle for accessible storage during early experimentation.
  • For large-scale, high-performance applications, migration to HDF5 for numerical data and SQL for production-grade results is recommended.
  • Alternative options like Feather and PyTables (an interface on top of HDF5) exist for specialized needs.

Summary

• HDF5 is optimal for numerical array storage due to its efficiency, built-in compression, and integration with major machine learning frameworks.
• Pickle accommodates arbitrary Python objects but is suboptimal for numerical data persistence or security.
• SQL databases and spreadsheets are used for disseminating results, especially when human consumption or application integration is required.
• The selection of a storage format is determined by data type, pipeline stage, and end-use requirements within machine learning workflows.