Every breakthrough in machine learning capability has been preceded by a breakthrough in data availability. ImageNet — 1.2 million labelled images — enabled the deep learning revolution in computer vision. The Common Crawl — petabytes of web text — fuelled the large language model era. The relationship between data scale and model capability is not accidental: machine learning algorithms, particularly deep learning, are data-hungry by nature. They extract statistical patterns, and patterns become clearer — and more complex patterns become learnable — as data volume grows.
The Big Data Infrastructure Stack
Processing data at the scale required by modern ML pipelines requires specialised infrastructure. The foundational layer is distributed storage: systems that distribute data across many nodes, providing both high capacity and high throughput. Apache Hadoop's Distributed File System (HDFS) pioneered this approach; today, cloud object stores such as Amazon S3, Google Cloud Storage, and Azure Blob Storage have largely superseded it for new deployments, offering near-unlimited capacity, high durability, and pay-as-you-go pricing.
Above the storage layer sits the distributed processing framework. Apache Spark is the dominant framework for large-scale data transformation, feature engineering, and model training orchestration. Spark distributes data processing across a cluster, representing data as resilient distributed datasets (RDDs) or, in the modern API, DataFrames. A Spark job that transforms petabytes of raw transactional data into model-ready features might run in minutes on a 500-node cluster — a computation that would take weeks on a single machine.
Feature Stores
A key innovation in production ML infrastructure is the feature store — a centralised system for storing, versioning, and serving ML features. Feature stores solve a critical problem: in large organisations, the same features (customer lifetime value, 30-day transaction count, account age) are computed redundantly by different teams using different code, leading to inconsistencies between training and serving environments — the so-called "training-serving skew" that causes model performance to degrade in production.
Feature stores like Feast, Hopsworks, and the managed offerings of cloud providers provide a shared catalogue of feature definitions. Training pipelines pull historical feature values from an offline store (a columnar data warehouse like BigQuery or Redshift); serving pipelines retrieve the latest feature values from an online store (a low-latency key-value database like Redis or DynamoDB) in milliseconds. This architecture ensures that training and serving use identical feature logic, eliminating a major source of production failures.
Data Quality and Validation
In ML pipelines, garbage in genuinely equals garbage out. Data quality issues — missing values, inconsistent encodings, schema drift, label noise — can silently degrade model performance in ways that are difficult to detect. Tools such as Great Expectations and Apache Griffin enable automated data validation: defining statistical expectations about data properties (ranges, distributions, null rates) and alerting when new data batches violate them.
Data versioning — tracking exactly which data was used to train each model version — is equally important for reproducibility and debugging. DVC (Data Version Control), LakeFS, and Delta Lake all provide mechanisms for versioning large datasets alongside model code.
Streaming Data and Real-Time ML
Many of the highest-value ML applications — fraud detection, real-time recommendation, dynamic pricing — require predictions in milliseconds, based on data that is itself generated in real time. This necessitates streaming data infrastructure. Apache Kafka is the dominant event streaming platform: it ingests millions of events per second from sources such as payment terminals, web servers, and IoT sensors, and distributes them to consumer applications with very low latency.
Frameworks such as Apache Flink and Spark Structured Streaming enable real-time feature computation from Kafka streams — aggregating events over sliding windows, joining streams with reference data, and updating feature stores continuously. The resulting architecture enables truly online learning systems where models are retrained or updated on new data without manual intervention.
MLOps: From Prototype to Production
The discipline of MLOps addresses the full lifecycle of ML systems in production: training, evaluation, deployment, monitoring, and retraining. A mature MLOps practice includes automated training pipelines triggered by new data or scheduled intervals; A/B testing frameworks for safe model rollout; model monitoring that tracks prediction drift, feature drift, and business metric degradation; and model registries that maintain versioned artefacts with metadata about training data, hyperparameters, and evaluation results. The investment in MLOps infrastructure pays dividends through reduced time to deploy new models, faster detection of production issues, and reproducible, auditable model lifecycle management.