Building Scalable ML Pipelines on Millions of Rows with Vaex

A Coding Guide to Build a Scalable End-to-End Analytics and Machine Learning Pipeline on Millions of Rows Using Vaex

Vaex enables high-performance exploratory analysis and machine learning workflows on datasets containing millions of rows without materializing data in memory. This technical guide demonstrates an end-to-end pipeline processing 2,000,000 records using lazy evaluation and approximate statistics to eliminate memory bottlenecks.

Why This Matters

In large-scale data science, materializing intermediate data frames in memory often leads to Out-Of-Memory (OOM) errors and significant latency. Vaex addresses this by using lazy expressions and memory mapping, ensuring that computations only occur when results are explicitly requested. This architectural shift allows engineers to perform complex feature engineering—such as city-level aggregations and statistical normalization—on millions of rows using standard hardware. By integrating with scikit-learn via specialized wrappers like Predictor, Vaex bridges the gap between big data processing and traditional machine learning frameworks without the overhead of Spark or Dask.

Key Insights

• Out-of-core execution: Vaex processes 2,000,000 rows without loading the entire dataset into RAM, utilizing memory-mapping for efficiency (Vaex 4.19.0).
• Lazy Evaluation: Features like income_k and value_score are defined as virtual columns, meaning they are calculated on-the-fly and consume zero additional memory.
• Approximate Statistics: Functions like percentile_approx enable fast binning-based aggregations across large categories without full data passes.
• Scikit-learn Integration: The vaex.ml.sklearn.Predictor wrapper allows training standard models like LogisticRegression directly on Vaex DataFrames.
• Pipeline Persistence: Preprocessing states, including LabelEncoder mappings and StandardScaler parameters, can be serialized to JSON for deterministic inference.

Working Examples

Demonstration of lazy feature engineering and scikit-learn model integration using Vaex.

import vaex, vaex.ml, numpy as np
from vaex.ml.sklearn import Predictor
from sklearn.linear_model import LogisticRegression

# Initialize lazy DataFrame
df = vaex.from_arrays(city=city, age=age, tenure_m=tenure_m, tx=tx, income=income, target=target)

# Define virtual columns (lazy expressions)
df['income_k'] = df.income / 1000.0
df['log_income'] = df.income.log1p()
df['value_score'] = (0.35*df.log_income + 0.10*(df.tenure_m/12.0) - 0.015*df.age)

# Scalable approximate statistics
n_cities = len(df.unique('city'))
p95_income = df.percentile_approx('income_k', 95, binby='label_encoded_city', shape=n_cities)

# Model Training via Sklearn Wrapper
model = LogisticRegression(max_iter=250)
vaex_model = Predictor(model=model, features=features, target='target', prediction_name='pred')
vaex_model.fit(df=df_train)

Practical Applications

• Financial Risk Modeling: Using percentile_approx to compare individual incomes against city-level benchmarks. Pitfall: Materializing intermediate joins can crash local environments if not handled lazily.
• Predictive Lead Scoring: Deploying LogisticRegression through vaex.ml for real-time inference on millions of records. Pitfall: Failing to persist scaler mean/std values leads to training-serving skew during deployment.

References:

• https://www.marktechpost.com/2026/03/02/a-coding-guide-to-build-a-scalable-end-to-end-analytics-and-machine-learning-pipeline-on-millions-of-rows-using-vaex/