EliminationSearchCV: A Smarter Alternative to GridSearchCV That Cuts Training Time by Up to 150x

Why GridSearchCV Wastes Most of Its Time — And What I Did About It

Thisal Dilmith released EliminationSearchCV, a drop-in replacement for scikit-learn’s GridSearchCV. On a full grid with DecisionTree, it achieved a 152x speedup with an accuracy difference of only -0.0008.

Why This Matters

GridSearchCV brute-forces every possible hyperparameter combination—nᵏ × cv_folds configurations—even when early results clearly show a value is dead. This exponential cost scaling means adding one new parameter with four values can quadruple total training time, turning small grids into hours-long waits. Dilmith’s elimination-based approach cuts bad values in early rounds, preventing them from multiplying into thousands of useless combinations.

Key Insights

• GridSearchCV evaluates nᵏ × cv_folds configurations without learning from early results; it treats round 1 and round 1000 as equally uninformed (Dilmith, June 2026).
• EliminationSearchCV uses iterative elimination: test per-parameter in isolation (Round 1), discard worst performers, then test surviving pairs (Round 2), triples (Round 3+), until one winner remains (Dilmith, June 2026).
• On a LogisticRegression with C [0.001–100], penalty [l1,l2], solver [liblinear,saga], max_iter [1000,2000], GridSearchCV runs 240 fits vs EliminationSearchCV’s just 23 fits—same best params found (Dilmith, June 2026).
• Benchmarked across five models on three datasets: speedups range from 4x (LogisticRegression) to over 150x (DecisionTree), with accuracy differences under ±0.02 across all models (Dilmith, June 2026).

Working Examples

# Before
search = GridSearchCV(model, param_grid, cv=5)
# After — just swap the class name
search = EliminationSearchCV(
    estimator=model,
    param_grid=param_grid,
    scoring='accuracy',
    cv=5,
    elimination_rate=0.8,
)
search.fit(X_train, y_train)
print(search.best_params_)
# → {'C': 1, 'penalty': 'l1', 'solver': 'liblinear', 'max_iter': 1000}
print(search.best_score_)
# → 0.9248
search.best_estimator_.predict(X_test)

Practical Applications

• (Company/system + behavior) Teams using scikit-learn for hyperparameter tuning can drop-in replace GridSearchCV with EliminationSearchCV on large grids—swap the class name and set elimination_rate=0.8 to cut training time dramatically while keeping similar best params.
• (Common anti-pattern + consequence) Using this approach on light or small search spaces actually increases runtime due to elimination overhead; developers should stick with traditional GridSearchCV when the grid has only a few combinations.

References:

• https://dev.to/thisald/why-gridsearchcv-wastes-most-of-its-time-and-what-i-did-about-it-2cak