Why Most Machine Learning Projects Fail to Reach Production

Why Most Machine Learning Projects Fail to Reach Production

The high failure rate of machine learning projects is a pressing concern, with studies showing that up to eighty-five percent of projects fail to reach production. According to Wenjie Zi, a machine learning engineer with experience across multiple domains, the five common pitfalls driving these failures are choosing the wrong problem, data quality and labeling issues, the model-to-product gap, offline-online mismatch, and non-technical blockers.

Why This Matters

The technical reality of machine learning projects is that they are inherently uncertain and experimental, with a lengthy, multi-step process involving numerous handovers across teams, which increases the risk of failure due to complexity. Ideal models often overlook the importance of data-centric optimization, feedback signals from data, models, and monitoring, and the need for cross-functional teams to align stakeholders, scope an MVP, and iterate based on production feedback. The cost of failure can be significant, with late pivots being costly due to heavy data engineering, objective-function design, and infrastructure investments.

Key Insights

• 85% failure rate of ML projects: a 2023 Rexer Analytics study found that only thirty-two percent of ML practitioners reported their projects reaching production.
• Data leakage is a critical pitfall: a 2022 Princeton University review found critical pitfalls in twenty-two peer-reviewed papers, including data leakage, which can lead to flawed conclusions.
• Cross-functional teams are essential: successful ML teams align stakeholders, scope an MVP, build end-to-end early for A/B testing, and iterate based on monitoring, as seen in Google’s ML system diagram.

Working Example

# No code example provided in the context

Practical Applications

• Use Case: Netflix’s recommendation system, which uses a combination of collaborative filtering and content-based recommendation to surface relevant content to users.
• Pitfall: Over-optimizing offline models, which can lead to diminishing impact in the merged system, as seen in the photo recommender example.

References:

• https://www.infoq.com/articles/why-ml-projects-fail-production/
• https://rexeranalytics.com/
• https://patterns.journalồi.com/patterns/article/100804/leakage-and-the-reproducibility-crisis-in-machine-learning-based-science
• https://research.google/pubs/pub43146.html
• https://dl.acm.org/doi/10.1145/2623330.2623342