Situation:
At my previous role, I was working on a churn prediction model for a food delivery subscription program. The goal was to proactively identify users likely to churn and design interventions to retain them. The safe route would have been to use a traditional logistic regression model with standard RFM (Recency, Frequency, Monetary) features, which we knew would give us “good enough” performance.
Task:
I believed we could significantly improve prediction accuracy—and ultimately retention—by incorporating session-level behavioral data using a more complex machine learning approach. But doing this came with risk: more complexity meant slower iteration, possible model interpretability concerns, and engineering resources for new feature pipelines.
Action:
I laid out the trade-offs to my team and stakeholders during a planning session. We held a working session with our product manager, engineering lead, and another DS to decide how much to invest upfront. I proposed a two-track strategy:
- In the short term, we'd ship a simple logistic regression using existing data to get a baseline.
- In parallel, I built a prototype using XGBoost with behavioral features like time of day ordering patterns, days since last session, and order scroll depth.
Throughout, I regularly shared updates with the team in Slack and in weekly check-ins. I also set up interpretable SHAP analysis so the model wouldn’t be a black box to marketing and product partners.
Result:
The baseline model gave us a lift in retention campaign targeting—about a 5% improvement in retention over random targeting. But the more complex model improved that by an additional 9 percentage points. After a successful A/B test, the marketing team adopted it for ongoing campaigns. Our engineering team later prioritized automating the behavioral feature pipeline due to the business impact.
Why it was a calculated risk:
We risked sunk time and complexity, but mitigated it with parallel development, stakeholder involvement, and clear success criteria. The team was involved throughout because buy-in and trust were essential for implementation—and I wanted to ensure everyone understood the “why,” not just the “what.”
