Analytics Case

What would you advise the product team to do if we are launching a new product but are seeing a decline in usage metrics? Would you continue with launching the product?

Data Scientist

Palo Alto Networks

Flexport

Meta

Google

Spotify

Grab

Did you come across this question in an interview?

Answers

Anonymous

4Strong
Uber's mission: reinvent how the world moves
Uber rides mission: get people from point A to B

I have some questions first about the decline in usage metrics.
What usage metrics specifically are we seeing decline?
When did the dropoff start? was it sudden or gradual?
How big was the change relative to what we've seen in the past?
How important is the change? Is it endangering the core function of the product?

If the viability of the core product is jeopardized, we should probably focus on fixing the core product rather than launching a new product (unless we think the new product could fix the usage decline). These questions will help us understand the dropoff and address it.

Next I have questions about motivation for the new product.
Do we have a mission for our new product? Do we know how it interacts with the existing core product? That will help us understand if the viability is jeopardized. If it remains viable, likely we want to continue with the launch, but we want to make sure we can evaluate it. It may be hard to evaluate the launch if our metrics are in a free-fall. So we should want to see our metrics have stabilized before launching the new product.

Do we know what KPIs we'd want to use to evaluate the new product? Perhaps % of our users who used the product. Given potential for network effects in the Uber marketplace, we may want to run this as a market-level A/B. We can either pick a few markets by hand where we'd like to implement this and then use a synthetic control to create a synthetic comparison group of other markets as a control. Or we can do a stratified random sample and implement the program in the areas selected for treatment. I would recommend the former so we can maximize the success of the new program.

We should also have success criteria, e.g., we will expand the experiment if utilization exceeds 20%.  Once the experiment has run, we can determine whether it met the success criteria, and if so, roll it out to a larger treatment aea for additional study.

Anonymous

4Strong
Clarifying questions - 
  • Which usage metrics are declining? Are these for the overall Meta site or only for the product about to launch? Are all metrics on decline or only this specific metric? 
  • What does launch mean? Does it mean to start A/B testing, or post testing? Is the Launch going to be ramped or big bang?
  • Is there a similar product currently existing? Internally or competitor? What is Meta's goal with launching this new product?
  • What is the time frame of the observed decline? Is it sudden or gradual?
Let us say, 
  • avg time spent by a user in a day on this product is declining. 
  • A/B testing is complete, a phased launch is planned with upcoming phase being all users in US.
  • Let us say there is a mature existing product in the market but no Meta product.
I would trend the avg time spent by user over time, with the x axis being a calendar day. Since this is still pre-launch the volume of users is not much and time frame is also small. So I would take all users who were in the experiment, from the start of the experiment. Say 2 months.

I observe that at the start of the experiment, there was high activity - avg time spent grew for the first 1 week, flattened for a week after and then started having a slow decline. The time plots seems to indicate novelty effect. The decline seems to have stabilized post novelty. 

If the goal of Meta is to take market share from competitor, this new product might still be a viable product to launch. Since the usage is stable and sustained, the product has a potential to grow from network effect post launch. 
  • What would you advise the product team to do if we are launching a new product but are seeing a decline in usage metrics? Would you continue with launching the product?
  • If a product's usage metrics take a hit leading up to its launch, what steps would you recommend taking?
  • How would you handle a scenario where the usage metrics for a product slump before its launch?
  • In the event of a product launch, but a noticeable decrease in user engagement metrics, what course of action would you suggest taking?
  • If a newly launched product seems to be struggling with user engagement metrics, what would be your next move as a product team?
  • If metrics suggest low user engagement in the lead-up to a new product launch, what steps would you recommend to the product team?
  • Suppose a new product is set to launch, but usage metrics appear to be in decline. What would you advise the product team to do next?
  • If a product launch is imminent, but the usage metrics are dropping, what would be your recommended course of action?
  • What would be your advice for a product team if usage metrics show a decline leading up to a new product launch?
  • In the case of a new product launch, but declining usage metrics, what actions would you suggest the product team takes?
  • Could you tell me how you would proceed if user engagement metrics for a new product decline prior to its launch?
Try Our AI Interviewer

Prepare for success with realistic, role-specific interview simulations.

Try AI Interview Now

Interview question asked to Data Scientists interviewing at Google, Evernote, Postmates and others: What would you advise the product team to do if we are launching a new product but are seeing a decline in usage metrics? Would you continue with launching the product?.