What would you advise the product team to do if we are launching a new product but are seeing a decline in usage metrics? Would you continue with launching the product?

Question

Anonymous · Accepted Answer

This would depend on the usage metrics that are in decline and the product
launching.
I would start with investigating why there is a drop in usage, as there are
several scenarios -
1. if low usage is overall and due to seasonality, I would generally hold off on
product launches.
2. if some of the usage metrics are down, and some are up, then it depends on
the product itself -
a. If the new product is more relevant to the ones that are up, no reason not to
go forward.
b. If the new product is more relevant to the declined metrics, then we need to
understand if it's a steady decilne (trend seen overtime) and the new product
might assist with it. Or, perhaps it's a sudden decline, and I would hold off
with product launch until I have an understanding of what is causing this sudden
decline.
Of course, if any publich commitments were made to the launch of this product,
it's also something we need to take into consideration, since changing a public
publish date is problematic for user trust.

Anonymous · Answer

Uber's mission: reinvent how the world moves
Uber rides mission: get people from point A to B

I have some questions first about the decline in usage metrics.
What usage metrics specifically are we seeing decline?
When did the dropoff start? was it sudden or gradual?
How big was the change relative to what we've seen in the past?
How important is the change? Is it endangering the core function of the product?

If the viability of the core product is jeopardized, we should probably focus on
fixing the core product rather than launching a new product (unless we think the
new product could fix the usage decline). These questions will help us
understand the dropoff and address it.

Next I have questions about motivation for the new product.
Do we have a mission for our new product? Do we know how it interacts with the
existing core product? That will help us understand if the viability is
jeopardized. If it remains viable, likely we want to continue with the launch,
but we want to make sure we can evaluate it. It may be hard to evaluate the
launch if our metrics are in a free-fall. So we should want to see our metrics
have stabilized before launching the new product.

Do we know what KPIs we'd want to use to evaluate the new product? Perhaps % of
our users who used the product. Given potential for network effects in the Uber
marketplace, we may want to run this as a market-level A/B. We can either pick a
few markets by hand where we'd like to implement this and then use a synthetic
control to create a synthetic comparison group of other markets as a control. Or
we can do a stratified random sample and implement the program in the areas
selected for treatment. I would recommend the former so we can maximize the
success of the new program.

We should also have success criteria, e.g., we will expand the experiment if
utilization exceeds 20%.  Once the experiment has run, we can determine whether
it met the success criteria, and if so, roll it out to a larger treatment aea
for additional study.

Anonymous · Answer

Clarifying questions - 
 * Which usage metrics are declining? Are these for the overall Meta site or
   only for the product about to launch? Are all metrics on decline or only this
   specific metric? 
 * What does launch mean? Does it mean to start A/B testing, or post testing? Is
   the Launch going to be ramped or big bang?
 * Is there a similar product currently existing? Internally or competitor? What
   is Meta's goal with launching this new product?
 * What is the time frame of the observed decline? Is it sudden or gradual?

Let us say, 
 * avg time spent by a user in a day on this product is declining. 
 * A/B testing is complete, a phased launch is planned with upcoming phase being
   all users in US.
 * Let us say there is a mature existing product in the market but no Meta
   product.

I would trend the avg time spent by user over time, with the x axis being a
calendar day. Since this is still pre-launch the volume of users is not much and
time frame is also small. So I would take all users who were in the experiment,
from the start of the experiment. Say 2 months.

I observe that at the start of the experiment, there was high activity - avg
time spent grew for the first 1 week, flattened for a week after and then
started having a slow decline. The time plots seems to indicate novelty effect.
The decline seems to have stabilized post novelty.

If the goal of Meta is to take market share from competitor, this new product
might still be a viable product to launch. Since the usage is stable and
sustained, the product has a potential to grow from network effect post launch.

Anonymous · Answer

We need to reschedule the launch date and recheck the planning stage.
Collaborate with the product team on how you can make sure that the usage
metrics will increase.

What would you advise the product team to do if we are launching a new product but are seeing a decline in usage metrics? Would you continue with launching the product?

Practice this question with AI

Go Premium

Community Answers

Unlock Full Access

Unlock Full Access

Unlock Full Access

Unlock Full Access

Practice More Questions