Share an experience making a key engineering decision with poor data quality.

Question

Anonymous · Accepted Answer

Every half we run consolidated experiments to measure the impact of our
launches. A week and half before our’s a team reached out asking us to revert
one of our key launches because they claimed it impacted one of their launches
readings negatively. This was a serious claim, very quickly i examined the
evidence and realized it was not causal but based on a timing correlation where
their launches’ readings got level worse sometime after our launch. I gathered
evidence from a number of our past experiments for the change to show that there
was no negative interaction in that segment. To push towards causality, I also
suggested we use the next week to run an experiment to validate their claim.
Unfortunately we could not secure a well powered experiment under the time
constraints, so I decided to pivot - we could not change the power of the
experiment, but the decision criteria before the experiment started. Convened a
meeting with the key stakeholders, showed them the totality of the evidence,
past experiments, and based on sampling variation there was a 16% of getting a
SS -ve result (type s error). I managed to get alignment that this underpowered
test would be used for exploration only, not to inform a revert decision, and
that our half back-tests would proceed as per plan without any changes. We did
pre-align that if the sufficiently powered test (that was secured 3 weeks later)
showed a negative interaction, we would revert our launch

Share an experience making a key engineering decision with poor data quality.

Practice this question with AI

Go Premium

Community Answers

Unlock Full Access

Practice More Questions