Think of a time things went wrong. How did you collaborate with engineers to break down the failure?

Question

Anonymous · Accepted Answer

"I'll share one from my Cisco days that I think about often — even now at
Amazon.

We were running a large-scale migration from a legacy Java and Oracle stack to a
cloud-native architecture. The goal was to dramatically improve system latency —
which we did, going from 15 seconds down to 500 milliseconds. The team had done
everything right from a testing standpoint — unit testing, QA, integration
testing, smoke testing — we had verified production readiness thoroughly. We got
to the Go/No-Go call feeling confident.

Then our Business VP asked a question I wasn't prepared for: 'How can you prove
that the new system's output is exactly the same as the old system's, even after
the technology change and logic optimizations?'

It stopped me cold. This was a new VP — the previous one had been with us
through the whole program and had built confidence along the way. This VP had
come in late and had no baseline. And I didn't have a ready answer. We had
tested that the new system worked — but I hadn't built the comparative evidence
to show a skeptical executive that it produced identical outputs to the system
it was replacing.

I consider that a failure on my part. Stakeholder transition was a known risk on
the program and I hadn't designed a confidence-building mechanism that would
hold up to a new executive asking the most basic question: 'How do I know this
is right?'

I asked the VP for 48 hours. I designed a parallel run framework — I launched
the cloud-native stack to production alongside the existing system, fed both
systems identical inputs, captured outputs and latency for each in logs, and
built a Kibana dashboard that showed old system vs. new system results side by
side in real time.

Then I came back to the VP twice a week with the dashboard and walked through
any mismatches — explaining the reason for each one. Week one we had about 10%
mismatch. Week two it was down to 2%. By week three we were at 0.5% — and every
remaining mismatch had a documented, expected reason. The VP could see the
trend, understand the exceptions, and build confidence progressively rather than
being asked to trust a single readiness checklist.

We got the green light.

But the bigger outcome was the framework itself. Since that day I've built
parallel run validation into every major launch I've run — at Cisco and at
Amazon. At Amazon Ad Exchange, before any significant supply-side change goes
live, I run a parallel validation period — typically 24 hours — where both the
old and new systems process the same requests and I compare outputs before
cutting over fully. It's caught issues that would have been production incidents
on three separate occasions.

The failure taught me that technical readiness and stakeholder confidence are
two different things — and as a TPM, I'm responsible for both."

Think of a time things went wrong. How did you collaborate with engineers to break down the failure?

Practice this question with AI

Go Premium

Community Answers

Unlock Full Access

Practice More Questions