How do you ensure that your engineering processes are robust and that mistakes are caught before they become serious issues?

Question

Anonymous · Accepted Answer

Well mistakes are inevitable but i believe in following processes which can
reduce mistakes and also we can foresee and plan to avoid them.
Engineering team should follow below processes:
1.  Thorough Requirement Analysis
2. Design and code review
3. Incremental development and testing
4. Automated testing
5. Monitoring and alerting system on production
6. Post deployment support with retrospective and RCA
I was handling one project where we were integrating a 3rd party api for payment
processing and we misunderstood a crucial part of their rate limiting policy of
api and didn't implement exponential back off correctly.
During high traffic event, out application hit the rate limit and instead of
retrying with exponential backoff we keep retrying immediately which caused a
spike in failed transaction and customer complained.
We immediately halt affected service and rollback previous stable version.  We
had analysed logs and confirmed that issue and changed logic and implemented
exponential backoff retrying and simulated testing with high traffic and
documented this as lesson learned .
We learned a lesson that rate limiting is resilience not performance and also
not to assume api behaviour based on doc. try and check edge cases to avoid such
issues. we have  added this learning in onboarding manual for new developers.

Anonymous · Answer

Bugs and issues are part of software development process. I look at this from 2
separate aspects

1. Proactive

2. Reactive

In my current team there are 2-3 parts to the proactive process review

1. We have weekly on-call rotations where we can discuss various issues we have
    seen. Part of the discussion is also open ended such that we end up trying
    to find the root cause. This is not always a guarantee to be a hit but
    provides options to the team to explore the root casuses.

2. This also can be linked with out ORR process where we look at project
    readiness before they are launched. This will help the team members
    familiarize with the changes to the update, provide feedbacck based on what
    they know or have experienced and the outcome will be a more stable and
    reliable launch.

3.

4. We also have annual process review meetings where we look at various SOPs
    and runbooks for our services to capture things that may not have been
    captured during the year. Or look at repetitive issue that have happened
    thorugh the year and see what needs to be done to improve or what can be
    done to simplify things. Out come of this meeting is to update our backlog
    with details of what needs to be changed.

5.

6. Additionally we lean on new members joining the team as a check point on
    these things as well. New members on the team look at things with new set of
    eyes and do not have the tribal knowledge of certain things. They also bring
    in new skill sets and tooling that they may be able to leverage to improve
    and suggest updates to existing processes.

Reactive is pretty straight forward. When ever there is an issue or an outage
that is large enough that needs analysis outside of our on-call and ORR we do a
through analysis of what the issues are with the 5 whys and what caused the
outage and what can be done to catch the issue earlier and what can be done to
avoid this earlier.

How do you ensure that your engineering processes are robust and that mistakes are caught before they become serious issues?

Practice this question with AI

Go Premium

Practice More Questions

Community Answers

Unlock Community Insights

Unlock Community Insights