Can you describe a time when you faced a major setback at work and how you recovered from it?
Engineering Manager
UX Designer
UX Researcher
Data Science Manager
TikTok
Square
DoorDash
Asana
Palantir Technologies
Stripe
Answers
Anonymous
2 months ago
When buildling the next generation instance, during the planning phase I budget the amount of servers we will launch the product with. Given each server cost was over 2 million dollar I biased to towards frugality and provided a bare minimum number of server we could launch with. But what I completely missed was during the developement some servers will end up in bad state and the engineering team will short of servers to test the software and firmware getting built which made meeting the project timeline riskly and heavily dependent on delivery schedules of servers and them staying healthy. This quickly became a bottleneck that I realized. I quickly put togther a plan to request adittional server and provide a detailed breakdown on where and how these would be used during and after developement. Synced with the hardware and ODM team and executives to get a approval and ordered the parts to get the servers and racks build at ODM.
Anonymous
3 months ago
I was the lead engineer on a project to enhance our event-driven messaging system. The system processes real-time messages and archives them into S3 buckets. These archived messages are then ingested by another team into Snowflake, which serves as the backbone for various downstream pipelines and analytics reporting.
We recently implemented an update to the S3 archiver to write data to multi-region buckets based on the message-producing country. This was a crucial update aimed at improving data redundancy and compliance with regional data regulations.
While the deployment of the multi-region S3 buckets was successful, I overlooked the need to inform the Snowflake team about the changes. As a result, their ingestor continued to point to the old S3 buckets, causing Snowflake to go out of sync. This led to a cascading failure in all downstream processes, including inaccurate analytics reporting and failed data pipelines.
The mistake led to significant disruptions in downstream processes, with incorrect reports being generated for critical business decisions. Once I realized the oversight, I immediately coordinated with the Snowflake team to update their ingestor to point to the correct S3 buckets. After the fix, I took steps to ensure better communication and coordination between teams, including setting up a process for notifying all stakeholders of changes that could impact their systems. This experience reinforced the importance of cross-team communication, especially in environments where multiple systems are interdependent.
This scenario effectively demonstrates your ability to own up to a mistake, address it quickly, and learn from the experience. It also highlights the complexity of the systems you're working with and your role in leading such projects. By discussing how you implemented better communication protocols afterward, you show your growth as a senior engineer who is proactive in preventing similar issues in the future
Interview question asked to Data Engineering Managers, ML Engineering Managers, Data Science Managers and other roles interviewing at SAP Concur, Workday, Convoy and others: Can you describe a time when you faced a major setback at work and how you recovered from it?.