Expert Answer
Anonymous
While working at ABC, I noticed that internal customers were creating expensive resources but still paying per-use rates. This meant they had large resources allocated but were not fully utilizing them, leading to inefficiencies in both our fleet and overall system performance. It became clear that there were misaligned incentives.
Task:
I needed to influence multiple internal teams and stakeholders to resize their resources to better align utilization with actual demand. This required gathering data, negotiating with internal customers, and proposing a solution that both addressed the technical inefficiencies and fit within their operational requirements.
Action:
- Data Gathering: I asked my engineers to gather data on resource utilization across internal customers. We discovered that most customers were using less than 5% of their provisioned capacity.
- Engagement with Stakeholders: I began by reaching out to two of the largest internal users—XYZ and PQR—both of which I had close relationships with. I framed the problem as a multi-phased solution that would allow them to resize their resources gradually without impacting their services.
- Cross-team Alignment: I then engaged with the Front-End (FE) team and its Senior Manager to inform them of this opportunity to shape the traffic and reduce strain on the backend fleet. This led to further discussions about aligning future traffic patterns to optimize fleet usage.
- Initiating Pricing Discussions: With the success of the initial efforts, I initiated discussions around refining the internal pricing model, presenting data to show that many more customers could benefit from resizing, thereby reducing backend load and unnecessary resource allocation.
Result:
- Backend Fleet Reduction: We successfully reduced the backend fleet size by 7-9%, leading to a reduction in overall system load and minimizing metadata overhead.
- Cost Savings: Customers were able to save costs as well by reducing their reliance on over-provisioned resources and lowering the number of workers required for their applications.
- System Improvements: By pushing out scaling cliffs for the load aggregator service, we avoided potential bottlenecks, improving operational stability.
- Pricing Strategy Refinement: This work led to ongoing refinements in ABC's internal pricing model, aiming to align more closely with public pricing or move towards a more robust per-use product.
Reflection:
One thing I would do differently is celebrate the success more broadly across teams. I later realized that while the FE team was actively working on resizing streams, the Orchestration team had been giving conflicting advice to customers, suggesting they over-provision when in doubt. Greater coordination and communication could have avoided this issue and created a more unified effort.
