Taking a Promotion to Customer

After 7 years, 8 months, and 13 days, I had my last day at Amazon Web Services on August 11, 2023.

Over that time, I managed to have 5 different job titles (2 as an individual contributor, 3 as a manager) and 5 different managers, working on 2 distinct AWS products.

A lot changed over (almost) 8 years. When I joined AWS, I was the second engineer on the fledgling RDS team in Sydney. As a Systems Engineer in an operationally focused team, I spent a lot of time diving into all aspects of a complicated distributed system. After a couple of years of seeing various recurring problems and patterns, I realised that I could fix more if I wasn't an IC fighting fires, so I turned to the dark side and became a manager. My manager and I started growing the team and building up a base of operationally excellent software for us to support. In September 2018 my manager moved to another part of AWS, and I took over the team (then 15 people). Although I'd had direct reports before, this felt like my first real management role, and it was a pretty big adjustment. The team continued to grow - by the end of 2018, I had 23 direct reports (a mix of Software Development Engineers, System Development Engineers, and Systems Engineers). Somewhere in there I got a promotion and helped hire 2 extra managers so I could split up the different teams¹.

Through 2019 I had a small team, trying to explore some of the big areas of RDS ripe for improvement after 10 years. This was a big year of exploring options for improvement, suggesting fixes large and small, and then seeing some internal re-organisations completely change the priorities of my team. After 4 years, I started to look for other options in AWS, and found an opportunity to work on a completely new service - AQUA for Redshift.

In February 2020 I started with no team, a bunch of headcount to hire, and a whole lot to learn. A few of my previous team from RDS asked to join me, which helped bootstrap the team. It's no secret that most of AWS is built on AWS. Building a control plane for a new system from scratch meant the team got to use a lot of AWS technology that didn't exist when RDS was designed (or was unusable due to itself having a dependency on RDS). For ~2 years I focussed exclusively on AQUA, hiring up teams in the US and in Australia, and working on awesome new stuff, all from the "comfort" of home during COVID. We saw AQUA launch, we iterated on the stuff we'd built, and I learned a lot about leading a growing and globally distributed team. The pace of growth was a little slower, but the challenges were bigger. I had to learn to delegate, balance the requirements of the job across the different skill sets in the team, and then deliver a bunch of stuff from scratch, with no existing codebase to extend. Turns out greenfield development is a blessing - no legacy to maintain, and you can ensure that things are done in an operationally sound manner from the get-go. It's a lot easier to convince engineers to be on call for 168 hours at a time when they know that (1) they wrote it, and (2) they are fully empowered to stop whatever they're doing and take the time to properly root cause and fix anything that pages them.

After a couple of years, my team and I shifted away from AQUA and started working more broadly on the Redshift Control Plane. I hired a manager to run part of the team, focussing on the old familiar area of operationally excellent software. The other part of my team and I embarked on a big medium bet to try to capture some of the pure complexity that comes out of operating a custom-built query engine that customers can use freely. Again we were building greenfield stuff, and this time I was working more broadly across a big organisation with lots of internal stakeholders. But by 2023 there was a bit of a niggle that I should be trying something new. I started to look around, and eventually made the decision to leave.

There were a lot of good things about my time at Amazon. Some of the highlights:

Scale

There's no other company working in cloud computing that matches the scale of AWS (although you could argue that Azure and GCP are near enough to be equivalent). The pace is relentless and the scope/impact of the work you do is epic, especially when you consider not just the direct customers of AWS, but their customers and the people impacted. This is keenly felt whenever there's an AWS outage, especially for one of the core services and/or in one of the bigger regions. This leads to a lot of pressure, and a never-ending backlog. But the feeling of knowing that your work has such a large impact across the internet is something amazing that's hard to describe.

Hiring

There's a lot written about the "2 Pizza Team" model of Amazon, and how it allows for a lot of autonomy and independence for teams. That aligns with my experience, but Amazon also comes with a lot of benefits of scale. There are a lot of internal tools, processes, and support teams that are not viable if you don't have a huge company to fund them. Hiring is one of those areas. Over (almost) 8 years, I did 395 interviews as either a hiring manager, bar raiser, or interviewer for other teams. Basically 1 a week. I love doing interviews, because people are (usually) excited about the opportunity, and when you're asking behavioural questions you get to hear cool stories about people doing awesome things. Not every candidate is a good one, but the general rule is that when you interview someone you hear awesome stories about them doing awesome things. The Bar Raiser role was even better, as I was often not interviewing people in roles I was familiar with, so I also got little peeks into different worlds like data centre construction in South East Asia, NBA retail operations in China, industrial HVAC installations in Australia, and more.

Great Leaders, Great Opportunities

I was lucky enough to have many great role models who either directly or indirectly taught me a lot about management, ownership, and leadership. I also had a few examples of anti-patterns that quite clearly showed me ways of working that I really didn't want to adopt. The thing that I appreciate the most though is that I had several managers and organisational leaders who saw what I was doing and liked it enough to give me opportunities and freedom to carve out my own path - especially in Redshift. Building the AQUA CP was the first time I'd driven a greenfield software project, and it was a risk for the Redshift leadership to invest in someone whose background was much more operational, but it paid off (if I do say so myself). Likewise, when I took my team and started working more in the Redshift CP, I was given leeway to define my own roadmap, propose my own projects, and given product management support and freedom to go build it. I still can't quite believe that I was able to take a group of 15 people and basically run completely on my own initiative for 2 years, and that was a remarkable show of faith from my manager and all the Redshift leadership.

The Leadership Principles

When I was first interviewing to join Amazon, I studied the leadership principles extensively, as it's a key part of the hiring process. As I was told in the interview, then realised as I joined, and eventually started telling candidates as I interviewed them - "The LPs aren't like corporate motivational phrases you put on the wall and forget about, in Amazon people actually believe them and use them". I wouldn't ever say the LPs are closer to the definition of "cult" than "corporate", but some might. To me, I found the LPs incredibly helpful, even when they conflicted with each other. Part of what made me so attracted to Amazon in the first place was that in the LPs I found the words to help define some of the things I felt deeply, but couldn't articulate. Earn Trust², Ownership³, and Insist on the Highest Standards⁴ all described how I liked to view myself (if not always describing how I was.) Cult or not, I felt the LPs deeply⁵, and ultimately one of the big motivators to move on from Amazon was when I started to feel like I cared about the LPs much more than the company and some of the people around/above me.

My Team

One of the things I was most proud of at Amazon was my ability to build teams and bring great engineers with me on the journey. The teams that I built were the engine behind all my individual success - even in a company that hires high achievers, my teams were often delivering above expectations (justified by performance ratings) and I was incredibly gratified that I built a core group of engineers who worked with me for years. There's nothing quite so good as a manager as having a strong engineer come with you when you move on to new and uncharted waters, and I was lucky enough to leave Amazon with a third of my team being engineers who had worked with me in RDS and chosen to join me in Redshift. I'm a firm believer that the team is much more important than the job, and that core group proved it to me. What hammered home the strength of these people was the fact that these engineers didn't just form a roaming exclusive clique, they just helped set the tone and build up a strong team culture as we grew.

Like any job, especially for a global mega-corp, there were some lowlights. There are a lot of things about how Amazon treats key inputs to the success of the retail business (namely: the people who work in warehouses and last-mile delivery, as well as the suppliers) that are awful. There are a lot of stories about the unrelenting pressure and intensity of working on AWS that aren't as far from the truth as I would've liked. The broader culture inside the company was a big part of what made me want to leave, and I'm not super happy with how the company is operating now. But this isn't that blog post.

In the end, I look back on my time at AWS with a lot of fondness and gratitude. I left an amazing team, and some great leaders and mentors. The common phrase at Amazon when someone left was taking a "promotion to customer", and now it's my turn.