Working at Amazon

Sometimes when we’re trying to hire a senior employee, I get asked to do a “sell call”, tell them what it’s like to work here. Since I’m coming up on three years and haven’t quit, I guess the supposition is that I’ll be positive. Since these candidates are outsiders and some of them don’t come to work for us, nothing I can say can be a secret. So why don’t I tell everyone?

Before I dive in any further, everything here relates to AWS, not Amazon as a whole. It might be true of the retail side too, but I don’t know because I don’t work there.

So, what is it like to work here?

Customers · That’s the big deal. AWS has over a million and you get to meet them all the time, and while there are lots of geek introverts who might not like that, I do. Anyhow, lots of the customers are geek introverts too.

When you ship low-level horizontal general-purpose technology, there’s no way to predict what people are going to do with it. Which in practice means that you’re going to be shocked by the way it gets used. For an old guy like me it’s hilarious to hear the kid engineers looking at a bug report or feature request and saying “WTF, why would anybody want to do that?”

Me, I like computers, and I like hearing about interesting things people do with them, and it makes me happy when the things they do work out well, especially when I helped build what they’re using. I could tell you stories that’d curl your hair (or straighten your wavy locks) but they’re not mine to tell.

Anyhow, if you like talking to customers you’ll like working here, and if not, probably not.

Uptime · It matters more than anything. Some of our services are cooler than others, but what I think customers care about most is confidence that the services, cool or boring, will be there 24/7/365. What that means is that everything has to be automated, and much of the most brilliant engineering at AWS, done by some of the smartest people, does its work behind the scenes where nobody will ever see it.

So if you’re the kind of person who, for example, thinks figuring out a better way to automate detecting hot-spots in back-end clusters and re-routing traffic to cool things down is interesting work, then you’ll like working here. If not, maybe not.

Only maybe not, because we have lots of people who wrangle JavaScript frameworks and mobile SDKs and build tools and database kernels all day. But operational automation, that’s the heart of the matter.

Being document-driven · Go ahead and Google “Amazon six-pager” if you haven’t already.

If you’re the kind of person who’s OK with spending a lot of time constructing carefully-written narratives, and being in meetings that start with 20+ quiet minutes while everyone reads the narrative, you’ll like working here, and if not, definitely not. Disclosure: I’ve written over a million words on this blog so I’m seriously predisposed to like this part.

There is one downside: Suppose you’re an author of the document being read. It can be simultaneously crushingly boring and twitchingly nervous while AWS’s senior leadership plows their way through your words in stony silence.

The actual software · I don’t think anyone would be surprised. The services are usually conventional RESTful services, usually composed of multiple microservices, usually in mainstream programming languages using mainstream programming frameworks, We use conventional compilers and debuggers and IDEs and frameworks for HTTP and unit testing and integration testing and dependency injection and so on.

Yes, there’s some secret sauce voodoo down in the infrastructure that’s pretty magical. But most of us don’t work on that most of the time.

What’s unusual is the proportion of the code focused on availability and data durability. But you already knew that. We also care a lot about doing better than O(N), because N is typically so freaking huge at Amazon. I personally seem to end up working mostly on message processing of one kind or another which is sort of stuck on O(N) in the number of messages, so we’re often left with micro-optimizations.

You know what makes me happy? One little library I wrote is currently being used by several different teams to efficiently do a useful thing on a million-ish messages per second in aggregate. There aren’t that many places where your code gets that kind of opportunity.

Organization and culture · The teams at AWS have a huge amount of independence. If you’re on a team that’s trying to ship something or operate something or improve something, that’s generally great; nobody will get in your way. On the other hand, if you’re trying to build something that works across multiple service teams, it can give you grey hairs. But I already had those; and it’s mostly a feature not a bug.

Corporate “culture” is a thing I find it hard to be articulate about. I find the asshole density nonzero but lower than average. We are trying to do very difficult things and we fail a lot; Mature technology companies try make those teachable moments, and we do a pretty good job of that. There is [*gasp*] politics. I’m going to argue that the proportion of really bad managers is unusually low; but that’s just an anecdote, wouldn’t know how to measure it.

My time rarely feels wasted.

Money? · If you work in a tech job at any large high-tech company, you’ll be well-paid by any reasonable standard. Want more than just “well-paid”? The brutal truth is that the way high-tech compensation is structured, you’re making a bet on your employer’s share price. If it goes up really a lot while you’re working there, you’re gonna make out like a bandit, and if not, not. All these things are as true at Amazon as anywhere else. Welcome to twenty-first century capitalism.

What sucks? · After I’ve talked through all this stuff, the person who might come to work for us wants to hear about the downside. Fair enough.

I guess the biggest one is that we’re not perfect at operational automation, so everything we fail to automate, every combination of compute and network and storage failure we weren’t smart enough to auto-remediate in advance, has to be dealt with manually. Which means people have to be on-call and yeah, sometime the phone goes off after midnight.

The good news is, we’re getting better, learning how to build automation in at the core. Which means that the fresh new AWS services are better at self-maintaining and self-healing, so on-call sucks less. On the other hand, the older services tend to have bigger teams and thus less on-call time per person, so it evens out.

But there are some people who sufficiently dislike the prospect of that occasional wee-hours page that they just don’t want to stick around with us. Fair enough.

I dislike our office-automation setup, but then I’m the kind of person who really likes Google Docs and Calendar and Gmail, so maybe I’m an outlier.

One more thing: Most jobs at AWS, you’ll never be able to explain to civilians what it is you do for a living. I generally say “I help keep the Internet running” and you know what? That’s not a total lie.

On balance · I could go on about diversity and work environments and benefits and process and promotion and so on, but at the end of the day it’s just another high-tech company, nothing surprising.

I don’t really need the money, but I haven’t quit.

---

---

Contributions