It'd be great to read about all the things this internal DevX org has been able to improve after some time.
Internal developer experience is extremely important. One codebase I worked on used to have a 24-hour, nondeterministic CI run. A teammate brought that time down to ~45 minutes (it's a big codebase with a ton of tests) and made it deterministic. Everything got better:
* Faster turnaround time for external contributions (it's OSS)
* Ability to be relied on to get a critical fix in and tested in under an hour
* Much faster local build times
* Removed reliance on some old busted internal infrastructure so we could use much better, different infrastructure, and make the build/test runs the same across any OS rather than rely on windows so much
* Everyone was happier in general
If we could have gotten a full CI run down from 45 minutes to 15 minutes, that'd be great. And there was still quite a lot of technical debt in the tests themselves, since they should be able to run a lot faster than they did. But even with that in mind, the difference was night and day.
I would love to find a full-time position of just doing this kind of work. Nearly every place I've worked had some hacked together system that mostly worked, but wasn't enough of a pain for management to dedicate time to. I'd have to steal time away from the feature mill to make some little improvement, and everyone would agree that, yes, it did make things a easier for the devs, but no, we can't just have you out there working on things willy-nilly when there are features to churn out!
This kind of stuff is right my ally and I'd love to connect with someone at Amazon about how they're tackling these problems. Does anyone know someone that would like to grab coffee to chat about it?
At almost every company I’ve worked for I would estimate that if I worked on or made changes like you have described the benefit to cost ratio would be multiples larger than any of the actual features I worked on. This was always perplexing until I realized my assumption that managers were given a pool of money and were judged on the ROI they generated on that money was false.
Hmm the title made me believe this was to fix the use-and-throw-workers culture. Didn't realise it was about fixing tooling. Wonder if anything else was done to fix the disregard for wellbeing and safety?
Amazon was a good gig to join (on the AWS side) 4 years ago, not so much now.
Reason being everyone I know who got in around 2018 has options that cost a fraction of the current stock price. The job is still boring/miserable but when you are clearing 600k+ annual income...
Now if you join, you can bet its a far worse deal and you will make far less than those who just happened to join earlier, even if they're checked out...
There in lies one of their problems, the incentives are for checked out people to stay and 'click buttons' while performing as much CYA as possible.
I'm hearing tales left and right like that; one part of the tales is that you get stocks or stock options, BUT you have to eat a shit sandwich for years before you can 'cash out'. This is the companies' golden handcuffs; they know the job is shit, but you'll get a big payout if you manage to keep it up for at least X time.
The other one is that compensation and competition is sobering up. I have one former colleague who claims that they earn 3x as much working for a US company than they did our previous job, but I don't see much exorbitant wages being thrown around. Second, that company he works for now has set a hiring stop; the explosive growth and demand from US based tech companies is slowing down or coming to a halt.
not sure what you are saying the stock lasts 4 years any new stock you getting in between will be at market price so you not gonna clear 600k after 4years but only base +new stock..
if amazon stock goes up the next 4years newcomer will get 600k too
> In an email to Insider, Amazon's spokesperson confirmed the existence of the team, saying it's part of an effort to maintain its Day 1 culture.
This terminology is bullshit. As someone who was actually there at Day 1 of amzn, it's completely clear that the term "Day 1 culture" is truly meaningless. It refers not to a point in time, but to some management-imposed aspiration in which things never actually settle down to a routine, but instead everyone feels "empowered" by running around constantly reinventing everything. Of course, this is sold as "you can still make a big difference here", when in reality, you almost certainly cannot.
Amazon's actual Day 1 culture meant bringing up a couple of SPARCstations and getting the T1 line functioning.
This type of bureaucracy is not only harmful to dev culture, it's also outrageously expensive. I've been on an engagement with a large healthcare provider and the dev culture there is very similar to Amazon, loads and loads of procedure and bureaucracy. Now i get a certain level of that at any company, but you often find yourself going: "does it really need to be this hard". Tasks that would take you an hour outside of this companies ecosystem would take you 3-4 inside the ecosystem.
It's annoying to devs but should be alarming to management. You're literally spending 2x to 3x the required amount on your labor because of your inefficiencies. Personally I found it easy to convince management of changes like this when you pitch it like: "How would you like to reduce your labor costs by 2x". Now granted many people at these places are salaried which makes this a bit of a moot point but most software shops contract out alot of work, and that's where you're really paying for your inefficiencies.
While it's true that arbitrary bureaucratic inefficiencies can get out of control, sometimes "good" bureaucratic requirements can get inflated with the arbitrary ones, simply because people don't understand their reason in the first place. This leads to a Chesteron's fence scenario [1].
Anytime somebody say's something like "does it really need to be this hard" or "all you've got to do..." should make sure they fully understand the rationale behind the requirements. A lot of times, those requirements can be met in a more efficient manner, but sometimes the seemingly "efficient" path will cause more second-order problems down the road because of those blind spots. What you often find in both camps (those bluntly instituting the requirements and those saying they should go away) is a lack of understanding of the entire system dynamics.
> Now granted many people at these places are salaried…
This shouldn’t make a difference because in theory they would be doing 2x more work which is essentially cutting their salary in half. I realize that the dev team being twice as efficient doesn’t necessarily double the unit’s revenue but it’s some multiple.
At the moment, the general impression I get from people working at Amazon is "Join for the experience, get it on your CV, move on after a year". This is even across departments and countries. The wife of a very good friend who lives in China is working there since nine months in some kind of operations role, and made up her mind months ago that she wouldn't take it any longer than a year. Here in Berlin, I just recently talked to a dev who left exactly a year after he started. He said it was interesting seeing the whole AWS machinery from the inside, and there was quite a lot to learn in terms of process and management, but shoving around legacy Java was just not worth it.
Current AWS employee of 1 year, opinions are my own
I largely agree that for me, working here feels "day 2", which just means it feels the same as working for any other mid to large sized corporation. Lots of meetings, building consensus, tasks that amount to overhead, and adjusting slowly moving pieces to advance whatever the corporate agenda is. Many of these things are directly contradictory to amazon's stated leadership principles (core values) but work happens at the scope of teams not at the scope of leadership principles.
Some of it is team-dependent (not unrelated, my team has really good work life balance. I slog through annoying corporate BS 9-5, but don't work much beyond that) and some of it is related to the builder tools as mentioned in the article.
Is it just the luck of the draw who ends up on the teams with good managers or can you dig your heels in if you end up on one of the bad teams and just refuse to stay late? Or can you easily switch teams until you find a good one?
I know a few senior engineers / manager types who left financial services tech to work at Amazon. Most of them left within two years, and the ones that stayed quickly got out of AWS org because it was too crushing.
In a way I kind of respect Amazon for treating their white collar tech staff as poorly as their blue collar warehouse staff.
What about the hire to fire, and crazy amount of pips they do? How are they going to fix that part of the culture? These two are the biggest reason why I turn down amazon recruiters.
Stripped of fluff, the main point of the article seems to be that their engineers are doing too much manual repetitive work, due to bad tooling and processes. Why would that be caused by their performance review system?
TBF, that was scrapped in 2017. Source: worked there.
There was a clear culture shock for some of the more seasoned Amazonians (and you could see a fall out effect with some "mails to your manager" every now and then if you crossed a Grand Poobah the wrong way), but to all of us "newer" hires, the times of adversarial review were just a story.
Have never worked at Amazon nor do I know anyone who did, what exactly did that entail? "Snitching" on your co-workers? I'm asking because a quick web search didn't help me too much.
This is purely a PR stunt for the shareholders confidence in Andy's leadership. We've been screaming from the rooftops for years, then they hire someone even worse as HR head, Beth Galetti, who clearly shares distain for the companies talent.
Andy, you've known about the culture problems for years, the fact that you're just now acknowledging them is only because the business is hurting, and burning out all the talent. You're scared shareholders will get involved if this isn't fixed. Sadly, this won't fix anything, this is the culture YOU built.
"...the retail giant has pledged to become the Earth's Best Employer, not just for frontline warehouse workers but for corporate and technical employees, too..."*
* As long as it doesn't interfere with the share price and exec bonuses
Barely a day goes by without a story about how badly Amazon treats it's non-tech workers. Combine that with the stack ranking and, like you, I'm not convinced.
I would argue you can't fix the tech employment without fixing the rest. One bleeds into the other. I would never consider working for a company that can't treat it's more replaceable employees well.
Internal tooling is a mess at most companies I've worked at. At a certain scale it's worth the savings to standardize, improve, and reduce the shared friction costs.
Stripe, as it has scaled, has ran into challenges and invests in the same efforts. FB internal tooling was (is?) a mess. It goes on. The surprise here is it's such a late investment from Amazon, but it's worked for them thus far. I'm guessing the returns on +1 engineer has plateaued or declined, leading to this effort.
I quit a few months back and hope it continues to crumble. You won’t be missed, Amazon. After 5 years and countless long nights, I was thanked with a pay cut and ignored when petitioned, even after rearchitecting the entirety of one of their floundering subsidiaries and leading a team of 10 thru it’s execution. The reason? One of my junior engineers missed a deadline. When petitioned, my cries were ignored and I was recommended to boomerang if I wanted a pay raise.
At the same time, a friend was interviewing for the exact same senior engineering role and received an offer. The total compensation he was offered in his first year was nearly double my pay. He didn’t even take it - another company offered him more.
To be frank, this compensation issue isn't a problem specific to Amazon. It's common knowledge that if you want significant raises (50%+) you have to jump among the FAANGs/Tier1 companies, mainly due to the external vs internal offer policies. An internal promo will get you into the bottom of the next pay band versus an external offer which usually puts you near the top. The compensation delta can be in the 6 figures for non-new grad roles. This holds true all the way up to director level roles in my observations.
Fair or not, this is currently the optimal career/comp progression strategy, although recent hiring downturns might change the meta(heh)game to favor internal candidates. We shall see.
Stories like this are the reason why I keep turning down Amazon recruiters, not the lack of shiny/speedy tools. This article is fluff. And I'm glad you got out.
Thank you for sharing your experience. I'm sure there might be some guy going "but, but, it depends on your team! I've never seen this" --but Amazon has a large variety of bitter ex-employees and horror stories.
RE: Boomerang, Amazon changed the policy recently. If you return within one year at the same level, you get the same comp you left at. A comp increase requires a new level and a full loop. The upshot is that you have to have proof of impact for that new level for boomerang to be worth it.
Lucky ducks. Development at my org largely consists of going around to every different possible "partner" to get buy in to have a meeting, to discuss approval of permissions to do something. I think the new metric used to asses an organization is PPS (Powerpoint Slides) to LOC ratio.
Jassy has used the "builders" term for years while leading AWS. It was a central theme in many of his keynote speeches. I think it was chosen specifically because it was not an industry-standardized (and watered-down) term.
Amazon has had an org called 'Builder Tools' or some variation on that for over a decade (maybe much longer?). That team has historically owned the systems for build, deployment, source control etc. AWS also uses the 'builder' terminology a lot in their marketing. E.g. Werner saying 'Now go build!' at the end of his keynotes, and the 'Build On AWS' campaign.
AWS console is one of the most poorly developed builder tools Ive ever encountered for a company like Amazon, from a UX perspective at least.
I may like some of the services like s3 but will never rely on AWS for anything more that that. Logging into the aws console makes my blood pressure go up.
What do people think of DevX groups in general? I'm not sold, but in my (only n=1) experience, it'd be best to try to address the cultural problems that led to the perceived need for DevX.
For instance : if build times are really bad, maybe it's because developers don't feel empowered to fix this as part of their day-to-day work.
Management at Amazon was the biggest problem. The don't index for soft skills. They want procedural robots that can memorize 100 stories to cover every LP.
Culture fit matters and you can't fix the culture when thats what they are hiring for.
[+] [-] firstSpeaker|3 years ago|reply
[+] [-] phillipcarter|3 years ago|reply
Internal developer experience is extremely important. One codebase I worked on used to have a 24-hour, nondeterministic CI run. A teammate brought that time down to ~45 minutes (it's a big codebase with a ton of tests) and made it deterministic. Everything got better:
* Faster turnaround time for external contributions (it's OSS)
* Ability to be relied on to get a critical fix in and tested in under an hour
* Much faster local build times
* Removed reliance on some old busted internal infrastructure so we could use much better, different infrastructure, and make the build/test runs the same across any OS rather than rely on windows so much
* Everyone was happier in general
If we could have gotten a full CI run down from 45 minutes to 15 minutes, that'd be great. And there was still quite a lot of technical debt in the tests themselves, since they should be able to run a lot faster than they did. But even with that in mind, the difference was night and day.
[+] [-] zach_garwood|3 years ago|reply
[+] [-] johnbellone|3 years ago|reply
[+] [-] 0x445442|3 years ago|reply
[+] [-] flashgordon|3 years ago|reply
[+] [-] bearjaws|3 years ago|reply
Reason being everyone I know who got in around 2018 has options that cost a fraction of the current stock price. The job is still boring/miserable but when you are clearing 600k+ annual income...
Now if you join, you can bet its a far worse deal and you will make far less than those who just happened to join earlier, even if they're checked out...
There in lies one of their problems, the incentives are for checked out people to stay and 'click buttons' while performing as much CYA as possible.
[+] [-] Cthulhu_|3 years ago|reply
The other one is that compensation and competition is sobering up. I have one former colleague who claims that they earn 3x as much working for a US company than they did our previous job, but I don't see much exorbitant wages being thrown around. Second, that company he works for now has set a hiring stop; the explosive growth and demand from US based tech companies is slowing down or coming to a halt.
[+] [-] ogn3rd|3 years ago|reply
[+] [-] retinaros|3 years ago|reply
[+] [-] PaulDavisThe1st|3 years ago|reply
> In an email to Insider, Amazon's spokesperson confirmed the existence of the team, saying it's part of an effort to maintain its Day 1 culture.
This terminology is bullshit. As someone who was actually there at Day 1 of amzn, it's completely clear that the term "Day 1 culture" is truly meaningless. It refers not to a point in time, but to some management-imposed aspiration in which things never actually settle down to a routine, but instead everyone feels "empowered" by running around constantly reinventing everything. Of course, this is sold as "you can still make a big difference here", when in reality, you almost certainly cannot.
Amazon's actual Day 1 culture meant bringing up a couple of SPARCstations and getting the T1 line functioning.
[+] [-] _fat_santa|3 years ago|reply
It's annoying to devs but should be alarming to management. You're literally spending 2x to 3x the required amount on your labor because of your inefficiencies. Personally I found it easy to convince management of changes like this when you pitch it like: "How would you like to reduce your labor costs by 2x". Now granted many people at these places are salaried which makes this a bit of a moot point but most software shops contract out alot of work, and that's where you're really paying for your inefficiencies.
[+] [-] bumby|3 years ago|reply
Anytime somebody say's something like "does it really need to be this hard" or "all you've got to do..." should make sure they fully understand the rationale behind the requirements. A lot of times, those requirements can be met in a more efficient manner, but sometimes the seemingly "efficient" path will cause more second-order problems down the road because of those blind spots. What you often find in both camps (those bluntly instituting the requirements and those saying they should go away) is a lack of understanding of the entire system dynamics.
[1] https://fs.blog/chestertons-fence/
[+] [-] foobarian|3 years ago|reply
[+] [-] 0x445442|3 years ago|reply
This shouldn’t make a difference because in theory they would be doing 2x more work which is essentially cutting their salary in half. I realize that the dev team being twice as efficient doesn’t necessarily double the unit’s revenue but it’s some multiple.
[+] [-] afroisalreadyin|3 years ago|reply
[+] [-] breitling|3 years ago|reply
I'm starting to feel very old
Is Java considered legacy now?
[+] [-] helen___keller|3 years ago|reply
I largely agree that for me, working here feels "day 2", which just means it feels the same as working for any other mid to large sized corporation. Lots of meetings, building consensus, tasks that amount to overhead, and adjusting slowly moving pieces to advance whatever the corporate agenda is. Many of these things are directly contradictory to amazon's stated leadership principles (core values) but work happens at the scope of teams not at the scope of leadership principles.
Some of it is team-dependent (not unrelated, my team has really good work life balance. I slog through annoying corporate BS 9-5, but don't work much beyond that) and some of it is related to the builder tools as mentioned in the article.
[+] [-] ghostpepper|3 years ago|reply
[+] [-] steveBK123|3 years ago|reply
In a way I kind of respect Amazon for treating their white collar tech staff as poorly as their blue collar warehouse staff.
[+] [-] ogn3rd|3 years ago|reply
This isn't always the case. Ask a TAM. With the right blend of customers I've known some to work about 10 hours a week.
[+] [-] champagnepapi|3 years ago|reply
[+] [-] endemic|3 years ago|reply
[+] [-] snarkerson|3 years ago|reply
[+] [-] jsnell|3 years ago|reply
[+] [-] choudharism|3 years ago|reply
There was a clear culture shock for some of the more seasoned Amazonians (and you could see a fall out effect with some "mails to your manager" every now and then if you crossed a Grand Poobah the wrong way), but to all of us "newer" hires, the times of adversarial review were just a story.
[+] [-] jejeyyy77|3 years ago|reply
[+] [-] paganel|3 years ago|reply
[+] [-] ogn3rd|3 years ago|reply
Andy, you've known about the culture problems for years, the fact that you're just now acknowledging them is only because the business is hurting, and burning out all the talent. You're scared shareholders will get involved if this isn't fixed. Sadly, this won't fix anything, this is the culture YOU built.
[+] [-] stevievee|3 years ago|reply
* As long as it doesn't interfere with the share price and exec bonuses
[+] [-] vertis|3 years ago|reply
I would argue you can't fix the tech employment without fixing the rest. One bleeds into the other. I would never consider working for a company that can't treat it's more replaceable employees well.
[+] [-] ctvo|3 years ago|reply
Stripe, as it has scaled, has ran into challenges and invests in the same efforts. FB internal tooling was (is?) a mess. It goes on. The surprise here is it's such a late investment from Amazon, but it's worked for them thus far. I'm guessing the returns on +1 engineer has plateaued or declined, leading to this effort.
[+] [-] mv4|3 years ago|reply
[+] [-] mmmeff|3 years ago|reply
At the same time, a friend was interviewing for the exact same senior engineering role and received an offer. The total compensation he was offered in his first year was nearly double my pay. He didn’t even take it - another company offered him more.
[+] [-] itsyaboi|3 years ago|reply
Fair or not, this is currently the optimal career/comp progression strategy, although recent hiring downturns might change the meta(heh)game to favor internal candidates. We shall see.
[+] [-] joezydeco|3 years ago|reply
[+] [-] oneepic|3 years ago|reply
[+] [-] coredog64|3 years ago|reply
[+] [-] spelunker|3 years ago|reply
[+] [-] bogomipz|3 years ago|reply
Is petitioning a process in AWS? What does boomerang mean here?
[+] [-] jollyllama|3 years ago|reply
You guys get to click buttons??
[+] [-] buscoquadnary|3 years ago|reply
[+] [-] twic|3 years ago|reply
[+] [-] sokoloff|3 years ago|reply
https://aws.amazon.com/builders-library/ among many other AWS docs that use the builders language.
[+] [-] discodave|3 years ago|reply
[+] [-] fjabre|3 years ago|reply
I may like some of the services like s3 but will never rely on AWS for anything more that that. Logging into the aws console makes my blood pressure go up.
[+] [-] jvolkman|3 years ago|reply
[+] [-] phillipcarter|3 years ago|reply
[+] [-] pyb|3 years ago|reply
For instance : if build times are really bad, maybe it's because developers don't feel empowered to fix this as part of their day-to-day work.
[+] [-] angryasian|3 years ago|reply
Culture fit matters and you can't fix the culture when thats what they are hiring for.