throwaway1280's comments

This is an incredible achievement, but as a musician, I wish this would go die in a fire.

I'm a bedroom hobby musician with no dreams of ever making it big, but even so, I'm looking at the hours I'm spending trying to improve my skills and thinking what's the point, really, when I could just type in 'heavy metal guitar solo at 160bpm, A minor' and get something much much better?

I know there is value in creating art for art's sake. I've always been up against a sea of internet musicians, even when I started back in 2000. But there's just something about this that's much more depressing, when it's not even other people competing with me, but a machine which hasn't had to invest years of its life in practice to beat me.

Eh. I'm in exactly the same group she's talking about (born in 84, so at the upper end of millennial age) and my inheritance has started already. My parents have calculated that they have enough to live on until the time they expect to die, so they're passing any extra onto me early to avoid inheritance tax.

Personally, it kinda grates on me. I'm very much a soft socialist, who believes in everybody paying their fair share of tax. I've also derived a lot of satisfaction over the years in becoming self-sufficient[1] and being able to pay my own way without them having to prop me up. But it's their money and they're free to do with it as they wish.

As it stands, I'm doing the privileged thing that the author mentions, and 'refusing' it: it's going in an investment account and I'll continue to live within the means that I earn myself. But I know that that still means that I'll never be afraid of having to live on the dole, or do a job that I utterly hate.

In any case, even without that money, as an ex-FAANG developer I'm still significantly more privileged than most of my peers: I don't have any debt, I own my (fairly cheap) home outright, and I own a (very cheap) car and motorbike, also outright. Amongst my friends, most of whom are renters working in precarious jobs, I just don't mention money.

[1] How does that work with the socialism? Well - I'm happy to take from the government, because everybody can get the same, and I've repaid that with taxes over the years.

That sounds incredibly rough.

But as a simple counterpoint to this, I'm getting a lot out of therapy, as are most of my friends who are in therapy. It's not doing what I thought it would - it hasn't fixed my depression overnight, or cured my irritability - but it's providing a different framework to think about the world and relate to people, and I like who I am an awful lot more than I like the person I was.

It's expensive for me, at 60 bucks a session on a not-particularly-high wage, and I regularly revisit the idea of stopping, but I think I'm still getting something useful and insightful out of most appointments, so I'm keeping on going.

Just letting you know that I downvoted you not because of the advice - which is pretty good - but because of the link to Jordan Peterson, who is generally a pretty nasty pseudo-intellectual piece of work.

Yup. There was a famous memo from Jeff that went round years ago that read something like "I want to buy a kettle. I search for 'kettle' and I get thousands of products. That sucks - now I have to comparison shop for kettles, when I don't particularly care what sort of kettle I get. Why can't Amazon just tell me 'buy this one'?"

Amazon's choice arrived years after my tenure at the big A, and I'm not sure it's directly connected, but it definitely helps solve this problem.

Yes, indeed.

I've had depression for nineteen years, and because of it I have giant holes in my memory - I know that I went places and did things when I was growing up, but I have next to no memories of any of them. My short-term memory malfunctions a lot - I can go through an entire day, and have zero memories of it the next day. I suspect I'd lose a few more memories with ECT but, really, I would happily pay those memories for having a life where I could actually function.

Speaking of twitchy lawyers and Move to AWS... one of the weirdest things we had to deal with inside Amazon was that, for many years after AWS launched, we weren't allowed to use it because it "wasn't secure enough".

Given that we were actively shopping it around to major financial institutions at the time, doesn't that strike you as particularly hypocritical? :)

Yeah. I did this once a few years ago, and it was quite unpleasant. Did get it done in the end, but it definitely put my team off looking for any other useful NPM packages.

I wonder if it's any more streamlined now?

Ex-Amazon engineer of several years here.

This is a pretty interesting article, but it's important to know that Amazon's internal tooling changes pretty fast, even if it's mostly several years behind state-of-the-art.

Exhibit A: Apollo

Apollo used to be insane. It was designed for the use case of deploying changes to thousands of C++ CGI servers on thousands of website hosts, worrying about compiling for different architectures, supporting special fleets with overrides to certain shared libraries, etc etc. It had an entire glossary of strange terms which you needed to know in order to operate it. Deployments to our global fleet involved clicking through tens of pages, copy-and-pasting info from page to page, duplicating actions left right and centre, and hoping that you didn't forget something.

When I left, most of that had been swept away and replaced with a continuous deployment tool. Do a bit of setup, commit your code to the internal Git repo, watch it be picked up, automated tests run, then deployments created to each fleet. Monitoring tools automatically rolled back deploys if certain key metrics changed.

Auto scaling became a reality too, once the Move to AWS project completed. You still needed budgetary approval to up your maximum number of servers (because for our team you were talking thousands of servers per region!) but you could keep them in reserve and only deploy them as needed.

Manually copying Apollo config for environment setup was still kind of a thing though. The ideas of CloudFormation hadn't quite filtered down yet.

Exhibit B: logs

My memory's a bit hazy on this one. There certainly was a lot of centralized logging and monitoring infrastructure. Pretty sure that logs got pulled to a central, searchable repository after they'd existed on the hosts for a small amount of time. But, yes, for realtime viewing you'd definitely be looking at using a tool to open a bunch of terminals.

The monitoring tools got a huge revamp about halfway through my tenure, gaining interactive dashboarding and metrics drill-down features which were invaluable when on-call. I'm currently implementing a monitoring system, so my appreciation for just how well that system worked is pretty high!

Exhibit C: service discovery

Amusingly, a centralized service discovery tool was one of the tools that used to exist, and had fallen into disrepair by the time this person was working there.

This was a common pattern in Amazon. Contrary to the 'Amazon doesn't experiment' conclusion, Amazon had a tendency to experiment too well - the Next Big Thing was constantly being released in beta, adopted by a small number of early adopters, and then disappearing for lack of funding/maintenance/headcount.

I can't think of any time I hard-wired load balancer host names though. Usually they would be set up in DNS. We did used to have some custom tooling to discover our webserver hosts and automatically add/remove them from load balancers, but that was made obsolete by the auto-scaling / continuous deployment system years before I left.

As for the question of "can we shut this down? who uses it?" - ha, yes, I seem to remember having that issue. I think that, before my time, it wasn't really a problem: to call a service you needed to consume its client library, so you could just look in the package manager to see which services declared that as a dependency. With the move to HTTP services that got lost. It was somewhat mitigated over the years by services moving to a fully authenticated model, with client services needing to register for access tokens to call their dependencies, but that was still a work in progress a few years ago.

Exhibit D: containers

Almost everything in Amazon ran on a one-host-per-service model, with the packages present on the host dictated by Apollo's dependency resolution mechanism, so containers weren't needed to isolate multiple programs' dependencies on the same host.

Screwups caused by different system binaries and libraries on different generations of host were a thing, though, and were particularly unpleasant to diagnose. Again, that mostly went away once AWS was a thing and we didn't need to hold onto our hard-won bare-metal servers.

'Amazon Does Not Experiment'

Amazon doesn't really do open source very well. The company is dominated by extremely twitchy lawyers. For instance, my original employment contract stated that I could not talk about any of the technology I used at my job - including which programming languages I used! Unsurprisingly, nobody paid attention to that. That meant that for many years, the company gladly consumed open source, but any question of contributing back was practically off the table as it might have risked exposing which open source projects were used internally.

A small group of very motivated engineers, backed up by a lot of open-source-friendly employees, gradually changed that over the years. My first ever Amazon open source contribution took over a year to be approved. The ones I made after that were more on the order of a week.

Other companies might regard open sourcing entire projects as good PR, but Amazon doesn't particularly seem to see it that way. Thus, it's not given much in the way of funding or headcount. AWS is the obvious exception, but that's because AWS's open source libraries allow people to spend more money on AWS.

Instead, engineers within Amazon are pushed to generate ideas and either patent them, or make them into AWS services. The latter is good PR and money.

As for different languages: it really depends on the team. I know a team who happily experimented with languages, including functional programming. But part of the reason for the pushback is that a) Amazon has an incredibly high engineer turnover, both due to expansion and also due to burnout, so you need to choose a language that new engineers can learn in a hurry, and b) you need to be prepared for your project to be taken over by another team, so it better be written in something simple. So you better have a very good justification if you want to choose something non-standard.

Overall, Amazon is a pretty weird place to work as an engineer.

I would definitely not recommend it to anybody whose primary motivation was to work on the newest, shiniest technologies and tooling!

On the other hand, the opportunities within Amazon to work at massive scale are pretty great.

One of the 'fun' consequences of Amazon's massive scale is the "we have special problems" issue. At Amazon's scale, things genuinely start breaking in weird ways. For instance, Amazon pushed so much traffic through its internal load balancers that it started running into LB software scaling issues, to the point where eventually they gave up and began developing their own load balancers! Similarly, source control systems and documentation repositories kept being introduced, becoming overloaded, then replaced with something more performant.

But the problem is that "we have special problems" starts to become the default assumption, and Not Invented Here starts to creep in. Teams either don't bother searching for external software that can do what they need, or dismiss suggestions with "yeah, that won't work at Amazon scale". And because Amazon is so huge, there isn't even a lot of weight given to figuring out how other Amazon teams have solved the same problem.

So you end up with each team reinventing their own particular wheel, hundreds of engineer-hours being logged building, debugging and maintaining that wheel, and burned-out engineers leaving after spending several years in a software parallel universe without any knowledge of the current industry state-of-the-art.

I'm one of them. I'm just teaching myself Docker at the moment. It's pretty great.

Worked for Amazon. It was a big issue internally, but we never got a good answer to that question.

Yeah. Amazon loses a small but significant number of potential hires when they learn about the open source clause - often very good people, too. When I left they were actively working on trying to find a solution, as it had been a thorn in the side for years.

Yeah, it's really weird. Amazon has other IP-producing divisions, and even though Amazon Studios is very powerful, there's nothing preventing people from making their own video content. I wonder if it's just down to whoever leads that game studio pushing for such a rule.

Yep. AGPL is explicitly forbidden within Amazon, even for non-customer-facing use, and their open source license lawyers are very twitchy about it.

It's amazing to see how modern conversions work - I'm seriously impressed.

One of my family members has muscular dystrophy, which affects her ability to use and feel her legs. She can still walk very slowly, but she has an adapted car with hand controls. I thought I'd describe it, for those interested in the spectrum of car adaptation.

Her car is a standard mid-size automatic compact, with no adaptations other than a push-pull accelerator/brake hand control.

Unlike Nicola's car above, it doesn't have an electronic bypass for the hand controls - they're completely mechanical, with an ingenious system of sliding linkages connecting the hand lever to the pedals. The sliding links are what allows a pull on the hand lever to push the accelerator without simultaneously pulling on the brake pedal (which would obviously stop the hand lever from moving). They also allow the pedals to be used without the hand control moving, so it's easy for me to swap in to drive the car on long distance trips.

This is amazing. One of my friends group was diagnosed as manic depressive recently. They quit their job, and have major issues with spending hundreds of pounds at a time on random cool gear. This would be a very useful tool to help them make those impulses less damaging.

A bit of a digression, but possibly a useful one for anybody wondering if amphetamines or other drugs might be an easy panacea for depression. The answer may well be "yes, and then very much no."

I had my first (and so far only) ecstasy pill a year or so ago, and it was wonderful - lit my brain on fire, caused me to experience the best music festival I've ever been to, and gave me the kind of intense connection with my friends I'd been missing from years of crippling depression. The following week was hell, and caused slight but permanent damage to my family relationships. It took a couple more weeks to recover to even my low standard of 'normal'.

It was only later that I learned that ecstasy, amphetamines, LSD, and basically anything that acts on the seratonin system can cause severe depression lasting a week or more - and if you've already got depression, that can get really bad.

It's worth being careful with these things.

On the other hand - a similar experiment a couple of months later with a small dose of synthetic psilocybin yielded a week of the best mood I've been in for years, with no obvious side effects.

I'd love to see more research on these things. I'm probably not going to do much more research on it using my own brain ;)

Forgot to point out: I'm piscetarian as well. Which seems like a ridiculous self-imposed challenge ;)

That makes getting ready-made food harder, too: my supermarket only does a couple of vegetarian/seafood ready-meals per category (two curries, three italian meals, no Chinese meals at all!) so I find myself eating a very monotonous diet anyway.

Nevertheless, I don't use Joylent full-time: I just have a box in my cupboard for when I need an emergency meal or just can't face another plastic tub of macaroni cheese.

I'm a pretty reasonable cook, actually. I spent a couple of years challenging myself to improve: I can make a reasonable pizza with home-made base and sauce (albeit American deep pan style rather than Italian), I'm fond of cooking Chinese food (because in the UK takeaway Chinese is very hit-and-miss), and I enjoy baking.

I haven't done any of that for years, because now I'm living with crippling depression, and I often face the choice of spending my energy for the day on work, on housework, on life maintenance, or on eating well. I only get to choose one at the most. So most days I live on ready meals, supermarket pizza, pasta, etc. I buy Joylent (EU version of Soylent) because I reckon it's got more nutritional value than most of the ready meals I'm eating, and it's very quick to make - useful on days when I have to drag myself out of the house and don't have time for breakfast. It's also cheap - which is good, because depression means that I don't get many billable hours worked these days.

So, yeah. I'm probably nowhere near the common case here, but I just wanted to present a case study to show that not all of us meal-replacement users are just "people who haven't learned to cook properly yet" or "people who don't care about food". We're a diverse bunch.