There are a bunch of comments saying that Google is just like any other big tech company and that the exciting engineering bit has gone. My experience is only from the last 2.5 years, but I've got a slightly different take.
Engineering from >10 years ago seems like it was a wild west. Some truly stunning pieces of technology, strung together with duct tape. Everything had its own configuration language, workflow engine, monitoring solution, etc. Deployments were infrequent, unreliable, and sometimes even done from a dev's machine. I don't want to disparage engineers who worked there during that time, the systems were amazing, but everything around the edge seemed pretty disparate, and I suspect gave rise to the "promo project" style development meme.
Nowadays we've got Boq/Pod, the P2020 suite, Rollouts, the automated job sizing technologies, even BCID. None of these are perfect by any means, but the convergence is a really good thing. I switched team, PA, and discipline 6 months ago, and it was dead easy to get up and running because I already knew the majority of the tech, and most of that tech is pretty mature and capable.
Maybe Google has become more like other tech companies (although I doubt they have this level of convergence), but I think people glorify the old days at Google and miss that a lot of bad engineering was done. Just one example, but I suspect Google has some of the best internal security of any software company on the planet, and that's a very good thing, but it most certainly didn't have that back in the day.
I left over ten years ago and it's hard to understand that perspective. Back when I was an SRE (~2006 to 2009) there were only one or two monitoring systems (which didn't overlap, so you could argue there was one) and a handful of config languages. Compared to anywhere else Google had military levels of discipline and order.
> Deployments were infrequent, unreliable, and sometimes even done from a dev's machine.
Deployments were weekly and done from a dev machine because that way someone was watching it and could intervene in case of unexpected problems. Some teams didn't do that and tried to automate rollouts completely. I could always tell which products weren't doing enough manual work because I'd encounter obviously broken features live in production, do a bit of digging and discover end user complaints had been piling up in the support forums for months. But nobody was reading them, and the metrics didn't show any problem, and changes flowed into prod so the team just ... didn't realize their product wasn't working. There's no substitute for trying stuff out for yourself. I encounter clearly broken software that never seems to get fixed way too often these days and I'm sure it's partly because the teams in question don't use their own product much and don't even realize anything is wrong.
Although there were some rough areas, Google has been considered to have pretty advanced software engineering for 20 years or more. There was plenty for a new engineer to boggle at even then.
On the other hand, someone who started in 2015 missed out on the years when Google was mostly considered to be the good guys and given the benefit of the doubt. That’s about when the culture started turning against “big tech” in general (rather than specific companies like Microsoft).
From my experience crossing eras; the tools have improved, and there are still some amazingly brilliant people, but the amount of overhead and the ratio of top level, passionate builders has dropped meaningfully. I have had a priority project stuck in legal hell for six months because it’s just a morass of logistics. The project is benign but it doesn’t matter. We are ossified.
I look at it as somewhat inevitable considering the path the company has taken, but it is certainly different. We are cranking out money and that’s fine, but it is a change.
"Bad engineering" in adhoc ways tends to mean new ideas are being explored. Sophisticated deployments, logistics, procedures, tends to mean you're optimizing or extending existing system.
That's not to disparage the latter. making things work at scale is hard engineering. But when people praise the glory days, it may be a preference for working on new ideas in small projects.
On the flip side, clueless execs are killing tools like Code Search because they don't understand the value. They're willing to layoff off a team to save money even if it reduces the productivity of 20,000 employees by 1%
What’s the point if the capacity to deploy good products has been lost in the process? What if what you see as unreliable was at the hearth of what enabled great things to happen. This is a typical caveat of a programmer centric organization. Things get more reliable but more frozen and innovation die.
So sure they can very reliably and safely deploy... It just doesn't feel like they've actually deployed anything interesting in 5+ years (except maybe some AI stuff). From the outside it looks like the company is perpetually refactoring
history's biggest yakshave...? not surprising for a programmer-run company
I'm sure it still matters to bring down the power and server bills. But one can't help feel like they could be doing much more
I've done the path from SWE to SRE and back to SWE. I was always happy to do production support and diagnose and fix production problems, so I naturally moved to SRE which is always looking for people.
It was a real mistake, SRE is hugely stressful and really unrewarding compared to SWE. Yes you learn some skills and get some occasional glory, but year after year of fighting fires really didn't build any long lasting career.
After switching back to SWE I've finally got promotions and pay rises again, as well as good night sleep and much less stress.
And for anyone still considering SRE: when interviewing, ask how incentives/bonuses/promotions work for software engineers and how does that compare to SREs. A lot of promotable activities for SWE (shipping new things constantly) have negative value for an SRE role, since continually changing infrastructure ensures on-call never develops mastery of those systems.
I've been running as SRE for the last decade, running critical stuff like authentication (which mostly all services depend on). I'm a software engineer.
I cannot disagree more: our team is healthy, oncall is quite a fine activity to do (and compensated, of course), we have plenty of engineering work to do.
I've had five promotions (and tripled salary) and done so working on plenty rewarding activities over time. I've done from deployment automation, to capacity planning, distributed system design, large data migrations, designing ietf standards for auth protocols, wrote client sdks, now we even do AI for different things (including model development).
I'd recommend to not generalize from "I didn't like it / the experience wasn't a match for me" to "the role is shitty".
I started in basic IT, worked into SRE, and once I had established an understanding of it, immediately did whatever I could to transition into full time SWE. I went in hoping it would be cool coding to automate infrastructure, but it felt mostly like being tier 4 support.
I miss working in software that also utilizes my skills in infrastructure, but I do not miss the constant escalations, terrible on-call schedules, and only about 20% of my time being spent on the rewarding parts of the job.
Thanks for recognizing what SRE/SysAdmins have to deal with all the time.
I think in general that type of experience would help Developers be more empathetic to the operations side of things. Those fires often come from trade-offs made in development.
Maybe you should clarify what company you worked in, because that's clearly not Google. In Google, SREs do 12 hour shifts. Night shifts (or rather 24 hour shifts) are (ironically enough) usually done by SWE teams. Almost every SWE team in Cloud has a 24 hour shift, and I agree, they are quite terrible.
SRE is an anti pattern that Google is unwilling to admit and is selling books on. Just like there should not be QA, release engineering, continuing engineering or DBA as separate departments/job titles, because these critical parts of software development should not be considered optional and thrown over the wall to take care of by someone with no stake in developing the product.
I've always wondered if privacy conscious engineers who work at Google do actually use Google's services for their personal lives (Google Drive, Google Photos, Gmail, Google Calendar, Google Keep, Google Docs, etc.)? And if so, do they continue to use them when after leaving the company?
I ask this question here because there seem to be quite some (ex-)Google employees in this thread.
I work at google and I use google products. Sure, some giant automated heap of code is processing your data and deciding how many grammerly adverts to show you, but your data is about as safe as it can be from loss or from humans. There are so many controls and checks in place when working with user data it's difficult to get things done some times (and quite rightly too).
Google is large enough that you will hear plenty of opinions on this. In my bubble (Zurich, Security) the overall sentiment seems to be "I trust Google handling my data way more now compared to before I joined".
All these things make me envious of people who get to work at Google or any other FAANG company, where they are both paid well and validated for their intelligence.
These are great benefits for sure, but one of the reasons I left is a totally ineffective and wasteful management structure that makes it extremely hard to actually do anything. It is very hard to feel like your work means anything at a company of that size, since the chances of having the goals of your work changed (or being laid off) at a moments notice makes it hard to stay motivated.
If I take another software gig it will certainly be at a small company where my daily work contributes directly to the company’s central goals.
The validated for their intelligence thing is a problem. Googlers in the early days constantly told each other that they were the "smartest people in the world". I noticed this immediately after joining and found it quietly troubling, because there's all kinds of smart and the interview processes were really only selecting for skill with computers or maths. A lot of Googlers wouldn't survive long on the streets or alone in the wilderness, as these require different kinds of smart to what they had. Some colleagues were troubled by those statements for other reasons: they didn't personally feel like one of the smartest people in the world, and this led to imposter syndrome. But we said nothing because, hey, the company was doing great and people did seem pretty smart overall back then. Plus nobody wants to call themselves out as an imposter, and Google had a certain degree of institutional humility to it as well. The company was very much about empowering everyone, no matter who or how "smart" they were.
What you're seeing from Google in the last ten years is a maybe predictable consequence of that culture, where some Googlers really do seem to think they're generically much smarter than everyone else, about everything. You started to see mass scale social engineering via manipulation of search results and products, driven apparently by the immense faith they have in their own wisdom. Is there any claim Googlers cannot immediately resolve as true or false given nothing more than a few ML models and a team of contractors in LatAm? Apparently some of them think that's all it takes.
This quasi-misanthropic culture is miles away from the trusting "make it universally available and useful" culture the company once had, but the seeds of that culture's end were clearly visible even at the start. You can't constantly validate people by telling them they're super smart before some of them come to actually believe it, and that leads naturally to the belief that if they're really the smartest people in the world then surely that means they should be running it.
its the proximity, pedigree, profile that you have to fit to get in to Google
I'm happy for those that made it. Not everybody gets to work for Google. But the work they do are no less challenging or more important than what the rest of us do.
If anything FAANG has contributed greatly to the American Firewall of Algorithms and have destroyed an entire generation's ability to reason and value common sense.
I remember this quote which I can't remember who said but "if they are paying you a large salary, what they take from you is far greater"
The traditional view is that young engineers should join startups in hope of a massive payoff if one goes big, but working for 10 years at BigCo with good salary and stock plans can set you up for life without any risk. One path to avoid is the one I took which was to work at a big tech co as a contractor. Good rates but nothing to show for it after 10 years other than the experience and whatever I had put in my 401K.
Another path to avoid is working "W2" for a BigCo via a contractor. You don't get any of the security or compensation, and you may not even get the benefit of the name for your resume.
I try not to be jaded having never cleared the dreaded google interview multiple times, but I'm very envious of the high compensation, amenities (which their recruiters argue is compensation), and probably getting paid doing very technically challenging work. For me in Australia, it seems to be a case of choose one.
> USA centric culture, if you are not in USA at Google and don’t have a big presence in a location it’s a bit like swimming upstream, it’s easy to feel isolated, sidelined or on the flipside overwhelmed with late meetings
It's like the fourth time I read/hear this. I understand that it's a tricky one to adress.
Thanks for writing this! I work on AppEngine / Serverless as a SWE. Nice to see that you worked on it as an SRE and I can totally relate to the cognitive complexity of the systems! :)
The money alone is good enough. I can probably just retire with all those money and stocks. Let alone tons of techs and a shiny CV. Kudos for starting from a height most people can't reach for life. I guess you are going to start another company eventually?
Having spent some time with Google as SWE, I think Google was by far the best engineering company I have worked with. Even Amazon, Microsoft were terrible when it comes to software engineering.
I am one of those engineers who do not care about culture as long as I am getting paid for the efforts I put in. Google in that sense beat others by HUGE margin.
The engineering work was however very different. We focused on right engineering solutions instead of just business aspect. While that kind of attitude hurts us in short term, it pays big in long term.
> working 60% or 80% were fantastic for my lifestyle and building relationships outside of work
Whoah, it seems fantastic! That alone seems like a good reason to work for Google. Unfortunately, none of the companies I worked for was interested in less than 100%. I told them many times, you can keep your money, I just want to spend 20% or 30% less time at work, but they always insisted on 100%. I have a feeling they would go for 120% if legally allowed.
I'd love to know how in a fast paced office environment you can improve those, none of the trainings or anything are about this (even the leadership like ones feel like just standardized template stuff rather than actually have an environment where you can practice social skills and get the correct timely feedback to improve it)
If you don't enjoy waking up at 4am, work on mobile. Once a new app release is on the phone, nothing is going to make it suddenly break and there is no hurry to send the new release to all users advance, go through internal employee testing and then roll out slowly so that any breakage doesn't affect a lot of users.
Perhaps a condescending take but I think the author got a bit of a big head from getting promoted quickly, and the subtext is that it was due to their amazing technical competence. It’s a noteworthy feat to get recruited out of school, but SRE is a godawful position with high attrition, so it’s easier than SWE to get promoted. That they regret not moving to SWE sooner ignores that SRE is a talent sink and considered a separate ladder by most companies. At this point, the ship has sailed. Eat humble pie, embrace your skillset, and move onward and upward
[+] [-] danpalmer|1 year ago|reply
Engineering from >10 years ago seems like it was a wild west. Some truly stunning pieces of technology, strung together with duct tape. Everything had its own configuration language, workflow engine, monitoring solution, etc. Deployments were infrequent, unreliable, and sometimes even done from a dev's machine. I don't want to disparage engineers who worked there during that time, the systems were amazing, but everything around the edge seemed pretty disparate, and I suspect gave rise to the "promo project" style development meme.
Nowadays we've got Boq/Pod, the P2020 suite, Rollouts, the automated job sizing technologies, even BCID. None of these are perfect by any means, but the convergence is a really good thing. I switched team, PA, and discipline 6 months ago, and it was dead easy to get up and running because I already knew the majority of the tech, and most of that tech is pretty mature and capable.
Maybe Google has become more like other tech companies (although I doubt they have this level of convergence), but I think people glorify the old days at Google and miss that a lot of bad engineering was done. Just one example, but I suspect Google has some of the best internal security of any software company on the planet, and that's a very good thing, but it most certainly didn't have that back in the day.
[+] [-] mike_hearn|1 year ago|reply
> Deployments were infrequent, unreliable, and sometimes even done from a dev's machine.
Deployments were weekly and done from a dev machine because that way someone was watching it and could intervene in case of unexpected problems. Some teams didn't do that and tried to automate rollouts completely. I could always tell which products weren't doing enough manual work because I'd encounter obviously broken features live in production, do a bit of digging and discover end user complaints had been piling up in the support forums for months. But nobody was reading them, and the metrics didn't show any problem, and changes flowed into prod so the team just ... didn't realize their product wasn't working. There's no substitute for trying stuff out for yourself. I encounter clearly broken software that never seems to get fixed way too often these days and I'm sure it's partly because the teams in question don't use their own product much and don't even realize anything is wrong.
[+] [-] skybrian|1 year ago|reply
On the other hand, someone who started in 2015 missed out on the years when Google was mostly considered to be the good guys and given the benefit of the doubt. That’s about when the culture started turning against “big tech” in general (rather than specific companies like Microsoft).
[+] [-] throwaway20222|1 year ago|reply
I look at it as somewhat inevitable considering the path the company has taken, but it is certainly different. We are cranking out money and that’s fine, but it is a change.
[+] [-] whiterknight|1 year ago|reply
That's not to disparage the latter. making things work at scale is hard engineering. But when people praise the glory days, it may be a preference for working on new ideas in small projects.
[+] [-] Rebelgecko|1 year ago|reply
[+] [-] tarsinge|1 year ago|reply
[+] [-] miki123211|1 year ago|reply
Could you elaborate on what these are for a non-googler? Ironically enough, Google isn't very helpful.
[+] [-] contrarian1234|1 year ago|reply
history's biggest yakshave...? not surprising for a programmer-run company
I'm sure it still matters to bring down the power and server bills. But one can't help feel like they could be doing much more
[+] [-] hiddencost|1 year ago|reply
[+] [-] aorloff|1 year ago|reply
[+] [-] rr808|1 year ago|reply
It was a real mistake, SRE is hugely stressful and really unrewarding compared to SWE. Yes you learn some skills and get some occasional glory, but year after year of fighting fires really didn't build any long lasting career.
After switching back to SWE I've finally got promotions and pay rises again, as well as good night sleep and much less stress.
[+] [-] adin8mon|1 year ago|reply
[+] [-] m3drano|1 year ago|reply
I cannot disagree more: our team is healthy, oncall is quite a fine activity to do (and compensated, of course), we have plenty of engineering work to do.
I've had five promotions (and tripled salary) and done so working on plenty rewarding activities over time. I've done from deployment automation, to capacity planning, distributed system design, large data migrations, designing ietf standards for auth protocols, wrote client sdks, now we even do AI for different things (including model development).
I'd recommend to not generalize from "I didn't like it / the experience wasn't a match for me" to "the role is shitty".
[+] [-] kimbernator|1 year ago|reply
I miss working in software that also utilizes my skills in infrastructure, but I do not miss the constant escalations, terrible on-call schedules, and only about 20% of my time being spent on the rewarding parts of the job.
[+] [-] orev|1 year ago|reply
I think in general that type of experience would help Developers be more empathetic to the operations side of things. Those fires often come from trade-offs made in development.
[+] [-] anon990|1 year ago|reply
[+] [-] advisory5739f2|1 year ago|reply
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] aaomidi|1 year ago|reply
[+] [-] p4bl0|1 year ago|reply
I ask this question here because there seem to be quite some (ex-)Google employees in this thread.
[+] [-] makerofthings|1 year ago|reply
[+] [-] googler_thrwy|1 year ago|reply
[+] [-] Balinares|1 year ago|reply
[+] [-] atleastoptimal|1 year ago|reply
[+] [-] enneff|1 year ago|reply
If I take another software gig it will certainly be at a small company where my daily work contributes directly to the company’s central goals.
[+] [-] mike_hearn|1 year ago|reply
What you're seeing from Google in the last ten years is a maybe predictable consequence of that culture, where some Googlers really do seem to think they're generically much smarter than everyone else, about everything. You started to see mass scale social engineering via manipulation of search results and products, driven apparently by the immense faith they have in their own wisdom. Is there any claim Googlers cannot immediately resolve as true or false given nothing more than a few ML models and a team of contractors in LatAm? Apparently some of them think that's all it takes.
This quasi-misanthropic culture is miles away from the trusting "make it universally available and useful" culture the company once had, but the seeds of that culture's end were clearly visible even at the start. You can't constantly validate people by telling them they're super smart before some of them come to actually believe it, and that leads naturally to the belief that if they're really the smartest people in the world then surely that means they should be running it.
[+] [-] tombert|1 year ago|reply
[+] [-] deisteve|1 year ago|reply
its the proximity, pedigree, profile that you have to fit to get in to Google
I'm happy for those that made it. Not everybody gets to work for Google. But the work they do are no less challenging or more important than what the rest of us do.
If anything FAANG has contributed greatly to the American Firewall of Algorithms and have destroyed an entire generation's ability to reason and value common sense.
I remember this quote which I can't remember who said but "if they are paying you a large salary, what they take from you is far greater"
[+] [-] dave333|1 year ago|reply
[+] [-] I_AM_A_SMURF|1 year ago|reply
[+] [-] AlotOfReading|1 year ago|reply
[+] [-] commandersaki|1 year ago|reply
[+] [-] incognito124|1 year ago|reply
[+] [-] lysace|1 year ago|reply
It's like the fourth time I read/hear this. I understand that it's a tricky one to adress.
[+] [-] foobarbaz569|1 year ago|reply
[+] [-] st3fan|1 year ago|reply
[+] [-] markus_zhang|1 year ago|reply
[+] [-] KorematsuFredt|1 year ago|reply
I am one of those engineers who do not care about culture as long as I am getting paid for the efforts I put in. Google in that sense beat others by HUGE margin.
The engineering work was however very different. We focused on right engineering solutions instead of just business aspect. While that kind of attitude hurts us in short term, it pays big in long term.
[+] [-] dvfjsdhgfv|1 year ago|reply
Whoah, it seems fantastic! That alone seems like a good reason to work for Google. Unfortunately, none of the companies I worked for was interested in less than 100%. I told them many times, you can keep your money, I just want to spend 20% or 30% less time at work, but they always insisted on 100%. I have a feeling they would go for 120% if legally allowed.
[+] [-] Aardwolf|1 year ago|reply
I'd love to know how in a fast paced office environment you can improve those, none of the trainings or anything are about this (even the leadership like ones feel like just standardized template stuff rather than actually have an environment where you can practice social skills and get the correct timely feedback to improve it)
[+] [-] cat_plus_plus|1 year ago|reply
[+] [-] keepamovin|1 year ago|reply
[+] [-] luckyone15|1 year ago|reply
[+] [-] georgeburdell|1 year ago|reply
[+] [-] bananapub|1 year ago|reply
[+] [-] lexlash|1 year ago|reply
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] dr_dshiv|1 year ago|reply
How much, do you think?