2005: your infrastructure is automated using a handful of Bash, Perl and Python scripts written by two system administrators. They are custom, sometimes brittle and get rewritten every 5 years.
2022: your infrastructure is automated using 10 extremely complex devops tools. You automated the two system administrators away - but then had to hire 5 DevOps engineers paid 2x more. The total complexity is 10x.
They wrote YAML, TOML, plus Ansible, Pulumi, Terraform scripts. They are custom, sometimes brittle and get rewritten every 3 years.
EDIT: to the people claiming that today's infra does more things... No, I'm comparing stuff with the same levels of availability, same deployment times, same security updates.
> If you look at a DevOps engineer job description, it looks remarkably similar to a System Administrator role from 2013, but...
> If DevOps was supposed to be about changing the overall culture, it can’t be seen as a successful movement. People on the operations side of the fence will...
As someone who was keenly watching this stuff back 15 years ago, parts of this article connect with my understanding, but the core problem I have is that this article itself is somehow bought into the mistake that led to the failure and so almost can't see the failure for what it is: the entire point of DevOps was that "operations" isn't a job or role anymore and has instead become a task that should be done by the developers.
Ergo, if you even still have operations people to comment on it--or certainly if you are somehow hiring dedicated "DevOps" people--you aren't doing DevOps and have already failed. The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps; you simply can't have it both ways, as that's just renaming the same two camps instead of merging them into a single unified group.
I've worked in ops roles since about 2000 after a few years in backend corp IT stuff.
I agree that what you've described was the original intent and goal of "devops" but in light of that failure, the "cross functional team" definition took over and then in light of that failure, the SRE was born and we're basically back where we started but now the ops people use git instead of rcs.
In my experience and opinion, developers are really bad at ops and sysadmins/ops are really bad a development. Anyone that is truly good at both is a unicorn that is probably carrying their team.
> the entire point of DevOps was that "operations" shouldn't exist, and that operations is a task that should be done by the developers.
Ergo, if you even have operations people to comment on it, or if you are hiring dedicated "DevOps" people, you aren't doing DevOps and have already failed.
This. My first thought when I was reading the article. Spot on
> the entire point of DevOps was that "operations" isn't a job or role anymore and has instead become a task that should be done by the developers.
This akin to saying "frontent developer isn't a role anymore - both frontend and backend should be handled by a full-stack developer". This works for small companies/projects, but bigger ones can benefit from specialization and division of labor. Body of knowledge required to be a decent software developer and a decent ops engineer is too big to fit into one head. I've seen ops work being done by developers without ops experience and more often than not it was ugly - they didn't had enough experience/knowledge (and time/incentives to gain them) to do ops work well.
To me the best part of DevOps isn't about roles but about team structure - splitting all Dev in one department and all Ops into another usually is a bad idea. And a failure of this split was a motivation to start DevOps movement. Having Ops embedded into Dev teams in my experience works much better.
> The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps
That's like saying "agile is firing your scrum masters and tell your developers you're agile now".
The idea behind DevOps is that applications are provisioned by the people who know the application best, the developers. If everything works out as it should, this gives additional load with creating the deployments, but also removes the overhead when dealing with operations when deploying, updating and debugging - so a net zero in workload, but a gain in the way the application is hosted better and fixing bugs is easier. You still need operations, both for providing the underlying platform (getting a server ready is not a developers core business and it shouldn't be) and for guiding the developers. It should be leaner, but you still need it.
Of course, you can also fire all of infra and tell the developers "that's your job now", but that's like calling biweekly deadlines scrum (and leads to equally bad outcomes).
This can be true, but I would argue not always. Some DevOps teams work in the old mode of “throwing code over to Ops to run” - this isn’t what DevOps intended, but happens.
When they work well, they’re doing things like authoring reusable (by product eng. teams) infrastructure modules, or helping to build “you build it, you run it” tooling like monitoring stacks etc. They’re also helpfully/hopefully subject matter experts on CI/CD, your cloud/hosting of choice, security stuff - things that general developers have mixed levels of interest or competence in.
That is utter BS. DevOps means Ops and Devs working hand in hand in a crossfunctional team. Nothing more, nothing less. The main idea was to tear down silos.
At all the places I worked previously in the last ~20 years there was always a sharp separation of development/testing and production environments. I as a developer never had access to any system in production, apart from one place which had a very sophisticated security system in place which could grant you temporary access during deployments. Just think about customer data, and you'll understand why.
So when I hear that someone thinks devops is developers running their own systems in production I always wonder where this is actually possible, let alone whether it is a good idea at all.
> The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps
That's a way to say you're doing DevOps, but it's not going to work very well.
> merging them into a single unified group
That's the right way to describe it. Just like the old "programmers build it to spec, then throw it over the wall to QA" is out of style, and now good teams have testing specialists in the same room (conceptually) as developers. the goal with DevOps was to stop doing the old "here's a build, deploy it" that caused so much wailing and gnashing of teeth and instead bring ops skills into the team as a first-class expertise. The CI/CD pipeline can mean that development, testing, and release are all together, rapidly iterating and responding to change.
By the way, pretty soon the AppSec/CyberSec people will be folded in, to, so instead of the old "it's done/deployed, run your pentests/analysis tools" secure by design will require those skills to be integrated, too.
> our role is to enable and facilitate developers in getting features into the hands of customers
The problem here is this creates the wrong kind of incentives for developers… somehow elevating the to a level where they don’t have to care about how their code works in production.
As someone that remembers being a developer back in the days of sysadmins, we were AFRAID of upsetting the operations people. If your code brought a server down, you were at least going to face some very awkward conversations. The cartoon “The Bastard Operator from Hell” immortalized that era.
Meanwhile at one company I worked at years ago - an airline - the development team was responsible for keeping the system running 24/7. Nothing makes you think more carefully about your code in production than meeting a colleague on Monday morning who got woken up at 2am by your code failing.
While I’m not arguing for hostility in the workplace, giving developers incentives to care about their code in production seems to me to be one of the things devops got wrong
There's a reason people end up doing plain ops and calling it devops: it's often too costly to handle this as “a task that should be done by the developers”.
Specialization makes people much more productive, because they face the same kind of issues over and over and know how to fix them quickly. When you distribute the load in your organization, everybody is going to face problems, struggle, learn and never reuse that knowledge again.
I was just reading the "Building Scalable Websites" book [1] released in 2006. At that time, "DevOps" was called SysAdmins. And there were also DBAs, Network engineers, among others.
> The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps; you simply can't have it both ways,
I think this points at what happened: Startup scrappy culture started permeating new technology companies, which meant no budget for DBAs, QAs, SysAdmins and other similar roles. So decision-makers fired all those roles and ask Programmers to fill the voids. At the same time "cloud computing" started to mature, so there was a change from hardware/operating-systems tinkering to software related tinkering.
One just has to see the decline of "SlashDot" which was a very SysAdmin/Operating-System focused website, in favor of news.ycombinator and similar more software-oriented forums.
I just quit a job partly because we lost our key DevOps guy and no serious effort was made to replace them. As a result I ended up wasting huge amounts of my time dealing with operations-level stuff that made it impossible to focus on the key parts of my role (feature development etc.). I subsequently turned down a job offer from elsewhere that explained their policy was not to have dedicated DevOps resources for their SaaS platform (devs themselves being responsible for all deployment and system maintenance), and would do so again. Good DevOps people are worth their weight in gold, and at least in many verticals (e.g. those involving payments ) it's virtually mandated that there is a separation of responsibilities between those writing the code and those responsible for delivering the product to customers. I can't see the need for dedicated DevOps resources going away any time soon.
The reason that you need to do things the ops way is because ops knows how to run applications in production. There's a reason the meme "worked in dev, ops problem now" exists. You need to meet all of the requirements of an app that's running in production from a technical, availability, security, and policy point-of-view. It's not easy and that's why this will never work.
Software is hard, it's just that a lot of developers used to cut their code, run it on their laptop, and let someone else worry about it. It's different these days (although not as much as I'd like).
We don't make you use these tools because we want to, we use these tools because we're required too. No one cared about ISO27001, SOC2, or PCIDSS compliance for your crappy PHP app you ran on your cpanel. They didn't care back then you were using md5 hashes to "secure" passwords. The world is fundamentally different to what it used to be 10-15 years ago, and the requirements from business are astronomically different.
Edit: and to people saying "oh you could just run it on a single server", no you can't because certifications like ISO27001 require certain levels of availability and DR. You're not going to be able to guarantee that with a single server running in a rack somewhere.
I'm assuming you mean your comment, not the post itself.
> The reason that you need to do things the ops way is because ops knows how to run applications in production
Stability in production is one metric. Ops overindexing on this metric is exactly what causes the friction with developers.
Developers are trying to ship value to customers.
Uptime is only one part of that equation and for most businesses, it's not even a very important one.
The author points this out near the end. DevOps can't convince devs to use ops techniques if all the reasons for using those techniques are based on the flawed assumption that development velocity isn't important.
Certifications are a very good point because afaik ISO 27001 is now far more achievable for far more companies of smaller sizes with not that many IT staff. Sometimes even 3 good engineers can set up everything needed to pass ISO in a small company in like half a year or something.
Meh…DevOps is just System Administration, and Systems Administration is just Sys Ops. They keep changing the title/role but the work remains largely the same. I think is a bit disingenuous to throw “dev” in title, as a “DevOps Engineer” myself I don’t consider anything I ever do “dev”. Ansible is not “dev”, terraform is not “dev”, ci/cd pipelines are not “dev”, helm charts aren’t “dev”. But for some reason companies seem to love the term.
> But for some reason companies seem to love the term.
It's possible that I'm just getting very pessimistic, but at this point I'm fairly confident that companies love it because it makes it way easier to attract candidates and describe one set of responsibilities/position in an interview process, and then bait-and-switch it into what is effectively a systems administrator role.
Depends on what you think Developer Operations should be. Our developers instantiate their buckets, databases, cache instances etc. themselves, deploy microservices themselves and update configuration, traffic management and scaling parameters themselves. No 'system' people required. The system people are mostly just keeping the automation running and add features as needed.
The work also really isn't the same. Unless you're stuck in the 90's we aren't building servers, installing operating systems, installing applications and installing patches anymore.
I agree with most of what you say, and in particular with companies' love for the term, but I disagree that "the work remains largely the same". When I got started on this line of work, we used cvs to track program code and we used backups to 'track' infrastructure, including code used to manage the infra (mostly shell scripts, though not only that).
There's a long path from that to ansible and terraform on SCM.
Another big difference I have experienced: we used to literally celebrate server uptime (I mean as a celebration, I have a distinct memory of gathering around an IBM "fridge" to celebrate the uptime birthday of a particular RS/6000) while now a piece of infra with too much uptime is a red flag about potential vulnerabilities.
What does largely remain the same, I think, are the skills needed to be good at this. Then, and now, we need people who don't mind reading manuals, searching online (this was already a thing when I started, I guess you'd have to go back to the mid 90s for this to not be the case?) , who can keep track of where they've been during a debugging/troubleshooting session, that sort of thing.
Another thing that changed is that in the past some people considered it a badge of honor to be assholes to others not in sysadmin, even more to others not in IT (remember those "select * from users where clue > 0" t-shirts, or the BOFH stories?), while now that's typically frowned upon and quite a few companies are explicit about a no assholes policy in their hiring material (or perhaps I've just been lucky with my teammates and smarter when picking where to work at than when I was younger).
I think the main problem is that the "DevOps" role significantly differs company to company. A company that desperately needs a solid administrator might not be able to attract the right talent and as a result, end up classifying the open positions as "DevOps engineer". At the same time, there are companies out there legitimately trying to bridge the divide between the two — software and system administrators — job families.
From my understanding, DevOps was never about technical solutions/processes. It was about giving the same business goals to Dev and Ops. The idea being to eliminate the tension netween these departments because the goal of Dev was to ship features and the goal of Ops was to ensure the system stayed up.
I think devops means something here, sysops would be about running infrastructure not dev infrastructure, devops would focus on producing dev envs, test envs, CI/CD. Not just setting up the runtime hardware / os configuration.
Meh... Dev is just clicking on whatever your IDE autocompletes for you. With copilot you do even less. I think some programmers out there have some big heads that need popping.
Building out and automating cloud infrastructure so your simple code can work is way more complex than most things you do every day. But ya, keep telling yourself how smart you are as your write "connect to database, return a value" for the 1000th time.
> the anecdotal evidence I’ve gathered has been that the conference are heavily attended by operations, and less attended by developers.
Most Ops people - fuck, even most actual DevOps Engineers - have no clue what the fuck DevOps is. They (rightfully) assume it's just a trendy new word for the same old Ops bull, but in the cloud and with Terraform.
DevOps failed because it never educated anybody except a very small handful of people who actively were looking to solve big organizational problems. It was too many things to too few people. It could succeed only if you brought everybody in the entire org through three different training courses. And that's because DevOps tries to make Operations uplift literally the entire technology organization.
DevOps is dead. The ideas are great, but we need to bring the ideas to people outside Ops. Until then it will just be a slightly-more-technologically-advanced Ops.
(disclaimer: I am a DevOps Engineer that hates the fact that this is my title)
My take is different. I think DevOps was wildly successful, most of our infrastructure is now software that can be managed by Software Engineers. The goal posts have shifted, we now have major software challenges where as before we had hardware and operational challenges.
Well written tools and cross-functional teams that do both operations, feature work and security are still the path forward IMO, we just need to refocus on developer experience.
We tried to hire senior devs to do DevOps work, but the ones that can pass the interview and have already been at a DevOps shop are too smart to be fooled a second time.
We still use all the DevOps buzzword stacks, but we stopped doing dev ops. Instead, we are building out a really good ops team. This makes it possible to hire developers again.
Personally, I'm one of the better ops people on the developer side of the fence, but I'd need at least 2x typical principle engineer comp to take another job at a DevOps shop, and also wouldn't get even a quarter of my normal productivity.
At that point, you may as well just hire a junior undergrad graduate and burn the rest of your cash.
As a software engineer I don’t want to touch your infrastructure code. I have been lucky so far and I have been doing pure product development instead of being a devops. I do believe though in the idea of cross functional teams: designer, developer, infra engineers, managers.
And now 50% time software engineers are writing infrastructure. Don’t know what is solution but cloud-native landscape has increased cognitive overload.
+1 tiny teams run infrastructure that previously was responsibility of whole departments at bigco and things mostly work well. Then as soon as they face some minor, totally solvable issues everyone loses their minds.
I disagree, i move in different circles from the author. In my little world DevOps is often more dev-heavy (and needs to mature it’s ops credentials), SRE has become the new-age Ops.
DevOps is primarily tackling a people problem, there’s certainly plenty of useful tools to help but at its core, it’s about people.
Encouraging people to work in frequent small increments (think XP) rather than quarterly releases. Getting rid of the change management beauraucracy, encouraging developers to consider security as they’re designing and writing software. DevOps is a broad church, it’s effective too.
People like to over-focus on the tools of DevOps but I’d take today’s world of version controlled (vs. “Ohh Brad has the config GUI code on his desktop, speak to him about making a change“), automated CI pipeline (vs. builds at the end of the quarter that require stop-the-world help and assistance from all developers, also RIP merge guy), automated deployment into many environments (vs. Excel list of tasks for QA team and different excel list of tasks for prod ops). I’ll take SOA over the-DB-is-the-network 00’s enterprise software architecture, oh and with automated DB deployment (albeit still without a real solution to automating stateful db rollback even today). These are all DevOps factors, from architecture to testing, from security to working and growing together as a team.
The tool-only view of DevOps is just yet another excuse to ignore the hard problem: helping large numbers of people work productively together.
Devops is a failure only when you have the wrong people on the job. Which is nearly all the time because only 1 in 20 engineers has the ability to do both operational and development work to a standard high enough not to put your organisation at risk.
If you have several of those 1 in 20 people they will self organise into a methodology which roughly resembles it.
The problem with Docker and friends, from my perspective as a developer, is that it's a whole other skill-set you have to learn, keep up with, and context-switch into and out of. I think that's why it usually ends up being its own job: maybe those technologies bring some order to a huge messy problem-space, but they don't make the problem-space simple. The abstraction is way, way too leaky to use it without knowing and thinking about all the ins and outs of everything that's going on underneath. And you can't even just use the existing knowledge you might have of scripting and systems: you have to learn whole new technologies on top of those. And as a dev, who's already thinking about all the ins and outs of my own whole complex system, that just kneecaps my ability to stay in a headspace where I can get things done.
If you want me to do my own ops, they need to have a Heroku level of simplicity. That's what's required for them to not detract from my other work, and not drive me to burnout. Anything less, and I'm going to just keep tossing things over the fence.
I think a lot of good ideas in tech get communicated to people who want a cargo cult and don't understand the underlying good ideas until agile becomes a flavor of waterfall, test driven development becomes development with tests, and devops becomes just another wall between developers and their end product.
The actual job title was meant to be “Agile Systems Administrator”.
People got the conference confused with the “10 deploys a day” talk from Flickr.
DevOps is a failure because it’s a meaningless term that changes depending on the bearers own understanding: the issue is that everyone is right. It’s a nebulous idea.
I wrote, rather frustratedly, about this. I even did the research and learned something new myself at the time.
Trying to do DevOps without any [sysadmins, network engineers, DBAs, security specialists] is like trying to write an accounting system without any CPAs, or manage a chemical processing plant without any chemical engineers. Reading the books as a substitute for experts with experience only works a little way.
The point of DevOps was to take that experience and multiply its effectiveness through the tools of modern software development.
It's interesting to read the proposed "SoftOps" approach - it's something I've been trying to do in my own little bubble, working on making the lives of developers easier (especially when getting their code from repo to prod), with the developers creating systems. I volunteer for a convention run entirely within VRChat, with a handful of supporting services (API for tracking players, instances and registration, frontends for admin and attendees, and so on), and while the department is named "DevOps" (too late to change it, and everyone knows what we mean), I landed on the infrastructure subteam. While part of the job was building out our new Kubernetes cluster (mostly for scaling. I'm still working on the blog post about this sort of stuff), I wanted to make sure that the developers could be completely abstracted away from _where_ their code was running, and as much as possible _how_. Of course, they're free to mess with whatever tooling I setup for them (mostly Docker images, GitHub Actions, etc), but I've found it very fulfilling to "exist to serve" (because infrastructure? servers?) the developers pushing out code. If we need some custom internal tooling to support that development cycle I'm more than happy to get something written and deployed (something for the blog post). It may be barebones but I wouldn't stop a developer from another team contributing (provided they don't have something more pressing).
All this to say... I really like supporting developers in their work, as a sort of meta-engineer.
edit: worth pointing out I do something sort of similar at my current employer, but it lands a bit more on the type of devops that the author describes as having failed, and I'm actively working to see if I can find a position that better suites my skillset!
Just use the cloud, honestly. "No, but there is this weird thing we do, and need custom yada yada", no just stop, conform to the cloud, stop being a snowflake. "Its so expensive!". No, its still cheaper than on prim and paying dedicated teams to manage it. Just use the cloud in a sane way. Teach your devs with the infinite guides and docs freely available online. Your life will be easier. If your devs aren't interested in your infrastructure, they are bad devs.
Critic: Even if Helm and Helm Charts makes it easy to install a set of apps into a Kubernetes cluster, surely some actions are more complicated than a simple install? What about upgrading a PostGres database?
Advocate: Way ahead of you! We worked out that problem a long time ago! You just use a Helm Operator! It’s all really simple!
Critic: Really? And this solves all the problems of upgrading PostGres in a stable and reliable way?
Advocate: Uh, well, it’s supposed to. It’s, uh, all really simple?
Critic: Supposed to?
Advocate: Sure, so long as you have a working Go environment, you just use the Operator SDK to generate the scaffold for your Operator.
Critic: The scaffold?
Advocate: Sure, the scaffold sets up the basics, figures out the permissions, the dependencies, everything needed to install the Helm Chart. Then you can build the Operator container, and install it in your Kubernetes cluster. It’s all really simple!
Critic: This sounds complicated.
Advocate: Way ahead of you! The good folks at RedHat knew people like you were going to whine about stuff, since you obviously like to whine about stuff, so they created the Operator Lifecycle Manager to make all of this a lot easier.
Critic: Doesn’t it seem like we keep piling new technology on top of new technology, to manage the excessive complications of the previous layer of technologies?
Advocate: Hey, think of the alternatives. You don’t want to go back to the bad old days of the past, do you?
Critic: You mean, the bad old days when stuff mostly worked and I didn’t have to learn 3 new alpha technologies each day?
Advocate: That’s a ridiculous exaggeration! Some of these technologies are beta.
Critic: And you seriously regard these piles of code, heaped upon piles of code, as an improvement on the old situation?
Advocate: Are you kidding? It’s like night and day. I’d rather drink arsenic than get dragged back to the bad old days when I had to write Ansible scripts. We live in the future now. We’ve escaped the old world where every attempt at devops became a painful, confusing, unmaintainable disaster after 2 years.
Critic: How long have you been using Docker/Kubernetes in production?
Mechanical engineers had DevOps figured out long ago. They call it DFM - Design For Manufacturing.
It doesn’t mean every designer has his own metal sheet press on his desk. It means he understands his component is used in a bigger picture, should fit into the robot assembling it, use the same type of plastic and standardized screw as all other components, and so on.
Same goes for software, running your own shit sounds romantic and all, but having every developer in your company picking their own monitoring stack or understanding the intricacies of kubernetes networking is really not effective. Some central guidance defining the standard screw head to use is needed here as well.
I agree that DevOps is a failure, but not for the reasons that OP states. Or maybe exactly for the reasons OP states.
People that would traditionally be called Ops were turned into DevOps overnight, and felt like they suddenly had the mandate to try and barge into development, offering true, but ultimately misguided help.
If you haven't asked me what I need yet, don't bring me a platform that is "going to solve all my problems".
I imagine this is why the developers in OP's story do everything themselves. At least they're able to fix something if it breaks, instead of having to rely on the ops team that is off chasing the next shiny thing.
Tired of working on infrastructure stuff while holding the “software engineer” role. Give me product features, bugs, interesting and boring product problems to solve. Keep your yaml, k8s, gitlab ci/cd, terraform, on call rotations. Or better, hire a dedicated infra engineer to handle all that stuff.
Both DevOps and Front-End hysteria are enormous. My conspiracy theory is that FAANG+MS does this to burn down resources and prevent anything interesting created outside of their sphere of influence. In one workplace I was barely keeping my head over the surface in the Angular+NGRX+Observable cesspool then started to talk about corresponding infra and the backend person was all like "let me configure and provision an autoscaling geodistributed self-healing supercluster". KILL. ME.
[+] [-] terpans|3 years ago|reply
2022: your infrastructure is automated using 10 extremely complex devops tools. You automated the two system administrators away - but then had to hire 5 DevOps engineers paid 2x more. The total complexity is 10x.
They wrote YAML, TOML, plus Ansible, Pulumi, Terraform scripts. They are custom, sometimes brittle and get rewritten every 3 years.
EDIT: to the people claiming that today's infra does more things... No, I'm comparing stuff with the same levels of availability, same deployment times, same security updates.
[+] [-] saurik|3 years ago|reply
> If DevOps was supposed to be about changing the overall culture, it can’t be seen as a successful movement. People on the operations side of the fence will...
As someone who was keenly watching this stuff back 15 years ago, parts of this article connect with my understanding, but the core problem I have is that this article itself is somehow bought into the mistake that led to the failure and so almost can't see the failure for what it is: the entire point of DevOps was that "operations" isn't a job or role anymore and has instead become a task that should be done by the developers.
Ergo, if you even still have operations people to comment on it--or certainly if you are somehow hiring dedicated "DevOps" people--you aren't doing DevOps and have already failed. The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps; you simply can't have it both ways, as that's just renaming the same two camps instead of merging them into a single unified group.
[+] [-] uberduper|3 years ago|reply
I agree that what you've described was the original intent and goal of "devops" but in light of that failure, the "cross functional team" definition took over and then in light of that failure, the SRE was born and we're basically back where we started but now the ops people use git instead of rcs.
In my experience and opinion, developers are really bad at ops and sysadmins/ops are really bad a development. Anyone that is truly good at both is a unicorn that is probably carrying their team.
[+] [-] nixlim|3 years ago|reply
This. My first thought when I was reading the article. Spot on
[+] [-] citrin_ru|3 years ago|reply
This akin to saying "frontent developer isn't a role anymore - both frontend and backend should be handled by a full-stack developer". This works for small companies/projects, but bigger ones can benefit from specialization and division of labor. Body of knowledge required to be a decent software developer and a decent ops engineer is too big to fit into one head. I've seen ops work being done by developers without ops experience and more often than not it was ugly - they didn't had enough experience/knowledge (and time/incentives to gain them) to do ops work well.
To me the best part of DevOps isn't about roles but about team structure - splitting all Dev in one department and all Ops into another usually is a bad idea. And a failure of this split was a motivation to start DevOps movement. Having Ops embedded into Dev teams in my experience works much better.
[+] [-] Sebb767|3 years ago|reply
That's like saying "agile is firing your scrum masters and tell your developers you're agile now".
The idea behind DevOps is that applications are provisioned by the people who know the application best, the developers. If everything works out as it should, this gives additional load with creating the deployments, but also removes the overhead when dealing with operations when deploying, updating and debugging - so a net zero in workload, but a gain in the way the application is hosted better and fixing bugs is easier. You still need operations, both for providing the underlying platform (getting a server ready is not a developers core business and it shouldn't be) and for guiding the developers. It should be leaner, but you still need it.
Of course, you can also fire all of infra and tell the developers "that's your job now", but that's like calling biweekly deadlines scrum (and leads to equally bad outcomes).
[+] [-] ebbp|3 years ago|reply
When they work well, they’re doing things like authoring reusable (by product eng. teams) infrastructure modules, or helping to build “you build it, you run it” tooling like monitoring stacks etc. They’re also helpfully/hopefully subject matter experts on CI/CD, your cloud/hosting of choice, security stuff - things that general developers have mixed levels of interest or competence in.
[+] [-] oweiler|3 years ago|reply
[+] [-] haspok|3 years ago|reply
So when I hear that someone thinks devops is developers running their own systems in production I always wonder where this is actually possible, let alone whether it is a good idea at all.
[+] [-] cratermoon|3 years ago|reply
That's a way to say you're doing DevOps, but it's not going to work very well.
> merging them into a single unified group
That's the right way to describe it. Just like the old "programmers build it to spec, then throw it over the wall to QA" is out of style, and now good teams have testing specialists in the same room (conceptually) as developers. the goal with DevOps was to stop doing the old "here's a build, deploy it" that caused so much wailing and gnashing of teeth and instead bring ops skills into the team as a first-class expertise. The CI/CD pipeline can mean that development, testing, and release are all together, rapidly iterating and responding to change.
By the way, pretty soon the AppSec/CyberSec people will be folded in, to, so instead of the old "it's done/deployed, run your pentests/analysis tools" secure by design will require those skills to be integrated, too.
Little by little, chipping away at the waterfall
[+] [-] harryf|3 years ago|reply
> our role is to enable and facilitate developers in getting features into the hands of customers
The problem here is this creates the wrong kind of incentives for developers… somehow elevating the to a level where they don’t have to care about how their code works in production.
As someone that remembers being a developer back in the days of sysadmins, we were AFRAID of upsetting the operations people. If your code brought a server down, you were at least going to face some very awkward conversations. The cartoon “The Bastard Operator from Hell” immortalized that era.
Meanwhile at one company I worked at years ago - an airline - the development team was responsible for keeping the system running 24/7. Nothing makes you think more carefully about your code in production than meeting a colleague on Monday morning who got woken up at 2am by your code failing.
While I’m not arguing for hostility in the workplace, giving developers incentives to care about their code in production seems to me to be one of the things devops got wrong
[+] [-] littlestymaar|3 years ago|reply
Specialization makes people much more productive, because they face the same kind of issues over and over and know how to fix them quickly. When you distribute the load in your organization, everybody is going to face problems, struggle, learn and never reuse that knowledge again.
[+] [-] xtracto|3 years ago|reply
> The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps; you simply can't have it both ways,
I think this points at what happened: Startup scrappy culture started permeating new technology companies, which meant no budget for DBAs, QAs, SysAdmins and other similar roles. So decision-makers fired all those roles and ask Programmers to fill the voids. At the same time "cloud computing" started to mature, so there was a change from hardware/operating-systems tinkering to software related tinkering.
One just has to see the decline of "SlashDot" which was a very SysAdmin/Operating-System focused website, in favor of news.ycombinator and similar more software-oriented forums.
[1] https://www.oreilly.com/library/view/building-scalable-web/0...
[+] [-] wizofaus|3 years ago|reply
[+] [-] StopHammoTime|3 years ago|reply
The reason that you need to do things the ops way is because ops knows how to run applications in production. There's a reason the meme "worked in dev, ops problem now" exists. You need to meet all of the requirements of an app that's running in production from a technical, availability, security, and policy point-of-view. It's not easy and that's why this will never work.
Software is hard, it's just that a lot of developers used to cut their code, run it on their laptop, and let someone else worry about it. It's different these days (although not as much as I'd like).
We don't make you use these tools because we want to, we use these tools because we're required too. No one cared about ISO27001, SOC2, or PCIDSS compliance for your crappy PHP app you ran on your cpanel. They didn't care back then you were using md5 hashes to "secure" passwords. The world is fundamentally different to what it used to be 10-15 years ago, and the requirements from business are astronomically different.
Edit: and to people saying "oh you could just run it on a single server", no you can't because certifications like ISO27001 require certain levels of availability and DR. You're not going to be able to guarantee that with a single server running in a rack somewhere.
[+] [-] dhzhzjsbevs|3 years ago|reply
I'm assuming you mean your comment, not the post itself.
> The reason that you need to do things the ops way is because ops knows how to run applications in production
Stability in production is one metric. Ops overindexing on this metric is exactly what causes the friction with developers.
Developers are trying to ship value to customers.
Uptime is only one part of that equation and for most businesses, it's not even a very important one.
The author points this out near the end. DevOps can't convince devs to use ops techniques if all the reasons for using those techniques are based on the flawed assumption that development velocity isn't important.
[+] [-] jen20|3 years ago|reply
The way most “enterprises” deploy distributed systems, I’d be surprised if a single server didn’t typically result in better uptime to be honest.
[+] [-] skjoldr|3 years ago|reply
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] hitpointdrew|3 years ago|reply
[+] [-] nrmitchi|3 years ago|reply
It's possible that I'm just getting very pessimistic, but at this point I'm fairly confident that companies love it because it makes it way easier to attract candidates and describe one set of responsibilities/position in an interview process, and then bait-and-switch it into what is effectively a systems administrator role.
[+] [-] oneplane|3 years ago|reply
The work also really isn't the same. Unless you're stuck in the 90's we aren't building servers, installing operating systems, installing applications and installing patches anymore.
[+] [-] fipar|3 years ago|reply
There's a long path from that to ansible and terraform on SCM.
Another big difference I have experienced: we used to literally celebrate server uptime (I mean as a celebration, I have a distinct memory of gathering around an IBM "fridge" to celebrate the uptime birthday of a particular RS/6000) while now a piece of infra with too much uptime is a red flag about potential vulnerabilities.
What does largely remain the same, I think, are the skills needed to be good at this. Then, and now, we need people who don't mind reading manuals, searching online (this was already a thing when I started, I guess you'd have to go back to the mid 90s for this to not be the case?) , who can keep track of where they've been during a debugging/troubleshooting session, that sort of thing.
Another thing that changed is that in the past some people considered it a badge of honor to be assholes to others not in sysadmin, even more to others not in IT (remember those "select * from users where clue > 0" t-shirts, or the BOFH stories?), while now that's typically frowned upon and quite a few companies are explicit about a no assholes policy in their hiring material (or perhaps I've just been lucky with my teammates and smarter when picking where to work at than when I was younger).
[+] [-] itsmemattchung|3 years ago|reply
[+] [-] damagednoob|3 years ago|reply
[+] [-] agumonkey|3 years ago|reply
[+] [-] dpz|3 years ago|reply
[+] [-] dilyevsky|3 years ago|reply
[+] [-] downrightmike|3 years ago|reply
[+] [-] user00012-ab|3 years ago|reply
Building out and automating cloud infrastructure so your simple code can work is way more complex than most things you do every day. But ya, keep telling yourself how smart you are as your write "connect to database, return a value" for the 1000th time.
[+] [-] throwaway787544|3 years ago|reply
> the anecdotal evidence I’ve gathered has been that the conference are heavily attended by operations, and less attended by developers.
Most Ops people - fuck, even most actual DevOps Engineers - have no clue what the fuck DevOps is. They (rightfully) assume it's just a trendy new word for the same old Ops bull, but in the cloud and with Terraform.
DevOps failed because it never educated anybody except a very small handful of people who actively were looking to solve big organizational problems. It was too many things to too few people. It could succeed only if you brought everybody in the entire org through three different training courses. And that's because DevOps tries to make Operations uplift literally the entire technology organization.
DevOps is dead. The ideas are great, but we need to bring the ideas to people outside Ops. Until then it will just be a slightly-more-technologically-advanced Ops.
(disclaimer: I am a DevOps Engineer that hates the fact that this is my title)
[+] [-] lox|3 years ago|reply
Well written tools and cross-functional teams that do both operations, feature work and security are still the path forward IMO, we just need to refocus on developer experience.
[+] [-] hedora|3 years ago|reply
We still use all the DevOps buzzword stacks, but we stopped doing dev ops. Instead, we are building out a really good ops team. This makes it possible to hire developers again.
Personally, I'm one of the better ops people on the developer side of the fence, but I'd need at least 2x typical principle engineer comp to take another job at a DevOps shop, and also wouldn't get even a quarter of my normal productivity.
At that point, you may as well just hire a junior undergrad graduate and burn the rest of your cash.
[+] [-] tkiolp4|3 years ago|reply
[+] [-] abhishektwr|3 years ago|reply
[+] [-] dilyevsky|3 years ago|reply
[+] [-] ketzo|3 years ago|reply
I’ve never touched a computer running my company’s production software, and I never will.
20 years ago, that would have been literally inconceivable with someone who was called a “Software Engineer.”
[+] [-] CraigJPerry|3 years ago|reply
DevOps is primarily tackling a people problem, there’s certainly plenty of useful tools to help but at its core, it’s about people.
Encouraging people to work in frequent small increments (think XP) rather than quarterly releases. Getting rid of the change management beauraucracy, encouraging developers to consider security as they’re designing and writing software. DevOps is a broad church, it’s effective too.
People like to over-focus on the tools of DevOps but I’d take today’s world of version controlled (vs. “Ohh Brad has the config GUI code on his desktop, speak to him about making a change“), automated CI pipeline (vs. builds at the end of the quarter that require stop-the-world help and assistance from all developers, also RIP merge guy), automated deployment into many environments (vs. Excel list of tasks for QA team and different excel list of tasks for prod ops). I’ll take SOA over the-DB-is-the-network 00’s enterprise software architecture, oh and with automated DB deployment (albeit still without a real solution to automating stateful db rollback even today). These are all DevOps factors, from architecture to testing, from security to working and growing together as a team.
The tool-only view of DevOps is just yet another excuse to ignore the hard problem: helping large numbers of people work productively together.
[+] [-] iasay|3 years ago|reply
If you have several of those 1 in 20 people they will self organise into a methodology which roughly resembles it.
[+] [-] brundolf|3 years ago|reply
If you want me to do my own ops, they need to have a Heroku level of simplicity. That's what's required for them to not detract from my other work, and not drive me to burnout. Anything less, and I'm going to just keep tossing things over the fence.
[+] [-] c3534l|3 years ago|reply
[+] [-] dijit|3 years ago|reply
The actual job title was meant to be “Agile Systems Administrator”.
People got the conference confused with the “10 deploys a day” talk from Flickr.
DevOps is a failure because it’s a meaningless term that changes depending on the bearers own understanding: the issue is that everyone is right. It’s a nebulous idea.
I wrote, rather frustratedly, about this. I even did the research and learned something new myself at the time.
http://blog.dijit.sh/devops-confusion-and-frustration
Someone on hackernews responded to me once that: “devops is how sysadmins get respect for their role” and that resonated with me.
Sysadmins today are what helpdesk was 15 years ago now though.
[+] [-] dsr_|3 years ago|reply
The point of DevOps was to take that experience and multiply its effectiveness through the tools of modern software development.
[+] [-] gmemstr|3 years ago|reply
All this to say... I really like supporting developers in their work, as a sort of meta-engineer.
edit: worth pointing out I do something sort of similar at my current employer, but it lands a bit more on the type of devops that the author describes as having failed, and I'm actively working to see if I can find a position that better suites my skillset!
[+] [-] strangescript|3 years ago|reply
[+] [-] lkrubner|3 years ago|reply
Critic: Even if Helm and Helm Charts makes it easy to install a set of apps into a Kubernetes cluster, surely some actions are more complicated than a simple install? What about upgrading a PostGres database?
Advocate: Way ahead of you! We worked out that problem a long time ago! You just use a Helm Operator! It’s all really simple!
Critic: Really? And this solves all the problems of upgrading PostGres in a stable and reliable way?
Advocate: Uh, well, it’s supposed to. It’s, uh, all really simple?
Critic: Supposed to?
Advocate: Sure, so long as you have a working Go environment, you just use the Operator SDK to generate the scaffold for your Operator.
Critic: The scaffold?
Advocate: Sure, the scaffold sets up the basics, figures out the permissions, the dependencies, everything needed to install the Helm Chart. Then you can build the Operator container, and install it in your Kubernetes cluster. It’s all really simple!
Critic: This sounds complicated.
Advocate: Way ahead of you! The good folks at RedHat knew people like you were going to whine about stuff, since you obviously like to whine about stuff, so they created the Operator Lifecycle Manager to make all of this a lot easier.
Critic: Doesn’t it seem like we keep piling new technology on top of new technology, to manage the excessive complications of the previous layer of technologies?
Advocate: Hey, think of the alternatives. You don’t want to go back to the bad old days of the past, do you?
Critic: You mean, the bad old days when stuff mostly worked and I didn’t have to learn 3 new alpha technologies each day?
Advocate: That’s a ridiculous exaggeration! Some of these technologies are beta.
Critic: And you seriously regard these piles of code, heaped upon piles of code, as an improvement on the old situation?
Advocate: Are you kidding? It’s like night and day. I’d rather drink arsenic than get dragged back to the bad old days when I had to write Ansible scripts. We live in the future now. We’ve escaped the old world where every attempt at devops became a painful, confusing, unmaintainable disaster after 2 years.
Critic: How long have you been using Docker/Kubernetes in production?
Advocate: 18 months.
http://www.smashcompany.com/technology/my-final-post-regardi...
[+] [-] Too|3 years ago|reply
It doesn’t mean every designer has his own metal sheet press on his desk. It means he understands his component is used in a bigger picture, should fit into the robot assembling it, use the same type of plastic and standardized screw as all other components, and so on.
Same goes for software, running your own shit sounds romantic and all, but having every developer in your company picking their own monitoring stack or understanding the intricacies of kubernetes networking is really not effective. Some central guidance defining the standard screw head to use is needed here as well.
[+] [-] Aeolun|3 years ago|reply
People that would traditionally be called Ops were turned into DevOps overnight, and felt like they suddenly had the mandate to try and barge into development, offering true, but ultimately misguided help.
If you haven't asked me what I need yet, don't bring me a platform that is "going to solve all my problems".
I imagine this is why the developers in OP's story do everything themselves. At least they're able to fix something if it breaks, instead of having to rely on the ops team that is off chasing the next shiny thing.
[+] [-] tkiolp4|3 years ago|reply
[+] [-] durnygbur|3 years ago|reply