top | item 17483640

The Bulk of Software Engineering in 2018 Is Just Plumbing

365 points| karlhughes | 7 years ago |karllhughes.com

258 comments

order
[+] alexanderdmitri|7 years ago|reply
When I was in college, one summer I worked up in Vermont putting together and maintaining trailer park homes. Plumbing was one of the best parts of the jobs in my opinion. I'd wager the person who wrote this article has never had to put a plumbing system together from scratch.

There is an art and a science to it. The guy I worked with put a lot of thought into the best way to design these things, often because he'd be the one who would then have to maintain them. He'd also get very upset when we go out to a trailer with a poor plumbing system. The main giveaway would be a mess of pipes going every which way without rhyme or reason. These systems were the ones that broke the most and were also the hardest to fix.

It might just be "plumbing" to someone that doesn't have to do it, but there is a whole lot that goes into it and you don't want a plumber who approaches his/her work with the catch phrase: "Well, it's just plumbing."

[+] rootusrootus|7 years ago|reply
And plumbers get paid pretty well, just like programmers. I talked to the plumber who did my house and he loved his job. He was reluctantly retiring and handing over the day-to-day operations to his son, but if he had been physically able he'd keep doing it.

He also said he refused to do work on existing plumbing. He loved to build new systems but hated maintaining old ones. Where have I heard that before...?

[+] rgrieselhuber|7 years ago|reply
We've been conditioned to believe that trades such as plumbing, carpentry, construction, etc. are beneath high IQ people so it's easy to see how this mentality carries through into software engineering.

Infrastructure, properly done, outlasts its own empires.

[+] neverminder|7 years ago|reply
But when you interviewed for this plumbing job, were you asked to explain in detail Bernoulli's principle of fluid dynamics? I think the author's point is ridiculous academically heavy interviews when the actual job is boring ass CRUD.
[+] kashyapc|7 years ago|reply
There's a reason why parts of the Linux community has an event named: The Linux Plumbers Conference :-)

https://linuxplumbersconf.org

A reputed community conference, co-organized by contributors and maintainers of the Linux kernel and related low-level "plumbing" utilities and libraries.

[+] PretzelFisch|7 years ago|reply
In the valley to do comercial plumbing you need to join the union and go through an apprentice program( 5 years and 1080 hours of classes) plus there is a lot of plumbing done for hospitals and clean rooms which takes more training and class work. After all of that is ready the trades have to work together and organize their schedule to get a given project done.

I don't know if the people making the programing is like plumbing comment realize how highly skilled a plumber is.

[+] cousin_it|7 years ago|reply
You just made me feel inspired in a new way about the job I've been doing for many years. Thank you!
[+] TAForObvReasons|7 years ago|reply
software development is "plumbing" now because many of the common problems have standard and high-quality solutions.

Plumbers don't have to craft their own pipes nowadays because the existing piping solutions are very well understood. It's not as important that the plumber knows exactly how the pipe is produced. They need to know how to set up the interface between the pipes.

Similarly in software, we have high quality solutions to many common problems ("the pipes"). While we do have some people focusing on designing the solutions, in 2018 people don't need to know how to make their own frameworks in order to use one. This reflects a maturity of the industry.

[+] eastbayjake|7 years ago|reply
... but you would be very upset if you'd spent years doing graduate research in fluid mechanics, gone through several interviews where you discussed theory, accepted an offer believing your role would be technically groundbreaking and use your knowledge of fluid mechanics -- then realize on your first day you had been hired as a plumber.

I think this is less a denigration of software engineering in 2018 than it is a denigration of software engineer interviewing in 2018.

[+] ken|7 years ago|reply
> I believe that many in the field are overpaid relative to the difficulty of the work they do.

I believe that every programmer is overpaid relative to the difficulty of the work they do -- or, more accurately, that The Market doesn't pay based on difficulty of work. Software pays so well because the product scales so well.

Successful pop musicians don't do work that's 100 times more 'difficult' than software engineering. They do work that scales even better (copying digital music; playing to arenas; branding on merch).

Having worked outside the software world, I absolutely do not believe that software people are any smarter, on average, than anyone else. Ever see a plumbing or electrical or structural system fail spectacularly? Other types of workers absolutely need to understand interactions between multiple complex systems, deal with obsolete and incompatible systems, and deal with changing and conflicting requirements and regulations.

If software is any more complex to deal with than physical systems, it's only because the architects and implementors let it get that way.

[+] ellius|7 years ago|reply
I'm reading Thomas Sowell's "Intellectuals and Society," and while I disagree with a lot in his arguments and his view of the world, I think he nails a key point about the philosophy behind free market economies. He says that conservative intellectuals don't view inequality and other social tragedies as mainly the consequence of policy, but rather as an inherent part of the human condition. Markets may act as a mechanism that conveys those inequalities, but they are not necessarily their cause. Prices are his first example. Commanding a higher salary doesn't mean you're a harder or even more skilled worker, it just means that you produce something that is judged by buyers (employers) to have a higher positive impact on their well-being.
[+] thermodynthrway|7 years ago|reply
I disagree on several points. A good chunk of why software engineers get paid well is because most people don't like writing code. It's not just because the businesses scale well, it's because it's an undesirable job for most.

Having worked many jobs I also disagree that it's easy, relatively. It's probably not due to IQ, it's because it requires constant thinking. Every other job I've worked is a light mental load compared to software and electrical engineering.

Most jobs you don't think about anything half the time. In hospitality you largely sit around waiting for people. Same with finance and sales in many cases. In HR most of your time is spent sending email, entering data on websites, and attending meetings. Or, your job is to talk to people most of the day, something many find more appealing.

Engineering is 6-8 hours of intense focus and fixing difficult to find bugs. As palatable as doing math homework for most people.

I disagree about intellectual difficulty as well. In school I was a teaching assistant and many, many students simply couldn't put working programs together. Even if you know all the constructs, software engineering is basically building something in your head then tanslating it. I've seen enough people try and fail to build any working programs for 6 months to believe that some people just aren't capable

[+] RobertDeNiro|7 years ago|reply
Keep in mind that it's easy to you because you're from that particular field. For a lot of people software development is still basically witchcraft.
[+] BurningFrog|7 years ago|reply
> I believe that every programmer is overpaid relative to the difficulty of the work they do -- or, more accurately, that The Market doesn't pay based on difficulty of work. Software pays so well because the product scales so well.

...which is another way of saying that writing software is a good use of relatively smart people's time, which is why the market rewards it.

[+] blablabla123|7 years ago|reply
> I believe that every programmer is overpaid relative to the difficulty of the work they do

What?? :D

No and no. People get paid for working on things that take an effort. Nobody would pay you if there was no reason for it. Even the "window watchers" (or how they're called) are payed because they didn't change jobs for 40 years, were loyal to their bosses, colleagues and company.

And yes of course, some jobs take a higher amount of "problem crunching" (whatever that means) than others. In fact sometimes this is the reason certain jobs can only be performed by certain programmers. Most programmer jobs cannot be done by none-programmers.

> If software is any more complex to deal with than physical systems, it's only because the architects and implementors let it get that way.

There are not many fields that are purely abstract, namely that's mathematics, logic and theoretical computer science. Those fields are notoriously difficult, every school kid knows this for mathematics and eventually gets that logic is also a hard nut. This completely misses the point that reasoning about completely artificial entities is an effort by itself. It not only gets harder with complexity but also depending how abstract it gets.

In fact their are even weird limitations that you do not experience in the physical world like paradoxes.

[+] lr4444lr|7 years ago|reply
Then any year now, the explosion of boot camps, nano-degrees, and general online courses should start to push the market to equilibrium with people looking to get a leg up, right? Any year now...
[+] laogai|7 years ago|reply
To use the plumbing analogy, even if most or all of the job will be maintenance, you still want engineers who knows the internals enough to build their own “plumbing” if/when needed. Maybe an apt comparison is how astronauts are trained heavily for the 1% of off script situations that might occur even if it doesn’t happen in the span of their careers.

Companies should feel able to innovate when appropriate by having hired engineers capable of doing so.

[+] wmccullough|7 years ago|reply
I’m not fully willing to accept the plumbing analogy myself. While I do agree that most of our work is CRUD over REST, I think there’s another element of skill that isn’t uncovered in the article. If we are humble plumbers, why does it take me six months to find engineers that understand basic concepts like cohesion and loose coupling beyond a text book level (and I live in a metropolitan area with several software schools!)?

Perhaps a more apt analogy is that we are more like home renovation specialists. If things go well, we can update your plumbing to get away from lead pipes, but if we have to route the new plumbing through a joist, we are going to have to educate you on why we should do something a certain way. If we find a water leak and rot that will put your home (business) as risk, count on us to tell you about the risk and let you decide if you want to address it.

I’d argue that we are simply doing plumbing when things are going really well, perhaps even when the house is small and was well built to start with. If plumbing was all we did, I’d argue that our field would be way more saturated than it is.

[+] Someone|7 years ago|reply
Are you suggesting all plumbers should know how to make siphons, valves, or even water pipes from scratch, just in case they ever get into a situation where shops that sell them for peanuts aren’t available?

Yes, you’ll need some people who know how to make siphons and valves, but that some is only a few, and most of them won’t even be plumbers, but work on building machinery that builds siphons and valves.

I think that in most jobs in the modern world, there’s plenty of low-level details that one doesn’t need to know about to do the job.

[+] daveFNbuck|7 years ago|reply
If your company has 100 employees, how many of them do you think need to be able to handle the 1% situations? The current standard hiring practice is to aim for all 100.
[+] adamsea|7 years ago|reply
Not to knock this comment, which makes a valid point, but I couldn’t help but smile at the implicit comparison of software engineers to astronauts when the article is preaching the value of developers thinking of themselves as the “humble plumber”.
[+] inertiatic|7 years ago|reply
>"This isn’t a popular opinion among software engineers, but I believe that many in the field are overpaid relative to the difficulty of the work they do."

But no one is paid relative to the difficulty of their job, dummy!

[+] TeMPOraL|7 years ago|reply
Or actual importance.

It's always a combination of what salary you can command (function of your soft skills) and how much money you can make the person paying you (function of market dynamics).

[+] arkis22|7 years ago|reply
And if people were paid relative to the difficulty of their job no one would have an incentive to make things simple or easy.
[+] tootie|7 years ago|reply
People should be paid relative to their accountability.
[+] Animats|7 years ago|reply
Overcomplication is sometimes an issue. Remember Soylent, the hipster's Ensure? They were always talking about their overdesigned server infrastructure, even though you could calculate from their sales volume that they did only a few transactions per minute. They could have used low-end shared hosting and CGI to handle that. But no, they had to be a "tech" company. (It looks like they've finally accepted that they're in the food business and are selling by the pallet to WalMart.)

Plumbing has standard ways of doing things, and standard parts that rarely change. Web services need to settle down like that. Really, few sites are doing anything very interesting.

[+] mborch|7 years ago|reply
He wants to make that case that software development is a more pedestrian affair than it might seem, without much of an argument. But it's not true.

Even a small shop needs to live up to pretty much a global standard, not least in terms of security. In the EU you must engineer with a security perspective in mind. The field is constantly moving. People come and go faster than most other industries.

I think there's an analogy to getting a better bike: it doesn't get easier, you just go faster.

[+] qaq|7 years ago|reply
Working at security related company even Fortune 500 are not really living up to "global standard" in terms of security. Smaller shops can't afford proper security solutions and def. can't afford a competent SOC.
[+] mannykannot|7 years ago|reply
Security is a valid counter-example, but, significantly, we are (collectively) very bad at it in practice, repeatedly re-implementing the same old mistakes in each new technology.
[+] betterworldb|7 years ago|reply
I recently got a better bicycle and it got easier and faster. Well lubricated bottom brackets and a new chain makes a massive difference in speed / resistance.
[+] tootie|7 years ago|reply
For most projects I've worked on in the past 10 years, the energy has majorly shifted towards front-end development. Building blocks have improved dramatically, but expectations have been rising even faster.

Backends are getting more and more homogenized. There's still a fair amount of work that goes into accurately implementing business rules and a much bigger focus on quality and maintainability than ever before. But there's not a ton of serious thinking involved. I can pretty much follow a script for implementing microservices that will scale up to millions of users. I somehow ended up with a high-paying enterprise architect job and when they ask how we can build a system that will have 1000 users, the answer is that you can do pretty much whatever the hell you want so long as the framework is less than 5 years old and it will be fine. Then I go get a coffee.

[+] funfunfunction|7 years ago|reply
This article has an interesting point. I think the corollary to the point made is that web software, which is what is providing most of the plumber-type jobs this article references, has reached a point of maturation where certain solutions have been accepted as best practices. This lowers costs for businesses and lowers the barrier to entry of the web software industry, which I think of as generally good, but it also cements the positions of certain organizations that are foundational to the infrastructure on which these best practices are built.

The saddest part about this for me is the realization that there is little room left for truly innovated web technologies that have real impact to businesses. Of course there are counter-examples to this (I would count WebRTC, WebSub, and a few others as having potential to impact some usecase specific best practices) but the majority of new software I see being hyped is either a new implementation of an old idea (React and Vue for FRP on the front end) or something touting decentralizing as the next major step which so far hasn't proved itself valuable in a commercial sense.

[+] AznHisoka|7 years ago|reply
It also doesnt help that building valuable software these days requires lots of data, especially if you are doing any machine learning.

And most startups lack any data. its all owned by the monopolies of Google, facebook, Amazon and Apple. So the rich will just get richer, and the chances of startups succeeding in any field is as low as ever.

[+] notacoward|7 years ago|reply
Makes me glad I work in a specialty (data storage) which provides the parts and tools the plumbers need. Sure, there's a shared grab-bag of techniques etc. that we all use, but it's far from plug and play. While the market for people like me will never grow anywhere near as big as the front-end plumbing stuff, demand and pay have been much more consistently good for the last couple of decades and are likely to remain so for a while.

My advice to new software engineers is to get into one of the more "boring" specialties. You might not be in the lottery for the really big bucks, but you probably weren't going to win anyway. There will still be plenty of interesting and fun challenges, with greater stability and (TBH IMO) more mature coworkers.

[+] jaiprabhu|7 years ago|reply
Genuinely curious - can you give me some examples of these specialities where one can go deeper? In am in this boat now where I have worked on pretty much on all layers of the software stack and I want to now narrow down my focus.
[+] segmondy|7 years ago|reply
This is a good way to offend most developers since most like to see themselves as rocket scientists instead of plumbers. But I have to agree, it's MOSTLY plumbing. Thus the reason we see people argue, "Don't make me leetcode during an interview bro! Who ever implements their own algorithm these days?"
[+] lgleason|7 years ago|reply
Beyond the ludicrous nature of the authors sttatement, a look at his Linkedin shows that he is a non-technical CTO. IE: A CTO who has has never really worked as an engineer for an extended period of time and din't come from that background. Thus, consider the source and move on...
[+] js8|7 years ago|reply
I think the article is quite right, and David Graeber made a similar observation when he talks about bullshit jobs (companies would rather hire someone to tapeduct the system rather than redesign it).

But I feel part of the problem is that the companies want to rather hire a plumber (I mean, software engineer) than to buy an off-the-shelf solution. It seems to me that off-the-shelf solutions are very costly and many companies think they can do it cheaper if they build it themselves. Which to me sounds like that the market is not functioning properly, for whatever reason.

[+] Daishiman|7 years ago|reply
"Just Plumbing" is not really an apt analogy.

Because, judging from all the people I've been working with throughout the years, it's amazingly difficult to write a clean, cohesive, maintainable, scalable CRUD app that will actually satisfy your customers' demands.

Making things that work is easy. Making things that work well, that's another story. And it's the reason why there's so many open spots and high salaries for senior web devs who can deliver.

[+] sonnyblarney|7 years ago|reply
"The Bulk of Software Engineering in 1XXX is Just Plumbing"

When was it not?

What were these guys thinking?

The vast majority of software is taking real world applications and dealing with the 'kind of a mess' of situations that can arise.

Even login/credentials can be a minor pain.

Doing it clearly and succinctly is the challenge.

Though it's often more 'solving problems' than 'building something' we should maybe think of ourselves more as construction workers than research scientists.

Upon graduation my skills were filled with so many holes I suggest that every CS grad takes a 3-month long 'power course' wherein the get all the basics, best practices and common tooling down.

[+] nobody271|7 years ago|reply
The writing has been on the wall for at least a decade. I remember hearing about how programmers would be more like plumbers all the way back in 2008. Sure enough, that really has come true for a lot of developers.

But I agree that this is good. The job is just changing. For example, maybe you wrote a custom DAL back in 2010. Now what you likely have is a mess that you have to pay several people a senior salary for because they are the only ones who understand how it works. Meanwhile a general purpose plug and play solution like entity framework is ubiquitously understood among .net developers. So now we don't need those "senior" DAL guys simply because they understand the mess they helped create a decade ago (also, about the last time they read any books).

I see it as more a levels of software development thing. low-level > medium-level > high-level > plumbing.

[+] amelius|7 years ago|reply
Yeah, in a way it is the curse of "open source": there are so many libraries and tools freely available that the only problem left is to tie them together.
[+] matz1|7 years ago|reply
I have no problem when getting paid engineer job when I'm just doing plumbing :-) but the problem is when I go look for a job or interview, there is literally no job ad for software plumber so I have to pretend to be engineer.
[+] zelos|7 years ago|reply
I constantly have this feeling: software development as a job, relative to the amount it pays, just feels ridiculously easy most of the time. Churn out some API that takes data from one place and exposes it in a slightly different form over here, write some unit tests and docs, done.

Then other days I'll spend an hour doing nothing but thinking, staring out the window, trying to figure out the best way to do solve some complex multi-threaded issue, before writing any code, and I feel like a real engineer again.

[+] falcolas|7 years ago|reply
I have to be honest - I don't see this. Maybe I'm lucky.

My current company is basically creating Microsoft Office on the web - because for the targeted niche, Microsoft Office (Google sheets, Libre office) wasn't cutting it. There aren't libraries for that.

My previous company was applying machine learning to the twitter firehose, and present the results in real time. The existing libraries weren't up to the task (or the volume).

Before that, we built tooling to allow ten people to administrate 40 plus companies' on-premise databases. There were a few libraries and tools we used, but nothing that we could just "plumb together".

I saw in the Reddit thread that the "plumbing" argument applies mostly to backend developers who just put REST libraries on top of databases. OK, I can kinda see that, but it's not like the complexity of the programs has disappeared, it was just moved to the frontend. And in a way, that's a benefit to the company (and the employees) - the resources previously tied up in re-creating REST libraries before REST libraries existed can now do other things.

The value of programmers has never been in the movement of data from point A to point B, it's been in the application of business logic to that data.

[+] ramoz|7 years ago|reply
If devs see their jobs as "plumbing" then that is a problem with their current culture & environment.

Most devs should be focused on delivering business/mission value. This means custom code, unique ip, algos, GUIs, etc.

However, the current state of devops has thrown all sorts of engineering challenges at devs that lose sight of the actual value that should be continuously delivered. It becomes plumbing when we get caught up in redundant tasks such as dependency management, tests, logging, containers, runtimes, integrations, cloud provider specifics for scaling/ha/sre, and connecting them all to create "the platform" that is supposed to enable true continuous delivery of value. We end up more focused on this internal platform development even though the tooling is pretty advanced. We still have to configure/orchestrate/manage that advanced tooling, and this, in turn, creates the plumping effect. We greatly underestimate the work involved with all of these things.

It is not plumbing when we are focused solely on the continuous delivery of value. We remain lean and agile, constantly testing a hypothesis and developing our "uniqueness".

[+] matte_black|7 years ago|reply
Very few developers deliver direct business value or revenue.

For every rockstar dev who gets paid a lot of money to make unique solutions and use his brain, there’s dozens of unsung heroes who do the grunt work reading from docs and setting up plumbing so the rockstar can deliver his value effectively.

These developers are not working in cool languages or solving new problems, and many of them will never do so in their life. They are just expendable workers swinging hammers and writing JavaScript.