Ask HN: Where can we find the unsexy jobs?

[throw1138]

Everything is double-digit YOY growth with impending exit, and kubernetes all-the-things, and DevSecOps, and SRE, and web-scale etc etc.

Where are the places running their setup out of a rack in a rando datacenter grandfathered into an affordable Edgecast plan running a LAMP stack on Debian using borg for backups?

This may read like satire, but I promise I'm seriously asking.

Or am I just way too old?

Related: https://boringtechnology.club/

[curiousllama]

They’re outside of the “tech” industry. Look at companies 200-300 on the Fortune 500; their IT group uses boring tech.

Just fair warning: you seem to be under the assumption that “boring” means “straightforward and just makes sense.” It does not. If you’re looking for that… good luck! Update if you find somewhere and I’ll join you.

[tinco]

You've got the wrong impression of what is "boring" tech. A setup out of a rack in a rando datacenter isn't boring, it's frustrating. Saying doing things the old way is boring, is like saying sending your wife out to carry water from the well in a jar on her head is boring. We've got running water now, that has nothing to do with double-digit YOY growth, it has to do with actually reliably and comfortably deploying and maintaining our infrastructure by leveraging modern technologies like Kubernetes.

I don't know how old you are, but now that I have running water I think to my old and boring setup I cringe at how much work it was to replace or add to an existing cluster, or to replicate it for testing out new features. How out of date the software was, and how easy it would be for an APT to exploit some known-for-years vulnerability in a transitive software dependency.

I wish I could afford an SRE in my current startup, but the fact that we can run a decently reliable setup that's easy to keep up to date, expand and modify is thanks to all of that web scale technology trickling down to us small fish.

And why not use borg for backups? We still do run Debian (or Ubuntu, but same diff), if it's good it's good.

[DrammBA]

I don't think he's saying he wants to go back to getting water from the well, instead he's saying he's got running water and he's trying to join company that won't expect him to terraform mars with the running water technology.

[zozbot234]

There's plenty of usable middle ground between "rack in a rando datacenter" and doing everything on the cloud. You can leverage VPS hosts, or run the bulk of your workload on prem and use the cloud for infrequent scalability needs.

[f0e4c2f7]

I try to make this point to people a lot and can never seen to quite get it across.

I really like your running water analogy.

Newer technology is usually easier. Especially if you pick the easy stuff.

I used to do all these tricks to work infra jobs but not be on call. Now - is it in AWS? Is the app in a container? Is it deployed with Kubernetes? Then I don't even care if there is on call!

I've woken up to fix Linux servers in the middle of the night dozens of times. And don't get me wrong - I love Linux. I have yet to have my first "kubernetes broke during the middle of the night" call.

Not to say that new tech doesn't have tradeoffs too. But good lord! It's so much harder with old tech. It's like you're purposely doing it harder to prove a point.

[muzani]

I've joined such "boring" companies before. Using SOAP instead of JSON/REST. Using email attachments for source control. Having to download every new version of a dependency and attaching it with Eclipse, instead of it auto downloading when I update the version number.

Also my grandma used to have gas lighting and wells. That was fun, but I can't go back to that.

I think people just don't like the uncertainty with new tech, but the old ones can be uncertain too.

[jf22]

Ok, I get your point, but the whole point about comparing mature tech to carrying water on your head before running water is ridiculous.

[Melatonic]

Who says you can't use Kubernetes and all of your running water tech running your own hardware? Plenty of places do it.

Sounds like you just have worked at places who sucked at updating, maintaining, and setting up their own hardware unfortunately. Which I agree can be a huge pain in the ass if done incorrectly.

[fxtentacle]

My impression is that there's a lot of Kubernetes jobs because all those unnecessary scaling tools come with additional maintenance work. So if you use new & messy technology, you just need more people. My boring deployments (Nginx+Ruby+PostgreSQL on Debian) can be maintained for multiple projects by just one employee, because there isn't that much to do apart from updating Ansible scripts and running them. But on the Ruby side, stuff gets deprecated/broken all the time, so we need much more manpower to maintain it. And Docker/Kubernetes (where everything depends on access to random GitHub URLs) is even a lot more fragile than that.

EDIT: I was considering doing this myself recently, so maybe you'll like the idea, too: Just walk physically to all the mom&dad shops in your area and ask them if they want a free website, if they allow you to put Google text ads on the side. If you have time, you can build those websites and super cheaply host them themselves, and it'll certainly help those shops get their feet wet in the digital age. Plus you build up some ad/link inventory which you can monetize later. The reason why I considered doing it is because I'm 100% sure that people will tell you a lot of technical stuff that they need solved, if you're there and willing to listen. And those small shops are going to need small cloud-free solutions.

[nolroz]

When I worked for a small web agency, I loved the fact that when you did a website for someone, you'd be introduced to their entire business model and everything within. It was a great opportunity to help them optimize all the things if they were willing. It lead to doing a lot of add on work like spreadsheet-> program work, physical network setup, ads, logos, kiosk applications, Christmas tree databases, etc...

I loved helping those small businesses but a lot of them don't have a lot of money to burn so going with a monthly rate they can plan on is powerful, as is managing expectations. A lot wanted Facebook for $500. There are patterns that emerge and one could build an in house stack to handle most everything I'm sure, just increase your monthly as they add services.

[nwellnhof]

> But on the Ruby side, stuff gets deprecated/broken all the time, so we need much more manpower to maintain it.

Maybe you should consider Perl? I'm only half joking. I have a few Perl projects running unchanged for 15 years and hopefully for the next 15 years as well.

[umvi]

Don't confuse Docker with K8s. Docker is more for solving dependency management problems than scaling and redundancy which are the problems k8s solves. I highly recommend some sort dependency management solution like Docker or Nix - it makes changing hardware or OSes down the road much easier.

[jthrowsitaway]

> And Docker/Kubernetes (where everything depends on access to random GitHub URLs)

Where is this true? I'd say this ecosystem is more dependent upon Docker Hub et al.

[froggertoaster]

Disclaimer: I own a boring consultancy.

Boring, however, is contextual. We have flex hours. We have unlimited PTO (and frequent reminders to take personal time - I already know the data related to offering unlim PTO and how often people then take PTO, thank you very much). We make over $1 million in revenue a year with 6 people and no plans to have exponential growth, coupled with high margins so that we aren't rugpulled by a rainy day (and to take appropriate profits of course). My employees report high levels of job satisfaction and happiness.

Our focus? Maintaining boring legacy software for low-risk clients. We also do some greenfield work, but we like the stability that comes with working on older cash cow software.

The jobs are out there. We're just not super flashy about it.

EDIT: I appreciate everyone who commented :) if this wasn't a burner I'd totally reach out to some of you!

[m348e912]

Unlimited PTO to me would only be interesting to me if you had a policy stating the minimum number of days/hours you must take a year. Reminders are nice and all but (written) policy is better.

[belmarca]

Can you share some more about what you do? Looks interesting! Like sibling comment, I'm also looking for remote part-time work. Such opportunities seem relatively rare.

[alongreyber]

I would love to hear a bit more about how you built your consultancy firm. I've done some consultancy work for local companies, and I'm looking to focus on similar work (helping local companies with ERP system integrations). If you have some time, I would love to connect and ask you some questions! If you can, please reach out to [email protected]. Thanks!

[py_or_dy]

uh, you got a careers page? :)

[rodolphoarruda]

I'm looking at the replies to your comment asking about jobs/career page etc.

Boring is the new sexy.

[throwaway787544]

I don't want to use K8s. But I also don't want to use a manually setup rack in a random DC.

Back in the day when we had to hire a guy to go down to the cage every day to repair yet another faulty machine, we would have given our left arm for the ability to magically spin up a new VM of any size with a baked image by just running an API call. It's freakin' magic.

I also love, love, love plain Fargate. Not as stupidly complicated as K8s, but I can still run containers without ever thinking about managing hosts. Just keep my containers up to date, push new ones, and trigger a deploy. Never have to "upgrade my cluster to keep from using EOL versions" or "migrate ingress to a new API version" or some other ridiculous thing. It just works, and all I have to think about is my app/container.

[dcminter]

So much this. I have painful memories of talking a not-very-technical "technical support" guy through the use of `lsof` on Linux (they'd never heard of it) to diagnose a deployment issue. Also of multi-day to multi-week waits to get hardware lined up for our projects.

I miss it not one jot.

[dotancohen]

Don't confuse faulty hardware with software abstractions of perfect hardware. Those cloud VMs are still running on some hardware somewhere, even it's been abstracted away from your control. And that hardware can and does fail.

[marvinblum]

> I also love, love, love plain Fargate.

Same goes for the HashiStack. I really don't want to work with Kubernetes anymore simply to deploy some containers.

[Melatonic]

You must have been running some damn cheap or old or faulty hardware because that is an insane amount of hardware maintenance.

[orev]

Whatever you were doing with those faulty machines that failed daily was a problem with you, not a problem with running remote hardware. I’ve run remote hardware for 20 years and only have to physically go to the data center a handful of times per year (and never need to use remote hands support). It just sounds like you were using crap quality equipment and/or didn’t have remote access setup correctly.

[bluetomcat]

Just leave your SF/VC bubble, move to a medium-sized European city and such jobs will be plentiful. Most likely you'll be maintaining internal business applications on uncool and legacy technology stacks. The value of these applications is mostly for the particular business and your experience wouldn't transfer easily to a new job. The upside is that job security is far greater there. Also, you'll have less opportunities to blog about what you are doing, or to write insightful HN comments because most of what you do is totally uninteresting for the world.

[ParksNet]

And between income tax, payroll tax, sales tax, you'll be taking a substantial cut in income - for housing which is more expensive, colder weather, and less sunlight.

[qwedf]

Industries that actually run the world and won't be disrupted by whatever VC-funded "Uber of X" whose business model is ultimately a Ponzi scheme. Think banking, engineering firms like Schneider/ABB, tools that are actually used in hospitals, etc. Look for major industry players that provide the backbone of basic infrastructure that you probably never heard of because they're too busy growing at a clip of 2% per year at $2B profit and don't need to crow. And of course the companies that support them

[sealeck]

Not sure that financial services (banks) are less likely to be perpetrating a Ponzi scheme than Uber, etc.

[rramadass]

Well Said!

I suspect this where the real "expertise" in Building Software Systems lies. Stable, Steady, Fault-Tolerant, least MTBF etc. characteristics are the name of the Game.

[Cerium]

Medical hardware is great. Boring time tested technology and good profits.

[olau]

We do, with a slightly charitable interpretation of LAMP.

We did have a brief stint with AWS, but that's thankfully over, partly because their Aurora Postgres-like cloud thing regularly would thrash to death because it doesn't have anywhere to store temporary data so a biggish join means out of memory, partly because the US law that says foreigners have no right to privacy from NSA with judicial review makes it illegal in Europe to use US providers unless data is encrypted client-side (see Schrems II).

I wish someone would start a sysadm contractor company offering setting up and maintaining servers at something like Hetzner. Take a service fee for being ready to respond to emergencies, bill for hours spent, and let Hetzner bill the hardware. Setting up a server isn't that bad, you can do that with a script, but ongoing maintenance and having someone always ready to respond to emergencies (that never occur, almost) is something you wouldn't mind having a competent sysadm take care of, as a developer.

I think there's a huge market for this in Europe with Schrems II, for those that aren't really deeply dependent on the software stack in cloud services. And you can get some beefy machines at Hetzner for pennies compared to Amazon or Google.

[joecot]

The answer is to talk to recruiters.

The job ads that catch your eye are specifically tailored for those results. They're usually posted from flashy startups who are high on their own supply. I had many an interview with those types of places, and nothing you could say would be good enough for them. That's why those job posting are always open.

What worked for me was happenstance -- I applied to a job that was posted by a recruiter, and once I was in that recruiting company's ecosystem, they started passing opportunities my way. The opportunities were interesting and had "boring" and relatable tech. If I looked up the job ads for those opportunities, the posting itself sucked and lacked the detail I'd expect. I'd have never applied without a recruiter talking to me about it.

There is a massive demand for developers at small and medium sized businesses. Those businesses have no idea how to effectively market their postings, which is why they use recruiters. The startups that know how to effectively market their postings on WeWorkRemotely and such have the applicant pool to be insanely choosy. If you're having trouble landing something, apply to jobs that are close to what you're looking for, but definitely posted by a recruiter. The recruiters will talk to you, and if they like what they hear, the interviews will pour in.

[bob1029]

Banking is pretty desperate for talent, probably because no one wants to suffer the complexity of the problem domain. I showed someone the models we use to track a customer implementation and they thought we were insane for even thinking this is a good business to be in.

Because the problem domain of banking sucks so much, the tooling is necessarily simple and "unsexy". We like things that fit into a single VM/folder and can be built in 1 click. Anything more complex than this is almost certainly going to fail by default under the combined weight of unnecessary technical complexity and vast (and necessary) business requirements.

It took us until the 3rd year of integration with one of our banking customers before someone in IT even gave enough of a shit to naturally ask the question "so what language/framework/technique did you write <product name> in?" Obviously, we are required to disclose this in legal materials prior to implementing our product, but our customer sees "Microsoft" and rubber-stamps it without much drama. No one else gives a shit about the tech after this phase of the project (which lasts approximately 1 phone call).

[jdmichal]

You are definitely working at a different bank than I was... Mine had an entire division focused just on internal technology and tooling. Which was actually a bit of a nightmare, because it led to things like building our own cloud with customized versions of off-the-shelf software. Because of course if you ask the internal technology group whether we should build a cloud or buy a cloud, what do you think the answer will be? And of course the publicly-available version is not good enough for our super special needs, so we'll customize it and just ignore the fact that we are now either (a) on a treadmill to keep up with upstream or (b) more likely just ignoring upstream. My company chose b, until eventually years later they realized that those super-special needs weren't so special and had a huge uplift project to get to the publically-available version.

Anyway, the group I worked in was an acquisition with a completely different tech stack. So we were basically completely ignored except for the occasional saber rattling that we should be moving to blessed tech stack. After which we would say, sure give us a project with time and budget reserved and we'll get right on that. Cue the crickets. Also, my group was PCI burdened, so we quickly became exempt from any of the afore-mentioned cloud stuff, because they were afraid we would make the cloud in scope for PCI.

Now, most of the people who I knew were decent have left. But it wasn't about the technology -- the paycheck is enough to get through that. It's mostly around back-to-office mandates. Which is a strange stance, after the company spent who-knows-how-much moving all our machines to be virtual desktops that we could log into from anywhere.

[eafkuor]

> Banking is pretty desperate for talent, probably because no one wants to suffer the complexity of the problem domain

I assume it's mostly because not many decent software developers want to suffer Banking's work environment

[Nextgrid]

> Banking is pretty desperate for talent

The salaries don't suggest that, unless I'm at the wrong locations and/or looking at the wrong banks.

[runekaagaard]

I totally agree, the range in number of servers/services/technologies different companies use to solve similar problems are SO wide.

After 12 years of growth we are still happily on:

  - 1 Ubuntu server
  - 1 MySQL db
  - 1 Django framework
  - Ansible for deployment

And thats mostly it. Due to Hetzner missing the 27002 certification we moved from metal to Amazon EC2 instances which was pretty uneventful. We also moved to Aurora and the point-in-time restores actually makes me sleep better at night. Relative to Hetzner it's a lot more expensive. Not at all a problem in the absolute sense.

Some years back I had a chance to talk with the CTO of a company very similar to ours, and their landscape was 400+ microservices, 50 servers managed by k8s on Azure Cloud, Service bridges, Workflow engines, 5 different DB engines, etc. It sounded very fascinating, but also quite complicated.

Some of my worst decisions over the years has been not having the balls to say no to current best-practices. I've learned the hard way that even though the pressure of using shining-new-thingTM can be immense it's so important to realise that everything always has pros and cons. The most important thing is trying to really understand _my_ specific problem and _then_ using the simplest and most obvious solution that can work.

Obligatory "Programmers are also human" link: https://www.youtube.com/watch?v=Uo3cL4nrGOk

[softwaredoug]

> Everything is double-digit YOY growth with impending exit

I'm going to say, on a random job board, this is actually the vast minority of jobs. Most, even at FAANG, have lots of unsexy maintenance that just needs to get done. Even in fancy machine learning land (where I work) 99% of the work is fairly 'boring' in the traditional sense.

Also, there is a strong bias to talk about 'sexy' work outside the company. For a variety of dev marketing reasons. Nobody does a conference talk about the uninteresting boring work that gets done.

[rufius]

I had a job with a lot of ML and Data Science a few years back.

I learned that the job really should’ve been called “Data Janitor”. 90-95% if the work was cleaning data sets and making sure that last update in some ML library we used didn’t mess up something.

[cupofpython]

> Nobody does a conference talk about the uninteresting boring work that gets done.

Id go to that

[manuelmoreale]

Small design studios. They still exist and they often need developers to develop “regular” websites using very down to earth stacks.

There’s a ton of business that simply need basic websites and those still need to be developed by someone.

[paxys]

Most tech jobs in the country are exactly the kind you describe. Of course if you are expecting a silicon valley salary to go with it then you will be disappointed. Look up any $50-60K IT job in a mid sized city on Indeed.

[Melatonic]

With the IT shortage it has actually shot up quite a bit - when no one will work your 50-60k job because everyone is chomping at the bit the salaries have risen quite a bit. Medium to large city will not be paying top end tech salary of course but you should be able to make between 100-200k depending on experience and the job.

[datalopers]

Sorry. We’re not hiring currently because our system is simple and reliable and we enjoy our excess profits.

[PascLeRasc]

Pittsburgh, PA. Every big company here - PNC, UPMC, Highmark, Fedex, Norfolk Southern, Duquesne Light - will pay you $65-70k/year at most. I met an IT director at one of these companies who told me that open source is a communist movement and so he's banned it. The dress code is very relaxed - if you stay late all week you can earn jeans Fridays!

At all of these companies you will be forced to use Internet Explorer 11 and you'll program in a remote desktop Notepad.exe on Windows Server 2012. Doing anything else is explicitly banned. The programming language is a proprietary version of Visual Basic with all the functions renamed and changed slightly so the vendor can control access to the $5k/year documentation, which is still horribly written. You will sign an agreement not to ever ask or answer questions about the language on Stack Overflow too. The vendor of these languages doesn't really exist beyond an email address like "[email protected]" - they will gaslight you that this is the real Python and python.org is a scam, "if you're not paying you're the product!". None of your coworkers will know what you're talking about when you mention Git or Linux.

[dimitrios1]

Work for a heavily regulated private industry: banking, finance, healthcare, energy.

I did startups for 10 years prior. Never ending rat races, constant high pressure deadlines, everything is "ASAP", compensation is iffy (your start ups got to make it).

In contrast, I work for a bank now. Easiest and most well compensated job I have ever had in my life. So easy that I am pursuing additional degrees (which they pay for), and moonlighting my own product offerings, virtually stress free.

They still dabble in some of the tech you mention. But they are much more deliberate about everything. And if they do go with something, it will be around for a long time.

[lifefeed]

Most jobs aren't sexy. This website focuses on Silicon Valley darlings and FAANGs and unicorn startups, but that's a minority of the available software jobs. There are a lot of good programming jobs available everywhere from banks to intellectual property protection to Domino's.

It can be frustrating working in companies that treat IT as a cost center, but there's still a lot of money to be made there. (And as a bonus, the interviews are ridiculously low-pressure compared to Google.)

[yalogin]

The problem is you will also want job security. The answer is to look in a company that is not tech. Lots of companies that have tech in the background, where it’s needed but not driving revenue. However even in that scenario there is a risk that the job could be cut because software wants to eat everything, move everything to a service oriented architecture or automate, or do the same thing using new buzzwords. So you need to look at a lot of parameters score you jump in.

[alkonaut]

If a company has 9 racks with their legacy products and 1 product sounding like the ones you describe, then the job ad will mention the kubernetes all the things product/job, but if you apply you'll be maintaining 10 products where 9 aren't very sexy at all.

So unless the company is extremely young (say younger than 10 years) then there will be tech of all ages and all levels of sexy.