This is what I tell engineers. Microservices aren't a performance strategy. They are a POTENTIAL cost saving strategy against performance. And an engineering coordination strategy.
Theoretically If you have a monolith that can be scaled horizontally there isn't any difference between having 10 replicas of your monolith and having 5 replicas of two microservices with the same codebase. UNLESS you are trying to underscale part of your functionality. You can't underscale part of your app with a monolith. Your pipe has to be big enough for all of it. Generally speaking though if you are talking about 10 replicas of something there's very little money to be saved anywhere.
Even then though the cost savings only start at large scales. You need to have a minimum of 3 replicas for resiliency. If those 3 replicas are too big for your scale then you are just wasting money.
The place where I see any real world benefit for most companies is just engineering coordination. With a single repo for a monolith I can make 1 team own that repo and tell them it's their responsibility to keep it clean. In a shared monolith however 0 people own it because everyone owns it and the repo becomes a disaster faster than you can say "we need enterprise caching".
> For example, at a recent generalist tech conference, there were six talks on how to build or deal with side effects of complex, microservice-based, architectures and zero on how one might build out a simple monolith.
Queue my favourite talk about microservices: David Schmitz - 10 Tips for failing badly at Microservices [1]
This guy has amazing delivery -- so dry and funny. He spends 45 minutes talking about all of his microservices mistakes!
Microservices are an organizational technology. They allow small teams to work independently from other teams.
The most frequent problem that I see is that people design Microservices that are "small as possible" instead of "Small enough than a small team can own them". For example, I have seen teams of 5-7 developers owning 20 microservices which is crazy IMHO. I blame the pre-fix micro which is highly miss leading. Now, if you merge together these 20 tiny microservices doesn't make monolith. A monolith would be a service or application that is owned by multiple/many teams.
The other important aspect is that microservices should be loosely coupled and what I see is highly coupled microservices. How do you know you have highly coupled microservices? I have seen a few things.
- They share business logic and developers end up creating libraries with business logic which is very problematic.
- They share access to the tables.
- Changes in one often require changes in other microservices.
- Integration and unit tests are not enough. You need E2E tests, but these are hard to write and brittle.
Monoliths aren’t very useful in many organisations where you need to build and connect 300+ systems. They also stop having simple architecture if you try. Most architecture conferences and talks tend to focus more on the enterprise side of things, and really, why would you need full time software focused architects if you’re building something like stackoverflow.
I do think things have gotten a little silly in many places with too much “building like we’re Netflix” because often your microservices can easily be what is essentially a bunch of containerised monoliths.
I think the main issue is that your IT architecture has or should have) very little to do with tech and everything to do with your company culture and business processes. Sometimes you have a very homogeneous focus maybe even on a single product, in which case microservices only begin to matter when you’re Netflix. Many times your business will consist of tens-thousands of teams with very different focuses and needs, and in these cases you should just never do monoliths unless you want to end up with a technical debt that will hinder your business from performing well down the line.
I think a lot of this comes from the additive cognitive bias, which is really deeply entrenched in most of our thinking.
Here's a concrete example, stolen from JD Long's great NormConf talk, "I'd have written a shorter solution but I didn't have the time", which is itself drawing from "People systematically overlook subtractive changes" by Adams, et al., 2021:
Give people a Lego construction consisting if a large, sturdy plinth with single small spire supporting a platform on top. The challenge is to get it to reliably support a brick. As is, it will collapse because that little spire is so flimsy. Ask participants to modify the structure so that it will support the brick. Incentivize simpler solutions by saying that each additional brick costs $0.50.
Given just this information, people universally try to fix the problem by adding a bunch more bricks to overbuild the structure, and they spend a lot of energy trying to come up with clever ways to reduce the number of bricks they add. But then the research team makes one small tweak to the instructions: they explicitly point out that removing bricks doesn't cost anything. With that nudge, people are much more likely to hit on the best solution.
In software, it's clear why we don't prefer subtractive changes - complexity. If there's any chance that the code in question has non-local effects that can't be reasoned about at the call site, it's risky to make subtractive changes.
Additive changes have the advantage of having a corresponding additive feature to test - you don't need to touch or even understand any of the existing complexity, just glue it on top.
So the cost structure is likely inverted from the study you describe. Additive changes are (locally) cheap. Subtractive changes are potentially expensive.
I'm not convinced that challenge shows what you say it shows. If someone set me a challenge to "modify this structure to support a brick" and the solution was "take the structure away and put the brick on the ground" I would feel like it was a stupid trick. "Remove the spire and put the brick lower" is less clear, but it's along those lines; "make this church and tower reliably support a clock at the top where everyone can see it", solution: "take the tower away and put the clock on the church roof", no, it's a Captain Kirk Kobayashi-Maru cheat where the solution to the "engineering challenge" is to meta-change-the-goal.
Yes you might get to change the goal in business software development, but you also know that the goal is 'working solutions' not 'build up this existing structure to do X'.
I believe it is as easy to over-engineer as it is to under-engineer. Architecture is the art of doing "just enough"; it must be as simple as possible, but as complex as necessary.
But that is hard to do, and it takes experience. And one thing that does not describe the IT industry well is "experience": don't most software developers have less than 5 years experience? You can read all the books you want and pass all the certifications you can like a modern Agile manager, but at the end of the day you will have the same problem they have: experience takes time.
Inexperienced engineers throw sexy tech at everything, inexperienced manager throw bullshit Agile metrics at everything. Same fight.
Also, the area of experience matters. I myself for instance have become pretty good at designing systems and infrastructure both from scratch up to corporate level. However, that will not help me too much when it comes to mega-corp level (think: FAANG) and other things such as designing public opensource libraries. (libraries and systems/applications require very different tradeoffs).
When was the last time you saw a system fail because it was under engineered? I don’t mean bad code. I mean something like a single file with 100k lines with structs and functions, or a MySQL instance serving millions of customers a day.
I'm going to agree with Dan Luu by asking where I can find more of these sane companies. I want to spend an honest 6h a working day improving a product (Terraform and Webpack are not the product) and spend the rest of my time tending to my garden.
I'd argue that this article could be read more like "start as a monolith, and then move towards microservices when sensible", except the penny hasn't fully dropped for the author that the sensible time for their organisation is right now.
The company appears dogmatically locked into their idling python code, their SQL making unpredictable commits, and their SQL framework making it difficult to make schema migrations.
This is a financial services company describing data-integrity bugs in their production platforms.
This one is a classic (instant classic), I can't say anything about this better than Dan did.
What I can offer is a personal anecdote about what an amazing guy he is. I had been a fan of his blog for a while, and at one point I decided to just email him and offer to fly to wherever he was because I wanted to hear his thoughts on a number of topics.
This was in maybe 2016, but even then I didn't expect someone who must get a zillion such emails to even reply, let alone invite me up to Seattle at my leisure! I think I had the flight booked on a Wednesday for a departure on Friday, for one night's stay, and Dan was profoundly generous with his time, we stayed up late into the night chatting and I learned a great deal, especially about the boundary between software and hardware (a topic on which he is a first-order expert with an uncommon gift for exposition).
I had the great fortune of spending some real time with Dan not just once but twice! When I went to NYC to get involved with what is now the Reels ML group, he happened to also be in NYC, and I had the singular pleasure to speak with him at length on a number of occasions: each illuminating and more fun than you can have without a jet ski.
Dan is a singularly rigorous thinker with the dry and pithy wit of a world-class comedian and a heart of gold, truly generous with his expertise and insight. I'm blessed to have met and worked with a number of world-class hackers, but few, few if any are such a joy to learn from.
I agree with the general sentiment that simple architectures are better and monoliths are mostly fine.
But.
I've dealt with way too many teams whose shit is falling over due to synchronous IO even at laughably low volumes. Don't do that if you can avoid it.
"Subtle data-integrity bugs" are not something we should be discussing in a system of financial record. Avoiding them should have been designed in from the start.
The most sophisticated arch ever needed for any scale in my FAANG and F500 jobs are a ssl-supported load balancer, multiple app servers with thread pools, a sharded database, and message queues. Everything else is just a dressed up version of this
You certainly can't fault the guy for trying, although it does make him seem a bit crazy. And I'm aware of scenarios where raising prices doesn't have the negative effect you might expect. You never know what people might spend but I can't imagine this is a tempting proposition for anyone.
Maybe it's all a ploy to get you to spend $16/month to see how many "Sponsor", "Major", and "Patron" level subscribers there are.
I find that architecture should benefit the social structure of the engineering team, and there are limits. I work on one of these “simple architectures” at large scale… and it’s absolute hell. But then, the contributor count to this massive monorepo + “simple architecture” hell numbers in the thousands.
Wave financial is only 350 people according to wikipedia - I doubt that’s 350 engineers. I know only of Google and Meta that can even operate with a massive monorepo, but I wouldn’t call their architecture “simple”. And even they do massive internal tooling investments - I mean, Google wrote their own version control system.
So I tend to think “keep it simple until you push past Dunbar’s number, then reorganize around that”. Once stable social relationships break down, managing change at this scale becomes a weird combination of incredible rigidity and absolute chaos.
You might make some stopgap utility and then a month later 15 other teams are using it. Or some other team wants to change something for their product and just submits a bunch of changes to your product with unforseen breakage. Or some “cost reduction effort” halves memory and available threads slowing down background processes.
Keeping up with all this means managing hundreds of different threads of communication happening. It’s just too much and nobody can ever ask the question “what’s changed in the last week” because it would be a novel.
This isn’t an argument for monoliths vs microservices, because I think that’s just the wrong perspective. It’s an argument to think about your social structure first, and I rarely see this discussed well. Most companies just spin up teams to make a thing and then don’t think about how these teams collaborate, and technical leadership never really questions how the architecture can supplement or block that collaboration until it’s a massive problem, at which point any change is incredibly expensive.
Early in my career one of the IT guys told me that one of the people on staff was "a technical magpie". I looked at him with a raised eyebrow and he said "He has to grab every shiny piece of tech that shows up and add it to the pile".
This is where we are.
I can't tell you how many times I have seen projects get done just to pad a PM or developers resume. Just because it was the lastest and greatest hot shit thing to use. No sense of if it would be better, faster, cheaper, useful.
When cloud was the hot new thing the company that I worked with launched a replatform on AWS. It gave us the ability to get through the initial scaling and sizing with ease. We left right away, because even then the costs did not make sense. Now we see folks crying about "exit fees" that were always there. That assumes that your institution even has the gall to own years of pissing away money.
Workman like functionality isnt sexy, it wont be the hot bullet point on your resume, it wont get you your next job, but it is dam effective.
> Workman like functionality isnt sexy, it wont be the hot bullet point on your resume, it wont get you your next job, but it is dam effective.
So, not fun, not rewarding, no intellectual challenge, no career benefit. Why exactly should I want to do it? This isn't the goddamn United Federation of Planets, nor is the company a church - why exactly should I go above and beyond what I agreed to in exchange for my salary? It's not like the bosses go above and beyond either, nor do they believe in company "mission".
To be clear: I understand the importance of actually doing your job right, and benefits of using boring tech, but you are not selling that well here. Employees need food and shelter and creature comforts, and so do their families. They are going to think beyond the current job, because if they won't, nobody else will.
That guy was just optimizing for future employability, albeit in a short sighted way. Being able to talk in an interview about how you have professional experience with various tech stacks is valuable. That being said, optimizing for that at the cost of current job performance and coworker alienation is just dumb, since job performance and networking leads are more important for landing good jobs. I'm guessing this guy was a serial job hopper who had no expectation of being able to progress up the ladder at the company you were at.
I'm immediately reminded of my favourite Kurt Vonnegut quote: "Another flaw in the human character is that everybody wants to build and nobody wants to do maintenance."
I've always felt that the magpie syndrome you describe is because of the desire to build new things, rather than maintain what's there.
I watched a repair show recently where a craftsman repaired an old 70s bedside clock. The pride, time and patience he took in repairing the clock was wonderful to see, actively avoiding rebuilding parts if he could reuse what was there, even if there was a crack or blemish.
I've always respected engineers that maintained and fixed software well, and also knew when to reach for the right tool in the toolbox. Better yet, those that knew when not to build something new. Perhaps that's something you learn through experience and doing, but I wonder if it's actively taught, encouraged, or rewarded in the workplace. It really should help get you your next job.
One of the best jobs I ever had was under "technical magpie." Did we get shit done? No. Did I get paid a lot of money and always have new stuff to do instead of shoveling CRUD? Absolutely. It was a blast.
Just be careful not to go too far in the opposite direction. There are new things coming all the time. You probably don't want to be writing new COBOL anymore even though it was once a good idea (you might have to maintain it, but you should already know what you replace it with and what your interoperability strategy is)
As a product manager, I am frequently confronted by UX people who declare something as „standard“, a feature that is supposed to be an absolute „must-have“, or else our organisation would loose even the last of our users. Unfortunately, developers tend to accept these things as interesting challenges and (knowingly or not) underestimate the effort and complexity needed to implement it.
My very lonesome role in these cases is to insist that this shiny thing is no standard at all and that our users would be quite happy without it.
> I have seen projects get done just to pad a PM or developers resume
This reminds me of the time that I complained that a sensor on the machine we were developing was way too complicated and that there were far simpler ways to accomplish the same thing.
Someone with more history on the project explained. The engineer who designed that sensor expected, and received, a patent for his work. The company was quite generous with patent royalties and no one's getting a patent for an off the shelf float switch!
It's the job of engineering management to stop this. We're supposed to say "why do you need this? Justify the need for this". I.E. "Why do you need kafka here? Will we have enough traffic volume to warrant it? Make a proposal." And they need to follow up and ask "Was that needed? Show how we're using it".
But engineering management is so busy filling out TPS reports they don't have time to actually do any oversight.
I'd argue it's something a little more fundamental than mere CV padding
Taking the CV aspect literally — this is a sleight of hand because it's a metaphor — I know lots of people who do this stuff that don't have a CV at all.
There's levels to it of course, but I don't really view it be any different to people who insist on using old machines as a bit (but in the other way obviously)
Don’t we get paid the big bucks precisely because we have to fix stuff like this? I mean, if maintaining and fixing software were easy, I guess we wouldn’t be earning 6 figures.
In software engineering We have all these principles and patterns and whatnot precisely because we have to deal with a pile of things that don’t work well together.
It’s not the tech, it’s the business - people pay for new and shiny things to be added, regardless of the actual value they bring. Engineering managers hire for shiny things on your resume precisely because of that business trend.
Tech trend will continue until this business mindset of burning money on shiny things changes.
It's a common behavior. When I started my last job as the software lead at a small company, I was warned that one of the senior engineers on my team was easily distracted by shiny things. They were not wrong. The dude was smart and effective, but I had to spend way too much time keeping him on task.
> Workman like functionality isnt sexy, it wont be the hot bullet point on your resume, it wont get you your next job, but it is dam effective.
When you see opportunities to do such work in a way that delivers something new and valuable, I recommend taking hold of them. I learned this somewhere along the line, but not until I’d missed a few from self-doubt and indulging the inner magpie.
Clear, simple and value focused engineering is _exactly_ what I’m looking for in candidates these days. Focus on the long term, not the short term — just like with investing.
deal with a bunch of this right now, no considerations for future growth of system and also dump everything in json and itll be ok ....tech debt in architectural designs is real... and it takes a lot to trim it back and say ok now we are moving to XYZ tool that works and doesnt need to be shiny. Had a chat with a client once and they needed something and i said this looks like itll be a report and they wanted some super duper dashboard but all you needed was a small db + csv extract for charts etc.
> When cloud was the hot new thing the company that I worked with launched a replatform on AWS. It gave us the ability to get through the initial scaling and sizing with ease. We left right away, because even then the costs did not make sense.
Cases like this always fascinate me. I've led a "move from Data Center to AWS" effort twice, and both times it was at > 50% cost savings. However, I think both were probably small infra footprints compared to many cases like many others.
I wonder if this effect ultimately advantages Google, and other companies that create their own internal tools rather than using the new shiny buzzword.
Not to take anything away from the angle (which is bang on), but the "we're X people valued at $YBILLIONS" is a weird way to open and detracts from the message. I suppose it's a proxy for.. seriousness? I dunno. Valuations are just bets on NPV given "what could go right" scenarios, so congrats on having a convincing story to tres commas your val. But opening a pitch on architecture with "we're a small team building a simple numbers app but lol $BILLIONS" and then using some other company (SO) as the case study ref.. ok yup what. This affected casual "#winning but like whatever" style is everywhere now. But I'm a salty old fuck so it's probably just me it offsides.
The problem is that most people have never actually built something in all the architectures they are considering.
They just read blog posts of what to do.
As an analogy, I’ve manufactured a lot of stuff out of wood, concrete, plastic (3D printed) and increasingly more metal. When I need a jig, I know what each material will be like and what I will be getting out of it.
As a young gun coming from working in games before touching "internet"/"enterprise" software back around 2006 I had an eye on performance matters and all the early Google papers caught my eye (esp as I was on a project with an overpositive sales CEO that had in his mind that we'd run to Google valuations within a year).
A sobering account was our second CTO who told us that their previous 65000 user application ran on a single database/server, so for the rest of that project we just kept most our application on a single DB and focused on tuning that where needed.
Highlight that 99% of companies should take away from this piece:
>since we’re only handling billions of requests a month (for now), the cost of this is low even when using a slow language, like Python [and simple synchronous code], and paying retail public cloud prices.
> Another area is with software we’ve had to build (instead of buy). When we started out, we strongly preferred buying software over building it because a team of only a few engineers can’t afford the time cost of building everything. That was the right choice at the time even though the “buy” option generally gives you tools that don’t work. In cases where vendors can’t be convinced to fix showstopping bugs that are critical blockers for us, it does make sense to build more of our own tools and maintain in-house expertise in more areas, in contradiction to the standard advice that a company should only choose to “build” in its core competency. Much of that complexity is complexity that we don’t want to take on, but in some product categories, even after fairly extensive research we haven’t found any vendor that seems likely to provide a product that works for us. To be fair to our vendors, the problem they’d need to solve to deliver a working solution to us is much more complex than the problem we need to solve since our vendors are taking on the complexity of solving a problem for every customer, whereas we only need to solve the problem for one customer, ourselves.
This is more and more my philosophy. I've been working on a data science project with headline scraping (I want to do topic modeling on headlines during the course of the election) and kept preferring roll your own solutions to off the shelf ones.
For instance, instead of using flask (as I did in a previous iteration of this project a few years ago) I went with Jinja2 and rolled my own static site generator. For scraping I used scrapy on my last project, on this one I wrote my own queue and scraper class. It works fantastically.
It was a genuine joy to see a website in this day and age which doesn't try to impose margins on text. I want my entire monitor to be used, and this site delivers.
You might be alone in this take. While I agree a lot of websites do it poorly, the only way I personally can read Danluu is when I make the window smaller.
[+] [-] from-nibly|2 years ago|reply
Theoretically If you have a monolith that can be scaled horizontally there isn't any difference between having 10 replicas of your monolith and having 5 replicas of two microservices with the same codebase. UNLESS you are trying to underscale part of your functionality. You can't underscale part of your app with a monolith. Your pipe has to be big enough for all of it. Generally speaking though if you are talking about 10 replicas of something there's very little money to be saved anywhere.
Even then though the cost savings only start at large scales. You need to have a minimum of 3 replicas for resiliency. If those 3 replicas are too big for your scale then you are just wasting money.
The place where I see any real world benefit for most companies is just engineering coordination. With a single repo for a monolith I can make 1 team own that repo and tell them it's their responsibility to keep it clean. In a shared monolith however 0 people own it because everyone owns it and the repo becomes a disaster faster than you can say "we need enterprise caching".
[+] [-] throwaway2037|2 years ago|reply
Queue my favourite talk about microservices: David Schmitz - 10 Tips for failing badly at Microservices [1]
This guy has amazing delivery -- so dry and funny. He spends 45 minutes talking about all of his microservices mistakes!
[1] https://www.youtube.com/watch?v=r8mtXJh3hzM
[+] [-] corpMaverick|2 years ago|reply
The most frequent problem that I see is that people design Microservices that are "small as possible" instead of "Small enough than a small team can own them". For example, I have seen teams of 5-7 developers owning 20 microservices which is crazy IMHO. I blame the pre-fix micro which is highly miss leading. Now, if you merge together these 20 tiny microservices doesn't make monolith. A monolith would be a service or application that is owned by multiple/many teams.
The other important aspect is that microservices should be loosely coupled and what I see is highly coupled microservices. How do you know you have highly coupled microservices? I have seen a few things.
- They share business logic and developers end up creating libraries with business logic which is very problematic.
- They share access to the tables.
- Changes in one often require changes in other microservices.
- Integration and unit tests are not enough. You need E2E tests, but these are hard to write and brittle.
[+] [-] devjab|2 years ago|reply
I do think things have gotten a little silly in many places with too much “building like we’re Netflix” because often your microservices can easily be what is essentially a bunch of containerised monoliths.
I think the main issue is that your IT architecture has or should have) very little to do with tech and everything to do with your company culture and business processes. Sometimes you have a very homogeneous focus maybe even on a single product, in which case microservices only begin to matter when you’re Netflix. Many times your business will consist of tens-thousands of teams with very different focuses and needs, and in these cases you should just never do monoliths unless you want to end up with a technical debt that will hinder your business from performing well down the line.
[+] [-] bunderbunder|2 years ago|reply
Here's a concrete example, stolen from JD Long's great NormConf talk, "I'd have written a shorter solution but I didn't have the time", which is itself drawing from "People systematically overlook subtractive changes" by Adams, et al., 2021:
Give people a Lego construction consisting if a large, sturdy plinth with single small spire supporting a platform on top. The challenge is to get it to reliably support a brick. As is, it will collapse because that little spire is so flimsy. Ask participants to modify the structure so that it will support the brick. Incentivize simpler solutions by saying that each additional brick costs $0.50.
Given just this information, people universally try to fix the problem by adding a bunch more bricks to overbuild the structure, and they spend a lot of energy trying to come up with clever ways to reduce the number of bricks they add. But then the research team makes one small tweak to the instructions: they explicitly point out that removing bricks doesn't cost anything. With that nudge, people are much more likely to hit on the best solution.
[+] [-] perrygeo|2 years ago|reply
Additive changes have the advantage of having a corresponding additive feature to test - you don't need to touch or even understand any of the existing complexity, just glue it on top.
So the cost structure is likely inverted from the study you describe. Additive changes are (locally) cheap. Subtractive changes are potentially expensive.
[+] [-] jodrellblank|2 years ago|reply
Yes you might get to change the goal in business software development, but you also know that the goal is 'working solutions' not 'build up this existing structure to do X'.
[+] [-] palata|2 years ago|reply
But that is hard to do, and it takes experience. And one thing that does not describe the IT industry well is "experience": don't most software developers have less than 5 years experience? You can read all the books you want and pass all the certifications you can like a modern Agile manager, but at the end of the day you will have the same problem they have: experience takes time.
Inexperienced engineers throw sexy tech at everything, inexperienced manager throw bullshit Agile metrics at everything. Same fight.
[+] [-] bjornsing|2 years ago|reply
Or as Albert Einstein phrased it: “Everything should be made as simple as possible, but not simpler!”
[+] [-] geodel|2 years ago|reply
[+] [-] IshKebab|2 years ago|reply
[+] [-] valenterry|2 years ago|reply
Also, the area of experience matters. I myself for instance have become pretty good at designing systems and infrastructure both from scratch up to corporate level. However, that will not help me too much when it comes to mega-corp level (think: FAANG) and other things such as designing public opensource libraries. (libraries and systems/applications require very different tradeoffs).
[+] [-] whiterknight|2 years ago|reply
[+] [-] gherkinnn|2 years ago|reply
[+] [-] Shrezzing|2 years ago|reply
The company appears dogmatically locked into their idling python code, their SQL making unpredictable commits, and their SQL framework making it difficult to make schema migrations.
This is a financial services company describing data-integrity bugs in their production platforms.
[+] [-] benreesman|2 years ago|reply
What I can offer is a personal anecdote about what an amazing guy he is. I had been a fan of his blog for a while, and at one point I decided to just email him and offer to fly to wherever he was because I wanted to hear his thoughts on a number of topics.
This was in maybe 2016, but even then I didn't expect someone who must get a zillion such emails to even reply, let alone invite me up to Seattle at my leisure! I think I had the flight booked on a Wednesday for a departure on Friday, for one night's stay, and Dan was profoundly generous with his time, we stayed up late into the night chatting and I learned a great deal, especially about the boundary between software and hardware (a topic on which he is a first-order expert with an uncommon gift for exposition).
I had the great fortune of spending some real time with Dan not just once but twice! When I went to NYC to get involved with what is now the Reels ML group, he happened to also be in NYC, and I had the singular pleasure to speak with him at length on a number of occasions: each illuminating and more fun than you can have without a jet ski.
Dan is a singularly rigorous thinker with the dry and pithy wit of a world-class comedian and a heart of gold, truly generous with his expertise and insight. I'm blessed to have met and worked with a number of world-class hackers, but few, few if any are such a joy to learn from.
[+] [-] 2d8a875f-39a2-4|2 years ago|reply
But.
I've dealt with way too many teams whose shit is falling over due to synchronous IO even at laughably low volumes. Don't do that if you can avoid it.
"Subtle data-integrity bugs" are not something we should be discussing in a system of financial record. Avoiding them should have been designed in from the start.
[+] [-] cityguy33|2 years ago|reply
[+] [-] nonethewiser|2 years ago|reply
$16/month "Short-form posts"
$256/month "Sponsor"
$2,048/month "Major Sponsor"
$16,384/month "Patron"
You certainly can't fault the guy for trying, although it does make him seem a bit crazy. And I'm aware of scenarios where raising prices doesn't have the negative effect you might expect. You never know what people might spend but I can't imagine this is a tempting proposition for anyone.
Maybe it's all a ploy to get you to spend $16/month to see how many "Sponsor", "Major", and "Patron" level subscribers there are.
[+] [-] mr_tristan|2 years ago|reply
Wave financial is only 350 people according to wikipedia - I doubt that’s 350 engineers. I know only of Google and Meta that can even operate with a massive monorepo, but I wouldn’t call their architecture “simple”. And even they do massive internal tooling investments - I mean, Google wrote their own version control system.
So I tend to think “keep it simple until you push past Dunbar’s number, then reorganize around that”. Once stable social relationships break down, managing change at this scale becomes a weird combination of incredible rigidity and absolute chaos.
You might make some stopgap utility and then a month later 15 other teams are using it. Or some other team wants to change something for their product and just submits a bunch of changes to your product with unforseen breakage. Or some “cost reduction effort” halves memory and available threads slowing down background processes.
Keeping up with all this means managing hundreds of different threads of communication happening. It’s just too much and nobody can ever ask the question “what’s changed in the last week” because it would be a novel.
This isn’t an argument for monoliths vs microservices, because I think that’s just the wrong perspective. It’s an argument to think about your social structure first, and I rarely see this discussed well. Most companies just spin up teams to make a thing and then don’t think about how these teams collaborate, and technical leadership never really questions how the architecture can supplement or block that collaboration until it’s a massive problem, at which point any change is incredibly expensive.
[+] [-] zer00eyz|2 years ago|reply
This is where we are.
I can't tell you how many times I have seen projects get done just to pad a PM or developers resume. Just because it was the lastest and greatest hot shit thing to use. No sense of if it would be better, faster, cheaper, useful.
When cloud was the hot new thing the company that I worked with launched a replatform on AWS. It gave us the ability to get through the initial scaling and sizing with ease. We left right away, because even then the costs did not make sense. Now we see folks crying about "exit fees" that were always there. That assumes that your institution even has the gall to own years of pissing away money.
Workman like functionality isnt sexy, it wont be the hot bullet point on your resume, it wont get you your next job, but it is dam effective.
[+] [-] TeMPOraL|2 years ago|reply
So, not fun, not rewarding, no intellectual challenge, no career benefit. Why exactly should I want to do it? This isn't the goddamn United Federation of Planets, nor is the company a church - why exactly should I go above and beyond what I agreed to in exchange for my salary? It's not like the bosses go above and beyond either, nor do they believe in company "mission".
To be clear: I understand the importance of actually doing your job right, and benefits of using boring tech, but you are not selling that well here. Employees need food and shelter and creature comforts, and so do their families. They are going to think beyond the current job, because if they won't, nobody else will.
[+] [-] CuriouslyC|2 years ago|reply
[+] [-] cmsefton|2 years ago|reply
I've always felt that the magpie syndrome you describe is because of the desire to build new things, rather than maintain what's there.
I watched a repair show recently where a craftsman repaired an old 70s bedside clock. The pride, time and patience he took in repairing the clock was wonderful to see, actively avoiding rebuilding parts if he could reuse what was there, even if there was a crack or blemish.
I've always respected engineers that maintained and fixed software well, and also knew when to reach for the right tool in the toolbox. Better yet, those that knew when not to build something new. Perhaps that's something you learn through experience and doing, but I wonder if it's actively taught, encouraged, or rewarded in the workplace. It really should help get you your next job.
[+] [-] duped|2 years ago|reply
[+] [-] bluGill|2 years ago|reply
[+] [-] zmgsabst|2 years ago|reply
We see it in the C-suite; we see it with engineers.
I think the travesty of so-called “principal engineers” and “engineering leaders” is their adamant refusal to make doing the Right Thing (TM) sexy.
Your employees are monkeys: act like it.
[+] [-] olav|2 years ago|reply
My very lonesome role in these cases is to insist that this shiny thing is no standard at all and that our users would be quite happy without it.
[+] [-] HeyLaughingBoy|2 years ago|reply
This reminds me of the time that I complained that a sensor on the machine we were developing was way too complicated and that there were far simpler ways to accomplish the same thing.
Someone with more history on the project explained. The engineer who designed that sensor expected, and received, a patent for his work. The company was quite generous with patent royalties and no one's getting a patent for an off the shelf float switch!
[+] [-] phendrenad2|2 years ago|reply
But engineering management is so busy filling out TPS reports they don't have time to actually do any oversight.
[+] [-] nunez|2 years ago|reply
[+] [-] mhh__|2 years ago|reply
Taking the CV aspect literally — this is a sleight of hand because it's a metaphor — I know lots of people who do this stuff that don't have a CV at all.
There's levels to it of course, but I don't really view it be any different to people who insist on using old machines as a bit (but in the other way obviously)
[+] [-] dakiol|2 years ago|reply
In software engineering We have all these principles and patterns and whatnot precisely because we have to deal with a pile of things that don’t work well together.
[+] [-] smugglerFlynn|2 years ago|reply
Tech trend will continue until this business mindset of burning money on shiny things changes.
[+] [-] HeyLaughingBoy|2 years ago|reply
[+] [-] mlhpdx|2 years ago|reply
When you see opportunities to do such work in a way that delivers something new and valuable, I recommend taking hold of them. I learned this somewhere along the line, but not until I’d missed a few from self-doubt and indulging the inner magpie.
Clear, simple and value focused engineering is _exactly_ what I’m looking for in candidates these days. Focus on the long term, not the short term — just like with investing.
[+] [-] tamiral|2 years ago|reply
[+] [-] fatnoah|2 years ago|reply
Cases like this always fascinate me. I've led a "move from Data Center to AWS" effort twice, and both times it was at > 50% cost savings. However, I think both were probably small infra footprints compared to many cases like many others.
[+] [-] phendrenad2|2 years ago|reply
[+] [-] robertlagrant|2 years ago|reply
[+] [-] popcorncowboy|2 years ago|reply
[+] [-] throwawayanothe|2 years ago|reply
[+] [-] disgruntledphd2|2 years ago|reply
Complexity for the sake of complexity is foolish.
[+] [-] treflop|2 years ago|reply
They just read blog posts of what to do.
As an analogy, I’ve manufactured a lot of stuff out of wood, concrete, plastic (3D printed) and increasingly more metal. When I need a jig, I know what each material will be like and what I will be getting out of it.
[+] [-] whizzter|2 years ago|reply
A sobering account was our second CTO who told us that their previous 65000 user application ran on a single database/server, so for the rest of that project we just kept most our application on a single DB and focused on tuning that where needed.
[+] [-] vendiddy|2 years ago|reply
When hiring someone, I love to see if they are visibly annoyed when something is over-engineered. Or when a user interface is too complex.
If they have strong emotional reaction, it's a good cultural fit :)
The inverse is true too. If someone "gets off" to complexity, I view it a red flag.
[+] [-] Zopieux|2 years ago|reply
>since we’re only handling billions of requests a month (for now), the cost of this is low even when using a slow language, like Python [and simple synchronous code], and paying retail public cloud prices.
[+] [-] annowiki|2 years ago|reply
This is more and more my philosophy. I've been working on a data science project with headline scraping (I want to do topic modeling on headlines during the course of the election) and kept preferring roll your own solutions to off the shelf ones.
For instance, instead of using flask (as I did in a previous iteration of this project a few years ago) I went with Jinja2 and rolled my own static site generator. For scraping I used scrapy on my last project, on this one I wrote my own queue and scraper class. It works fantastically.
[+] [-] bigstrat2003|2 years ago|reply
[+] [-] MH15|2 years ago|reply