[rossdavidh]

So, a couple years ago Microsoft was the first large, public-facing software organization to make LLM-assisted coding a big part of their production. If LLM's really delivered 10x productivity improvements, as claimed by some, then we should by now be seeing an explosion of productivity out of Microsoft. It's been a couple years, so if it really helps then we should see it by now.

So, either LLM-assisted coding is not delivering the benefits some thought it would, or Microsoft, despite being an early investor in OpenAI, is not using it much internally on things that really matter to them (like Windows). Either way, I'm not impressed.

[Someone1234]

I know blaming everything on LLMs is in vogue right now; but this is much more to do with Microsoft very publically firing the QA department[0][1] as a cost savings measure and claiming developers will do their own QA (long before LLMs were on the scene). It started in 2014 and the trickle never stopped.

Microsoft has a cultural problem; it went from an "engineers" company to an MBA directed one, trying to maximize short-term shareholder value at the cost of long-term company reputation/growth. It is very common and typical of US Corporate culture today, and catastrophic in the long-run.

[0] https://arstechnica.com/information-technology/2014/08/how-m...

[1] https://www.reuters.com/article/business/microsoft-expected-...

[mancerayder]

The arstechnica article was very good as a history of waterfall v sprint using MS as a case study. However the firing the QA department narrative is not supported:

Prior to these cuts, Testing/QA staff was in some parts of the company outnumbering developers by about two to one. Afterward, the ratio was closer to one to one. As a precursor to these layoffs and the shifting roles of development and testing, the OSG renamed its test team to “Quality.”

Two QA per dev?? That seems ginormous to me. What am I missing about the narrative about evil corp sending all of QA packing, that seems not supported here?

The second, Reuters article seems like it's saying something different than the QA firing narrative - it seems to talk about Nokia acquisition specifically and a smattering of layoffs.

Not supporting layoffs or eliminating QA, and I'm deeply annoyed at Windows 11. I just don't see these as supportive of the narrative here that QA is kaput.

[debugnik]

That was in 2014, doesn't explain the timing of these increasingly common broken patches. I had never gotten as many calls over Windows Update messes from my non-techie family as last year.

[namcheapisdumb]

> I know blaming everything on LLMs is in vogue right now; but this is much more to do with Microsoft very publically firing the QA department

A move no doubt encouraged by c-suites to demonstrate how effective LLMs are in the budget tally.

[throwaway85825]

There's a great talk that explains how code structure ends up looking like the org chart, and every subsequent organization chart layered on top producing spaghetti code. Windows is now old and full of spaghetti code. Then Microsoft layed off all the expensive seniors who knew the stack and replaced them with cheaper diverse and outsourced staff. Then the people who can't maintain the code use AI and just ship it without any testing.

edit: https://en.wikipedia.org/wiki/Conway%27s_law

[steve1977]

According to Microsoft's top brass, Copilot (one of them) should easily be able to handle QA. So OP's point remains.

[thewhitetulip]

On a contrary note: if LLMs really are that helpful why are QA teams needed? Wouldn't the LLM magically write the best code?

Since LLMs have been shoved down everyone's work schedule, we're seeing more frequent outages. In 2025 2 azure outage. Then aws outage. Last week 2 snowflake outages.

Either LLMs are not the panacea that they're marketed to be or something is deeply wrong in the industry

[ASalazarMX]

It has been an MBA company for most of its life. If I had to draw the line, IMO seems Windows 2000 was the last engineer-driven product, and by then it had already developed predatory habits.

[Datagenerator]

Let's hope for the catastrophic scenario. A world without Microsoft.. no telemetry or backdoors. Please continue on this track to disaster!

[teleportmepls]

> Microsoft has a cultural problem; it went from an "engineers" company to an MBA directed one

I don’t think this is just Microsoft. Few engineers and visionaries that started these big companies are still at the helm.

It’s an opportunity for other companies to take over imo.

[wnevets]

> but this is much more to do with Microsoft very publically firing the QA department[0][1] as a cost savings measure and claiming developers will do their own QA (long before LLMs were on the scene). It started in 2014 and the trickle never stopped.

We know this was the correct move because Microsoft's stock price has gone up tremendously since 2014, those in the c-suite received massive bonuses and the worlds most efficient system for resource allocation has deemed it so.

[ferguess_k]

I think all companies eventually mutate into a MBA company. For MSFT there was a culture from very early that PMs should lead the project instead of engineers. I read in "Showstoppers" that Cutler was very against of the idea and he pushed back. So that means even in the late 80s MSFT was already a MBA-centered company. The only reason that it has not degraded yet, was because it has not achieved the monopoly position. Once it does it started to chew on its success and quickly degraded into a quasi-feudal economic entity.

[boshomi]

The shift from an engineer-led corporation to an MBA-led corporation has brought Boeing close to the brink of collapse.

[miohtama]

At least we get Visual Studio Code for free

[user____name]

There seems to be a lot of internal factionalism that's showing up in the final product. I think this is a chromic disease that flares up every couple of years and is then clamped down on... but for whatever reason the lessons are never learned for long.

[razodactyl]

So essentially, they need to turn quality around or suffer the thousand cuts of death like Intel?

Although. These companies don't "die" - it's more the consumers end up being abandoned in favour of B2B?

[themafia]

No one is blaming LLMs.

Their presence in this situation casts a conspicuous shadow though.

[joe_the_user]

Microsoft has a cultural problem; it went from an "engineers" company to an MBA directed one

Every simplistic analysis of failing company X uses a hackneyed cliche like this. But in the case of MS, this is completely ridiculous. MS has been renowned for shitty software, since day one. Bill Gates won the 90s software battle based on monopoly, connections and "first feature to market" tactics.

If anything, the heyday of MS quality was the mid 2000s, where it was occasionally lauded for producing good things. But it was never an engineers company (that's Boeing or whoever).

[fzeindl]

> I know blaming everything on LLMs is in vogue right now; but this is much more to do with Microsoft very publically firing the QA department.

Yes, yes, "agile" everything...

I remember clicking on a perfectly honest button in Azure Dev Ops (Production) and it told me that the button is completed but the actual functionality will be probably delivered in Sprint XY.

[Night_Thastus]

Microsoft fired their QA because at the end of the day, they are beholden to shareholders. And those shareholders want higher profits. And if you want higher profits, you cut costs.

It's not a culture problem. It's a 'being a business' problem, which unfortunately affects all publicly-traded companies.

[HumblyTossed]

> but this is much more to do with Microsoft very publically firing the QA department[0][1] as a cost savings measure and claiming developers will do their own QA (long before LLMs were on the scene).

I will never ever understand this. Development and QA are two different mindsets. You _can_ do both, but you* can't be great at both.

* There's always exceptions, yes, yes.

[rolandog]

Wholeheartedly agree.

I can't wait until we can live in a better era where we look back with collective disgust at the blatant white-collar crime time period that was ushered by Friedman and Welch.

That, plus the current era, feels to me like a massive dog whistle for people who can't read satirical stories like A Modest Proposal without taking them as instructions.

[SloppyDrive]

I fully believe highly skilled people can get a great benefit from LLM tools; probably not 10x; but enough that its noticeable.

The key thing for me is that it only works when the LLM is used for tasks below the devs skill level; It can speed up somebody good, but it also makes the output of low-skill devs much harder to deal with. The issues are more subtle, the volume is greater, and there is no human reasoning chain to follow when debugging.

So you combine that with a company that has staff in low skill regions, and uses outsourcing, and while there might be some high skill teams that got a speed up, the org is structured in a way that its irrelevant.

[claysmithr]

I think they keyword is "highly skilled." However, not everyone using the LLM will be highly skilled, especially juniors new to the industry.

[bodge5000]

The argument I usually hear is that you only truly get the 10x improvements with <new-model> (right now, Opus 4.5), so they've only had a few months, not years. In a few months, it'll turn out that <new-model> wasn't actually capable of that, but <new-new-model> is, and as that's not been out long its unfair to judge so early. And so the cycle begins anew

[rossdavidh]

Very true. Look at the wikipedia article for "AI Winter" to see how old this cycle truly is...

[adamrezich]

Imagine a world where Microsoft was pushing “Copilot” integration everywhere, just as they are in this one—but the proof was, actually, in the pudding. Windows was categorically improving, without regression, with each subsequent update. Long-standing frustrations with the operating system experience were gradually being ironed out. Parts of the system that were slow, frustrating, convoluted, or all three, were being thoughtfully redesigned without breaking backwards compatibility, and we were watching this all unfold in real time, in awe of the power of “AI”, eyes wide with hope for the future of software, and computing in general.

Think of how dramatically this hypothetical alternate reality differs from the one we live in, and then consider just how galling it is that these people have the nerve to piss on our leg and then tell us it's raining. Things are not getting better. This supposedly-magical new technology isn't observably improving things where it matters most—rather, it's demonstrably hastening the decline of the baseline day-to-day software that we depend upon.

[Telaneo]

The distance between the promise and the reality really is huge. On some level I wished they'd just promise less, because it's not like LLMs compleatly useless. I don't find much use in them, but some clearly do. They do them. But since the entire economy has apparently bet the farm on AI, underpromising isn't really an option, while underdelivering is a problem for future Microslop and co.

[rossdavidh]

Interesting thought experiment. In that alternate reality, their shareholders would probably be shouting "why would you give competitors access to this awesome tool?!"

[csomar]

I guess you haven't tried ZZK-5.6 with Maverick Agent? What prompt did you use? If it doesn't work, you can always try a swarm of agents with model hot-reload and re-spin. That will solve all your problems, write all your code and then make you a cup of coffee.

[heliumtera]

But web people can write css faster so I think it is a net positive?

[rich_sasha]

They weren't great before LLMs either.

Also, it seems from the outside like a dysfunctional organisation, or at least with incentives heavily misaligned with their users. Replace LLMs with a bunch of 10x engineers and it will still be bad in an environment like this.

So not sure how much to blame the LLMs - or in fact how much MS is really using them. Poor souls have to use MS AI tools, I almost feel sorry for them.

[BizarroLand]

They hit peak with Windows 7 and will never have an operating system that good again.

Some flavors of Linux are approaching the Windows 7 peak as well as far as ease of use for newbies, software "just working", and for familiarity for users of other OS's.

Their days as the default OS for most people are numbered unless they pull an incredible heel turn.

[MrBuddyCasino]

Everyone should read "The Bear Case for AI" thread:

"The bear case for AI is that bringing 10x or 100x or 1000x more intelligence to America will not change anything because U.S. institutions are already designed to ignore or waste intelligence and have no idea what to do with any more of it."

https://twitter.com/mmjukic/status/2014255931215716545

[ManlyBread]

I would like to point out that not even a week ago Satya Nadella stated that someone should finally do something really useful with AI because if no one does then they'll lose the social permission to burn all the energy on training and running the models: https://www.pcgamer.com/software/ai/microsoft-ceo-warns-that...

Mr. Nadella, why not lead by example and make Windows the most amazing operating system ever created with the help of Copilot? What's the holdup?

[baka367]

Oh it did help.

Microsoft went all in on do more with less and fired/reorged significant part of the company.

Wouldn’t be surprised if the outage is caused by new team taking something over with near zero documentation while all the tribal knowledge was torched away

[Someone]

Or LLMs weren’t good enough yet years ago, but the growth curve looked so promising that an investment seemed a good idea.

Also: do you have a reference for “a couple years ago Microsoft [made] LLM-assisted coding a big part of their production”?

I know they started investing, mentioning future benefits, but don’t remember them saying their Windows development team (heavily) relying on it.

[felixgallo]

(1) a couple of years ago, LLMs for coding sucked pretty bad.

(2) LLMs are a force multiplier. If you start with a negative number, then your coefficient makes things worse.

(3) Microsoft has never been a place of quality. It's not organized for that, it doesn't have that as its philosophy, and so you should never be surprised that it doesn't deliver that.

[austin-cheney]

Its a misunderstanding of costs. Its the same misunderstanding of thermodynamics applied to dieting. Inputs do not equal outputs, or even scaled outputs. In any dynamic system there are costs to storage and costs to processing. This is how you can increase your calorie intake and still rapidly lose weight, yes, I have lost 30 pounds that way.

In the case of LLMs the only way LLM use becomes profitable is if this condition are achieved:

    processing + storage + input + validation + maintenance < manual output + manual support

If you want to see a 10x savings then multiply the cost of manual output by 10. While LLM profit is achievable in some scenarios a t0x improvement in most scenarios is highly improbable.

[isk517]

People are pointing their fingers at QA, and while that is a big part of it I think the bigger issue is, like you said, them not really caring about some of their core products. Windows 11 seems to exist purely so that they can earn passive ad revenue while vacuuming up user data, Office 365 is now just a pile of mature applications that are slowly getting worse and new applications that are too unfinished to be actively useful.

[serf]

this reasoning is flawed.

wouldn't a for-profit company just balance the workforce for the productivity gained to increase overall profit?

some person is 10x 'more productive' (whatever that means) , let's cut 9 jobs.

Although to your grander point, employment during the LLM-embrace period seems fairly stable.[0]

[0]: https://stockanalysis.com/stocks/msft/employees/

[j1elo]

It's not LLMs. It's returns-driven-development.

[ljm]

Growth at any cost. Once growth is unable to increase the wealth of the shareholders the money has to be diverted from elsewhere, via cuts. Money gotta keep flowing upwards.

[heliumtera]

But the second was always the case, windows and everything else is getting shittier so fast it would require a prompt explanation if we didn't have one.

[pawelduda]

If they used copilot and it was years ago, I'm actually impressed there are no reports of Windows PC's exploding

[dzonga]

exactly my thoughts as well - if LLM really were massive productive booms - then we would see the number of bugs in major software platforms going down - we would see more features - but neither is happening

so yeah we're being sold a bag of air

[gchamonlive]

I think it's naive to believe AI is used primarily for productivity boost. It's used mainly for cost reduction and to increase profits, even if quality and productivity take a hit in the process.

[rk06]

Microsoft is not even using dotnet core and what not, internally. SLT is very hard on adopting AI, but not much on getting results

[steve1977]

If anything, we see a decrease, not an increase.

[Octoth0rpe]

> If LLM's really delivered 10x productivity improvements, as claimed by some, then we should by now be seeing an explosion of productivity out of Microsoft. It's been a couple years, so if it really helps then we should see it by now.

That productivity may not be visible. I think MS's move-everything-to-rust initiate would be one hell of an endorsement if they manage to make visible progress on that in the next couple of years.

[mattgreenrocks]

> That productivity may not be visible.

I'm not sure what your take is, but this reads like goalpost shifting.

If one of the biggest orgs that practically mandates some amount of LLM use cannot surface productivity gains from them after using them for several years, then that speaks volumes.

Reality has a way of showing itself eventually.

[Someone1234]

Microsoft has no "move-everything-to-Rust initiative" and never did. That was a bunch of clickbait created based on the personal comments by a single Microsoft developer.