[Loic]

I think the OpenAI deal to lock wafers was a wonderful coup. OpenAI is more and more losing ground against the regularity[0] of the improvements coming from Anthropic, Google and even the open weights models. By creating a chock point at the hardware level, OpenAI can prevent the competition from increasing their reach because of the lack of hardware.

[0]: For me this is really an important part of working with Claude, the model improves with the time but stay consistent, its "personality" or whatever you want to call it, has been really stable over the past versions, this allows a very smooth transition from version N to N+1.

[bakugo]

Is anyone else deeply perturbed by the realization that a single unprofitable corporation can basically buy out the entire world's supply of computing hardware so nobody else can have it?

How did we get here? What went so wrong?

[makeitdouble]

> unprofitable

I'm assuming you wouldn't see it as fine if the corporation was profitable.

> How did we get here?

We've always been there. Not that it makes it right, but that's an issue that is neither simple to fix nor something most law makers are guaranteed to want to fix in the first place.

Nothing in the rules stops you from cornering most markets, and an international companies with enough money can probably corner specific markets if they'd see a matching ROI.

[zozbot234]

They're simply making a bet that they can put the DRAM dies to more valuable use than any of the existing alternatives, including e.g. average folks playing the latest videogames on their gaming rig. At this kind of scale, they had better be right or they are toast: they have essentially gone all-in on their bet that this whole AI thing is not going to 'pop' anytime soon.

[hodgehog11]

I don't see this working for Google though, since they make their own custom hardware in the form of the TPUs. Unless those designs include components that are also susceptible?

[jandrese]

That was why OpenAI went after the wafers, not the finished products. By buying up the supply of the raw materials they bottleneck everybody, even unrelated fields. It's the kind of move that requires a true asshole to pull off, knowing it will give your company an advantage but screw up life for literally billions of people at the same time.

[frankchn]

TPUs use HBM, which are impacted.

[UncleOxidant]

Even their TPU based systems need RAM.

[bri3d]

Still susceptible, TPUs need DRAM dies just as much as anything else that needs to process data. I think they use some form of HBM, so they basically have to compete alongside the DDR supply chain.

[Grosvenor]

Could this generate pressure to produce less memory hungry models?

[hodgehog11]

There has always been pressure to do so, but there are fundamental bottlenecks in performance when it comes to model size.

What I can think of is that there may be a push toward training for exclusively search-based rewards so that the model isn't required to compress a large proportion of the internet into their weights. But this is likely to be much slower and come with initial performance costs that frontier model developers will not want to incur.

[lofaszvanitt]

Of course and then watch those companies reined in.

[lysace]

Please explain to me like I am five: Why does OpenAI need so much RAM?

2024 production was (according to openai/chatgpt) 120 billion gigabytes. With 8 billion humans that's about 15 GB per person.

[GistNoesis]

What they need is not so much memory but memory bandwidth.

For training, their models have a certain number of memory needed to store the parameters, and this memory is touched for every example of every iteration. Big models have 10^12 (>1T )parameters, and with typical values of 10^3 examples per batch, and 10^6 number of iteration. They need ~10^21 memory accesses per run. And they want to do multiple runs.

DDR5 RAM bandwidth is 100G/s = 10^11, Graphics RAM (HBM) is 1T/s = 10^12. By buying the wafer they get to choose which types of memory they get.

10^21 / 10^12 = 10^9s = 30 years of memory access (just to update the model weights), you need to also add a factor 10^1-10^3 to account for the memory access needed for the model computation)

But the good news is that it parallelize extremely well. If you parallelize you 1T parameters, 10^3 times, your run time is brought down to 10^6 s = 12 days. But you need 10^3 *10^12 = 10^15 Bytes of RAM by run for weight update and 10^18 for computation (your 120 billions gigabytes is 10^20, so not so far off).

Are all these memory access technically required : No if you use other algorithms, but more compute and memory is better if money is not a problem.

Is it strategically good to deprive your concurrents from access to memory : Very short-sighted yes.

It's a textbook cornering of the computing market to prevent the emergence of local models, because customers won't be able to buy the minimal RAM necessary to run the models locally even just the inferencing part (not the training). Basically a war on people where little Timmy won't be able to get a RAM stick to play computer games at Xmas.

[mebassett]

large language models are large and must be loaded into memory to train or to use for inference if we want to keep them fast. older models like gpt3 have around 175 billion parameters. at float32s that comes out to something like 700GB of memory. newer models are even larger. and openai wants to run them as consumer web services.

[daemonologist]

The conspiracy theory (which, to be clear, may be correct) is that they don't actually need so much RAM, but they know they and all their competitors do still need quite a bit of RAM. By buying up all the memory supply they can, for a while, keep everyone else from being able to add compute capacity/grow their business/compete.

[Phelinofist]

> By creating a chock point at the hardware level, OpenAI can prevent the competition from increasing their reach because of the lack of hardware

I already hate OpenAI, you don't have to convince me

[codybontecou]

This became very clear with the outrage, rather than excitement, of forcing users to upgrade to ChatGPT-5 over 4o.

[beAbU]

I'm not too keyed into the economics of this supposed AI bubble, but is this not an unfathomably risky move on OpenAI's part? If this thing actually pops, or a competitor like Google actually pulls ahead and comes out victorious, then OpenAI will sit holding a very expensive bag of expensive but unusable raw materials that they'll have to sell of at a discount?

[hnuser123456]

Sure, but if the price is being inflated by inflated demand, then the suppliers will just build more factories until they hit a new, higher optimal production level, and prices will come back down, and eventually process improvements will lead to price-per-GB resuming its overall downtrend.

[nutjob2]

Memory fabs take billions of dollars and years to build, also the memory business is a tough one where losses are common, so no such relief in sight.

With a bit of luck OpenAI collapses under its own weight sooner than later, otherwise we're screwed for several years.

[malfist]

Micron has said they're not scaling up production. Presumably they're afraid of being left holding the bag when the bubble does pop

[mholm]

Chip factories need years of lead time, and manufacturers might be hesitant to take on new debt in a massive bubble that might pop before they ever see any returns.