[mackopes]

For some time I have a feeling that Apple actually IS playing the hardware game in the age of AI. Even though they are not actively innovating on the AI software or shipping products with AI, their hardware (especially the unified memory) is great for running large models locally.

You can't get a consumer-grade GPU with enough VRAM to run a large model, but you can do so with macbooks.

I wonder if doubling down on that and shipping devices that let you run third party AI models locally and privately will be their path.

If only they made their unified memory faster as that seems to be the biggest bottleneck regarding LLMs and their tk/s performance.

[ChocolateGod]

> You can't get a consumer-grade GPU with enough VRAM to run a large model, but you can do so with macbooks.

You can if you're willing to trust a modded GPU with leaked firmware from a Chinese backshop

[Firerouge]

Short of flying to China and buying in person, how can an American find/get one of these?

[gmays]

True, but Apple is a consumer hardware company, which requires billions of users at their scale.

We may care about running LLMs locally, but 99% of consumers don't. They want the easiest/cheapest path, which will always be the cloud models. Spending ~$6k (what my M4 Max cost) every N years since models/HW keep improving to be able to run a somewhat decent model locally just isn't a consumer thing. Nonviable for a consumer hardware business at Apple's scale.

[csomar]

This. If we plateau around current SOTA LLM performance and 192/386Gb of memory can run a competitive model, Apple computers could become the new iPhone. They have a unique and unmatched product because of their hardware investment.

Of course nobody knows how this will eventually play out. But people without inside information on what these big organizations have/possess, cannot make such predictions.

[karmakaze]

On a hypothetical 70b q4 model, the Ryzen AI Max+ 395 (128GB memory with 96GB allocated to iGPU) delivers ~2–5 tokens/sec, slightly trailing the M4 Max’s ~3–7 tokens/sec. The next generation for AMD I expect can easily catch up to or surpass the M4 Max.

A pair of MaxSun/Intel Arc B60 48GB GPUs (dual 24GB B580's on one card) for $1200 each also outperforms the M4 Max.

[tyleo]

This isn’t a great point. “A hypothetical model with hypothetical hardware will beat Apple on a hypothetical timeline.”

The tangible hardware you point out is $2,400 for two niche-specific components vs the Apple hardware which benefits more general use cases.

[insane_dreamer]

> A pair of MaxSun/Intel Arc B60 48GB GPUs (dual 24GB B580's on one card)

please point me to the laptop with these

[orbifold]

I think it is a given that they are aiming for a fully custom training cluster with custom training chips and inference hardware. That would align well with their abilities and actually isn't too hard to pull off for them given that they have very decent processors, GPUs and NPUs already.

[vonneumannstan]

>I think it is a given that they are aiming for a fully custom training cluster with custom training chips and inference hardware.

It is? I haven't seen anything about this.

[billbrown]

They're working—almost done—on a CUDA backend for their Apple Silicon framework:

https://github.com/ml-explore/mlx/pull/1983

[stefan_]

Memory is not in any way or shape some sort of crucial advantage, you were just tricked into thinking that because it's used for market segmentation and nobody would slaughter their datacenter profits cash cow. The inference and god forbid training on consumer Apple hardware is terrible and behind.

[mackopes]

Show me another consumer hardware that handles inference and/or training better. How many RTX5090s would you need?