Ask HN: Who's running local AI workstations in 2026?

[Blue_Cosma]

After three years working on private LLM infrastructure, I still can't pin down who and how big the market is.

The ecosystem has matured: DGX Spark, high-end Mac Studios, AMD Strix Halo, upcoming DGX Station. Models are getting smaller and more efficient. Inference engines (llama.cpp, vLLM, SGLang) and frontends (Ollama, LMStudio, Jan) have made local deployment accessible. Yet I keep meeting more people researching this than actually deploying it.

For those running local inference: - What's your setup and use case? - Is it personal or shared across a team? - What's the real driver — privacy, regulation, latency, cost, tinkering?

I'm skeptical on cost arguments (cloud inference scales better, plus API subsidies, for now at least!), but curious if I'm missing something.

What would make local AI actually worth it for you?

[delaminator]

I have a 3090 24Gb Twin Xenon 64Gb RAM sat on a machine in our server room.

I do local AI with Qwen, Whisper and another I can't remember right now.

These are all QWEN:

We do AI Invoice OCR - PDF -> Image -> Excel. Works much better than other solutions because it has invoice context so it looks for particular data to extract and ignores others. Why local? I proved it worked, no need to send our data outside for processing and it works,

We deal with photos of food packaging - I do a "photograph ingredients list and check them against our expected ingredients" - downside is it takes 2 mins per photo, I might actually push this one outside.

Ingredients classifier - is it animal (if so what species), vegetarian, vegan, halal, kosher, alcoholic, is nut based, peanuts and more - simply no need to send it outside.

I've got a Linux chatbot helper on the "test this" pile with Qwen Coder - not evaluated it but the idea will be "type command, get it wrong, ask Qwen for the answer" - I use Claude for this but it seems a bit heavy weight and I'm curious.

tbh some of it is solution hunting - we spent $1000 on the kit to evaluate if it was worth it so I try and get some value out of it.

But it is slow, 3 hours for a recent task that took Claude API 2 minutes.

My favourite use is Whisper. I voice->text almost all of my typing now.

I've also bought a Nvidia Orin Nano but I haven't set it up yet - I want to run Whisper in the car to take voice dictation as I drive.

[andy99]

Just bought a Strix Halo (framework desktop), waffled a long time between that and a Mac Studo but I got tired of waiting for the M5 and don’t really like Apple.

I work with ML professionally, almost all in cloud, I just wanted something “off grid” and unmetered, and needed a computer anyway so decided to pay a bit more and get the one I want. It’s “personal” in that it’s exclusively for me, but I have a business and bought it for that.

Still figuring out the best software, so far it looks like llama.cpp with Vulcan though I have a lot of experimenting to do and don’t currently find it optimal for what I want.

[Blue_Cosma]

Thanks a lot for sharing. Haven't tested Strix Halo myself. Did you consider DGX Spark as well?

What is your target use case? Curious what feels suboptimal about llama.cpp + Vulkan so far.

[01092026]

Well, Mac chips are badass for training / inference - super underrated. I mean, I've literally run epochs on cloud Nvidia GPU Servers...compared to running them locally (M chip) - and look, not trying to burn any houses down but...eh...Apple does really really well.

The good news for you, you can chain like a bunch / couple of them together and run the largest open source models around. But extremely expensive route - but probably the easiest and smoothest way.

If you're planning on running this on Apple - you can do some stuff with Metal directly...in PyTorch it's 'mcu' if I remember?

I think your llama.cpp route is good - I wouldn't go the Ollama route - I mean great to start, but IMHO: get the models directly, learn the layers and how the heads work as best as you can, make an effort to understand what's going on - well you don't have to, but, I think the models appreciate the effort - respect goes far.

[01092026]

You asked us...well, first tell us what's your real driver? You have three years on local infrastructure? What does that even mean - you're running Ollama Llama_70b for 3 years?

Whats your stack?

And none of that hardware can run larger models, smaller tiny ones, or highly quantized versions of larger ones sure. Or do you have something important to say?

[Blue_Cosma]

Our main driver and hypothesis was to work with regulated industry. We worked with a few large enterprise clients in defence and industry for R&D and IP use cases mostly.

Our stack changes per project, adapting to client needs and infra: Llama 70B on a Mac Studio M1 with Ollama in 2024, vLLM on 4xH100 private cloud for larger deployments. Most recently, we've been working on a custom workstation with 2x RTX PRO 6000 Blackwell Max-Q + 1.1TB DDR5 to run larger models locally using SGLang and KTransformers.

The question isn't rhetorical, I'm trying to understand if the demand we see in regulated sectors is the whole market or if there's broader adoption I'm missing.