[lowbatt]

If I wanted to run this locally at somewhat decent speeds, is an RK3588S board (like OrangePi 5) the cheapest option?

[ThatPlayer]

The RK3588 is a bit interesting because of its NPU. You can find models that have been converted to take advantage of that on HuggingFace: https://huggingface.co/models?search=rk3588 .

No clue how performance compares. Not sure it's worth dealing with the lesser software support compared to getting an AMD mini PC and using Vulkan on llama.cpp for standard GGUF models.

[babl-yc]

I'm going to attempt to get it running on the BeagleY-AI https://www.beagleboard.org/boards/beagley-ai

Similar form factor to raspberry pi but with 4 TOPS of performance and enough RAM.

[jm4]

It depends on your idea of decent speeds and what you would use it for. I just tried it on a laptop with an AMD HX 370 running on battery in power save mode and it's not especially impressive, although it runs much better in balanced or performance mode. I gave it the prompt "write a fizzbuzz program in rust" and it took almost a minute and a half. I expect it to be pretty terrible on an SBC. Your best bet is to try it out on the oldest hardware you have and figure out if you can tolerate worse performance.

[lowbatt]

good idea, will test that out

[zipping1549]

Tried with S25+ (SD 8 elite). 0.82tok/s(4B L model). It's barely useful speed but it's pretty impressive either.

[ac29]

RK3588 uses a 7 year old CPU design and OrangePi 5 looks expensive (well over $100).

A used sub-$100 x86 box is going to be much better

[lowbatt]

you're right. For my purposes, I was thinking of something I could use if I wanted to manufacture a new (smallish) product