[ivan_gammel]

More expensive than what? How much equivalent low latency inference costs today?

I think you completely miss the UX point here. In 1997 CRT screens were mainstream, LCD was in the early stage, phones had antennas. In 2007 an iPhone with LCD touch screen changed the UX of computing forever. This tech that we see today is a precursor of technology that will dominate tomorrow. Today local inference is painful and expensive, it consumes a lot of energy. NPUs/GPUs solve nothing here, and they will always be less effective than hardwired models - by design. So only question is, when the consumer performance expectation for open-weight models will cross the price curve of specialized chips. It may happen earlier than for generic NPUs.