[tomschwiha]

The "not optimized" self hosted deployment is 3x slower and costs 34x the price using the cheapest GPU / a weak model.

I don't see the point in self hosting unless you deploy a gpu in your own datacenter where you really have control. But that costs usually more for most use cases.

[ToucanLoucan]

> I don't see the point in self hosting unless you deploy a gpu in your own datacenter where you really have control. But that costs usually more for most use cases.

Not wanting to send tons of private data to a company who's foundation is exploiting data it didn't have permission to use?

[Incipient]

Is there actually some scale magic that allows the 34x cost saving (over 100x when you include performance), or is it just insane investment allowing these companies to heavily subsidise cost to gain market share?

[tomschwiha]

Calculating without energy costs: The A10 Gpu itself costs 3200$. With a 3 year usage that is 0,002$ per minute. From the blog post the cost per minute is charged at 0,02$, so a premium of 10x. So with energy if you can load the GPU at minimum 15-20% self hosted becomes cheaper. But you need to take care of your own infrastructure.

With larger purchases the GPU prices also drop so that is the scaling logic.