There's also the availability factor. OpenAI has been known to go down on occasion and without warning. If a product relies on an LLM, I wouldn't feel great about the observed uptime of OpenAI APIs.
FWIW, OpenAI's availability seems to have gotten significantly better since May when we launched with them. I monitor our availability Service Level Objective and we keep needing to increase the success rate because they keep improving things.
This doesn't take away from high availability being a legitimate need to host your own LLM, though.
Another side of availability is that they'll make changes to the model without warning, which alters the results of the prompts you already have written. Developing against their API is developing against a moving target.
phillipcarter|2 years ago
This doesn't take away from high availability being a legitimate need to host your own LLM, though.
lolinder|2 years ago