top | item 38609106

(no title)

weekay | 2 years ago

If I were to self host an open source model like mistral or llama , are there options similar to this as an api gateway to proxy and authenticate , create api keys , monitor spends by api etc .,? How are people running open source LLM”s in production ? Thanks

discuss

donovan-so|2 years ago

We do support self-hosted models, as long as they're exposed as an API. Would this endpoint work for you? https://github.com/bricks-cloud/BricksLLM?tab=readme-ov-file...