- LLMs writing code that pros struggle with isn't a real thing (yet), and i've tried them all
- except, weirdly, lisp
- lisp can call tree-sitter now
- if you can write lisp, you can write/refactor most anything
- a far better strategy than comparing MMLU between `gpt-4-1106-preview` and `dolphin-8x7b-v0.1-q4_km` is to run them both (along with others) and have them review each other and make this easy in every tool (from emacs to vscode to zed)
- `llama.cpp` and `ollama` and `lmstudio` et al. are awesome, and the OpenAI protocol support is necessary and useful (for now), but we need a real `protobuf3`/`gRPC`/`connectRPC`/`buf.build`/etc. standard with a good reference implementation in (probably C++, but maybe Rust if Candle or whatever gets fast soon) that does all the model-server shit we did at FAANG and doesn't require the user to remember what 2^n context window is on pain of breakage
- once you have an extensible, open, fast protocol with great docs and trivial linkage courtesy of `buf.build`, ASR (`whisper.cpp`), TTS (probably StyleTTS but I haven't gotten that far yet), and other modalities now win big enough to not be non-starters in JSON
- other people will write better model servers, at least for certain applications, and everyone wins
This is obviously a pile of work, and there will probably be a commercial venture that's semi-adjacent, but the core stuff will be MIT (or maybe Apache 2 or something else that means "do what you want"), but it's coming along.
Right now this is a bunch of code not ready for release and like, a domain name that 404s, but if there's interest we'd be open to firing up a Discord or something. You can email me at `b7r6@pm.me` if you want to rap about it.
benreesman|2 years ago
- LLMs writing code that pros struggle with isn't a real thing (yet), and i've tried them all - except, weirdly, lisp - lisp can call tree-sitter now - if you can write lisp, you can write/refactor most anything - a far better strategy than comparing MMLU between `gpt-4-1106-preview` and `dolphin-8x7b-v0.1-q4_km` is to run them both (along with others) and have them review each other and make this easy in every tool (from emacs to vscode to zed) - `llama.cpp` and `ollama` and `lmstudio` et al. are awesome, and the OpenAI protocol support is necessary and useful (for now), but we need a real `protobuf3`/`gRPC`/`connectRPC`/`buf.build`/etc. standard with a good reference implementation in (probably C++, but maybe Rust if Candle or whatever gets fast soon) that does all the model-server shit we did at FAANG and doesn't require the user to remember what 2^n context window is on pain of breakage - once you have an extensible, open, fast protocol with great docs and trivial linkage courtesy of `buf.build`, ASR (`whisper.cpp`), TTS (probably StyleTTS but I haven't gotten that far yet), and other modalities now win big enough to not be non-starters in JSON - other people will write better model servers, at least for certain applications, and everyone wins
This is obviously a pile of work, and there will probably be a commercial venture that's semi-adjacent, but the core stuff will be MIT (or maybe Apache 2 or something else that means "do what you want"), but it's coming along.
Right now this is a bunch of code not ready for release and like, a domain name that 404s, but if there's interest we'd be open to firing up a Discord or something. You can email me at `b7r6@pm.me` if you want to rap about it.
Appreciate the interest!