[grbsh]

Ooh, I really like the idea about deciding whether to use the big or small model based on task specificity.

[tough]

You might like https://pypi.org/project/llm-predictive-router/

[anerli]

Oh this is interesting. In our case we are being very specific about which types of prompts go where, so the planner essentially creates prompts that will be executed by Moondream, instead of trying to route prompts generally to the appropriate model. The types of requests that our planner agent vs Moondream can handle are fundamentally different for our use case.