I do in similar way, connect claude code to litellm router that dispatches model requests to different providers: bedrock, openai, gemini, openrouter and ollama for opensource models. I have special slash command and script that collect information about session, project and observed problems to evaluation dataset. I can re-evaluate prompts and find models that do a job in particular agent faster/cheaper, or use automated prompt optimization to eliminate problems.
No comments yet.