[nickreese]

I'm just now moving my main workflows off openai over to local models and I'm starting to find that these smaller models main failure mode is that they will accept edgecases with the goal of being helpful.

Especially in extraction tasks. This appears as inventing data or rationalizing around clear roadblocks.

My biggest hack so far is giving them an out named "edge_case" and telling them it is REALLY helpful if they identify edgecases. Simply renaming "fail_closed" or "dead_end" options to "edge_case" with helpful wording causes qwen models to adhere to their prompting more.

It feels like there are 100s of these small hacks that people have to have discovered... why isn't there a centralized place where people are recording these learnings?

[rotexo]

Can you describe this more? Is “edge_case” a key in the structured output schema?

[nickreese]

Exactly. Using lmstudio you can get structured output like with openai... I have everything in a zod schema then wrote an adapter to both openai and lmstudio instance.

[alach11]

Just curious - are you using Open WebUI or Librechat as a local frontend or are all your workflows just calling the models directly without UI?

[nickreese]

I run lmstudio for ease of use on several mac studios that are fronted by a small token aware router that estimates resource usage on the mac studios.

Lots of optimization left there, but the systems are pinned most of the time so not focused on that at the moment as the gpus are the issue not the queuing.