[thewataccount]

From my experience "single prompt classification" isn't as simple as "type in sentence and it works" in practice. But you can use some methods to massively improve it's consistency/output.

I cannot recommend guidance enough. You can use shockingly small Llama models for some tasks with guidance while only actually generating a handful of tokens.

You should highly consider some form of guidance/logit bias for classification especially if you have a known set of classes. This will ensure you get it in the format that you want, with the correct classes that you want.

Keep in mind LLMs perform much better with COT. So you make it explain what the text/image is, then explain the possible classifications, then list its final decision. Again guidance can ensure it follows the correct format to do this.

LLM's still massively benefit from finetuning, especially if you want too classify it in a particular format. Notebook tags vs SFW/NFSW vs important subjects, etc. Existing alignment can sometimes mess with some of these classifications too which finetuning helps smooth out.

[rckrd]

We use a similar trick and expose it via an API. Much easier to parse when you can guarantee the shape of the output

[0] https://thiggle.com/

[IanCal]

Last time I used guidance it didn't work even for the examples in the repo, has it matured?

[thewataccount]

Eh I had some issues with microsoft's "guidance", eventually got it working with an older version. They haven't updated it in a month too.

lmql might be a decent alternative.

Any form of logit bias should work though.