[contrast]

I recognise the poster as someone actively working in the field. That’s exactly why it’s interesting that Simon is saying he hasn’t seen the benefits of fine tuning and would like a demo of it working.

Drawing an analogy to the scientific method, he’s not asking for anything more than a published paper he can read.

We don’t expect every scientist and engineer to personally test every theory and method before we grant them permission to ask questions. The world progresses by other people filling in gaps in our knowledge.

[antonvs]

Which field? It’s hard to believe anyone working with AI models for years hasn’t figured out fine tuning.

There are plenty of published papers on the subject.

One possible reason you may not see many side by side comparisons between tuned and untuned models is because the difference can be so dramatic that there’s no point.

I’m not objecting to asking questions, but rather to how the question was phrased as some sort of shortcoming of the world around him, rather than an apparent lack of any meaningful investigation of the topic on his part.

[simonw]

The reason I ask questions like this is that I know that most people are too scared to admit their ignorance... because of the risk that others might post comments like you've posted here!

I'm confident enough in my own reputation that I'll take that risk.

It's the same reason I try to step up in meetings and ask the "stupid questions" - things like "what do we mean by agent here?". There are always people in the room with the same question who are embarrassed to reveal any gaps in their knowledge.

My recommendation to you is to avoid that temptation to belittle others and instead try to engage in conversations like this in good faith.

It sounds like you've seen the evidence that fine-tuning is valuable and effective and have useful information to add to the conversation. So do that instead!

As far as I can tell, for most teams working on most problems fine-tuning is something of a trap: they assume that it's a good solution, spend months trying to get it to work by and get out-performed by their competitor who invested in better prompting and then got to benefit from the latest release of a frontier model.

In this particular case I was trying to do people who promote fine-tuning as a solution a favor - I am extremely confident that the first vendor to provide a useful side-by-side demo will see a great deal of return on that investment, because I know I'm not the only person who wants to see the benefits of fine-tuning shown with more than just a paper with some benchmark scores.