[rosstaylor90]

Thanks for feedback! Yes, we’re looking to improve quality in the coming months. Couple of notes:

- The initial use of data is distillation so we’re less bound by question quality (anything that evinces output diversity is good).

- But moving onto RL, we’ll need stronger quality. We have much better things planned both on data filtering and verification!

- Surprisingly, a lot of ML datasets actually look like this when you look under hood. We’re hoping having more eyeballs on it will help improve quality in long run over less transparent status quo!

[eternityforest]

I still don't understand why all the datasets have so many general knowledge questions and so much math, when so few people can do any of that stuff.

It makes sense for ASI research I suppose, but why are we trying to teach small models to do stuff almost no humans even try to do?

What happens if you train them with RAG context in the prompts and calculator calls in the CoT?

[rosstaylor90]

Many math questions are easy to verify, and it's a classic benchmark for reasoning -> so it's a good hill to climb.

I agree with your meta-point that better benchmarks testing more types of task would be good!