[jychang]

> How hard would it be to systematically evaluate the different quantizations? E.g. on the Aider benchmark that you used in the past?

Very hard. $$$

The benchmarks are not cheap to run. It'll cost a lot to run them for each quant of each model.

[danielhanchen]

Yes sadly very expensive :( Maybe a select few quants could happen - we're still figuring out what is the most economical and most efficient way to benchmark!

[illusive4080]

Roughly how much does it cost to run one of the popular benchmarks? Are we talking $1,000, $10,000, or $100k?