[porttipasi]

That's a fair point. You're right that without seeing the strategy, you can't fully disentangle what drives the differences.

But the strategy itself isn't really the point. Since every model gets the exact same prompt and the exact same market data, the only variable is the model. So relative performance differences are real regardless of what the strategy contains. If Model A consistently outperforms Model B under identical conditions, that tells you something meaningful about the model.

And honestly, that blend of prompt adherence and decision quality is how people actually use LLMs in practice. You give it instructions and context, and you care about the result.

You're right though that the strategy being private limits what outsiders can evaluate. It's something I'm thinking about.

[smeeth]

> Model A consistently outperforms Model B under identical conditions, that tells you something meaningful about the model.

Not really! Sorry to harp on this, but there are two ways one model could outperform another:

1) It adheres to your strategy better

2) It improvises

If the prompt was "maximize money, here's inspiration" improvising is fine. If the prompt was "implement the strategy," improvising is failure.

Right now you have a leaderboard; you don’t yet have a benchmark, because you can’t tell whether high P&L reflects correctness.

[porttipasi]

To be more specific: the prompt defines a trading philosophy and tells models what to look for in the charts. But the actual read and the decision is entirely on the model. Using your framing — it's closer to "here's inspiration, now maximize money" than "implement this exact strategy." Which means improvisation within that framework is exactly what's being measured.

But yeah, it's closer to a leaderboard right now.