[PTRFRLL]

> We were cautious to only run after each model’s training cutoff dates for the LLM models. That way we could be sure models couldn’t have memorized market outcomes.

[stusmall]

Even if it is after the cut off date wouldn't the models be able to query external sources to get data that could positively impact them? If the returns were smaller I could reasonably believe it but beating the S&P500 returns by 4x+ strains credulity.

[cheeseblubber]

We used the LLMs API and provided custom tools like a stock ticker tool that only gave stock price information for that date of backtest for the model. We did this for news apis, technical indicator apis etc. It took quite a long time to make sure that there weren't any data leakage. The whole process took us about a month or two to build out.

[plufz]

I know very little about how the environment where they run these models look, but surely they have access to different tools like vector embeddings with more current data on various topics?

[endtime]

If they could "see" the future and exploit that they'd probably have much higher returns.

[disconcision]

you can (via the api, or to a lesser degree through the setting in the web client) determine what tools if any a model can use