[roykishony]

Thanks much for these thoughtful comments and ideas.

I can’t but fully agree: pre-registered hypothesis is the only way to fully guard against bad science. This in essence is what the FDA is doing for clinical trials too. And btw lowering the traditional and outdated 0.05 cutoff is also critical imo.

Now, say we are in a utopian world where all science is pre-registered. Why can’t we imagine AI being part of the process that creates the hypotheses to be registered? And why can’t we imagine it also being part of the process that analyzes the data once it’s collected? And in fact, maybe it can even be part of the process that help collects the data itself?

To me, neither if we are in such a utopian world, nor in the far-from-utopian current scientific world, there is ultimately no fundamental tradeoff between using AI in science and adhering to fundamental scientific values. Our purpose with data-to-paper is to demonstrate and to provide tools to harness AI to speed up scientific discovery while enhancing the values of traceability and transparency and make our scientific output much more traceable and understandable and verifiable.

As of the question of novelty: indeed, research on existing public datasets which we have currently done cannot be too novel. Though scientists can also use data-to-paper with their own fascinating original data. It might help in some aspects of the analysis, certainly help them keep track of what they are doing and how to report it transparently. Ultimately I hope that such co-piloting deployment will allow us delegating more straight forward tasks to the AI and letting us human scientists to engage in higher level thinking and higher level conceptualization.

[uniqueuid]

True, we seem to have a pretty similar perspective after all.

My concern is an ecological one within science, and your argument addresses the frontier of scientific methods.

I am sure both are compatible. One interesting question is what instruments are suitable to reduce negative externalities from bad actors. Pre-registration works, but is limited to few fields where the stakes are high. We will probably similarly see a staggered approach with more restrictive methods in some fields and less restrictive ones in others.

That said, there remain many problems to think about: E.g. what happens to meta-analyses if the majority of findings comes from the same mechanism? Will humans be able to resist the pull of easy AI suggestions and instead think hard where they should? Are there sensible mechanisms for enforcing transparency? Will these trends bring us back to a world in which trust was only based on prestige of known names?

Interesting times, certainly.