[catelm]

> First, we evaluate, for each voxel, subject and narrative independently, whether the fMRI responses can be predicted from a linear combination of GPT-2’s activations (Fig. 1A). We summarize the precision of this mapping with a brain score M: i.e. the correlation between the true fMRI responses and the fMRI responses linearly predicted, with cross-validation, from GPT-2’s responses to the same narratives (cf. Methods).

Was this cross checked against arbitrary Inputs to GPT-2? I gather, with 1.5 Billion parameters, you can find a representative linear combination for everything.

The Bible Code comes to mind (https://en.wikipedia.org/wiki/Bible_code).

[mjburgess]

They assume linearity. They map their choice of GPT-properties to their choice of (brain blood flow!) properties. They then claim there are correlations, with a few fMRI datasets.

If something serious was on the line, with this type of analysis, you'd be fired.

Reading this it feels like we might as well give up on there being any science any more, tbh. For this to appear in Nature -- it feels like the rubicon has been crossed.

How can we expect the public not to be "anti-vax" (etc.), or otherwise scientifically competent in the basic tennets of modern science (experiment, refutation, peer review) -- if Nature isnt?

[svara]

It's not Nature, it's Scientific Reports. The bar to publication in the two couldn't be more different. Nature is one of the premier high impact journals, Sci. Rep. is a pretty middle of the road somewhat new open access journal.

[feet]

That's a pretty extreme statement about the entire field of scientific research being based off one paper