MindEye2: Shared-Subject Models Enable fMRI-to-Image with 1 Hour of Data

[thorum]

From the article,

> The present work demonstrates that it is now practical for patients to undergo a single MRI scanning session and produce enough data to perform high-quality reconstructions of their visual perception.

> Such image reconstructions from brain activity are expected to be systematically distorted due to factors including mental state, neurological conditions, etc.

> This could potentially enable novel clinical diagnosis and assessment approaches, including applications for improved locked-in (pseudocoma) patient communication (Monti et al., 2010) and brain-computer interfaces if adapted to real-time analysis (Wallace et al., 2022) or non-fMRI neu-roimaging modalities.

> As technology continues to improve, we note it is important that brain data be carefully protected and companies collecting such data be transparent with their use.

[flurb]

While decoding dreams, as some here suggested, would certainly be an intriguing venue of research, I believe, decoding the minds eye of people who lack the minds eye would be an even more interesting topic. How accurate are their "view", for a lack of a better word? Would there even be an image to capture?

[kordlessagain]

It's an interesting set of questions to explore. Having thought about this for many years now (since Blake Ross' post on Aphantasia) my hypothesis is that there wouldn't be any decoding because there is no attention on a visual based internal mind object. As a comparative, if we could decode audio from a person's mind and then you took someone who didn't do echoic recall, then you would expect to "hear" static. The percentage of the population who doesn't do echoic recall is, in some references, estimated to be about 60%.

[monkpit]

The fact that this is possible is so bizarre to me. Impressive work.

[wiz21c]

What about doing the opposite: get an image and stimulate the brain to perceive it ?

[rl3]

For that you’ll need a first class airline ticket with Tom Hardy and Leo.

[GaggiX]

Looking at the result (figure 4), the model looks more like a good classifier than an image retriever, the generated images have the same subject as the original but are very different, even the colors of the subjects.

[teaearlgraycold]

Exactly. The whole image generation bit, at this stage, is noise. But you could easily describe what someone is looking at. Even that is very interesting. And perhaps this will scale up to get high quality, accurate, outputs.

[srslyskptcl]

Why does anyone think this type of scientific inquiry should be pursued at all.

[extheat]

Same reason we study and do research in biology and fields in general. Understanding more about biology, whether in neuroscience or elsewhere, is the fundamental backbone of medicine. Do you want to live in a world where we don't pursue scientific research into things because they seem scary to you? Does it not concern you us building AI smarter than humans, while knowing little to nothing about our own brains? Before we start fading in relevance, we ought to understand how our own bodies (and minds) work! So in fact I think the exact opposite. We should be doing much more work in understanding our own biology (as long as it's ethical).

[teaearlgraycold]

I definitely believe that this technology will get misused. But I expect the researchers are thinking of helpful medical applications. Perhaps to aid people with locked-in syndrome?

Edit: More interestingly, it would be amazing if this could show us someone’s perception rather than what they’re looking at. That would be tremendously useful to therapists.

[reliablereason]

Cause it has huge medical implications?

[ilaksh]

I hope someone will do a similar experiment with visualization (imagination) only. Some people are better at this than others so maybe focus on those people who easily create vivid images in their mind's eye.

Or maybe text.

[dewarrn1]

It's remarkable that this works, and it's equally remarkable that the authors succeeded using a (deliberately) naive approach with the fMRI data. I was just teaching the visual system to students over the last few weeks: cognitive neuroscience/neuropsychology can tell us a lot about what brain systems should be involved here. It's such a tempting research domain to dive into using a more sophisticated approach, but then again, maybe the authors' simpler data-driven method is a strength.

[shrubble]

So here is a question: what would happen if this process faced a person who was specifically hostile to it? Could you "trick" the process by intensely thinking of, for instance, a yellow ball while viewing a cat picture, as one example?

[keenmaster]

I assume the two would be discernible from each other in your brain data to a sufficiently advanced AI. One is your visual stream and one is your visual imagination. On top of that would be signatures of intention to deceive, which are also detectable in theory :)

[rossant]

From the article:

> fMRI is extremely sensitive to movement and requires subjects to comply with the task: decoding is easily resisted by slightly moving one’s head or thinking about unrelated information (Tang et al., 2023)

[ukuina]

This is insane! If capture latency is reduced, can this be used to visualize dreams?

[boardwaalk]

As someone with damage to my visual cortex with a very specific effect I’m very curious what this would come up with for me. Not something useful on its own, but more understanding means more possibilities for treatments.

[nunodonato]

aphantasia?

[artninja1988]

Very cool work. Can something similar be done for audio?

[GoblinSlayer]

https://www.nytimes.com/2023/08/15/science/music-brain-pink-...

[swagatkonchada]

very exciting tech. also very scary, if we imagine how authority can misuse this.

[mt_]

Impressive