Projecting the mind's eye

Professor Jack Gallant, a neuroscientist working in a lab at Berkeley, has released a truly remarkable research paper. He and his team have mapped the brain's visual activity and can construct images based on the results of their research. Above is a demonstration of their work. You can read more about it at Scientists use brain imaging to reveal the movies in our mind.

I'll try to explain what I think that they've done, but be forewarned that after reading the article several times my grasp of it is still pretty foggy.

Scientists use brain imaging to reveal the movies in our mindPreviously, Gallant and fellow researchers recorded brain activity in the visual cortex while a subject viewed black-and-white photographs. They then built a computational model that enabled them to predict with overwhelming accuracy which picture the subject was looking at.

In their latest experiment, researchers say they have solved a much more difficult problem by actually decoding brain signals generated by moving pictures.

“Our natural visual experience is like watching a movie,” said Shinji Nishimoto, lead author of the study and a post-doctoral researcher in Gallant’s lab. “In order for this technology to have wide applicability, we must understand how the brain processes these dynamic visual experiences.”

They placed their test subjects in an fMRI and recorded their brain's activities as they watched sets of movie trailers. By dividing the brain into cubes, and measuring the blood flow through each of those cubes they were able to map brain activity. They then fed that data into a computer program that associated the images, second by second, with the measured brain activity.

Once this set of data was compiled they then fed 18 million seconds of YouTube videos into the program and predicted what brain activity would correlate to the images.

I think, but am not sure, that what they next did was feed a new video into the program where it selected a number of similar clips it had stored, and the merged them into a composite of what the brain activity would indicate the person had seen. That is, the reconstructions aren't from actual brain scans, but rather from predictions of how a real brain would behave if it had watched the clips.

Fascinating stuff, although, if I understood what they had done in the experiment, obviously a step is missing -- taking actual measured signals and converting them into the resultant images. They hope to be able to eventually use this process on comatose people, and perhaps even for people dreaming.

Here is a link to the Gallant Lab at UC Berkley website which has more information for those who want to dig into this deeper.