I work at OpenBCI. It's been great getting to work with Gabe and the Valve team. Can't overstate how unique they are as partners on a project like this. Also cool to see OpenBCI (sortof) in the top 10 today :)
What is the timeframe you imagine for us to see the first consumer BCI-based products?
Gabe seems to talk a lot about "inserting" data (like feelings) into the brain, instead of reading it. Is the technology really there already? Can we reliably read data from the brain (i.e. using it as input for a digital system)? And regarding inserting, what is the coolest thing you've done that you can share with us?
I don't know much about the tech, but I'm curious if neural networks and/or machine learning is used to process data and find connections, correlations, or...cool stuff like that? I just think machine learning is neat man, and bio tech is also very neat. The two together....super neat...haha.
I know the example they put out, a horror game that responds to your fears by reading the data you produce. Is the tech actually at a point were a dedicated team can accurately and consistently identify bio markers correlated with a fear response? Or diffrentiate between fear, anger, happiness, etc?
Has there been any research into using this tech effectively to treat anxiety/depression? Really interested in this focus once "writing" information is possible, especially with more severe mental health issues that aren't related to physical brain deterioration. But it seems (per your comments) that that's still a ways a way, haha. Would be really cool to be able to see a psychatrist and have them treat my anxiety with a few bip bops from an industrial headset, or go home with a prescription program to run once a day, that just sounds like science fiction.
I often wonder about the parallels between reverse engineering games using memory inspection software like 'cheat engine' to trying to reverse engineer the brain using a BCI.
For example if you want to find the memory address for your guns ammo you search for a start value in memory say '30', get all addresses that match that value, fire the gun and then find which of those addresses now have the value 29. Continue the search until you narrow down the memory address to just one. Then you can use that address to query the ammo for a 3rd party program that alerts you that you're low on ammo or even write to the memory address to give yourself more ammo..
Obviously the brain isn't as discrete but I feel like if I could play around with a BCI I could find fun signals for when I'm thinking about 'apples' vs 'oranges' and slowly build up an interface.
Have you been able to use a BCI to detect when you're thinking about something specific?
All the talk of implants makes me wonder what's possible without a direct physical link to the brain. Like using an EEG to measure brain signals and transform them into instructions, images, etc. Here's some research where they use the EEG to train a model that reconstructs what people are seeing in their visual cortex [0]
All the reconstructions are super low resolution, but I wonder how much better they can get? I'd be much more open to brain-computer interfaces if it didn't involve drilling through my skull.
Most if not all of the very interesting signals are in the 100hz - 10Khz range. These signals are also very low voltage so. Add in that the skulls acts as a low pass filter and you end up in a situation where you just will not be able to recover the information needed to decode what the brain is doing at the required resolution.
You can use ECoG which is like EEG but placed on under dura-mater (a protective bag around the brain) but this is also highly invasive. Also the spatial resolution is pretty bad. You can get a hundred or so sensors in a few square centimeters.
Tangentially related, but I wanted to post an article and a paper.
I remember hearing way back around 2010 that Valve was messing around with BCI's. Valve has as recently as 2019 talked about EEG's and gaming [0]. Also recall reading about similar concepts over a decade ago. [1]
I have an OpenBCI headset that I have been messing with. Signal is definitely noisy, but they've been able to get a degree of mind control working, like flying little helicopters.
Doesn't reconstruction like this rely on the decoding mechanism being trained on the original images? This doesn't scale to reconstructing any visual image, only to differentiating between those the decoder has already seen.
It doubt very much that it reconstructs from the visual cortext. That is a non-trivial task with fMRI which has a much higher SNR than EEG.
Let's say you have certain nero indicators which indicate person X may do Y .
Do you we then minority report lock up people.
At the same time , it's going to be amazing for people in a coma or unable to communicate. We could have a fantastic amazing world, where we effectively live forever via some type of matrix like interface.
Worse yet, while the government may be banned from looking at this data intrusively. We're likely to freely give it away to a few large tech companies. Without limits on their use, their power will only continue to grow.
To me the primary issue of our day at this moment, is we have no users bill of rights. We are forced to use the services of large tech companies to participate in modern life yet there are few restrictions on their powers.
It's bad enough when it's our written thoughts. But the idea that it can be my unwritten thoughts too... that scares the heck out of me.
Yes, they will do. People are claiming for other people to be fired, disbarred or imprisoned for stupid tweets. Can you imagine the cancel brigade and the big social media companies in charge of this technology and its consequences?
We're already there on foreign soil with Palantir's monitoring software. You don't need to read people's brains to invasively spy on them and arrest them without traditional probable cause
I think the question is the maximum possible range. If it requires something that's a few centimeters (or less) away from the skull, then a government would either have to force you / make you agree to a scan, or would have to tap data generated by existing consumer devices (like they currently often do with phones/computers).
Is it theoretically plausible that this activity could be read from many feet away, or further?
Eh, we'll have software for detecting and correcting criminal intent before the government gets its act together enough to start arresting people preemptively.
Heh. Remember back in the day when pacemakers were being hacked remotely because admin passwords weren't even changed from the default, on top of outdated wireless protocols? Oh right that's today!
Hardware makers often aren't good at software, not to mention software updates. And even that gets wonky, like when Microsoft update got hijacked.
Point is, if we can't get IoT 100% right, or even 90% right, how can we trust IoT with physical interfaces into our bodies? That's the problem. And then what happens if the company who made your implant goes out of business? What do you do when those updates stop? Look at cellphones, supporting a cellphone for 2 years is too much for most hardware makers, they rather never update it.
Point is, even if the tech is 100% possible, we're way too far from business setups that allow for this to happen.
These are all valid worries, but they all have the same fundamental solutions as current products.
> if we can't get IoT 100% right, or even 90% right, how can we trust IoT with physical interfaces into our bodies?
Trust is a risk-reward calculus for any product, whether it's BCI, the microcontrollers in your car, or the (hopefully uninfected) produce in your supermarket. Many folks will find the BCI value outweighs the risks for some feature set that matters to them.
> And then what happens if the company who made your implant goes out of business? What do you do when those updates stop?
Certain applications will have to be designed and evaluated with longevity or long-term support in mind. You don't need that when talking about cellphones. You do for today's pacemakers.
So, everyone needs to go back and watch the Ghost in the Shell: Stand Alone Complex series. In that world, pretty much everyone has a few enhancements, visual and audio overlay options are pretty much universal. Having your equipment hacked is a super rare, state-actor-level event. But, the story is focused on people who deal with events of that magnitude, so it happens all the time in the plotline.
Updates of any sort to the SW are generally very difficult to do. The certification process for any medical device is hard enough, let alone for implants, let alone for life-critical implants, let alone the recertification process for an already implanted life-critical device. This is why you'll see sonogram machines running XP still, completely disconnected from the larger internet [0].
The questions listed are mostly already considered and have many mitigation strategies per the regulatory agency in charge. There are many other questions that people like the FDA demand answers to.
One thing about the business side is that risk/reward ratio. In medical development, FDA authorization typically occurs at the 10-12 year mark for a product (though it varies widely). Meaning that your start-up only gets the go-ahead to make money after about a decade of investment. That said, once you get to sell, you have an effective monopoly on the market. Hence the costs of new drugs and devices being so insanely high; it's that risk/reward imbalance on the business side.
[0] Another reason why IoT medicine is very difficult, among many.
And that's not even getting into the privacy aspect. I, for one, reserve no excitement whatsoever for a world where we can interface directly with computers. I'm just hoping that unlike smartphones, it remains viably optional.
Even worse, what if you have artificial eyes and one day the manufacturer blocks your vision functionality because you did something they disagree with politically? It used to be your account would get banned, now with cancel culture you can be banned from participating in media and society altogether, tomorrow maybe your ability to participate in your own body's functionality may be subject to the whims of the censors and the morality police
I'm not a huge anime person, but there's one I've watched that seems a bit relevant. In _Accel World_, they wear this thing on their neck called a "Neurolink" and basically are able to connect to things like the Internet or private networks (school, home, hospital) wirelessly with it. I believe it takes over signals coming to and from the brain. But they also have a wired connection mode so they can do that or do a special direct, wired connection with someone they are close to.
The ramifications of access like that are described surprisingly well in the show.
For instance, two close friends (they're in a relationship but young enough I'd just consider them close friends) use the direct wired connection feature because they feel comfortable enough doing so. But at some point one of them decides to use that private access to plant a virus of sorts. While the purpose is irrelevant here, it does show how brain connections, especially between people, can become increasingly vulnerable.
The main plot point of the anime though, is this game application that can be installed to the Neurolink, and basically it has the ability to also wipe away or hide memories.
While what Newell is talking about is still a ways away, it's interesting to consider exactly what we may need to be worried about when the time comes.
There's a better version of that in Ghost in the Shell SAC. You could have a cyber brain and connect to networks which could include websites, devices, other people, and even have a direct wired connection too.
Regarding security, I think people would be really freaked out by the scope of attacks. One infusion pump I worked on this year was succesfully attacked over 200 times.
This sounds like the spaceship problem. Whenever you build a spaceship to take people to another planet, it will be passed by a later ship which was built to go faster.
I don't think that's quite right - it's more that they're a lab and not a product manufacturer, so they can't decide to go mass-market.
Even if they decided that they want to go public with a virtual keyboard/mouse/controller and virtual heads-up display that you use by wearing an electrode net, the current team would not be able to make that pivot. End users won't be debugging Labview sketches, spending weeks training a neural network to recognize virtual keystrokes, or shaving spots on their scalp.
Personally, I expect that the V1 product here is approximately just that: a game controller. A few buttons, a pointer, maybe a little haptic feedback. It would be great for me if it supported text input and output faster than a keyboard and terminal, but I don't think that's likely to happen in an early version.
Even less likely is that we're going to jump straight to The Matrix. Valve needs to admit that, and be happy with a limited version, instead of letting it fizzle out in the lab.
One of the things I wish the most technologically is simply being able to record my thoughts. Sometimes in the middle of a stream of thought going back and putting it on paper, taking notes disrupts the process somewhat. Also obviously translating visual imagination and recollection is more difficult (more for someone that wasn't trained artistically). My memory is imperfect too and I've forgotten a few ideas I thought were amazing because I didn't write them down at the right time.
I, for one, welcome the mind-reading devices.
I'm sure it'll be limited by physical contact so I'm not much worried about some science fiction scenarios. However, the implications for police are indeed worrisome, enough it should warrant specific legislation as soon as possible (i.e. non-consent mind reading should be made a grave crime, and a war crime, as soon as the technology is available)
I thought that it was interesting that Gabe was using The Matrix as a comparison for BCIs since that movie was a warning about what this kind of tech could bring, but he did express a lot of concern around more real-world ramifications around what this technology could bring if it wasn't properly used or secured. What he was describing then reminded me more of the movie Strange Days where people would use neural connectors to record and play back experiences from others, but amplifying the signal too much would result in scrambled neurons.
Also reminds of "Better Than Life" from a cheesy, teen-sci-fi Shadowrun novel. The story starts out assuming Strange Days-style neural recording and playback is a commonplace consumer good. But, then quickly gets into highly edited experiences, generated experiences, and eventually into overwhelming, drug-like experiences.
> the experiences that you will have are things that you will be created and edited for you [..]
> sleep will now become an app that you will run where you say I need this much sleep, I need this much REM
I find this to go in a bizarre awkward direction. We as humans have self regulating capabilities that are to be tapped with some "self work". Is that a bad thing? Why are we trying to change that and create an app to control bodily functions? Our brain is perfectly capable of handling that and has been for millions of years in our ancestors
I think VR could use non-fresnel lenses and better ways to adapt your vision in the headset, ironically the DK1&2 had lenscups which now are found in zero headsets.
I keep my old glasses that are smaller to be able to fit them in my vive, but that makes development inconcievable: Remove terminal/monitor glasses and put on old glasses and then the headset every time I need to test something in VR. Nope I'm not doing that.
Also why are the media business still adding fake laughter and annoying background music to everything they touch?
HP is getting into basic (compared to GabeN's ideas) work in this area with the Omnicept VR headset. It looks like eye tracking, pupillometry sensor, heart rate, and a face camera. Not direct neurotechnology but none the less using data to better understand and adapt to the user's response to VR.
The link below shows up on occasion and is a good reminder that not only do we need to keep up with physical/practical trust but philosophical as well.
"While technically hostile, Adbugs do not directly attack Vault Hunters. Instead, they follow their targeted Vault Hunter around and beam a semi-transparent advertisement at them, which is placed in a random position on the field of vision, obscuring part of the field of view."
That's a recurring thing in Deus Ex: Human Revolution. Outside of the protagonist, most people need a costly antirejection drug for the augmentations that they're societally pressured to have. Because they're otherwise uncompetitive in their work.
Reduced to a life of just using a crowbar and not talking t anyone.
But all jokes aside, it could be brain damage, but drug addiction might be a better analogy. It could be something that takes time for neuroplasticity to repair or correct.
johnnybaptist|5 years ago
Happy to answer any questions people have about OpenBCI https://openbci.com/
farias0|5 years ago
Gabe seems to talk a lot about "inserting" data (like feelings) into the brain, instead of reading it. Is the technology really there already? Can we reliably read data from the brain (i.e. using it as input for a digital system)? And regarding inserting, what is the coolest thing you've done that you can share with us?
darkwolf4999|5 years ago
I know the example they put out, a horror game that responds to your fears by reading the data you produce. Is the tech actually at a point were a dedicated team can accurately and consistently identify bio markers correlated with a fear response? Or diffrentiate between fear, anger, happiness, etc?
Has there been any research into using this tech effectively to treat anxiety/depression? Really interested in this focus once "writing" information is possible, especially with more severe mental health issues that aren't related to physical brain deterioration. But it seems (per your comments) that that's still a ways a way, haha. Would be really cool to be able to see a psychatrist and have them treat my anxiety with a few bip bops from an industrial headset, or go home with a prescription program to run once a day, that just sounds like science fiction.
mentos|5 years ago
For example if you want to find the memory address for your guns ammo you search for a start value in memory say '30', get all addresses that match that value, fire the gun and then find which of those addresses now have the value 29. Continue the search until you narrow down the memory address to just one. Then you can use that address to query the ammo for a 3rd party program that alerts you that you're low on ammo or even write to the memory address to give yourself more ammo..
Obviously the brain isn't as discrete but I feel like if I could play around with a BCI I could find fun signals for when I'm thinking about 'apples' vs 'oranges' and slowly build up an interface.
Have you been able to use a BCI to detect when you're thinking about something specific?
andygmb|5 years ago
bwood|5 years ago
All the reconstructions are super low resolution, but I wonder how much better they can get? I'd be much more open to brain-computer interfaces if it didn't involve drilling through my skull.
[0] https://scitechdaily.com/image-reconstruction-from-human-bra...
bognition|5 years ago
You can use ECoG which is like EEG but placed on under dura-mater (a protective bag around the brain) but this is also highly invasive. Also the spatial resolution is pretty bad. You can get a hundred or so sensors in a few square centimeters.
malwarebytess|5 years ago
I remember hearing way back around 2010 that Valve was messing around with BCI's. Valve has as recently as 2019 talked about EEG's and gaming [0]. Also recall reading about similar concepts over a decade ago. [1]
[0] https://venturebeat.com/2019/03/24/valve-psychologist-explor...
[1] https://vpa.sabanciuniv.edu/phpBB2/vpa_views.php?s=4&serial=...
yters|5 years ago
PH01|5 years ago
It doubt very much that it reconstructs from the visual cortext. That is a non-trivial task with fMRI which has a much higher SNR than EEG.
offtop5|5 years ago
Let's say you have certain nero indicators which indicate person X may do Y .
Do you we then minority report lock up people.
At the same time , it's going to be amazing for people in a coma or unable to communicate. We could have a fantastic amazing world, where we effectively live forever via some type of matrix like interface.
swalsh|5 years ago
To me the primary issue of our day at this moment, is we have no users bill of rights. We are forced to use the services of large tech companies to participate in modern life yet there are few restrictions on their powers.
It's bad enough when it's our written thoughts. But the idea that it can be my unwritten thoughts too... that scares the heck out of me.
cambalache|5 years ago
VRay|5 years ago
meowface|5 years ago
Is it theoretically plausible that this activity could be read from many feet away, or further?
px43|5 years ago
Justsignedup|5 years ago
Hardware makers often aren't good at software, not to mention software updates. And even that gets wonky, like when Microsoft update got hijacked.
Point is, if we can't get IoT 100% right, or even 90% right, how can we trust IoT with physical interfaces into our bodies? That's the problem. And then what happens if the company who made your implant goes out of business? What do you do when those updates stop? Look at cellphones, supporting a cellphone for 2 years is too much for most hardware makers, they rather never update it.
Point is, even if the tech is 100% possible, we're way too far from business setups that allow for this to happen.
a_wild_dandan|5 years ago
> if we can't get IoT 100% right, or even 90% right, how can we trust IoT with physical interfaces into our bodies?
Trust is a risk-reward calculus for any product, whether it's BCI, the microcontrollers in your car, or the (hopefully uninfected) produce in your supermarket. Many folks will find the BCI value outweighs the risks for some feature set that matters to them.
> And then what happens if the company who made your implant goes out of business? What do you do when those updates stop?
Certain applications will have to be designed and evaluated with longevity or long-term support in mind. You don't need that when talking about cellphones. You do for today's pacemakers.
psyc|5 years ago
corysama|5 years ago
Balgair|5 years ago
Updates of any sort to the SW are generally very difficult to do. The certification process for any medical device is hard enough, let alone for implants, let alone for life-critical implants, let alone the recertification process for an already implanted life-critical device. This is why you'll see sonogram machines running XP still, completely disconnected from the larger internet [0].
The questions listed are mostly already considered and have many mitigation strategies per the regulatory agency in charge. There are many other questions that people like the FDA demand answers to.
One thing about the business side is that risk/reward ratio. In medical development, FDA authorization typically occurs at the 10-12 year mark for a product (though it varies widely). Meaning that your start-up only gets the go-ahead to make money after about a decade of investment. That said, once you get to sell, you have an effective monopoly on the market. Hence the costs of new drugs and devices being so insanely high; it's that risk/reward imbalance on the business side.
[0] Another reason why IoT medicine is very difficult, among many.
matz1|5 years ago
Either accept that it can broke or always have a backup plan.
>then what happens if the company who made your implant goes out of business?
The company goes out of business doesn't always mean the product stop working anymore.
If the product stop working then I'll look for removal/replacement.
brundolf|5 years ago
silicon2401|5 years ago
criddell|5 years ago
The theme of CES for the past decade has been making products worse through software.
spike021|5 years ago
The ramifications of access like that are described surprisingly well in the show.
For instance, two close friends (they're in a relationship but young enough I'd just consider them close friends) use the direct wired connection feature because they feel comfortable enough doing so. But at some point one of them decides to use that private access to plant a virus of sorts. While the purpose is irrelevant here, it does show how brain connections, especially between people, can become increasingly vulnerable.
The main plot point of the anime though, is this game application that can be installed to the Neurolink, and basically it has the ability to also wipe away or hide memories.
While what Newell is talking about is still a ways away, it's interesting to consider exactly what we may need to be worried about when the time comes.
accounted|5 years ago
5440|5 years ago
5440|5 years ago
johnnybaptist|5 years ago
VikingCoder|5 years ago
LeifCarrotson|5 years ago
Even if they decided that they want to go public with a virtual keyboard/mouse/controller and virtual heads-up display that you use by wearing an electrode net, the current team would not be able to make that pivot. End users won't be debugging Labview sketches, spending weeks training a neural network to recognize virtual keystrokes, or shaving spots on their scalp.
Personally, I expect that the V1 product here is approximately just that: a game controller. A few buttons, a pointer, maybe a little haptic feedback. It would be great for me if it supported text input and output faster than a keyboard and terminal, but I don't think that's likely to happen in an early version.
Even less likely is that we're going to jump straight to The Matrix. Valve needs to admit that, and be happy with a limited version, instead of letting it fizzle out in the lab.
gnramires|5 years ago
I, for one, welcome the mind-reading devices.
I'm sure it'll be limited by physical contact so I'm not much worried about some science fiction scenarios. However, the implications for police are indeed worrisome, enough it should warrant specific legislation as soon as possible (i.e. non-consent mind reading should be made a grave crime, and a war crime, as soon as the technology is available)
Minor49er|5 years ago
corysama|5 years ago
tartoran|5 years ago
I find this to go in a bizarre awkward direction. We as humans have self regulating capabilities that are to be tapped with some "self work". Is that a bad thing? Why are we trying to change that and create an app to control bodily functions? Our brain is perfectly capable of handling that and has been for millions of years in our ancestors
rowanG077|5 years ago
bullen|5 years ago
I keep my old glasses that are smaller to be able to fit them in my vive, but that makes development inconcievable: Remove terminal/monitor glasses and put on old glasses and then the headset every time I need to test something in VR. Nope I'm not doing that.
Also why are the media business still adding fake laughter and annoying background music to everything they touch?
gen3|5 years ago
pugworthy|5 years ago
https://www8.hp.com/us/en/vr/reverb-g2-vr-headset-omnicept-e...
meee|5 years ago
https://marshallbrain.com/manna
(It explores implants and using them vs. being used by them.)
m463|5 years ago
Valve collects lots of data - that you can't turn off - and doesn't let you opt out.
(see if you can ask them to stop collecting time played in games or "achievements" or whatever)
piracy1|5 years ago
fyver|5 years ago
marmalar|5 years ago
ghostbrainalpha|5 years ago
Old_Paris|5 years ago
Geee|5 years ago
m463|5 years ago
"While technically hostile, Adbugs do not directly attack Vault Hunters. Instead, they follow their targeted Vault Hunter around and beam a semi-transparent advertisement at them, which is placed in a random position on the field of vision, obscuring part of the field of view."
pizza|5 years ago
Seanambers|5 years ago
tpoacher|5 years ago
nabla9|5 years ago
Imagine living only half-life.
redwall_hp|5 years ago
pugworthy|5 years ago
But all jokes aside, it could be brain damage, but drug addiction might be a better analogy. It could be something that takes time for neuroplasticity to repair or correct.