Written By | Torch Media Africa
For the first time, a single implant on the surface of the brain has read a paralysed man's words AND his hand gestures at the very same instant.
Search
SAN FRANCISCO
He could not speak. He could not lift his arm. For years the only way he could answer a question was to spell it out, letter by exhausting letter, with his eyes.
Then, on a screen in a laboratory at the University of California, San Francisco, a digital version of him raised a hand and waved — and said hello at the same moment.
No hand moved. No voice sounded. The only thing that happened was a flicker of electricity across the surface of his brain, caught by a postage-stamp strip of 253 electrodes and translated, in under a second, into a body he no longer controls.
That moment, published today in the journal Nature Neuroscience, is being described by researchers as the first time any brain-computer interface has decoded speech and body language simultaneously — the first time a machine has been handed the full, messy, gesturing, shrugging reality of how human beings actually talk to each other.
"Conversation is about much more than the words being spoken," said Dr Edward Chang, the UCSF professor of neurological surgery who led the work. "It's a multilayered, dynamic process involving the whole motor cortex. This proof-of-concept shows us it's possible for a BCI to restore some of this freedom and flexibility."
THE PROBLEM NOBODY HAD SOLVED
Brain-computer interfaces, or BCIs, are no longer science fiction. Over the past three years, implants have let paralysed people type, move a cursor, steer a robotic arm, and — in headline-grabbing demonstrations — produce synthetic speech from nothing but the intention to talk.
But every one of those systems had the same blind spot. They did one thing.
"Most of these studies have focused on restoring one of these functions at a time," said Samantha Brosler, the PhD student in bioengineering at UCSF who is first author on the new paper.
And here is the cruel irony: for people who have lost speech to a brainstem stroke or to motor neurone disease, the gestures often go too. Amyotrophic lateral sclerosis, or ALS, and brainstem strokes can strip away both the voice and the body in one blow. A device that restores only words gives back a fraction of what was taken.
Think about the last real conversation you had. Count the nods. The shrug that meant "I'm not sure". The thumbs-up you gave instead of saying yes. The hand that went up when you wanted to interrupt. Strip all of that out and what is left is not conversation — it is dictation.
The current standard of care for these patients is eye-tracking technology, which turns eye movements into text and then into synthetic speech. It works. It is also, as the National Institutes of Health bluntly put it in announcing the study, slow, limited in the forms of expression it allows, and physically exhausting to use.
253 ELECTRODES AND A VERY BIG BET
Chang's laboratory has spent years building towards this. In earlier work, the team implanted a thin sheet of sensors — an electrocorticography array, or ECoG — onto the surface of the brain's cortex, and used machine-learning models called decoders to drive a digital head and face.
The new study takes the head and gives it a body.
The implant itself is an array of 253 electrodes laid on the surface of the sensorimotor cortex — not pushed into the brain tissue, but resting on it. That placement matters enormously.
"It covers a fairly large surface area of the sensorimotor cortex, so it can capture signals related to both speech and body movements," Brosler explained.
In other words: one implant, one operation, one strip of sensors — but wide enough to eavesdrop on the parts of the brain that handle the mouth and the parts that handle the arms at the same time.
Three volunteers, identified in the paper only by the study codes Bravo-1r, Bravo-3 and Bravo-6, took part. All three had paralysis of the vocal tract and limbs. In the acknowledgements, the authors thank them "for their tireless dedication and generous commitment to this research and without whom this work would not be possible."
One participant had impaired speech and movement following a brainstem stroke. Researchers asked him to silently attempt five phrases, and to attempt to wave, nod, shake his hands and clap.
A second participant had ALS, which had progressively taken his speech and impaired his movement. At the time of the study he could still type on a computer and phone. The team asked him to vocalise ten phrases and to imagine making ten gestures — a shrug, a thumbs-up — without moving anything except his facial muscles.
Then came the part that mattered: the researchers asked them to do both at once. To pair the word "hello" with a wave. To pair "yes" with a nod.
THE SURPRISE HIDING IN THE DATA
Here is where the science got genuinely strange — and where, according to the researchers, the breakthrough actually lives.
For years, the working assumption was simple arithmetic. If you know what the brain looks like when a person tries to speak, and you know what it looks like when a person tries to wave, then a person doing both should produce something close to the two patterns stacked on top of one another. Speech plus gesture equals speech-plus-gesture.
It doesn't.
Chang and his colleagues found that multimodal communication is more than the sum of its parts. Yes, there was overlap between the signals. But the picture painted by data from simultaneous expressions was, in the NIH's phrase, "very different" from the pictures created by speech or gestures on their own.
This is not a technical footnote. It explains a frustrating and well-documented failure in earlier BCI research, where asking a user to gesture while speaking seemed to actively degrade the quality of the decoded speech. The decoders were not broken. They were simply looking for the wrong thing — trying to match live, blended brain activity against templates learned from neat, isolated tasks.
So the team changed the training. They fed the decoders data captured while participants performed speech and gesture together, rather than separately.
Performance went up. Mistakes went down.
"Training the system using both speech and gestures together improved performance and reduced mistakes," the study's publisher summary states.
THE NUMBER: 100 PER CENT
In two of the three participants, the researchers then ran parallel speech and gesture decoders to animate a full-body avatar in real time. Participants were asked to attempt specific gestures, to respond to conversational prompts, or both at once.
In the conversational tasks, the system accurately decoded both speech and gesture in both participants. And one participant achieved median accuracies of 100 per cent — for speech and for gesture — across three conversational blocks.
One hundred per cent. In live conversation. From electricity on the surface of a brain.
It is worth being precise about what that figure is and is not. It is a median across three blocks of a conversational task, in one person, using a defined set of phrases and gestures. It is not a claim that the device is flawless, and the authors do not make one.
But it is a startling result for a technology that, three years ago, was celebrated for producing recognisable words at all.
WHAT THE EXPERTS SAY
The reaction from outside the laboratory has been notably warm — and notably forward-looking.
"This gives us hope for neural prosthesis that will enter day-to-day usage," said Christian Herff, a computational neuroscientist at Maastricht University in the Netherlands, who was not involved in the work.
That phrase — day-to-day usage — is the one that matters. The BCI field has never been short of dazzling demonstrations. What it has been short of is devices a person can actually live with.
Dr Debara Tucci, director of the NIH's National Institute on Deafness and Other Communication Disorders, which funded the research, framed it in terms of what patients lose and what they might get back.
"These promising results give me hope that in the future, patients with severe paralysis will be able to recapture the holistic nature of human communication," she said.
Brosler herself was careful to keep expectations grounded. The system, she said, is a proof of concept, but one that aims to be "a step towards more multifunctional BCIs that can capture more of the expressivity and communication that is possible" than existing technology allows.
THE CAVEATS THAT DESERVE A HEADLINE OF THEIR OWN
Tabloid rules say you bury the bad news. This story is too important for that, so here it is up front.
First, the numbers are tiny. Three participants contributed data; two drove the avatar. In medicine, three is not a population — it is a set of case studies. The authors say plainly that testing in more participants, and with larger sets of words and gestures, will be needed before the technology can be used in practice.
Second, the vocabulary is small. We are talking about five to ten phrases and up to ten gestures, not open-ended conversation. A wave, a nod, a clap, a shrug, a thumbs-up. Expressive — but a long way from the infinite improvisation of ordinary speech.
Third, and most practically: the thing is plugged in. The BCI used in this study was a wired system, connecting implanted sensors to external processing units sitting outside the body. Nobody is walking around with this. Chang says his team will soon test a fully implantable, wireless version with far better prospects for long-term use — but that test has not happened yet.
There is also a small discrepancy worth flagging, because different outlets have reported different figures. The NIH release describes machine learning being used to decipher brain activity in three patients, with two participants' thoughts translated into avatar commands. Nature's own news coverage describes the avatar work as involving two participants. The publisher's summary splits the difference: neural activity recorded from three participants, with the full-body avatar driven in two. All three accounts are describing the same study — the difference is which stage of the work is being counted.
WHAT THIS MEANS FOR YOU
Unless you or someone you love is living with ALS or the aftermath of a brainstem stroke, you might reasonably file this under "interesting, not urgent". That would be a mistake, for three reasons.
The first is scale. ALS alone affects hundreds of thousands of people worldwide at any given moment, and strokes are among the leading causes of long-term disability on the planet. Locked-in syndrome — full awareness inside a body that will not respond — is one of the most feared outcomes in all of medicine. Anything that widens the channel out of that silence matters.
The second is the shape of the technology. Notice what is doing the heavy lifting here. It is not a new electrode or a new surgical technique. It is the machine-learning insight that you have to train the decoder on the messy combined behaviour rather than on tidy isolated components. That is a lesson that will travel — to prosthetic limbs, to rehabilitation devices, to any system trying to read a human being who is doing more than one thing at once. Which is to say: any system trying to read a human being.
The third is the question this raises and does not answer. A device that reads your intended words and your intended gestures is, unavoidably, a device that reads intention. The field is already grappling with what happens when neural data becomes a category of personal information — Nature has covered brain implants shipping with password protection. As these systems move from laboratory to living room, who owns the signal will stop being a philosophical question and start being a legal one.
For now, there is no product, no price and no timeline. This is a research result, not a purchase. The realistic path runs through a wireless implant, then larger trials, then regulators — and that is years, not months.
THE LAST WORD
Strip away the electrode counts and the accuracy medians and what is left is a very old idea dressed in very new hardware: that a person is more than the words they can produce.
For decades, the goal of assistive communication technology has been to get the message out. Faster typing. Cleaner synthesis. More words per minute. It is a reasonable goal, and it has helped enormous numbers of people.
But it has also quietly accepted a lossy compression of what a person is. It has treated the shrug, the nod, the open hand as decoration rather than content — the packaging around the real message, disposable once the words arrive.
Chang's team started from the opposite assumption. They treated the gesture as part of the sentence. And when they built a decoder that took that assumption seriously, the decoder got better at everything, including the words.
There is a long road ahead — more volunteers, bigger vocabularies, a version of the device that does not trail a cable. The researchers are the first to say so.
But somewhere in a laboratory in San Francisco, a man who has not moved his arm in years watched a screen, and waved, and said hello, and the two arrived together, the way they are supposed to.
That is not a small thing. That is a conversation.
