Written By | Torch Media Africa
A bombshell audit finds a yawning gap between 'cleared' and 'proven', and the AI tools already scanning your X-rays are at the heart of it
Search
TECH & SCIENCE
Picture this. A hospital rolls out a shiny new artificial intelligence tool that reads your scan, flags a possible problem and nudges your doctor toward a decision. It carries an official stamp of approval from the US Food and Drug Administration. Surely that means it has been proven to help patients?
Not necessarily. Not even close, according to a new audit that is rattling the medical technology world. Researchers from the University of Toronto and MIT combed through 1,357 FDA-cleared or approved AI medical devices and found that just three, a mere 0.2 per cent, had ever been evaluated for patient-centred outcomes such as death, hospital readmission or quality of life.
The study, published in PLOS Digital Health on 19 August 2026, has been echoing around the tech and health worlds ever since, and it is back in the headlines this week as AI newsletters and policy watchers pick over its implications while lawmakers and AI firms wrestle with how to keep the technology safe.
The Numbers That Stopped Doctors In Their Tracks
The audit, led by Rawan Abulibdeh and colleagues, makes for uncomfortable reading. Of the 1,357 devices examined, only 34, or 2.5 per cent, were linked to a registered clinical trial. Just 12, or 0.9 per cent, had posted results on ClinicalTrials.gov, and only 12 had peer-reviewed publications to show for it.
Among the 1,059 radiology and imaging tools in the dataset, as summarised in a roundup of the work, just three had registered trials. That matters because imaging is where AI medicine lives. Radiology accounts for the overwhelming majority of the FDA's AI-enabled device list, which had reached 1,524 devices through March 2026, with 1,163 of them, or 76 per cent, in radiology, according to one analysis. Clearances are arriving at around 30 a month.
So the picture is a boom in approvals, dominated by scan-reading software, with a trickle of the kind of evidence that tells you whether any of it actually helps people live longer or feel better.
'Clearance Is Not Proof'
The researchers put the problem bluntly. "Clearance tells you a device resembles something already on the market," they wrote. "It does not tell you it helps anyone."
That line gets at how the system works. Most AI devices reach the market through the FDA's 510(k) pathway, which relies on a principle called "substantial equivalence": show that your product is similar to one already approved, and you can sell it. Nothing in that standard requires a company to demonstrate that patients end up healthier.
Much of the evidence that does exist leans on retrospective accuracy measures or so-called surrogate endpoints. In other words, the software got the scan right in an old data set, which is not the same as improving the outcome for the person lying in the scanner. The authors conclude that whether these devices can meaningfully improve patients' lives "remains largely unanswered based on publicly available patient-outcome evidence."
When AI Gets It Wrong In The Real World
It would be comforting to think that the gap is only theoretical. The study points to real examples where tools looked good on paper and stumbled in practice. Epic's sepsis prediction model, widely used in hospitals, was found to miss 67 per cent of sepsis cases while still triggering alerts in 18 per cent of hospitalisations, meaning plenty of false alarms alongside plenty of misses.
IBM's Watson for Oncology showed concordance with expert treatment decisions as low as 12 per cent for gastric cancer in China and 33 per cent in Denmark. Those figures are reminders that an AI trained in one place can fall flat in another.
Geography is itself part of the problem. The audit found 68 per cent of registered trials took place only in the United States. Most studies were run in well-resourced healthcare settings and tended to exclude vulnerable groups, including pregnant women, adults over 75 and non-English speakers. Those are exactly the people for whom a wrong answer can be most costly.
Recalls: The Canary In The Coal Mine
A separate body of research suggests that thin evidence has a price tag. A 2025 study in JAMA Health Forum, led by Tinglong Dai of Johns Hopkins University's Carey Business School, examined 950 AI devices cleared through the 510(k) route and found that 60 of them, or 6.3 per cent, were linked to 182 recall events affecting more than 935,000 units, largely because of diagnostic or measurement errors.
Nearly half of those recalls happened within 12 months of clearance. Crucially, devices that lacked reported validation were almost twice as likely to be recalled as those backed by retrospective or prospective data. The authors suggested that "requiring prospective evaluation or issuing time-limited clearances that lapse without confirmatory data may reduce these risks."
Other analyses paint a similar picture. Research in JAMA Network Open found only 1.6 per cent of cleared devices cited randomised trial data, and about half reported no clinical performance study at authorisation. Public documentation is patchy too: study design went unreported for 47 per cent of devices, training sample size for 53 per cent and demographic representation for a striking 96 per cent.
Why Nobody Is Running The Big Trials
If the evidence is so thin, why not simply gather more? The researchers point to a familiar set of obstacles. Commercial incentives reward getting to market fast. Multi-centre clinical trials are expensive and slow. And AI evolves so quickly that a product may be updated before a lengthy trial even finishes.
There is a money angle on the buying side as well. Clearance does not guarantee anyone will pay for a device. Only 26 billing codes exist for clinical AI solutions, and just three have permanent Category I status, according to the analysis from AI2Work. Medicare had assigned payment to only around ten of the more than 1,000 cleared devices by early 2026. Tools that did win reimbursement, such as those for coronary artery analysis, did so on the strength of robust outcomes evidence, suggesting the market may be building incentives that regulation did not require.
The Proposed Fix
The authors do not argue for slamming the brakes on medical AI. Instead they propose a staged validation framework. First comes retrospective testing on diverse datasets. Next come prospective studies. Finally, for devices that claim to change care, come trials with at least 500 to 2,000 participants that measure patient outcomes across different population subgroups and healthcare settings.
Their wider call is to redesign authorisation policy so that it requires "demonstration of effectiveness across diverse patient subgroups and healthcare settings." Dai's team has separately floated time-limited clearances that expire unless confirmatory evidence arrives, along with stronger post-market surveillance in the style of drug safety monitoring.
Expert Perspective: Promise Versus Proof
It is important to be fair here. Many AI tools in radiology really are impressive, and plenty of clinicians say they catch things that tired human eyes can miss. Nothing in the audit says these devices are dangerous across the board. Its point is narrower and, for that reason, harder to dismiss: we mostly do not know whether they make patients better, because for the most part nobody has checked.
The findings also come with caveats. The audit relies on publicly available information, so a company might hold outcome data it has never published. And a missing registered trial is not proof that a tool does not work. But absence of evidence in a field where lives are at stake is, as the researchers imply, a gap worth closing.
The timing matters too. The same news cycle that carried this story also included a high-profile debate about AI risk, with OpenAI chief executive Sam Altman telling Politico there is "a lot of daylight" between his company and rival Anthropic on regulation, and a former Anthropic employee due to testify at a New York City Council hearing on AI safeguards. Healthcare may be where the abstract argument over how much risk to accept becomes very concrete indeed.
What This Means For You
If you are a patient, there is no need to panic or to refuse an AI-assisted scan. But it is entirely reasonable to ask your doctor questions. Is this tool used to support the decision, or to make it? What evidence is there that it improves outcomes for people like me?
If you are a clinician or hospital manager, the audit is a nudge to treat an FDA clearance as the start of due diligence rather than the end of it. Local validation, monitoring and clear human oversight matter, especially if your patients differ from the populations the software was built on.
And if you are a voter or a lawmaker, the story is a case study in how fast technology can outrun the rules that were written for an older generation of gadgets. With around 30 new AI clearances arriving every month, the evidence gap is growing by the week.
The Bottom Line
The FDA's green light tells us a medical AI device is similar to something already on the shelf. It does not tell us the device will help the person on the table. With 1,357 devices audited and just three tested for real patient outcomes, the authors have made their case that the gap between regulatory authorisation and clinical validation is wide. Whether regulators, manufacturers and hospitals move to close it may decide how much we can trust the AI doctors that are already in the building.
