In March, a team at Stanford put out a paper that should unsettle anyone building AI for healthcare or wellness. MIRAGE: The Illusion of Visual Understanding shows that today’s frontier visual language models, including GPT-5, Gemini 3 Pro, and Claude Opus 4.5, will confidently answer questions about medical images they cannot see. Not slightly off. They’ll be wrong in a way that sounds measured, confident, and medically believable.
In one case, a model ranked at the top of a standard chest X-ray benchmark without being shown a single X-ray. In another, the researchers explicitly told the model it did not have access to the images and asked it to guess. Performance went down. The model did better when it was allowed to imagine a “mirage image” and reason from that fiction than when it was asked to reason honestly about missing data.
For a field now pivoting from chatbots to autonomous agents, that is a structural problem. And it is a data problem before it’s a model problem.
From Assistants to Agents: The Stakes on Every Data Source Just Rose
For most of the last few years, consumer health AI has mostly meant a chatbot window. A user asks, the system answers, and the user decides what to do with it. Even outside healthcare this has been the m.o. (I am thinking about my least favorite hotel app which offers me an opportunity to chat with the front desk, except they are never there and the answers not really helpful). In that setup, the user acts as the final check on a confident mistake. Agentic AI changes the shape of the system.
BCG’s 2026 outlook describes agents that “observe, plan, and act on their own,” and those systems are already showing up in care coordination, protocol adjustments, and the next layer of personalization in digital health. That is a meaningful jump in capability and shrinks the buffer between a model’s output and a real-world consequence.
In that new architecture, every upstream data source turns into a trust boundary. If the data is noisy, inconsistent, or unverified, the agent doesn’t pause. It still produces a plan, and worse still, speaks about it very confidently, especially when the instruction is to act.
Health and Wellness Has the Most at Stake
The IQVIA Institute estimates more than 350,000 digital health apps are already on consumer app stores. Mental health coaches, sleep trackers, nutrition apps, fertility trackers, chronic condition managers — hundreds of thousands of products, reaching the better part of a billion people.
And nearly all of them are layering in AI, usually on top of data they don’t fully control and haven’t independently checked against any hard ground truth.
Sleep is the area I know best, and it makes the problem easy to see because the mismatch between what consumers think they’re getting and what the data really represents can be huge. Put on four different wearables tonight, and you may wake up to four different Sleep Score™ numbers. Some popular rings can overestimate total sleep time by as much as an hour a night. Some general-purpose wearables can underestimate overnight wakefulness by a comparable margin. App-only sleep tracking, without a validated measurement layer, can be off by nearly 99% on something as basic as how many times you woke up in the night. You can probably guess, these numbers can be the difference between “you are perfectly fine” or “you need urgent help.”
Now, feed that into an LLM and ask it to coach someone. You’ll get a confident weekly plan, it will reference the numbers, and it will even sound like a clinically grounded recommendation. But the signal it’s reasoning over was already broken before the model ever saw it.
Scale that across health categories where wearables, consumer sensors, and self-reported logs are the main inputs, and the issue stops being niche. Agents are about to be asked to make more decisions, across more domains, using data that has never been reconciled against any reliable standard or indeed understood in the first place.
The Data Layer Is Becoming the Moat
For the last couple of years, many teams assumed the defensible advantage in AI products would be the model, but that’s stopped being true. Frontier capability has become interchangeable faster than most people predicted. Major infrastructure providers already treat models like swappable components, sold through multi-model, pay-as-you-go APIs.
When the model becomes a commodity, differentiation moves elsewhere. Product experience matters, sure, but the other place it lands is the data layer.
For years, health APIs mostly meant plumbing: HL7 and FHIR pipes, device SDKs wired into dashboards, records moved from one system to another. What’s emerging now is a different kind of API that delivers signal an AI agent can safely ground itself on: a health data API validated against a gold standard, steady across input sources, clear about where its numbers come from, and willing to admit when it doesn’t know.
MIRAGE, at its core, is what happens when a system has no clean way to say, “I don’t know.” The data layer has to make that answer possible.
What to Demand from a Health Data API for AI Agents
If you’re building products in this space, there are five data checks that matter.
Validation against an accepted ground truth. If the API outputs sleep, activity, glucose, or any physiological measure, ask what it was benchmarked against. For sleep, polysomnography. For glucose, CGM. In my humble opinion, validation needs to be far greater than N=30 (which is sadly often the case). The comparison should be published and peer-reviewed, and it should be against the consumer devices the API claims to replace or reconcile. This cannot live as a marketing line; it needs to be a study. Actually, many studies.
Cross-device consistency. If the same biological event yields different numbers depending on which device or app someone uses, the API should reconcile that, not relay the noise. Agent-grade APIs give you one best answer, not five incompatible ones.
Transparent provenance. The downstream agent should be able to trace a value back to its source, understand how it was derived, and see a confidence signal. Without that metadata, every step the agent takes becomes less connected to evidence.
A practical way to represent uncertainty. This is the MIRAGE lesson in production form. When data is missing or low confidence, the API should report that explicitly in a structured way the agent can route on. Quiet extrapolation is where bad decisions start.
Compliance and privacy posture that fits the category. GDPR, HIPAA where it applies, ISO certifications, and policies like not retaining raw sensor data for sensitive signals. In regulated settings, this is baseline, and in consumer health it’s quickly becoming expected.
The Next Phase Will Be Built on Grounded Systems
MIRAGE isn’t arguing that frontier models are useless. It’s showing something more specific: they can be too confident without evidence, and the more fluent their reasoning sounds, the easier it is to miss that there was nothing underneath it. That may be fine when you are trying to figure out where to go to dinner, but it is nowhere near good enough in health.
But teams shipping products now don’t get to wait for model-side fixes. Agents that coach, route, recommend, and act in health decisions are being built today.
So, the practical answer is grounding. Foundations. A data layer beneath the agent that is validated, consistent, transparent, and honest about uncertainty, delivered through APIs that make it easy to build correctly. Those who get that right in health and wellness, and there are serious teams working on it, will support a generation of AI products that actually match their claims. More importantly, products which really deliver better health outcomes.
Everyone else will ship mirages. Or do they disappear?
