2026-07-11 · 10 min read · HAAM Notes / Medical AI and product assurance
The System Around the Medical Model
AI can identify a pattern, draft a note, or flag a patient. Healthcare reveals how much product work begins after the model produces its answer.
A medical AI demo often ends with a prediction.
A tumour is outlined on a scan. A consultation becomes a structured note. A phone detects an irregular heartbeat. A pathology slide receives a probability score.
The output appears, the screen looks convincing, and the technology seems complete.
The real product begins at that moment.
Someone must decide whether the input was suitable, whether the result is reliable enough to act on, whether another professional should review it, what the patient should be told, and what happens when the system is wrong. The result may need to enter a medical record, trigger a referral, survive an audit, and remain understandable months later.
A model produces an answer. A medical product creates a safe path around that answer.
This distinction matters far beyond healthcare. Medical AI makes the surrounding product layer unusually visible because weak assumptions can become clinical decisions. Evidence, boundaries, escalation, monitoring, and recovery move from background infrastructure into the centre of the experience.
A small Nordic window into the problem
Estonia and Finland offer a useful cross-section of what AI health products are becoming.
Estonia's Better Medicine has developed BMVision Kidney, a Class IIa medical device that detects and characterises kidney lesions in CT scans. A clinical study found that the system reduced radiologists' reporting time by an average of 33 percent, with savings reaching 52 percent in some tasks. It also improved sensitivity for benign renal lesions and increased agreement between radiologists.
In Finland, Gosta Labs turns doctor-patient conversations into clinical documentation and structured data. The company says its system is already used across tens of thousands of clinical appointments. In February 2026, Gosta announced that its underlying AI operating system had been registered in EUDAMED as a Class I medical device.
Aiforia supports pathologists analysing digital tissue samples. Its prostate cancer application is CE-IVD marked, and a 2026 project is deploying the system across additional Paris hospitals to evaluate its use in routine pathology at a scale of around 3,000 prostate biopsy cases per year.
CardioSignal turns the motion sensors in a smartphone into a one-minute measurement for signs of atrial fibrillation. Its software was certified as a Class IIb medical device under the European Medical Device Regulation.
These products work in different places: radiology, clinical documentation, pathology, and personal monitoring. Yet they face the same product question:
How does a probabilistic output become something a person can safely depend on?
The model is one moment in a longer chain
Medical AI is often discussed as though the model sits at the centre of the system.
In practice, the model occupies one position inside a much longer sequence:
capture → validate → interpret → communicate → decide → act → record → monitor
Every step introduces its own risks.
A scan may use the wrong protocol. A microphone may miss part of a conversation. A pathology slide may be stained differently from the training data. A patient may move during a smartphone measurement. A generated note may sound clinically polished while quietly omitting an important detail.
Even a statistically strong model can enter a weak product system.
This is where the idea of a product harness becomes useful. A harness combines the guides that constrain a system with the sensors that reveal whether it is still behaving as intended.
For medical AI, the guides include the intended patient population, accepted input formats, permissions, confidence thresholds, workflow rules, and clear limits on what the software is allowed to conclude.
The sensors include error reports, clinician corrections, unusual input patterns, performance across different populations, referral outcomes, safety incidents, and changes in real-world use.
The harness turns a one-time validation exercise into an operating system for continued responsibility.
Evidence becomes part of the interface
Most software hides its assurance work.
Users rarely see the accessibility tests, deployment checks, security reviews, or performance monitoring that sit behind an ordinary digital product. They experience the result indirectly through reliability.
Healthcare pushes some of that evidence closer to the surface.
A clinician needs to know what the system analysed, which findings it produced, how those findings relate to the source material, and where uncertainty remains. A patient may need to understand whether an alert represents a diagnosis, a screening result, or a reason to seek further assessment.
The interface therefore has to carry more than information. It must carry the conditions under which that information deserves trust.
This changes the role of explanation. An explanation cannot simply decorate the output with a confidence score or a heat map. It must help the user make a better decision.
Useful evidence might include:
- the source scan, passage, signal, or tissue region behind a result
- missing or low-quality input
- the system's intended use and limitations
- changes made by a clinician
- the model and product version used
- the next appropriate action
- the route for challenging or correcting the result
The strongest medical AI interface may sometimes make a decision slower. A pause, comparison, or required confirmation can protect judgment from the speed and apparent certainty of automation.
Friction becomes a safety feature.
Human oversight can become theatre
Medical AI companies usually describe their products as tools that support professionals. The radiologist, pathologist, or physician remains responsible for the final decision.
That principle sounds safe. Its value depends on how the workflow is designed.
A professional who must approve hundreds of machine-generated outputs may gradually become a confirmation layer. When the AI is usually correct, checking becomes less attentive. When the system presents a polished recommendation before the human forms an independent view, it can anchor the decision.
A human click does not automatically create meaningful oversight.
Meaningful oversight requires enough time, information, authority, and interface support to disagree.
The product should help the professional detect where attention is needed. It should reveal uncertainty, surface contradictions, preserve access to original evidence, and make correction easier than passive acceptance. It should also learn from disagreement without silently turning every correction into new training data.
The human role is judgment. The product must protect the conditions in which judgment can still happen.
A health record describes the past
Predictive systems learn from history. Healthcare adds an uncomfortable fact: bodies change.
A person starts a medication. A tumour grows. Pregnancy changes physiological baselines. Surgery alters anatomy. A sensor is replaced. A chronic condition develops. A previous diagnosis is corrected.
Historical data may remain accurate as a record while becoming misleading as a prediction.
A personalised health product therefore needs more than memory. It needs a way to recognise transition.
This creates an important design principle:
A person should be able to revise the model of themselves.
The system should make its assumptions visible enough to challenge. Clinicians and patients should be able to mark a new baseline, correct outdated information, explain an unusual event, and distinguish temporary change from long-term deterioration.
Without this, personalisation can harden into a permanent judgment based on an earlier version of the person.
Health AI should repeatedly ask: What changed?
The clinical product continues outside the screen
A model may correctly identify a risk and still produce a poor outcome.
A patient receives an alert but cannot access a doctor. A screening product creates more referrals than the health system can process. A false positive causes months of anxiety. A hospital buys an efficient documentation system and fills the saved time with additional administrative requirements.
These are product effects, even when they happen outside the software interface.
Health products participate in systems with limited capacity, uneven access, professional hierarchies, reimbursement incentives, and emotional consequences. Increasing detection can create value, workload, or both. Faster documentation can improve care or accelerate an already overloaded workflow.
The relevant unit of design is therefore larger than the individual interaction.
A responsible product team has to observe downstream effects:
- Did clinicians act on the output?
- Did patients understand what happened?
- Did waiting times change?
- Did one population receive more false alerts than another?
- Did the tool reduce workload or redistribute it?
- Did easier detection lead to useful intervention?
- Could people recover from an incorrect result?
The answer may require collaboration between designers, clinicians, engineers, regulators, procurement teams, and patients. The interface is only one surface of the product.
Regulation already assumes a living product
European medical-device regulation treats assurance as a lifecycle process.
The European Commission's guidance states that manufacturers must maintain post-market surveillance systems, gather real-world information, update technical documentation, and use what they learn to support preventive or corrective action. Surveillance continues throughout the lifetime of the device.
EUDAMED is designed to provide a living picture of medical devices across their lifecycle and improve transparency for professionals, authorities, and the public.
The EU AI Act adds another layer. AI systems embedded in regulated products can fall into the high-risk category when they require third-party conformity assessment, creating an overlap between AI governance and existing product-safety regimes.
This regulatory architecture points toward a broader product idea:
A trustworthy product carries an evidence trail through time.
The approved version, current version, training assumptions, incidents, corrections, and observed outcomes form part of the product's identity.
Certification provides a threshold for entering the market. Continued assurance determines whether the product keeps deserving its place there.
Designing the whole medical product
A medical AI team may begin with a model. Product design expands the frame.
The work includes deciding where the system belongs, who can use it, how results are reviewed, how uncertainty appears, what counts as failure, and how reality flows back into improvement.
This creates several design responsibilities.
Design the claim
Define exactly what the product promises. Detection, screening, prioritisation, documentation, and diagnosis carry different meanings and consequences.
Design the boundary
Show where the system should stop. A product needs a clear handover point when the input is unsuitable, the confidence is low, or the case falls outside its intended use.
Design the disagreement
Make correction visible and easy. Capture why a professional disagreed, preserve both views when relevant, and avoid treating human judgment as anonymous noise.
Design the evidence
Connect outputs to their sources. The user should be able to inspect what produced the result and what remains uncertain.
Design the escalation
A warning without a viable next step transfers responsibility to the user. The product must connect detection to a realistic care pathway.
Design the monitoring
Measure performance after launch across users, contexts, populations, versions, and outcomes. Watch for quiet degradation as workflows and data change.
Design the recovery
Errors will occur. The product needs a way to correct records, contact affected people, revise decisions, and learn without concealing what happened.
Together, these elements form the clinical product around the model.
Healthcare is showing where AI products are going
Medical AI feels like a specialised category. Its lessons are increasingly general.
As AI moves into finance, education, employment, government, and infrastructure, more products will make decisions that persist beyond the interaction. Outputs will influence access, eligibility, diagnosis, reputation, and opportunity.
The product will need to explain itself across time. It will need to support disagreement, preserve evidence, monitor consequences, and respond when the world changes.
Healthcare reaches this future early because the stakes make shortcuts harder to hide.
The demo may end when the model produces an answer.
The product continues through every person, institution, and decision that follows.
AI authorship note
This essay was developed from Kris Haamer's exploration of AI health startups in Estonia and Finland and their connection to Raw Drafts themes around product harnesses, assurance, trust, and human judgment. AI supported research, synthesis, drafting, and implementation. Kris is responsible for the framing, editorial direction, and decision to publish it.