Procurement of AI in healthcare: what should you pay attention to legally?

In Sweden, AI in breast cancer screening is a game changer: where normally two radiologists review each mammogram, AI enables this to be done with just one radiologist—without any loss of quality. In the UK, false positive results in mammography decreased by 5.7% and false negatives by 9.4%. In the United States, an AI tool for tumour delineation achieved an accuracy of 84.7%, compared to 67.2% for experienced specialists. And the Netherlands Cancer Institute (Antoni van Leeuwenhoek) demonstrated that AI can increase survival rates in immunotherapy by 24%.

Outside oncology, the results are equally promising. AI can now automatically calculate coronary calcium scores from CT scans—at a level of accuracy comparable to a specialised radiologist. In acute stroke care, AI detects vascular occlusions within seconds and immediately alerts the clinical team.

The administrative gains are also significant. Transcription tools that automatically convert patient consultations into written reports save physicians 24% in time and reduce cognitive burden by 81%. It is therefore no surprise that 65% of European healthcare providers plan to implement such tools in the coming years.

The European Commission has also assessed the financial impact. AI-supported clinical decision-making could save EU healthcare systems up to €252 billion over the next decade. Automated image analysis could generate €192 billion in savings, and digital mental healthcare platforms €164 billion. The digital health market is expected to grow from €11 billion (2023) to €61 billion by 2035.

Widespread implementation is still some way off. Data is fragmented and non-standardised (45% of healthcare providers identify this as a key barrier). AI tools only deliver value when integrated with existing systems such as electronic health records (EHRs). Regulation is complex: medical AI falls under both the AI Act and the Medical Device Regulation (MDR), and 60% of suppliers cite this overlap as a significant obstacle. Algorithmic explainability is often lacking, and 34% of healthcare providers report a shortage of digital skills as a barrier.

In the coming years, many new AI tools will be procured in healthcare. Smart procurement is also legal procurement. When selecting and contracting AI solutions, the following questions are essential:

• Does the tool comply with regulation?
  Medical AI is classified as high-risk under the AI Act and often also qualifies as a medical device under the MDR. This imposes requirements relating to safety, clinical validation, human oversight, and documentation. Verify whether the supplier complies with both frameworks and ensure this is contractually secured.
• Is the system explainable?
  Healthcare professionals must understand how an algorithm reaches its conclusions. Prefer tools that offer “explainable AI” and ensure the supplier clearly communicates performance and limitations.
• Does it integrate with existing systems?
  AI only adds value if it integrates with electronic health records and clinical workflows. Integration requirements should be part of both selection criteria and contractual terms.
• Is there a training plan?
  AI literacy is not optional: the European AI Regulation requires organisations using AI to ensure adequate training. Ensure the supplier supports implementation and provides a concrete training plan.
• Are proper legal arrangements in place?
  Clear contractual arrangements on liability, data use, audit rights, and exit strategies help prevent disputes later on.

Do you have questions about the legal aspects of AI and procurement? We are happy to support healthcare organisations. Please contact Menno de Wijs, attorney IT, Privacy & Cybersecurity.

Would you like a monthly overview of updates and blog posts in your inbox? Subscribe to the newsletter.