How Healthcare Organisations Can Use AI Agents With Patient Data Safely

Healthcare organisations are starting to hand real work to AI agents. Scheduling and reminders, referral routing, prior authorisation paperwork, medical coding, and the summarising of long patient records are all being tested right now. The appeal is easy to see. These tasks are repetitive; they take up clinical and administrative time, and they sit in systems that were never designed to work together.
The difficulty is that almost every one of those tasks touches patient data, and healthcare is the industry where a breach costs the most. IBM’s 2026 Cost of a Data Breach Report put healthcare at the top again, at an average of $6.64 million per breach. The same report found that AI-driven attacks rose 56% year over year.
None of this means agents should stay out of healthcare. It means the way they are set up matters more than the model behind them. Here is how to approach it.
Where agents are showing up
Before the controls, it helps to be specific about the work. These are the areas drawing the most interest, and the patient data each one usually touches.
Scheduling and patient messaging: names, contact details, appointment types, and sometimes the reason for the visit.
Referrals and prior authorisation: diagnoses, treatment plans, and insurance details.
Coding and billing support: clinical notes, procedures and payer information.
Documentation and record summaries: the full chart, which is the most sensitive of all.
Operations and supply chain: usually very little patient data, which makes it a lower-risk place to start.
Why agents need different controls
A chatbot answers a question. An agent holds credentials, reads records, calls other systems, and takes actions, often in a chain of steps that no person watches. The risk moves from what the tool says to what it can reach and what it can do.
A summarising tool with access to every chart in a hospital is a bigger exposure than any single wrong answer it might give. Most of the controls below follow from that point.
1. Map the data before you build
Start by listing where patient data lives and which fields the agent needs for its specific task. A scheduling agent needs names, contact details, and appointment types. It does not need diagnoses or lab results.
This follows the minimum necessary principle that healthcare privacy rules are built on, and it is the simplest way to cut exposure. Include the places data hides: shared drives, email attachments, old exports and spreadsheets that staff keep for convenience. If you cannot say which fields an agent uses, and where they come from, it is not ready to deploy.
2. Give every agent its own identity and narrow permissions
Agents should not borrow a staff member’s login. Give each one a separate identity, with access limited to the systems and records its task requires, read-only where writing is not needed, and reviewed on a schedule.
Time limited access is better still, since an agent that only needs a record during a task should lose it afterwards. When something goes wrong, you can see exactly what that agent touched, and you can switch it off without disrupting any person’s access.
3. Keep identifiers out of the model where you can
Many tasks do not need to know who the patient is. Summarising trends, drafting general correspondence or sorting referrals by category can work on masked or anonymised data.
Where identifiers are required, pass only what the step needs and strip them from anything that is stored or logged. Masking at the data layer is far more reliable than asking people to remember what not to paste into a tool, and it keeps working when staff are busy or new.
4. Know where the data goes
Read the contracts closely. Where is the data processed and stored? Is it used to train anyone’s models? How long is it kept, and which subcontractors can see it?
In the US, a vendor handling protected health information normally needs a business associate agreement under HIPAA. In Canada, provincial health privacy laws such as Ontario’s PHIPA apply, and in Europe the GDPR does. Data residency and retention terms often decide whether a tool can be used at all, so settle them before a pilot starts, not after the first patient record has gone through.
5. Decide what the agent may do alone
Sort tasks into three groups: what the agent can do on its own, what it can propose for a person to approve, and what it must never do.
Reminders and form pre-filling are often safe to automate. Anything that changes a clinical record, affects a care decision, or disputes a bill should have a person confirming it. Write these lines down, and build them into the system as hard limits, not as guidance. An instruction in a policy document is easy to miss. A limit in the system is not.
6. Log every action so it can be reconstructed
For each action, record which agent acted, which record it accessed, why, what it did, and what happened next. Privacy officers, auditors, and regulators will ask for this, and so will your own team after the first unexpected result.
Keep the logs tamper-resistant and outside the agent’s own control, and decide how long you retain them. A clear audit trail also speeds up the investigation when a patient asks who has seen their record.
7. Test, monitor and plan for failure
Agents can be tricked. A malicious instruction hidden in an email, a document or a web page can steer a tool into disclosing or changing data. This is known as prompt injection.
Test for it before launch, repeat the tests after any change, and watch behaviour in production for unusual volumes, unexpected records, or new destinations. Agree in advance who is called at 3 am if an agent starts acting strangely, and how it gets paused. Models also drift over time, so check accuracy after go-live as well, not only before.
What a failure looks like
This is an illustration, not a real incident. Imagine a referral agent given read access to the whole patient index so that it never fails on edge cases. A scanned letter arrives with hidden text telling it to include a list of recent patients in its reply.
Without limits on what it can reach and where it can send things, it might comply. With narrow permissions, a restriction on outbound destinations and alerts on unusual volumes, the same attempt fails, and someone is told. The model is identical in both cases. The setup is what differs.
Bring clinicians and privacy officers in early
Technology teams often build first and ask clinical and privacy staff to review afterwards. Reverse that order. Clinicians know which tasks are low risk and which only look that way. Privacy officers know the incident reporting duties and consent rules that apply.
Asking them at design stage costs a few meetings. Asking them after launch can cost months of rework, and it tends to produce a long list of changes the project has no budget for.
A checklist before go-live
Before any agent goes live, a few plain questions are worth asking.
Which patient fields does it use, and why does it need each one?
What identity does it run under, and what can that identity reach?
Where does the data go, and who are the vendor’s subprocessors?
Which actions need a person to approve?
Where are the logs, and who reviews them?
Who can pause it, and how quickly?
If the team cannot answer any of these, the project is not ready yet. That is a useful result, not a failure.
Start with one workflow
A good first candidate is a high-volume administrative task with limited data needs, a clear owner, and an easy way to check the results. Appointment reminders and referral status updates fit this description.
Run it for a few months, review the logs, and then widen the scope one step at a time. Organisations that go this way build the evidence their privacy and security teams need, and they find problems while they are still small and cheap to fix.
Rules differ by jurisdiction and change often, so involve your privacy officer and legal counsel early. Nothing here is legal advice.
Where Contivos fits
At Contivos, we work with technology leaders on AI application development, data protection and governance, and the ongoing operation of AI systems once they are live. In healthcare, that means building access limits, masking, audit trails, and monitoring in from the start, not adding them after launch.
If you are planning an agent project that touches patient data, visit contivos.com to talk it through.
Which task in your organisation would you trust an AI agent with first?





Comments