The demo was never the hard part.

The model works. The output looks good. Someone has already shown it to a customer. And now the question is whether it can touch real patient data, and the room goes quiet.

The instinct is to write an AI policy.

But a policy that engineering cannot implement is a document, not a control. The gap is not intent. It is that nobody has decided where the boundaries sit, who reviews what, and how you would prove any of it a year from now.

Three controls decide whether an AI feature in a PHI-bearing system is shippable. Everything else is commentary.

1. PHI boundaries: where the data may go

The question is not "are we HIPAA compliant." It is "which specific fields may leave which specific process, and to whom."

• Name every system that can receive PHI, including model providers, logging, error tracking, analytics, and evaluation tooling. The unglamorous ones are where it leaks.
• Decide what is sent versus what is referenced. A record identifier and a de-identified summary are a different risk tier than a raw chart.
• Confirm the contractual posture matches the technical one. A vendor without a BAA is not made acceptable by good intentions.
• Verify retention on the receiving side. "We do not train on your data" and "we do not retain your data" are different promises.
• Test the failure path. A stack trace containing a patient record is still a disclosure.

2. Provenance: what produced this, and from what

If a clinician or an auditor asks why the system said something, "the model generated it" is not an answer.

• Record the model, version, prompt or template version, and retrieval inputs for every generated artifact.
• Make the output attributable to its sources, not merely plausible.
• Keep the chain queryable. Provenance that exists only in logs you cannot search is provenance you do not have.
• Version the governance itself. When the prompt changes, the behavior changes, and you need to know which outputs came from which regime.

3. Human-in-the-loop: a decision, not a checkbox

Most teams claim human review. Fewer can say what the human is accountable for.

• Define what the reviewer is approving: the facts, the phrasing, the clinical judgment, or only the formatting.
• Give them enough to review. A reviewer who cannot see the sources is rubber-stamping.
• Make declining easy and observable. If the fast path is approval, you have built an approval machine.
• Instrument override rates. A review step nobody ever rejects is a control that is not working.
• Decide what happens on disagreement, before it happens.

Why this is leadership work

None of the above is a model problem. It is architecture, contracts, data flow, and accountability — decided before the feature is built, because retrofitting boundaries into a shipped system is where the cost lives.

The reframe: do not ask whether your AI feature is accurate enough to ship. Ask whether you could reconstruct, a year from now, what it did and why — and who was accountable when it was wrong.

Governance is not the tax on AI velocity in regulated software. It is the thing that lets you keep the velocity when someone finally asks.

Next step

If your AI features are shipping faster than your governance, that ordering is fixable — and it is cheaper to fix now than after an audit. Book a fit review.