Article

De-risking enterprise
AI in healthcare

A cybersecurity perspective

Vipin Varma

SVP and Head of Cybersecurity,
CitiusTech

Deepesh Singh

AVP, CitiusTech

15-Jul-2026

Insights

AI in healthcare has moved from experimentation to systems that influence decisions and trigger actions at scale.
Traditional cybersecurity models are not designed for probabilistic, adaptive, and autonomous AI systems.
AI expands the attack surface across data, models, prompts, APIs, and agent-driven workflows.
De-risking enterprise AI requires continuous, lifecycle-driven security and governance.

Across healthcare systems, AI has moved decisively out of experimentation mode. What began as narrowly scoped pilots, with use cases like automating documentation, assisting with coding, or summarizing clinical notes, has rapidly evolved into systems that influence decisions, trigger actions, and interact directly with sensitive data. AI is now embedded^[1] across administrative workflows and operational systems in healthcare, increasingly influencing clinical operations and patient-facing processes, often operating continuously and at scale.

Yet while AI capabilities have advanced quickly, the security models governing them have not kept pace. Most healthcare cybersecurity programs were designed for deterministic software systems - applications whose behavior is predictable, whose logic can be traced step by step, and whose failure modes are well understood. AI systems do not behave this way. They are probabilistic, adaptive, and increasingly autonomous. They learn from data, evolve over time,^[2] and respond to inputs in ways that cannot always be anticipated in advance.

This mismatch is creating a growing gap between how AI behaves and how security is designed to control it. As AI becomes foundational to care delivery and operations, that gap becomes a material risk - one that traditional controls alone cannot close.

The new AI attack surface: From data and models to autonomous agents

In traditional IT environments, the attack surface is relatively well defined. Security teams focus on infrastructure, networks, endpoints, and user access. AI expands that surface in multiple directions at once. Organizations are aware of that and 66% of them expect AI to have the most significant impact on cybersecurity.^[3]

Risk now begins with data. Training datasets, fine-tuning inputs, and retrieval sources often contain highly sensitive information, including patient records, clinical narratives, imaging metadata, and genomic data. If this data is not properly classified, sanitized, or governed, it can be unintentionally exposed through model behavior rather than through a direct breach.

The model itself becomes a target. Models can be manipulated, probed, or influenced through adversarial inputs. Inference behavior can leak information that was never intended to be shared. Updates and retraining cycles introduce new vulnerabilities, especially when models evolve faster than the controls around them.

Interfaces add another layer of exposure. APIs, plugins, and natural-language prompts create new entry points where misuse does not resemble traditional attacks. Prompt injection, misuse of context, and unintended chaining of instructions can produce outcomes that bypass established access controls without ever triggering a conventional security alert.

The most significant shift, however, comes with agentic AI. As AI systems are given the ability to act - triggering workflows, accessing multiple systems, or making chained decisions - the risk profile changes fundamentally. Misconfigurations or adversarial inputs no longer result only in incorrect answers; they can lead to unauthorized actions that propagate across applications. In these environments, traditional perimeter-based security struggles because the problem is no longer just who has access, but how behavior unfolds over time.

The need for AI security frameworks

Healthcare AI operates in a uniquely high-stakes context. Patient records, clinical notes, imaging, and genomic data are not simply sensitive; they are consequential. Decisions influenced by AI systems can affect downstream operational workflows and, indirectly, clinical and patient outcomes. When failures occur, they do not remain contained within a single application. They ripple across workflows, influence downstream decisions, and create regulatory and reputational exposure.

As AI becomes embedded into day-to-day healthcare operations, security can no longer be treated as a back-office IT function. It becomes a core element of clinical governance. Leaders must be able to answer not only whether systems are secure, but whether AI-driven decisions are accountable, explainable, and auditable.

This is where AI security frameworks enter the picture. Their emergence is a response to failure modes enterprises are already encountering as AI scales. These frameworks recognize that AI risk is not confined to deployment. It begins well before a model goes live and continues long after it is in production.

Training data selection introduces risk if sensitive information is not properly handled. Model updates and fine-tuning can alter behavior in ways that are difficult to predict. Access patterns and prompt behavior can expose data indirectly. Drift can cause models to behave differently over time, even when no explicit changes are made. Deprovisioning and retirement introduce their own risks if models and data are not fully disconnected.

In healthcare, where AI systems evolve alongside clinical workflows and regulatory expectations, this lifecycle view is essential. Security must be continuous rather than episodic, and governance must extend across the entire lifespan of an AI system, not just its initial rollout.

What de-risking AI really means for healthcare leaders

De-risking AI does not mean slowing innovation or surrounding AI initiatives with friction. It means being intentional about how AI is designed, deployed, and governed. It means acknowledging AI Systems are not neutral tools, but active participants in healthcare workflows.

For leaders, this requires:

Clarity on where autonomy is appropriate and where human oversight remains essential.
Mechanisms to log and explain AI-driven decisions, embedded as core design principles.
Clearly defined accountability when AI systems act, especially when those actions influence clinical or operational outcomes.

Most importantly, it requires accepting that AI security is not a one-time certification exercise. It is an ongoing discipline that evolves alongside models, data, and use cases. In healthcare, where trust and safety are foundational, this discipline becomes a prerequisite for scale rather than a barrier to it.

This perspective sets the stage for the next phase of the conversation. Once organizations understand why AI must be de-risked and how that risk spans the entire lifecycle, the question becomes practical: how can AI itself be used to strengthen cybersecurity operations, enforce guardrails, and manage complexity at scale?

References

This article is also published on Cyber Defence Magazine

Frequently Asked Questions

How do "probabilistic" failures in AI differ from the "deterministic" bugs we manage in legacy healthcare software?

Deterministic bugs in legacy healthcare software are predictable and traceable step by step, with failure modes that stay contained to the system that produced them. AI failures are probabilistic and adaptive, so the same input can produce different outputs, and errors ripple across workflows, influence downstream decisions, and create regulatory and reputational exposure. That's why legacy, rule-based monitoring alone can't be treated as a sufficient safeguard for AI-driven clinical systems.

What are the most common "hidden" attack surfaces in healthcare AI that bypass traditional SOC monitoring?

The attack surface for healthcare AI extends well past the perimeter traditional SOC tooling is built to watch. It spans the data layer, where training, fine-tuning, and retrieval sources often contain sensitive information; the model layer, where adversarial inputs can probe or manipulate model behavior; and the interface layer, made up of APIs, plugins, and natural-language prompts. Prompt injection, misuse of context, and unintended chaining of instructions matter most here, since they can bypass established access controls without ever triggering a conventional security alert.

How does the "agentic" shift in AI change our liability profile for patient-facing applications?

Once an AI system can act, triggering workflows, touching multiple applications, or chaining decisions, the liability question shifts from whether the output was wrong to who is accountable for the action the system took. A misconfiguration or adversarial input can propagate an unauthorized action across systems before anyone reviews it, so this is a governance problem as much as a technical one. Before deploying AI agents, organizations should define accountability, oversight, and liability boundaries. Consult with CitiusTech to assess patient-facing workflows.

What architectural patterns (e.g., RAG, API firewalls) are most effective at preventing data exfiltration via natural language interfaces?

A retrieval-augmented generation (RAG) pattern that gives fine-grained control over what internal information is searched and retrieved keeps the LLM as an interface layer rather than a direct data store, which limits what a crafted prompt can pull out. Layering an API gateway or firewall in front of that interface closes the other main gap, since APIs, plugins, and natural-language prompts create entry points where misuse doesn't resemble a traditional attack. Enforcement needs to sit at the interface layer itself, not just around the network perimeter.

How can we design "human-in-the-loop" checkpoints that don't introduce unacceptable latency into clinical operations?

The guidance is to decide upfront, as a core design principle, exactly where autonomy is appropriate and where a human checkpoint is non-negotiable, so review only sits on the small set of high-risk decisions rather than every model output. CitiusTech Knewron platform applies this by pairing agentic workflows with audit logs and reasoning trails that let a reviewer validate a decision quickly using the model's own rationale, rather than re-deriving it from scratch. Getting that checkpoint placement right is workflow-specific, so it's worth consulting with our healthcare experts to design the oversight points before latency becomes a production problem.

Can we build a security layer that sanitizes prompts in real-time without degrading the model’s clinical accuracy?

Keeping the LLM as a pure interface layer over a controlled RAG retrieval pipeline lets prompt sanitization happen at the boundary, filtering what reaches the model and what it's allowed to retrieve, rather than inside the model's reasoning, which is what protects clinical accuracy. Tracking prompt efficiency, hallucination, accuracy, and relevance as ongoing quality dimensions also means a sanitization layer can be tuned and monitored against real metrics instead of deployed once and left alone.

Which specific AI security frameworks or tools should we integrate into our existing CI/CD pipelines?

There's no single named CI/CD security toolchain to point to here. The underlying guidance frames AI security as a lifecycle discipline embedded from training-data selection through fine-tuning, deployment, and drift monitoring, with policy enforcement (CitiusTech Knewron's approach uses OPA-based policy-as-code) and audit-ready trust frameworks built in rather than bolted onto the pipeline afterward. Because the right combination of tools depends on which lifecycle stage and existing pipeline you're integrating into, it's best to consult with CitiusTech experts directly to map specific frameworks to your CI/CD environment.

What specific telemetry data should we collect to prove AI compliance to an auditor during a clinical incident?

An auditor needs three telemetry types captured continuously rather than reconstructed after the fact: input and output logs of what the model saw and produced, reasoning or decision trails explaining why the system acted as it did, and access and action logs showing which systems or workflows the AI touched.

What is the most cost-effective way to implement real-time "guardrails" for high-stakes, agent-driven workflows?

The most cost-effective path is to build guardrails as policy-as-code embedded in the agent orchestration layer. OPA-based policy enforcement and healthcare-native guardrails can be applied at the point where agents act, rather than through a separate, custom-built monitoring system layered on afterward. This allows the same policy engine to be reused across every new agentic workflow, keeping the marginal cost of extending guardrails low as the agent footprint grows. Since the appropriate policy set depends on whether workflows are patient-facing or internal, organizations should consult with our AI experts to scope a guardrail implementation aligned to their specific agent landscape and risk profile.