A recent Becker’s Hospital Review article highlights a troubling trend as reported by the New York Times: patients are increasingly uploading complete medical records, images, and physician notes to generative-AI chatbots like ChatGPT to get faster answers or second opinions. The article specifically notes  — “HIPAA doesn’t apply to popular chatbots.”

What does that mean in practice? Under the Health Insurance Portability and Accountability Act (HIPAA), “covered entities” (e.g., hospitals, clinics) and their “business associates” must protect patient records and other Protected Health Information (PHI). But a generative-AI “answer engine” — unless specifically configured and contracted as a HIPAA-compliant tool — typically does not fall under these protections. Indeed:

• Some privacy experts warn that patient records uploaded to these systems “could be employed to train the large language models or otherwise exposed by them.”
  
• A recent HIPAA Journal review concluded standard ChatGPT is not HIPAA compliant — in large part because the model’s developer (e.g., OpenAI) will not enter into a Business Associate Agreement (BAA) with covered healthcare entities.

In short: just because a healthcare facility has a BAA with a front-end vendor doesn’t mean the underlying AI they employ is HIPAA-safe — especially if that AI is a third-party large language model (LLM) like ChatGPT or Gemini. 

Most Front-End AI Agents Don’t Own Their Models

One of the most important but overlooked realities: many companies marketing themselves as “AI agents” for scheduling, patient communication, or clinical support don’t actually own or build the underlying AI technology. Instead, they act as front-end interfaces — using APIs from major model providers (e.g., ChatGPT, Gemini) to deliver their services.

That means:

• The front-end company may claim to offer “AI-powered patient scheduling,” “virtual clinical assistants,” or “automated documentation,” but behind the scenes they are simply relaying data to and from external AI platforms.
• Since those platforms are typically not HIPAA-compliant, uploading PHI through the front-end effectively bypasses HIPAA protections. The front-end vendor’s BAA offers little — if any — real safeguard, because the “back end” is outside the scope of that agreement.
• Even if the front-end vendor claims encryption or “secure handling,” the fundamental control and data processing reside with the external model provider, who may store, reuse, or process the data in ways that jeopardize privacy.

In short: many front-end providers offer convenient wrappers — but they are effectively overlaying their brand on external, non-compliant AI engines. For healthcare data, that’s a critical distinction.

Why That Matters: Real Risks for Healthcare Providers

The implications for healthcare organizations are profound:

• If a patient uploads their full medical record to one of these AI agents, the hospital or clinic could face serious liability, especially if the data is used beyond the immediate session (e.g., to train the model).
• Even well-meaning uses — like drafting letters to patients, summarizing notes, or generating care plans — risk exposing PHI to systems not covered by HIPAA as detailed in a recent article “Why doctors using ChatGPT are unknowingly violating HIPAA
• And because many AI-agent vendors don’t disclose they are relying on external models, healthcare leaders may believe they are HIPAA-compliant when they’re not.

As legal guide Bricker Graydon puts it, “properly de-identifying health data before uploading to ChatGPT can mitigate risk — but many users don’t do that, and even then, outputs should be reviewed by a human expert.”

The Solution? American Health Connection (AHC): A Different Approach

This is where AHC stands apart. AHC offers a full-service, outsourced virtual call center and patient communication solution — with a strong emphasis on secure, compliant, human-centered operations.

• AHC combines human agents (Patient Service Representatives) with select AI and automation tools — like call analytics, virtual assistants, and proactive outreach — but always under carefully monitored, human-supervised workflows. You can learn more in our blog post “9 Good Reasons to Outsource Your Healthcare Call Center”
• AHC focuses on centralized scheduling, appointment reminders, intake calls, insurance verification, and other administrative support — functions that do not require uploading detailed PHI into a general-purpose LLM.
• Because AHC builds and manages its own patient-communication infrastructure — rather than relying on third-party LLM APIs — healthcare providers can feel confident that communications are handled under proper compliance frameworks, rather than risking exposure through publicly hosted AI chatbots.

In other words: AHC offers the benefits of modern automation — improved scheduling efficiency, reduced no-show rates, enhanced patient access, 24/7/365 availability — without compromising patient privacy or regulatory compliance.

What This Means for Healthcare Organizations

If you are a healthcare leader evaluating AI-enabled solutions, here are important takeaways:

1. Ask whether a vendor owns or has built its own AI tech. If the answer is “no,” and they rely on third-party LLMs like ChatGPT or Gemini, HIPAA protections are likely void.
2. Treat any system that ingests full patient medical records as high risk. Uploading comprehensive PHI to non-HIPAA-compliant models is virtually a guaranteed compliance violation.
3. Prioritize vendors that combine human oversight with purpose-built communication infrastructure. Companies like AHC demonstrate that you don’t need public LLMs to modernize scheduling and patient outreach — and still stay compliant.
4. Don’t confuse branding with compliance. Just because a vendor markets itself as an “AI agent” doesn’t mean it’s safe. The underlying architecture matters far more than the label.

Offshore AI Models Increase Risk as Well

Another emerging concern is that some healthcare organizations are now experimenting with offshore AI models such as Kimi, a free, China-developed answer engine created to compete with ChatGPT. While these tools may appear attractive due to cost, they are not built with U.S. healthcare regulations in mind. They are known to return highly inaccurate medical information. Beyond the clinical danger of incorrect outputs, these systems operate in foreign jurisdictions, where patient privacy, handling of sensitive data, and backend data processing are completely outside HIPAA protections. For providers, the legal and reputational risks are enormous — simply using one of these engines for scheduling data, insurance information or discharge instructions can expose PHI to an offshore model with no enforceable regulatory compliance.

Summary

The recent Becker’s Hospital Review and New York Times articles are wake-up calls: many “AI agents” used in healthcare today are little more than front-ends built on top of third-party large language models that have no HIPAA obligations. That creates a serious compliance risk for providers and — more importantly — a potential privacy nightmare for patients.

In this landscape, a company like American Health Connection — which invests in its own infrastructure, maintains human agents, and selectively deploys AI under controlled, compliant frameworks — offers a far safer, more responsible path forward. For healthcare organizations committed to protecting patient privacy while embracing efficiency, that distinction matters.

To learn more or set up a no obligation meeting, get in touch with us today