AI Agents in Digital Health Are Not Neutral — And That's the Problem

A few weeks ago, someone on a product call I had the chance to sit in on said something that I haven't been able to shake. They were talking about possibly adding an AI agent for an intake flow for a healthcare app. The framing was efficiency. However, nobody in that room, including me, was asking what “efficient” felt like to the patient on the other end. The app in question was being tailored through the lens of LGBT+ patients and what followed was the reminder that the same tool that removes friction for one population can compound harm for another.

The Population You're Building For Has Already Been Failed

Before configuring a single model or making a single design decision, you need to understand the reality many LGBT+ patients are coming into any health tech product with.

Many have already experienced discrimination in clinical settings — misgendering, invasive identity-specific questioning that crowds out the broader health conversation that should be happening, and providers who were simply never trained to care for this population. Many are not, or cannot be, out in all areas of their lives, which means they may self-censor during intake. Not because they don't want help, but because past experiences have taught them that disclosure carries real risk.

You are not dropping a tool into a neutral environment. You are introducing it into a relationship that may already be defined by mistrust. That context shapes everything about how your AI agent needs to be designed.

The Double-Edged Reality of AI in This Context

AI can do something genuinely useful here. It can act as a safe intermediary before a human enters the session.

A well-designed AI agent can capture identity information once and carry it forward, reducing the exhausting repetition of explaining yourself to every new provider. But this only works as a feature if the patient controls it. For many LGBT+ patients who are not out in every area of their life, a system that automatically forwards sensitive identity data across care settings isn't reducing burden. It's creating exposure. The architecture has to reflect that reality. Patients need to understand exactly what gets shared, with whom, and when. They also need a meaningful way to limit that. Consent here isn't a terms-of-service checkbox. It's the difference between a tool that feels safe and one that feels like a liability.

Unlike a static intake form, an AI agent can also explain in real time why it's asking what it's asking. That shift matters more than it might seem. There's a meaningful difference between a form that asks about current medications including HRT and an agent that says: we're asking about your current medications, including hormone therapy, to check for potential interactions with any new prescriptions. One feels like surveillance. The other feels like care.

But here is where the double edge appears. The same technology that can create a safer intake experience can also systematically harm the patients you're trying to serve, if the underlying model was never built with them in mind.

Bias Doesn't Disappear When You Deploy a Model. It Scales.

Most large clinical datasets underrepresent LGBT+ patients. In a model trained on those datasets, this population may register as statistical noise rather than as a distinct group with distinct clinical needs. The model isn't intentionally excluding them. It's just optimizing for the majority. And in health tech, that can look like a diagnosis driven by a patient's stated gender rather than their presenting symptoms, because the model was never taught to separate the two.

Bias in these models is not always the result of bad intentions. It's often the result of insufficient attention. The humans who build these systems carry assumptions, conscious and unconscious, and models have a tendency to amplify those assumptions rather than correct for them.

Build the Safeguards In. Don't Add Them Later.

What struck me on that call wasn't what was missing from the conversation about safeguards. It was how easy it would be for the gaps to get there. Not through negligence. Through prioritization. Auditing gets scheduled for after launch. Provider training becomes a phase two item. The work that protects the patient is often the first to move when something else doesn't fit the timeline.

Representative training data is the one everyone agrees on in the room and the first thing that gets scoped down when the dataset costs more than the budget assumed.

Auditing is the one that gets framed as ongoing — which in practice means it gets revisited at the next planning cycle, and then the one after that. By the time a patient surfaces the problem, you're not preventing harm anymore. You're responding to it.

Keeping a human in the loop is the right instinct, but it rests on an assumption we've already challenged — that the human is the safer option. For many LGBT+ patients, the harm didn't come from an algorithm. It came from a provider who wasn't trained or wasn't aware. A biased human reviewing the output of a biased model doesn't create a safeguard. It creates a second opportunity for the same mistake. Human oversight only works as a check if the humans providing it have the cultural competency to catch what the model gets wrong. That means your HITL strategy has to be paired with genuine investment in provider training for interacting with this population.

Explainability isn't a nice-to-have in this context. It's the mechanism by which every other safeguard actually functions. If the model can't show its reasoning, the data points it used, and the logic it applied, you have no way to audit for bias, no way to catch when it's reverting to heteronormative assumptions, and no way to build the patient trust that the whole system depends on. A black box recommendation in a general consumer app is a product problem. In a clinical setting serving a population that has historically been misdiagnosed and underserved, it's an ethical one.

Trust Is Not a Feature You Ship

It gets built by involving LGBT+ patients, the clinicians, and advocates who work alongside them in the design process before the product ships, not in a feedback session after. It gets built by sharing your bias audit results openly, including the ones that show where the model fell short. It gets built by giving patients a genuine mechanism to flag when something felt wrong and showing them what you did about it. 

What you're trying to fix isn't just a system problem. It's a relational one. You don't fix a relational problem with a better onboarding flow.

The patients you're building for have already decided, more than once, that it wasn't safe to be honest with a system that was supposed to help them. What you build either earns a different answer or confirms the one they already had.