Patient-Governed AI: The Trust Infrastructure Rare Disease Will Need Next

What This Article Argues

In rare disease, the biggest risk in AI is not that it will have a lack of data to provide insights — it is whether those insights can be trusted.

As AI moves healthcare beyond information and toward interpretation, prediction, guidance, and advice, the stakes rise. In this article, we are specifically considering AI applied to the patient journey and to real-world data and evidence — not AI in rare disease in the abstract.

Key Takeaways

• The AI opportunity is to help patients and communities leverage fragmented data, surface meaningful signals over time, and generate more useful evidence.
• In rare disease, trust shapes participation, evidence quality, and long-term value.
• Governance, transparency, and patient agency are not add-ons. They are essential parts of the infrastructure.

Artificial intelligence has arrived in healthcare. For rare disease communities, the central question is not whether AI will be powerful. It is whether it will be worthy of trust.

That question carries unusual weight in rare disease. Our rare disease communities are small. Their histories are personal. Their data often sits in fragments across hospitals, registries, advocacy groups, research programs, and family-held records. Participation in any new AI model depends less on novelty than on whether patients and families believe it is transparent, accountable, and genuinely aligned with their interests. In a field where trust is hard won and populations are limited in number, getting that wrong is not a side issue. It is an obstacle to participation, weakens evidence, and limits the value of the system itself.

Here, we are not talking about AI in rare disease broadly. We are talking specifically about AI applied to the patient journey and to generation of real-world data and evidence, helping patients and communities leverage fragmented information, surface meaningful changes over time, and generate more useful longitudinal insight for care, research, and community decision-making. The FDA’s patient-focused drug development framework reflects the same reality: patients are uniquely positioned to inform therapeutic context because they are the experts in what it means to live with their condition.¹

What the AI layer is actually for

The AI layer is not primarily about discovering new drugs or diagnosing rare diseases from imaging scans or clinical data. It is about applying AI to the real-world patient journey: organizing fragmented records, registry data, and lived-experience data, surfacing patterns over time, and turning those signals into more useful evidence for care, research, and community decision-making.

That matters because rare disease patients and caregivers often live inside disconnected systems. Clinical data sits in one place. Registry data sits in another. Patient-reported experience may be captured intermittently or not at all. Important signals like changes in function, symptom burden, treatment tolerability, and quality of life often emerge between visits, not during them.

Applied effectively, AI could help reduce that fragmentation. It could help patients make better sense of their own history, help communities identify patterns earlier, and help generate more useful longitudinal evidence around what actually matters in daily life. But that promise only becomes real if patients and communities trust how the system works, what it is doing with their data, and whose interests it is built to serve.

Why trust matters more now

In the early internet era, patients went online looking for health information but were often left to navigate sources of uneven quality and credibility on their own.¹² Social media changed the landscape again, adding communities, anecdotes, influencers, and algorithmic amplification to how people encountered health perspectives. Now AI is pushing that evolution further. Patients are no longer just finding information or encountering influence; they are increasingly receiving summaries, interpretations, and medical advice-like guidance. In healthcare, that raises the stakes. The challenge is no longer only whether information is available. It is whether the systems shaping decisions are trustworthy.

U.S. policy is beginning to reflect the same shift. In March 2024, the U.S. Department of Health and Human Services (HHS’s) Assistant Secretary for Technology Policy / Office of the National Coordinator for Health Information Technology (ASTP/ONC) put new AI transparency requirements into effect for certified health IT, while NIST’s AI Risk Management Framework reinforced the importance of accountability, transparency, explainability, privacy, and safety.^2,4 These are important signals, but they describe a floor, not a ceiling. For rare disease, the governance question goes well beyond regulatory compliance.

Rare disease has a trust problem and an AI opportunity

Healthcare technology discussions often begin with capability: What can a system predict? What can it summarize? What can it automate? Those are useful questions. But in rare disease, they are not usually the first questions patients and families ask.

More often, the experience is this: a family shares information with a specialty clinic, enters data into a registry, responds to an advocacy survey, repeats the same history to a new provider, and still has little visibility into how that information connects, who learns from it, or whether the community benefits in a meaningful way. That is not only a data-fragmentation problem. It is a trust-design problem.

For many highly engaged rare disease families, this is a familiar routine. Data is collected across multiple institutions, each with its own systems, consents, and governance. The family’s role remains largely the same: provide information, sign consent, and hope it leads somewhere useful.

That is the environment into which AI is now being introduced. If an AI system enters this picture without addressing the underlying trust architecture, it risks compounding the problem rather than solving it. If governance remains fragmented, transparency is absent, and families still have little visibility into how their data informs the intelligence being generated, the structural pattern has not changed. It has only been automated.

Trustworthiness matters here not only as an ethical principle, but as a practical one. As described in a recent article by Kraft and colleagues, “Trustworthiness is not only an intrinsic ethical value, but is also of instrumental value, in that it can increase research participation and improve the public’s perception of research.”⁹ In rare disease, where participation is foundational to evidence generation, that point carries real weight.

Patients and advocacy groups are not inputs. They are governance partners.

Rare disease patients are not just sources of information. Their lived experience, judgment, and continued engagement shape whether any evidence or intelligence layer will work over time. The FDA’s patient-focused drug development program explicitly centers patients’ experiences, needs, and priorities in development and evaluation, while patient-centered real-world evidence guidance increasingly calls for patients and patient representatives to help shape research questions, study protocols, and dissemination.^1,7

As former FDA commissioner Janet Woodcock put it, “Patients are true experts in their disease.” That should not be a slogan attached to rare disease AI. It should be a design principle.

The same is true at the community level. In rare disease, advocacy organizations often serve as trusted conveners across the ecosystem. They understand where care breaks down, what families are navigating, and how the community has been engaged over time. As EURORDIS notes, independent patient groups play a crucial role both in direct support and in improving conditions for the wider rare disease community.¹¹

For AI, that means advocacy organizations should be involved upstream — where the model’s purpose, guardrails, permissions, and definitions of value are set. Governance is not a communications task added at the end. It is part of the operating model.

Strong governance reduces the risk of AI extracting benefit and value from rare disease communities

The distinction between extractive AI and governed AI is critical. It shows up in the practical moments rare disease families navigate every day.

It shows up when families are asked to share the same records across registries, studies, and providers without understanding how those contributions connect. It shows up when care transitions reset years of history. It shows up when treatment decisions must be made with limited real-world evidence about what happens outside the clinical trial. It shows up when patients contribute to research but hear little about what was learned. And it shows up between visits, when the signals that matter most in daily life often never enter the formal record.

In a governed model, those moments would look different. Families would have clearer visibility into how data is used. Longitudinal history would be more continuous. Evidence would reflect the questions communities actually care about. Participation would be designed to return value to families, not just capture value from them.

Without intentional governance, AI systems can extract enormous value from patient communities while returning very little to them. The pattern is familiar from other industries: a platform provides a better service, but the economic and strategic value it generates flows upward to the platform operator rather than staying within the community that made it possible. In rare disease, the risk is the same. If patient data, lived experience, and community participation power an AI layer that patients and advocacy groups do not govern, the scientific, economic, and strategic value created through that collaboration can be captured by others. Governance is what ensures that the value generated by the community is directed back to the community, enabling its growth, strengthening its advocacy, and increasing its benefit to patients over time.

None of this requires futuristic technology, but it requires immediate design choices.

The best rare disease AI will feel less like surveillance and more like support

The most valuable AI in rare disease will not simply observe patients more efficiently. It will help support them more effectively.

ASPE’s recent work on data and health information for patient empowerment emphasizes that digital tools can promote patient empowerment by giving people access to their data, facilitating shared decision-making, and supporting more personalized care.⁸ That is a more useful framing for rare disease than the familiar AI storyline centered only on efficiency or automation.

Patients do not need another opaque system operating around them. They need tools that help them navigate care, understand options, contribute their experience meaningfully, and remain connected to a trusted community and evidence framework over time. Providers do not need black boxes they cannot evaluate. They need support that is transparent enough to assess and practical enough to use. Advocacy organizations do not need another technology layer that extracts participation while centralizing value elsewhere. They need models that respect patient agency, align with community governance, and create visible benefit for the ecosystem itself.

The scientific and economic value created through the collaboration between the patient organizations, patients, clinicians and researchers must be directed back to the community to enable its growth and increase its benefit to patients over time. Intentional design of trust and community governance retains these elements for stakeholders to leverage rather than outsource AI but risk extraction of these essential opportunities.

This is why patient-governed AI is not a branding phrase. It is a practical design requirement.

Six questions that will decide whether AI earns trust in rare disease

If rare disease wants AI to create durable value, these are the questions that need to move to the center of the conversation:

1. 1.Who governs how patient data is used in AI systems?
2. 2.What permissions are truly meaningful to patients, not just in theory, but in practice?
3. 3.What role should advocacy organizations play in defining appropriate use?
4. 4.How much transparency is enough to earn trust?
5. 5.Who benefits when patient data becomes intelligence, and how is that value returned to patients and communities?
6. 6.How do we ensure that AI strengthens patient agency rather than weakening it?

These are not technical side questions. They are the questions that will shape legitimacy, adoption, and long-term value.

What this can look like in practice

These questions are not only theoretical. They are already shaping how some rare disease AI models are being built.

AiRare, by Lumiio, is one example. It begins from the premise that governance, patient agency, and community-defined value should be built into the model from the start — not added later as safeguards. In practice, that means advocacy organizations helping shape acceptable use, patients having better access to and context for their own information, and evidence being built around what communities identify as meaningful over time.

This is not the only way to build AI for rare disease. But it reflects the larger argument of this article: in rare disease, governance is not an add-on. It is part of the infrastructure that makes trust, participation, and durable value possible.

The real opportunity

Rare disease has an opportunity to help define a better path.

Because rare disease communities are smaller, more connected, and often more practiced in questions of trust, governance, and shared purpose, they may be especially well positioned to show what responsible AI looks like when patient participation is treated as something to earn, not assume.

The most important AI systems in health will not simply be the ones that process the most data. They will be the ones that earn trust, preserve agency, and create value in ways patients and communities can actually recognize.

That is the real opportunity in rare disease: to build systems that are not only intelligent, but permissioned; not only predictive, but transparent; not only technically interoperable, but community-governed.

Patients and families with rare diseases already live with enough complexity. The next generation of health AI should reduce that burden, not deepen it. It should help create a model of progress that communities can believe in and choose to be part of.

That is the trust infrastructure rare disease will need next.