For years, the promise of artificial intelligence in medicine has been framed as the ultimate equalizer. We were told that algorithms, devoid of human prejudice, would diagnose diseases faster, predict patient risks more accurately and distribute life-saving resources based on objective need rather than the subjective whims of a provider. In theory, a machine does not see a patient’s bank balance or their zip code; it sees data.
However, the reality emerging from clinical settings is far more troubling. Rather than erasing human bias, AI is often acting as a mirror, reflecting and amplifying the systemic inequalities already embedded in our healthcare systems. For the world’s most vulnerable populations, particularly those living in poverty, this “objective” technology is creating a new, invisible barrier to care. AI discrimination in healthcare for low-income patients is not a glitch in the system; it is often a direct result of how these systems are designed and the data they are fed.
As an editor who has spent over a decade covering the intersection of human rights and geopolitics, I have seen how technology can either liberate or marginalize. In the case of medical AI, we are seeing a dangerous trend where the “digital divide” is translating directly into a “health divide.” When algorithms are tasked with deciding who gets extra care or who is prioritized for a transplant, they are frequently relying on proxies that penalize the poor, effectively automating neglect.
The core of the problem lies in the distinction between “health need” and “healthcare utilization.” In many advanced AI models, the system is not programmed to look for the severity of a disease, but rather for the amount of money spent on a patient’s care. The logic is superficially simple: the sicker a person is, the more they spend on doctors, tests, and medications. High spending equals high need. But this logic collapses when applied to low-income populations who may be desperately ill but cannot afford the care that would generate those data points.
The Proxy Trap: How Spending Masks Sickness
The most stark illustration of this failure comes from a landmark study published in the journal Science, which analyzed a widely used healthcare risk-prediction algorithm. The researchers discovered that the AI consistently underestimated the health needs of Black patients compared to white patients with the same chronic conditions. The reason was systemic: because of long-standing socioeconomic disparities, less money was spent on Black patients on average, even when they were sicker. The AI interpreted this lower spending as a sign of better health, thereby denying thousands of high-risk patients access to care-management programs designed to prevent hospitalization (Science, 2019).
This phenomenon is known as “proxy bias.” A proxy is a piece of data used to represent something else—in this case, using “cost” as a proxy for “health.” For a wealthy patient, a high number of specialist visits is a clear signal of illness. For a patient living below the poverty line, the absence of those visits is not a signal of health, but a signal of a lack of insurance, transportation, or childcare. When AI uses these proxies, it creates a feedback loop: the poor are seen as “healthier” because they cannot afford care, so they are denied the extremely resources that would help them access that care.
This is not limited to a single algorithm or a single country. Across the globe, the integration of predictive analytics into public health is risking the creation of a two-tier system. In one tier, the affluent receive proactive, AI-driven preventative care. In the other, the poor are managed by systems that only recognize their illness once it reaches a crisis point—usually in an emergency room—because the AI failed to flag them as “high risk” during the preventative phase.
The Socio-Economic Determinants of Health and Data Gaps
To understand why AI discrimination in healthcare for low-income patients persists, we must look at the “training data.” AI learns by identifying patterns in historical data. If the historical data is biased, the AI will be biased. This is often referred to as “garbage in, garbage out.”
Healthcare data is rarely neutral. It is a record of who had access to the system, who was welcomed by providers, and who could afford to seek help. Low-income individuals are frequently underrepresented in the clinical trials and datasets used to train AI. This means the “norm” the AI learns is based on a demographic that is wealthier, healthier, and more likely to have stable housing and nutrition. When the AI encounters a patient from a marginalized background, it may misinterpret their symptoms or fail to account for the socio-economic determinants of health (SDOH)—such as food insecurity or environmental pollution—that exacerbate their condition.
For example, an AI trained on data from urban academic medical centers may not accurately predict outcomes for a patient in a rural “healthcare desert.” The algorithm might suggest a treatment plan that requires frequent follow-up visits, which is functionally impossible for a patient who lacks reliable transportation. By failing to integrate SDOH into its logic, the AI produces “optimal” medical advice that is practically useless for the poor, further widening the gap in health outcomes.
Regulatory Responses and the Fight for Algorithmic Justice
Governments are beginning to realize that algorithmic bias is a civil rights issue. In the United States, the Department of Health and Human Services (HHS) has taken steps to address these disparities. The HHS has emphasized that the non-discrimination requirements of Section 1557 of the Affordable Care Act apply to the use of clinical algorithms, meaning that providers who use biased AI that results in discriminatory care could be in violation of federal law (HHS.gov).
In Europe, the approach is more structural. The European Union’s AI Act, the first comprehensive AI law of its kind, classifies AI used in healthcare as “high-risk.” This designation mandates strict requirements for data quality, transparency, and human oversight. Under these rules, developers must demonstrate that their training datasets are representative and that they have implemented safeguards to prevent discriminatory outcomes before the technology can be deployed in a clinical setting (EU AI Act).
However, legislation often lags behind innovation. While the EU AI Act provides a framework, the challenge remains in the enforcement. Auditing a “black box” algorithm—where even the developers cannot fully explain how the machine reached a specific conclusion—is a monumental technical challenge. Without mandatory “algorithmic impact assessments,” the burden of proof remains on the patient to prove they were discriminated against, a task that is nearly impossible for someone without the resources to challenge a medical institution.
Pathways to Equitable AI: Moving Beyond the Proxy
Correcting AI discrimination requires a fundamental shift in how we build medical technology. We cannot simply “tweak” the code; we must change the philosophy of the data.
- Replacing Cost with Clinical Markers: Developers must move away from using financial proxies. Instead of looking at how much was spent on a patient, AI should be trained on clinical markers—such as lab results, vital signs, and documented comorbidities—which provide a more accurate picture of health regardless of income.
- Inclusive Data Sourcing: There must be a concerted effort to include data from community clinics, rural hospitals, and underserved populations in training sets. This ensures the AI understands the diverse ways illness manifests across different socio-economic strata.
- Human-in-the-Loop Oversight: AI should never be the final arbiter of care. A “human-in-the-loop” system ensures that a clinician reviews AI recommendations, specifically looking for signs of bias or impracticality based on the patient’s life circumstances.
- Transparency and Explainability: Patients and providers have a right to know why an AI flagged a person as “low risk.” Moving toward “explainable AI” (XAI) allows doctors to see the factors driving a decision, making it easier to spot when a proxy like “low spending” is unfairly influencing the outcome.
The goal is not to remove the human element from medicine, but to use AI to augment human empathy and clinical judgment. When a doctor knows that an algorithm has a tendency to overlook low-income patients, they can consciously counteract that bias, using the tool as a starting point rather than a final verdict.
What This Means for the Future of Global Health
As AI continues to proliferate in the Global South and in developing healthcare systems, the risks of automated discrimination increase. In regions where healthcare resources are extremely scarce, the temptation to rely on AI for “triage” is high. If these systems are imported from wealthy nations without being recalibrated for local socio-economic realities, they risk institutionalizing a form of digital colonialism, where the “standard of care” is defined by a data set that does not include the local population.
The intersection of poverty and technology is one of the most critical frontiers of human rights in the 21st century. If we allow AI to automate the biases of the past, we are not advancing medicine; we are simply making the machinery of inequality more efficient. The true measure of AI’s success in healthcare will not be how well it serves the healthiest and wealthiest, but how it protects and uplifts those the system has historically ignored.
The fight for health equity is now a fight for data equity. We must demand that the algorithms governing our lives are as transparent as they are intelligent, and as fair as they are fast.
Next Checkpoint: The implementation of the EU AI Act’s high-risk requirements will begin to roll out in phases over the coming years, with specific compliance deadlines for healthcare providers and developers expected to be formalized as the act enters full effect. Further guidance from the US HHS on the enforcement of Section 1557 regarding algorithmic bias is also anticipated as more case law emerges.
Do you believe AI can ever be truly unbiased in a biased world? We invite you to share your thoughts in the comments below or share this article to join the conversation on digital health equity.