Addressing Bias in VLMs for Glaucoma Detection Without Protected Attribute Supervision

Fairness when the protected attributes are hidden

Glaucoma is a leading cause of irreversible blindness and hits underserved populations hardest. Vision–language models can read fundus images alongside clinical notes, but pretraining data are imbalanced, and these models can perform worse for some demographic groups. Most debiasing methods, FairCLIP among them, need the protected attribute during training. That is often impossible for legal, ethical or regulatory reasons, and such methods usually handle one attribute at a time.

This work debiases a CLIP-style model without any protected-attribute labels. It follows the worst-case (Rawlsian) view of fairness: rather than equalizing known groups, it reduces the loss on the hardest examples, wherever they come from.

Abstract

Vision–Language Models (VLMs) have achieved remarkable success on multimodal tasks such as image–text retrieval and zero-shot classification, yet they can demonstrate demographic biases even when explicit protected attributes are absent during training. In this work, we focus on automated glaucoma screening from retinal fundus images, a critical application given that glaucoma is a leading cause of irreversible blindness and disproportionately affects underserved populations. Building on a reweighting-based contrastive learning framework, we introduce an attribute-agnostic debiasing method that (i) infers proxy subgroups via unsupervised clustering of image–image embeddings, (ii) computes gradient-similarity weights between the CLIP-style multimodal loss and a SimCLR-style image-pair contrastive loss, and (iii) applies these weights in a joint, top-k weighted objective to upweight underperforming clusters. This label-free approach adaptively targets the hardest examples, thereby reducing subgroup disparities. We evaluate our method on the Harvard–FairVLMed glaucoma subset, reporting Equalized-Odds Distance (EOD), Equalized Subgroup AUC (ES-AUC), and Groupwise AUC to demonstrate equitable performance across inferred demographic subgroups.

Surrogate weighting from gradient alignment

1. Focus on the hardest pairs. Instead of averaging the CLIP loss over the batch, take the mean of the top-k per-sample losses. This is a smooth surrogate for minimizing the worst latent group’s loss.

   ℓ_topk-clip = ½ Σ_m [ L_CLIP(x̃_m, t_m) − λ_clip ]₊ / k + λ_clip

   where λ_clip is the k-th largest loss.
2. Score each image pair. Two augmentations of every image give a SimCLR-style contrastive loss ℓ_ctr. For each view m, the weight is the alignment between its gradient and the top-k CLIP gradient, clipped at zero:

   w_m = max{ (∇_θI ℓ_topk-clip)^⊤ ∇_θI ℓ_ctr(x̃_m), 0 }

3. Reweight. The weights are normalized into a distribution and applied to the contrastive loss, so the image pairs whose updates most reduce the hardest CLIP losses get the most emphasis.

Harvard–FairVLMed

10,000 de-identified records from 10,000 patients, each with a fundus image, a clinical note (11–332 words, mean 147) and a glaucoma label. The original split is used: 7,000 train, 1,000 validation and 2,000 test. Demographics are used only for evaluation: 7,690 White, 1,491 Black and 819 Asian patients; 56.3% female; 4.0% Hispanic; 92.5% English-speaking and 1.7% Spanish-speaking.

CLIP ViT-L is fine-tuned on image–note pairs and then evaluated zero-shot with the prompts “This image contains glaucoma” and “This image does not contain glaucoma”. Metrics, all in percent:

• EOD (Equalized Odds Distance, lower is better): the largest gaps in TPR and FPR between subgroups.
• ES-AUC (Equalized Subgroup AUC, higher is better): overall AUC penalized by subgroup disparity.
• Group-wise AUC: AUC within each subgroup.

CLIP vs. Debiased CLIP across six attributes

Debiased CLIP lowers EOD on four of six attributes (ethnicity, language, age, marital status) and raises ES-AUC on five (all but gender). Small minority groups often gain the most. Race and gender get worse on EOD, and gender also loses ES-AUC.

Full table

BibTeX

Copy@article{akash2025debiasedvlm,
  title   = {Addressing Bias in VLMs for Glaucoma Detection Without
             Protected Attribute Supervision},
  author  = {Akash, Ahsan Habib and Murray, Greg and Amireskandari, Annahita and
             Palko, Joel and Laxson, Carol and Bhattarai, Binod and Gyawali, Prashnna},
  journal = {arXiv preprint arXiv:2508.09087},
  year    = {2025}
}

Supported by the Defense Advanced Research Projects Agency under Grant AI-CRAFT AWD16069, and by the National Institutes of Health under Grant P20 GM144230 from the National Institute of General Medical Sciences (NIGMS).