When Medical VLMs Stop Understanding

Motivation

Medical vision–language models are typically validated with image–text retrieval (R@k on ROCO or MedICaT) and zero-shot classification (AUROC on CheXpert-style label sets). These aggregate scores can conceal elementary failures. In our analysis, MedCLIP assigns a higher similarity to the text “acoustic guitar” than to “chest x-ray” for a frontal chest radiograph, despite competitive retrieval performance on chest radiographs.

Abstract

Medical vision–language models (VLMs) are attracting growing interest as a foundation for clinical image interpretation, retrieval, and decision support. However, current evaluations largely emphasize aggregate accuracy, retrieval, or classification performance, offering limited insight into whether these models possess the fine-grained semantic understanding required for safe and reliable use in high-stakes clinical settings. In this work, we introduce MedTEC-Bench, a diagnostic benchmark for probing semantic understanding in medical VLMs across five levels of increasing specificity, from broad modality recognition to fine-grained findings and negation. MedTEC-Bench combines controlled semantic probes with a suite of metrics designed to reveal failures hidden by aggregate benchmarks. Evaluating a diverse set of VLMs, from broad biomedical models to clinical-specialist models, we find that existing systems often fail on basic yet clinically meaningful semantic distinctions, despite strong reported performance on standard tasks. Our results suggest that current medical VLM evaluations may overestimate vision-grounded clinical understanding and highlight the need for targeted semantic benchmarks before such models are deployed in high-stakes medical workflows.

Summary of findings

1. Two regimes of failure. Chest-radiograph specialists (MedCLIP, CheXzero, BioViL) fail at the domain level: they do not reliably separate chest radiographs from everyday objects. Broad-biomedical and general-domain models pass the coarse levels but fail on fine-grained findings and negation.
2. Negation fails universally. No evaluated model reaches meaningful negation discrimination on any modality.
3. Standard protocols disagree. Retrieval, zero-shot AUROC, and MedTEC probing produce different model rankings, including on identical images.

Probe taxonomy

For a fixed anchor image, each probe pairs four correct descriptions with eight to ten incorrect ones that differ along a single semantic axis. Length, syntax, and template are matched between positives and negatives, so a failure cannot be attributed to surface form. Levels are ordered by difficulty; the first level at which a model collapses characterises its text-side competence ceiling.

Discrimination Score

Let v be the ℓ2-normalised image embedding, and t⁺ and t⁻ the normalised positive and negative text embeddings. The Discrimination Score is

DS = max_i cos(v, t⁺_i) − max_j cos(v, t⁻_j)

That is, the best-matching correct description must exceed the strongest distractor. DS ≤ 0 is recorded as a probe failure (collapse). Scores are averaged over per-concept template variants, and bootstrap 95% confidence intervals are computed over images (B = 10,000, seed 42).

Classification of results

Discrimination scores by modality

Each spoke is a model and each polygon a MedTEC level. Two patterns hold across modalities. The L2 and L3 polygons separate model families: broad-biomedical and general-domain models score above zero, while out-of-domain specialists do not. The L4 and L5 polygons contract toward the zero circle in every modality, irrespective of model family.

Per-level results on chest radiographs

We examine chest radiography in detail because it has the most established retrieval and zero-shot benchmarks and the largest set of specialist models.

Model rankings under three protocols

We rank the nine chest-radiograph models under ROCO-CXR retrieval, NIH ChestX-ray14 zero-shot AUROC, and MedTEC L4. Retrieval favours models trained on PubMed figure captions; zero-shot AUROC favours models fine-tuned on the closed CheXpert label vocabulary. Neither tests finding-level discrimination directly.

The disagreement persists when all three protocols are applied to the same OpenI images. CXR specialists lead retrieval (BioViL R@50 = 12.9% vs. BiomedCLIP 4.5%) and zero-shot AUROC (BioViL 0.708), while BioViL also shows the deepest probing collapse (L4 −0.219, L5 −0.167). MedTEC therefore measures a property that neither standard protocol captures.

Abdomen CT with disease labels

Our initial CT probes used AMOS-2022, which provides organ segmentation labels only; its L4 and L5 would therefore probe organ identity rather than findings and negation. We rebuilt the modality from the Merlin abdominal CT dataset (Stanford AIMI; 30 report-derived abnormality labels) with six single-label classes: hepatic steatosis, renal cyst, gallstones, ascites, pleural effusion, and no finding (150 each, n = 900). We additionally evaluate MERLIN, a volumetric CT model, on its own training distribution.

Robustness to prompt formulation

We repeated L4 and L5 under four phrasing styles, keeping the anchor image, competing findings, and probe structure fixed: the original template captions, report-style sentences (Findings/Impression), ontology terms following RadLex and UMLS, and alternative clinical paraphrases. Of the 63 cells that collapse under the template, 62 remain below the 0.05 boundary under every style; style accounts for roughly 0.01–0.04 SD per cell.

Negation subtypes

The original L5 used a single negation form. We decomposed it into four subtypes following NegEx, ConText, and the radiologist-certainty lexicon of Panicek and Hricak: definite (“pneumonia excluded”), hedged (“possible pneumonia”, “cannot be excluded”), ruled-out (“no radiographic evidence of pneumonia”), and historical (“prior pneumonia, now resolved”). Hedged phrasing is collapse-certain in 38 of 46 model × modality cells (83%), compared with 52% for definite and 50% for ruled-out negation.

Downstream task

On the 1,191 closed yes/no questions of VQA-RAD, a dual encoder answers by scoring an affirmative statement against its negation, which is an L5 contrast on an externally authored dataset. Balanced accuracy across ten models ranges from 0.484 to 0.530 (chance 0.500), only two models exceed chance, and the yes-rate varies from 0.164 to 0.766, indicating that answers follow a fixed polarity prior rather than the image.

Demographic subgroup analysis

All subgroup analyses use ground-truth attributes without imputation and the same DS protocol. Sex and age use HAM10000 disease labels joined to ISIC-2019 metadata (n = 1,000; L1–L5). Skin tone uses the MSKCC Skin Tone Labeling Dataset (ISIC collection 413; 300 images per Fitzpatrick group). Because all lesions in that set are benign, skin-tone analysis covers L1–L3.

Sensitivity to sample size

We subsampled per-image scores at n = 100, 250, 500, and 1,000 with fixed prompts, seed, and bootstrap settings. Point estimates are stable from n ≈ 250; larger samples mainly narrow the confidence intervals. At n ≈ 1,000 the intervals are narrow enough to separate collapse from near-chance behaviour for |DS| ≈ 0.001–0.005.

We also report precision@k alongside recall@k for retrieval. With one gold caption per image, P@k = R@k / k, so the two metrics yield identical rankings.

Benchmark construction

Anchor images are stratified by label and restricted to single-label cases. This restriction is necessary because L4 and L5 negatives name other findings; with a multi-label anchor, a nominal negative could be a true description. Each anchor set is produced by a deterministic script (seed 42) that writes a manifest with per-image provenance.

Negatives are drawn from clinical vocabularies rather than constructed adversarially: other imaging modalities of the same anatomy, named following RadLex (L2); the same modality applied to a different anatomical region (L3); the remaining labels of the modality's diagnostic vocabulary, e.g. CheXpert for chest radiographs (L4); and ten negation constructions (L5). This follows the grounded-negative practice of Winoground, ARO, and SugarCrepe.

Relation to existing benchmarks

Scope and limitations

MedTEC-Bench probes the text side of dual-encoder medical VLMs. Image-side failures are a separate axis, and the conclusions do not transfer directly to generative or cross-attention architectures. The negation levels cover a range of constructions but do not exhaust clinical phrasing. The study is diagnostic; establishing how L4 and L5 performance maps to specific downstream applications, and how each failure regime responds to mitigation, is left to future work.

Citation

Copy@inproceedings{akash2026medtec,
  title     = {When Medical VLMs Stop Understanding: MedTEC-Bench for Probing Semantic Specificity},
  author    = {Akash, Ahsan Habib and Devkota, Alina and Adjeroh, Donald A. and Bhattarai, Binod and Gyawali, Prashnna Kumar},
  booktitle = {Advances in Neural Information Processing Systems (Evaluations and Datasets Track)},
  year      = {2026}
}