Do Composed Image Retrieval Benchmarks Require Multimodal Composition?
By Matteo Attimonelli, Alessandro De Bellis, Aryo Pradipta Gema, Rohit Saxena, Monica Sekoyan, Wai Chung Kwan, Claudio Pomo, Alessandro Suglia, Dietmar Jannach, Tommaso Di Noia, and Pasquale Minervini, May 14, 2026
In NeurIPS 2026
Composed image retrieval asks a model to find a target image from a reference image and a textual modification. Although benchmark success is intended to measure composition across both inputs, this work finds that many queries can instead be solved from only one modality.
The study audits four established benchmarks using eleven generalist multimodal embedding models, then applies human validation to queries that resist the identified shortcuts. Of 4,741 shortcut-free queries reviewed, 1,689 are judged well-formed; common defects include ambiguous edits and targets that do not match the requested modification.
On the validated subset, single-modality retrieval no longer suffices and successful systems rely more strongly on both image and text. The results show that existing benchmark scores mix genuine multimodal composition with unimodal shortcuts and annotation noise, potentially overstating compositional capability.
Paper: https://arxiv.org/abs/2605.14787
Stay ahead with research-backed solutions
From papers to production, we translate cutting-edge AI research into practical systems that give your business a competitive edge.
Book a Consultation