Transactions on Machine Learning Research (TMLR) / presented at ICML 26 · 2026
mSOP-765k - A Benchmark For Multi-Modal Structured Output Predictions
Why this publication matters
Turning a product advertisement into reliable database fields requires understanding images, words, and numbers together. mSOP-765k provides more than 765,000 annotated examples and a common evaluation framework for that task. It makes the remaining gap between impressive demonstrations and accurate structured extraction easier to measure.
Abstract
This paper introduces mSOP-765k, a large-scale benchmark for the evaluation of multi-modal Structured Output Prediction (mSOP) pipelines. Besides novel evaluation metrics, the benchmark provides combined training and test datasets with over 765,000 images taken from real-world product advertisements. Each of these images contains product visualizations, textual information like product name or brand, and numerical data such as product weight, price, and discount. All images are annotated with the corresponding structured information in form of dictionaries containing key-value pairs. An initial baseline evaluation, including various LLMs and VLMs, as well as multi-modal RAG approaches, shows that the proposed benchmark provides a challenging problem which can not yet be solved completely by state-of-the-art mSOP methods. The benchmark and dataset are available under a creative-commons license: https://huggingface.co/datasets/retail-product-promotion/mSOP-765k.
Figures
Additional figures and captions will be added when the full paper is available.
Cite this paper
@article{lamm2026msop765ka78,
title = {{mSOP-765k}: A Benchmark For Multi-Modal Structured Output Predictions},
author = {Bianca Lamm and Janis Keuper},
journal = {Transactions on Machine Learning Research},
year = {2026},
url = {https://openreview.net/forum?id=H7eYL4yFZS},
issn = {2835-8856},
note = {J2C Certification}
}
Figures and abstract are reproduced from the linked research sources. Credit remains with the authors and publishers.