Keuper Labs
← All publications

MUCG Workshop at ECCV 26 · 2026

Combining Fine-Tuning and RAG for Structured Output Prediction on a Multimodal High Velocity Retail Dataset

Bianca Lamm, Janis Keuper

Retrieval-Augmented GenerationVision-Language ModelsStructured PredictionRetail

Why this publication matters

An advertisement shows some product information but leaves other details unstated. This study finds that a fine-tuned model is strongest at reading visible information, while retrieving relevant examples helps with attributes that are missing from the image. Combining the two according to the information needed gives the best overall results, offering a practical design for changing retail catalogs.

Abstract

Domain-specific fine-tuning and retrieval can improve Vision Language Model (VLM) performance on specialized visual tasks. In this work, we study Structured Output Predictions (SOPs) for retail advertisement images with Qwen2.5-VL under Fine-Tuning and Retrieval Augmented Generation (RAG) settings. We evaluate both settings on the multimodal mSOP-765k dataset, which pairs advertisement images with SOPs. The images are cropped from retail leaflets with a frequently changing product assortment, yielding a high velocity dataset. The fine-tuned VLM performs best for information visible in the image, while the Visual RAG Pipeline is more accurate for attributes not shown in the image. A hybrid strategy that generates predictions based on whether the required information is present in the image achieves the strongest overall performance.

Abstract source ↗

Figures

Fine-tuning, visual retrieval, and a hybrid pipeline produce structured predictions from retail advertisements.
Figure 1. Structured Output Predictions (SOPs) from high velocity retail advertisements. We compare three approaches: a Fine-Tuned VLM, the Visual RAG Pipeline [13], and a combination of Fine-Tuning and Visual RAG Pipeline. View in source ↗
A product advertisement paired with a table of its product and promotion data.
Figure 2. Data item of the multimodal mSOP-765k dataset. It consists of an advertisement image and the corresponding structured output of the product and promotion data. View in source ↗

Cite this paper

Download .bib 
@inproceedings{lamm2026combiningfinetuningand96,
  title = {{Combining Fine-Tuning and RAG for Structured Output Prediction on a Multimodal High Velocity Retail Dataset}},
  author = {Bianca Lamm and Janis Keuper},
  booktitle = {MUCG Workshop at ECCV 26},
  year = {2026},
  url = {https://openreview.net/pdf?id=H4fdIUP8Jd}
}

Figures and abstract are reproduced from the linked research sources. Credit remains with the authors and publishers.