Synthetic Data for Computer Vision Workshop at CVPR 2026 · 2026

Fréchet Inception Distance is Failing to Preserve Rank Consistency for Synthetic Out-of-Distribution Samples

Linghui Liu, Henrike Stephani, Janis Keuper

Why this publication matters

Researchers often use one score to decide which image generator is better. This study shows that the ranking can become unreliable when the score relies on visual features learned from a different kind of image. Adapting those features to the target domain produces more consistent comparisons, helping researchers evaluate scientific and other specialized imagery more meaningfully.

Abstract

Fréchet Inception Distance (FID) is widely used to evaluate how well synthetic images match a real target distribution. However, FID is known to depend strongly on the ImageNet-pretrained Inception representation, and it remains unclear whether its similarity rankings remain reliable when data move away from the ImageNet distribution. We study this question through controlled natural-image shifts, non-natural image datasets, and synthetic-to-real comparisons. Across these settings, we find that FID-based rankings become less reliable outside the ImageNet domain, and that this instability is driven mainly by the evaluation feature space. Our controlled shift experiments show that ranking consistency is largely preserved under some augmentations like photometric perturbations, but breaks down under others including Gaussian blur and noise. This instability is mainly caused by the mismatch between the evaluated data and the ImageNet-pretrained feature space. With domain-adapted representations, rankings become much more consistent across backbones and discrepancy metrics, and align better with the underlying data structure. Preliminary synthetic-to-real results further suggest that synthetic data evaluation can benefit from target-domain representations and complementary metrics.

Abstract source ↗

Figures

Three experimental settings compare image similarity rankings under shifts and domain adaptation. — **Figure 1.** Overview of the three experimental settings used to study representation effects on FID-style similarity rankings. Left: For natural images, we first obtain a clean reference ranking in the ImageNet-pretrained feature space. After induced distribution shifts (like Gaussian blur), we compare rankings obtained in the same feature space with those from blur-adapted models to examine whether the clean reference structure is preserved. Middle: For non-natural images, we compare ImageNet-pretrained and domain-adapted feature spaces on medical data using FID, D&C, and MMD. Right: For synthetic images, we use the same metrics to compare synthetic samples against real data in pretrained and domain-adapted feature spaces. View in source ↗

Example image transformations with increasing severity used to test ranking stability. — **Figure 2.** Illustration of the perturbations used in the natural-image experiments. Starting from the original image, we apply six common image transformations and gradually increase their severity to produce controlled distribution shifts. View in source ↗

Cite this paper

Download .bib

@inproceedings{liu2026frechetinceptiondistance85,
  title = {{Fréchet Inception Distance is Failing to Preserve Rank Consistency for Synthetic Out-of-Distribution Samples}},
  author = {Linghui Liu and Henrike Stephani and Janis Keuper},
  booktitle = {Synthetic Data for Computer Vision Workshop at CVPR 2026},
  year = {2026},
  url = {https://openreview.net/pdf?id=jWqcKaZdTS}
}

Figures and abstract are reproduced from the linked research sources. Credit remains with the authors and publishers.