BMVC 26 · 2026

Beyond String Matching - Semantic Evaluation of PDF Table Extraction

Pius Horn, Janis Keuper

Why this publication matters

A table can be extracted with slightly different formatting and still preserve its meaning, or look similar while containing a serious error. This benchmark evaluates that distinction using controlled documents and a semantic assessment checked against human judgments. It gives users a more useful basis for choosing tools that turn PDF tables into reliable data.

Abstract

Reliably extracting tables from PDFs is essential for large-scale scientific data mining and knowledge base construction, yet existing evaluation approaches rely on rule-based metrics that fail to capture semantic equivalence of table content. We present a benchmarking framework based on synthetically generated PDFs with precise LaTeX ground truth, using tables sourced from arXiv to ensure realistic complexity and diversity. As our central methodological contribution, we apply LLM-as-a-judge for semantic table evaluation, integrated into a matching pipeline that accommodates inconsistencies in parser outputs. Through a human validation study comprising over 1,500 quality judgments on extracted table pairs, we show that LLM-based evaluation achieves substantially higher correlation with human judgment (Pearson r=0.93) compared to currently used Tree Edit Distance-based Similarity (TEDS, r=0.68) and Grid Table Similarity (GriTS, r=0.70). Evaluating 21 contemporary PDF parsers across 100 synthetic documents containing 451 tables reveals significant performance disparities. Our results offer practical guidance for selecting parsers for tabular data extraction and establish a reproducible, scalable evaluation methodology for this critical task.

Abstract source ↗

Figures

Semantic table-evaluation examples, human-judgment correlations, and the synthetic PDF benchmark. — **Figure 1.** Rule-based metrics both over-reward semantically broken outputs and penalize benign representational variation (a); an LLM-as-a-judge approach validated on 1,554 human ratings aligns substantially better with human judgment (Pearson r=0.93 vs. r=0.68 for TEDS) (b); and used to benchmark 21 parsers on 100 synthetic PDFs that embed 451 real arXiv tables with their LaTeX source as exact ground truth (c). View in source ↗

**Figure 2.** Overview of the benchmark generation pipeline: arXiv LATEX tables are collected, cleaned, and validated, then placed alongside filler text under a sampled layout, yielding a PDF page paired with a JSON ground truth. View in source ↗

Cite this paper

Download .bib

@inproceedings{horn2026beyondstringmatching99,
  title = {{Beyond String Matching - Semantic Evaluation of PDF Table Extraction}},
  author = {Pius Horn and Janis Keuper},
  booktitle = {British Machine Vision Conference},
  year = {2026},
  url = {https://openreview.net/pdf?id=8KZckKjivl}
}

Figures and abstract are reproduced from the linked research sources. Credit remains with the authors and publishers.