ICPR 2026 · 2026
Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs
Why this publication matters
A mathematical expression can be represented in several ways without changing its meaning. This benchmark evaluates whether PDF extraction preserves that meaning, rather than merely reproducing a matching character string. It helps researchers choose tools that retain the scientific content needed for downstream analysis.
Abstract
Correctly parsing mathematical formulas from PDFs is critical for training large language models and building scientific knowledge bases from academic literature, yet existing benchmarks either exclude formulas entirely or lack semantically-aware evaluation metrics. We introduce a benchmarking framework centered on synthetically generated PDFs with precise LaTeX ground truth, enabling systematic control over layout, formulas, and content characteristics. For evaluation, we apply LLM-as-a-judge to assess semantic equivalence of parsed formulas, capturing mathematical meaning beyond surface-level notation differences. We validate this approach through a human study (250 formula pairs, 750 ratings from 30 evaluators), showing a Pearson correlation of r=0.78 with human judgment, compared to r=0.34 for character-level matching (CDM) and r≈0 for text similarity. Our robust two-stage matching pipeline combining LLM-based extraction with fuzzy validation reliably aligns parsed formulas with ground truth despite format inconsistencies across parsers. Evaluating 20+ contemporary PDF parsers across 100 synthetic documents with 2,000+ formulas reveals significant performance disparities, providing actionable guidance for practitioners selecting parsers for downstream applications.
Figures
Cite this paper
@inproceedings{horn2026benchmarkingdocumentparsers87,
title = {{Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs}},
author = {Pius Horn and Janis Keuper},
booktitle = {International Conference on Pattern Recognition},
year = {2026},
url = {https://arxiv.org/pdf/2512.09874}
}
Figures and abstract are reproduced from the linked research sources. Credit remains with the authors and publishers.