LLM OCR & Layout Analysis
Teaching large language models to actually read a page — and figuring out how to tell if they're reading it correctly.
The Short Version
Since June 2025, I've been part of a Virginia Tech research group studying how large language models read and understand scanned documents. The work centers on layout analysis — figuring out where headers, paragraphs, tables, and figures actually sit on a page — and on benchmarking how well different detection tools agree with each other. It's led to a paper, Improving LLM-Based OCR Through Document Segmentation and Semantic Prompting, accepted to ICTAI 2026.
The Bigger Picture
Classic OCR read a page as a flat string of text and ignored structure. Now that documents get processed by LLMs instead of rule-based pipelines, layout matters again — a model that doesn't know "this is a table" will misread it. Tools like Detectron2, LayoutParser, and MinerU each detect layout differently, and benchmark datasets like PRImA and DocLayNet don't always agree on what "correct" layout even looks like. That disagreement is itself an open research problem, not just an engineering inconvenience.
What I'm Exploring
The parts of the problem I actually spend time on:
- Building a benchmarking pipeline that puts Detectron2, MinerU, and LayoutParser on common ground
- Studying why layout-detection models disagree — how much is real performance difference vs. measurement artifact
- Testing how different prompting strategies affect an LLM's ability to transcribe a document accurately
- Co-authoring the resulting paper alongside undergraduate researchers and a faculty advisor