Explainable AI for Vision-Language Models
If a model can read a document, can we tell why it got a word wrong?
The Short Version
Alongside the OCR benchmarking work, I've been digging into explainability — trying to understand what's actually happening inside a vision-language model as it processes a document, rather than just measuring whether its final answer is right or wrong.
The Bigger Picture
As models get bigger and more capable, understanding why they produce a given output gets harder — and more important, especially anywhere a wrong answer has real consequences. Explainable AI (XAI) is the umbrella term for techniques that try to open that black box: saliency and heatmap-based attribution methods that highlight what a model "looked at," probing techniques that test what information is encoded in a given layer, and mechanistic interpretability, which tries to reverse-engineer what individual neurons or attention heads are actually doing.
What I'm Exploring
Where this intersects with the OCR research:
- Reviewing existing mechanistic interpretability literature to see which techniques transfer to document-understanding models
- Building heatmap-based visualizations of a model's attention as it moves through preprocessing, the vision tower, and the language model
- Tracing where in that pipeline a model's understanding of a document starts to diverge from the ground truth
- Treating this as exploratory groundwork that runs alongside the layout-analysis benchmarking work