Complete PDF OCR: clean markdown with structure and visual content for RAG and LLM pipelines
PDF OCR is a critical bottleneck in AI pipelines. It’s often mentioned in passing, as if it’s a trivial step. Practice shows it’s far from it. Poorly converted PDFs mean garbage-in-garbage-out for downstream AI-system (RAG, …).
When Mistral AI released their state-of-the-art OCR model in March 2025, it opened new possibilities for large-scale document processing. While alternatives like datalab.to and docling.ai offer viable solutions, Mistral OCR delivers exceptional accuracy at a compelling price point.
mistocr emerged from months of real-world usage across projects requiring large-scale processing of niche-domain PDFs. It addresses two fundamental challenges that raw OCR output leaves unsolved:
Heading hierarchy restoration: Even state-of-the-art OCR sometimes produces inconsistent heading levels in large documents—a complex task to get right. mistocr uses LLM-based analysis to restore proper document structure, essential for downstream AI tasks.
Visual content integration: Charts, figures and diagrams are automatically classified and described, then integrated into the markdown. This makes visual information searchable and accessible for downstream applications.
Cost-efficient batch processing: The OCR step exclusively uses Mistral’s batch API, cutting costs by 50% ($0.50 vs $1.00 per 1000 pages) while eliminating the boilerplate code typically required.
In short: Complete PDF OCR with heading hierarchy fixes and image descriptions for RAG and LLM pipelines.
Note
Want to see mistocr in action? This tutorial demonstrates real-world PDF processing and shows how clean markdown enables structure-aware navigation through long documents—letting you find exactly what you need, fast.
import osos.environ['MISTRAL_API_KEY'] ='your-key-here'os.environ['ANTHROPIC_API_KEY'] ='your-key-here'# for refine features (see Advanced Usage for other LLMs)
Complete Pipeline
Single File Processing
Process a single PDF with OCR (using Mistral’s batch API for cost efficiency), heading fixes, and image descriptions:
from mistocr.pipeline import pdf_to_mdawait pdf_to_md('files/test/resnet.pdf', 'files/test/md_test')
Saved descriptions to /tmp/tmp62c7_ac1/resnet/img_descriptions.json
Adding descriptions to 12 pages...
Done! Enriched pages saved to files/test/md_test
This will:
OCR the PDF using Mistral’s batch API
Fix heading hierarchy inconsistencies
Describe images (charts, diagrams) and add those descriptions into the markdown Save everything to files/test/md_test
To print the the processed markdown, you can use the read_pgs function. Here’s how:
Then to read the fully processed document:
from mistocr.pipeline import read_pgsmd = read_pgs('files/test/md_test')print(md[:500])
# Deep Residual Learning for Image Recognition ... page 1
Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun<br>Microsoft Research<br>\{kahe, v-xiangz, v-shren, jiansun\}@microsoft.com
## Abstract ... page 1
Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, ins
By default, read_pgs() joins all pages. Pass join=False to get a list of individual pages instead.
Advanced Usage
Batch OCR for entire folders:
from mistocr.core import ocr_pdf# OCR all PDFs in a folder using Mistral's batch APIoutput_dirs = ocr_pdf('path/to/pdf_folder', dst='output_folder')
Custom models and prompts for heading fixes:
from mistocr.refine import fix_hdgs# Use a different model or custom promptfix_hdgs('ocr_output/doc1', model='gpt-4o', prompt=your_custom_prompt)
Custom image description with rate limiting:
from mistocr.refine import add_img_descs# Control API usage and customize descriptionsawait add_img_descs('ocr_output/doc1', model='claude-opus-4', semaphore=5, # More concurrent requests delay=0.5) # Shorter delay between calls
For complete control over each pipeline step, see the core, refine, and pipeline module documentation.
Known Limitations & Future Work
mistocr is under active development. Current limitations include:
No timeout on batch jobs: Jobs poll indefinitely until completion. If a job stalls, manual intervention is required.
Limited error handling: When batch jobs fail, error reporting and recovery options are minimal.
Progress monitoring: Currently limited to periodic status prints. Future versions will support callbacks or streaming updates for better real-time monitoring.
Contributions are welcome! If you encounter issues or have ideas for improvements, please open an issue or discussion on GitHub.
Developer Guide
If you are new to using nbdev here are some useful pointers to get you started.
Install mistocr in Development mode
# make sure mistocr package is installed in development mode$ pip install -e .# make changes under nbs/ directory# ...# compile to have changes apply to mistocr$ nbdev_prepare
Documentation
Documentation can be found hosted on this GitHub repository’s pages. Additionally you can find package manager specific guidelines on conda and pypi respectively.