Local OCR and Document AI: Self-Host baidu Unlimited-OCR for Private Document Parsing

Document parsing is the one AI task where the privacy argument writes itself. The pages people most want to OCR are invoices, receipts, contracts, medical forms and bank statements, which is the exact pile of paper you least want to upload to someone else's server and pay per page to read. Cloud Document AI from Azure and Google works well, but it bills per page and every document leaves your network. baidu's Unlimited-OCR, open-sourced in June 2026 under the MIT license, is the first credible open-weight model that makes self-hosting this category practical. This guide covers what it is, where it fits against Tesseract and vision LLMs, the hardware reality, and how to get from a scanned image to structured output you can drop into a spreadsheet.

Why run OCR and Document AI locally

Two forces push document OCR onto your own hardware. The first is privacy and data residency: scanned contracts and invoices carry names, account numbers and signatures, and many businesses are simply not allowed to send that off-box. The second is cost. Cloud Document AI is priced per page, so a workflow that processes tens of thousands of pages a month turns into a recurring bill that scales with your volume forever. A local model is a fixed hardware cost that does not care how many pages you push through it.

• Privacy: documents never leave your machine or VPC, which clears most data-residency and compliance hurdles outright.
• Cost: no per-page fee, so high-volume batch jobs stop being a metered expense.
• Latency and availability: no network round-trip and no vendor rate limits on a busy day.
• Reproducibility: a pinned local model gives identical output for the same input, which a versioned cloud endpoint does not guarantee.

What baidu Unlimited-OCR actually is

Unlimited-OCR is a 3-billion-parameter Mixture-of-Experts model with only about 500 million parameters active per token, released by baidu under the MIT license. Its headline trick is in the name: it is built to parse long documents in a single pass. Instead of the usual decoder attention, whose memory cost climbs with every page, it uses Reference Sliding Window Attention (R-SWA) that keeps the KV cache flat. That is why it can read dozens of pages, or a whole PDF, in one inference call without the memory blowing up the way a long context normally would.

It outputs a markdown-style layout with grounding: detected regions come back tagged with a type and a bounding box, for example a title or a table cell with its pixel coordinates. That means you get both the text and where it sat on the page, which is what makes structured extraction from forms and invoices possible rather than just a flat text dump.

Where it sits: vs Tesseract and vs vision LLMs

Unlimited-OCR lives between the two tools people already reach for. Classic OCR like Tesseract is fast and runs on a potato, but it reads characters without understanding layout, so it falls apart on multi-column pages, tables and forms. A general vision LLM like a Gemma vision model understands layout and can answer questions about a page, but it is a generalist that costs far more memory and is slower for bulk page extraction. Unlimited-OCR is the document specialist: it understands structure like a vision LLM but is purpose-built and cheap enough to batch.

Hardware: does it need a GPU?

At 3B total parameters with only ~500M active, this is a small model by 2026 standards, which is the whole point. A 4-bit quant of the weights is a few gigabytes, so it fits an 8 GB consumer GPU comfortably and leaves room for the image and context. It will also run on CPU, just slower per page, which is fine for an overnight batch job but painful for interactive use. There is also an NVFP4 variant for NVIDIA Blackwell cards (the RTX 50-series) if you want the faster low-precision format.

Get the model and a runner

The model card lists support for Hugging Face Transformers, vLLM, SGLang and Docker Model Runner. The simplest first path is plain Transformers for a one-off script; switch to vLLM once you want to serve it as an endpoint and batch. Pull the weights from Hugging Face and install the inference dependencies the model card pins (Transformers, Pillow for images, PyMuPDF for PDFs).

First run: a scanned image and a scanned PDF to text

Loading follows the standard pattern for a custom-code multimodal model: bfloat16 weights, safetensors, trust_remote_code on. The prompt embeds an image placeholder and a task instruction. For plain extraction the task is simply document parsing, and the model returns the page as grounded markdown.

import torch
from transformers import AutoModel, AutoTokenizer

MODEL = "baidu/Unlimited-OCR"

tokenizer = AutoTokenizer.from_pretrained(MODEL, trust_remote_code=True)
model = AutoModel.from_pretrained(
    MODEL,
    trust_remote_code=True,
    use_safetensors=True,
    torch_dtype=torch.bfloat16,
).eval().cuda()  # drop .cuda() to run on CPU

# Task prompt; the image placeholder is filled by the inference helper.
prompt = "<image>\ndocument parsing."

# Follow the README for the exact helper name/signature.
result = model.infer(
    tokenizer,
    prompt=prompt,
    image_file="invoice_scan.png",
    image_size=1024,        # Base mode for multi-page throughput
    max_length=32768,
)
print(result)

For a scanned PDF, render each page to an image first with PyMuPDF, then feed the pages. Because the model holds the KV cache flat, you can pass a whole multi-page document in one call rather than looping page by page, which is the feature the long-horizon design exists to deliver.

import fitz  # PyMuPDF

def pdf_to_images(path, dpi=200):
    doc = fitz.open(path)
    paths = []
    for i, page in enumerate(doc):
        # Higher DPI = sharper input for OCR (see accuracy tips below).
        pix = page.get_pixmap(dpi=dpi)
        out = f"page_{i:03d}.png"
        pix.save(out)
        paths.append(out)
    return paths

pages = pdf_to_images("contract.pdf", dpi=200)
# Pass the page list to the model in a single call (Base mode, 1024).

Structured extraction: tables, forms and key-value pairs

Plain text is the easy half. The reason to use a document model over Tesseract is that the output is grounded: each region comes back with a type and a bounding box, so a table stays a table and a labelled field keeps its label. A detected region looks roughly like this, with the type tag, pixel coordinates, then the text:

<|det|>title [37, 64, 464, 130]<|/det|>INVOICE #2026-0623
<|det|>text [40, 150, 300, 180]<|/det|>Bill To: Acme Holdings Ltd
<|det|>table [36, 220, 760, 540]<|/det|>| Item | Qty | Unit | Total |
| Widget A | 12 | 4.00 | 48.00 |
| Widget B |  3 | 9.50 | 28.50 |

Because tables come back as markdown and fields keep their position, turning a parsed invoice into key-value pairs is a parsing step, not another model call. Pull the table block, split the rows, and map the labelled lines (Bill To, Invoice #, Total) into a dict. The bounding boxes let you disambiguate when two fields share a label by their position on the page.

Batch a folder of documents

For a real workflow you point the model at a directory and let it grind. Load the model once, reuse it across files, and write one JSON sidecar per source document. Running this as an overnight job is exactly where the no-per-page-fee economics pay off against a cloud API.

import json, pathlib

IN_DIR = pathlib.Path("inbox")
OUT_DIR = pathlib.Path("parsed")
OUT_DIR.mkdir(exist_ok=True)

for src in sorted(IN_DIR.glob("*.pdf")):
    pages = pdf_to_images(str(src), dpi=200)
    text = model.infer(
        tokenizer,
        prompt="<image>\ndocument parsing.",
        image_file=pages,          # whole document, one pass
        image_size=1024,
        max_length=32768,
    )
    out = OUT_DIR / f"{src.stem}.json"
    out.write_text(json.dumps({"source": src.name, "markdown": text}, indent=2))
    print(f"parsed {src.name} -> {out.name}")

Accuracy tips: DPI, preprocessing and language

OCR quality is decided before the model ever runs. The single biggest lever is input resolution: render scans and PDF pages at 200 to 300 DPI, because a model cannot recover characters that were never captured. A few cheap preprocessing steps clean up the rest.

• Render at 200-300 DPI. Below ~150 DPI, small print and table digits start to disappear and no model recovers them.
• Deskew tilted scans. A page rotated even a few degrees hurts line detection; auto-rotate before parsing.
• Convert to grayscale and bump contrast for faded receipts or thermal-printer paper.
• Crop away large blank margins so more of the model's resolution budget lands on the actual text.
• For dense single pages with tiny print, switch from Base to Gundam dynamic resolution.

Wire the output into a spreadsheet or database

The end of the pipeline is turning grounded markdown into rows. Parse the table and key-value lines into a flat dict per document, then append to a CSV for a spreadsheet or insert into a database. Keeping the raw markdown alongside the structured fields means you can re-parse later without re-running the model.

import csv, json, pathlib

rows = []
for f in pathlib.Path("parsed").glob("*.json"):
    doc = json.loads(f.read_text())
    md = doc["markdown"]
    # Map the labelled lines you care about into columns.
    record = {
        "source": doc["source"],
        "invoice_no": extract_field(md, "INVOICE #"),
        "bill_to": extract_field(md, "Bill To:"),
        "total": extract_total(md),
    }
    rows.append(record)

with open("invoices.csv", "w", newline="") as fh:
    w = csv.DictWriter(fh, fieldnames=["source", "invoice_no", "bill_to", "total"])
    w.writeheader()
    w.writerows(rows)

Cost and privacy vs paid Document AI

Cloud Document AI from Azure and Google is metered per page, so the cost grows linearly with volume and never stops. A self-hosted model flips that into a fixed cost: you pay once for hardware (or rent a GPU by the hour) and the marginal cost per page is electricity. The crossover depends entirely on volume. For a few hundred pages a month a cloud API is cheaper and zero-maintenance. For tens or hundreds of thousands of pages, or for any workload that legally cannot leave your network, self-hosting wins on both axes at once.

Limits and when to fall back to a vision LLM

Unlimited-OCR is a parser, not a reasoner. It transcribes and lays out a document; it does not answer questions about it, summarise it, or judge whether an invoice is fraudulent. When your task is reasoning over the content rather than extracting it, hand the parsed markdown to a general vision LLM or a text model. Handwriting, extreme low resolution, and heavily stylised layouts also remain hard for any OCR, so keep a human-review step for the documents where a wrong digit actually costs money.