LightOnOCR-3-4B (GGUF)
GGUF quantizations of lightonai/LightOnOCR-3-4B — LightOn's most accurate LightOnOCR-3 model (Qwen3.5 vision-language architecture): end-to-end OCR with layout and visual understanding — optimized for local inference with llama.cpp and Ollama, with its vision projector. All quants use an importance matrix.
Quick Start
With Ollama
ollama run richardyoung/lightonocr-3-4b
# or straight from Hugging Face
ollama run hf.co/richardyoung/LightOnOCR-3-4B-GGUF:Q4_K_M
With llama.cpp
huggingface-cli download richardyoung/LightOnOCR-3-4B-GGUF \
--include "*Q4_K_M*" "mmproj*" --local-dir ./models
./llama-mtmd-cli -m ./models/LightOnOCR-3-4B-Q4_K_M.gguf \
--mmproj ./models/mmproj-LightOnOCR-3-4B-F16.gguf \
--image page.png -p "grounding"
Prompts: send the image with an empty prompt for plain transcription (Markdown), or with the prompt grounding to get every block with a label and a 0–1000 bounding box, image descriptions and chart data as tables. LightOn notes that other instructions are out of distribution. Recommended sampling: temperature 0.2, top_p 0.9 (set in the Ollama tags).
Why This Model?
- Most accurate of the LightOnOCR-3 family — LightOn's recommendation for most OCR tasks
- Two modes: plain transcription, or
groundingwith labels and bounding boxes for every block - Visual understanding: short descriptions for images, charts turned into tables of their data
- Tables, receipts, forms, multi-column layouts and math notation
- Apache-2.0: research and commercial use
Use Cases
- Digitizing scanned documents and PDFs locally
- Layout-aware extraction with bounding boxes
- Chart and table data extraction for RAG
Available Quantizations
| File | Size | Use Case |
|---|---|---|
LightOnOCR-3-4B-IQ3_M.gguf |
2.2 GB | Smallest |
LightOnOCR-3-4B-IQ4_XS.gguf |
2.5 GB | Great quality-size balance |
LightOnOCR-3-4B-Q4_K_M.gguf |
2.7 GB | Recommended — good balance |
LightOnOCR-3-4B-Q5_K_M.gguf |
3.1 GB | Higher quality |
LightOnOCR-3-4B-Q6_K.gguf |
3.5 GB | Very high quality |
LightOnOCR-3-4B-Q8_0.gguf |
4.5 GB | Near-lossless |
mmproj-LightOnOCR-3-4B-F16.gguf |
0.67 GB | Vision projector (required for images) |
LightOnOCR-3-4B-imatrix.gguf |
– | Importance matrix used for the quants (calibration_datav3) |
Verified
Tested through Ollama before upload (Q4_K_M and Q8_0) on a synthetic invoice with an empty prompt; Q4_K_M returned: 'INVOICE 4471 / Date: 2026-09-28 / Widgets x3 ....... $42.00 / TOTAL: $86.20'. Converted with --no-mtp: the upstream config declares a multi-token-prediction layer whose weights are not in the release.
Also Available
- LightOnOCR-3-1B — LightOnOCR-2 architecture, drop-in upgrade
- LightOnOCR-3-0.8B — fastest
Other Models by richardyoung
- New Heretic models (reproducible, listed in the Heretic Grimoire): LFM2.5-1.2B | Granite-4.2-3B | Granite-4.2-8B | Olmo-3-7B-Think | Phi-4-mini | DeepSeek-R1-0528-Qwen3-8B | Agents-A1-4B | FrogNano-4B
- Abliterated/Uncensored models: Qwen2.5-7B | Qwen3-14B | DeepSeek-R1-32B | Qwen3-8B
- OCR & Vision: MiniCPM-V-4.6 GGUF | dots.mocr GGUF | Unlimited-OCR GGUF | olmOCR GGUF | LightOnOCR-3-4B GGUF | WeVisDoc-4B GGUF | Xiaomi-OCR-0 GGUF
- MLX quantizations (Apple Silicon): Kimi-K2 series | olmOCR MLX
- Healthcare/Medical: Synthea 575K patients dataset | CardioEmbed
- Research: LLM Instruction-Following Evaluation (arxiv:2510.18892)
Built & maintained by Richard Young · DeepNeuro